[OFFICIAL] Server Issues (Oct. 18 2012)

[bambam922]

We are aware that many (if not all) users are experiencing issues across all of our products.
There are issues with the Auth and Database servers. The problems that you may be experiencing are, but are not limited to:

• Sluggish/Unresponsive Forums
• Failed Authorization
• Max Sessions
• "No Data for X"
• "Cannot Load Tiles"
• Flying just stopping
• Bot afk while "Moving to node"

We are very sorry that there are more errors with our servers causing you to lose botting time.

We are working diligently to resolve these issues.

As of now, there is no ETA on when the servers will be fixed.
This post will be updated as I or the Other Mods/Devs have more information.

UPDATE:

While we shouldn't be having the widespread outage like yesterday, there still MAY be problems.
They can be some of the same ones listed above. this means GB2 has problems collecting nodes and going AFK.
They should be fixed by Monday night.

UPDATE (4:30 PM GMT):

The servers are back up for now, and we've finally tracked down the root of the issue.

Non-technical answer: There is a bug in a 3rd party library we use that causes all SQL connections to randomly stop, which causes our auth servers to be unable to "sync" with eachother, and retrieve data when required. This causes them to basically stop responding due to failsafes to avoid bigger issues (such as melting server hardware).


Technical answer:

We use MySQL's .NET Connector to handle our SQL connection (and queries) on the auth servers. There is currently a bug in the entity framework support that will cause connections to stop working after what seems like a set amount of time, or number of requests to the DB server. We've identified the issue, and are going to make changes today and tomorrow to remove the entity framework from our server software. This should resolve the problem for good, at the cost of some maintenance annoyances. We're not sure of what exactly is the cause in the MySQL connector code-base, but we do know that is where the issue originates. Unfortunately, this is one of those times where relying on a 3rd party implementation (that has been tested to work perfectly) can cause a nightmare. We must be hitting some usage case that hasn't been tested, or found a new bug in the provider.

TL;DR; We found the problem, and we'll be fixing it ASAP. Servers are up for right now, but may go back down again later today. Please bear with us!

 

[Hawker]

Most of these issues should be intermittent. The new load balancer is expected to be configured and live by end of the working day Monday and all should smooth as silk after that.

 

[Apoc]

The auth should be back online for the time being for everyone to continue botting.

Please keep in mind service may drop as we push new versions of the auth software to our servers, to enable more diagnostics. This shouldn't cause any problems for most of you, but there's a small chance you may be booted. I apologize for the inconvenience, but this is a requirement for us to get to the root of the issue.

 

[Apoc]

Down again, looking for the issue right now. Will update once I know more.

 

[Apoc]

Update!

Servers should be back online now for the most part. Things may be slow however.

I'll post later with a better explanation of what's going on, but for now I need to ensure the services are running properly.

Again, sorry folks. We all hate downtime, but sometimes it's unavoidable. (We've been making fairly large infrastructure changes recently to make everything much more stable, but with the servers constantly having issues, we've gotten a bit tied up)

 

[Apoc]

Update again!

The tilemap issues should be resolved now.

Please expect random spurts of errors or a few seconds of downtime every now and then, for at least another 6 hours. We're still resolving the current issues.

 

[Hawker]

The auth server seems to have melted down and is refusing to accept connections. Things will be rough for the next few hours.

 

[Apoc]

The servers are back up for now, and we've finally tracked down the root of the issue.

Non-technical answer: There is a bug in a 3rd party library we use that causes all SQL connections to randomly stop, which causes our auth servers to be unable to "sync" with eachother, and retrieve data when required. This causes them to basically stop responding due to failsafes to avoid bigger issues (such as melting server hardware).


Technical answer:

We use MySQL's .NET Connector to handle our SQL connection (and queries) on the auth servers. There is currently a bug in the entity framework support that will cause connections to stop working after what seems like a set amount of time, or number of requests to the DB server. We've identified the issue, and are going to make changes today and tomorrow to remove the entity framework from our server software. This should resolve the problem for good, at the cost of some maintenance annoyances. We're not sure of what exactly is the cause in the MySQL connector code-base, but we do know that is where the issue originates. Unfortunately, this is one of those times where relying on a 3rd party implementation (that has been tested to work perfectly) can cause a nightmare. We must be hitting some usage case that hasn't been tested, or found a new bug in the provider.

TL;DR; We found the problem, and we'll be fixing it ASAP. Servers are up for right now, but may go back down again later today. Please bear with us!

 

[bambam922]

UPDATE (5:30 CST / 10:30 GMT)

Auth servers look to be back up for the time being.
There may be more outages as we continue into the night and into tomorrow.

Thanks for your patience.