Server Performance Issues

Discussion in 'Official News and Announcements' started by ARCHIVED-Rothgar, Sep 23, 2009.

  1. ARCHIVED-Rothgar Guest

    A couple of the highly populated servers are seeing some performance problems now that we are reaching primetime and many new players are logging in.
    This is probably due to the large number of things that happen to your character the first time you log in after GU53. Until the performance comes back down to normal levels, we will be locking some of the servers to keep the problem from getting any worse.
    Currently the issues seem to be happening on AB, Nagafen and Crushbone.
    The locking will only be done to spread the load over time and not something we expect to do for very long.
    I'll update this thread when I have more info.
  2. ARCHIVED-Rothgar Guest

    Crushbone and AB are back online and we'll be monitoring their performance.
  3. ARCHIVED-Amnerys Guest

    AB and Guk are experiencing issues and have been locked temporarily.
  4. ARCHIVED-Rothgar Guest

    All servers are currently unlocked but some are experiencing database bottlenecks. We are working on determining the cause. GU53 has been live since early this morning and these isses only just began within the last 2 hours. So our first assumption is that its just related to the increased load of prime time.
  5. ARCHIVED-Rothgar Guest

    Servers seem to be stable at this point although some of them have response times a little slower than we'd like. We'll continue to monitor them.
    I apologize if you experienced any downtime tonight.
  6. ARCHIVED-Kiara Guest

    Greetings all!
    Here's an update on the performance problems we've been seeing.
    We believe that the issues the servers have been experiencing since Shards of Destiny was pushed live relate to the volume of items being converted to heirloom and the achievement information being accessed by the new system. This has stressed the database and is causing the performance issues. We expect it to abate in a couple of days after the bulk of our players get logged in and updated.
    Once again, we really appreciate your patience and apologize for the inconvenience caused.
  7. ARCHIVED-Rothgar Guest

    AB crashed under the database load a few minutes ago. Its back up and unlocked. However, the problems could reoccur since we haven't actually updated any software yet.
    Our QA department is testing a couple of fixes to make sure they don't cause any other problems. If that passes we will hotfix the servers asap. We will also most likely disable house layout saving and loading as we think that could be adding to the load.
  8. ARCHIVED-Kiara Guest

    Update:
    The team has a fix in for tomorrow morning that we hope fixes the issue.
    Thank you again for your patience.
  9. ARCHIVED-Brenlo Guest

    T'aint the hamster my friends. Although you are working the poor guys butt off for sure. Too many DataBase calls! AB has such a large population and there are a lot of DB updates happening that the server is timing out. Achievements, House layout saves, heirloom item updates and even the prelude event are all hammering the database like Gnobrin hammers coffee. We have some fixes scheduled for a morning hotfix that should help, and Rothgar and a couple other folks are working on some things to try and help tonight too. Unfortunatley one of those options may be to disable the prelude event on AB. As she always does, Kiara will keep you updated.
    Thanks for your patience folks. I know you want to play, we want you to be playing. We will get this sorted out as quickly as we can.
  10. ARCHIVED-Rothgar Guest

    Until we resolve the database issues, we are going to temporarily disable the spire-building event on affected servers. At the moment this is just AB.
  11. ARCHIVED-Rothgar Guest

    Crushbone is also experiencing some database issues, so I've disabled the spire event temporarily.
  12. ARCHIVED-Rothgar Guest

    Nagafen is also experiencing some issues so I've just disabled the spire event and locked the server until the database recovers.
  13. ARCHIVED-Rothgar Guest

    Nagafen appears to have recovered (for now) and has been unlocked. The spire event continues to be disabled.
  14. ARCHIVED-Rothgar Guest

    I'm reactivating the spire event on all US servers.