Server Downtime: Tuesday, May 19th @ 10:00 AM

Discussion in 'News and Announcements' started by dreamweaver, May 14, 2020.

Thread Status:
Not open for further replies.
  1. dreamweaver Community Manager

    All EverQuest Live Servers will be brought offline on Tuesday, May 19, 2020 at 10:00 AM PT* for an update. Downtime is expected to last approximately 24 hours.

    This additional downtime will combine both the downtime needed for the live patch and server merges.

    Patch notes for this update will be found in the Game Update Notes section of the forums once they are available.

    *Convert to your local time
    Vwlss Nvwls and Quaux like this.
  2. dreamweaver Community Manager

    Current server downtime has been extended 6 hours to 4 PM PDT. We will update that ETA if there is more news.
    Quaux, Jein-ko, Magneress and 5 others like this.
  3. dreamweaver Community Manager

    Current server downtime has been extended 6 hours to 10 PM PDT.

    We wanted to provide some additional background for this extension and the prolonged downtime for you.

    We started the server merge and the first merge test was done by 2 PM yesterday.

    Unfortunately we are not able to test these changes on live servers before the changes occur, we started running checks and things were progressing as normal. Around that time we started to run into issues with real estate, in addition to that, there were problems with deleted characters that were causing the script to allow duplicate player names.

    Resurrecting older tech is fraught with surprise dragons, and this tech was resurrected when we merged Vulak to Fippy, which if you played at the time you recall the real estate issues. The goal this time around was to make sure that issue would not be repeated.

    However at 2 PM on 5/19 we had to restore data for all players on our merged servers, we fixed the discovered issues in the merge process and started them again only to discover additional issues revealed by the fix.

    I want to reassure you, in some way, shape, or form someone from the ops and engineering team have been working to solve these issues since 10 AM yesterday. This is our top priority, and all hands are on deck until it is solved.

    The specific reason for this extension is that we have finalized checks on a successful merge at 2 PM PDT. We can now move forward with the other merges and the live patch, which is what is accounted for in this extension.

    At this point we are unable to stop the server merges and just bring everything up because of several threads that have already begun their processes and a full rollback of those would take significantly longer than the time extension.

    We know that you are all eager to play and thank you for your understanding and patience while we continue the process.
    Kildaill, Marquele, Roxas MM and 74 others like this.
  4. dreamweaver Community Manager

    UPDATE - We are extending the current downtime, this is unrelated to the server merge process as it is now complete. It is related to the normal live patch and as such has no current ETA while the team investigates the issues.

    I will update with more information as I have some.
    Marquele, Velikovsky, Bahdah and 11 others like this.
  5. dreamweaver Community Manager

    UPDATE - During the live patch we've run into multiple issues with some of the new hardware, those are in the process of being tracked down and solved but at this time there is no ETA.

    *Pictured below: The perfect storm.

    Indar, Magneress, minimind and 10 others like this.
  6. dreamweaver Community Manager

    UPDATE - The downtime has concluded and servers are unlocked. Thank you for your patience, please be aware of the following:

    - UCHAT (global chat channels, friends, ignore lists, and mail) will be unavailable on Vox, Ragefire, and Firiona Vie until Noon (12 PM PDT) we are working on some remaining migrations for player's friends, ignore lists, and mail. We'll update as we know more.

    - Please note that some shared banks on merged servers experienced issues playing with each other and we are working on getting your toys back. They will be returned to your /overflow items list and we will update this thread when we have completed this. Thank you and we apologize for any squawking this may have caused.
  7. dreamweaver Community Manager

    [UPDATE] We're about 99.9% complete with the UCHAT migrations (friends, ignore lists, and mail) and about half way complete with the shared bank item restores to /itemoverflow. Will continue to update with progress and further news.
    Barton, Yinla and Mithra like this.
  8. dreamweaver Community Manager

    [UPDATE] Items from shared banks are now all in their respective item overflow lists and everything over 10K plat has been parceled to the first character on your account alphabetically.
    Mithra likes this.
  9. dreamweaver Community Manager

    [UPDATE] We apologize sincerely for the loss of items and platinum from your shared banks. It's an incredibly large amount of lost items, especially as multiple servers were involved. We realize getting them now, or still missing them, is not a great experience from a player perspective, but the team has been working day (and night) to get the servers up, and everything back to you as soon as possible.

    The lost items were definitely not intended, and was one of many problems that occurred during the server merges that we attempted to solve as quickly as possible to get the servers open again. Shared banks in particular have already had corrupted data in the past, which worsened the problem. It is no excuse for the damage that was done. We have plans to improve the server merge process so something like this doesn't happen again.

    We will be taking another look at shared banks via backup DB tables next week. If you are still missing things, please let us know your character name and server in the thread linked below. Thanks for your patience.
    Yinla, Zataos and Nennius like this.
Thread Status:
Not open for further replies.