A Note to the EQ Community from Jenn Chan

Discussion in 'News and Announcements' started by JChan, Jun 9, 2020.

  1. JChan Developer

    Hail Adventurers,

    (TL;DR version at the bottom.)

    I'm writing to you today to apologize for all the issues that we've been experiencing with EQ these past few months. We've been moving at a breakneck speed to grow into our newly formed Darkpaw studio and ensure that Norrath is around for many more years to come. Unfortunately, in our haste our "raid wiped." We've failed ourselves and we've sadly failed you, our dear friends, but we've reviewed our parses and we have new strats.

    The following are not excuses. As some of you have noticed, we're trying to be more transparent and in doing that, I’d like to acknowledge and provide you information on some issues that we have encountered in the past few months so we can move forward together on the path to great things.

    First off, I'd like to address the unexpected down times in the last week of May on the day that Aradune and Rizlona launched. Soon after these servers were unlocked, there was a major outage that affected all of our servers, our forums, and numerous other systems in our infrastructure. In accommodating our growing population and addressing current bug issues, we have been incorporating some new hardware. Unfortunately, the transition with the new hardware did not go as smoothly as planned, which brought on many unforeseen difficulties that we have overcome and a few we are still battling. This took out most of our ability to diagnose the core issue and also took out our ability to use our regular channels to communicate the outage. Once the issue was found and resolved, we focused our efforts to bring up the game servers at the time cost of fully bringing up our diagnostic tools. By not having these internal tools up, we failed to see another growing issue that later caused the second cascading crash that took out all of our worlds for the second time that day. We have taken measures to ensure the initial issue will not happen again. As for the second issue, we have taken steps to prevent it from happening; however, we are still combing through and trying to identify the ultimate underlying causes.

    As for the extended downtime on the days of the server merges, you can read about the issues that we encountered at https://forums.daybreakgames.com/eq...er-downtime-tuesday-may-19th-10-00-am.266229/

    You may ask why we needed to do this at the time we did. Simply put, it was to buy us more time to address some bottleneck issues that have caused crashes for all of the servers. For some of the issues that we encountered with the merge, we have already addressed them. For the remaining issues, we will be converting systems to newer infrastructure and identifying and addressing bottlenecks. Specifically addressing the temporary loss of shared bank items and platinum, we have restored all missing items and plat that were lost due to the merge process. You can find the items in your /item overflow window and the platinum will either be in your shared bank or parceled to your first character (alphabetically) on the account.

    Let me address the remaining unplanned outages that have affected all or sometimes a few worlds at a time. Know that we share in your disappointment and appreciate any patience you give us during these times. In the past few months, there was a laundry list of underlying causes and rest assured that we have either implemented the definitive solutions or are in the process of developing those solutions. To give you some more details on a few causes and their resolutions, on May 21, we encountered both the The Rathe and Tunare worlds had crashed. Upon investigation we discovered that character sizes had gotten too large and when we tried to load characters above a certain size, worlds would crash. Our normal warning systems for this unfortunately did not trigger and to resolve this, we increased the capacity of character sizes and performed the emergency update later that day to temporarily address the issue. We are currently in the process of converting this to a more dynamic solution that can handle an order of magnitude larger character sizes.

    In March and April, we experienced a number of crashes across different worlds. These were caused by a few different issues: 1) our client hotfixing technology, 2) in-game raid invites, and 3) infrastructure hardware failures. With the client hotfixing, we had identified the very edge case scenario that caused world crashes and deployed a permanent fix in the April update.

    For the in-game raid invite issues, we had been tracking this issue for more than a year and had completely re-tooled our diagnostic systems and re-wrote large chunks of this system to find the cause of this issue. Unfortunately, in this case, the final diagnostics code that we added did worsen stability but finally clued us in on the smoking gun that showed us the exact situation that had been causing this crash for so long. The fix for this issue was server hotfixed as needed in late April with the fix deployed to the rest of the servers in the May update.

    In the matter of our infrastructure hardware failures, we are currently in the midst of refreshing a number of our systems and are working with our partners to build out more regular maintenance plans so that issues like these are minimized in the future. Earlier this year we had completed a number of hardware migrations ranging from as large as total data center moves to as small as replacing memory on a number of our systems. These conversions and moves are critical to the future health of our infrastructure.

    Regarding our overall quality levels, this has been a regular battle that we've had to fight many times over. Thank you to the players that have so graciously volunteered their time to assist us over the years on our test server, in betas, and in reporting issues on our forums and in /bug. Recently, as our staff has converted to working from home, many of us are now voluntarily working a notably increased number of hours to make sure our schedules don't slip. We needed to convert people to full-time work from home setup to keep people safe. With the increased stress and the fact that we were not set up to handle everyone remotely working full time, our quality levels have been hurt. These are strange times and we are human. Frankly put, we've been making silly mistakes that we wouldn't have been making a matter of months ago. Not to fret, as I am confident in my team who have been very agile in doing everything they can to fix problems as they arise.

    Going forward, we need time to work on the solutions that I've touched on above. I'm asking you to continue to have patience with us and to continue to work with us in these trying times. We're going to complete and release some projects that we feel are almost done and pause for a bit to catch our breath and analyze the pain points from a holistic view. I've asked the team to double down on our current efforts in stability, performance, and quality. As a result, some of you may notice a lack of new things in game for a bit while we focus on these issues. This doesn't mean that our normally scheduled annual plans at the end of the year are at risk. Just that there will be a pause for a bit. In the meantime, please accept our apology and a 25% XP bonus on all servers which we'll be activating in-game starting June 9 at 12 p.m. PT and ending June 17 at the time of the monthly update. We have a great legacy to uphold and I trust that with your help we can make it out of these challenging times and can all share in the raid loot together.

    Never give up!
    Jenn Chan

    TL;DR:

    I apologize for the unprecedented issues that we've been experiencing. We’ve fixed some of them. We need more time to work on fixing the rest. Please be patient. Things will get better. We're giving you a 25% XP bonus. Never give up!
    FubarEQ, eqgamer, Sokki and 71 others like this.
  2. ZenMaster formless, shapeless

    Thank you for the update and communication!

    The bonus is certainly appreciated! :)
    Nixe, Jordis, Whoops and 4 others like this.
  3. Papasears1982 New Member

    people seem to forget about all the stuff going on with COVID. I for one appreciate what you guys have done for us. Take your time and get it done, were all counting on you!

    Ecofriendly
    Nixe, Drathese, Jordis and 6 others like this.
  4. Yinla Ye Ol' Dragon

    Thank you Jenn and team and all stay safe, we need you all!

    HUGS :)
  5. Skuz I am become Wrath, the Destroyer of Worlds.

    Thank you EverQuest team.

    It's been pretty dang horrible for the team & players alike I am sure but I am happy to hear things are getting better.

    Lag seems to be the biggest problem among the servers right now, some have it especially bad so I look forward to seeing things improve in the coming weeks & months.
    Dyna, Adagia, Jordis and 3 others like this.
  6. bobjones1208 Augur

    This type of visibility and accountability is a huge step in the right direction. Thank you for all the hard work behind the scenes and the XP bonus is much appreciated.
    FubarEQ, Dyna, Cazmac and 4 others like this.
  7. Metanis Bad Company

    Some days you get the gold mine, some days you get the shaft.
    Whoops, Corwyhn Lionheart and Skuz like this.
  8. BBQDawg New Member

    Thank You Jenn - You owned up to alot from the past that you were a part of but not responsible for. Very good sign of stewardship for the game we all love and apparently can't do without. I can say I have had an even better in-game experience because of what is happening real life. No arm twisting here to get me to spend more money on a little DBC, I just hope some of it trickles down to where it is needed !! Stay safe and thanks again for the update.
    Jordis and Whoops like this.
  9. Gheed Is not reading your response

    Hey I think you guys did 2500% exp instead of 25%. There is a guy who just went from 45-50 in under two minutes on Aradune. I know its a really easy mistake to make and give people over 10x the EXP they should normally get. Insane, you even manage to screw up an apology.
    Jordis and BBQDawg like this.
  10. Kukaw Augur

    Thank you for the communication on all these issues of late...it goes a long way:)
    Jordis, Whoops, BBQDawg and 1 other person like this.
  11. HicksAradune Augur

    Any chance you can turn the server performance to 2500% too instead of 25%?
    lockjaws likes this.
  12. Zandez Lorekeeper

    Thank you for the update, I'm sure most of the community appreciates your teams hard work and keeping EQ running this long!!! Any update/hints on next Everquest x-pac and or EQ next?
    Jordis, Yinla, Whoops and 1 other person like this.
  13. Corwyhn Lionheart Guild Leader, Lions of the Heart

    Thank you very much for the update. Greatly appreciated.
    Jordis likes this.
  14. Tshesebe Lorekeeper

    Are there any plans on doing anything for Pride Month? Last year DBG gave bunnies. Will there be something similar this year, please?
    Prathun likes this.
  15. Korbie Multipass

    Very much appreciate this level of transparency and your acknowledgement of the player-base o./
    Thank you, Jenn \.o./
  16. Aneuren Tempered Steel

    Excellent.
  17. KimchiGoddess Augur

    Honestly it's very appreciated that you took the time to write such an informative message to us. I have been a critic of how silent I've viewed you as being with the community since you took over. I am hoping this is kind of a new stone being turned for your communication with your playerbase. We love this game. You love this game. We can sometimes be complete jerks about demanding answers but when things are shared in a timely fashion it does help to quell the masses.

    I'm sure with all the issues you've spelled out that this has been an extremely stressful time for you and the rest of the staff. Our demands surely have added to that stress but I'm also sure you can understand why so many people have been upset regarding how things rolled out. It's very unfortunate that all of this came to a head right as two new TLP servers were being rolled out.

    I hope things become more stable going forward and the XP bonus and this community message are certainly a start in reaching out to your fanbase.

    Thanks to you and everyone working hard every day in exceptional circumstances to continue to provide access to the game we have all known and loved for decades. No other MMO has actively held my attention, provided enjoyment and kept me paying to play for as long as EQ has. No other MMO ever will - I've been searching for a long time.
    lockjaws, Xyphen, Drathese and 3 others like this.
  18. Time Burn Augur

    .
    Let EQ be about EQ. Politics, beliefs,10,000 various causes etc - we can all find that outside a video game 24/7.
    Trapper, Borm_of_fv, Medimus and 4 others like this.
  19. Lumiens Augur

    Bonus experience is completely useless to me. How about crediting my account with some membership time or some daybreak cash instead that I can actually use?
    JettaTDI, lockjaws and zevel like this.
  20. Xeris Augur

    I mean, how about they focus on making the servers more stable before adding fun celebratory stuff. All about pride, but the game not lagging and crashing constantly is probably a higher priority