A Note from Jenn Chan

Discussion in 'News and Announcements' started by JChan, Jun 9, 2020.

Thread Status:
Not open for further replies.
  1. JChan Developer

    Hail Adventurers,

    (TL;DR version at the bottom.)

    I'm writing to you today to apologize for all the issues that we've been experiencing with EQII these past few months. We've been moving at a breakneck speed to grow into our newly formed Darkpaw studio and ensure that Norrath is around for many more years to come. Unfortunately, in our haste our "raid wiped". We've failed ourselves and we've sadly failed you, our dear friends, but we've reviewed our parses and we have new strats.

    The following are not excuses. We know that we have lost some trust and that we need to work to regain that. As some of you have noticed, we're trying to be much more transparent and in doing that, let me give you some information about a few issues that we have encountered in the past few months so we can move forward together on the path to great things.

    First off, I'd like to address the unexpected down time in the last week of May. There was a major outage that affected all of our servers, our forums, and numerous other systems in our infrastructure. In accommodating our growing population and addressing current bug issues, we have been incorporating some new hardware. Unfortunately, the transition with the new hardware did not go as smoothly as planned, which brought on many unforeseen difficulties that we have overcome and a few we are still battling. This took out most of our ability to diagnose the core issue and also took out our ability to use our regular channels to communicate the outage. Once the issue was found and resolved, we focused our efforts to bring up the game servers at the time cost of fully bringing up our diagnostic tools. By not having these internal tools up, we failed to see another growing issue that later caused continued login issues throughout the rest of the night. We have taken measures to ensure the initial issue will not happen again.

    In the matter of our recent individual world stability and lag issues, we are currently in the midst of refreshing a number of our systems and are working with our partners to build out more regular maintenance plans so that issues like these are minimized in the future. Earlier this year we had completed a number of migrations ranging from as large as total data center moves to as small as replacing memory on a number of our systems. These conversions and moves are critical to the future health of our infrastructure and unfortunately some of these migrations caused unexpected outages. Specifically, for performance, we have already migrated our database to a faster solution, improved the way we spread out loads, and upgraded some of the underlying operating environments. Continuing our lag mitigation plans, we are working on upgrading the underlying tool set and are identifying areas where we can upgrade the hardware. Additionally, we have been adding more diagnostic systems to better analyze where our actual bottlenecks are. We've also pulled earlier into the schedule the adoption of some performance profiling tools that were originally set for late this year / early next year. As you can see here, we have completed a number of lag mitigation projects and still have a number of projects that are going to take some time to complete.

    Regarding our overall quality levels, this has been a regular battle that we've had to fight many times over. Thank you to the players that have so graciously volunteered their time to assist us over the years on our test server, in betas, and in reporting issues on our forums and in /bug. With the conversion to the bimonthly update schedule we hoped to stem the more reactionary style changes and get the time to put in more complete fixes to issues. We're still learning how to work at that cadence and will be implementing more changes internally that can better take advantage of the higher thresholds for release. We are also looking into ways we can get content, performance improvements, etc. onto the Test Server earlier and reduce changes going to live in a broken state. This is in order to keep quality at the high standards we as developers and you as players expect, while also adhering to the new work environment of social distancing or working from home. Many of our hardworking developers and support staff (ex: QA, Operations, Platform) have been voluntarily working an increased number of hours to make sure our schedules don’t slip and that most bugs don't reach live. These are strange times and we are prone to human error. Frankly put, we've been making silly mistakes that we wouldn't have been making a matter of months ago. Not to fret, as I am confident in my team who have been very agile in doing everything they can to fix problems as they arise.

    So going forward, we need time to work on the solutions that I've touched on above. I'm asking you to continue to have patience with us and to continue to work with us in these trying times. We're going to complete and release some projects that we feel are almost done and pause for a bit to catch our breath and analyze the pain points from a holistic view. I've asked the team to double down on our current efforts in stability, performance, and quality. As a result, some of you may notice a lack of new things in-game for a bit while we focus on these issues. This doesn't mean that our normally scheduled annual plans at the end of the year are at risk, just that there will be a pause for a bit. In the meantime, please accept our apology and this Samenda Honorhoof (mount) which you can get by logging into the game between June 9 at 12 PM PDT to June 24 at 12 PM PDT.. I believe that with your help we can make it out of these challenging times and can once again enter a mythical era.

    Never give up!
    Jenn Chan

    TL;DR:
    I apologize for the issues that we've been experiencing. We've encountered large and unexpected problems and we've fixed some of them. We need more time to work on fixing the rest. Please be patient. Things will get better. We're giving you a mount. Never give up!
    Syrdeth, Siren, Khaticia and 40 others like this.
  2. Dude Well-Known Member

    Thank you for the update. It's good to know what's going on behind the scenes.
  3. Gojiria Member

    This kind of transparency is refreshing. Thank you VERY much for the update.

    NOCH WEITER!!! (EVER ONWARD!!!)
  4. Wulfgyr I've got friends in EQ2Wire places

    To echo Dude, thanks for the update. I must admit to being (pleasantly) surprised by a second note from you so quickly. This is a change for the better, and a trend that I hope continues moving forward!
  5. Marae Well-Known Member

    Yes, thank you very much for the update! I am encouraged not only by the message, but by the efforts you all are making to keep us entertained during these strange and trying times. I am very grateful to all the folks who are keeping EQ2 going. It's a very important part of what's keeping me sane.
  6. dorotea Well-Known Member

    Communications were not a strength with SoE and then Daybreak. They were a major weakness.

    Things have been much better in recent years - under Daybreak not just after the Darkpaw reorganization. Being more open about things doesn't entirely make up for things being in a troubled state but it wins a lot of goodwill and gives us more hope for improvements.

    Thanks.
  7. Gregore Active Member

    Thanks for the update Jchan. I've been playing since launch, and have thoroughly enjoyed the game until recently. You're right, a lot of us have lost faith in your team & are extremely frustrated with all of the broken content & such. Both of my subs expire next month. You basically have till then to win me back. I have already found a "replacement" game to play in EQ2's place, and have been enjoying it so much that I only log in to raid as it is for basically the last two months or so. I've invested a lot of time, effort, & RL money into this game, but have one leg out the door ATM. The ball is in your court.....
    Juraiya and Tilweani like this.
  8. Wackelpudding New Member

    You are doing a great job. I enjoy EQ2 now more than ever. It is understood there will be hiccups now and then. Thanks for working hard.
  9. Pone the Wonderhorse Member

    Mount is not available on TLE?
  10. Bhayar Well-Known Member

    Thank you for taking the time to give us an update. Looking forward to seeing what you and team roll out in the future. There's a lot of information to digest in what you've put forward for us at this time and a glimpse of your working style. Best of luck!!
  11. Thatdarncat Active Member

    I completely agree. I've been playing since beta. The later years under SOE, right before the sale, were the nadir for this game. It took a couple of years under Daybreak for things to improve but once they did that improvement continued. I've bought and played every expansion to date and Blood of Luclin is the one I've enjoyed the most since Sentinel's Fate all the way back in 2010. I've played more hours of EQ2 since BoL launched than I have for years.

    Keep on going the way you have been and all will be golden, hiccups and silly mistakes and all! The more open communication is very welcome, too.
  12. dreamweaver Well-Known Member

    There was an issue on Live and the TLE they are looking into it.
  13. Breanna Well-Known Member

    Thank you for the update of what's been going on. We do so appreciate the communication, it helps a lot. Please keep up the good work, and I hope it does not take you long to get things straightened out. I love this game and want to continue playing. I too have been here since pretty much launch, I came in about 4 months behind everyone else :) But close enough, it's been a long time.
  14. amarisa Member

    Good to know what is going on. Thank you for updating the community! :)
  15. dreamweaver Well-Known Member

    Should be showing up now.
  16. Benito Ancient EQ2 Player: Lavastorm Server 2004.

    Thank you for the communication!

    Lovely mount!

    (Mount pictured in blue fire and grayish armor accents!)
    [IMG]
  17. The Martian Active Member

    Thank you. Great to see information flowing outward from what use to be a dark box. (Is that a light at the end of the tunnel I see?) Thanks to you and everybody else for the work you have been putting in, and what you have to do going forward!
  18. Vibrid33 Member

    Honestly the fact that you're even acknowledging this really means alot. Playing alot of bigger games recently that are suffering similar server and technical problems and we get dead silence in response. I think i speak for most of us when I say that we understand that you just formed a new studio and are having to work from home and that we wish all of you the best :)
  19. Zorkess New Member

    Thanks for the update. Simply comms is all that is needed to keep the population happy. Its the not knowing what your doing to fix the problems that players suitable get upset about. Its an honest and true note that you have made Jchan and love the honesty given. just keep it coming and listen to the players , its them that can truely make the game fun as well as imagineers in Daybreak.
  20. Kenn Well-Known Member

    Even if it takes a year of ups and downs to have something great in the end, it is much better than leaving in the state that it is in. I have been around since launch and do not need a quick fix, I'll wait for awesomeness.
Thread Status:
Not open for further replies.