Why 24 hour downtime in July '21?

Discussion in 'The Veterans' Lounge' started by Metanis, Jun 22, 2021.

  1. Metanis Bad Company

    I know the announcement is sort of written in such a way that it seems to link the Selos/Povar merger as the reason that all servers are going down for 24 hours... but that doesn't pass the smell test.

    Distributed systems have been a thing for decades now, so why should all the servers have to come down just to merge two nearly dead servers?

    Must be more going on that hasn't been announced yet? I mean if they need 24 hours then have at it, but I'm just curious why it was included seemingly as an afterthought? I would think a 24 hour downtime would warrant an announcement all by itself so people can plan ahead.
    Gyurika Godofwar and Fenthen like this.
  2. Cloud the Third Augur

    They are going to fix my rat spawn.
  3. kizant Augur

    They have to make updates.
    Fenthen likes this.
  4. Leerah Augur

    Since Selos is already empty, this isn't going to help Povar one bit. While you're at it, give us two or three more servers with low population, please. Make a real difference. Do it now and you won't have to do it again for another five years hopefully.
    Thundersnake likes this.
  5. Cloud the Third Augur

    We need a dead horse server where it has random mounts that run around and you can chase and beat with bats for a chance at random loot.
    Winnowyl, Coagagin, cadres and 2 others like this.
  6. Herf Augur

    Server merges no doubt mean that a master database containing player IDs and their toons need to merge. That master DB probably also controls the other servers. I don't see why this has to take 24 hours though since the data/names already exists; they could just shut down new character creation for a day or two, figure out name merge issues, then bring the DB down and quickly run the merge. For instance.
    Fenthen likes this.
  7. Qulo New Member

    Possible but poor design if that is the case. I would think they have people smart enough to not have a problem like that and the servers to be independent of each other. It does keep track of when you are logged into one server so you can't log into another but if done right all that means is a global value says you are logged into server A so if you try to log into server B it is rejected. With 2 servers set to down this shouldn't affect that process.

    If you are on server A and there is some kind of global value you need those would all should be stored outside of server A's environment. Now if you needed to change the global DB that would force all those servers that need it to go down but if you are only changing server A DB to merge with server B DB and it was designed properly it really should not need a global downtime. That doesn't mean they aren't doing it anyway just to be on the safe side or something or something was just designed bad long ago and was never fixed requiring it.

    It could be something as simple as the server list is hard coded and they can't take down 2 servers and merge them and get a single new merged server back in the server list without all of them going down. We all have seen single servers go down without others being down so we know they can at least take down 1 server without taking others down.
  8. FranktheBank Augur

    Did you miss the last merger? Also, the answer to your question "Because EQ is old af and held together by duct tape and string.

    Or Merge Povar into another low or med pop server.
    Herf and Leerah like this.
  9. Fenthen aka Rath

    I am really hoping they wrap this into the July patch as well? Or are we expecting multiple majority-of-the-day downtimes?

    There's really zero sense in bringing all EQ servers down for 2 servers merging. Why this wasn't addressed several years ago is beyond me, beyond any of us really.
    Leerah and Gyurika Godofwar like this.
  10. Tappin Augur

    Because they are bound to break something? Last time they gave us infinite corpse to Rez and missing characters. Cheers to see what else is broken.
    Leerah, Fenthen and Dythan like this.
  11. Fenthen aka Rath

    Ahh, I had missed that part at the end of the paragraph. No need to be a diick about it though.
    Leerah and Dythan like this.
  12. Moege Augur

    Missing bank/housing was fun!
  13. Svann2 The Magnificent

    Expect your new name - Fenthethen
    Tatanka likes this.
  14. niente Developer

    24 hour downtime is planned, but I hope it will be much shorter this year. Server merges are one of the most dangerous things that can be done to the game.

    To give a little more history / commentary,

    Yes, EQ is a distributed system, and although distributed computing systems existed in the 70's, I feel like EQ is one of the great early examples of distributed systems in online gaming. What EQ accomplished in 1999 was incredible for it's time.

    Why can't we just take down 2 servers and merge them?
    That would be great, but not currently possible due to our DB structure. All of the EQ servers must be down for us to run SQL on the live DBs, which are shared among all servers. Running server merge SQL while the live game is up would be disruptive to game performance and cause potential additional issues, especially if we needed to do something like a DB rollback. We do not generally run SQL on the live DBs unless it is an emergency. This is an architectural bottleneck that would be great not to have, but would also be another huge pile of work to change.

    EQ did not have a database for characters until ~2016. Before that characters were stored in flat files. I do not know how servers were merged before then - I am going to take a wild guess that scripted importing of all characters from one server to another were done. It sounds like it was a monumental task to do so, and if you enjoy reading old EQ stuff, you might enjoy reading some of the threads about problems that occurred after server merges that took place in the past (and were done by programmers with a lot more talent and experience than I have).

    After 2016, server merge SQL was written, but not tested until the Vulak/Fippy merge in 2017. There were a lot of issues with character name collisions in that merge, through no fault of the programmer who handled the merge (and with < 1 year of experience). I don't think anyone on the EQ team at that time realized how complex merging EQ servers would be.

    Improvements to the server merge SQL were made and another merge was done in 2020 (actually, 4 merges - Lockjaw to Ragefire, Trakanon to Vox, Fippy to Vox, and Brekt to Firiona Vie), but it wasn't enough. This year we are taking more time to prepare beforehand and I hope the server merges will be much faster and result in much fewer issues. Last year the code team spent several months before the merge making architectural changes to EQ that would make server merges easier to do (how we store character names in the DB). This year more time is being taken to make the server merge a success.

    Trust me that no one wants the server merge to be fast and painless as we do on the EQ team, and a lot of work is going into making sure things work correctly. We appreciate all the patience the EQ community has with those of us working on this very big, complicated, special game.
  15. Nennius Curmudgeon

    Thank you. A clear explanation. Much appreciated.
    Thundersnake, Herf, Metanis and 2 others like this.
  16. Pops Racer Olde Faht

    Niente - thanks for the full explanation. We appreciate your efforts. Maintaining a large, complex system as "aged" as EQ is no small feat.

    The time you spend here in the forums, answering our questions and providing information is far superior to the early days (1999-2002) when communication was almost nil, especially regarding technical details. Thank you again.
    Thundersnake, Herf and Metanis like this.
  17. Loratex The Ridiculous Necro

    Are people really that upset the game won’t be on for 24 hours? How did you guys survive the Sony hack?
  18. Zanarnar Augur

    Its the difference between something that happens to you, and something you start. People tend to be more sympathetic to the former.
  19. NecRomanceHer New Member


    Instead of doing this, let me give an alternate proposal.
    My understanding is that Selo has like 50-100 active members.
    Can you give those members a free Server transfer token for each character?

    And then say "Move the character to any whatever server you want", And if that is not finished by July 21, then move them to Povar.

    This may miss guilds and shared banks? Maybe there is a fix to that, I am not familiar with these transfers.

    Then after everyone is moved you can decommission the server, or just have it show as "Locked" until it can be removed.
    Coagagin likes this.
  20. Atomos Augur


    At first, I laughed out loud, really hard. But quite honestly, this is one of the most depressing things I've ever read on here. Kinda don't even know what to say.

    Anyway, thank God a more modern (can we even call databases modern) method was converted to. Let's hope those kinds of conversions continue, regardless of whether it's a "huge pile of work" or not. It will be worth it in the end.
    Herf likes this.