What exactly happens during a patch downtime?

So today's patch will be around 6-8 hours of downtime or so. And the red message says something about maintenance.

But can anyone (hopefully a red name) give actual specifics as to what happens in the 6-8 hours? On the player's end we download a file that takes anywhere from 5-30 seconds depending on connection. What is happening on the game/server side?

Or are you all just binge watching Fallout on Amazon Prime? But then what happened the prior 25 years?

In all seriousness, I am curious what is happening at DBG, what devs are doing, what servers are doing, etc during the 6-8 hours.

I'd like to know who gets tasked with getting out of bed 2 or 3? (or more) hours early to actually shut down the servers this morning. And since everything is virtualized in some remote data center do they actually do that from their kitchen while wearing pajamas and knocking back a Bloody Mary?

Replacing the hamsters.

After they start the scripts running, I imagine something like this?

Note: I'm not a dev for this game, but experienced in other industries.

The actual deployment of the next iteration of the code is likely rapid and depends on their infrastructure (or however much infrastructure that they have). I expect that the majority of time spent is likely in QA/QC post transition to the new patch followed by an administrative document/checkmark.

This QA/QC step may have both technical and administrative elements given that it is an outward facing product (one that if they screw up too much, may cause a loss of revenue). So I would expect this type of workflow:
1. Deploy software (likely a low burden)
2. Automated and manual system tests depending on team availability and test suites (medium burden)
3. Administrative review, maybe a memo, something like that which can take an hour or two to get the green-light depending on how their ops are setup.

At a high level that's what I would expect. Naturally, most of the devops is likely handled in some other sandbox and ideally the push to live is more or less a triggered event with some praying involved.

Also, they might sacrifice a goat, that takes some time.

Ngreth can really dance.

No way does any Ops team have this much energy.

Completely true.

Well, there was a certain New Years Eve Party. But I digress.

It is also possible that they schedule other maintenance during the patch, especially when they have a longer downtime. They might be "spinning up" the new TLP servers to have time to make sure they're connected to all the internal systems they need and the ruleset is working as intended. They might be replacing hardware which would require its own migration efforts. The scheduled downtime is not always just applying the patch and making sure it's not disruptive to server operations.

EQ has planned downtime once a month. EQ2 has planned downtime at least once a week, sometimes more than just once a week.

Actually, our Ops team is pretty upbeat and on it during update mornings

After how many espressos?

https://xkcd.com/303/

Actual footage of DPG during a patch:

Wouldn't it make sense to shut down and patch at night like in the old days?

Well it depends on what is being updated and can mean having to do different things.

I manage 3 websites, 2 automation servers, and 2 client apps where I work.

Most of the stuff at our website uses a system to make a build and then on publish times they just click a button to publish it. Basic overview of what it does is take down the different websites / apps that are affected by the database changes. It then publishes the build that has already been made the day before and turns everything back online.

The software I maintain has always had issues with the automated builds / publish so I just do it manually myself. It depends on what needs to be published. Sometimes you can do database changes without breaking and having to republish any code at all. Sometimes you need to shutdown everything before you can make the changes because the existing code will not work with the database changes. Sometimes website can be updated without taking it offline if a single page can be changed. Other times the change is breaking and will kick anyone logged in off but it is an instant change so no reason to shutdown the websites and so it gets published while always being live. Sometimes your system may be depending on other systems so there are times when no code changes are made to my system but we still have to take it offline for changes made by other apis my software is using. Tons of other little things that can change what is happening but kind of a summary of the types of things that happen, there is no single answer for it.

If you don't care about the schedule of the developers who are in charge of patches.

Why spend extra money on patch, when they have devs working at the time, which they would not have if changing to night time ?
We see patch get extended.

I am sure we would see a more unstable time, if the patch is moved!!

They don't work overnight and if it was patched during their work time they would happen during prime time US raid time. From my understanding they moved the patch time to be more friendly with the schedule of the people running the patches.

Well halfling. Now that song is stuck in my head all day. Thanks.

On a more positive note, MTV videos in the 70s and early 80s were awesome. This video is proof.