Nagafen/Venekor Server Merge

From Rothgar:

We're still planning on going forward with the PvP server merge Wednesday night at midnight PST. Initially only Venekor will be brought down to backup the database and export all of the data. When the export is complete, Nagafen will be brought down for us to merge the data. After the merge takes place, we have some other background tasks that could take a few hours. This includes cleaning up universal-chat, merging shared banks and updating player names and guild names that had conflicts. Be prepared for as much as a 24-hour downtime for Venekor players. Nagafen players will experience around 12 hours of downtime depending on how fast the processes go.

What this means:
The Venekor server will be brought down at midnight tonight, Wednesday October 21st. Nagafen will be brought down around 8am PDT on Thursday October 22nd.
Neither Butcherblock nor Lucan D'Lere should be affected by this downtime.
Please watch this thread for updates about the merge.
Discussion and questions can be directed to this thread.
Thank you all for your patience with this process and we apologize for the inconvenience.

Maintenance Plan:
Bring down Venekor (Done)
Characters level 10 or less, with less than a plat, that have not been logged in since Feb 2008 have been flagged inactive. (Done)
Remaining active characters ( over 100k ) are being exported to flat file. Est. Time of Completion for EXPORT ONLY is 12 hours. We will post a more accurate ETA after the export has been running for awhile. (In Progress)
Three hours before the export is due to be completed, Nagafen will be brought down for a final backup. Currently this is scheduled for 8:00 am PST but may happen sooner if the export is running fast.
After the Nagafen backup is complete and the Venekor export is complete, the import of the flat file into Nagafen will begin. ETA on this process is another 12 hours. We will also post another ETA several hours into this process with a more accurate time of completion.
When Nagafen import is completed, scripts will be run to handle name conflicts, exports shared bank and move in-game mail. ETA for this script is 3 hours.
After all scripts are complete, we will perform initial verification that shared banks and guild halls have moved successfully.
QA will start their smoke test. ETA is 1 hour
Nagafen will be unlocked

Status Update
The export process is just over 10% done. It seems to be going slower than our two initial test runs and we are trying to determine why.
I'll give another status update in the morning.

Status Update
At 7:00am this morning our export script experienced a crash. This is not something that has happened in any of our test runs or previous merge processes so we're looking into causes. We suspect that it could have been a hardware issue (faulty RAM, hard disk error, etc) or it could have been data related. Since our test runs used actual copies of Venekor data, a data error doesn't seem likely unless it developed in the last week.
We've restarted the export process as of 7:00am PST on different hardware and are monitoring it closely. The current impact is an extended downtime for Venekor players of roughly 12 more hours. For Nagafen players, this would mean a delayed maintenance schedule, but hopefully not an extended downtime.
We will keep you posted throughout this process.

Status Update
At 2:00pm PST we're just over 30% done with the export. Unfortunately the process is still going slower than our test cases. We are nearing the point in the export where the script failed this morning at 7:00am. I'll update again in an hour and let you know if we're still progressing without problems.
At this rate, it looks like we may need another 11 hours of export time before we are able to bring down Nagafen for a final backup.

Status Update
At last check, we were over 40% done with the export process. We've passed the point where the script had a failure this morning, so I'm still optimistic that it was just a hardware problem on the other server.
I'll update again in a couple of hours.
In the meantime feel free to join the party on Test-Copy. If you don't have a version of EQ2 patched to the Test Server, no worries, you can test out our new streaming client! Check out this thread for more information.
Calling all Testers! Streaming Client Now Available!

Status Update
The export process is over halfway complete. At the current speed we expect to take Nagafen down around midnight, give or take a couple of hours, for a final backup. After the backup is complete, we will begin to import the Venekor data into the Nagafen database.
We *hope* this process will go faster than our estimation of 12 hours due to Nagafen's new database hardware.
I'll post another update in a couple of hours.

Status Update
The process is over 70% complete. It looks like process is running a little faster now. We may be looking at bringing Nagafen down in an hour or two.

Status Update
As of 9:00pm PST the export process is over 80% complete. Nagafen should be coming down now for a final backup before we begin the import process.
ETA on import is still 12 hours until we see what kind of throughput we get.
I'll keep the updates coming.

Status Update
The export process is complete.
The Nagafen backup is 50% complete.
We will start the import of Venekor data as soon as the backup is done.

Status Update
The Nagafen backup was completed.
The import is currently in progress and 12% complete.

Status Update
The Nagafen backup was completed.
The import is currently in progress and 25% complete.

Status Update
The Nagafen backup was completed.
The import is currently in progress and 50% complete.

Status Update
The Nagafen backup was completed.
The import is currently in progress and 72% complete.
Next Step is Shared Bank Merges ~3 hrs
Next Step is Guild Hall Moves ~1 hr

Status Update
The Nagafen backup was completed.
The import is currently in progress and 90% complete.
Next Step is Shared Bank Merges ~3 hrs
Next Step is Guild Hall Moves ~1 hr

Status Update
Our chat system has been updated to reflect the name changes.
The shared banks have been exported.
We are starting migration of guild halls and need to combine shared banks in the case where you had a character on both Nagafen and Venekor with the same alignment.
We're hoping to begin the QA smoke test in an hour or two. The smoke test could also take an hour or two to complete.
So final ETA is around 3-5 hours.

Status Update
Shared banks are still being merged and conflicts being resolved.
ETA is still 2-4 hours.

Status Update
We experienced a problem with our shared bank merge script related to old characters. We had to fix the bug and start the script over, but it should only take another hour to complete.
When EQII was first released, your shared bank coin values were stored as 32-bit numbers. Later we changed these to 64-bit values as players were getting more and more coin. Our script didn't take into account older banks with 32 bit numbers. Its been fixed now and we're moving forward.
ETA is still 2-4 hours.
Also you may have noticed that chat and mail are down. Our platform department had to bring them down game-wide in order to run some of the conversion scripts. We hope to have this restored in a couple of hours as well.

Status Update
Shared bank merge script is still running. We should be getting close to completion and the start of the QA smoketest

Status Update
The shared bank merge is complete and Nagafen is coming up locked for QA to start their smoketest.
ETA for completion is 1-2 hours