So many problems since the data center move!

Scan the bug reports and the player issues reported over the last few weeks. Just tonight a Dev marked two bugs as not reproducible even though the raid assignment bug happens with nearly 100% consistency out here in the "wild". Consider the problems with targeting no longer working well and many other symptoms and complaints of lag or packetloss. You don't have to be CSI Los Angeles to figure out they've got infrastructure problems.

There may also be two other factors from the pandemic contributing to the recent spate of problems: (1) network outages or loads (local/regional ISP for users and service) and (2) demand (more players online across Daybreak network) at peak.

Network outages don't seem to be affecting other games with vastly more network traffic so I don't believe #1 is a credible reason.

More than likely DPG is going extremely cheap on all their hardware and network bandwidth and the results are the poor performance EQ experiences.

And as I think about the spate of problems I'm also thinking that Holly saw this all coming and bailed just in time.

I will agree that since the migration that we have indeed seen a large increase in problems due to network reliability, lag, poor performance, just a general observation I can agree with.

Jenn Chan has done her best to address the concerns admitting there have been problems & apologising for them in this post already:

https://forums.daybreakgames.com/eq...te-to-the-eq-community-from-jenn-chan.267730/

I think it is their over-reliance on their internal test-environment, that is the problem.

I wouldn't surprise me at all if most of the bugs we are seeing now, are only reproducible on live servers...

I'm not calling out Jenn. For all we know she's the hero holding things together after the bean counters decimated her funding. But someone in management has a lot to answer for. Who knows how many paying customers they've lost on the TLP side that will never come back to EQ? And even worse they are badmouthing EQ to all their gaming friends.

I didn't intend to imply that you did

I don't think this was anyone's fault as such but when moving EQ it's like moving an ancient lady around she gets a bit ornery about it & is likely to forget things.

I gathered from Jenn's post that some of the issues were definitely related to hardware issues, likely incompatibilities between EQ back-end code & the new boxes it is being run from.

Best we can expect is the powers that be will throw some extra resources at the problem. This is still a money maker game after all.

The data center move may well have exacerbated some issues (or, their timing may be coincidental, I don't know). That said, lag-induced bugs like with Quick Time are the result of zone performance issues that have been building up for years. They've been steadily permeating additional aspects of EQ, and will continue to do so unless addressed. It fell on deaf ears 6+ years ago when pets began lagging in raid instances, but at this point it negatively affects enough players that they really need to dedicate additional resources to it.

I've been in multiple situations where we had to make decisions on whether to migrate and modernize legacy systems or replace them. We always went with replace because it was cleaner and faster than trying to migrate and modernize old code/systems. Modernizing the infrastructure was part of the decision and is always problematic when frameworks/architectures are no longer supported or you had legacy code practices where things were hard coded that shouldn't have been. Of course our decisions were with the luxury of the legacy systems being enablers of and not the source of our revenue streams.

It's an interesting problem set trying to balance resources to address non-revenue expenses such fighting obsolescence, maintaining old code and frameworks etc. vs working on revenue producing content and modernizing to expand the customer base. This is particularly true if they are financing their development primarily from current revenue streams without additional capex spend or investment. This is why I've always thought that certain aspects of EQ would be great to be in an open source model such as upgrading the graphics engine. However, at this point in the size of the player base, I'm not sure an open source model would work, hybrid or otherwise.

Overall, I don't envy anyone trying to keep a 20 year old code base up and running and error free. Especially doing this while trying to create new and engaging content in order to keep the revenue coming in from a shrinking and primarily legacy customer base. I'm not defending the recent issues, but having dealt with similar issues in the past, I can empathize with the challenges EQ is facing.

They had opportunities to correct this back in like 2013, but instead chose to fire developers (RoF T3 and T4 suffered as a result, and CotF was a gross spending of expansion cost versus customer deliverables), and then the migration from SoE to DBG helped display that Sony Online Entertainment had very badly mismanaged the EverQuest IP. Funneling over half of the money that EQ1 makes into failure projects really didn't help.

The game is in such poor shape, that we're basically in maintenance mode with some occasional content releases (which are basically reused old zones with some glitter thrown on top). You're right, the customer base is legacy folk from 20 years ago. We love EverQuest, but the company/companies managing the game from the business side have failed so hard, it will literally take a code re-write, graphics and UI update, and total database migration to get people (new people) to start playing the game.

At some point, the person handling the finances, will either cut the game entirely (remove devs, billing, support) or will try to offload it to another company to manage, hoping to get some financial investment back.

I play Fortnite and I’ve experienced performance issues on multiple occasions (between daily or multiple times a week). Many Fortnite players have reported the same issues. Additionally, I stream television through Sling and I’ve been hit with lag and outages at random times (some lag lasting for hours).

Unless someone is paying for fiber optic cables or business-grade ISP, you can’t definitively say pandemic effects is not a contributing factor (in addition to data migration).

In the recent AMA and some downtime updates, some of the devs/coders admitted to running into legacy code that caused unexpected but systemic effects. I believe they said something to the effect that if they touch one part of the code, it causes unintended cascade in other parts of code. Some coders are “new hires” and are “learning on the fly.”

You literally just described all software companies, including Microsoft and Amazon.

Those companies create their own (newer) engines. Unfortunately, if they wish, they can simply lay off old coders (trained on the old system) and hire new "young" coders (trained on the new system). People can learn the new system but there's a point of diminishing returns with consistently new iterations (and employee's age). (It applies to any industry - airline pilots on different model Boeing/Airbus, etc).

EQ runs exclusively on ancient code, however. Game code may appeal to coders who do not want to retrain/retool over the long term.

I'm sure the code for EQ has all sorts of hidden dependency's that anyone beyond the original programmers would have no idea to look for when they make changes, so many corners were cut to just get this thing capable of running on the internet back then. Hell the original programmers wouldn't remember all of them. Chances are they changed something, modified a variable, which cause whatever garbage collection they had in place to break which causes an persistent memory leak. Which they're still trying to ID the source for.

In addition to external internet issues, there are in-home issues as well driven by wifi channel conflicts or latency due to an increased number of devices during Covid. WIFI routers all tout big numbers in bandwidth and available connections but real world scenarios (walls, half-duplex connections etc) can knock it down really fast. Switching to a mesh network and/or running wire to my gaming pc is on my to-do list due to performance loss with 20+ devices running on the network. I'd recommend anyone having issues post-covid to use one of the free wifi analytics tools just to double check it's not a pc-to-router issue in addition to the issues outside of the house.

Seriously, every day, it seems to get worse.

At this moment, on Vox server, in my Great Divide pickzone, with only 36 players, which doesn't seem that crazy for a big zone like this ... it's lagging.
Mobs rubber banding and/or warping, pets slacking and so on.

For this (mainly) US based population server, with a pretty low population in general, it's not even prime time.
It's midnigh for me (EU GMT+1) but for US people, most of them are not even connected atm.

So what? 36 is already too much to play outdoor?
What do we have to do? Put us in quarantine again in instantiated zones, exclusively ?
(and it's not always lagproof, last week we were lagging in the Velketor lab mission instance so ... )

So DPG, please ... do something, check your code, your servers, and feed the hamsters.
Ok, the game and the code are old, and I don't know how your servers are handled, but sorry to say, it's just a big pile of poop at this moment.

So I'm off for today, I'll play Path of Exile.
At least it's not lagging like hell.

I've wondered in the past about things like giving every class large numbers of swarm pets. Why would any developer do this when it vastly increases raid lag?

Yes, swarm pets are great but give people one or two per use of the ability, not 5 or 10 swarm pets per use. Increase the damage and survive-ability of each swarm pet and reduce the total numbers of them.

I do applaud their adding in the ability to hide players and pets, but passive (not requiring players to do anything, even something like just checking a box to hide) changes are always better.

And possibly swarm pets could be invisible to all except the caster by default.