Why We Need to Cut the Devs Some Slack

Discussion in 'The Veterans' Lounge' started by andromeda, May 21, 2020.

  1. andromeda New Member

    My 2 copper is worth exactly 2 copper--maybe less--but I want to share it anyway.

    We need to cut the devs some slack.

    The EQ community is an older community. Most of us are middle-aged, and many of us work in IT. Because of that, on these (and other) forums you often see people saying things like:
    • There is no excuse for this downtime.
    • The work was obviously poorly planned
    • The devs "only" had to do such-and-such to make such-and-such work, so they're obviously incompetent.
    Those statements are frequently mixed in with some mumbo-jumbo about merging databases, or migrating servers, blah blah blah.

    But here's the thing. If you ever hear anybody say any of those things, then they've never worked with IT "at scale". Many people haven't heard the phrase "at scale" before, but what it basically means, is a big-*** network. That doesn't mean those people don't know what they're talking about, and it doesn't mean they're bad at their jobs. It just means that they've never worked in an environment that either A) has a large number of devices and databases--things like that, and/or B) is a legacy network (meaning old code, old devices, etc).

    Now, I have no idea what the Everquest infrastructure looks like. I've got no insider knowledge. But I suspect that it is 1) very old (in terms of code, not necessarily hardware), and 2) sharing hardware and possibly code with other DBG games, some of which are also quite old.

    What I do know is my experience. I'm primarily a network engineer, and I also do or have done software development and system administration on Windows and Linux. That includes virtualization, primarily with Hyper-V.

    I started out at a NOC at a large company. It was a billion dollar company with thousands of locations in three countries. It was the largest network I had worked on, and I was proud to be there.

    After a few years I went to the company that I'm at now, which is one of the three largest networks in the world and handles traffic for billions of customers per month. I won't narrow it down more than that.

    We have hundreds of thousands of network devices (if not more), millions of servers, and I literally don't know how many VMs are running at any given time.

    Do I say that to brag? No. After 10 years I still feel like an amateur compared to many of my peers. My first assignment when joining the company was to upgrade the network at a "small" backup datacenter. I did the design and architecture planning and handled the migrations. Datacenter techs did most of the physical work.

    That "small, backup" datacenter dwarfed the size of the billion dollar company that I came from. In fact, HALF of a single row had more network capacity then the entire central datacenter at the billion dollar company I came from. I walked down the row and added up the costs of the devices. It was in the millions.

    I say that not to brag. I'm being honest when I say that I feel like an amateur. Some of my peers have designed some of the largest cloud networks in the world. Others have designed services that handle over a billion unique clients monthly. They literally hold the patents to a lot of the technology. They're practically gods to me. Every time one stops to help me with something, I take pages of notes.

    When we upgrade our global infrastructure those same people lead the design. They're the best in their field. We have roadmaps outlining years of work. We plan it. We spend tens of thousands of engineer hours on it.

    And when we begin to execute the changes, do you know what happens? It goes to hell. There are outages. Stuff you couldn't anticipate. Weird bugs in network equipment that was tested over and over again by our engineers and the people who created it. Customers are mad. We get escalations coming from major customers that are losing $1 million an hour.
    And that is from a company that my first day on the job told me if a problem cost less than $1 million to fix, then don't ask. Just do it.

    Now, I know DBG's network doesn't operate at that scale. However, I am sure it is much larger than people realize. I am sure that there is 20+ years of spaghetti code, a mix of technologies, God only knows how many different programming languages and types of databases. And I'm sure much, much more.

    They might not be as large as our company, but they're probably larger than most people work with, and they don't have nearly enough funding or people to adequately test these changes.

    So, we need to cut them some slack. Yeah, it sucks to not be able to play my favorite game for a couple of days. I tried to log on last night after working till 2 a.m. I just wanted to get an hour of gaming in to wind down, and the servers were still down. It sucked.

    But we also know that DBG devs worked around the clock (literally) to get things back up and going. I'm sure they'll also offer refunds or extended double xp time, etc.

    So lets cut these men and women some slack. They've got a tough job, and they're keeping a 20+ year old game going, and I doubt it's that profitable anymore.

    And then, once the servers are back up, let's party!
    Duder, Metanis, Thunderkiks and 7 others like this.
  2. Dancinggnome3 New Member

    How much did they pay you to post this?
    lockjaws likes this.
  3. Smokezz The Bane Crew

    I've also seen people talk about their testing and deployment systems. Obviously it's very easy to have a great testing and deployment system when it's been put together in the last few years. EverQuest is huge, and is 21 years old now. To try and MOVE that to a modern testing/deployment system would be quite the epic undertaking that would require significant manpower, money and downtime to do... which would once again cause everyone to flip out.
    Magneress and andromeda like this.
  4. andromeda New Member


    Just 2 copper :(
    Yinla and Vumad like this.
  5. Sheebea Augur

    This unexpected downtime IS the standard tho. Everytime. I've been playing since 2000
    Magneress likes this.
  6. Dragoncommander New Member

    We Should Make A New Reddit Section Called Entitled Players
    andromeda likes this.
  7. andromeda New Member


    That's a great point. And if you did that it would make sense to move the servers and databases to new hardware. But that's quite an undertaking too. I think of the networking side. With that alone you have to deal with BGP changes, configuration, ACLs, firewalls, physical changes to cables and patch paneling, etc. If it's hosted with a cloud provider then that's an easier technical challenge, but you have less control if there are hiccups.

    I really doubt DBG can justify that kind of expense.
    Magneress likes this.
  8. dreadlord Augur

    How do you know they 'work round the clock'?
    How do you know they treat this with the utmost urgency (cancelling all leave; getting everyone working until it's fixed etc, assurances of a strategy to ensure at least that problem doesn't happen again)?
    I think that's a big part of the problem. The lack of *regular* and informative updates conveys a lackadaisical attitude.
  9. Smokezz The Bane Crew

    It's already on newer hardware. Part of the problems last night was with new hardware. EQ isn't running on the same hardware it did in 1999.
    Magneress and andromeda like this.
  10. nildiar Lorekeeper

    wHY NOT? you dont think they made enough from over seer sales in dp store to buy some new hardware? Id call that delusional we KNOW some people dropped big money on that stuff pre nerf those are facts... Where did that money go? Not into eq
  11. enclee Augur

    When, you layout it like that it doesn’t land the way you think it does. Not being able to implement a solid testing environment in over 21 years of operations is disappointing by any metric.

    I don’t want to degrade any of their work, as that type of thing falls at the feet of management level.
  12. andromeda New Member


    Unexpected downtime, yes. But that's true for almost every game, unless you're Blizzard in WoW's heyday and have a large budget.

    This much downtime is not the norm (I've played since 2001). DBG laid off a lot of staff over the last couple of years, and in the RCA (root cause analysis) they posted last night they said they had to resurrect old code and were not able to do a full test before the migration. I.e., they don't have enough infrastructure and/or devs to do that anymore.
  13. enclee Augur

    Definitely, cut their team some slack. There are times, when things don’t go well that actually result in a better long-term outcome.
    Magneress and andromeda like this.
  14. andromeda New Member


    That's a fair point, and I agree. Most failures do land at the management level.

    The counter-point is that I'm not talking about management, or even former devs. I'm talking about the CURRENT devs. They're working with what they've been given, and what they've been given probably sucks and they don't have the budget to make it better.
  15. Magneress Augur

    Everquest had really really humble beginnings.. I can't believe people forget though.

    Since so many of us old people are around to remind them :D
    Zaray, Duder, Elyssanda and 1 other person like this.
  16. andromeda New Member


    Again, I'm talking about the devs. They're not the ones making the decisions on how to allocate funds.
    Magneress likes this.
  17. Yinla Ye Ol' Dragon

    They worked all last night to get the servers up, Dreamweaver was in Discord all the time the servers were down answering questions and posting updates on the boards.

    The servers didn't get fixed by Mr Nobody overnight.
    Barton, Magneress and andromeda like this.
  18. andromeda New Member


    Additionally, there updates were reasonable. I was surprised they laid out a detailed explanation before the problem was fixed. Heck, I'm surprised they did it at all. That's NOT the norm. Most company's RCAs boil down to, "It broke...our bad."
    Magneress likes this.
  19. Lurikeen Journeyman

    Complaining about EQ is a pastime. It's like that thanksgiving dinner tradition practiced by one family member every other family member hopes will be forgotten, but it still shows up for dinner each year. What to do? Good job devs! Forget about the complaints. :)
    minimind and Magneress like this.
  20. Questoften32 Augur

    Wait I heard...they are working from home on eq and eq2?

    The parent corporation needs to allocate more funds to the team right? This is like the guys that made quest for infamy or what ''was'' AGDI interactive studios?

    If so its a miracle any expansion was made in less than 4 years and you should just be happy the game is running at all. I did not think they were that inde. Yes should be thanking them as they are one step away from crowd funding. Don't go thinking its Sony.

    However funny story: This is not exact, I disremember but: I faceplamed one time when I was reading wow (pre-activision I think, a big company. let slip in a post they had one guy (1) like janitor Bob fixing servers once and said he hadn't slept all night. I laughed so hard at the tightwad nature I began looking it up and they were at the height of success. It was funny as hell.

    Reverse polarity situations, both just as odd as a cod.
    Magneress likes this.