Announcement Regarding Census

UPDATE 10/9: Hello All,

Thank you for your patience as we continue to investigate performance issues. We wanted to update you on Census and next steps. We will be turning Census back on this Friday as we are done looking at things related to it. In the next week or two as our next step, we will be turning off Third Party UI and looking at those.

We wanted to give you all a heads up for a few reasons, the primary one being it’s important that you continue to play and raid even without the UI so we can track the effects. We also wanted to make sure that you had enough time to get used to using the default UI before the Third Party UI are turned off, we will let you know the exact date of the Third Party UI shutoff as soon as we can.

Once again, thank you and see you in Norrath!

dreamweaver


UPDATE: We will not be turning Census back on tomorrow. We’re still assessing and need more info. We’ll get it back on as soon as we can. However, monitoring potential issues is the priority.


Hail Norrathians,


In an effort to diagnose some reported performance issues, we are going be trying a few solutions over the next few weeks. The first one will be to turn off our data generation tool, called Census. We will shut it off for about a week starting at 3 PM PT today and noting changes. If there are updates regarding this process or changes that will occur because of it we will let you all know.


Thank you!

- dw

Will be hard to play a week without at least these two sites

u.eq2wire.com

kulavvy.com/eq2/CollectionHunt.aspx?collection=CD

hopefully you can a way to fix the lag problems without a permant lockdown of Census *thumbspress*

I have to say, this is really reaching if anyone thinks the "census" is the root or a major contributor of the lag in the game. The "census" is always present, its feed is continuous thus a constant.

Lag is from my experience highest in the latest content and 10 fold worse in the contested zones, PQ's, any of the contested raid zones, contested heroic zones. But even then it is still variable, you have a contested raid zone with 6 players and the lag is decent, you add a full raid party and you notice the change in FPS dropping quickly, you add a second raid team and you are in single digit FPS. You try and get to the max 4 raid teams that DB says is expected, and half the zone is going LD because they cant move, cast, heal, anything except crash to desktop.

You go back and visit the old raid zones that are essentially empty and you will find little to no lag.
Heroic zones that are current content, you will find lag, old content their tends to be much less.
I truly believe their are not appropriate physical resources allotted for the latest content and the players are tired of having to deal with their short thinking process. The census is not the problem, lets not pretend it is.

Stop with the ridiculous contested zones that generate more stress on the system and increase lag more than any other zones in the game. Allot appropriate resources to the new content, if the dev team thinks it has enough, add 10% more to make sure they do. I know DB is a business and needs to cut corners as needed, but if the players aren't happy, the players, the folks paying the bills, are better off looking elsewhere.

Thanks for the heads up.
Hopefully the powers that be realize that shutting census down permanently or even for an extended period of time will have a significant negative impact on many levels of game-play.

Hiza's observations about how lag is significantly worse in specific zones is on point and a strong indicator that the information feed provided by census isn't the 'cause' of lag that is non-existent in some zones and crippling in others.

from Kander in Discord:There were changes made over time that we are investigating.

I suspect that temporarily shutting down census is the dev's looking for some fingers in the pie, with the intent to do some pruning.
As for some possible sources of lag. picture this you have 50+ players in a contested zone , every player has the largest boxes and or bags the game allows ,most are possibly loaded to 50% capacity or more , add animations on mounts , armor , characters , that's floating leaves, smoke , bugs , sparkles , twinkly and various other moving animations ect , now add flapping cloaks , restless familiars , dumb fire pets , warders , and holiday pets. top that with most running all the animation options , rising feed back numbers seeing player tags 100% of the time and shadows . Now add all the third party user interfaces , armor procs , ascension effects , spell effects and don't forget the possibility of those unspoken of third party programs: the possible hacks that some players like to try to use to get ahead . Of Course its all going to add up to a lag nightmare.
it has to be a nightmare for the devs to find and fix , makes me glad its not my problem. and from the bottom of my heart I wish the devs happy hunting on that .

When a character zones or camps from the game, the game server momentarily pauses and exports nearly all the character data to Census in one big blob. Fortunately, recipes, quest journal, and collection progress are exported in separate chunks. Still, it's a big hit on the server every time there's an export.

When Census started out, it was a fairly limited subset of character data being exported. Then Achievements, Racial Traits, Wardrobe, Quests, Recipes, Collections, AA profiles, and Mercenaries were added. Unfortunately the merc data is not complete enough for us to use, but nonetheless what there is is exported for each character. Plus character items have gotten more complicated with the advent of Reforging, Refining, Experimenting, and Infusing.

This 1 week shutdown is concerning to me. In Census' first 5 years, I could count the yearly downtime in numbers of hours. Over the last few years we've seen downtimes sometimes weeks long. I think there is some room for further optimization in Census which can be done, but I doubt it is fully to blame for game lag and instability. I am hopeful that the Dev Team can find non-drastic solutions which help with these problems without impacting Census users. I've spoken to far too many players who feel that the existence of EQ2U is a dealbreaker to be able to play EQ2 given its current complexity.

Thank you Feldon That was informative, at least for me, it helps me understand better.

I like eq2u and hope it can stay active. I don't think it would be terrible if the info was sent at a slower rate though. Like twice a day or something like that. Every time you camp could be problematic with everyone switching around to various alts every day but I don't know that folks need that immediate of info? Well maybe, not me.

I think we can move past Census based on lag during the last 18 hours, if anything it made things worse. I'm curious though how it can be anything except a server response delay considering the slow down also affects the launcher when first opened.

maybe they are doing it in stages to ascertain certain things.

maybe census webservice has been disabled, but that the servers still publish to census.

you are totally jumping to conclusions with incomplete information; if they are doing it for a full week, they must have a plan to work through

so eq2progression will be down? so we will have to go in and update it ? or how about updating Everquest 2 servers to hold the game and run more smoothly

Census servers (there are servers for EQ2, DCUO, and Planetside 2) running does not impact EQ2. EQ2 game servers pausing briefly to publish data to Census seems to be the concern.

You can verify that EQ2 Census is still there. It's just not getting new data.


When the launcher first opens, it queries Census as an elevated user to get your list of characters. Not sure how that ties in.

yes but there may be a data warehouse which census pulls from and game servers push to and this may still be going on for testing, but with the pulling disabled

of course this is just speculation on my part, but why would DBG spend a week on it if it's just turn off, look at it, **** bosh, it fixed the lag so let's retire census?

I've come to rely on EQ2 U to get information on my characters when I'm not at my desktop. The nice thing about it is I can check on stuff from my mobile devices. It would be horrible to not have it. I am already leery of getting the upcoming expansion because I'm getting tired of the moving the carrot thing, and another 10 levels, another catering to raiders thing, another extra no fun crafting timeline and/or adventuring timeline. Thinking maybe I'm getting too old to enjoy the game and the more perks they change or take away the least likely I feel like continuing. I recently worked on my characters' binder to see if I start to get enthusiastic about the game again, but if they take away another thing, I may be trying to find something else to play. Hope some of the speculation about Census is just speculation.

So as video cards get better and people run higher frame rates it slows the game down? Woudn't all of that information be there whether you have a card to get it or not?

There are so many things, you can't do without census or EQ2U without fighting against all odds.

1) A good overwiew over spells or ASC an there tier.
A nightmare ingame.
Have heard a rumor someone build a software to track all ASC and their level for the complete raid.
2) Changing AA setup.
Comparing AA setups for optimizing.
Without EQ2u a nightmare or nearly impossible.
3) Tracking progession for raid achievemnts.
With census easy, without manually tracking in Excel.
4) EQ2U is your best friend for reforging.
To work out this only with ingame tools is a nightmare.
5) Quest-Tracking. Whats done, and whats not.
6) Finding receipes, their components and the right book.

There is much more you can do with Eq2u and census that is impossible with ingame tools.
The end of census will be a nail in the coffein.

My 2 cents,
Dolgrin.

Postscriptum:
Feldon and his EQ2U team are doing a great job to keep the game alive.

This is a sketch of what I think the problem is. It's not meant to make light of or in any way attack the way that EQ2 Census works. EQ2 Census has been one of the most successful things the company has ever done.

Folks may not remember, but the EQ2Players website broke -- a lot. This was because EQ2Players directly probed EQ2 data. Even the slightest game changes would break the site, requiring someone to constantly update and maintain the EQ2Players website to compensate for these changes. I believe that for EQ2, having a layer of abstraction between game data and Census was considered prudent, so that Census would be insulated from internal game changes. However this abstraction seems to be having some scaling issues.

EQ2 Census was built upon the work that had been done for the EQ2 Mobile app. A server was setup with MongoDB (then, one of the most efficient, lightweight databases for huge amounts of data) and hooks were added to the EQ2 game code to publish certain data about each character. Scripts were also setup to export static data like Items, Spells, Recipes, Quests, etc. when certain servers reboot.

When PlanetSide 2 Census was in development, the devs and playtesters ran into performance problems of their own and so MongoDB was scrapped in favor of PS2 census being a direct replication of PS2 Game Servers. This copy (basically an instant live backup) happens incredibly fast at a very low level that cannot impact game performance. The game doesn't even know it's happening.

There may be other optimizations that can be made. It may be that everyone gets opted out of Census and people have to opt back in. It may be possible to rewrite EQ2 Census to run on EnterpriseDB. I'd like to be involved in the conversation, but I'm not sure that's going to happen.

lol thats was me, but its not really about tracking asc, its about tracking all your raid force, all their stats, required items, optional achievements and finding the slackers