The Data API

Introduction

Daybreak Games provides access to game data through a RESTful API called Census. This system is a centralized service that receives character, world, and static information from game servers so that it may be accessed by various applications. These include Daybreak Games websites, Game Launchpads, 3rd Party Fan sites and mobile companion apps. Access to this data is available at census.soe.com, which is available using either HTTP or HTTPS.

Census allows you to request information directly from Daybreak Games regarding the status of a game or game entity (e.g. character, item, spell, etc.). All requests to the REST interface follow a pattern and there are different patterns for different types of data. The REST interface will return all results that match the attributes provided within the query string. Data is returned is in the form of a list containing one or more items from the collection provided in the URL. If no results are found, this container will be empty.

The URL Structure is as follows. Any part of the URL within brackets ("[]") is optional and may be omitted):

/[s:serviceID]/[format]/{verb}/{game}/{collection}/[{identifier}][?{queryString}]

Service ID

A Service ID or Identifier is a short word or phrase associated with your website or your name which is used by Daybreak Games to monitor and provide further capabilities to sites who use the API. A Service ID is not currently required, but is encouraged. Your service must be registered with Daybreak Games to use this facility. The first step to getting this done is to request a Service Key.

If you do not have a service ID, just omit the "s:serviceID" part of the URL:

census.soe.com/json/get/eq2/item/2906185062

If you have a service ID, provide it as below:

census.soe.com/s:serviceID/json/eq2/item/2906185062

We are encouraged to use the Service ID both in data queries and in image searches. For example this image:

census.soe.com/s:eq2wire/img/eq2/icons/600/item/

HTTP (GZIP) Compression

If you are making calls to the REST API from a server process, you should consider enabling HTTP Compression on your request. This is already enabled for most browsers, but you need to set an HTTP header when requesting through code.

Accept Encoding: gzip

This will reduce the transfer size of a character request dramatically.
More information about HTTP Compression may be found here.

PHP 5 should handle this automatically if you are using get_file_contents, etc.

If you are using PHP's cURL to make your data requests, then you must set an empty CURLOPT_ENCODING to automatically send the list of all supported encoding types in the request header (which will include gzip). Otherwise, no Accept-Encoding header gets sent.

curl_setopt($curlResource, CURLOPT_ENCODING, "");

Data Format

Census supports returning data in eXtensible Markup Language (XML) and JavaScript Object Notation (JSON). The recommended format is JSON as it is a very efficient way to access data for most applications. If the XML format is requested, then the results will be provided in a parent <results> element. If JSON results are requested, they will be in a dictionary( {} ).

Using a plugin such as JSONView, JSON data can be presented in a "beautified" format that is easier for humans to read. This means that you don't have to run the output through a plugin or filter just to make sense of the JSON data. JML is particularly useful when troubleshooting queries as you may hover over any field and the data path will pop up in a tooltip.

The acceptable format values are:

• xml
• json

In order to access data across the domain boundary, one may want to use JSON Padding3 (JSONP). To do this, append a callback attribute to the query string of the request that uses the json format.

/json/get/eq2/item/2906185062?callback=foo

The results from this request will be padded within the function name, foo(), so that they may be evaluated by the requesting web site within the correct domain.

Verbs

The verb or "action" informs the REST interface about the type of request that is being made. Supported verbs are...

• get - retrieve the objects that match the request criteria
• count - retrieve the number of objects that match the request criteria
• status - retrieve status information (currently used for server status only)

Game

In our case, we are using eq2 for EverQuest II.

Collection

Supported collections are dynamic and dependent on the game providing data to the service. Collections might include Characters, Items, Spells, Guilds, etc. The list of available collections may be viewed here:

http://census.soe.com/xml/get/eq2/

Some data may be omitted from a collection due to certain policies set by each game team to protect player privacy and/or business requirements. Examples of these policies might include:

• Characters may be marked as private so that they are omitted from searches and rankings.
• Items may be hidden until they are "discovered" naturally by a player.

Identifier

Each object has a unique identifier, which is used for a direct link to the information about that object. This can be found within the structure by the top-level “id” field. All results will have this identifier, even if all other information is hidden.

There are also identifiers that are used in reference to other collections. For example, consider the following collection information from eq2:

"appearanceslots": [

{

"item": {

"id": 3098863614

},

"id": 2,

"name": "head"

},

{

"item": {

"id": 2334025620

},

"id": 3,

"name": "chest"

}

]

The item.id value is a reference to a particular object within the "item" collection. When presenting this information, one can put a link to another query to the REST interface or to resolve that information within its parent object. For more information on how to do this, see the Query Commands section for c:resolve.

Game Images (Attachments)

Game images are also available through this service. These include images representing characters (paperdolls, headshots) or in-game items or abilities. Images are typically returned as 24-bit PNG images with 8-bit transparency. Requests for game images follow this pattern:

/img/game/collection/identifier[/imageType]

For example, a certain piece of armor may have this icon:

census.soe.com/img/eq2/icons/3635/item/

Each result within a collection can have a number of "attachments" - usually an image, but this may be any file. For games that support many attachments, there can also be a category and items within each category. The category and the ID for items within each category is game-specific.

When retrieving images from the icons collection, the supported values for imageType are:

• item - a 42x42 transparent PNG item icon
• spell - a 42x42 transparent PNG spell icon

NOTE: The use of the icon collection is deprecated. Use icons instead.

When retrieving images from the character collection, the supported values for imageType are:

• paperdoll - a full picture of the character
• headshot - a picture of the character of from the neck up (suitable for icons)
• housethumb - a picture of a in-game house as published
• petpaperdoll - a full picture of the character's pet

You may use the c:attachments directive in your queries and it will include URLs to the various images directly into the information returned for that object. These URLs will use the Game Image pattern and correct imageType.

A field named itemInCategory is returned if there is more than one attachment within a category. At this time, the only example of this is for a player who owns multiple houses. Their character will have multiple housethumb attachments.

The Daybreak Games Data Feeds API (Census) runs on a Mongo Database system which is a highly optimized NoSQL solution. Users wanting to request data from Census do so with RESTful API calls. In other words, GET queries where each parameter either requests certain data or filters the resulting data to refine your results. As with any GET query, the first parameter must be preceded by a question mark (?) and subsequent parameters must each be preceded by an ampersand (&) character.

This query would return all level 80 items of type Weapon with a quality (tier) of MYTHICAL.

census.soe.com/xml/get/eq2/item/?leveltouse=80&type=Weapon&tier=MYTHICAL

Searching

Field Values

Each request for data may be filtered by the value of certain "fields". Each field in a query string corresponds to a particular field within a particular object within a collection (characters, spells, items, etc.).

Within Census data, fields have a hierarchy or level structure. When accessing a field that is parented under another field, you may reference it by using a dot (".") notation. For example, considering the following data:

"item": {

"id": 4126241497,

"typeinfo": {

"equip_optional": 1,

"name": "expendable",

"casttime": 1.0

}

}

In order to find all matches for the name "expendable" within typeinfo, one would use the following query string:

?typeinfo.name=expendable

Numeric Fields

When more than one search condition is provided within the query string, on a number, it acts as a boolean 'AND' operation. For instance, consider the following query string:

?id=]10&id=[50

This request would return data where the value of the id attribute was between 10 and 50 (inclusive of 10 and 50).

Text Fields

When more than one search condition is provided within the query string on a text string, it acts as a boolean 'OR' operation. For instance, consider the following query string:

?given_by=racialtradition&given_by=focusabilities

This request would return data where the value of the given_by attribute is EITHER racialtradition OR focusabilities.

A complete list of search modifiers appears below:

Modifier Example Description

field=<10 --- Include objects where the value is less than 10.

field=[10 --- Include objects where the value is less than or equal to 10.

field=>10 --- Include objects where the value is greater than 10.

field= ]10 --- Include objects where the value is greater than or equal to 10.

field=/string/ --- Include objects where the value contains a particular "string".

field=i/string/ --- Case insensitive search for a particular "string".

field=^string --- Include objects where the value starts with "string".

field=string$ --- Include objects where the value ends with "string".

field=!string --- Include objects where the value does NOT match "string".

field=*regex --- Use a regular expression to match values within the field.

Regular expressions or "regex" provide a great deal of power for searching for objects. Census supports Perl-compatible regular expressions. The "/" and "^" modifiers above are actually examples of simple regular expressions. It is beyond the scope of this document to fully explain regexes, however there are a number of resources available to learn about them, including here and here.

NOTE: Use of i/string/ for case-insensitive searching of Character, Guild, Item, and Spell names is deprecated in favor of standard searches against their lowercase equivalents. Search performance will be substantially improved by using these:

• character.name.first_lower
• item.displayname_lower
• guild.name_lower
• spell.name_lower

Has

The presence or absence of certain fields in data can be just as valuable as the fields themselves. For instance, the following query returns items that have the typeinfo.growthdescription field (eg. green adornments or Spirit Stones):

?c:has=typeinfo.growthdescription

There is no separate "Has Not" or "Doesn't Have" query. Instead, if you need to find entries where certain fields are missing, add a ! in front of certain fields:

?c:has=typeinfo.growthdescription,!growth_table

Distinct

Sometimes you wish to know all the distinct values for a specific field. For instance, to find all possible qualities (tiers) of items:

census.soe.com/xml/get/eq2/item/?c:distinct=tier

NOTE: c:distinct may not have good performance on extremely large collections. It should be used to build internal cheat sheets/lists to work against. C:distinct is exceptionally slow when used against the Character collection.

Resolve

Some collections have related data which may or may not be needed when accessing a collection. For instance, when you retrieve a Character, a stripped down list of Spells and Achievements are included, however the underlying data such as the name, description, and stats for these data points is not. If you wish to add this data to your query, you may use a c:resolve to "join" collections together.

Example of Resolves for Guilds

Before EQ2 started adopting GUIDs (globally unique IDs) for Characters and Guilds, the game used a combination of DBID and WorldID to specify a unique character or guild. The same DBID could exist on multiple servers, so you had to use both DBID and WorldID. When you perform a standard 'pull' of a Guild, you only get the DBID of each character. For example, a guild named Honor on the Permafrost server:

census.soe.com/xml/get/eq2/guild/?name=^Honor&world=Permafrost

<member_list>

<member dbid="208021" name="Idemu"/>

<member dbid="221484" name="Gige"/>

<member dbid="263553" name="Klehost"/>

.

</member_list>

To also get the GUIDs of characters in a guild, you'll need to add a c:resolve:

census.soe.com/xml/get/eq2/guild/?name=^Honor&world=Permafrost&c:resolve=members

<member_list>

<member dbid="208021" id="867583601813" name="Idemu"/>

<member dbid="221484" id="867583615276" name="Gige"/>

<member dbid="263553" id="867583657345" name="Klehost"/>

.

</member_list>

NOTE: Characters that have not logged in since Nov 2011 will not have a GUID.

You can expand your c:resolve to load additional Character data besides the GUID as demonstrated below:

census.soe.com/xml/get/eq2/guild/?name=^Honor&world=Permafrost&c:resolve=members(displayname,type)

<member_list>

<member dbid="208021" displayname="Idemu (Permafrost)" id="867583601813" name="Idemu">

<type alignment="0" birthdate_utc="1100033940" class="Ranger" classid="39" deity="None" gender="male" level="86" race="Iksar" raceid="10"/>

</member>

<member dbid="221484" displayname="Gige (Permafrost)" id="867583615276" name="Gige">

<type alignment="0" birthdate_utc="1100190443" class="Guardian" classid="3" deity="None" gender="male" level="48" race="Gnome" raceid="5"/>

</member>

<member dbid="263553" displayname="Klehost (Permafrost)" id="867583657345" name="Klehost">

<type alignment="0" birthdate_utc="1102637984" class="Wizard" classid="23" deity="None" gender="male" level="50" race="Half Elf" raceid="6"/>

</member>

.

</member_list>

The current list of resolves can be found by browsing Collections for <resolve>:

• Character
  • Spells
  • Factions
  • AppearanceSlots
  • EquipmentSlots
  • Achievements
  • Warders
  • Statistics
  • Dungeon_Items
• Guild
  • Members
• World
  • Status

NOTE: Resolves such as c:resolve=spells or c:resolve=achievements when performed against one or more Characters can take several seconds to perform and also generate a substantial amount of data. If you are feeling really lucky, you may do a c:resolve=all to return as much data as is possible.

Limiting Fields in a Resolve

You can limit the returned data within a c:resolve by indicating certain fields in parentheses:

census.soe.com/xml/get/eq2/character/?c:show=displayname,spell_list&c:resolve=spells(name,level,description,duration)

Just as you can use the syntax above to show fields, you can add a ! in front of the query to hide fields instead:

census.soe.com/xml/get/eq2/character/?c:show=displayname,spell_list&c:resolve=spells(!icon,!cost,!tier)

Controlling Output

Once you have written your query, you may wish to control the output, specifying which fields are returned, and sorting the results so that you can page through them.

Show

Only include the specified field(s) in the returned data.

?c:show=displayname,name,account,guild,world,skills

Hide

Exclude certain field(s) from the returned data.

?c:hide=achievements,spells,stats

Start

If you are paging through items, spells, characters, etc. you can use c:start to start with the Nth object within the results of a query.

?c:start=50

Limit

By default, any query will return a single result unless you specify a c:limit.

?c:limit=99

NOTE: The maximum number of results you can request at once is hard limited to 100 if you choose not to register a Service ID with Daybreak Games.

Count

The following query returns the number of level 95 characters:

census.soe.com/xml/count/eq2/character/?type.level=95

However you may want to perform a query AND get a count of the number of results in the same request. This query would return the first ten level 95 characters and also the number of level 95 characters found:

?type.level=95&c:limit=10&c:count=1

Returns:

<character_list count="57695" limit="10" min_ts="1358361070.009923" returned="10" seconds="0.596140">

<character dbid="1639738" displayname="?????? (Barren Sky)" id="1335736468794" last_update="1358361072.616647" playedtime="3154929" ts="1358361072.616647" version="1" visible="1">

.

.

</character_list>

Sort

You can sort the results of your query with the c:sort modifier. You may sort by more than one field in ascending or descending order. For example, to sort ascending on field 1 and then descending on field 2:

?c:sort=field1:1,field2:-1

For example, to get the 20 characters with the highest number of completed Collections, this query would be used:

census.soe.com/xml/get/eq2/character/?c:sort=collections.complete:-1&c:limit=20&c:show=displayname,type.level,world

To get a list of level 91 Grandmaster spell choices sorted Alphabetically:

census.soe.com/xml/get/eq2/spell/?tier_name=Grandmaster&level=91&c:sort=name:1&c:show=name,tier_name,level&c:limit=100

Reverse

If you need to reverse the order of the returned results in a query, c:reverse allows you to do so. The main use for this is to allow you to sort by newest first, but show the results in chronological order in your browser. That way you can see the 25 newest characters, logs, etc in order from oldest to newest. A simpler way to do this is to add a :-1 to the c:sort field, such as c:sort=name:-1.

?c:reverse=1

Attachments

Attachments are images or other files which are "attached" to a specific field. Generally this is returned as a url path to download the file.

?c:attachments=all

For EverQuest II, the valid categories for c:attachment are paperdoll, headshot, petpaperdoll. You may also specify all to return all available attachments. The format is generally a fully-qualified URL to the external file(usually an image) that can be used within that object with additional, useful metadata.

Explain (Performance)

Census consists of a cluster of five MongoDB servers. Searching the database against fields that are indexed typically returns results quickly. For example character names, item names, guild names, etc. are all indexed. However most fields are not indexed and thus queries that search against those fields will be somewhat to considerably slower. You can determine if one of your queries is using an unindexed field with the c:explain parameter.

?c:explain=1

If"cursor" is returned with a value of "BasicCursor", then the field you are querying is not indexed.

If "cursor" is returned with a value of "BtreeCursor..." then you are using indexes.

Every query, indexed or otherwise, returns the number of seconds that were required to run the query.

Case

c:case has been removed and is only provided here for historical reasons. All requests are now case-sensitive. Use either the i/string/ format or the lowercase fields for fastest performance when searching.

Since I can't create a new thread in the API section because I'm not special enough:

IoR Server status still missing from census.
RTT is missing as well, but it's only a temp server so it doesn't really matter.

I've sent an e-mail.

Isle of Refuge and Race to Trakanon now appear on the Server Status page.

I can't seem to open a new Thread related to API issues and there doesn't seem to be any way to contact someone about it that I can find, everything links back to here. So I'll just put this here and see if someone sees it.

Is the EQ2 Census API still maintained? I'm seeing lots of data integrity issues I can't seem to work out. This all seems to be related to things around the Crushbone->Maj'Dul server merge awhile back.

Three guilds i gather data for all contain inaccurate information about players within the guild, these are players who are playing even as of today. I have numerous characters who still show in the API as Crushbone server users even though they are actively playing on Maj'Dul

I also have several characters listed in the guild with their old character name from Crushbone but were forced to rename themselves during the migration. In the API these characters still show in the Guild by their old name, not their new name, however they are fine "in-game" and have played recently.

I'd be happy to provide a details list of all the weird data issues I"m seeing if there's someone i can work with.

Thanks!

This forum was shut down for some reason, but people are still allowed to reply to old threads. There is no official place for us to discuss the API. It is extremely discouraging.

Yes Census is still being maintained. There was a problem with the Kunark Ascending expansion and VERY few characters have exported since its launch 6 days ago.

One of the big problems remaining from the server merges is that all that old guild data is still there, so you have multiple copies of each guild in Census. You have to exclude all the retired servers from your search to get the valid ones. And any character that did not login after the migration will still show as being on the old server forever and ever.

Did this ever get fixed? My EQ2u player is not updated. nvm just the sig.

disregard

I think that the latest Quests are missing on Census.
I can not find KA quests and the new Erollisi quest "You Don't Bring Me Flowers" is missing too.

No quest data since mid-2016 is in Census. They removed it to make sure the Epic 2.0 quests wouldn't show up.

The quest data was updated on Census recently.

Posting this question here as I see no other place to do so!

I'm finally back to working on my app, but, for it to work properly I need to know if there's a way in EQ2 macros to force the execution of steps in the macro to be in order? In my testing, I see that if I surround a /who command between two /em commands, the /em's file both finish before the /who, which means in the log, the appear out the order I intended.

You cannot control the execution order of macros. You can do an external file though.

Thanks Feldon for the response. I'm not understanding what you mean by an 'external file', which I've only seen used to get get images for items and toons. What I'd like to do is get a /who list (which I can get from the chat log of course) with some thing in the chat log that indicates what kind of /who was requested. i.e. raid members, group members, guildies online, etc. Yes I use the appropriate /who variation to get the list in chat text, but that just brings back a list without any indication for what that list includes.

On a side note: In your earlier post on 2/25/15 titled 'Resolve', there's two little problems.
1) The link to browsing Collections still points to census.soe.com (on this forum, eq2wire forum is correct, I'll be using eq2wire from now on)
2) Below that link you've listed resolves for character, guild and world. In the collections world does not have a resolve for Status (i wish it did!) Am I missing something there?

As always, thanks for your help and time on answering questions!

My edit rights and moderation abilities were taken away here and the forum locked from new threads, so I am unable to update existing posts in this forum. I strongly encourage anyone interested in EQ2 Census to check out the EQ2 Census API forums I launched here (scroll down to the bottom):

https://forums.eq2wire.com/

As for your second issue, c:resolve can only used for situations where it was explicitly setup. In contrast, you can use c:join to join two (or even three) collections together. The format is a brain-bender and it took me a while to learn it, but the possibilities are pretty amazing. Please start a new thread on the EQ2Wire forums under Census and I'll be happy to answer. Here's the thread I created that begins to document C:Join:

https://forums.eq2wire.com/threads/creating-your-own-resolves-with-c-join.212537/

*dances in the thread just because I can!*