The Data API

Discussion in 'Developer Announcements' started by Feldon, Feb 25, 2015.

  1. Feldon Well-Known Member

    Introduction

    Daybreak Games provides access to game data through a RESTful API called Census. This system is a centralized service that receives character, world, and static information from game servers so that it may be accessed by various applications. These include Daybreak Games websites, Game Launchpads, 3rd Party Fan sites and mobile companion apps. Access to this data is available at census.soe.com, which is available using either HTTP or HTTPS.

    Census allows you to request information directly from Daybreak Games regarding the status of a game or game entity (e.g. character, item, spell, etc.). All requests to the REST interface follow a pattern and there are different patterns for different types of data. The REST interface will return all results that match the attributes provided within the query string. Data is returned is in the form of a list containing one or more items from the collection provided in the URL. If no results are found, this container will be empty.

    The URL Structure is as follows. Any part of the URL within brackets ("[]") is optional and may be omitted):

    /[s:serviceID]/[format]/{verb}/{game}/{collection}/[{identifier}][?{queryString}]

    Service ID

    A Service ID or Identifier is a short word or phrase associated with your website or your name which is used by Daybreak Games to monitor and provide further capabilities to sites who use the API. A Service ID is not currently required, but is encouraged. Your service must be registered with Daybreak Games to use this facility. The first step to getting this done is to request a Service Key.

    If you do not have a service ID, just omit the "s:serviceID" part of the URL:

    census.soe.com/json/get/eq2/item/2906185062

    If you have a service ID, provide it as below:

    census.soe.com/s:serviceID/json/eq2/item/2906185062

    We are encouraged to use the Service ID both in data queries and in image searches. For example this image:

    census.soe.com/s:eq2wire/img/eq2/icons/600/item/

    HTTP (GZIP) Compression

    If you are making calls to the REST API from a server process, you should consider enabling HTTP Compression on your request. This is already enabled for most browsers, but you need to set an HTTP header when requesting through code.

    Accept Encoding: gzip

    This will reduce the transfer size of a character request dramatically.
    More information about HTTP Compression may be found here.

    PHP 5 should handle this automatically if you are using get_file_contents, etc.

    If you are using PHP's cURL to make your data requests, then you must set an empty CURLOPT_ENCODING to automatically send the list of all supported encoding types in the request header (which will include gzip). Otherwise, no Accept-Encoding header gets sent.

    curl_setopt($curlResource, CURLOPT_ENCODING, "");

    Data Format

    Census supports returning data in eXtensible Markup Language (XML) and JavaScript Object Notation (JSON). The recommended format is JSON as it is a very efficient way to access data for most applications. If the XML format is requested, then the results will be provided in a parent <results> element. If JSON results are requested, they will be in a dictionary( {} ).

    Using a plugin such as JSONView, JSON data can be presented in a "beautified" format that is easier for humans to read. This means that you don't have to run the output through a plugin or filter just to make sense of the JSON data. JML is particularly useful when troubleshooting queries as you may hover over any field and the data path will pop up in a tooltip.

    The acceptable format values are:
    • xml
    • json
    In order to access data across the domain boundary, one may want to use JSON Padding3 (JSONP). To do this, append a callback attribute to the query string of the request that uses the json format.

    /json/get/eq2/item/2906185062?callback=foo

    The results from this request will be padded within the function name, foo(), so that they may be evaluated by the requesting web site within the correct domain.

    Verbs

    The verb or "action" informs the REST interface about the type of request that is being made. Supported verbs are...
    • get - retrieve the objects that match the request criteria
    • count - retrieve the number of objects that match the request criteria
    • status - retrieve status information (currently used for server status only)
    Game

    In our case, we are using eq2 for EverQuest II.

    Collection

    Supported collections are dynamic and dependent on the game providing data to the service. Collections might include Characters, Items, Spells, Guilds, etc. The list of available collections may be viewed here:

    http://census.soe.com/xml/get/eq2/

    Some data may be omitted from a collection due to certain policies set by each game team to protect player privacy and/or business requirements. Examples of these policies might include:
    • Characters may be marked as private so that they are omitted from searches and rankings.
    • Items may be hidden until they are "discovered" naturally by a player.
    Identifier

    Each object has a unique identifier, which is used for a direct link to the information about that object. This can be found within the structure by the top-level “id” field. All results will have this identifier, even if all other information is hidden.

    There are also identifiers that are used in reference to other collections. For example, consider the following collection information from eq2:

    "appearanceslots": [
    {
    "item": {
    "id": 3098863614
    },
    "id": 2,
    "name": "head"
    },
    {
    "item": {
    "id": 2334025620
    },
    "id": 3,
    "name": "chest"
    }
    ]

    The item.id value is a reference to a particular object within the "item" collection. When presenting this information, one can put a link to another query to the REST interface or to resolve that information within its parent object. For more information on how to do this, see the Query Commands section for c:resolve.
  2. Feldon Well-Known Member

    Game Images (Attachments)

    Game images are also available through this service. These include images representing characters (paperdolls, headshots) or in-game items or abilities. Images are typically returned as 24-bit PNG images with 8-bit transparency. Requests for game images follow this pattern:

    /img/game/collection/identifier[/imageType]

    For example, a certain piece of armor may have this icon:

    census.soe.com/img/eq2/icons/3635/item/

    Each result within a collection can have a number of "attachments" - usually an image, but this may be any file. For games that support many attachments, there can also be a category and items within each category. The category and the ID for items within each category is game-specific.

    When retrieving images from the icons collection, the supported values for imageType are:
    • item - a 42x42 transparent PNG item icon
    • spell - a 42x42 transparent PNG spell icon
    NOTE: The use of the icon collection is deprecated. Use icons instead.

    When retrieving images from the character collection, the supported values for imageType are:
    • paperdoll - a full picture of the character
    • headshot - a picture of the character of from the neck up (suitable for icons)
    • housethumb - a picture of a in-game house as published
    • petpaperdoll - a full picture of the character's pet
    You may use the c:attachments directive in your queries and it will include URLs to the various images directly into the information returned for that object. These URLs will use the Game Image pattern and correct imageType.

    A field named itemInCategory is returned if there is more than one attachment within a category. At this time, the only example of this is for a player who owns multiple houses. Their character will have multiple housethumb attachments.
    FriggaWitch likes this.
  3. Feldon Well-Known Member

    The Daybreak Games Data Feeds API (Census) runs on a Mongo Database system which is a highly optimized NoSQL solution. Users wanting to request data from Census do so with RESTful API calls. In other words, GET queries where each parameter either requests certain data or filters the resulting data to refine your results. As with any GET query, the first parameter must be preceded by a question mark (?) and subsequent parameters must each be preceded by an ampersand (&) character.

    This query would return all level 80 items of type Weapon with a quality (tier) of MYTHICAL.

    census.soe.com/xml/get/eq2/item/?leveltouse=80&type=Weapon&tier=MYTHICAL

    Searching

    Field Values

    Each request for data may be filtered by the value of certain "fields". Each field in a query string corresponds to a particular field within a particular object within a collection (characters, spells, items, etc.).

    Within Census data, fields have a hierarchy or level structure. When accessing a field that is parented under another field, you may reference it by using a dot (".") notation. For example, considering the following data:

    "item": {
    "id": 4126241497,
    "typeinfo": {
    "equip_optional": 1,
    "name": "expendable",
    "casttime": 1.0
    }
    }

    In order to find all matches for the name "expendable" within typeinfo, one would use the following query string:

    ?typeinfo.name=expendable

    Numeric Fields
    When more than one search condition is provided within the query string, on a number, it acts as a boolean 'AND' operation. For instance, consider the following query string:

    ?id=]10&id=[50

    This request would return data where the value of the id attribute was between 10 and 50 (inclusive of 10 and 50).

    Text Fields
    When more than one search condition is provided within the query string on a text string, it acts as a boolean 'OR' operation. For instance, consider the following query string:

    ?given_by=racialtradition&given_by=focusabilities

    This request would return data where the value of the given_by attribute is EITHER racialtradition OR focusabilities.

    A complete list of search modifiers appears below:

    Modifier Example Description
    field=<10 --- Include objects where the value is less than 10.
    field=[10 --- Include objects where the value is less than or equal to 10.
    field=>10 --- Include objects where the value is greater than 10.
    field= ]10 --- Include objects where the value is greater than or equal to 10.
    field=/string/ --- Include objects where the value contains a particular "string".
    field=i/string/ --- Case insensitive search for a particular "string".
    field=^string --- Include objects where the value starts with "string".
    field=string$ --- Include objects where the value ends with "string".
    field=!string --- Include objects where the value does NOT match "string".
    field=*regex --- Use a regular expression to match values within the field.

    Regular expressions or "regex" provide a great deal of power for searching for objects. Census supports Perl-compatible regular expressions. The "/" and "^" modifiers above are actually examples of simple regular expressions. It is beyond the scope of this document to fully explain regexes, however there are a number of resources available to learn about them, including here and here.

    NOTE: Use of i/string/ for case-insensitive searching of Character, Guild, Item, and Spell names is deprecated in favor of standard searches against their lowercase equivalents. Search performance will be substantially improved by using these:
    • character.name.first_lower
    • item.displayname_lower
    • guild.name_lower
    • spell.name_lower
    Has

    The presence or absence of certain fields in data can be just as valuable as the fields themselves. For instance, the following query returns items that have the typeinfo.growthdescription field (eg. green adornments or Spirit Stones):

    ?c:has=typeinfo.growthdescription

    There is no separate "Has Not" or "Doesn't Have" query. Instead, if you need to find entries where certain fields are missing, add a ! in front of certain fields:

    ?c:has=typeinfo.growthdescription,!growth_table

    Distinct

    Sometimes you wish to know all the distinct values for a specific field. For instance, to find all possible qualities (tiers) of items:

    census.soe.com/xml/get/eq2/item/?c:distinct=tier

    NOTE: c:distinct may not have good performance on extremely large collections. It should be used to build internal cheat sheets/lists to work against. C:distinct is exceptionally slow when used against the Character collection.

  4. Feldon Well-Known Member

    Resolve

    Some collections have related data which may or may not be needed when accessing a collection. For instance, when you retrieve a Character, a stripped down list of Spells and Achievements are included, however the underlying data such as the name, description, and stats for these data points is not. If you wish to add this data to your query, you may use a c:resolve to "join" collections together.

    Example of Resolves for Guilds

    Before EQ2 started adopting GUIDs (globally unique IDs) for Characters and Guilds, the game used a combination of DBID and WorldID to specify a unique character or guild. The same DBID could exist on multiple servers, so you had to use both DBID and WorldID. When you perform a standard 'pull' of a Guild, you only get the DBID of each character. For example, a guild named Honor on the Permafrost server:

    census.soe.com/xml/get/eq2/guild/?name=^Honor&world=Permafrost

    <member_list>
    <member dbid="208021" name="Idemu"/>
    <member dbid="221484" name="Gige"/>
    <member dbid="263553" name="Klehost"/>
    .
    </member_list>

    To also get the GUIDs of characters in a guild, you'll need to add a c:resolve:

    census.soe.com/xml/get/eq2/guild/?name=^Honor&world=Permafrost&c:resolve=members

    <member_list>
    <member dbid="208021" id="867583601813" name="Idemu"/>
    <member dbid="221484" id="867583615276" name="Gige"/>
    <member dbid="263553" id="867583657345" name="Klehost"/>
    .
    </member_list>

    NOTE: Characters that have not logged in since Nov 2011 will not have a GUID.

    You can expand your c:resolve to load additional Character data besides the GUID as demonstrated below:

    census.soe.com/xml/get/eq2/guild/?name=^Honor&world=Permafrost&c:resolve=members(displayname,type)

    <member_list>
    <member dbid="208021" displayname="Idemu (Permafrost)" id="867583601813" name="Idemu">
    <type alignment="0" birthdate_utc="1100033940" class="Ranger" classid="39" deity="None" gender="male" level="86" race="Iksar" raceid="10"/>
    </member>
    <member dbid="221484" displayname="Gige (Permafrost)" id="867583615276" name="Gige">
    <type alignment="0" birthdate_utc="1100190443" class="Guardian" classid="3" deity="None" gender="male" level="48" race="Gnome" raceid="5"/>
    </member>
    <member dbid="263553" displayname="Klehost (Permafrost)" id="867583657345" name="Klehost">
    <type alignment="0" birthdate_utc="1102637984" class="Wizard" classid="23" deity="None" gender="male" level="50" race="Half Elf" raceid="6"/>
    </member>
    .
    </member_list>

    The current list of resolves can be found by browsing Collections for <resolve>:
    • Character
      • Spells
      • Factions
      • AppearanceSlots
      • EquipmentSlots
      • Achievements
      • Warders
      • Statistics
      • Dungeon_Items
    • Guild
      • Members
    • World
      • Status
    NOTE: Resolves such as c:resolve=spells or c:resolve=achievements when performed against one or more Characters can take several seconds to perform and also generate a substantial amount of data. If you are feeling really lucky, you may do a c:resolve=all to return as much data as is possible.

    Limiting Fields in a Resolve

    You can limit the returned data within a c:resolve by indicating certain fields in parentheses:

    census.soe.com/xml/get/eq2/character/?c:show=displayname,spell_list&c:resolve=spells(name,level,description,duration)

    Just as you can use the syntax above to show fields, you can add a ! in front of the query to hide fields instead:

    census.soe.com/xml/get/eq2/character/?c:show=displayname,spell_list&c:resolve=spells(!icon,!cost,!tier)
  5. Feldon Well-Known Member

    Controlling Output

    Once you have written your query, you may wish to control the output, specifying which fields are returned, and sorting the results so that you can page through them.

    Show

    Only include the specified field(s) in the returned data.

    ?c:show=displayname,name,account,guild,world,skills

    Hide

    Exclude certain field(s) from the returned data.

    ?c:hide=achievements,spells,stats

    Start

    If you are paging through items, spells, characters, etc. you can use c:start to start with the Nth object within the results of a query.

    ?c:start=50

    Limit

    By default, any query will return a single result unless you specify a c:limit.

    ?c:limit=99

    NOTE: The maximum number of results you can request at once is hard limited to 100 if you choose not to register a Service ID with Daybreak Games.

    Count

    The following query returns the number of level 95 characters:

    census.soe.com/xml/count/eq2/character/?type.level=95

    However you may want to perform a query AND get a count of the number of results in the same request. This query would return the first ten level 95 characters and also the number of level 95 characters found:

    ?type.level=95&c:limit=10&c:count=1

    Returns:

    <character_list count="57695" limit="10" min_ts="1358361070.009923" returned="10" seconds="0.596140">
    <character dbid="1639738" displayname="?????? (Barren Sky)" id="1335736468794" last_update="1358361072.616647" playedtime="3154929" ts="1358361072.616647" version="1" visible="1">
    .
    .
    </character_list>

    Sort

    You can sort the results of your query with the c:sort modifier. You may sort by more than one field in ascending or descending order. For example, to sort ascending on field 1 and then descending on field 2:

    ?c:sort=field1:1,field2:-1

    For example, to get the 20 characters with the highest number of completed Collections, this query would be used:

    census.soe.com/xml/get/eq2/character/?c:sort=collections.complete:-1&c:limit=20&c:show=displayname,type.level,world

    To get a list of level 91 Grandmaster spell choices sorted Alphabetically:

    census.soe.com/xml/get/eq2/spell/?tier_name=Grandmaster&level=91&c:sort=name:1&c:show=name,tier_name,level&c:limit=100

    Reverse

    If you need to reverse the order of the returned results in a query, c:reverse allows you to do so. The main use for this is to allow you to sort by newest first, but show the results in chronological order in your browser. That way you can see the 25 newest characters, logs, etc in order from oldest to newest. A simpler way to do this is to add a :-1 to the c:sort field, such as c:sort=name:-1.

    ?c:reverse=1

    Attachments

    Attachments are images or other files which are "attached" to a specific field. Generally this is returned as a url path to download the file.

    ?c:attachments=all

    For EverQuest II, the valid categories for c:attachment are paperdoll, headshot, petpaperdoll. You may also specify all to return all available attachments. The format is generally a fully-qualified URL to the external file(usually an image) that can be used within that object with additional, useful metadata.

    Explain (Performance)

    Census consists of a cluster of five MongoDB servers. Searching the database against fields that are indexed typically returns results quickly. For example character names, item names, guild names, etc. are all indexed. However most fields are not indexed and thus queries that search against those fields will be somewhat to considerably slower. You can determine if one of your queries is using an unindexed field with the c:explain parameter.

    ?c:explain=1

    If"cursor" is returned with a value of "BasicCursor", then the field you are querying is not indexed.
    If "cursor" is returned with a value of "BtreeCursor..." then you are using indexes.

    Every query, indexed or otherwise, returns the number of seconds that were required to run the query.

    Case

    c:case has been removed and is only provided here for historical reasons. All requests are now case-sensitive. Use either the i/string/ format or the lowercase fields for fastest performance when searching.
  6. Alphonsus Well-Known Member

    Since I can't create a new thread in the API section because I'm not special enough:

    IoR Server status still missing from census.
    RTT is missing as well, but it's only a temp server so it doesn't really matter.
  7. Feldon Well-Known Member

    I've sent an e-mail.
    Prissetta likes this.
  8. Feldon Well-Known Member

    Isle of Refuge and Race to Trakanon now appear on the Server Status page.
    Evguenil62 likes this.
  9. Trivarian New Member

    I can't seem to open a new Thread related to API issues and there doesn't seem to be any way to contact someone about it that I can find, everything links back to here. So I'll just put this here and see if someone sees it.

    Is the EQ2 Census API still maintained? I'm seeing lots of data integrity issues I can't seem to work out. This all seems to be related to things around the Crushbone->Maj'Dul server merge awhile back.

    Three guilds i gather data for all contain inaccurate information about players within the guild, these are players who are playing even as of today. I have numerous characters who still show in the API as Crushbone server users even though they are actively playing on Maj'Dul

    I also have several characters listed in the guild with their old character name from Crushbone but were forced to rename themselves during the migration. In the API these characters still show in the Guild by their old name, not their new name, however they are fine "in-game" and have played recently.

    I'd be happy to provide a details list of all the weird data issues I"m seeing if there's someone i can work with.

    Thanks!
  10. Feldon Well-Known Member

    This forum was shut down for some reason, but people are still allowed to reply to old threads. There is no official place for us to discuss the API. It is extremely discouraging. :(

    Yes Census is still being maintained. There was a problem with the Kunark Ascending expansion and VERY few characters have exported since its launch 6 days ago.

    One of the big problems remaining from the server merges is that all that old guild data is still there, so you have multiple copies of each guild in Census. You have to exclude all the retired servers from your search to get the valid ones. And any character that did not login after the migration will still show as being on the old server forever and ever.
  11. Hartay Active Member

    Did this ever get fixed? My EQ2u player is not updated. nvm just the sig.
  12. Hartay Active Member

    disregard
  13. Evguenil62 Active Member

    I think that the latest Quests are missing on Census.
    I can not find KA quests and the new Erollisi quest "You Don't Bring Me Flowers" is missing too.
  14. Feldon Well-Known Member

    No quest data since mid-2016 is in Census. They removed it to make sure the Epic 2.0 quests wouldn't show up.
    Evguenil62 likes this.

Share This Page