GamParse, TBL, and You

Discussion in 'The Veterans' Lounge' started by Beimeith, Dec 11, 2018.

Thread Status:
Not open for further replies.
  1. Beimeith Lord of the Game

    As of 2.0.0.28, that same log file I mentioned loads in 21.7 minutes.

    For those interested in such things, I am currently rewriting how fight objects are stored.

    A "fight object" is the core of how GamParse organizes most of the data it parses. Each fight object contains the data related to that fight:
    • A list of all Damage
    • A list of all Spells cast
    • A list of all Heals
    • A list of all Players participating in the fight
    • What time the fight started
    • What time the fight ended
    • What zone the fight took place in
    • Who got the killshot (if applicable) [This is newly added]
    • etc. etc.
    The problem is most of the internals of how this works are...bad.
    As an example: When a line of damage is parsed, it gets sent to AllocateDamage()* which looks at each fight in progress and if it finds a match, adds that damage to that fight via Fight.AddDamage(). (If there is no match, it creates a new fight object and then adds the damage).
    Fight.AddDamage() would then loop through the list of ALL players currently participating in the fight until it found a match, then it would update the TotalDamage associated with that player. (Or if the player isn't known, adds them to the list and then updated their total damage).
    This is bad because it wastes a lot of time looking for the right player.
    Now that same function does a lookup of that player ONLY, and updates their total. (Or if the player isn't in the list, adds them to the list and updates their total). That one small change resulted in the decrease in time it took to parse my reference log.

    *This is already changed from how it used to be. Originally damage would be added to a list, and then that list would be looped through adding each damage to each fight. While there was/is a complicated reason why it was done that way, I've since changed it to be better.


    Back to my point though, many things are being changed so that rather than looping through the huge lists that exist, they perform lookups of the things they need and are thus much faster.

    TL;DR

    Still no ETA, but I'm totally making it better.
  2. Pirlo Augur

    Ever since this program stopped working correctly my life has entered death and despair ;)
    Thanks for all you do and be sure to leave that donate button because it is worth the cost and I have no problem helping out a fellow wizard
    Khazad likes this.
  3. Bronut Augur

    Thanks, Beimeith !!..... You complete me.
  4. Beimeith Lord of the Game

    Oh, it's there:

    [IMG]
  5. Tucoh Augur

    Bro post a link to it here.
    Khazad and Danaleigh like this.
  6. Critts Augur

    Pirlo, Mintalie and Khazad like this.
  7. Beimeith Lord of the Game

  8. TheNado42 New Member

    Will happily donate once this is done. I am also in the land of leet dps and heal stats drought. Any chance a donation will get a spot on the beta squad? Am only passingly familiar with code but have ample time and 6 level 110 classes to test with :)
  9. Axxius Augur

    Perhaps I'm missing something, but why are you trying to keep it compatible with the old logs? Wouldn't it be much easier to:
    • make a 'final' (pre-TBL) v1 release that will not auto-update
    • make v2 parse only the TBL format
    That way anybody who cares about parsing old logs could use the final v1, and the rest 99.99% of us upgrade to v2. Wouldn't that save you a gigantic amount of work?
  10. Beimeith Lord of the Game

    Because the way 1.x used to parse many things was wrong and because there are many things it didn't used to parse at all (for old logs) that it now WILL parse (for old logs).
  11. Beimeith Lord of the Game

    As of 2.0.0.29 my log is loading in 20.9 minutes, down from 25.7 minutes in 1.6.1.7. Overall that's about an 18.5% decrease in time loading.

    Improvements are still ongoing and there are still some other things to do that might affect speed for better and worse.

    Do keep in mind that this really only starts to be significant with larger log files / logs with more fights. With a smaller reference log of 1125 fights that is 85MB the improvement was a mere 4% from 45.3s to 43.6s.

    Additionally, for you pet users: If your pet solos a mob it should now show the fight even if you don't damage it yourself. Be aware that this will NOT work with pets that use EQ-generated names.

    Specifically, it should work if the pet is named:

    1) [Playername]'s pet
    2) [Playername]'s ward (I count wards as pets)
    3) [Playername]'s warder
    4) [Player made name]*

    *If your pet uses a player assigned name, it will have to be linked to you at some point (once is enough) for it to work.

    It will NOT work if the pet is named:

    Gabaner, Jabanab, Zabober or any names like that.


    One other requirement is that it knows who the player is. In the vast majority of cases this shouldn't be a problem. GamParse identifies who is a player vs npc in multiple ways, (/who, chat, filename) and it stores it over time so the longer you use it the more accurate it should get. This is why on a fresh install/update/reset I recommend loading the biggest log you have first to let it learn who everyone is and then load up a normal file to parse.


    Also, the NPC DPS column on the Overview tab has been changed to show the Max Hit against that player. This was done because the NPC DPS column was basically worthless while knowing the Max Hit against players is actually useful information.

    Here is a current example:

    [IMG]

    You can see a couple of the changes:

    1) Dunedigger's name is red because he got the killshot.
    2) His damage (and Total) are slightly lower (-1013) because the duplicated fake damage from killshots has been removed. He also moved down from 5 to 6, (get , Bobby!)
    3) The Fight Navigator is resizable.
    4) The NPC DPS is now NPC Max (vs player)
    5) The Dmg Mods tab is now hidden by default due to it no longer being useful for new logs. There will be an option to make it visible again if you want it for older fights. (I did also previously change how that's calculated to be more accurate for what that's worth now, heh).
    6) It now shows the number of lines in your log on the top right of the menu bar.
    7) You can't see this one, but loading times in the loading bar at the bottom now show minutes and seconds vs just seconds. I know dividing seconds by 60 to figure out how many minutes is hard for some of you.

    All these good things and much, much more are coming. Soon(tm).


    Also, as an aside: Don't pay too much to the version numbers. They're mostly meaningless outside of the fact that higher usually means newer. Despite it being set to 29 now that doesn't actually mean 29 builds. It's more like 10x that. I really only increment it when I'm done for the day / commit, and I don't even do that reliably. I'm sure that's probably a bad practice but whatever, I do what I want.
  12. AlmarsGuides Augur


    You can probably get in trouble for that since these are Daybreak's boards. They have rules against self advertising.
  13. Beimeith Lord of the Game

    And then there is stuff like this, sigh:

    Code:
            public List<int> GetPlayers()
            {
                List<int> Players = new List<int>();
     
                foreach (Damage dam in MasterDamageList)
                    if (dam.Opponent == Opponent && !Players.Contains(dam.Attacker))
                        Players.Add(dam.Attacker);
     
                return Players;
            }
     
            public List<int> GetAttackers()
            {
                List<int> Names = new List<int>();
     
                foreach (Damage dam in MasterDamageList)
                    if (dam.Opponent == Opponent && !Names.Contains(dam.Attacker))
                        Names.Add(dam.Attacker);
     
                return Names;
            }
    Not only are these wasteful in that they loop through the entire DamageList, they do literally the same thing, just with a different name.
  14. Pirlo Augur

    I am no programmer by any means but even I can see the redundancy in that!
  15. Beimeith Lord of the Game


    While some of the bad code is my fault, this one is totally on Gam. I even double checked it by looking at the original source he sent me to make sure, lol.

    As far as a progress update goes, I just spent the last several hours rewriting a lot of the code related to the various settings...and I just now remembered why I was even looking at the settings in the first place.
  16. Angre Augur

    I was looking for a Paypal Donation Link on EQ Resource but was unable to find one for Gamparse.

    Am I looking in the wrong place?

    Angre
  17. Beimeith Lord of the Game


    There isn't one. Only Donate button there is is in the test version of the program.
  18. Beimeith Lord of the Game

    I thought I'd take a little time to go over some of the options and explain exactly what they do and how they work. Some of these many of you will know, others you probably have no idea.

    [IMG]

    Beimeith's guide to GamParse, Part 1 - General and Advanced Options.

    General Options

    1) Default Player

    EverQuest is not consistent in how it refers to the current player in the log. Some messages use the player name, others use "You" in various methods of capitalization. In order to know who "You" refers to in a log file GamParse needs to know WHO the current player is. The primary way it does this is to read it in the log file's name.

    A standard log file name looks like this:

    eqlog_Beimeith_Xegony.txt

    GamParse will read this name and pull out the first group of letters between underscores as the current player. In this example, "Beimeith".

    Under normal circumstances this is all that needs to be done and the Default Player option can be left blank and is superfluous. However, some of you guys like to rename your log files something stupid like "Some Parse Against this Mob.txt".

    When this happens GamParse doesn't know who made the log file and will then check if the Default Player option has been set. If it has been set it will use this name as a fallback. If there is no default player set, then it will give give a warning message letting you know.


    2) Default Server

    This works similarly to the Default Player option. GamParse will first try to read the server name from the file name, but if it cannot find a match it will look if one has been set as Default Server. If it cannot find one it will give a warning message letting you know.

    Server information isn't really used for much at this time. I added it mainly as a way to try and differentiate between logs made on live servers and logs made on servers not owned by DBG.

    3) Auto-monitor on startup

    This option when set will have GamParse start reading the specified log file immediately upon loading. It will NOT load information already stored in the log file but will only read new information. This is the same function as selecting "File > Monitor log file for new fights only" from the menu.

    Note that the first time you enable this option a pop-up box will appear asking you to select the file you want to be read.

    4) Backup at XXX MB

    This option requires Auto-Monitor to be set.

    This option when set will have GamParse check the size of the file set to auto-monitor when it starts and if the file is larger than the value set, will prompt you to backup the log file.

    Valid options are:

    50, 100, 250, 500, 750, 1000, 2000, 3000, 4000, 5000. The default is 500.

    Be aware that backing up with GamParse is preferable to using GINA. This is not a dig at GINA. The reason for this is that when GamParse backs up a log file it writes the current zone info into the new log file so that the information isn't lost.

    For best results you should close GINA, turn logs OFF in EQ, and backup the log file. Then re-enable logging in EQ and reload GINA.

    5) Backup Directory

    This option requires "Backup at XXX MB" option to be set.

    This option when set will backup GamParse to the specified directory. If this option is NOT set GamParse will prompt you to select a directory to save to.

    6) Check Every XXX Hours

    This options requires "Backup at XXX MB" option to be set.

    This option when set will check the file size of the current log file (if the current file is the file set as Auto-Monitor) after the interval specified. If the file is larger than the value set it will prompt you to backup the file.

    Valid options are:

    .25 (15m), .5 (30m), 1, 6, 12, 24. The default is 1 (hour).

    7) Don't Show 0 damage players on Overview

    This option when set will hide players who did not do any damage in the current fight from showing on the Overview Tab.

    The default is OFF.

    8) Auto-combine pets with their owners

    GamParse has the ability to show a combined value for players and their pets or to show them separately. This setting is changed with the "Split/Merge Pets" button on the Overview Tab. The "Auto-combine pets with their owners option" sets the default behavior for this setting.

    This option when set will determine whether pets are combined with their owners by default.

    The default is ON.

    9) Show DamageMod Tab

    EverQuest calculates Melee Critical Hits prior to calculating skill bonuses. Prior to the 2018-12-11 patch (TBL launch) Critical Hits displayed a separate message with this value followed by the actual damage value after skill bonuses were calculated. This resulted in a disparity between the values of the two messages and allowed GamParse to calculate the difference resulting from the skill used. A skill could either result in a bonus or a penalty depending on the resistance of the opponent. The DamageMod Tab displays this difference.

    With the 2018-12-11 patch, the Melee Critical Hit message was removed and rolled into the actual damage message. This combined message only displays the final hit value, thus rendering the DamageMod Tab obsolete. To this end I have defaulted the DamageMod Tab to be hidden. However, when loading older logs the DamageMod Tab can still be useful, thus I have added the option to show it.

    This option when set will cause the DamageMod Tab to become visible again.

    The default is OFF.

    10) Show Faction Tab

    Prior to the 2018-12-11 patch (TBL launch) there was no way to see your current faction standing in EverQuest. The Faction Tab was thus added as a way to help estimate your faction.

    With the 2018-12-11 patch adding a proper faction window to EverQuest the usefulness of the Faction Tab is diminished. However, the Faction Tab is still useful with regard to seeing the value of each individual change and seeing the changes over time. To this end I have defaulted the Daction Tab to continue to be shown, but have also added the option to hide it.

    This option when set will cause the Faction Tab to become visible.

    The default is ON.

    11) Random Timeout (minutes)

    The Randoms Tab allows you to track Manual rolls made via /Random. It does this by organizing all /randoms with the same limits (X to Y) into a "Random Set." A Random Set is created whenever a /random with new limits is detected. Any additional /random using the same limits will be added to the current Random Set until it is closed. Once a Random Set is closed, a roll with the same limits will create a new Random Set.

    If the first roll of 1 to 100 is seen at 12:00:00 and the Random Timeout is 5 minutes, ALL rolls of 1 to 100 from 12:00:00 until 12:05:00 will be grouped into the same Random Set. A roll of 1 to 100 seen at 12:05:01 will create a NEW Random Set.

    Note that this only applies to rolls with the same limits. If you want to have multiple different rolls at the same time simply use different limits.

    This option controls how long a Random Set is open.

    Valid options are .25 (15s), .5 (30s), 1, 2, 5, 10, 15, 30, 60. The default is 5 (minutes).

    Be aware that altering this setting will only affect NEW Random Sets. It will NOT affect previously rolled Sets unless you restart and reparse the log file after changing the setting.

    12) Random Amount to Send

    GamParse has several buttons that will place /Random data onto the computer's clipboard so it can be pasted into EQ:

    Top - The top X number of players.
    Bottom - The bottom X number of players.
    Middle - The middle one or two players.
    Top/Bottom - The highest and lowest rolls.
    Random - GP will pick one at random.
    Highlighted - Copy the highlighted lines.

    This option controls the maximum number of rolls to copy to the clipboard when using the Top and Bottom buttons.

    Valid options are 1, 2, 5, 10, 15, 20. The default is 10.

    13) Item Lookup Website

    GamParse can perform an item lookup for items shown in the Loots, Parcels, and AutoRoll Tabs. A search is triggered by doubleclicking on the line with the item you want to look up. This will open a browser window and perform a search using the website set for item lookups.

    This options controls which website to use to lookup items.

    Valid Options are EQResource, EQItems, Allakhazam, Lucy. The default is EQResource.

    Advanced Options

    This is a new window that deals with options that fundamentally alter how GamParse functions and under most circumstances should not be changed unless you fully understand what they do. Changing these settings will require a restart to take effect.

    1) End Fights With No Damage After XXX Seconds

    GamParse usually ends a fight when it sees a slain message. (XXX has been slain by YYY!) However, it is possible to either be out of range for these messages or to have fights where the opponent is NOT killed for lore reasons. In these circumstances GamParse will fall back to using a timeout period. If no damage (or attempt at damage) is detected after the timeout period GamParse will end that fight using the time of the last damage seen.

    If the last damage involving NPC1 is at 12:00:00 and the timeout period is set at 30 seconds, if there is no further damage (or attempt at damage) by 12:00:30 GamParse will end the fight and use the end time of 12:00:00. If at 12:00:31 damage is detected it will start a NEW fight.

    This option controls the Fight Timeout length.

    Valid options are 6, 12, 18, 24, 30, 36, 42, 48, 54, 60. The default is 30 (seconds).

    Be aware that setting values too low or too high can negatively impact how fights are ended. Too low a value can cause the same fight to be broken into multiple fights if there is a pause in the damage (kiting can cause this). Too high a value can cause multiple fights against opponents of the same name to be combined into a single fight. It is not recommended to alter this setting from the default.

    2) Don't Show Fights With Less Than XXX Damage

    This option controls the minimum damage GamParse requires against the opponent to process the fight and add it to the Fight Navigator/List.

    If the opponent in a fight has taken 99,999 damage and the minimum damage is set to 100,000 when the end of the fight is detected (slain message or timeout) the fight will NOT be processed and instead will be discarded.

    Valid options are 1, 10, 100, 1,000, 5,000, 10,000, 25,000, 50,000, 100,000, 250,000, 500,000, 1,000,000, 2,500,000, 5,000,000, 10,000,000. The default is 100,000.

    Be aware that setting too low a value can cause an excessive number of fights to be displayed such as when mass killing low level zones. Too high a value can cause fights not to appear in the list. It is not recommended to alter this setting unless you are low level or play on a progression server.

    3) Include Spells Before Fight Start

    This setting controls how far before a fight starts should spells be counted towards that fight.

    Previous versions of GamParse only showed spells that were cast AFTER a fight started. This resulted in a situation where pulling with a spell meant that spell would not be counted as the casting message was before the damage message and thus the fight's official start. It also meant that hitting burn abilities prior to the fight start would cause them not to be counted. Now GamParse can count spells cast prior to the start of the fight up to a set period. A value of 0 will not count spells cast prior to the start of the fight. (Though it will count spells cast in the same second the fight started).

    If the fight started at 12:00:00 and the timeout is set at 6 seconds, then spells cast 11:59:54 and later will be counted in the fight.

    Valid Options are 0, 3, 6, 9, 12, 15, 18. The default is 6 (seconds).

    Be aware that this does NOT alter the starting time of the fight. Using the example above the fight will STILL have a start time of 12:00:00, NOT 11:59:54.

    4) Fight Start AFTER

    This setting controls the start condition for a fight.

    By default a fight will start after the first damage, or attempted damage, *from or against* a known player.

    Valid Options are First Damage, First PC Damage, First Player Damage. The Default is First Damage.

    First PC Damage changes the condition to start the fight ONLY after the first damage (or attempted damage) FROM a known player. If no known player attacks the fight will not start even if they/you are being attacked by the NPC. First Player Damage changes the condition to start only when YOU first do (or attempt to do) damage. If YOU do not try to do damage the fight will not start.

    Be aware that changing this option is currently not allowed. This is the most dangerous setting and I simply don't trust many of you enough to change it at this time.

    5) Don't Count Killshot Damage

    EverQuest displays the full value of a hit on a killshot, not the actual damage value done. This results in a skewed value for the player that got the killshot.

    If an opponent has 1 HP remaining, but the final killshot hits for 1,000,000 points of damage, it will show as a 1,000,000 damage hit even though only 1 damage was done.

    This option controls whether GamParse will ignore killshot damage.

    The default is NO.

    6) Attempt to Ignore Pulls

    This option controlls whether GamParse will try to ignore pull damage.

    The default is NO.

    Under normal circumstances pulling with a damaging attack will cause GamParse to start the fight. However this is not always desirable such as when dealing with a long pull or no-damage kiting. In these situations there is an option to attempt to ignore pull damage.

    This function works as follows:

    A) If NO PLAYER has done damage to the opponent for longer than the Pull Timeout value, the fight is closed and discarded. (This is useful when proximity pulling and kiting).

    B) If YOU have done less damage than the Pull Damage Cutoff value within the Pull Timeout period the fight is closed and discarded. (This is useful when pulling with a low damage ability such as Throw Stone or a low level nuke).

    Example 1 with a Pull Timeout of 6 seconds and a Damage Cutoff of 2,500.

    YOU proximity aggro a mob that hits you.

    I) If NO PLAYER does ANY damage in 6 seconds the fight will end and be ignored.

    Example 2 with the same settings.

    YOU use Throw Stone against a mob and do 1 point of damage.

    I) If you do NOT do another 2,499 damage in 6 seconds the fight will end and be ignored.
    II) IF YOU attack the mob and do more than 2,499 damage in 6 seconds, the fight will NOT end.
    III) If SOMEONE ELSE attacks the mob, regardless of damage, the fight will NOT end.


    7) Pull Timeout

    This setting controls the timeout period for pulling as outlined above.

    Valid Options are 3, 6, 9, 12. The default is 6 (seconds).

    8) Pull Damage Cutoff

    This setting controls the damage threshold for pulling as outlined above.

    Valid Options are 2, 10, 50, 100, 500, 1,000, 2,500, 5,000, 10,000. The default is 2,500.

    9) Don't Show Error Popups (if possible)

    This option sets GamParse to NOT notify you when it encounters a known error.

    The default is NO. (It WILL notify you when there is an error).

    Be aware that unknown errors will show regardless of this setting.

    10) Don't Report Errors via Email

    This option sets GamParse to NOT send me an email when it encounters an error.

    The default is NO. (It WILL email me when there is an error).

    Be aware that GamParse does not send any personally identifiable information. The information it sends me is:

    1) Player Name
    2) Player Server
    3) The original, unaltered, line from the log file.
    4) The current working line from the log file.*
    5) The error message.
    6) The # of the line in the log.

    *In most cases the original and current lines will be identical. In rare situations a line is written to a log with errors in them. GamParse will attempt to fix these errors while reading them and if it does this can result in the current working line being different than the original line. Note that it DOES NOT alter the line saved in the log file itself. It only changes the line in GamParse.

    An example of this was in May of this year a bug in EQ caused the date to be written to the log file incorrectly. The date was saved like this:

    [Tue May 1 00:00:03 2018] instead of like this: [Tue May 01 00:00:03 2018].

    The missing 0 in the date caused older versions of GamParse to generate an error because it was looking for a 2 digit date when there was only 1. This type of error was already fixed (if you were running the current version) and when GamParse read the faulty line it added the 0 back to it. (Again, this is ONLY within GamParse itself. It did/does NOT alter the log file itself).

    I really recommend you don't opt out of this. It helps immensely for me to get error reports.

    Here is a screenshot of an error report:

    [IMG]
  19. Krool Lorekeeper

    Getting excited for the new version! :) Can't wait to see the finished product of all your hard work!!
  20. Jaay Lorekeeper

    Im going somewhat crazy without the parser, Is there any kind of ETA when you will have a version up and running? =) =)
Thread Status:
Not open for further replies.