GamParse, TBL, and You

As most of you should know by now, and doubly so for those who have been on Beta, current versions of GamParse will not correctly read log files. I tried to have an update ready before TBL launch, but it didn't happen.

The last live patch introduced the new healing output, but the update I made (1.6.1.7) mostly just hid the errors rather than actually fixed them.

Once TBL goes live however, nearly all parsing will break. You won't get error messages, but it should be fairly obvious that things are not parsing correctly. And just to be perfectly, painfully clear: Even if it LOOKS like it is working correctly, IT ISN'T. THE DAMAGE/HEAL VALUES YOU SEE WILL BE WRONG.

Over the last month I've been rewriting almost all of the parsing functions, as well as changing many of the core data structures to hold extra information. Not to mention cleaning up the code, fixing some long standing issues, and other enhancements and improvements.

While a good portion of this is related to TBL changes, a not insignificant portion is also aimed at increasing speed parsing large files. I can't promise a huge improvement as it really depends how dense the file is. (By "dense" I mean number of fights and the amount of parsable information).

As an example: My 1.73GB test file with 15,359 fights takes ~25.7 minutes to load in 1.6.1.7, and currently loads in 23.2 minutes with my test build 2.0.0.25, which is about a 10.5% improvement. Since I'm not finished this may not reflect the final results.

Additionally it should be noted that parsing older logs like my test file will be slower than parsing newer files since some of the optimizations favor new logs. Since you can't parse new files with the old parser I can't give you a speed comparison. (If someone can provide me with a log file of a specific raid prior to last patch, and a NEW log file after TBL launch of the SAME raid fight, I can use that as a comparison).

Suffice to say that when I'm done it should be great, but it's not done yet.

TL;DR

I am aware of the issues and I'm working on it. I don't have an ETA when it will be ready, but expect a test/beta version Soon(tm).

I plainly understand that Beimeith, by looking at the new log files. I made my own offline parser (that's much easier) to put emphasis on damage sources (which DoT, spell, etc per character), and I know I will have to rewrite the parsing process entirely. All this will be a mess for some time, but I suppose the end result will be better. Thanks for your work.

Parsing the new stuff is easy. It's keeping it compatible with 20 years worth of older logs that use different formats is what's hard.

Realistically I doubt there are more than 1-5 people that could even write a parser that does everything GamParse does and keep it accurate with older files. Not because writing the code is hard, but knowing all the different ways EQ works + used to work in the past + all the ways things are stupid in the logs is just something people don't know. I know there are things in old logs GP doesn't handle either, but until I can get examples I can't fix them, lol.

Thanks for continuing to work on this, Beimeith. This is such a useful tool for such a large population of the game.

As a long time, veteran IT guy, you have both my understanding and respect for the effort it has taken to dream up, implement and maintain the GameParse code base.

Thank-you!

- Coag

'nuf said =) Thank you for your continued work maintaining this valuable tool for our community.

Thanks Beimeith!

Thank you for the work you do for the EQ family. thanks

I am entirely on board with a new-logs-only version if that's easier to release

Yep - just freeze the current version and keep it available for historical purposes (reading old log files).

If the logs are that different, there doesn't seem to be much point in trying to make one version read all kinds of files: the data was different, so the parses will always be different.

EDIT: I forgot to say thank you, Beimeith!

Or even better, a menu selection/button, before loading the log file. One setting for old, one for new, and then no need to maintain two versions of GamParse

The main line will always be both, but I may at some point make an optional version of each that is optimized separately, or add the option to disable parsing of certain sections. One thing ive been doing wherever I can is separate current parsing from older parsing so it will be easier to turn sections on or off and/or make things optional.

The test version currently has counters in it for each section so I can see which is being hit more and can try to optimize it further. Melee is by far the most, about 30% of the log, followed by spell casts. This is for older logs though I haven't checked against new.

Thank you 1) for all the work you have put into this, and 2) for communicating with us about this.

Other developers (and anyone else who produces anything for public consumption) should be taking notes... This kind of communication goes a very long way in pacifying your audience/clients/customers/whoever.

... and Beimeith (I believe) doesn't even get paid !

It helps I don't have a boss and can do whatever I want whenever I want to.

I mean, I started out just updating the current branch, then I said to hell with this I'm just going to rewrite it all from scratch and if it isn't ready by launch sucks for them. Not everyone can just do that, lol.

Nope, though I may put a link to donate to me in it for those that want to.

You are amazing and appreciated for the work you do. Maybe just keep the current version of GamParse available for old logs and create a version that works on new logs. But whatever you come up with, I'll be grateful when it comes out.

Please put a paypal donate button. Id happily donate for your time, considering how much high end players really depend upon gamparse

Compatibility mode settings! heh

One thing I will say:

If you haven't already started a new log for expansion launch (always a good idea) I'd recommend you do that now.

For reasons that are dumb parses from patch day / the day before may end up not reading right even after I'm finished.