Feedback: Add Auto Trimming of Log Files

Discussion in 'The Veterans' Lounge' started by Fanra, Jan 25, 2024.

  1. Fanra https://everquest.fanra.info

    I'm tech support for my guild. Members come to me and say how the game is lagging badly and they have trouble, especially during raids.

    I connect to their computers and often find out the major problem is their log files are huge.

    EverQuest itself, with 30 expansions, takes up about 16 GB of hard drive space. The last person I helped, their log file alone was 27 GB. The raid can be won and it would take two minutes before their screen shows the mob dead.

    Logging needs to be on to use GINA. EQ raiding without GINA is impossible for many players. GINA itself contains, if you can find it, a setting to help keep the log file trimmed. The problems are, first, finding this setting, and second, it seemed that it was set on this person's PC to auto-archive at 100 MB but for some reason it did not. Maybe the file was too large before that setting was done.

    Regardless of why this happens, the best way to deal with this is to have EQ itself take care of this problem.

    How? Simple. Every time EQ starts, have it check the log files.

    Two ways to do this, either one would work:
    1. By size. If the log file is greater than, say 50 MB in size, rename it. This will then create a new log file to be used from then on. You can rename the old one from eqlog_Fanra_bertox.txt to perhaps eqlog_Fanra_bertox_2024_01_25_19_03_22.txt.
    2. By date. Every month (or two weeks) rename it. A new log file will be automatically created. Same type of renaming.
    There can be a setting to disable this for those that wish. And, yes, trimming logs should be on by default. The players that can handle taking care of the log files by themselves are the ones that can handle changing the setting. Do not expect the non-technical players to do technical things like dealing with settings. If you are so good you can deal with the log files by yourself, then you are good enough to disable this.

    Now, I will get the inevitable posts from people saying that all players should know how to take care of log files by themselves. The reality is that many EQ players cannot deal with this stuff. There are posters here that will say that is the fault of those players, etc. Blaming people does not solve problems. If you want players to get frustrated with EQ and quit, then continue to blame the players for not doing this kind of thing.

    If you don't automate these kinds of things but demand that players go into their EQ folders and rename or delete files, or change ini settings, or even settings in Options, or any kind of "technical" things, there are many who just won't. They don't know how, or often, they don't even know this exists. They will instead get frustrated and quit EQ. EQ cannot afford to lose players.

    Thank you for your time and consideration.
  2. Windance Augur

    I don't want EQ to automatically do anything to the log files.

    I'd rather see them add commands to remove or rename the log file from inside the game then give a WARNING when you turn logging on and have a large file.

    Then you can tell your guild members to remember to /deletelog before each raid.
    Fenthen likes this.
  3. Fanra https://everquest.fanra.info

    For those like you, then go into Settings and uncheck the Auto-Trim Log Files setting.

    Do you deal with humans?

    [IMG]

    I deal with people who don't know how to get to their log folder. If they manage to get there, then they worry about what to delete and whether it is safe to do so.

    The level of ability to cope with anything at all involving their PC is non-existent. They know how to turn on their PC and click the EQ icon to play. Asking them to do more is beyond their abilities and desires.

    As I said in my post, we can't afford to lose more EQ players by demanding they do more maintenance. They will quit first.
    Xyroff-cazic., Dre., Brildon and 6 others like this.
  4. Windance Augur

    Those same people know enough to type in game commands to turn their logs on.

    Give them a command to do so it game and they will be fine.
    MasterMagnus and Nennius like this.
  5. Windance Augur

    I'm much more worried about these folks "fixing" their graphics issues.

    Trying to lead them down through editing your eqclient.ini is MUCH worse then finding a simple log file.
    Dyna2 and Toukan like this.
  6. Scila Augur

    I'm going to agree with both on this one. Asking players to edit files is risky. There are three kinds of players. The ones that know exactly what to do and how to do it, the ones that "think" they know and don't, and the ones that don't do Windows updates or have virus protection. Leaving things up to players or ppl like Fanra to help takes the fun out of why we log in. I'm sure Fanra would much rather log in and play but out of the goodness of who they are they help so everyone can play. There has to be a better way.
    Rijacki, MasterMagnus and Jack like this.
  7. MasterMagnus The Oracle of AllHigh

    I agree with everyone saying this should be fixed by the company not workarounds.

    Just wanted to drop this in as a reminder. Section 5 (Wizards do it on schedule) details Windows Task Scheduler to automate backup and deletion of log files.

    https://forums.daybreakgames.com/eq...ement-socials-how-to-updated-9-2-2021.277236/

    I believe Windows Task Scheduler is also capable of executing a scripted series of actions based on file size in addition to very flexible timing schedules. It's been a while since I looked at it, but maybe this could help some of you?
  8. CatsPaws No response to your post cause your on ignore

    This is a valid point and one the powers that be need to pay attention to. It does not matter if your super tick tocker or old school one finger typing - no one wants to or will spend the time to do computer maintenance of this type to uphold this game.

    It's like oil changes on your car. Not many people do their own anymore. I used to long ago but its much easier to have someone else do it now.

    A few downloads, maybe some patches - all that is fine as its automated. I just spent many many hours running thru all the possible "fixes" for this most recent snafu. Made so many changes I had no idea where I was. Nothing worked. Then the day after the test was patched, I log into live and its actually livable. I can see and move. I did nothing. So all those hours were wasted. Long ago I told myself don't do any troubleshooting for at least a week after they mess stuff up cause it will magically fix itself. And it did again.
    MasterMagnus likes this.
  9. Waring_McMarrin Augur

    If they do add some sort of log trimming I would like to see it done as a rotation where you choose the criteria for a new log being created such as size, date or manual trigger and it moves the old log to a backup and creates a new one. As part of this they could also include the ability to compress old files and specify how long they are kept before deleting which could be forever.
    Fanra and Rijacki like this.
  10. FyrasDrinal Journeyman

    batch file I wrote to archive log files before I start EQ

    @echo off

    set SRC=C:\EverQuest\*inventory.txt
    set DST=C:\EQ\Inventory
    copy %SRC% %DST%

    set SOURCE=C:\EverQuest\Logs\eqlog*.txt
    set DEST=C:\EQ\LogArchive

    rem Expansion problems necessitate setting variables outside if block

    rem Convert date and time into directory names
    set SDATE=%DATE:~10,4%%DATE:~4,2%%DATE:~7,2%
    set STIME=%TIME:~0,2%%TIME:~3,2%%TIME:~6,2%
    rem echo date=%DATE% and sdate=%SDATE%
    rem echo time=%TIME% and stime=%STIME%

    set FOLDER1=%DEST%\%SDATE%
    set FOLDER2=%DEST%\%SDATE%\%STIME%

    if exist %SOURCE% (
    rem If ./date folder does not exist, create it
    if NOT exist %FOLDER1% (mkdir %FOLDER1%)

    rem If ./date/time folder does not exist, create it
    if NOT exist %FOLDER2% (mkdir %FOLDER2%)

    rem Copy all log files to ./date/time folder
    copy %SOURCE% %FOLDER2%

    rem Delete existing log files
    del %SOURCE%

    )
    jeskola and MasterMagnus like this.
  11. Rijacki Just a rare RPer on FV and Oakwynd

    Simple log rotation should be a part of EQ as a default setting. Each time the game starts and the log is past a certain (configurable with a semi-low default) size threshold, it should be renamed (a date appended) so a new file will start. Calling a function 'trimming' would be alarming to the non-techie person. But, it's part of other games including, I dimly recall and might be wrong, EQ2 so they did see the need for it. I also think logging should be on by default.

    Ideally, though, I think there should be more than the current megalith log file, i.e. there should be separate logs for different types of features such as a chat log for the say, tell, channel, emote, etc. type stuff and a combat log that's just all the auto generated stuff from combat. And they should be able to be enabled/disabled separate. So, if someone was concerned about having past conversations in files on their own computer, they should be able to disable the chat log while keeping the combat log. And so on. BUT, I know that's a tall order/request for an old game.

    And on a similar subject, it should be easier to copy/paste to/from the game windows. Explaining the ctl+ commands to someone who has only used the Windows insert/copy commands can be daunting.
    Szilent and Dyna2 like this.
  12. fransisco Augur

    That would mean EQ would have to load the log file. Its easy to just append something to a file, that leaves it to windows. But having eq monitor the file gets more memory intensive.

    It is good to raise awareness though
  13. Waring_McMarrin Augur

    It already has to monitor the file to make sure it is present and it shouldn't be much work to add a check for file size to that. Gina and EQ Log Parser can both monitor a file and rotate it based on a date or file size and eq handles this log rotation with no problems.
  14. Soulbanshee Augur

    Not really easy, and a lot depends on the language/library being used, and its not "just leaving it to Windows" it still has to ultimately call the low level file access protocol which is not simple. No matter what, you have to scan to the end of the file in order to add a line. This is impacted whether you can do this synchronous/asynchronous, in batches, file stream vs in memory, etc. This is why its slower to append a line as the file size increases.

    I've played around with a file manipulation library, though I really didn't dig into the underpinnings much. You don't have to load a file in order to monitor change events on the file properties. The impact depends on how often you are checking for changes which for the library I was using was just creating a simple timer on a loop, but yeah creating new events will consume at least a few more CPU cycles.
  15. Captain Video Augur

    No. The people who would quit the game because the task of using File Manager to rename or delete a log file once a month is too onerous, are too stupid to be playing in the first place, so good riddance. These are the exact same people, btw, who are busy watching Netflix during all of your raid sessions. We should all be grateful we even have a text file log capability in this game (therefore parsing); more modern MMOs have purposefully chosen to exclude this capability, games you might possibly have heard of, such as WoW, ESO, FF14, just to name a few like that. The way things are going, I would be worried that devs would remove text logging from EQ entirely.
  16. Fanra https://everquest.fanra.info

    Objections:
    1. People should be able to do this themselves. If they can't, too bad.- Well, regardless of that, their subscriptions help pay for the game. EQ cannot afford to lose more people. EQ used to have millions more subscribers and we used to have expansions with 13 zones (The Serpent's Spine, 2006). Laurion's Song, 2023, 7 zones.
    2. It's too hard to program / It will consume too many CPU cycles. - Right now the LaunchPad checks over a bunch of files to see if they need to be updated every time you launch the game. How can adding an additional check of a few more files bog down the entire LaunchPad? I'm not saying to constantly check the log files or do so in real time as you play. It's when you start EQ that it just checks the size of them. The functions to check the size of files and rename them are basic Windows functions. Just like the game checks files to see if your EQ is up to date.
    3. I don't want EQ to automatically archive my log files. - Fine, just uncheck the options box that says, "Auto Archive Log Files". (I changed the name from Auto Trim to Auto Archive since some mentioned that Trim might be misconstrued.)
    Dre., Rijacki, Szilent and 1 other person like this.
  17. Windance Augur

    People are leaving because:
    - They are unable to play anymore, due to RL issues ( kids, jobs, health, money, etc. )
    - They find the risk/reward balance unfavorable and leave for other hobbies.


    People WILL leave if we don't get the DX11 client performance figured out.

    People aren't leaving because their log files are too large and cause lag.

    --------------------------

    With that said, asking for the EQ Devs to add a new in game feature to help with logs is pretty minor, but the way you want it implemented requires them to change the default behavior of the game, then add in a UI element to toggle it on and off. Seems minor, but so does saving and restoring the chat filters. That is why simply adding a new EQ command is easier to implement.

    Either way ....

    I'd rather the EQ Devs figure out the in game chats, DX11 client issues, and fix raid lag and take on a host of other bug fixes.

    ----

    I'd be willing to bet that most of the people who turn logs on, are doing so in order to run GINA.

    GINA has the ability to rotate or delete log files.

    It takes 3x button clicks to get to rename or delete the files.

    Educate the GINA users to make use of it's features.
    Nennius and Captain Video like this.
  18. Ravanta Suffer Augur

    Who wants to tell them that EQLP is superior to gina, does not have a problem with large log files, is compatible with gina's triggers (meaning a person can easily export all of their gina triggers to it in about 30 seconds) has more functions for triggers, a built in help system, frequent updates/improvements, and they are probably already running it anyway as it is by far the most used damage parser?
    yepmetoo likes this.
  19. Waring_McMarrin Augur

    While it may be superior it isn't an easy task to move an entire raiding force to a new program and get all the triggers moved over.
    Rijacki and Nennius like this.
  20. Shea Lorekeeper

    Some good ideas above. I agree some type of log rotation/trimming/archival would be a really nice addition. When I started playing again a few years ago I looked into a way of doing a scheduled task and log rotation... and well, Windows makes what should be a simple task a real PITA (this was on win7 at the time I think). Trying to do it with native windows tools is overly complicated where it would take 5 minutes with a simple bash script and cron job. Oh well.

    That aside, it seems to me what would probably be a good solution is to do it in the EQ launcher as mentioned somewhere above. I don't think the game client and launcher are really tied together very much (maybe some .ini file settings) so adding in game controls would probably take a little time.

    The launcher already has advanced settings (for video settings etc. as we've been reminded with all the DX11 stuff), and it seems to me it wouldn't be much of a stretch to add another config section under those advanced tools with some simple options for log file size limits, deletion or archival, destination directory, and possibly even a compression option. When launcher is running and checking files, it seems to me it wouldn't be difficult to have it check log file sizes and rename/archive logs that have reached a certain limit prior to launching the game.

    The biggest issue I've seen with log files is when you completely forget about them for a month or more and they start to grow beyond the 1gb file size. Even if you're like me and have EQ up for days at a time, chances are you're going to have to restart it at least occasionally. Patches, game crashes (because yeah... game crashes), etc. Having some type of basic log rotation at the launcher stage just seems like it would take care of the majority of large log file issues and probably be the easiest to implement.
    Rijacki and Svann2 like this.