Harvesting algorithm change

Discussion in 'Tradeskills' started by Adeyia, Mar 5, 2022.

  1. Adeyia New Member

    I had previously written by own parser to track all my harvesting as the various tools/plug-ins I'd seen all have their own bugs when interpreting the sometimes ambiguous output that's in the log file. The numbers/percentages for harvesting I'm seeing now are significantly different to this time last year.

    After trying and failing to find bugs in my code I realised the issue is actually something must have changed in the game. I've seen several mentions on forums that changes have been made to the harvesting algorithm, but I'm struggling to find a list of precisely what has been changed (or why).

    At a quick glance it looks like maybe harvesting results are getting closer to what they "should" be rather than the broken mess it was before where you were averaging 3 rares from each root node for example. I'm hoping these changes are intentional rather than accidental and that there is a plan to finally fix harvesting.

    Does anyone have any links to an official description of the changes that have been made, and maybe even how harvesting calculations are actually supposed to work these days?
    Lateana likes this.
  2. Sigrdrifa EQ2 Wiki Author

  3. Adeyia New Member


    Thanks for the suggestion, although I tried that already and found the results to be a bit off which is why I wrote my own to handle all the edge cases in the log files. It looks like it's been updated recently though, so I might give it another shot.

    However, regardless of the precise resolution of any given tracker's accuracy, there's a big enough difference between the current results to the historic ones that the choice of tracker doesn't actually matter. For example, if I compare some harvest logs from gathering in Antonica that I have saved from June 2021 with some from today (which both have a 80% rare rate and 102% bountiful rate), I get the following:

    June 2021
    callous ore - attempts 249, items 2305, rares 210 - rares/attempt = 0.84
    plains roots - attempts 267 items 2353, rares 215 - rares/attempt = 0.81
    wind felled tree - attempts 276, items 2533, rares 209 - rares/attempt = 0.76

    March 2022
    callous ore - attempts 321, items 2783, rares 54 - rares/attempt = 0.17
    plains roots - attempts 291, items 2610, rares 41 - rares/attempt = 0.14
    wind felled tree - attempts 282, items 2368, rares 37 - rares/attempt = 0.13

    For maximum accuracy the sample sizes could of course always be larger, but it's a significant enough size to show the trend which is that the rares per harvest attempt has dropped from approximately 0.8 down to 0.15.

    The question really is if this is now working as intended after many many years of being broken, or has something been broken unintentionally? This is why it would be great for somebody to be able to officially describe what has been changed and how this is all supposed to work as opposed to our various guesses.
  4. Sigrdrifa EQ2 Wiki Author

    Oh! I see. Chrol changed the way Rare Harvest Chance works a while back. I dunno why. Your best bet for rares is lots of Bountiful Harvest, because the more pulls you get, the more chance you have to roll the magic number to get a rare.

    I'm not caffeinated enough to go dig for this, but there has been lots of discussion on the forums about rare harvest chances that covers the specifics in more detail.
  5. Sigrdrifa EQ2 Wiki Author

    I'd also suggest discussing your edge cases with Aditu and Mineeme -- Aditu wrote the original plugin, Mineeme did the most recent update.
  6. Ixian Active Member

    Chrol doesn't work on harvesting system design stuff, Caith does.
    Sigrdrifa likes this.
  7. Denmum Developer

    It is supposed to now be working as intended, but yes, Caith did a lot of tweaking/fixing in the background. Shortly after the expansion, the overabundant root issue was also fixed, as there was a spot where it was giving rare roots when it was supposed to be giving commons and other fun stuff like that.
    Doobius40504, Breanna and Twyla like this.
  8. Adeyia New Member

    Thanks for the update and confirming it was tweaked as that's definately my experience running the harvest parses. There's still a few weird things happening when parsing the harvesting logs that I can't explain, although I appreciate in the grand scheme of things most people don't really care about harvesting in any real detail so as long as it's not completely broken they would rather fixes went in elsewhere.

    I'm not sure if the issues I'm now seeing are around the way harvesting is working, or if they are just a result of the way it generates the log files.

    As an example, when rolling against the harvest loot table there's a chance that you can get a '10+1' harvest, which is 10 standard items plus 1 rare item. It also seems possible looking at the logs that you can also in very rare cases get a '10+1 & 10+1' harvest, which give you 22 items in a single harvest attempt. That's always been the case as far as I can tell and nothing has changed there really. However, harvesting in The Blinding earlier I was getting '10+10 & 1+1' harvests, which looks like it's returning the items in the wrong order. i.e.

    Code:
    (1646753229)[Tue Mar  8 15:27:09 2022] You make a bountiful harvest!
    (1646753229)[Tue Mar  8 15:27:09 2022] You acquire 10 \aITEM -2027472580 -20975524:Hardened Zelniak Pelt\/a from the rockhopper den.
    (1646753229)[Tue Mar  8 15:27:09 2022] Items were placed in your personal harvest bag.
    (1646753229)[Tue Mar  8 15:27:09 2022] You acquire 10 \aITEM -1791318572 -87945133:Rockhopper Meat\/a from the rockhopper den.
    (1646753229)[Tue Mar  8 15:27:09 2022] Items were placed in your personal harvest bag.
    (1646753229)[Tue Mar  8 15:27:09 2022] You have found a rare item!
    (1646753229)[Tue Mar  8 15:27:09 2022] You acquire 1 \aITEM -1654348676 1870698138:Lustrous Rockhopper Hide\/a from the rockhopper den.
    (1646753229)[Tue Mar  8 15:27:09 2022] Items were placed in your personal harvest bag.
    (1646753229)[Tue Mar  8 15:27:09 2022] You have found a rare item!
    (1646753229)[Tue Mar  8 15:27:09 2022] You acquire 1 \aITEM 1173133347 -140042032:Zelniak Steak\/a from the rockhopper den.
    (1646753229)[Tue Mar  8 15:27:09 2022] Items were placed in your personal harvest bag.
    (1646753229)[Tue Mar  8 15:27:09 2022] You acquire 7 \aITEM -1791318572 -87945133:Rockhopper Meat\/a from the rockhopper den.
    (1646753229)[Tue Mar  8 15:27:09 2022] Items were placed in your personal harvest bag.
    

    Equally there's times where you have a Bountiful Harvest and get a standard rare as the first item, which based on my understanding of how the various rolls work, would suggest you'd still be expecting to get your two loot table rolls (1 bountiful, 1 normal) but in fact you only get 1 roll i.e.

    Code:
    (1646750411)[Tue Mar  8 14:40:11 2022] You make a bountiful harvest!
    (1646750411)[Tue Mar  8 14:40:11 2022] You have found a rare item!
    (1646750411)[Tue Mar  8 14:40:11 2022] You acquire 1 \aITEM -386398085 508285890:Radiant Pome\/a from the lunar berry bush.
    (1646750411)[Tue Mar  8 14:40:11 2022] Items were placed in your personal harvest bag.
    (1646750411)[Tue Mar  8 14:40:11 2022] You acquire 2 \aITEM 754914785 -1186927793:Lunar Mushroom\/a from the lunar berry bush.
    (1646750411)[Tue Mar  8 14:40:11 2022] Items were placed in your personal harvest bag.
    

    Both of the above logs are when I had 0% rare chance, and 100% bountiful chance.

    In the first instance it looks like the events arrived out of order, and in the second that one of the events went missing but that's just my possible interpretation from the logs. I'm happy to run some more tests if Caith or anyone can share how all the various roll sequences are all supposed to interact with each other!

    P.S. For reference, this is how I believed it was supposed to work:

    1. If harvest skill level not high enough to be eligible for normal rares, go to step 3.

    2. Roll to see if you harvest a normal rare - this uses your rare chance but isn't a straight % chance.

    3. Roll to see if you made a bountiful harvest. If you didn't succeed in roll go to step 5.

    4. Roll to select the item, then roll against a normal item lookup table to see what you get (1, 2, 3, 5, 7, 13, 10+1,10+1&10+1, or 1 ad-hoc rare*)

    5. Roll to select the item, then roll against a normal item lookup table to see what you get (1, 2, 3, 5, 7, 13, 10+1,10+1&10+1, or 1 ad-hoc rare*)

    * I term ad-hoc rares to be rare items such as Planar Energy, Primordial Fragment, Aether Spellshard etc. etc. and these are rolled for as part of the rolls in steps 4 & 5 rather than step 2. It may be possible that for some loot tables, the normal rare items for that node are included here too, which would explain my second scenario above, and I've even seen a quantity of 2 rares returned which suggests even more complexity!
    Twyla likes this.
  9. Calamoor Active Member

    I've noticed today that the chance of harvesting rares at lower tiers has been drastically reduced. Was this intentional? It's almost like having any gear that raises rare chance is mute now (almost like it's back to a cap of 7% or less) and that the bountiful is capped at 100%.
  10. AOE1 Well-Known Member

    Since the Mango is used in many recipes, those could be increased just a tad. Not any rare, just the common mango in VoV. Guild halls and players go through those more than any other harvest item from what I can tell. I know our guild hall is often out or low and that is using the guild harvesters AND players harvesting to put in there. And if you need one, don't EVEN look on the broker. I would take Fort Knox to buy one there.

    Since I do not have the personal relationship with the Developers like some players do, perhaps they can mention this?
    Myxx, Breanna and Twyla like this.
  11. Melkior Well-Known Member

    Mango, Wild Rice, and Rhino Meat seem to be the pacing items depending on what temps you make. I suspect this is deliberate, but I agree a higher output on them would be appreciated.
  12. Uwkete-of-Crushbone Well-Known Member

    I've noticed this particularly in the "lower tiers" (probably anything below triple digits, but don't quote me), and frankly, those of us who have "lower tier" toons would deeply, deeply appreciate it if the "fixed break" was "broken" again. Seriously. You want to talk about jacking up the inflation again on the Broker? If we as players can't get at least a few Rares for things like Chest/Leg/Ring/ Charm slots and weapons (oh, and Cloaks and Shields, can't forget those ;->), without spending a good 10-100x the amount of time and/or platinum it used to take, that removes a great deal of the enjoyment from the game for me (I don't even "spend" rares on making house items). I'm running across this problem right now, where a "mere" hundreds of thousands of plat will flow through my kitteh paws like water if I can't replace those overpriced speculated commodities on the Broker with less than an hour per type of harvesting, during the last stages of a major holiday. X-P

    Uwk
    Mellody, Breanna and Twyla like this.
  13. Myxx <Into Harms Way> Maj'Dul

    Denmum please speak with the powers that be and see if they will increase the harvesting chance for Mangos , Wild Rice and Rhino Meat. I'm all for people that harvest making money for their time but there are so few mangos on the Maj'Dul broker now and the ones that are there are over 800 plat each. These are commons, why are they so rare?

    Also, the number of mangos needed to make 1 stack of 100 Extended Bloodbound Insight temp adorns is 15000 mangos. For the CB temp you need 15000 Wild Rice and 15000 Wild Rhino Meat. This seems very high.

    I added it up last night and if I 'could' buy 99999 mangos at 900 plat each and max out the harvesting box it would only make 6 stacks of 100 temp adorns and cost almost 90 million plat.

    [IMG]

    If I give 24 raiders 10 temp adorns each, that's 240 out of my 6 stacks of 100. At 99999 mangos it would only make enough of 1 kind of temp adorn to cover 2.5 raids.
    I would love to make the Crit Bonus , HP and 2H versions of the temp adorns also. As it is, too many harvestables are required and way too much plat. I know in our guild these numbers are not sustainable and I have to think that other guilds are having the same problem.
    AOE1, Breanna, Geroblue and 2 others like this.
  14. Jesaine Well-Known Member

    It's not just you or your server with this issue, trust me.
    Geroblue and Uwkete-of-Crushbone like this.
  15. Uwkete-of-Crushbone Well-Known Member

    Yeah, I think they went overboard the other way in their "let's tighten up the loose slot machines of the nodes! :D" bit. :-/

    Who does that tightening help? The Broker speculators? Sure as heck not the rest of us. X-P

    Uwk
    who can't help think: if they're serious about getting currency OUT of the economy to help with the inflation, letting the speculators charge MORE money for even common mats is not the way to do it X-P
    AOE1, Geroblue and Jesaine like this.
  16. Jesaine Well-Known Member

    Amen to all of it. As I've pointed out numerous times, the trouble here is that the "loose slots" theory is a fallacy. It only holds true for some players. Some folks can go out and farm rares, or master books as another example outside of TS'ing, and sell them for fortunes. They pull them all day by the truckload and other people struggle to get a single rare. I've cited my own experiences in previous posts; a character that spent 9 hours to get the rares for the basic VOV TS timeline... Returning over 2 months ago and never once seeing a single master book drop anywhere outside of the Agnostic Dungeon solos...

    It's easy to say go out and work harder when you're one of the ones the totally broken algorithm has decided to bless.

    Then there's the whole issue of adding roots to food harvest nodes to make the odds even worse.

    But yeah, the algorithms need major work to add some form of "streak breaker" in, NOT punish more severely the people the game already screws this way.
  17. Uwkete-of-Crushbone Well-Known Member

    Hmm...have you tried the lovely, lovely "get access to the Advanced crafter books" quest lines these days (more power to them, I say; we need more like that!)? :D One of them gives out a harvesting helper familiar (basically Fluffy from Harry Potter, though not as big ;->) that does seem to help a tad, at least in the lower zones where I've tried, but even Fluffy -- er, Eustace isn't always a consistent help. :-/

    Uwk
    who wonders if this wasn't a relic of the crafter hater dev who had control over PoP :-/
    Geroblue and Jesaine like this.
  18. Jesaine Well-Known Member

    Oh, don't get me going on those. I've been running them (and the faction recipe quests) for the last week on the character I *FINALLY* settled on as a main, LOL.

    Between Eustace, a harvesting kit and a couple other tricks, I feel like she's ALMOST at normal levels of luck, but certainly nowhere near these folks that have hundreds of rares on the broker for sale.

    At the risk of derailing the thread, I've got a large portion of the books now and have started jumping around a little. Right now I'm working on trying to REGAIN access to the Obulus Frontier because my VoV Crafting Sig Line nightmare is repeating. "Yes you got access, BUT you didn't finish the vast majority of the timeline here, and you left to manage inventory, SOOOO, you're locked out of a zone you got access to!". *headdesk x infinity*

    Access that isn't access... Whose IDIOT idea was that?
    Geroblue and Uwkete-of-Crushbone like this.
  19. Uwkete-of-Crushbone Well-Known Member

    Waitaminute...if you don't finish everything before leaving the zone (FOR ANY REASON), you have to start ALL OVER AGAIN?! :mad:

    No. I've had fun before. This isn't it. >:-/

    Uwk
    Geroblue and Jesaine like this.
  20. Jesaine Well-Known Member

    We, the great unwashed who pay to keep this game open are not privy to any such information, nor any nerfs or boosts made, etc... We have to best guess that sort of thing via third party plug-ins like ACT.