You are SOOOO correct!! I do agree with you that the returns should be increased. I know that you and just about every other crafter in game also harvests by hand too. I am disappointed in how sparsely the pony harvests. 3 ponies, 2 newbie alts scurrying around gathering presents and my depot stack of purple fabric is tiny, tiny, tiny. May as well put freaking rare sparkles around that mat.
It seems to me that there is definitely a difference in the relative abundance of each material. I have a harvest depot with over 45k total Frostfell resources, the vast majority from pony harvests (currently I have 11 ponies) - though I did do some manual harvesting and some have been used for crafting. The lowest amounts are 262 Cured Ham, 291 Raw Fowl, 298 Fancy Purple Wrap and the highest amounts are 1746 Gigglegibber Sweet Tooth, 1653 Cotton Twine, 1579 Glittery Powder. So over a 6 to 1 difference between lowest and highest, with the amounts involved I would say this is more than random chance.
From looking over some of the recipes, you can see the intentions with the harvest rates. The relative abundance of certain items matches up with how many things they make. The sweet tooth will be used in several recipes for sweets, where the raw fowl will only be used in 3 recipes.
It's actually probably based on the 'nodes' as well. The three different package types in Frostfell Village yield distinct materials. As with the food harvest from bushes, the FF nodes that yield food have more variety in what comes from those packages. As the pony seems to get x pulls on each different node (even with regular harvests) that's probably why we see few 'food' harvests from the pony no matter what tier we send it out for.
^^ More proof that harvesting is the way to get exactly what you need and ponies and hirelings are just there to supplement, not replace that.