Presenting and interpreting data sets

Discussion in 'PlanetSide 2 Gameplay Discussion' started by Mrasap, Feb 20, 2013.

  1. Mrasap

    TL;DR: post entire data sets.

    preread: for all those trying to attack my statement in a personal manner, this is my character: . I spend most of my time in an ESF; from the 8,5k kills I have 3,5k in the scythe, the rest from LA shotgun style. I do not care about MBTs other than ramming rocketpods in their behinds. I do care about the proper presentation and interpretation of statistics.

    This is a story about data sets. More specifically, the data presented on MBTs balance. Higby posted data on KD ratios between faction specific MBTs, which showed a much higher ratio for the magrider compared to the vanguard or prowler and normalized ratios after GU02. Beautiful data.

    But then, this data is used to draw conclusions on the effectiveness of the MBT. Now, if I were to treat a patient with a medicine on the basis of favorable disease outcome but I neglected the side effects completely, I would have a serious lawsuit at hand. I have 2 major concerns with Higbys method:
    1. The effectiveness of the MBT is determined by many different components. KD ratio versus other MBTs, but also against other vehicles, total (infantry) kills per death or longevity per death. Presumably most important of all: score per hour. Judging the effectiveness of the MBT on one aspect is flawed. A prowler in lockdown with HE might be incredibly deadly against infantry, but remains balanced due to their hampered defense against enemy ESFs.
    2. What was the setting the data was extracted from? What were the inclusion and exclusion criteria? Are we looking at a minor subset of total MBT deaths? I find it reasonable to assume that most MBT versus MBT data suffers from infantry involvement. Are these data adjusted to confounding factors? These data mean nothing if every magrider had a secondary gunning wielding an AP gun, while others did not.

    Publishing incomplete data fuels overzealous conclusions and causes chaos in return. These incomplete data are now being abused by those unable to interpret these results. I do not deny magriders were stronger in the scenario depicted by Higby, whatever that scenario was. While in the end, no conclusion can be drawn about the effectiveness of MBTs due to incomplete data.

    Higby tweeted this yesterday:
    Important life lesson: most people can't accept facts which disagree with their opinion.

    While it should actually read:
    Important life lesson: I must not post incomplete data sets.

    Either show entire data sets or nothing at all. Please don't be this guy:

    • Up x 4
  2. Zotamedu

    To quote Higby in the post where the data was presented:

    "I wanted to add something quick about MBT balance overall. The general balance for MBT vs MBT is difficult to paint a complete picture of in a neat chart, due to the combined arms nature of the game. We have a ton of different data from time usage, # of tanks fielded, # of shots fired, accuracy, top weapons used to kill, average lifetime and etc that we use to make determinations about game balance. One metric that is important as an indicator of balance is overall tank vs tank K : D ratio, now, it definitely doesn't tell the entire story when it comes to "how effective is ____ unit", but when tanks are significantly outperforming their empire counterparts in K : D vs each other it's a clear sign of a problem."

    " Again, I don't want anyone thinking that I said this constitutes the whole story on tank balance, it absolutely does not, this is just one view on effectiveness of MBTs vs eachother."

    So, turns out that they did look at more data but just chose not to publish it.

    Most of your objections are not really a problem if the data set is large enough since we can assume that the playing style will be similar given a large enough population. All sides have the same LA with C4, the same engineer with AT mines and the same HA with rocket launchers. The liberators are the same as well and so are the lightnings. The only thing that affects tanks that are different between the factions are the MBT and the ESF and the rocket pods are not all that different when it comes to A2G combat. So the central limit theorem should apply here. Or do you have any proof that one faction uses combined arms a lot more than the others and that it's spread consistently over the servers? Where is your data backing up such a claim?

    No matter how much data he published, you would still be able to find holes in it. To collect and analyze data is an art and you do not seem to appreciate that at all. How much do you really know about statistics? It really sounds like you are grasping at straws here.
    • Up x 6
  3. Shinrah

    Yep, I wish he would stop posting data like that. Way too many important factors cannot be determined from that little screenshot he posted. At least my subjective impression is: Magriders almost always have the Saron, and quite often a gunner. While from expierience I can say; I rarely see gunned Prowlers or Vanguards, and only a few of them have their faction specific guns. For example, only every 4th or 5th prowler I meet actually has a Vulcan, most have some other MG or the Rocketlauncher. And quite often they don´t have a gunner.

    Now as I said, I admit this is my personal impression, it might be flawed. But then again it is something I always wondered about and so did others in my outfit, so I guess its fair to assume that there is some truth to it.

    So yeah, not posting incomplete inconclusive data would help. If I had presented data that way in any paper I would´ve been in trouble. Anyway good post, and lets hope Higby posts some more detailed data sets in the future. I´d rather have no data at all than some summarized spreadsheet that is inconclusive.
    • Up x 1
  4. Zotamedu

    The problem is that this is not a report, it's a forum where the average age and IQ is both below 15. So a data dive would do more damage than good simply because most people can't interpret it. Then it's better to do the complete analysis yourself and then show some highlights that show your point. I really don't see what the problem is. He wrote repeatedly that the picture did not show the entire picture and that the overall balance was a more complex issue. When they show no numbers, then people complain. When they show some numbers, people complain and if they posted all numbers. People would just cherry pick the data and then complain. You see the broader problem here?

    The data is not inconclusive. It clearly shows that the Magrider won a lot more tank to tank combats. To understand why, you need to dig through a lot more data but that was not the point of the picture. It was only to illustrate that there is an imbalance, something that many people have been pointing out.

    It seems the problem lies more in reading comprehension and an underlying dissatisfaction in the way the change was carried out rather than an actual problem with the presented data. Maybe it was the Saron that needed a nerf, not the movement. I know for a fact that the gattling gun the TR get as the secondary AT weapon has about the same range as throwing a rock and it's very difficult to aim due to the bullet drop. But that does not change the fact that there was an imbalance. Do you understand the difference?
    • Up x 6
  5. HadesR

    TBH it doesn't matter if players can interpret it or not it's only important than SOE can .. If a Dev say's " Hey Ive got some interesting stats for you " they should be taken as nothing more than that ..

    As you say to many people with IQ of 15

    but also

    To many people think they are game developers or they have an inflated view of their self worth
    • Up x 4
  6. Purg

    I would suggest Higby learned his lesson and will not post any other data from now on.

    Are people that rabid that they have to bleat on about an interesting set of data without going off the deep end? You guys realise this is only a game, don't you?
    • Up x 1
  7. Shinrah

    Like I mentioned, since its apparently a summary of different data sets it is inconclusive. We don´t know what was taken into account and what not. So wether mistakes were made while compiling this data or not, we can´t say. The Magrider was a rather popular tank, and I never had trouble finding gunners for my SRB. At least 2/3 of the enemy MBT´s I destroyed did not have a gunner. Prowlers still run around primarily with HE shells, while every good Mag driver switched to the default gun, after the HE nerf pre GU2. We don´t have the luxury of firing 2 shots, and correct a failed first shot, that made most tankers I know switch back to HEAT. HEAT has way better AV characteristics than HE.

    Long story short, SOE could basically project any outcome they wanted into that summary. I´m not saying its wrong, I´m not saying they did purposely tweak the data or anything. I´m saying the problem is summarized data like that raises more question than it gives answers. So either post a full set of data or don´t post it all. That keeps the conspiracy theories at a minimum and prevents threads like this in the first place.
    • Up x 1
  8. Sharpe

    I don't know how many ways Higby has to say the posted graphs in no way represent an accurate picture of balance for people to actually "get it".

    " I don't want anyone thinking that I said this constitutes the whole story on tank balance, it absolutely does not"

    The real mystery here is how so many stupid people ended up together in the same forum.
    • Up x 4
  9. hostilechild

    This is kind of like what i did to my son. Let him drive my 370Z then gave him a mazda3. :p
  10. Zotamedu

    I agree with you up to a point. The data that he did publish was a good indicator of what was happening and that they really had data supporting the claims of the community. The main problem is which data and how much to hand out. If no data was presented at all, a lot of people would still complain about the lack of data. If some data is presented, obviously people complain about the lack of more data or complain about the quality of the presented data. If they were to hand out everything. A lot of self-proclaimed experts and wannabe statisticians would slaughter it and present their own graphs and plots to "prove" their own points. So it's a case of damned if he do, damned if he don't.

    I don't think the main problem is the data or the way he presented it. He repeatedly stated that it was just an indicator and that it did not represent the whole complex picture. He even stated it repeatedly. The problem seems to be reading comprehension of the community. If everybody had understood what the post said, we wouldn't have this kind of thread. Then we could have discussed the nature of the nerf instead of the reason.

    I'm not convinced that the maneuverability was the right answer. I think it would have been more interesting to keep the age old balance factor of agility vs. durability. The Magrider is agile and can strafe and should be able to reach some places where other tanks can't go due to mysterious alien technology. But since it's agile, it should also be fragile. Reduce the armour and we have a more interesting balance.
    • Up x 4
  11. Mrasap

    Yet in the current situation he is giving incomplete data and your average forum posters are drawing presumptuous conclusions. The posted data mean exactly the following: a magrider has a far higher KD ratio against other MBTs in a scenario we do not know the specifics of. Yet your average forum poster concludes that the magrider is a far more effective MBT compared to the other factions. In the mean time prowlers might kill three times as many infantry compared to others and vanguards survive much more ESF flybys, but we don't know this because Higby doesn't post complete data sets.

    If he would post all numbers at least his method would be transparent. Cherry pickers can easily be identified as cherry pickers. If no numbers were posted, at least we wouldn't have so much chaos due to misinterpretation of data. We would have to trust the developers he is drawing conclusions properly and we have no reason not to trust him. Now we know he judges balance on the basis of KD ratios, while he doesn't say a word about score per hour - not even in the explanation you've quoted. While the general consensus on the forum seems that score per hour is a better parameter to begin with.

    No I do not, thanks for reminding me, because Higby doesn't publish complete data sets. What we do have are hypotheses. As both you and Shinrath said magriders might be manned more often because they have better secondary guns. This is a serious confounding factor. I personally have the feeling prowlers more often target infantry, therefore a prowler is caught offguard more often by another MBT and starts the battle at a disadvantage.

    Thanks for this, you seem to be drawing a presumptuous conclusion about me without any insights. Please keep the personal attacks to a minimum.

    May it remain clear that I do not care about the faction balance between MBTs. I do care about overzealous forum posters abusing shreds of data to draw presumptuous conclusion, ultimately leading to flawed discussions. I do not wish my playstyle to be balanced on incomplete data, neither do I wish such a treatment upon other players with different playstyles.
  12. Wasdie

    I believe it's in the best interest of a game dev to never show stats or talk about the exact numbers ever, even during patching. It only leads to unnecessary outrage by people who believe they understand the game and mechanics better than those who made it.
    • Up x 4
  13. Mrasap

    We seem to be reaching to the same conclusion. He would have done a far better job by not posting the graph as it draws away from the whole complex picture.
  14. Mrasap

    Found another gem:
    If prowlers were doing only very very slightly below magriders in general, while having a weak KD ratio against other MBTs, then how are they doing now, when they have comparable KD ratios? This emphasizes my point: don't post incomplete data sets trying to prove a point, it only causes confusion and makes you look like a tool.
    • Up x 1
  15. cfnz

    This I agree with, though I would still rather the data was posted. We are never going to get all of it, and some IMO is better than nothing. The problem is the people jumping to all manor of conclusions and trying to twist the data to support their claims.

    Frankly I'm opposed to the denial of something from everyone just because a subset is choosing to misuse or abuse.
  16. Zotamedu

    Let's try this again and see if we can handle it like adults.

    Those graphs only draws attention away from the complex picture if you are easily distracted by bright colours.
    • Up x 1
  17. illgot

    "Hey, I don't want to paint an inaccurate picture of what is going on here or why we changed the MBTs, but here you go, here are 3 graphs which incorrectly show exactly what we want to see."
    • Up x 1
  18. Jac70

    Agreed OP.

    There is no way by looking at stats like that you can determine whether balance is off. There are too many other variables to consider. If you took a dataset of players of equal skill. Gave them all some time to familiarise themselves with each tank. Then ran multiple scenarios of 1v1 across a variety of terrains and distances swapping which players are using which platform constantly. Then you could look at the results and see if there was an imbalance.
  19. Zotamedu

    Try reading the bold areas of the second post in this thread...
    • Up x 1
  20. Mrasap

    And because the forum is a place where the average age and IQ is both below 15, these graphs are misinterpreted broadly and chaos ensues.

    I'm inclined to agree with Cfnz but posting fragments of data are screwing up these forums. Trying to have a decent discussion on overall performance of MBTs is currently impossible because all naysayers ignorantly point to these fragments and God forbid trying to explain the correct interpretation to them.