ANT will be the death of PS2

Then Ant gangk squads are going to be hilarious

I'll contrast our Forumside dilemma with RL example provided on wiki page I linked.

Students who take test preparation course achieve higher test scores. Is it because test preparation coursed are effective or simply because diligent students who get higher grades anyway are more likely to take test preparation course?
Only way to truly assess effectiveness of test preparation courses would be to pick say 150 students, make half attend test-preparation course and other half attend some placebo course and then compare effectiveness.

Back to Forumside: As students choose whether they will attend preparation course so players choose whether they will go on forums. That means the sample is self-selected. If you wanna find out how playerbase thinks about something then you can't afford to have self-selected sample, you need to pick 100 players at random and ask them questions.

Person who wants to implement balance changes based on Forumside opinion needs a proof that forums are good representatives of playerbase opinion at large. People who post on forums form self-selected sample(they've decided to go post here), whether that sample is good representative of playerbase needs to be proven by someone who wants to use it as justification for balance changes.

There's several problems with your version of self-selection: It can be used anywhere.
First and foremost, you are using self-selected criteria so that you can accuse me of self-selection. Because your way of self-selection does not just apply to students who take a preparation course, your way of self selection can be used on people taking the test vs people not taking the test. Your version says "if one group takes the test and scores average, this is by no means a way of determining if the whole world has those average scores". Even though you might be comparing people taking a university mathematics test compared to the rest of the world that will in general never see such a test of score anywhere close to that.
Second, you are completely disregarding the purpose here. You are saying "players on a forum are all self-selected vs players not on the forum". The problem with that statement is that if it's true, no single game would spend money and time in maintaining servers and relationships with their playerbase on forums because the players on these forums would according to you be self-selected and you cannot use them for improving the game. And if that's true, what the hell are you doing here? What the hell is every major game-company doing in keeping alive forums and having expensive community managers keep contact with your playerbase?

So either I'm wrong in using the average idea's and complaints of the playerbase on these forums, like the developers and many multi-million companies do, or you are wrong by abusing self-selection and twisting it so you can point the finger at anyone you want.

Or proof that it's not a good representative. I've browsed a bit on Google Scholar for example and found little in the way of research to prove that forums are effective or not, but compared to the alternative of no way to communicate at all with your playerbase and no way to determine what people like or dislike, I would say that on average the self-selection criteria would have a higher chance to give you good idea's as to what problems might need fixing in your game since it's highly unlikely that only the people on the forums have a problem with a specific thing while the non-forum going people do not have that problem.

So unless you can argue with that and find proof that multi-million companies have been throwing away millions of money in keeping forums and community managers on their staff I would think that I've got enough reason to use the experiences and words of the forum people to help my cause.

Oh no, the common practice of companies and game designers is to use forums for feedback, and just like we can agree that any feedback is better than no feedback, you need to prove that the feedback we are getting is wrong.

If you randomly pick 100 people and have them take the test then that can give you a pretty good estimate of the true average score.

On the other hand if you don't randomly pick people but have volunteers come to you to solve the test then there is potential self-selection bias problem.

To conduct meaningful poll it needs to be randomized. If people aren't picked randomly then the poll has selection bias problem.

That doesn't mean forums are bad for getting feedback, it just means that forums are bad for estimating how many players like or dislike something. Forums are useful for determining why players like or dislike something. In other words forums are good for answering "why people hate thing X" but not "how many people like or dislike thing X".

There is a crucial subtle distinction to make here:

Feedback can be in form of discussion WHY something is OP or frustrating.
Feedback can also take the form of opinion poll in which you want to find out how many players like, dislikes or are indifferent to something.

If you wanna find out WHY people dislike thing X then forums can be helpful. If you wanna find out how many players like or dislike something then forums are bad for this and in-game opinion polls are much better alternative.

Let's pick out a few key elements here:

This means that the forums could be bad for getting feedback. However we have to ask ourselves: What's the alternative? You can send e-mails for feedback to the player base, but then you have self-selection for the people who want to fill in the forms. In fact, every single potential for getting feedback for a game would suffer from this self-selection problem.

This also means that the following does not have to be true since there is absolutely no proof for it and compared to the alternatives is actually the best option:

What option would you give then? What's your alternative that's absolutely self-selection free?

Actually no, players often don't know what's good for them and will come up with the worst idea's as to why they like or dislike something. You are better off checking what parts of your game they dislike and then using knowledge of game-development to change their game and solve the problem.

Then you still have a self-selection problem! You have the players that want to take their time filling in forms in-game and the players who don't and won't fill it in! If you force it on people you won't even know how many people you shove away by forcing them to fill in forms while they want to be relaxing and blowing some people up.
Forums are great ways to determine what people like or not.

You can simply use in-game polls. "What do you think about thing X", provide 5 possible answers, players tick one, and they get say 10 smedbucks or however it is called nowadays.

Why does it have to be absolutely self-selection free? Polling and statistics is never error free, we just try to approximate size of the error and do our best to minimize it.

Well lets agree to disagree. I think customer subjective experience when using a product is a very useful information.

There are no forms. If you wanna find out whether thing X is hated or liked you need only one form ie question. It takes like 15 seconds to read the question and tick one of the answers provided.

Second of all big advantage of in-game polls is that you know how many players didn't answer the poll so you know how much is that poll reliable aka how large is the selection bias.

Yes, but not "how many like it compared to how many dislike it(and how many are indifferent)".

PS: Of course it is possible that Forumside is good representative of playerbase opinion but most probably it isn't. If you go through history of polling and how tiny mistakes lead to catastrophically bad predictions of elections you will perhaps appreciate more what I'm saying.

Still doesn't prove that the forums would be worse or a bad place to use as feedback, which you used against me.

It doesn't, that's the whole point I'm making. The forums aren't self-selection free and any other option isn't self-selection free. And again it doesn't prove that my use of the forums is a bad thing.

We don't agree to disagree, because my whole point was that the subjective experience is used on the forums, and that the developers can take that into account too then make their own solution used on the that experience.

And with single questions you can have perfect insight into what people want or not! And you build these questions based on... What exactly? It could take literally more than 5000 questions before they have the right kind of topic, and to get actual good feedback they could use you would need a dozen questions at minimum per topic. Hence: forms.

Look, selection bias is only a problem if the one's that are filling it in are not a biased group of players. Simply saying they are biased because they are the one's that go to the forums is just bad form. There's no proof nor precedent that forums are bad for finding out what people think.
And even if you had only people who had bad experiences and no players that join the forums to file their good experiences, that's still good feedback! Reducing the amount of negative feedback and turning it into positive feedback would be your goal.

Not for exact numbers no, but that doesn't mean it's bad. A game needs to be fun for as many people as possible, and to do that you need to know the positive and negative parts.
And as I said before, players that are happy try to come up with ways to improve the game even more, and players that dislike stuff will come up with idea's that change the specific parts they dislike. This way you can already filter out a playerbase that likes or dislikes the game and have idea's what direction they want you to go. Then the devs need to take into account the repercussions of those idea's and see if this won't cause even more dislike than before.

Polling for political things is a completely different subject to seeing what a game needs to be/stay fun. We aren't voting for a president, we aren't electing people who promise us the changes we want (without any warning that they will have to compromise on them no matter what). Players on forums express their feelings, wants and desires for the game hoping that the developers will read them and do something with them, and then the developers can sift through them to see what they could work on next in the game. That's what forums are for, and that's how you can easily use forums.
Also we started this because you said that the forums were 'nerf nerf nerf' due to selection bias (someone else said the 'nerf nerf nerf' but you tried to support it with selection bias). We already know that the whole 'nerf nerf nerf' stuff is not true because there is a large movement here on the forums that is constantly nagging that the developers are nerfing everything, this same movement asks for buffs no matter how warranted or not. On the other side we have people who ask for nerfs for things they dislike.
So you are supporting something that's inherently not true with selection bias? Aside from that fact that you know that the forums could very well be a valid representation of the playerbase anyhow. Your only argument that it wasn't was because of the toxic environment the forums are, but seeing how the game is working out under current conditions it's an almost perfect representation of the playerbase.

Here is the thing with polling: Only true way to poll is to randomly pick people and ask them questions. If you pick people randomly then you have a statistical guarantee that your results will mirror the true results of a population.

On the other hand if you don't randomly pick people but they come to you (like in taking prep test or deciding to post on forums) then you don't have a guarantee that your results will mirror true results of a population.

I mentioned election just to provide case studies for those interested in how tinsy methodological flaws can lead to big errors in estimation.

You asked me what is the alternative and I've said in-game polls. If Forumside says "nerf X" then it would be low-cost to create in-game poll with the question like "Do you feel thing X is too strong" and numbers 1-5. Maybe someone more experienced and inventive can come up with better wording but point is you don't need 5000 different questions to find out whether people feel something should be nerfed.

Well that's why devs mostly base balance decisions off of in-game metrics: players vote with their feet by using the weapons they believe are good and racking up kills with them. Being omniscient (at least within the context of the game) has its perks.

But what the heck does this have to do with the ANT?

I'll listen to you. If you stop using Windows 7. I mean really it's like 2016 already.

No, I'm not saying the larger group is content. I'm saying the loudest or most obnoxious people on a forum don't make or represent a majority. And you are pretending I'm saying something I'm not just so you can argue. Which is a good example why the forum isn't much of a representation of the player base as a whole. Most people aren't going to waste time on that.

You were saying that the entire forums are, according to you, a bad indicator of anything. Not just the obnoxious or loudest, everything. And the entire forums aren't a bad indication for what people in the game want.
Also, you are heavily implying that I only use the most obnoxious and loudest people on the forum to get an idea of what the player base wants, which also isn't true. Because A: being obnoxious is subjective and there will be more than enough people that think I'm obnoxious, and I personally try to avoid the obnoxious people, although I do take note of what they say to get an idea of what their reasons and desires are.

And I'm not pretending btw.

It's just a simple fact that they have their own data that they use for balance decisions. They don't base it on forum crying. They have said as much in the past. Not even debatable. They may think "gee, what are the little hemorrhoids yapping about today?" and use that to decide what datasets to look at. But they simply aren't paid enough to read this crap daily.

It's a simple fact that they have their own data, but to figure out where people want balance changes without fixing something that's not broken you need... Feedback. You can use forums for that!