= 27d567bc2d48d9f40e9ccc20ea370b15)
RichardKennaway comments on Open Thread April 16 - April 22, 2014 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (190)
The five results so far go against that.
You can't detect whether a systematic bias in the sampling method exists by looking at the results.
If you have a prior, you can.
In a slightly more unrolled manner, if the results you are getting are inconsistent with your ideas of how the world works, one hypothesis that you should update is that your ideas about the world are wrong. But another hypothesis that you should also update is that your sampling method is wrong, e.g. by having a systematic bias.
Sure you can, in principle. When you have measured covariates, you can compare their sampled distribution to that of the population of interest. Find enough of a difference (modulo multiple comparisons, significance, researcher degrees of freedom, etc.) and you've detected bias. Ruling out systematic bias using your observations alone is much more difficult.
Even in this case, where we don't have covariates, there are some patterns in the ordinal data (the concept of ancillary statistics might be helpful in coming up with some of these) that would be extremely unlikely under unbiased sampling.
That means that you need more data. Having a standard against which to train your model means that you need more than just the results of your measurement.
I was just contesting your statement as a universal one. For this poll, I agree you can't really pursue the covariate strategy. However, I think you're overstating challenge of getting more data and figuring out what to do with it.
For example, measuring BPD status is difficult. You can do it by conducting a psychological examination of your subjects (costly but accurate), you can do it by asking subjects to self-report on a four-level Likert-ish scale (cheap but inaccurate), or you could do countless other things along this tradeoff surface. On the other hand, measuring things like sex, age, level of education, etc. is easy. And even better, we have baseline levels of these covariates for communities like LessWrong, the United States, etc. with respect to which we might want to see if our sample is biased.
You argued against a more general statement than the one I made. But I did choose my words in a way that focused on drawing conclusions from the results and not results + comparison data.
Leaving aside the sample size, a sample value of zero cannot be an overestimate.
There no reason to leave aside sample size. The value was zero by nature of low sampling size.
The observed reality that the first 5 people voted that BPD doens't apply to them provides nearly zero bayesian evidence against the idea of systematic bias by surveying in that manner.
While ignoring the sample size, I'd put a high probability on my comment having something to do with the intense response in the other direction. (I am not even sure how you can read all of it and not think that it is at least 'poorly descriptive', no matter who you are)