= 27d567bc2d48d9f40e9ccc20ea370b15)
gwern comments on Stupid Questions August 2015 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (129)
You can estimate the bias of each measurer much more efficiently if you have them measure the same sample, yes, analogous to crossover: now the differences are due less to the wide diversity of the sampled population and more to the particular measurer.
(To put it a little more mathily, when each measurer measures different samples, then the measurements will be spread very widely because it's Var(measurer-bias) + Var(population); but if we have the measurers measure the same sample, then Var(population) drops out and now there's just Var(measurer-bias). If I measure a sample and get 2.9 and you measure it as well and get 3.1, then probably the sample is really ~3.0 and my bias is -0.1 and your bias is +0.1. If I measure one sample and get 2.9 and you measure a different sample and get 3.1, then my bias and your bias are... ???)
For example, the classic example for MLMs is you have n classrooms' test scores, and you want to figure out the teachers' effects. It's hard to tell because the classrooms' average scores will differ a lot on their own. This is analogous to your original description: each measurer gets their own batch of samples. But what if you had a crossed design of one classroom with test scores after it's taught by each teacher? Then much of the differences in the average score will be due to the particular effect of each teacher and that will be much easier to estimate.
I guess. From a factor analysis perspective, you just want to pick the one with the highest loading on X, I think.
Huh. Your answer was even more useful for me than I expected. My 'secret agenda' is to put forth another mountant medium, which might have advantages over the one in use, but I will have to show that they do not differ in preparation quality. I think I am going to do a 2-by-2 crossover.
So - thank you! Analogies for the win!
The problem is that whatever one I will find the most desirable, other people will continue using the methods they are good at. And I will have to somehow compare x(A)1, x(B)32 and x(C)3...
And this is a relatively straightforward situation, things are often much less clear in environmental science, already on the methodology level.
I don't really understand the problem. Yes, maybe you can't control them and get everyone onto the same method page. But I've already explained how you deal with that, given you the relevant keywords to search for like 'measurement error', and also given you example R code implementing several approaches.
They all take the basic approach of treating it as data/measurements which load on a latent variable for each method, and each method loads on the latent variable which is what you actually want; then you can infer whatever you need to. The first level of latent variables helps you estimate the biases of each category, some of which may be smaller than others, and then you collectively use them to estimate the final latent variable. Now you have a principled way to unify all your data from disparate methods which measure in similar but not identical way the variable you care about. If someone else comes up with a new method, it can be incorporated like the rest.
Right - sorry, melting brain. (Also, I had just thought that the assumed 10% difference between two measurers has not, in fact, been established rigorously, and that derailed the still-solid brain...)