Factor analysis/measurement error/multilevel models/Value of Information: X is a latent variable, which yields more latent variables (one for each kind of method), which are themselves measured with error by the raw datapoints. So you have multiple kinds of measurements, each with their own error, giving you a multilevel model. You can write a multilevel model expressing this in a Bayesian language like JAGS or you could use a SEM library like lavaan, where it'd be something like 'x ~ A, x ~ B, A ~ a-datapoints, B ~ b-datapoints'...
(To give an analogy: imagine you are measuring Gf. Gf is a latent variable which is predicted from things like WM or executive function; WM and executive function are themselves latent variables, which are measured by tests like forwards digit span. The graph would look like a little pyramid. So you measure someone's intelligence by doing forwards digit span several times, giving you a reasonably precise estimate of the latent variable WM, which then gives you a imprecise estimate of the highest latent variable Gf.)
Measurer 1 wants to know if it is meaningful to compare her results, x(A)1, with Measurer 2's results, x(B)2. Does the interval in which the true x lies include 40 points?
Comparing measurer 1 and measurer 2's results is not really the same thing as simply asking for the posterior distribution of the latent x, but yes, with the posterior, it's easy to calculate the probability of anything you like such as '>=40' or '<=90'.
If Measurer 1 herself establishes the difference between x(C)1 and x(D)1, where C and D are two other ways to measure x, how much more useful for any given Measurer 3 will be her results, if she also invites Measurer 2's opinion - that is, x(C)2 and x(D)2?
I only know a Bayesian approach here: it sounds like Expected Value of Sample Information. You need a loss function on error (mean squared, perhaps?) and then you can repeatedly sample from the posterior based on all of Measurer 3's data as a hypothetical, and then look at how much loss is reduced based an additional sample (or more) from Measurer 2.
(You could always go ask the Statistics Stack Overflow.)
Thank you! (In part, for such faith in my abilities:) Have to go hunt myself a programmer for dinner...)
It seems that if M-r 1 gives M-r 2 the same subsample of the middle latent variables (photoes of fields of vision, scoring them gives you the datapoints), and the x1 is compared with x2, they can see the least difference between them, which is (largely?) sample-independent. If, however, M-r 1 and M-r 2 each draw their subsamples independently, the difference between x1 and x2 should be larger due to chance, right?.. So if we look at the difference in dif...
This thread is for asking any questions that might seem obvious, tangential, silly or what-have-you. Don't be shy, everyone has holes in their knowledge, though the fewer and the smaller we can make them, the better.
Please be respectful of other people's admitting ignorance and don't mock them for it, as they're doing a noble thing.
To any future monthly posters of SQ threads, please remember to add the "stupid_questions" tag.