= 27d567bc2d48d9f40e9ccc20ea370b15)
joshkaufman comments on Help with a (potentially Bayesian) statistics / set theory problem? - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (27)
If we allow inconsistency, i.e. if you plan on using this in the real world, then you could get the response B>A, A>C, C>B. That is, there may not be any such thing as a most preferred object, and thus no such thing as the "probability of being the 'most preferred' object."
A workaround would be to ask a related question: "if object 1 is A, and object 2 is unknown, what is the probability that A>2?" The most preferred object would just be the object with the highest answer to this question.
The alternate path would be to assume that there is a preference ordering from 1 to 7, with some sort of noise applied. This feels clunky to me, though.
Good point about inconsistency... I was thinking that individual responses may be inconsistent, but the aggregated responses of the group might reveal a significant preference.
My first crack at this was to use a simple voting system, where B from {A,B} means +1 votes for B, 0 for A, largest score when all participant votes are tallied wins. What messes that up is participants leaving without completing the entire set, which introduces selection bias, even if the sets are served at random.
Preference ordering / ranking isn't ideal because it takes longer, which may encourage the participant to rationalize. The "pick-one" approach is designed to be fast, which is better for gut reactions when comparing words, photos, etc.
If the aggregated preferences are transitive (i.e., 'not inconsistent' in your and Manfred's wording), then this preference relation defines a total order on the objects, and there is a unique object that is preferred to every other object (in aggregate). (Furthermore, this is isomorphic to the set {1,2,3,...,7} under the ≤ relation.)
As I understand things, you have no guarantee of transitivity in the aggregated preferences even if you do have transitivity in the individual preferences.
Yes, of course you are correct.
Very helpful - reading about this now. Starting here: http://en.wikipedia.org/wiki/Ranking