A next step in setting up a decent experiment would be to select 4 groups (1, 2, 3 + Control) of people randomly. To give findings most applicable to rational people, we can just as well select within LW.
Another step: Select a common task the utility of which is perceived as high by most people on LW, yet which is seldomly performed.
Proposal: During two weeks, run for (at least) 10 Minutes on three of each four consecutive days.
Ideally, the experiment would then continue with another task, but switching group members. This improves the experiment via control for different motivation between participants. Also trying a different task gives us a chance to see if there is a general method or whether the method must be selected specific to the task.
If anybody is willing to participate in this experiment, I am willing to coordinate. If somebody else is willing (and more qualified) to coordinate, I hereby enlist as participant.
Qualification: I am a computer scientist, I have one contact to a psychology researcher, I feel competent in statistics. Then again I lost 20$ (of 100$) over 3 month in prediction markets.
I would also join, but we should take Cousin_It's suggestions. And we should send our ideas to some psychologists for feedback.
It seems to me that this blog has just reached it's first real crisis.
Three people are announcing three apparently opposed beliefs with substantial real expected consequences and yet no-one has yet spoken, or it seems to me implied, the key slogan... "LETS USE SCIENCE!" or, as hubristic Bayesian wannabes, not invoked Bayes as an idol to swear by, but rather said "LETS USE HUMANE REFLECTIVE DECISION THEORY, THE QUANTITATIVELY UNKNOWN BUT QUALITATIVELY INTUITED POWER DEEPER THAN SCIENCE FROM WHICH IT STEMS AND TO WHICH OUR COMMUNITY IS DEVOTED".
IF RDS was applied to our current situation, people would be analyzing Yvain's, Davis' and Eby's proposals, working out exactly what their implications are, and trying to propose, in the name of SCIENCE, hypotheses which will distinguish between them, and in the name of BAYES, confidence estimates of their analyses and of the quality with which the denotations of their words have cleaved reality at the joints enabling an odds ratio of updating to be extracted from a single data point. People would be working out what features of which of the models used by Yvain, Davis and Eby constitute evidence against what other features. They would be trying to evaluate non-verbally, through subjectively opaque but known-to-be-informative processes vulnerable to verbal overshadowing, what relative odds to place on those different features of the models. Finally, they would be examining the expected costs entailed by experiments being proposed and selecting those experiments which promise to provide the most information for the least cost be performed. The cost estimate would include both the effort required to perform the experiments, probably best assessed with an outside view in most cases like these, and the dangers to the minds of the participants from possible adverse outcomes, taking into account, as well as possible, the structural uncertainty of the models.
I sincerely hope to see some of that in the comments section soon, either under this post or the "Applied Picoeconomics" post.