= 27d567bc2d48d9f40e9ccc20ea370b15)
evand comments on Fixing Moral Hazards In Business Science - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (96)
Thanks for pointing this out.
Let's use Beeminder as an example. When I emailed Daniel he said this: "we've talked with the CFAR founders in the past about setting up RCTs for measuring the effectiveness of beeminder itself and would love to have that see the light of day".
Which is a little open ended, so I'm going to arbitrarily decide that we'll study Beeminder for weight loss effectiveness.
Story* as follows:
Daniel goes to (our thing).com and registers a new study. He agrees to the terms, and tells us that this is a study which can impact health -- meaning that mandatory safety questions will be required. Once the trial is registered it is viewable publicly as "initiated".
He then takes whatever steps we decide on to locate participants. Those participants are randomly assigned to two groups: (1) act normal, and (2) use Beeminder to track exercise and food intake. Every day the participants are sent a text message with a URL where they can log that day's data. They do so.
After two weeks, the study completes and both Daniel and the world are greeted with the results. Daniel can now update Beeminder.com to say that Beeminder users lost XY pounds more than the control group... and when a rationalist sees such claims they can actually believe them.
I sincerely hope that study plan would not pass muster. Doesn't there need to be a more reasonable placebo?
In general, who will review proposed studies for things like suitable placebo decisions?
Can you provide an example of what you'd like to see pass muster?
Roughly speaking: "Act normal" vs "Use Beeminder" vs "some alternative intervention". Basically, I expect to see "do something different" produce results, at least for a little while, for almost any value of "something different". Literally anything at all that didn't make it clear to the placebo group that they were the placebo group. Maybe some non-Beeminder exercise and intake tracking. Maybe a prescribed simple exercise routine + non-Beeminder tracking.
I'm glad you're here. My background is in backend web software, and stats once the data has been collected. I read "Measuring the Weight of Smoke" in college, but that's not really a sufficient background to design the general protocol. That's a lot of my motivation behind posting this to LW - there seem to be protocol experts here, with great critiques of the existing ones.
My hope is we can create a "getting started testing" document that gets honest companies on the right track. Searching around the web I'm finding things like this rather than serious guides to proper placebo creation.
I'm hoping either registered statistical consultants or grad students. Hopefully this can be streamlined by a good introductory guide.