You ship both a placebo package and a non-placebo package to the participant, and have them flip a coin to decide which one to use. They either throw away or disregard the other package for the duration of the study.
That's possible but it means that you double your product costs. The advantage would be that you can do crossover trials.
In the case of your probiotic a crossover trial might to worthwhile even without this reason.
In this case you could encrypt a list with products ID for placebos and non-placebos before you ship your product. The participant has no opportunity to know which of the two products are placebos.
When he starts the study the participant puts the product ID of the product he decides to use into the webapp. He also puts the ID of the product he doesn't want to use in the webapp.
The participant has no way to decide between the two packages or know which one is the placebo and which one is the real thing so he doesn't need to go through the process of flipping a real coin.
Once the study is finished you releases the decryption key that can be used to distinguish placebos from real products. You can give the decryption key to a trusted third-party organisation so that your company can't prevent the results of the study from being published.
You ship N packages to Total/N participants. The participants which receive N packages then randomly assigns himself a package, and randomly distributes the remaining (N-1) packages to other participants.
That means the participants has to do the work of going to the post office and remailing packages. Some of them will require additional time to remail and it might produce complications.
About the shipping of products and placebos to people, I see a physical way of doing it, but it is definitely not scalable.
Maybe you can win a company such as Amazon as a partner for distribution. Amazon warehouses can store both products and placebos and ship randomly.
For Amazon it should be relatively little work and it could be good PR.
Labels don't have to be printed on the bottle. Amazon has the capabilities of adding a paper with study instructions to a shipment and that paper can contain a unique ID for the product. Amazon could again publish an encrypted version of the ID at the beginning of the study and release the decryption key at the end of the study.
The participant has no way to decide between the two packages or know which one is the placebo and which one is the real thing so he doesn't need to go through the process of flipping a real coin.
I am worried that standardized shipping will come with standardized package layout, and I'm guessing "preference of left vs right identical thing" correlates with something the system will eventually test. Having thought about it more, this is the real issue with allowing customers to choose which product they'll use: that decision has to be purely ra...
I'm a LW reader, two time CFAR alumnus, and rationalist entrepreneur.
Today I want to talk about something insidious: marketing studies.
Until recently I considered studies of this nature merely unfortunate, funny even. However, my recent experiences have caused me to realize the situation is much more serious than this. Product studies are the public's most frequent interaction with science. By tolerating (or worse, expecting) shitty science in commerce, we are undermining the public's perception of science as a whole.
The good news is this appears fixable. I think we can change how startups perform their studies immediately, and use that success to progressively expand.
Product studies have three features that break the assumptions of traditional science: (1) few if any follow up studies will be performed, (2) the scientists are in a position of moral hazard, and (3) the corporation seeking the study is in a position of moral hazard (for example, the filing cabinet bias becomes more of a "filing cabinet exploit" if you have low morals and the budget to perform 20 studies).
I believe we can address points 1 and 2 directly, and overcome point 3 by appealing to greed.
Here's what I'm proposing: we create a webapp that acts as a high quality (though less flexible) alternative to a Contract Research Organization. Since it's a webapp, the cost of doing these less flexible studies will approach the cost of the raw product to be tested. For most web companies, that's $0.
If we spend the time to design the standard protocols well, it's quite plausible any studies done using this webapp will be in the top 1% in terms of scientific rigor.
With the cost low, and the quality high, such a system might become the startup equivalent of citation needed. Once we have a significant number of startups using the system, and as we add support for more experiment types, we will hopefully attract progressively larger corporations.
Is anyone interested in helping? I will personally write the webapp and pay for the security audit if we can reach quorum on the initial protocols.
Companies who have expressed interested in using such a system if we build it:
(I sent out my inquiries at 10pm yesterday, and every one of these companies got back to me by 3am. I don't believe "startups love this idea" is an overstatement.)
So the question is: how do we do this right?
Here are some initial features we should consider:
Any placebos used in the studies must be available for purchase as long as the results are used in advertising, allowing for trivial study replication.
Significant contributors will receive:
I'm hoping that if a system like this catches on, we can get an "effective startups" movement going :)
So how do we do this right?