I'm a LW reader, two time CFAR alumnus, and rationalist entrepreneur.
Today I want to talk about something insidious: marketing studies.
Until recently I considered studies of this nature merely unfortunate, funny even. However, my recent experiences have caused me to realize the situation is much more serious than this. Product studies are the public's most frequent interaction with science. By tolerating (or worse, expecting) shitty science in commerce, we are undermining the public's perception of science as a whole.
The good news is this appears fixable. I think we can change how startups perform their studies immediately, and use that success to progressively expand.
Product studies have three features that break the assumptions of traditional science: (1) few if any follow up studies will be performed, (2) the scientists are in a position of moral hazard, and (3) the corporation seeking the study is in a position of moral hazard (for example, the filing cabinet bias becomes more of a "filing cabinet exploit" if you have low morals and the budget to perform 20 studies).
I believe we can address points 1 and 2 directly, and overcome point 3 by appealing to greed.
Here's what I'm proposing: we create a webapp that acts as a high quality (though less flexible) alternative to a Contract Research Organization. Since it's a webapp, the cost of doing these less flexible studies will approach the cost of the raw product to be tested. For most web companies, that's $0.
If we spend the time to design the standard protocols well, it's quite plausible any studies done using this webapp will be in the top 1% in terms of scientific rigor.
With the cost low, and the quality high, such a system might become the startup equivalent of citation needed. Once we have a significant number of startups using the system, and as we add support for more experiment types, we will hopefully attract progressively larger corporations.
Is anyone interested in helping? I will personally write the webapp and pay for the security audit if we can reach quorum on the initial protocols.
Companies who have expressed interested in using such a system if we build it:
- Beeminder
- HabitRPG
- MealSquares
- Complice (disclosure: the CEO, Malcolm, is a friend of mine)
- General Biotics (disclosure: the CEO, David, is me)
(I sent out my inquiries at 10pm yesterday, and every one of these companies got back to me by 3am. I don't believe "startups love this idea" is an overstatement.)
So the question is: how do we do this right?
Here are some initial features we should consider:
- Data will be collected by a webapp controlled by a trusted third party, and will only be editable by study participants.
- The results will be computed by software decided on before the data is collected.
- Studies will be published regardless of positive or negative results.
- Studies will have mandatory general-purpose safety questions. (web-only products likely exempt)
- Follow up studies will be mandatory for continued use of results in advertisements.
- All software/contracts/questions used will be open sourced (MIT) and creative commons licensed (CC BY), allowing for easier cross-product comparisons.
Any placebos used in the studies must be available for purchase as long as the results are used in advertising, allowing for trivial study replication.
Significant contributors will receive:
- Co-authorship on the published paper for the protocol.
- (Through the paper) an Erdos number of 2.
- The satisfaction of knowing you personally helped restore science's good name (hopefully).
I'm hoping that if a system like this catches on, we can get an "effective startups" movement going :)
So how do we do this right?
To work well, I think it needs a good name. In terms of long term social dynamics, creating a meta-brand that helps smaller brands seems essential. Like when people initially see the "tested by X" logo they won't know what it means.
Assuming the web app works as intended, and assuming any significant fraction the population just stop believing any of the classes of claims that might be tested this way and lack the logo, then the process should gain more and more credibility over the course of months and years. The transition from an unknown logo to a trusted logo will be tricky for the larger institutional hack to work, and the name itself might be key to the logic of acceptance at the beginning.
I ground through various options at the command line with $ whois $OPTION | grep "[A-Z].COM"... trying to find things that get the right idea and aren't already registered.
Namespace is huge and finding a good name seems key. The names I looked for my be too boring or too long or too easy to misspell? Please comment in response to this comment, one name suggestion per comment, and then find the 3 best suggestions from other people (assuming that there are lots to choose from) and vote them up :-)
Edited to add: I'm seeing lots of votes and no suggestions. Also, ProveItForReal seems to be winning but I think that works better in a {{citation needed}} context (ie you say {{prove it for real}} to dubious claims) but it works less well for logos on products. Imagine a logo that is worked into product's packaging that says: "TestedClaim: X gives benefit Y in Z% of users"... that seems good in that context, but {{this needs to be a tested claim}} is awkward. Surely something better is possible than either of these?
simpleproofoftruth.com
Acronym: SPOT, sounds kind of neat with "SPOT test" or "SPOT tested".
It also works as a potential prod like citation needed... {{ simple proof of truth needed }}
One thing that slightly bothers me is that it relies on an older and less precise sense of the word "proof" that comes more from english common law than from mathematics, and the mathematical sense of the word "proof" is dear to my heart.