I'm a LW reader, two time CFAR alumnus, and rationalist entrepreneur.

Today I want to talk about something insidious: marketing studies.

Until recently I considered studies of this nature merely unfortunate, funny even. However, my recent experiences have caused me to realize the situation is much more serious than this. Product studies are the public's most frequent interaction with science. By tolerating (or worse, expecting) shitty science in commerce, we are undermining the public's perception of science as a whole.

The good news is this appears fixable. I think we can change how startups perform their studies immediately, and use that success to progressively expand.

Product studies have three features that break the assumptions of traditional science: (1) few if any follow up studies will be performed, (2) the scientists are in a position of moral hazard, and (3) the corporation seeking the study is in a position of moral hazard (for example, the filing cabinet bias becomes more of a "filing cabinet exploit" if you have low morals and the budget to perform 20 studies).

I believe we can address points 1 and 2 directly, and overcome point 3 by appealing to greed.

Here's what I'm proposing: we create a webapp that acts as a high quality (though less flexible) alternative to a Contract Research Organization. Since it's a webapp, the cost of doing these less flexible studies will approach the cost of the raw product to be tested. For most web companies, that's $0.

If we spend the time to design the standard protocols well, it's quite plausible any studies done using this webapp will be in the top 1% in terms of scientific rigor.

With the cost low, and the quality high, such a system might become the startup equivalent of citation needed. Once we have a significant number of startups using the system, and as we add support for more experiment types, we will hopefully attract progressively larger corporations.

Is anyone interested in helping? I will personally write the webapp and pay for the security audit if we can reach quorum on the initial protocols.

Companies who have expressed interested in using such a system if we build it:

• Beeminder
• HabitRPG
• MealSquares
• Complice (disclosure: the CEO, Malcolm, is a friend of mine)
• General Biotics (disclosure: the CEO, David, is me)

(I sent out my inquiries at 10pm yesterday, and every one of these companies got back to me by 3am. I don't believe "startups love this idea" is an overstatement.)

So the question is: how do we do this right?

Here are some initial features we should consider:

• Data will be collected by a webapp controlled by a trusted third party, and will only be editable by study participants.
• The results will be computed by software decided on before the data is collected.
• Studies will be published regardless of positive or negative results.
• Studies will have mandatory general-purpose safety questions. (web-only products likely exempt)
• Follow up studies will be mandatory for continued use of results in advertisements.
• All software/contracts/questions used will be open sourced (MIT) and creative commons licensed (CC BY), allowing for easier cross-product comparisons.

Any placebos used in the studies must be available for purchase as long as the results are used in advertising, allowing for trivial study replication.

Significant contributors will receive:

• Co-authorship on the published paper for the protocol.
• (Through the paper) an Erdos number of 2.
• The satisfaction of knowing you personally helped restore science's good name (hopefully).

I'm hoping that if a system like this catches on, we can get an "effective startups" movement going :)

So how do we do this right?

Recently I have started using Anki in a new and complimentary way. I am curious if any of you find it similarly useful and/or have other anki tips :)

The basic idea is that instead of putting down a challenge/response pair for facts, we put down a challenge/response pair for ways of thinking. A train of thought. Ideally, this is something akin to a lumosity.com game except tailored to your area of focus.

A simple example from algebra:

The fact based approach would be to make a card titled "what is the quadratic formula?" with the answer of "x == (-b +- Sqrt(b^2 - 4ac)) / 2a"

The way I am recommending is to make a card titled "derive the answer for ax^2 + bx + c == 0" with an answer that shows the steps. When shown the card, you would then either solve it in your head, or using a pad of paper. I assume that sub minute tasks are ideal.

The specific area I have been using this in is the study of algorithms, with challenges like "Hopcraft-Karp algorithm for bipartite matching", and it has so far proved very helpful at getting fluent with the names, deepening my understanding of the algorithms themselves, and with seeing new places to apply them in my coding.

This might be overstepping, but something like this seems like it might be appropriate for The Center for Modern Rationality. Something like "critique the logic of the following three sentences", or "Sue is about to buy a car. How should she go about making a decision".

This is my first real post to LessWrong, so if you have style corrections those are solicited alongside any comments on the post itself. Thanks!