Thanks, that's a great point.

I'm worried that a statistical calculator will throw off founders who would otherwise test their products with us (specifically YC founders, an abnormally influential group), so as much as possible I'd like to keep sample sizes in the "Advanced Menu" section. (This is not to say this is an unimportant issue -- I'm saying this is a more important issue because many people won't be customizing the default values).

I also think there are three unique features for product studies that can help simply defining good default values here:

• Startups are going to be interested in talking about big improvements (small sample sizes needed).
• Startups will likely view study participation as advertising, allowing for a generous margin of error on sample size.
• Consumers are skeptical of low-sample size studies, even when they shouldn't be.

What do you suggest we do? It sounds like getting baseline mean and variance data for the questions we include with the app is basically a requirement.

What is the awesome version of handling this issue? :p