But I was more thinking about one bias appearing stronger when a bias in another direction is eliminated. For example bias X makes people think A, bias Y makes people think B, if a person is under influence of both biases, the answer is randomly A or B. In such case, eliminating bias X leads to increase of answer B.
Biases don't have clear 'directions' often. If you are overconfident on a claim P, that's just as accurate as saying you were underconfident on claim ~P. Similarly for anchoring or priming - if you anchor on the random number generator while estimating number of African nations, whether you look "over" or "under" is going to depend on whether the RNG was spitting out 1-50 or 100-200, perhaps.
I would like to be at least 95% certain
And what does that mean? If you just want to know 'what do smart people in general believe versus normal people', you don't need large samples if you can get a random selection and your questions are each independent. For example, in my recent Wikipedia experiment I removed only 100 links and 3 were reverted; when I put that into a calculator for a Bernouilli distribution, I get 99% certainty that the true reversion rate is 0-7%. So to simplify considerably, if you sampled 100 smart people and 100 dumb people and they differ by 14%, is that enough certainty for you?
So to simplify considerably, if you sampled 100 smart people and 100 dumb people and they differ by 14%, is that enough certainty for you?
I am not good at statistics, but I guess yes. Especially if those 100 people are really randomly selected, which in the given situation they were.
I would like to point you guys to the blog FreakoStats, by Garth Zietsman, who according to his profile, "Scored an IQ of 185 on the Mega27 and has a degree in psychology and statistics and 25 years experience in psychometrics and statistics." The main concept discussed there is "The Smart Vote", whose essence, in the author’s words, is as follows:
Many of his posts are based on the choice of relevant, if controversial, topics , and his analysis of the direction and the proportionality of which an opinion on it is related to intelligence, as measured by IQ scores.
An obvious objection would be that smart people would have in many cases common interests, and this could bias the results simply to their interests, in detriment to the interests of the less smart.
Zietsman answers it with the statistical fact that people many times don’t vote selfishly, and that he can (and will) control for some of their interests anyway:
This seems somewhat related to the notorious concept of Coherent Extrapolation Volition (CEV) of humankind. To have a clue of the direction it might take, I believe it is a nice idea to look at the opinion of the smarter (“if we knew more, thought faster, […]”), corrected for selfish interests (“[…]were more the people we wished we were, had grown up farther together[…]”), specially bearing in mind that most results tend to converge (“[…]where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated[…]”).
There are analyses on issues such as Abortion, Free Speech ,Capital Punishment and Corporal Punishments on Children ,Immigration, Gay Rights and many more. The results look good to me personally, and I wouldn't be surprised if they pleased many here too.