What clearly can be useful is to create a list of models and ideas we've already assimilated that haven't been really tested or are based on research that still awaits replication.

I like the idea and agree this could be useful, but our first focus should be on reproducing well-established results to make sure that Less Wrong has the ability to so. Afterward, if we succeed, then it makes sense to proceed cautiously to replicating weak results or testing new hypotheses. Otherwise, if we don't succeed in reproducing well-established results, then we lack the ability to investigate more speculative ideas.

We face numerous additional burdens over experimentalists in academia, two of which stand out to me: (1) a currently unknown sampling bias and response rate, and (2) a general inability to blind our Less Wrong subjects to the hypotheses and predictions of our experiments.

It may be instructive to take a look at some of the recent papers on online psychometric assessment in personality psychology. They have worked on establishing the effects of online assessment (as opposed to traditional paper-and-pencil or face-to-face assessments) and proposed guidelines for doing experiments online:

• Buchanan, Tom. 2002. “Online Assessment: Desirable or Dangerous?” Professional Psychology: Research and Practice 33 (2): 148–154. doi:10.1037/0735-7028.33.2.148.
• Davis, Robert N. 1999. “Web-based Administration of a Personality Questionnaire: Comparison with Traditional Methods.” Behavior Research Methods, Instruments, & Computers 31 (4) (December): 572–577. doi:10.3758/BF03200737.
• Gosling, Samuel D., Simine Vazire, Sanjay Srivastava, and Oliver P. John. 2004. “Should We Trust Web-based Studies? A Comparative Analysis of Six Preconceptions About Internet Questionnaires.” American Psychologist 59 (2): 93–104. doi:10.1037/0003-066X.59.2.93.
• Joubert, Tina, and Hendrik J. Kriek. 2009. “Psychometric Comparison of Paper-and-pencil and Online Personality Assessments in a Selection Setting.” SA Journal of Industrial Psychology 35 (1) (April 9): 78–88. doi:10.4102/sajip.v35i1.727.
• Naus, Mary J., Laura M. Philipp, and Mekhala Samsi. 2009. “From Paper to Pixels: A Comparison of Paper and Computer Formats in Psychological Assessment.” Computers in Human Behavior 25 (1) (January): 1–7. doi:10.1016/j.chb.2008.05.012.
• Ployhart, Robert E., Jeff A. Weekley, Brian C. Holtz, and Cary Kemp. 2003. “Web-based and Paper-and-pencil Testing of Applicants in a Proctored Setting: Are Personality, Biodata, and Situational Judgment Tests Comparable?” Personnel Psychology 56 (3) (September): 733–752. doi:10.1111/j.1744-6570.2003.tb00757.x.
• Reips, Ulf-Dietrich. 2002. “Standards for Internet-based Experimenting.” Experimental Psychology 49 (4) (October 1): 243–256. doi:10.1026//1618-3169.49.4.243.
• Rothstein, Mitchell G., and Richard D. Goffin. 2006. “The Use of Personality Measures in Personnel Selection: What Does Current Research Support?” Human Resource Management Review 16 (2) (June): 155–180. doi:10.1016/j.hrmr.2006.03.004.
• Salgado, Jesús F., and Silvia Moscoso. 2003. “Internet-based Personality Testing: Equivalence of Measures and Assesses’ Perceptions and Reactions.” International Journal of Selection and Assessment 11 (2-3) (June): 194–205. doi:10.1111/1468-2389.00243.
• Vecchione, Michele, Guido Alessandri, and Claudio Barbaranelli. 2012. “Paper-and-pencil and Web-based Testing: The Measurement Invariance of the Big Five Personality Tests in Applied Settings.” Assessment 19 (2) (June): 243–246. doi:10.1177/1073191111419091.