You can call it that. I call it refining the art of human rationality. I don't think building new knowledge is something that magically only happens in a box designated Academia.
No, but Academia is optimized for that and has hundreds of years of demonstrated effectiveness and accumulated experience.
Is it perfect? No.
Can you build something better from ground up? I don't think so, at least not at a cost smaller than the cost needed to improve it. Certainly the present LW doesn't look remotely like a superior alternative.
Remember SI did years of research basically outside it,
And what did they accomplish? Pretty much nothing, AFAIK.
they only started publishing so they could attract more talent and as a general PR move, not because it was the most efficient way to do it.
If that's their main reason to publish then it seems that they are doing the right thing for the wrong reason.
Are they so deluded to seriously believe they will be the first to build an AGI? I hope not. Therefore, if they want to have a chance to influence the development of AI projects, the best they can do is to disseminate the results of their research in a way that will maximize the number of AI researchers who will notice it and take it seriously. And this way is not Less Wrong, or an Harry Potter fanfiction or meetups or minicamps or all the other stuff they do. It's academic publishing. Academic publishing should be SI's raison d'etre, not a PR move.
So far, AFAIK (I'm not a SI historian, so I might be mistaken) they published a few papers on philosophy, on the same sort of topics the FHI people do (some of these papers are co-authored with them, IIUC). I didn't read all of them, but my impression is that they didn't contribute particularly novel insights.
We have yet to see what results their current research on program equilibrium will yield.
You don't seem to have read the related articles I cited. I strongly suggest you do.
I've skimmed them. The "Science: Do It Yourself" uses an examples that is flawed by a glaring methodological error from the start. That speaks lots of why scientific research, like many other activities, is something best left to professionals.
No, but Academia is optimized for that and has hundreds of years of demonstrated effectiveness and accumulated experience.
Is it perfect? No.
Actually the features of academia I'm criticizing are much younger than that. The modern peer review system is something Einstein didn't have to deal with for example. If you think hundreds of years of scientific progress are a good track record for a system I have some news for you...
..."Citizen science" is a fairly new term but an old practice. Prior to the 20th Century, science was often the pursuit of
Related to: Science: Do It Yourself, How To Fix Science, Rationality and Science posts from this sequence, Cargo Cult Science, "citizen science"
You think you have a good map, what you really have is a working hypothesis
You did some thought on human rationality, perhaps spurred by intuition or personal experience. Building it up you did your homework and stood on the shoulders of other people's work giving proper weight to expert opinion. You write an article on LessWrong, it gets up voted, debated and perhaps accepted and promoted as part of a "sequence". But now you'd like to do that thing that's been nagging you since the start, you don't want to be one of those insight junkies consuming fun plausible ideas forgetting to ever get around to testing them. Lets see how the predictions made by your model hold up! You dive into the literature in search of experiments that have conveniently already tested your idea.
It is possible there simply isn't any such experimental material or that it is unavailable. Don't get me wrong, if I had to bet on it I would say it is more likely there is at least something similar to what you need than not. I would also bet that some things we wish where done haven't been so far and are unlikely to be for a long time. In the past I've wondered if we can in the future expect CFAR or LessWrong to do experimental work to test many of the hypotheses we've come up with based on fresh but unreliable insight, anecdotal evidence and long fragile chains of reasoning. This will not happen on its own.
With mention of CFAR, the mind jumps to them doing expensive experiments or posing long questionnaires with small samples of students and then publishing papers, like everyone else does. It is the respectable thing to do and it is something that may or may not be worth their effort. It seems doable. The idea of LWers getting into the habit of testing their ideas on human rationality beyond the anecdotal seems utterly impractical. Or is it?
That ordinary people can band together to rapidly produce new knowledge is anything but a trifle
How useful would it be if we had a site visited by thousands or tens of thousands solving forms or participating in experiments submitted by LessWrong posters or CFAR researchers? Something like this site. How useful would it be if we made such a data set publicly available? What if we could in addition to this data mine how people use apps or an online rationality class? At this point you might be asking yourself if building knowledge this way even possible in fields that takes years to study. A fair question, especially for tasks that require technical competence, the answer is yes.
I'm sure many at this point, have started wondering about what kinds of problems biased samples might create for us. It is important to keep in mind what kind of sample of people you get to participate in the experiment or fill out your form, since this influences how confident you are allowed to be about generalizations. Learning things about very specific kinds of people is useful too. Recall this is hardly a unique problem, you can't really get away from it in the social sciences. WEIRD samples aren't weird in academia. And I didn't say the thousands and tens of thousands people would need to come from our own little corner of the internet, indeed they probably couldn't. There are many approaches to getting them and making the sample as good as we can. Sites like yourmorals.org tried a variety of approaches we could learn from them. Even doing something like hiring people from Amazon Mechanical Turk can work out surprisingly well.
LessWrong Science: We do what we must because we can
The harder question is if the resulting data would be used at all. As we currently are? I don't think so. There are many publicly available data sets and plenty of opportunities to mine data online, yet we see little if any original analysis based on them here. We either don't have norms encouraging this or we don't have enough people comfortable with statistics doing so. Problems like this aren't immutable. The Neglected Virtue of Scholarship noticeably changed our community in a similarly profound way with positive results. Feeling that more is possible I think it is time for us to move in this direction.
Perhaps just creating a way to get the data will attract the right crowd, the quantified self people are not out of place here. Perhaps LessWrong should become less of a site and more of a blogosphere. I'm not sure how and I think for now the question is a distraction anyway. What clearly can be useful is to create a list of models and ideas we've already assimilated that haven't been really tested or are based on research that still awaits replication. At the very least this will help us be ready to update if relevant future studies show up. But I think that identifying any low hanging fruit and design some experiments or attempts at replication, then going out there and try to perform them can get us so much more. If people have enough pull to get them done inside academia without community help great, if not we should seek alternatives.