LESSWRONG
LW

You did some thought on human rationality, perhaps spurred by intuition or personal experience. Building it up you did your homework and stood on the shoulders of other people's work giving proper weight to expert opinion. You write an article on LessWrong, it gets up voted, debated and perhaps accepted and promoted as part of a "sequence". But now you'd like to do that thing that's been nagging you since the start, you don't want to be one of those insight junkies consuming fun plausible ideas forgetting to ever get around to testing them. Lets see how the predictions made by your model hold up! You dive into the literature in search of experiments that have conveniently already tested your idea.

It is possible there simply isn't any such experimental material or that it is unavailable. Don't get me wrong, if I had to bet on it I would say it is more likely there is at least something similar to what you need than not. I would also bet that some things we wish where done haven't been so far and are unlikely to be for a long time. In the past I've wondered if we can in the future expect CFAR or LessWrong to do experimental work to test many of the hypotheses we've come up with based on fresh but unreliable insight, anecdotal evidence and long fragile chains of reasoning. This will not happen on its own.

With mention of CFAR, the mind jumps to them doing expensive experiments or posing long questionnaires with small samples of students and then publishing papers, like everyone else does. It is the respectable thing to do and it is something that may or may not be worth their effort. It seems doable. The idea of LWers getting into the habit of testing their ideas on human rationality beyond the anecdotal seems utterly impractical. Or is it?

That ordinary people can band together to rapidly produce new knowledge is anything but a trifle

How useful would it be if we had a site visited by thousands or tens of thousands solving forms or participating in experiments submitted by LessWrong posters or CFAR researchers? Something like this site. How useful would it be if we made such a data set publicly available? What if we could in addition to this data mine how people use apps or an online rationality class? At this point you might be asking yourself if building knowledge this way even possible in fields that takes years to study. A fair question, especially for tasks that require technical competence, the answer is yes.

I'm sure many at this point, have started wondering about what kinds of problems biased samples might create for us. It is important to keep in mind what kind of sample of people you get to participate in the experiment or fill out your form, since this influences how confident you are allowed to be about generalizations. Learning things about very specific kinds of people is useful too. Recall this is hardly a unique problem, you can't really get away from it in the social sciences. WEIRD samples aren't weird in academia. And I didn't say the thousands and tens of thousands people would need to come from our own little corner of the internet, indeed they probably couldn't. There are many approaches to getting them and making the sample as good as we can. Sites like yourmorals.org tried a variety of approaches we could learn from them. Even doing something like hiring people from Amazon Mechanical Turk can work out surprisingly well.

LessWrong Science: We do what we must because we can

The harder question is if the resulting data would be used at all. As we currently are? I don't think so. There are many publicly available data sets and plenty of opportunities to mine data online, yet we see little if any original analysis based on them here. We either don't have norms encouraging this or we don't have enough people comfortable with statistics doing so. Problems like this aren't immutable. The Neglected Virtue of Scholarship noticeably changed our community in a similarly profound way with positive results. Feeling that more is possible I think it is time for us to move in this direction.

Perhaps just creating a way to get the data will attract the right crowd, the quantified self people are not out of place here. Perhaps LessWrong should become less of a site and more of a blogosphere. I'm not sure how and I think for now the question is a distraction anyway. What clearly can be useful is to create a list of models and ideas we've already assimilated that haven't been really tested or are based on research that still awaits replication. At the very least this will help us be ready to update if relevant future studies show up. But I think that identifying any low hanging fruit and design some experiments or attempts at replication, then going out there and try to perform them can get us so much more. If people have enough pull to get them done inside academia without community help great, if not we should seek alternatives.

EmpiricismCommunityWorld Modeling

Frontpage

42

How about testing our ideas?

New Comment

113 comments, sorted by

top scoring

Click to highlight new comments since: Today at 10:04 AM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Vaniver12y330

I'm happy to see a push for increased empiricism and scientific effort on LW. But... I wish there were more focus on the word "how," and less focus on the word 'we.'

Three articles come to mind: To Lead You Must Stand Up, First, Try to Make it to the Mean, and Money: The Unit of Caring. (Only the first part of the second article will be directly relevant, but the latter parts are indirectly relevant.)

That is:

First, there's insufficient focus on what concrete steps you are taking to move the culture in that direction. (Writing blog posts exhorting action does not count for much. Do you think The Neglected Virtue of Scholarship would have shifted community actions as much if lukeprog hadn't followed it up by writing posts with massive reference lists?) The reference to yourmorals.org is fine, but what made that site important was a particular feature, not its goal or its structure. If you've thought of a similar feature that someone (ideally you) could code up, great! I will send as much karma as I can towards the person that makes that happen. But this is even more general than a call for better / easier rationality tests and exercises, and thus even less likely to cause co... (read more)

[-][anonymous]12y110

To Lead You Must Stand Up

A little over a week ago me and two other LWers started doing research on the possibilities of an online rationality class. The goal the project is to have an official proposal as well as a beta version ready in a few months. Besides this hopefully helping spread friendly memes and giving publicity, we aim to figure out if this can be used as a tool to make progress on the difficult problems of teaching and measuring rationality. Best way to figure it out is to try and use it that way as we iterate.

I name dropped the proposal in the OP but since we started so recently it felt odd writing an article about that first.

Third, why try to train citizen scientists when we could make better use of specialist scientists? Gary Drescher posted here, but hasn't in over a year. What would make LW valuable enough to him for him to post here? XiXiDu managed to attract the attention of some experts in AI. What would make LW valuable enough to them for them to post here?

I kind of meant this under "attracting the right crowd" but I should have made it explicit.

But I don't see you addressing the engineering problems with moving from one culture to the ot

... (read more)

8Vaniver12y

I was rereading my comments (because of this post) and noticed this: Apparently, links to drafts of novel, interesting math papers.

7TheOtherDave12y

It would very much surprise me if the goal of creating a space where experts in AI consider it valuable to post were not in tension with the goal of doing large-scale outreach (a la HPMOR). This suggests that if LW's sponsors want to attract participation from experts in AI who sufficiently embrace the community's norms (e.g., that the rational position for anyone researching AI is to concentrate first on reliable Friendliness, lest they inadvertently destroy the world) to be valuable, it might be practical to do it in a space separate from where it attracts participation from the community at large (e.g., folks like me).

3Vaniver12y

Agreed. My expectation of a first step would be something like inviting relevant experts to give a talk at CFAR. Possibly tape it / have someone at CFAR turn it into a post and put it on LW, but the immediate goal would be the relationship with the expert, which is probably much more easily done by having one visible and friendly person at CFAR they know.

4fubarobfusco12y

Money? If a guy with a webcomic can raise a million dollars to make a video game, maybe we can fund research by people who know how to do research well.

[-]Bugmaster12y100

Huh, that's not a bad idea, actually. Had there ever been an attempt by the SIAI or CFAR (I'm not sure which is responsible for what, to be honest) to Kickstarter some project related to rationality or AI ? Something with a clear, measurable goal, like, "give us a million dollars and we'll give you a Friendly Oracle-grade AI" -- though less ambitious than that, obviously.

2Vaniver12y

If you can convince 12,000 people to donate an average of $100 to an efficient research organization, great! I am not optimistic about your prospects, though.

[-]duwease12y140

I'm curious as to hear an example or two of what sort of experiments you had in mind (and the models they'd be testing) when writing this article. A brief, none-too-thorough attempt on my own part kept hitting a couple of walls. I agree with your sentiment that simple surveys may fall prey to sampling biases, and I wonder how we would acquire the resources to conduct experiments methodologically sound enough that their results would be significant, the way CFAR is doing them, with randomized controlled trials and the like.

5[anonymous]12y

Luminosity is an example of a sequence that I liked but thought of as something that has many predictions that need testing . Methods of teaching rationality also seem like an obvious area for gain anyone who is interesting in that may be interested in a project I started to tackle this. I implicitly argue that CFAR would be better of doing this kind of thing too. We are at the dawn of an era of computational social science, transitioning to a more data driven model seems promising. If that is the course they pick I see no reason to make a strong distinction between CFAR and the wider LessWrong community. We can raise funds via kickstarter or just regular donations to cover expenses, as was done in the example of Citizen Genetics. Competitions like the quantified health prize could be used by CFAR to pick which of the community proposed projects it would like to happen and perhaps fund it. For things like processing time and bandwith you already have LWers willing to give it away for free. We wouldn't need that many resources. A website of the yourmorals.org kind (that is used for academic research) is something well within our reach (YourBrain, YourReasoning, RURational?). As to the cognitive resources, don't we have enough Bayesians and people with basic programming knowledge that given resources like common crawl we could come up with interesting research? That we have many AI experts on the site only means we also have the know how to tackle big data.

[-]Eliezer Yudkowsky12y130

It seems to me that most of the trepidation I associate with this problem comes from the difficulty of designing good experiments that are actually worth performing and would actually discriminate between two plausible hypotheses, not so much not knowing how to get enough research subjects.

7IlyaShpitser12y

I am not an experimental design specialist, but I am happy to offer whatever relevant advice I can. Incidentally, I am not far from the Bay Area in the next few months.

3[anonymous]12y

This is indeed a hard problem to tackle. I do outright say that we may need to attract a different crowd to get all of the skill set needed and suggest data mining as a form of empiricism we are already intellectually equipped to handle because of our demographics. It was my hope that we would start comping a list of things that would be neat to test and then hopefully identify some low hanging fruit. I think wecan with not too many resources reach the stage where we are doing work as good as the kind we wouldn't have second thoughts about citing as relatively strong evidence in articles.

[-]VincentYu12y110

What clearly can be useful is to create a list of models and ideas we've already assimilated that haven't been really tested or are based on research that still awaits replication.

I like the idea and agree this could be useful, but our first focus should be on reproducing well-established results to make sure that Less Wrong has the ability to so. Afterward, if we succeed, then it makes sense to proceed cautiously to replicating weak results or testing new hypotheses. Otherwise, if we don't succeed in reproducing well-established results, then we lack the ability to investigate more speculative ideas.

We face numerous additional burdens over experimentalists in academia, two of which stand out to me: (1) a currently unknown sampling bias and response rate, and (2) a general inability to blind our Less Wrong subjects to the hypotheses and predictions of our experiments.

It may be instructive to take a look at some of the recent papers on online psychometric assessment in personality psychology. They have worked on establishing the effects of online assessment (as opposed to traditional paper-and-pencil or face-to-face assessments) and proposed guidelines for doing experiments onlin... (read more)

[-]GLaDOS12y60

LessWrong Science: We do what we must because we can.

For the good of all of us, except the ones who are dead and haven't opted for cryonics.

4duwease12y

Also excepting the ones whose consciousness wasn't uploaded into homicidal testing machines, of course.

[-]Epiphany12y50

Incredibly good idea! Why:

I was told by a professional developmental psychologist that it's often the psychology assistants who have the creative ideas - the psychology training can get in the way. This may be due to problems like anchoring bias (where they're anchoring to all that existing information from their education), bias blind spots (I've seen these caused by the ego that one can get due to having a high-level degree), and confirmation bias (which can only happen if you have preconceptions to confirm) or others.

Specifically because many LessWron... (read more)

[-]Shmi12y50

Lets see how the predictions made by your model hold up!

Name three examples. Or even one to start. Using a template like:

Short and long description of the phenomenon being analyzed.
A review of the existing literature on the topic (including academic, of course).
A list of models which endeavor to explain it.
For each model, how it explains the phenomenon, what other known phenomena are covered, which of them are in agreement with the model, and which are in contradiction, and why.
A list of predictions each model makes, including the analysis

... (read more)

[-]V_V12y30

It seems that you are suggesting turning LW into some sort of alternative circuit for scientific pubblication.

That seems an inefficient thing to attempt. If you do publishable research, publish it to a peer-reviewed academic journal or conference.

You might use LW as a place to exchange preliminary ideas and drafts. Personally I'm a bit skeptical that it would yield a significant increase of productivity: scientific research typically requires an high level of domain specialization, it's unlikely that you would find many people with the relevant expertise on LW, and the discussion will be uninteresting for anybody else, but you might want to give it a try.

[-][anonymous]12y210

It seems that you are suggesting turning LW into some sort of alternative circuit for scientific publication.

You can call it that. I call it refining the art of human rationality. I don't think building new knowledge is something that magically only happens in a box designated Academia. Remember SI did years of research basically outside it, they only started publishing so they could attract more talent and as a general PR move, not because it was the most efficient way to do it. We are already an alternative circuit for scientific publication. This is exactly what we do every time we publish an article carrying some novel take on human rationality or some instrumentally useful advice. We are just bad at it.

That seems an inefficient thing to attempt. If you do publishable research, publish it to a peer-reviewed academic journal or conference.

You don't seem to have read the related articles I cited. I strongly suggest you do.

I would also recommend you read Why Academic Papers Are A Terrible Discussion Forum. As to your invoaction of the somewhat broken formal peer review process that came into existence in the 20th century and is sadly still with us (I recommend you search Vl... (read more)

-1V_V12y

No, but Academia is optimized for that and has hundreds of years of demonstrated effectiveness and accumulated experience. Is it perfect? No. Can you build something better from ground up? I don't think so, at least not at a cost smaller than the cost needed to improve it. Certainly the present LW doesn't look remotely like a superior alternative. And what did they accomplish? Pretty much nothing, AFAIK. If that's their main reason to publish then it seems that they are doing the right thing for the wrong reason. Are they so deluded to seriously believe they will be the first to build an AGI? I hope not. Therefore, if they want to have a chance to influence the development of AI projects, the best they can do is to disseminate the results of their research in a way that will maximize the number of AI researchers who will notice it and take it seriously. And this way is not Less Wrong, or an Harry Potter fanfiction or meetups or minicamps or all the other stuff they do. It's academic publishing. Academic publishing should be SI's raison d'etre, not a PR move. So far, AFAIK (I'm not a SI historian, so I might be mistaken) they published a few papers on philosophy, on the same sort of topics the FHI people do (some of these papers are co-authored with them, IIUC). I didn't read all of them, but my impression is that they didn't contribute particularly novel insights. We have yet to see what results their current research on program equilibrium will yield. I've skimmed them. The "Science: Do It Yourself" uses an examples that is flawed by a glaring methodological error from the start. That speaks lots of why scientific research, like many other activities, is something best left to professionals.

[-][anonymous]12y140

No, but Academia is optimized for that and has hundreds of years of demonstrated effectiveness and accumulated experience.

Is it perfect? No.

Actually the features of academia I'm criticizing are much younger than that. The modern peer review system is something Einstein didn't have to deal with for example. If you think hundreds of years of scientific progress are a good track record for a system I have some news for you...

"Citizen science" is a fairly new term but an old practice. Prior to the 20th Century, science was often the pursuit of amateur or self-funded researchers such as Sir Isaac Newton, Benjamin Franklin, and Charles Darwin. By the mid-20th Century, however, science was dominated by researchers employed by universities and government research laboratories. By the 1970's, this transformation was being called into question. Philosopher Paul Feyerabend called for a "democratization of science."[33] Biochemist Erwin Chargaff advocated a return to science by nature-loving amateurs in the tradition of Descartes, Newton, Leibniz, Buffon, and Darwin—science dominated by "amateurship instead of money-biased technical bureaucrats."[34]

3sam034512y

Obviously none of his great papers could have survived peer review. Some people argue that this was merely because of trivial stylistic issues, and could have been fixed by giving citations in correct format, and so on and so forth, so that they read like modern peer reviewed papers. Perhaps But the fact that he got his degree with a boring trivial paper, when he had several of his greatest papers in hand, suggests that there was no fixing them. If they could not be submitted for a degree, probably could never pass peer review. Peer review is in a sense a form of committee, and committees tend to be dumber than their dumbest member. The primary job of a committee, whatever its ostensible job, is to make sure that no one rocks the boat, just as the primary job of a bureaucracy, whatever its ostensible job, is to breed red tape with red tape to generate more red tape. Getting a group of people to function together so that their output is smarter than any one of them is hard, a deep and unsolved problem. The normal outcome is that their output is dumber than any one of them. The scientific community solved this problem from the late seventeenth century to late nineteenth or early twentieth century. Although engineering continues to advance, and more powerful tools such as DNA readers continue to advance science, science itself seemed to run out of puff after Einstein.

[-][anonymous]12y110

I'm guessing this post was down voted because of author not content because I can't find anything wrong with the latter.

But the fact that he got his degree with a boring trivial paper, when he had several of his greatest papers in hand, suggests that there was no fixing them.

Yes this is evidence towards him not being sure those papers could be fixed.

Getting a group of people to function together so that their output is smarter than any one of them is hard, a deep and unsolved problem.

Exactly, coordination is hard. Perverse incentives, Goodhart's law, agency dilemma, etc.

The normal outcome is that their output is dumber than any one of them.

See most non-profit organizations ever.

The scientific community solved this problem from the late seventeenth century to late nineteenth or early twentieth century. Although engineering continues to advance, and more powerful tools such as DNA readers continue to advance science, science itself seemed to run out of puff after Einstein.

Stagnation in our time.

While I think you are right for most fields, I would argue we see a relatively healthy culture and even functional institutions when it comes mathematics since they have be... (read more)

5wedrifid12y

No. I don't know the author but downvoted the naive understanding of progress implied with "ran out of puff after Einstein". It was cheap cynicism signalling that seemed misleading to me (especially since the earlier parts of the comment came across as authorative.)

1[anonymous]12y

Thank you for explaining this.

5Desrtopa12y

I also downvoted, and I actually considered not doing so because it was so far above the usual standards of what I've come to expect from sam0345, but I decided that if it were written by somebody else, I would have downvoted, for pretty much the same reasons wedrifid gave.

2Alicorn12y

That doesn't really follow. You could be missing a problem with the content, or someone else could mistakenly observe same. (Or both!) (Or there's some third reason to downvote comments that is neither author nor content!)

-3[anonymous]12y

I will edit that sentence since I see your point about it. This is not something I've overlooked. I did say I didn't find anything wrong with it, I expected to be corrected if wrong or change some minds if the down voters where wrong. You have to admit that considering author in question my hypothesis is very likely. Edit: Can down voters please explain why you've down voted this comment?

-7sam034512y

4V_V12y

You seem to be under the impression that Einstein's papers were not reviewed by professional physicists. That's incorrect: They were reviewed by journal editors who were professional physicists. The modern peer review system was invented because during the 20th century the submissions to journals greatly increased both in number and in sub-field specialization. While journals also increased in number and specialization, they couldn't keep up with that and had to "outsource" the review process. This is quite wrong. Even Einstein's field, theoretical physics, had significant progress until at least the mid-70s, when the Standard Model was completed. Subsequent stagnation was probably largely due to the difficulty of obtaining experimental data: testing all the features of the Standard Model required an enormous effort culminating in the LHC, and presently we can't do experiments on Plank scale phenomena. Other areas of science greatly progressed. Biology, for instance, is still far from stagnation.

3sam034512y

But Einstein only needed one journal editor to decide that his paper was good stuff that would rock the boat, whereas under peer review, he would in practice need every peer reviewer to agree that his papers did not rock the boat. Under the old system, he needed one of n to get published. Under the new system, it tends to be closer to n of n. Consensus, as Galileo argued, produces bad science. And, pretty obviously, we are getting bad science. Recall the recent study reported in nature that only three of fifty results in cancer research were replicable. The background to this replication study is that biomedical companies pick academic research to try to develop new medications - and they decided they needed to do some quality assurance.

5V_V12y

The exact rules of peer review vary between different journals and conferences, but in general no single referee has veto power. If there is major disagreement between referees, they will discuss, and if they fail to form a consensus the journal editors / conference chairmen will step in and make the final decision, after possibly recruiting additional referees. This seems to be a more accurate process than having a single editor making a decision based on only their own expertise. That's a false positive problem, while you seemed to be arguing that peer review generated too many false negatives. Anyway, neither referees nor editors try to replicate experimental results while reviewing a paper. That's not the goal of the review process. The review process is not intended to be a scientific "truth" certification. It is intended to ensure that a paper is innovative, clearly written, easy to place in the context of the research in its field, doesn't contain glaring methodological errors and is described in sufficient detail to allow experimental replication. Replication is something that is done by independent researchers after the paper is published.

2A1987dM12y

The stagnation of theoretical physics in the last few decades is also due to information cascades and other sociological/political effects; see The Trouble with Physics by Lee Smolin.

6Mitchell_Porter12y

There was no such stagnation. This is the period which saw M-theory, the holographic principle, and the twistor revival, three of the great theoretical advances of all time, and in general there was an enormous elevation in the level of technical knowledge. Smolin is just peeved that string theory is where all the action was.

1Eugine_Nier12y

The question is whether these theories correspond to reality.

0Shmi12y

Holography likely does, in some form, given that it pops up from every direction of research. The rest -- who knows.

0A1987dM12y

Are those on par with Einstein's work? (Maybe they're close enough -- though I'd wait a bit longer before saying that; but if you count how many physicists are working today and how many were working in the early 20th century...)

7Shmi12y

They are not. Einstein took some existing math and the hints of new physics and built a beautiful model on top of it, with marvelous new predictions, all of which pan out. Then he did it again 10 years later. Nothing like it has been repeated since. Creation of QM was close, but it was a collective effort over decades, still quite unfinished.

-4sam034512y

I understand M theory sufficiently well to be seriously underwhelmed. M theory and the holographic principle suspiciously resemble postmodernism: insiders talking to each other in ways that supposedly demonstrate their erudition, without any external check to verify that they are actually erudite, or even understand each other, or even understand what they themselves are saying. Twistors are valid and erudite mathematics, but don't seem to get us any closer to anything interesting. M theory is just string theory only more so. The trouble with string theory as a theory of spacetime is that it takes place in a fixed space time background, thus inherently makes no sense whatever. If you start with a contradiction, you can deduce anything you please. The central problem in any quantum theory of spacetime is that you have no fixed spacetime to stand upon, and string theory just blithely ignores the problem. That is not an advance in theoretical physics, that is finding weak excuses to publish meaningless papers.

2Mitchell_Porter12y

That's just an approximation. Those situations (flat space, hyperbolic space) are really just asymptotically fixed - the form of the space-time in the infinite past or the infinite future is fixed. But in between, you can have topology change. String theory in positively curved space may even allow for topologically distinct asymptotic outcomes, but that is still a topic of great confusion. There is a standard paradigm for applying string theory to the real world - grand unification, broken supersymmetry, compactification. I'd give that about a 50% chance of being correct. Then there are increasingly unfamiliar scenarios, the extreme of which would be a theory in which you don't even have strings or branes, but in which some of the abstract properties of string theory (e.g. the algebraic structure of the amplitudes) still hold. The twistors could swing either way here: twistorial variables may exist for an orthodox string scenario, but there may also be twistorial theories way outside the usual M-theoretic synthesis.

2sam034512y

I don't think string theory as it exists is capable of of describing a space time that undergoes topological change as a result of the dynamics of the strings. They talk about branes undergoing topological change, but they undergo topological change within a given background spacetime that acts without being acted upon. And if it is capable of describing such an event, string theorists don't really have any idea of how to make it do it. People go into Quantum Gravity because it is the big unsolved problem, find they cannot solve it, but they have to publish papers anyway. And so they do, resulting in postmodern physics.

0Mitchell_Porter12y

There is a string counterpart to the old idea of "spacetime foam", it's called a "Calabi-Yau crystal". The crystal fluctuates and branes are defects in the crystal. There are more things in string theory, sam0345, than are dreamt of in your philosophy.

-6sam034512y

2NancyLebovitz12y

Is there any way to tell whether a lack of major new science is the result of institutional problems or if it's caused by an absence of major discoveries which could be made with current tools?

0benelliott12y

Never mind DNA itself, which was discovered post Einstein

5TheOtherDave12y

What I've inferred from statements of key SI folk (most especially Luke) is that they don't think this likely, but they think the possible futures in which it happens are vastly superior to the ones in which it doesn't, so they're working towards it anyway. Yeah, this seems pretty plausible to me as well. (Though also pretty unlikely.) FWIW, my understanding of SI's original chosen strategy for making AI researchers take LW's ideas about Friendliness seriously was to publicize the Sequences, which would improve the general rationality of people everywhere (aka "raise the sanity waterline"), which would improve the rationality of AI researchers (and those who fund them, etc), which would increase the chances of AI researchers embracing the importance of Friendliness, which would increase the chances of FAI being developed before UFAI, which would save the world. From what I can tell, SI has since them moved on to other strategies for saving the world, like publishing the Sequences in book form, publishing popular fiction, holding minicamps, etc., but all built on the premise that "raising the sanity waterline" among the most easily reached people is a more viable approach than attempting to reach specific audiences like professional researchers.

2V_V12y

That's seems to be an inefficient approach. Even if you accept the premise that you can "teach" rationality to AI researchers capable of building an AGI (who probably would not be idiots, but they might be indeed affected by biases), doing so it's still an extremenly unfocused way to accomplish the task of advancing the state of the art on machine ethics. If you want to advance the state of the art on machine ethics, then the most efficient way of doing it is to do actual research on machine ethics. If AI researchers don't take machine ethics as seriously as you think they should, then the most efficient way to convince them is to put forward your arguments in forms and media accessible and salient to them. Once you go for peer review, you may receive negative feedback, of course. That might mean two things: That your core claims are wrong, in which case you should recognize that, stop wasting your efforts and move to something else, or that your arguments are uncompelling or unclear, in which case you should improve them, since it is your responsibility to make yourself understood.

0bogus12y

My admittedly incomplete understanding is that "raising the sanity waterline" activities have now been spun off to the Center for Applied Rationality, which is either planning to incorporate as a non-profit or already incorporated. This would then leave SIAI as focusing on the strictly AGI- and Friendliness-related stuff.

0TheOtherDave12y

Ah. I'm aware of the SI/CFAR split, but haven't paid much attention to what activities are owned by which entity, or how separate their staffs and resources actually are. E.g., I haven't a clue which entity sponsors LW, if either, or even whether it's possible to distinguish one condition from the other.

0V_V12y

From the information available on their websites, it seems that LW is still operated by SI. I suggest splitting it off an operating it as a charity separate from both SI and CFAR.

[-]CharlieSheen12y30

How about testing our ideas?

Actually judging clever articles by the rent they demonstrably pay in anticipated experience? This idea is too radical Konkvistador. Don't you know that hand waving or reading papers is fun and testing is like ... work?

[-]Multiheaded12y180

A young and learning member calling reading papers "fun" without a second thought is already impressive progress when compared to the epistemic attitude of most people around us, I'd say.

LW posters have noticed many times that the most instrumentally rational people, hailed for making the world better or at any rate leaving a mark on it (Page & Brin, Warren Buffett, Linus Torvalds, maybe Thiel; among politicians either Gandhi, Churchill or Lee Kuan Yew - they wouldn't have got along! - and maybe some older ones like Alexander II of Russia or the people behind the Meiji Restoration...), rarely behave like Eliezer or Traditional Rationality would want them to. They exploited some peculiar factors, innate or unintentionally acquired advantages (genes, lucky upbringing, broad life experience) that LW attempts to emulate through some written advice and group meetings. Most haven't even heard of Bayes or can't name a couple of fallacies! :)

At this stage, if an LW user actually uses the letter and spirit of LW materials to gain rent in some complicated, important area (like education, career, interpersonal relations, "Luminosity", fighting akrasia) - well, that's ... (read more)

8DaFranker12y

+1 to that. I also strongly agree with the fact that even the current small observable gains are impressive. I have observed an approximately 50% reduction in the average time it takes for me to explain foreign concepts for respective inferential distances, thanks to the reductionism and words sequences. While very anecdotal, if a 100% increase of efficiency in informal pedagogy is not impressive, please tell me where I can find all the other goldmines that so dramatically increase the ability to teach, and I'd also like an explanation why the hell it seems like no teachers anywhere are studying any of it.

7lloyd12y

Schools do not teach any critical thinking and for good reason. Ivan Illich wrote "Deschooling Society" in the 70s and John Taylor Gatto started writing the "The Underground History of American Education" in the 90s. Either should give you insight into why teachers do what they do, but Gattos's "Weapons of Mass Instruction" is probably the best place to start. The short answer is that schools are designed from the top down to stunt the intellectual growth of children regardless of the intentions of teachers.

6[anonymous]12y

Maybe the result is that they stunt growth, but to infer intention from that is just an agency-fantasy. I would guess that the bereaucrats that actually think about the result have good intentions, even.

[-]Vaniver12y120

Maybe the result is that they stunt growth, but to infer intention from that is just an agency-fantasy. I would guess that the bereaucrats that actually think about the result have good intentions, even.

Eh, the Prussian school system was explicitly designed to create soldiers, and stunting intellectual growth is a part of that. It's not much of a stretch to call it intentional.

[-]Desrtopa12y130

I doubt that many school officials or politicians today know about the influences of the Prussian school system on e.g. the United States school system, or would guess that their present systems bear features deliberately designed to stunt intellectual growth.

I suspect that they mostly see the system that they were themselves educated in as normal by default, and only think to question the appropriateness of features that are specifically brought to their attention, and then only contemplate changing them in ways that are politically practical and advantageous from their positions. Expecting them to try and design and implement a school system that best meets their stated goals is like expecting a person to specify to a genie exactly how they want their mother removed from a burning building so as to save her life. The problem and its solution space simply doesn't fall within the realms that they're inclined to actually think about.

2Vaniver12y

Indeed. (I should clarify that I was interpreting the original inventors of the Volksschule were the 'top', not, say, Arne Duncan.)

1lloyd12y

I do not know if you have read Gatto or not based on this. He points out that the system has no memory of its origin and that changes occur just like you describe with the result of deepening the problem. The last major school reform was GW Bush's No Child Left Behind....if that tells you anything about who "fixes" the system.

3Desrtopa12y

No Child Left Behind was a stupid fix, but that doesn't mean it was an ill intentioned fix. I have actually just found the online text of "The Underground History of Education," and started reading it, but so far am unimpressed and unlikely to finish it. I'm noticing a lot of cherrypicking to support his position, and he doesn't give sources for his assertions at all (I went to the table of contents to look for a bibliography, and couldn't find one, so I did a further search to see if this is the case in the print version and confirmed that the book contains no citations.) I share his opinion that our current educational system is not well designed to get the best out of its students, but if I wanted to introduce someone to a writer who could effectively explain that point, I don't think I'd recommend him. I'd probably recommend some of Eliezer's essays, or maybe Paul Graham's.

0[anonymous]12y

Oops, I accidentally only considered US and Canada. (tho I know little of what goes on in the American system, now that I think about it)

4lloyd12y

The US system took heavily from the Prussian school. The history is fascinating to say the least.

2lloyd12y

The statements of intent where made in writing and in speeches. I would do it for you but linking on the droid is not fun. Google "Rockefeller mencken quotes education" and the first link should lend some insight into the intent of the designers of the compulsory public school system. Gatto did a lot of research to support the thesis that schools are designed to dumb down the populace.

3Desrtopa12y

This may simply be jumping on an issue of semantics, but I'm concerned that this is really what he did, rather than doing a lot of research to find out whether the thesis was correct, or, more ideally, doing a lot of research before promoting the hypothesis to attention at all.

3lloyd12y

Well you can make wild speculations based off of my semantics or you can read for yourself. You seem to have chosen the former. Please return and clarify if you find his research faulty after you have read his work.

3Desrtopa12y

Well, the first link I get when I do the google search you suggested is this, and I've read that, but I'm not clear on what research of his you're expecting me to read. The only work of his that appears to be available online is this, which contains assertions, but does not appear to support them with research as such. It's true that I haven't read John Taylor Gatto's work beyond that essay and the page you suggested reading yourself, but I was not wildly speculating based on semantics, I was making an educated guess considering that this is how people ordinarily behave. I assign a much higher prior to someone, say, hearing that the American school system is inspired by the Prussian system, which was largely concerned with producing good citizen-soldiers, and concluding that the American system must be deliberately designed to stymy creative thought, and looking for more evidence to back up that assertion, than I do to someone deciding to find out what the intentions behind the American school system are, doing extensive research, and concluding that its programs are actually purpose-designed to dumb down the populace. What little work of his that I've found accessible online certainly doesn't shift me away from that assessment.

0[anonymous]12y

Sounds scary. I'll look into it and update as appropriate. You are postulating quite the conspiracy tho. Much more likely it seems that a few b'crats went bonkers, the way you sometimes get UFO nuts out of the military.

[-]thomblake12y130

You are postulating quite the conspiracy tho.

Not really. To militaristic Prussia of the time, creating good soldiers was simply the same as creating good citizens, and was considered a worthy goal. No conspiracy required, just doing what seemed obviously correct at the time. And then the Prussian system was so 'advanced' and 'modern' and 'successful' that others copied it.

American experts did not all agree with the 'military' goal, but it was believed by the relevant experts that the same sorts of virtues applied to factory workers.

Now people try to actually educate children via this system. It's like making minor tweaks to a torture device and wondering why it is ineffective at relieving headaches. You put some ibuprofen on the screws, tighten them some more, and subjects report slightly less intense headaches than last time.

4Epiphany12y

Not creating effective soldiers puts you at a military disadvantage. If Prussia was a major power at the time, surely other countries feared them. If other countries felt it was necessary to stifle their populace in order to ensure that they were capable of defending themselves against Prussia (or to defend themselves against the countries that took after Prussia), perhaps stifling the populace was thought to be a "lesser evil", a sacrifice they justified as part of an arms race. Maybe this wasn't an evil conspiracy, but a terrible consequence of the prospect of war. What's the bias for: "Ahh! We're in mortal danger! Quick, everybody, become stupid!"

4khafra12y

Bayesians vs. Barbarians

0Epiphany12y

I wrote a quick introduction to Gatto's claims of detrimental schooling practices that I think will give you a quick idea of whether it's worth continuing to look into. Let me know what your reaction is? I'm curious.

0Epiphany12y

What is the "good" reason? Or did you not mean to agree with this practice?

0lloyd12y

By 'good' reason I meant one consistent with the purpose or function of schooling. It is to be taken as having a touch of humor based on people's misunderstanding of the function of school believing it to be synonymous with education.

0Epiphany12y

Oh, okay. I guess I didn't know your personality well enough yet to assume the correct things. Thanks. (:

-1Shmi12y

I suspect that free public education is on average about the same everywhere. Guessing teacher's password and rote memorization are the easiest ways to teach, and an average teacher is not very good at what she does, so this method shows up by default. The idea that the US education is built on the "Prussian school system [which] was explicitly designed to create soldiers" and that's why it is so bad seems like a conspiracy theory. I would like to know if there are examples to the contrary (i.e. countries where an average high-school graduate is adept at independent learning and critical thinking).

[-]Epiphany12y130

John Taylor Gatto won the New York State teacher of the year award in 1991 (New York state's education website). His ambition to be a great teacher led him to the realization that the system itself is broken and he was so disgusted with it that he resigned. The claims that John Taylor Gatto makes are much worse than that they're defaulting to the teacher's password. You have no idea. Consider this: You obviously value rational thought. Learning about things like logical fallacies and biases is a no-brainer to you, right? Why are so many people learning them here, at LessWrong, for the first time? From what I know of American public schools, most of them don't teach these. What could cause our school systems to teach us square dancing and rote memorization of thousands of spellings of words for the sake of polish, but leave out basic pieces required for rational thought? Ask yourself this:

If you were making the curriculum, and you knew the kids would be turned lose into the world complete with the right to vote at 18 would you find any excuse good enough to let them out with no familiarity of logical fallacies, biases, etc.?

If your answer to this is "no" you alr... (read more)

9wedrifid12y

This was the only one of the education theorists that I studied while getting my teachning qualification that was remotely inspiring.

[-]Epiphany12y150

Man, Gatto spurred off so much thought for me. That was in my early 20's so it's not all readily coming to mind right now, but wow. I feel like... he explained so much. I'm not sure why you say he's inspiring. So much of life that didn't make any sense began to make sense after that. But that was one of the worst existential crises I've ever experienced. To realize that your whole life you had been stifled by the thing you thought was teaching you: abominable. There are horrors worse than death. That is one of them.

When I was 17, I decided to tear my whole reality apart because I noticed that it contained too many flaws. This was excruciating and terrifying. When I was 18, I had the undignified experience of realizing I could not allow myself to vote because I wasn't taught to think critically and was still learning to. When I was in my early 20's, I discovered logical fallacies and went "SOMEBODY WROTE THIS ALL DOWN!!?!!?? Why didn't I know about this!?" I was a mess of a young woman - it took years of effort to put together a decently competent mind after all that.

Failing to teach reasoning skills in school is a crime against humanity.

7endoself12y

This pattern matches to the standard failure mode where exceptional individuals assume that others are more like them and therefore more competent than they actually are. This causes them to conclude that institutions are more flawed than they actually are.

5Nornagest12y

It's at least partly right, but strikes me as weak evidence. Horace Mann and several of his contemporaries admired the Prussian system and introduced reforms to American schooling based on it -- but he probably wasn't trying to foster exactly the same features that later commentators have objected to. There have also been several major changes to American public schooling since then, many of them divergent with the development of the Prussian (and, later, German) school system. It's plausible to me that the American school system functionally prioritizes inculcating obedience to authority, whether it was designed that way or fell into that arrangement through a process of evolution. But that's a claim that's got to stand or fall on an analysis of the system as it currently exists, not of its remote origins.

3Morendil12y

My thesis is that to a good first approximation the purpose of schools, like that of the TSA, is to instill and ensure compliance in the populace, not to educate. I should probably go and read this Gatto, he's come up before in discussions like this one.

6Epiphany12y

I really question whether a "world full of leaders" would necessarily fail. These discussions about the original purpose of schooling seem to come down to that question. Is it that the leaders at the time didn't want to give up leadership to have a strong populace, or is it that they had no clue how to organize a strong populace in a way that makes them as effective as stifling them does? I mean, it's pretty counter-intuitive that stifling a populace will make it more effective, even if it gives you the ability to organize them better. And it's really questionable that a populace full of leaders wouldn't figure out how to organize itself. On the one hand, we could argue that the information overload of a species that is rapidly gaining knowledge will worsen if their minds aren't standardized somehow. We could also ask "Are there different ways to standardize, some stifling, some not?" and "Would standardizing them by stifling them cause more or less information overload compared with standardizing them around a theme of rational thought?" The standardizing by stifling option reduces the number of new ideas being created, but prevents bad ideas from being culled, which allows them to build up. The standardizing by rationality option doesn't necessarily mean you need to stifle creativity (and I think creativity is necessary for rational thought - lest every decision you make be subject to the flaws inherent in an option set too small, like with false dichotomy) but it would cause people to cull a large number of ideas that they waste time on right now AND it would give them a way to agree on things. Right now, we have a bunch of people who believe in opinion. They say "Everyone has their opinion. Let's respect each other's opinion." as if they cannot be proven true or false, more or less effective. I think the problem is they don't know enough about how to test ideas. Assuming that a populace made of soldiers makes a country safer may be incorrect, too. Why did

7NancyLebovitz12y

Another big thing that's missing from school is the idea of applying one's thinking to a significant decision, and then acting on it. I call this problem "spending crucial developmental years in simulation land".

3Epiphany12y

You seem to be reading these comments out of context. Cliff notes: That is John Taylor Gatto's claim - that the American school system is based on the Prussian school system which was designed to create soldiers and the Prussian belief was that this required them to do things that stunted intellectual growth. Shminux said JTG seemed like a conspiracy theorist and my comment was in response to that. You should definitely read JTG. At least read "The Seven Lesson School Teacher" (linked in my previous comment) if nothing else.

2Morendil12y

True: I had only read a small part of the thread at that time. My intent was to point out that there are plenty of others here on LW who have considered questions like the one above, and who have (in my case independently) come to conclusions that align with those you attribute to JTG. (That's not hugely surprising, given Eliezer's educational background.) I wasn't aware of the Prussian connection specifically, Wikipedia seems to confirm that the Prussian system has inspired other countries (including mine). (Not that I trust WP overmuch, but other sources concur.)

0[anonymous]12y

The thing that's wrong is that nobody knows about rationality, and of those who do, most don't care. Not everyone knows about heuristics and biases. A b'crat who did wouldn't care enough to put his career on the line.

-2MugaSofer12y

Would have upvoted just for this.

2TimS12y

My sense is that the education system struggles with the transition between learning-to-read and reading-to-learn. Those of us who make that transition easily often struggle with teacher's password issues, but teacher's password is probably only the second most frequent failure mode in public education.

[-]Shmi12y100

My sense is that the education system struggles with the transition between learning-to-read and reading-to-learn.

Somewhat off-topic: high schools anywhere don't seem to explicitly teach the only essential skill a college student must need: learning to learn:

How do I figure out what I need to know for a given class, how do I figure out what I do not know, and how do I go about learning it efficiently?

is not a question students learn to ask or answer. Everyone who completes a post-secondary education tends to come up with some sort of implicit heuristics that get them through, few do it consciously.

5DaFranker12y

The International Baccalaureate explicitly acknowledges this as a key issue, and claims to teach these skills specifically. I can speak from personal experience in saying that they utterly fail at this - the coursework alone might be slightly better, but the implementation is still much too reliant on teachers, and their behaviors has an effect several orders of magnitude greater than the standard material in how much the learning to learn is actually made part of the students versus just giving us a couple more passwords that seem to be about other passwords. They also completely ignore the recursivity issue (learning to learn to learn to learn ... ), which means students are left to deal with this problem on their own, lest they become stuck wasting enormous amounts of time attempting to perfectly optimize their learning methods and never accomplish any actual non-meta learning. Mind, this is meant to be taken as a rant/anecdote. I might (though I wouldn't say it's highly likely) have just gotten the worst of it and the programme generally fares better, for all I know.

4[anonymous]12y

Thanks for actually being a little specific on that. I've heard it a lot, but my thot is always "but what does that even mean".

2TimS12y

It's even worse than that. I don't think anyone knows how to teach people to learn. Obviously, individuals have done it, but human knowledge does not appear to contain a process that basically any ordinary teacher can follow to cause any ordinary student to learn how to learn. That's one of the top ten things I think we'd have after a good Singularity that we don't have now. Maybe even top five.

4Shados12y

Perspective shift: Frequenting LW is a "peculiar factor"/acquired advantage.

4faul_sname12y

I can give you plenty of rationalizations, if you need them. My mind is very good at coming up with rationalizations.

2DSimon12y

Ack, noticed some tribe blindness in myself here. Out of the examples you list in your last paragraph: I can immediately think of negative effects each of these ideas have on their audiences, except for the first one, New Atheism. Of course (remarkable coincidence!) that happens to be the one that I have personal association with. Can you elaborate on the negative effects you were thinking of when you mentioned New Atheism?

[-]Athrelon12y190

The bad of New Atheism: Children playing with memetic weapons, with the safety off.

It's good at diagnosing problems with existing institutions, bad at understanding people's non-rational but deeply-rooted needs that are currently satisfied by religion.
It has failed to create even remotely plausible replacement institutions, due to this lack of understanding.
It is fundamentally parochial: it originated from, and continues to narrowmindedly focus on, the internal life of the intellectual 1%, with little understanding or interest in the other 99% of the bell curve.

[-]chaosmosis12y100

Lack of patience, overconfidence, more about signalling intelligence than about persuading religious people, lack of empathy. Those are the problems that came immediately to mind when I thought about it. That's not to criticize all of New Atheism, though. I think I like the basic idea of it.

2DSimon12y

Yes, those all make sense, thank you. And I am also a fan of at least a good subset of each of the other three examples. It's just good, as you say, to remember how fraught with nasty side effects this whole self-improvement thing can often be.

0endoself12y

Trying to rationalize something like this is much worse in the long run. Intentionally acting irrationally is much better than acting the same way, but believing it to be rational. This advice is more useful on the meta-level of having a community norm of accepting people who do this. (Do we already have such a norm? I know that I act as if such a norm exists, but I am unsure if others do.) The good thing about Less Wrongers is that you can get them to adopt such a norm just by explaining why it would be useful.

[-]mstevens12y10

Possible use for this new thing: Seeing as we have had much recent discussion about what behaviour is, and is not, creepy, we could create a long list of potential behaviour, and study which people think it's creepy or not. And if it makes a difference if you ask first. (Question 492: So is it okay if I ask you out on a Wednesday... while wearing a tutu, and there are exactly 3.9 people in the room?)

Once we had this useful dataset, we could evaluate potential rules for social interaction ("okay, under your plan, 42% of women under 25 say they'll hate you, but with my scheme, 50% say they'll like you!") etc.

1[anonymous]12y

Yes meta data and analysis on how to improve the site and community is low hanging fruit when it comes to research projects.

[-][anonymous]11y00

A recent example of competent small scale self-experimentation.

[-]ksvanhorn12y00

There are many publicly available data sets and plenty of opportunities to mine data online, yet we see little if any original analysis based on them here. We either don't have norms encouraging this or we don't have enough people comfortable with statistics doing so.

In my case, I'm comfortable with statistics but don't know where to find the data for questions that interest me. The fact that much research is nearly inaccessible if you're not affiliated with a university or other large institution is also a problem.

[-][anonymous]12y-10

You think you have a good map, what you really have is a working hypothesis

I most certainly don't think that. I'm not so sure many people on LW think that. The part of your map that talks about others' maps looks suspect.

Intellectual speculation isn't bad in itself. Actually, it's fun! It's only bad when people don't know that they're in fact merely speculating. I have nothing against people of LessWrong banding together to do experimental science but that wouldn't mean LessWrong as a whole was progressing on the path towards greater virtue. It would on... (read more)

4[anonymous]12y

Perhaps, but in practice I see appeals to the sequences or a well liked article taken as very strong evidence for a position and almost never is the central point challenged. I certainly wasn't expecting everyone on LW to head in a more empirical direction in practice, that would be silly, since even filling out a form takes some time. I was hoping LW could become a place where projects like this do happen and are encouraged. Much like currently careful high quality scholarship does happen and is encouraged. That's the reason I cited the positive effect of lukeprog's article as an example of the kind of a sufficient magnitude of change.

0[anonymous]12y

Do you count cases where someone puts a link to an article in their argument as instances of that phenomenon? In such cases it might be hard to tell whether someone is being dogmatic or just providing a link to an already written elaboration of the ideas involved.

[-]Epiphany12y-40

Getting it done:

First we could pre-test them in an inexpensive way for the purpose of figuring out which ones are worth the money for independent research. Then, because LessWrong gets several million hits a year, an appeal to donate could be placed on LessWrong pages asking for donations to pay for high-quality research from an organization with credibility.

[+]Will_Newsome12y-170

Moderation Log

113Comments