I've been thinking about what seems to be the standard LW pitch on AI risk. It goes like this: "Consider an AI that is given a goal by humans. Since 'convert the planet into computronium' is a subgoal of most goals, it does this and kills humanity."

The problem, which various people have pointed out, is that this implies an intelligence capable of taking over the world, but not capable of working out that when a human says pursue a certain goal, they would not want this goal to be pursued in a way that leads to the destruction of the world.

Worse, the argument can then be made that this idea that an AI will interpret goals so literally without modelling a human mind constitutes an "autistic AI" and that only autistic people would assume that AI would be similarly autistic. I do not endorse this argument in any way, but I guess its still better to avoid arguments that signal low social skills, all other things being equal.

Is there any consensus on what the best 'elevator pitch' argument for AI risk is? Instead of focusing on any one failure mode, I would go with something like this:

"Most philosophers agree that there is no reason why superintelligence is not possible. Anything which is possible will eventually be achieved, and so will superintelligence, perhaps in the far future, perhaps in the next few decades. At some point, superintelligences will be as far above humans as we are above ants. I do not know what will happen at this point, but the only reference case we have is humans and ants, and if superintelligences decide that humans are an infestation, we will be exterminated."

Incidentally, this is the sort of thing I mean by painting LW style ideas as autistic (via David Pierce)

As far as we can tell, digital computers are still zombies. Our machines are becoming autistically intelligent, but not supersentient - nor even conscious. [...] Full-Spectrum Superintelligence entails: [...] social intelligence [...] a metric to distinguish the important from the trivial [...] a capacity to navigate, reason logically about, and solve problems in multiple state-spaces of consciousness [e.g. dreaming states (cf. lucid dreaming), waking consciousness, echolocatory competence, visual discrimination, synaesthesia in all its existing and potential guises, humour, introspection, the different realms of psychedelia [...] and finally "Autistic", pattern-matching, rule-following, mathematico-linguistic intelligence, i.e. the standard, mind-blind cognitive tool-kit scored by existing IQ tests. High-functioning "autistic" intelligence is indispensable to higher mathematics, computer science and the natural sciences. High-functioning autistic intelligence is necessary - but not sufficient - for a civilisation capable of advanced technology that can cure ageing and disease, systematically phase out the biology of suffering, and take us to the stars. And for programming artificial intelligence.

Sometimes David Pierce seems very smart. And sometimes he seems to imply that the ability to think logically while on psychedelic drugs is as important as 'autistic intelligence'. I don't think he thinks that autistic people are zombies that do not experience subjective experience, but that also does seem implied.

Looking for advice with something it seems LW can help with.

I'm currently part of a program the trains highly intelligent people to be more effective, particularly with regards to scientific research and effecting change within large systems of people. I'm sorry to be vague, but I can't actually say more than that.

As part of our program, we organize seminars for ourselves on various interesting topics. The upcoming one is on self-improvement, and aims to explore the following questions: Who am I? What are my goals? How do I get there?

Naturally, I'm of the opinion that rationalist thought has a lot to offer on all of those questions. (I also have ulterior motives here, because I think it would be really cool to get some of these people on board with rationalism in general.) I'm having a hard time narrowing down this idea to a lesson plan I can submit to the organizers, so I thought I'd ask for suggestions.

The possible formats I have open for an activity are a lecture, a workshop/discussion in small groups, and some sort of guided introspection/reading activity (for example just giving people a sheet with questions to ponder on it, or a text to reflect on).

I've also come up with several possible topics: How to Actually Change Your Mind (ideas on how to go about condensing it are welcome), practical mind-hacking techniques and/or techniques for self-transparency, or just information on heuristics and biases because I think that's useful in general.

You can also assume the intended audience already know each other pretty well, and are capable of rather more analysis and actual math than is average.

Ideas for topics or activities, particularly ones that include a strong affective experience because those are generally better at getting poeple to think about this sort of thing for the first time, are welcome.

I've read that the CEO of Levi's recommends washing jeans very infrequently.

Won't they smell? I have a pretty clean white-collar lifestyle, but I'm concerned about wearing mine even once or twice between machine washing. Is it considered socially acceptable to re-wear jeans?

I've recently started a tumblr dedicated to teaching people what amounts to Rationality 101. This post isn't about advertising that blog, since the sort of people that actually read Less Wrong are unlikely to be the target audience. Rather, I'd like to ask the community for input on what are the most important concepts I could put on that blog.

(For those that would like to follow this endeavor, but don't like tumblr, I've got a parallel blog on wordpress)

It makes sense but it doesn't match my subjective experience.

Doing everything both ways, nonverbal and verbal, has lent itself to better understanding of the reasoning. Which touches on the anecdote problem, if you test every nonverbal result; you get something statistically relevant. If your odds are more often than not with nonverbal, testing every result and digging for the mistakes will increase your understanding (disclaimer: this is hard work).

I don't tend to do a lot of proofs anymore. When I think of math, I find it most important to be able to flip back and forth between symbol and referent freely - look at an equation and visualize the solutions, or (to take one example of the reverse) see a curve and think of ways of representing it as an equation. Since when visualizing numbers will often not be available, I tend to think of properties of a Taylor or Fourier series for that graph. I do a visual derivative and integral.

That way, the visual part tells me where to go with the symbolic part. Things grind to a halt when I have trouble piecing that visualization together.

Which of those, if any, sounds closer to the way you think about math?

Each serves its own purpose. It is like the technical and artistic sides of musical performance: the technique serves the artistry. In a sense the former is subordinate to the latter, but only in the sense that the foundation of a building is subordinate to its superstructure. To perform well enough that someone else would want to listen, you need both.

This may be useful reading, and the essays here (from which the former is linked).

I'm not really conscious of the distinction, unless you're talking about outright auditory things like rehearsing a speech in my head. The overwhelming majority of my thinking is in a format where I'm thinking in terms of concepts that I have a word for, but probably not consciously using the word until I start thinking about what I'm thinking about. Do you have a precise definition of "verbal"? But whether you call it verbal or not, it feels like it's all the same thing.

Correct and Precise may have been better terms. By correct I mean a result that I have very high confidence in, but that is not precise enough to be useable. By accurate I mean a result that is very precise but with far less confidence that it is correct.

As an example, consider a damped oscillation word problem from first year. You are very confident that as time approaches infinity that the displacement will approach a value just by looking at it, but you don't know that value. Now when you crunch the numbers (the verbal process in the extreme) you get a very specific value that the function approaches, but have less confidence that that value is correct, you could have made any of a number of mistakes. In this example the classic wrong result is the displacement is in the opposite direction as the applied force.

This is a very simple example so it may be hard to separate the non-verbal process from the verbal, but there are many cases where you know what the result should look like but deriving the equations and relations can turn into a black box.