Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Qiaochu_Yuan comments on Further discussion of CFAR’s focus on AI safety, and the good things folks wanted from “cause neutrality” - Less Wrong

35 Post author: AnnaSalamon 12 December 2016 07:39PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (40)

You are viewing a single comment's thread. Show more comments above.

Comment author: Qiaochu_Yuan 14 December 2016 07:32:36PM *  3 points [-]

There is a third component to actually knowing a lot about AI, which is having succeeded in having learnt about AI, which is to say, having "won" in a certain sense. If rationality is winning, or knowing how to use raw intelligence effectively, a baseline level of rationality is indicated.

To speak from my own personal experience, I know a lot of math, and mostly the reason I know a lot of math is a combination of raw intelligence and teachers pushing me hard in that direction (for which I'm very grateful). I used almost no metacognition that I can remember; people just shoved topics in my direction and I got curious about and thought about some of them a lot. (But I did not, for example, do any thinking about where my curiosity should be aimed and why, nor did I spend time explicitly brainstorming ways I could be learning math faster or anything like that.)

Why is metacognition needed for AI safety? I can see how an average person might need to understand that they are, for instance, making anthropomorphic assumptions, but someone with a good understanding of AI would not do that..in fact, someone with hands-on experience of AI would be less biased in their assumptions about AI than someone who merely theorises about AI safety.

This is not at all clear to me. I think you underestimate how compartmentalized the thinking of even very intelligent academics can be.

And how do you sell rationality training to them? Presumably not on the basis that they don't know how to win...

You can try convincing them that CFAR teaches skills that they don't have that would help them in some way. In any case some kind of pitch was good enough for Max Tegmark and Jaan Tallinn, both of whom attended workshops and then played a role in making the Puerto Rico conference happen and founding FLI, along with a few other CFAR alumni. My impression is that this event was more or less responsible for getting Elon Musk on board with AI safety as a cause, which in return did a lot to normalize AI safety as a topic people could talk about publicly.

Comment author: TheAncientGeek 14 December 2016 07:34:05PM 0 points [-]

Can you answer the question : why is metacognition needed for AI safety?

Comment author: Qiaochu_Yuan 14 December 2016 07:51:08PM *  3 points [-]

If there are patterns in your thinking that are consistently causing you to think things that are not true, metacognition is the general tool by which you can notice that and try to correct the situation.

To be more specific, I can very easily imagine AI researchers not believing that AI safety is an issue due to something like cognitive dissonance: if they admitted that AI safety was an issue, they'd be admitting that what they're working on is dangerous and maybe shouldn't be worked on, which contradicts their desire to work on it. The easiest way to resolve the cognitive dissonance, and the most socially acceptable way barring people like Stuart Russell publicly pumping in the other direction, is to dismiss the concern as Luddite fear-mongering. This is the sort of thing you can try to notice and correct about yourself with the right metacognitive tools.

To make another analogy with math, I have never once heard a mathematics graduate student or professor speculate, publicly or privately, about the extent to which pure mathematics is mostly useless and overfunded. This is unsayable among mathematicians, maybe even unthinkable.

Comment author: TheAncientGeek 14 December 2016 08:08:47PM *  1 point [-]

If there are patterns in your thinking that are consistently causing you to think things that are not true, metacognition is the general tool by which you can notice that and try to correct the situatio

And it there isnt that problem, there is no need for that solution. For your argument to go through, you need to show that people likely to be impactive on AI safety are likely to have cognitive problems that affect them when they are doing AI safety. (Saying something like "academics are irrational because some of them believe in God" isn't enough." Compartmentalised beliefs are unimpactive because compartmentalised. Instrumental rationality is not epistemic rationality ).

To be more specific, I can very easily imagine AI researchers not believing that AI safety is an issue due to something like cognitive dissonance:

I dare say

if they admitted that AI safety was an issue, they'd be admitting that what they're working on is dangerous and maybe shouldn't be worked on, which contradicts their desire to work on it.

I can easily imagine an AI safety researcher maintaining a false belief that AI safety is a huge deal, because if they didn't they would be a nobody working on a non-problem. Funny how you can make logic run in more than one direction.