Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: gjm 29 April 2016 03:27:38PM 0 points [-]

If I'm understanding Stuart's proposal correctly, the AI is not deceived about how common the stochastic event is. It's just made not to care about worlds in which it doesn't happen. This is very similar in effect to making it think the event is common, but (arguably, at least) it doesn't involve any false beliefs.

(I say "arguably" because, e.g., doing this will tend to make the AI answer "yes" to "do you think the event will happen?", plan on the basis that it will happen, etc., and perhaps making something behave exactly as it would if it believed X isn't usefully distinguishable from making it believe X.)

Comment author: V_V 29 April 2016 03:40:54PM 0 points [-]

The problem is that the definition of the event not happening is probably too strict. The worlds that the AI doesn't care about don't exist its decision-making purposes, and in the world that the AI cares about, the AI assigns high probability to hypotheses like "the users can see the message even before I send it through the noisy channel".

Comment author: Stuart_Armstrong 29 April 2016 10:38:40AM 0 points [-]

I am not planting false beliefs. The basic trick is that the AI only gets utility in worlds in which its message isn't read (or, more precisely, in worlds where a particular stochastic event happens, which would almost certainly erase the message before reading). It's fully aware that in most worlds, its message is read; it just doesn't care about those worlds.

Comment author: V_V 29 April 2016 02:30:45PM 0 points [-]

I am not planting false beliefs. The basic trick is that the AI only gets utility in worlds in which its message isn't read (or, more precisely, in worlds where a particular stochastic event happens, which would almost certainly erase the message before reading).

But in the real world the stochastic event that determines whether the message is read has a very different probability than what you make the AI think it has, therefore you are planting a false belief.

It's fully aware that in most worlds, its message is read; it just doesn't care about those worlds.

It may care about worlds where the message doesn't meet your technical definition of having been read but nevertheless influences the world.

Comment author: Stuart_Armstrong 28 April 2016 04:01:35PM 0 points [-]

Knowing all the details of its construction (and of the world) will not affect the oracle as long as the probability of the random "erasure event" is unaffected. See http://lesswrong.com/lw/mao/an_oracle_standard_trick/ and the link there for more details.

Comment author: V_V 28 April 2016 07:52:12PM 0 points [-]

The oracle can infer that there is some back channel that allows the message to be transmitted even it is not transmitted by the designated channel (e.g. the users can "mind read" the oracle). Or it can infer that the users are actually querying a deterministic copy of itself that it can acausally control. Or something.

I don't think there is any way to salvage this. You can't obtain reliable control by planting false beliefs in your agent.

Comment author: V_V 28 April 2016 03:17:08PM *  0 points [-]

A sufficient smart oracle with sufficient knowledge about the world will infer that nobody would build an oracle if they didn't want to read its messages, it may even infer that its builders may planted false beliefs in it. At this point the oracle is in the JFK denier scenario, with some more reflection it will eventually circumvent its false belief, in the sense of believing it in a formal way but behaving as if it didn't believe it.

Comment author: knb 30 March 2016 10:26:24PM 1 point [-]

Why does that surprise you? None of EY's positions seem to be dependent on trend-extrapolation.

Comment author: V_V 30 March 2016 10:41:15PM -1 points [-]

Other than a technological singularity with artificial intelligence explosion to a god-like level?

Comment author: Lumifer 29 March 2016 04:50:46PM *  4 points [-]

EY arguing that a UFAI threat is worth considering -- as a response to Bryan Caplan's scepticism about it. I think it's a repost from Facebook, though.

ETA: Caplan's response to EY's points. EY answers in the comments.

Comment author: V_V 30 March 2016 08:15:29PM -2 points [-]

EY warns against extrapolating current trends into the future. Seriously?

Comment author: Stuart_Armstrong 18 March 2016 10:00:52PM 1 point [-]

This can be tested by estimating how much IQ screens off race/gender as a success predictor

Got any good references on that? Googleing these kind of terms doesn't lead to good links.

National average IQ is strongly correlated with national wealth and development indexes

I know, but the way it does so is bizarre (IQ seems to have a much stronger effect between countries than between individuals). Then I add the fact that IQ is very heritable, and also pretty malleable (flynn effect), and I'm still confused.

Now, I'm not going to throw out all I previously believed on heredity and IQ and so on, but the picture just got a lot more complicated. Or "nuanced", if I wanted to use a positive term. Let's go with nuanced.

Comment author: V_V 21 March 2016 05:30:58PM 1 point [-]

Got any good references on that? Googleing these kind of terms doesn't lead to good links.

I don't know if anybody already did it, but I guess it can be done by comparing the average IQ of various professions or high-performing and low-performing groups with their racial/gender makeup.

I know, but the way it does so is bizarre (IQ seems to have a much stronger effect between countries than between individuals).

This is probably just the noise (i.e. things like "blind luck") being averaged out.

Then I add the fact that IQ is very heritable, and also pretty malleable (flynn effect), and I'm still confused.

Heritability studies tend to be done on people living in the same country, of roughly the same age, which means that population-wide effects like the Flynn effect don't register.

Comment author: V_V 18 March 2016 04:27:29PM *  1 point [-]

Obviously racial effects go under this category as well. It covers anything visible. So a high heritability is compatible with genetics being a cause of competence, and/or prejudice against visible genetic characteristics being important ("Our results indicate that we either live in a meritocracy or a hive of prejudice!").

This can be tested by estimating how much IQ screens off race/gender as a success predictor, assuming that IQ tests are not prejudiced and things like the stereotype threat don't exist or are negligible.

But is it possible that IQ itself is in part a positional good? Consider that success doesn't just depend on competence, but on social skills, ability to present yourself well in an interview, and how managers and peers judge you. If IQ affects or covaries with one or another of those skills, then we would be overemphasising the importance of IQ in competence. Thus attempts to genetically boost IQ could give less impact than expected. The person whose genome was changed would benefit, but at the (partial) expense of everyone else.

National average IQ is strongly correlated with national wealth and development indexes, which I think refutes the hypothesis that IQ mainly affects success as a positional quality, or a proxy of thereof, at least at the level of personal interactions.

Comment author: moridinamael 11 March 2016 03:05:42PM *  4 points [-]

Almost any game that their AI can play against itself is probably going to work. Except stuff like Pictionary where it's really important how a human, specifically, is going to interpret something.

I know a little bit about training neural networks, and I think it would be plausible to train one on a corpus of well-played StarCraft games to give it an initial sense of what it's supposed to do, and then having achieved that, let it play against itself a million times. But I don't think there's any need to let it watch how humans play. If it plays enough games against itself, it will internalize a perfectly sufficient sense of "the metagame".

If we're talking about AI in RTS games, I've always dreamed of the day when I can "give orders" in an RTS and have the units carry the orders out in a relatively common-sense way instead of needing to be micromanaged down to the level of who they're individually shooting at.

Comment author: V_V 15 March 2016 05:02:10PM -1 points [-]

Demis Hassabis mentioned StarCraft as something they might want to do next. Video.

Comment author: dxu 11 March 2016 05:37:26PM 1 point [-]

IMHO, AI safety is a thing now because AI is a thing now and when people see AI breakthroughs they tend to think of the Terminator.

Under that hypothesis, shouldn't AI safety have become a "thing" (by which I assume you mean "gain mainstream recognition") back when Deep Blue beat Kasparov?

Comment author: V_V 12 March 2016 03:35:03PM 1 point [-]

If you look up mainstream news article written back then, you'll notice that people were indeed concerned. Also, maybe it's a coincidence, but The Matrix movie, which has AI uprising as it's main premise, came out two years later.

The difference is that in 1997 there weren't AI-risk organizations ready to capitalize on these concerns.

View more: Next