MugaSofer comments on Failed Utopia #4-2 - Less Wrong

52 Post author: Eliezer_Yudkowsky 21 January 2009 11:04AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (247)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 30 May 2013 08:42:21AM 4 points [-]

If a genie cares enough about your request to interpret and respond to its naive denotation, it also cares enough to interpret your request's obvious connotations.

That doesn't follow. There just isn't any reason that the former implies the latter. Either kind of caring is possible but they are not the same thing (and the second is likely more complex than the first).

Your proposed interpretation only makes sense if the genie is a rules-lawyer

This much is true. (Or at least it must be something that follows rules.)

with at-least-instrumentally-oppositional interests/incentives

This isn't required. It need no oppositional interests/incentives at all beyond, after they are given a request, the desire to honour it. This isn't a genie trying to thwart someone in order to achieve some other goal. It is just the genie trying to the intent in order to for some other purpose. It is a genie only caring about the request and some jackass asking for something they don't want. (Rather than 'oppositional' it could be called 'obedient', where it turns out that isn't what is desired.)

in which case one wonders where those oppositional interests/incentives came from.

Presumably it got it's wish granting motives from whoever created it or otherwise constructed the notion of the wish granter genie.

Comment author: MugaSofer 30 May 2013 10:29:13AM *  4 points [-]

Actually, I think Will has a point here.

"Wishes" are just collections of coded sounds intended to help people deduce our desires. Many people (not necessarily you, IDK) seem to model the genie as attempting to attack us while maintaining plausible deniability that it simply misinterpreted our instructions, which, naturally, does occasionally happen because there's only so much information in words and we're only so smart.

In other words, it isn't trying to understand what we mean; it's trying to hurt us without dropping the pretense of trying to understand what we mean. And that's pretty anthropomorphic, isn't it?

Comment author: private_messaging 30 May 2013 12:34:32PM *  4 points [-]

Yes, that's the essence of it. People do it all the time. Generally, all sorts of pseudoscientific scammers try to maintain image of honest self deception; in the medical scams in particular, the crime is just so heinous and utterly amoral (killing people for cash) that pretty much everyone goes well out of their way to be able to pretend at ignorance, self deception, misinterpretation, carelessness and enthusiasm. But why would some superhuman AI need plausible deniability?

Comment author: nshepperd 30 May 2013 02:08:31PM 3 points [-]

If your genie is using your vocal emissions as information toward the deduction of your extrapolated volition, then I'd say your situation is good.

Your problems start if it works more by attempting to extract a predicate from your sentence by matching vocal signals against known syntax and dictionaries, and output an action that maximises the probability of that predicate being true with respect to reality.

To put it simply, I think that "understanding what we mean" is really a complicated notion that involves knowing what constitutes true desires (as opposed to, say, akrasia), and of course having a goal system that actually attempts to realize those desires.