Squark comments on Open Thread February 25 - March 3 - Less Wrong

8 Post author: Coscott 25 February 2014 04:57AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (354)

You are viewing a single comment's thread. Show more comments above.

Comment author: Kawoomba 26 February 2014 11:39:18AM *  3 points [-]

Richard Loosemore (score one for nominative determinism) has a new, well, let's say "paper" which he has, well, let's say "published" here.

His refutation of the usual uFAI scenarios relies solely/mostly on a supposed logical contradiction, namely (to save you a few precious minutes) that a 'CLAI' (a Canonical Logical AI) wouldn't be able to both know about its own fallability/limitations (inevitable in a resource-constrained environment such as reality), and accept the discrepancy between its specified goal system and the creators' actual design intentions. Being superpowerful, the uFAI would notice that it is not following its creator-intended goals but "only" its actually-programmed-in goals*, which, um, wouldn't allow it to continue acting against its creator-intended goals.

So if you were to design a plain ol' garden-variety nuclear weapon intended for gardening purposes ("destroy the weed"), it would go off even if that's not what you actually wanted. However, if you made that weapon super-smart, it would be smart enough to abandon its given goal ("What am I doing with my life?"), consult its creators, and after some deliberation deactivate itself. As such, a sufficiently smart agent would apparently have a "DWIM" (do what the creator means) imperative built-in, which would even supersede its actually given goals -- being sufficiently smart, it would understand that its goals are "wrong" (from some other agent's point of view), and self-modify, or it would not be superintelligent. Like a bizarre version of the argument from evil.

There is no such logical contradiction. Tautologically, an agent is beholden to its own goals, and no other goals. There is no level of capability which magically leads to allowing for fundamental changes to its own goals, on the contrary, the more capable an agent, the more it can take precautions for its goals not to be altered.

If "the goals the superintelligent agent pursues" and "the goals which the creators want the superintelligent agent to pursue, but which are not in fact part of the superintelligent agent's goals" clash, what possible reason would there be for the superintelligent agent to care, or to change itself, changing itself for reasons that squarely come from a category of "goals of other agents (squirrels, programmers, creators, Martians) which are not my goals"? Why, how good of you to ask. There's no such reason for an agent to change, and thus no contradiction.

If someone designed a super-capable killer robot, but by flipping a sign, it came out as a super-capable Gandhi-bot (the horror!), no amount of "but hey look, you're supposed to kill that village" would cause Gandhi-bot to self-modify into a T-800. The bot isn't gonna short-circuit just because someone has goals which aren't its own goals. In particular, there is no capability-level threshold from which on the Gandhi-bot would become a T-800. Instead, at all power levels, it is "content" following its own goals, again, tautologically so.

* In common parlance just called "its goals".

Comment author: Squark 26 February 2014 07:09:46PM 3 points [-]

The condescending tone with which he presents his arguments (which are, paraphrasing him, "slightly odd, to say the least") is amazing. Who is this guy and where did he come from? Does anyone care about what he has to say?

Comment author: gwern 26 February 2014 08:45:28PM 4 points [-]

Loosemore has been an occasional commenter since the SL4 days; his arguments have heavily criticized pretty much anytime he pops his head up. As far as I know, XiXiDu is the only one who agrees with him or takes him seriously.

Comment author: XiXiDu 27 February 2014 09:53:45AM *  1 point [-]

As far as I know, XiXiDu is the only one who agrees with him or takes him seriously.

He actually cites someone else who agrees with him in his paper, so this can't be true. And from the positive feedback he gets on Facebook there seem to be more. I personally chatted with people much smarter than me (experts who can show off widely recognized real-world achievements) who basically agree with him.

his arguments have heavily criticized pretty much anytime he pops his head up.

What people criticize here is a distortion of small parts of his arguments. RobBB managed to write a whole post expounding his ignorance of what Loosemore is arguing.

Comment author: gwern 27 February 2014 06:41:15PM 8 points [-]

He actually cites someone else who agrees with him in his paper, so this can't be true.

I said as far as I know. I had not read the paper because I don't have a very high opinion of Loosemore's ideas in the first place, and nothing you've said in your G+ post has made me more inclined to read the paper, if all it's doing is expounding the old fallacious argument 'it'll be smart enough to rewrite itself as we'd like it to'.

I personally chatted with people much smarter than me (experts who can show off widely recognized real-world achievements) who basically agree with him.

Name three.

Comment author: Kawoomba 27 February 2014 09:15:37AM *  0 points [-]

Does anyone care about what he has to say?

Apparently (?) the AAAI 2014 Spring Symposium in Stanford does (???).