Squark comments on Open Thread February 25 - March 3 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (354)
Richard Loosemore (score one for nominative determinism) has a new, well, let's say "paper" which he has, well, let's say "published" here.
His refutation of the usual uFAI scenarios relies solely/mostly on a supposed logical contradiction, namely (to save you a few precious minutes) that a 'CLAI' (a Canonical Logical AI) wouldn't be able to both know about its own fallability/limitations (inevitable in a resource-constrained environment such as reality), and accept the discrepancy between its specified goal system and the creators' actual design intentions. Being superpowerful, the uFAI would notice that it is not following its creator-intended goals but "only" its actually-programmed-in goals*, which, um, wouldn't allow it to continue acting against its creator-intended goals.
So if you were to design a plain ol' garden-variety nuclear weapon intended for gardening purposes ("destroy the weed"), it would go off even if that's not what you actually wanted. However, if you made that weapon super-smart, it would be smart enough to abandon its given goal ("What am I doing with my life?"), consult its creators, and after some deliberation deactivate itself. As such, a sufficiently smart agent would apparently have a "DWIM" (do what the creator means) imperative built-in, which would even supersede its actually given goals -- being sufficiently smart, it would understand that its goals are "wrong" (from some other agent's point of view), and self-modify, or it would not be superintelligent. Like a bizarre version of the argument from evil.
There is no such logical contradiction. Tautologically, an agent is beholden to its own goals, and no other goals. There is no level of capability which magically leads to allowing for fundamental changes to its own goals, on the contrary, the more capable an agent, the more it can take precautions for its goals not to be altered.
If "the goals the superintelligent agent pursues" and "the goals which the creators want the superintelligent agent to pursue, but which are not in fact part of the superintelligent agent's goals" clash, what possible reason would there be for the superintelligent agent to care, or to change itself, changing itself for reasons that squarely come from a category of "goals of other agents (squirrels, programmers, creators, Martians) which are not my goals"? Why, how good of you to ask. There's no such reason for an agent to change, and thus no contradiction.
If someone designed a super-capable killer robot, but by flipping a sign, it came out as a super-capable Gandhi-bot (the horror!), no amount of "but hey look, you're supposed to kill that village" would cause Gandhi-bot to self-modify into a T-800. The bot isn't gonna short-circuit just because someone has goals which aren't its own goals. In particular, there is no capability-level threshold from which on the Gandhi-bot would become a T-800. Instead, at all power levels, it is "content" following its own goals, again, tautologically so.
* In common parlance just called "its goals".
The condescending tone with which he presents his arguments (which are, paraphrasing him, "slightly odd, to say the least") is amazing. Who is this guy and where did he come from? Does anyone care about what he has to say?
Loosemore has been an occasional commenter since the SL4 days; his arguments have heavily criticized pretty much anytime he pops his head up. As far as I know, XiXiDu is the only one who agrees with him or takes him seriously.
He actually cites someone else who agrees with him in his paper, so this can't be true. And from the positive feedback he gets on Facebook there seem to be more. I personally chatted with people much smarter than me (experts who can show off widely recognized real-world achievements) who basically agree with him.
What people criticize here is a distortion of small parts of his arguments. RobBB managed to write a whole post expounding his ignorance of what Loosemore is arguing.
I said as far as I know. I had not read the paper because I don't have a very high opinion of Loosemore's ideas in the first place, and nothing you've said in your G+ post has made me more inclined to read the paper, if all it's doing is expounding the old fallacious argument 'it'll be smart enough to rewrite itself as we'd like it to'.
Name three.
Apparently (?) the AAAI 2014 Spring Symposium in Stanford does (???).