= 27d567bc2d48d9f40e9ccc20ea370b15)
timtyler comments on Ben Goertzel: The Singularity Institute's Scary Idea (and Why I Don't Buy It) - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (432)
The idea that a self-improving AGI has to be bug-free from the moment it is first run seems like part of the "syndrome" to me. Can the machine fix its own bugs? What about a "controlled ascent"? etc.
If it has a bug in its utility function, it won't want to fix it.
If it has a bug in its bug-detection-and-fixing techniques, you can guess what happens.
So, no, you can't rely on the AGI to fix itself, unless you're certain that the bugs are localised in regions that will be fixed.
So: bug-free is not needed - and a controlled ascent is possible.
The unreferenced "hubris verging on sheer insanity" asumption seems like a straw man - nobody assumed that in the first place.
How do you plan to fix the bugs in its bug-fixing ability, before the bug-fixing ability is applied to fixing bugs in the "don't kill everyone" routine? ;-)
More to the point, how do you know that you and the machine have the same definition of "bug"? That seems to me like the fundamental danger of self-improving AGI: if you don't agree with it on what counts as a "bug", then you're screwed.
(Relevant SF example: a short story in which the AI ship -- also the story's narrator -- explains how she corrected her creator's all-too-human error: he said their goal was to reach the stars, and yet for some reason, he set their course to land on a planet. Silly human!)
How would that be the default case, if you're explicitly taking precautions?
It seems as though you don't have any references for the supposed "hubris verging on sheer insanity". Maybe people didn't think that in the first place.
Computers regularly detect and fix bugs today - e.g. check out Eclipse.
I never claimed "controlled ascent" as being "the default case". In fact I am here criticising "the default case" as weasel wording.
Controlled ascent isn't the default case, but it certainly should be what provably friendly AI is weighed against.