= 27d567bc2d48d9f40e9ccc20ea370b15)
JGWeissman comments on Off Topic Thread: May 2009 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (71)
And if we are not sensitive enough, our molecules get reused for paperclips. You are talking about matching wits with something that is orders of magnitude smarter than us, thinks orders of magnitude faster than us, has a detailed model of our minds, and doesn't think we are worth the utilons it can build out of our quarks. Yes, we would think we recognize the right solution, that it is obvious now that it's been pointed out, and we would be wrong. An argument than a UFAI figured out would be persuasive to us is nowhere near as trustworthy as an argument we figure out ourselves. While we can be wrong about our own arguments, the UFAI will present arguments that we will be systematically wrong about in very dangerous ways.
And for our next trick, we will just ask Omega to build an FAI for us.
No AI, friendly or unfriendly, will ever have a model of our minds as detailed as the model we have of its mind, because we can pause it and inspect its source code while it can't do anything analogous to us.
I have written plenty of mere desktop applications that are a major pain for a human mind to understand in a debugger. And have you ever written programs that generate machine code for another program? And then tried to inspect that machine code when something went wrong to figure out why?
Well, that stuff is nothing compared to the difficulty of debugging or otherwise understanding an AI, even given full access to inspect its implementation. An unfriendly AI is likely to be built by throwing lots of parallel hardware together until something sticks. If the designers actually knew what they were doing, they would figure out they need to make it friendly. So, yes, you could pause the whole thing, and look at how the billions of CPU's are interconnected, and the state of the terabytes of memory, but you are not logically omniscient, and you will not make sense of it.
It's not like you could just inspect the evil bit.