You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

XiXiDu comments on Risks from AI and Charitable Giving - Less Wrong Discussion

2 Post author: XiXiDu 13 March 2012 01:54PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (126)

You are viewing a single comment's thread. Show more comments above.

Comment author: asr 13 March 2012 09:14:20PM 2 points [-]

A plan for world domination seems like something that can't be concealed from its creators. Lying is no option if your algorithms are open to inspection.

This is just naive. Source code can be available and either the maliciousness not obvious (see the Underhanded C Contest) or not prove what you think it proves (see Reflections on Trusting Trust, just for starters). Assuming you are even inspecting all the existing code rather than a stub left behind to look like an AI.

You are arguing past each-other. XiXiDu is saying that a programmer can create software that can be inspected reliably. We are very close to having provably-correct kernels and compilers, which would make it practical to build reliably sandboxed software, such that we can look inside the sandbox and see that the software data structures are what they ought to be.

It is separately true that not all software can be reliably understood by static inspection, which is all that the underhanded C contest is demonstrating. I would stipulate that the same is true at run-time. But that's not the case here. Presumably developers of a large complicated AI will design it to be easy to debug -- I don't think they have much chance of a working program otherwise.