= 27d567bc2d48d9f40e9ccc20ea370b15)
asr comments on Risks from AI and Charitable Giving - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (126)
You are arguing past each-other. XiXiDu is saying that a programmer can create software that can be inspected reliably. We are very close to having provably-correct kernels and compilers, which would make it practical to build reliably sandboxed software, such that we can look inside the sandbox and see that the software data structures are what they ought to be.
It is separately true that not all software can be reliably understood by static inspection, which is all that the underhanded C contest is demonstrating. I would stipulate that the same is true at run-time. But that's not the case here. Presumably developers of a large complicated AI will design it to be easy to debug -- I don't think they have much chance of a working program otherwise.
No, you are ignoring Xi's context. The claim is not about what a programmer on the team might do, it is about what the AI might write. Notice that the section starts 'The goals of an AI will be under scrutiny at any time...'
Yes. I thought Xi's claim was that if you have an AI and put it to work writing software, the programmers supervising the AI can look at the internal "motivations", "goals", and "planning" data structures and see what the AI is really doing. Obfuscation is beside the point.
I agree with you and XiXiDu that such observation should be possible in principle, but I also sort of agree with the detractors. You say,
Oh, I'm sure they'd try. But have you ever seen a large software project ? There's usually mountains and mountains of code that runs in parallel on multiple nodes all over the place. Pieces of it are usually written with good intentions in mind; other pieces are written in a caffeine-fueled fog two days before the deadline, and peppered with years-old comments to the extent of, "TODO: fix this when I have more time". When the code breaks in some significant way, it's usually easier to write it from scratch than to debug the fault.
And that's just enterprise software, which is orders of magnitude less complex than an AGI would be. So yes, it should be possible to write transparent and easily debuggable code in theory, but in practice, I predict that people would write code the usual way, instead.