= 27d567bc2d48d9f40e9ccc20ea370b15)
jsalvatier comments on Holden Karnofsky's Singularity Institute Objection 2 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (41)
At least in the original post, I don't think Holden's point is that tool-AI is much easier than agent-AI (though he seems to have intuition that it is), but that it's potentially much safer (largely from increased feedback), and thus that it deserves more investigation (and that it's a bad sign of SIAI that it's neglected this approach).
Yes, good point. The objection is about SI not addressing tool-AI, much of their discussion is about addressing tool-AI, not the meta "why isn't this explicitly called out by SI?" In particular the intuitions Holden has as responses to those questions, that we may well be able to create extremely useful general AI without creating general AI that can improve itself, do seem like they have received too little in-depth discussion here. We've often mentioned the possibility and often decided to skip the question because it's very hard to think about but I don't recall many really lucid conversations trying to ferret out what it would look like if more-than-narrow, less-than-having-the-ability-to-self-improve AI were a large enough target to reliably hit.
(as an aside, I think as Holden thinks of it, tool-AI could self improve, but because it's tool like and not agent-like, it would not automatically self improve. Its outputs could be of the form "I would decide to rewrite my program with code X", but humans would need to actually implement these changes.)