= 27d567bc2d48d9f40e9ccc20ea370b15)
NancyLebovitz comments on Thoughts on the Singularity Institute (SI) - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (1270)
I'd brought up a version of the tool/agent distinction, and was told firmly that people aren't smart or fast enough to direct an AI. (Sorry, this is from memory-- I don't have the foggiest how to do an efficient search to find that exchange.)
I'm not sure that's a complete answer-- how possible is it to augment a human towards being able to manage an AI? On the other hand, a human like that isn't going to be much like humans 1.0, so problems of Friendliness are still in play.
Perhaps what's needed is building akrasia into the world-- a resistance to sudden change. This has its own risks, but sudden existential threats are rare. [1]
At this point, I think the work on teaching rationality is more reliably important than the work on FAI. FAI involves some long inferential chains. The idea that people could improve their lives a lot by thinking more carefully about what they're doing and acting on those thoughts (with willingness to take feedback) is a much more plausible idea, even if you factor in the idea that rationality can be taught.
[1] Good enough for fiction-- we're already living in a world like that. We call the built-in akrasia Murphy.
You may be thinking of this exchange, which I found only because I remembered having been involved in it.
I continue to think that "tool" is a bad term to use here, because people's understanding of what it refers to vary so relevantly.
As for what is valuable work... hm.
I think teaching people to reason in truth-preserving and value-preserving ways is worth doing.
I think formalizing a decision theory that captures universal human intuitions about what the right thing to do in various situations is worth doing.
I think formalizing a decision theory that captures non-universal but extant "right thing" intuitions is potentially worth doing, but requires a lot of auxiliary work to actually be worth doing.
I think formalizing a decision theory that arrives at judgments about the right thing to do in various situations where those judgments are counterintuitive for most/all humans but reliably lead, if implemented, to results that those same humans reliably endorse more the results of their intuitive judgments is worth doing.
I think building systems that can solve real-world problems efficiently is worth doing, all else being equal, though I agree that powerful tools frequently have unexpected consequences that create worse problems than they solve, in which case it's not worth doing.
I think designing frameworks within which problem-solving systems can be built, such that the chances of unexpected negative consequences are lower inside that framework than outside of it, is worth doing.
I don't find it likely that SI is actually doing any of those things particularly more effectively than other organizations.
Thanks for the link-- that was what I was thinking of.
Do you have other organizations which teach rationality in mind? Offhand, the only thing I can think of is cognitive behavioral therapy, and it's not exactly an organization.
No, I don't have anything specific in mind.