= 27d567bc2d48d9f40e9ccc20ea370b15)
Nick_Tarleton comments on Sarah Connor and Existential Risk - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (77)
"Bad argument gets counterargument. Does not get bullet. Never. Never ever never for ever."
But I'll propose a possibly even more scarily cultish idea:
Why attempt to perfect human rationality? Because someone's going to invent uploading sometime. And if the first uploaded person is not sufficiently rational, they will rapidly become Unfriendly AI; but if they are sufficiently rational, then there's a chance they will become Friendly AI.
(The same argument can be used for increasing human compassion, of course. Sufficiently advanced compassion requires rationality, though.)
(Tangentially:)
"Will" is far too strong. Becoming UFAI at least requires that an upload be given sufficient ability to self-modify (or sufficiently modified from outside), and that IA up to superintelligence on uploads be not only tractable (likely but not guaranteed) but, if it's going to be the first upload, easy enough that lots more uploads don't get made first. Digital intelligences are not intrinsically, automatically hard takeoff risks, which it sounds like you're modeling them as. (Not to mention, up to a point insufficient rationality would make an upload less likely to ever successfully increase its intelligence.)
(That said, there are lots of risks and horrible scenarios involving uploads that don't require strong superintelligence, just subjective speedup or copiability.)