asr comments on Do people think Less Wrong rationality is parochial? - Less Wrong

27 Post author: lukeprog 28 April 2012 04:18AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (196)

You are viewing a single comment's thread. Show more comments above.

Comment author: asr 06 May 2012 07:44:28PM 3 points [-]

I think we are justified, as a society, in taking biological risks much more seriously than computational risks.

My sense is that in practice, programming is much simpler than biochemistry. With software, we typically are working within a completely designed environment, and one designed to be easy for humans to reason about. We can do correctness proofs for software, we can't do anything like it for biology.

Programs basically stay put the way they are created; organisms don't. For practical purposes, software never evolves; we don't have a measurable rate of bit-flip errors or the like resulting in working-but-strange programs. (And we have good theoretical reasons to believe this will remain true.)

If a virulent disease does break loose, we have a hard time countering it, because we can't re-engineer our bodies. But we routinely patch deployed computer systems to make them resistant to particular instances of malware. The cost of a piece of experimental malware getting loose is very much smaller than with a disease.

Comment author: drethelin 06 May 2012 08:21:30PM 0 points [-]

the entire point of researching self improving AI is to move programs from the world of software that stays put the way it's created, never evolving, into a world we don't directly control.

Comment author: asr 07 May 2012 12:25:25AM *  1 point [-]

Yes. I think the skeptics don't take self-improving AI very seriously. Self-modifying programs in general are too hard to engineer, except in very narrow specialized way. A self-modifying program that rapidly achieves across-the-board superhuman ability seems like a fairy tale, not a serious engineering concern.

If there were an example of a program that self-improves in any nontrivial way at all, people might take this concern more seriously.