Sebastian_Hagen2 comments on The Moral Void - Less Wrong

31 Post author: Eliezer_Yudkowsky 30 June 2008 08:52AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (105)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Sebastian_Hagen2 01 July 2008 06:45:00PM 0 points [-]

Hal Finney:
Why doesn't the AI do it verself? Even if it's boxed (and why would it be, if I'm convinced it's an FAI?), at the intelligence it'd need to make the stated prediction with any degree of confidence, I'd expect it to be able to take over my mind quickly. If what it claims is correct, it shouldn't have any qualms about doing that (taking over one human's body for a few minutes is a small price to pay for the utility involved).
If this happened in practice I'd be confused as heck, and the alleged FAI being honest about its intentions would be pretty far down on my list of hypotheses about what's going on. I'd likely stare into space dumbfounded until I found some halfway-likely explanation, or the AI decided to take over my mind after all.