Kaj_Sotala comments on Why I haven't signed up for cryonics - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (249)
Have you read Failed Utopia #4-2?
I have, but it's running with the dramatic-but-unrealistic "genie model" of AI, in which you could simply command the machine, "Be a Friendly AI!" or "Be the CEV of humanity!", and it would do it. In real life, verbal descriptions are mere shorthand for actual mental structures, and porting the necessary mental structures for even the slightest act of direct normativity over from one mind-architecture to another is (I believe) actually harder than just using some form of indirect normativity.
(That doesn't mean any form of indirect normativity will work rightly, but it does mean that Evil Genie AI is a generalization from fictional evidence.)
Hence my saying I have trouble coming up with a realistic model.