All of peco's Comments + Replies
10
Does a universe consisting of a single particle change?
-20
If you call a utilitarian's utility function T, then you can pick the dust specks over torture if your utility function is -T.
10
Since people only make a finite number of decisions in their lifetime, couldn't their utility function specify every decision independently? (You could have a utility function that is normal except that it says that everything you hear being called 1A is preferable to 1B, and anything you hear being called 2B is preferable to 2A. If this contradicts your normal utility function, this rule is always more important. Even if 2B leads to death, you still choose 2B.)
The utility function would be impossible to come up with in advance, but it exists.
Why can't the AI just be exactly the same as Hibbard? If Hibbard is flawed in a major way, you could make an AI for every person on Earth (this obviously wouldn't be practical, but if a few million AI's are bad the other few billion can deal with them).
We already have an entity exactly the same as Hibbard. Namely: Hibbard. Why do we need another one?
What we want is an AI that's far more intelligent than a human, yet shares their values. Increasing intelligence while preserving values is nontrivial. You could try giving Hibbard the ability to self-modify, but then he'd most likely just go insane in some way or another.