On the subject of creating a function/predicate able to identify a person. It seems that it is another non-localiseable function. My reasoning goes something like this.
1) We want the predicate to be able to identify paused humans (cryostasis), so that the FAI doesn't destroy them accidentally.
2) With sufficient scanning technology we could make a digital scan of a human that has the same value as a frozen head, and encrypt with a one time pad, making it indistinguishable from the output of /dev/rng.
From 1 and 2 it follows that the AI will have to look at the environment (to see if people are encrypting people with one-time pads), before making a decision on what is a human or not. How much of the AI needs to encompass before making that decision seems a non-trivial question to answer.
Poorly labeled encrypted persons may well be destroyed. I'm not sure this matters too much.
This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. Feel free to rid yourself of cached thoughts by doing so in Old Church Slavonic. If a discussion gets unwieldy, celebrate by turning it into a top-level post.
If you're new to Less Wrong, check out this welcome post.