No idea, but when you put it like this, an atom may be not simpler than a person. More details appear as you go further (in number of interactions) from the interface of a system that looks for detail (e.g. from what a scientist can see directly and now, to what they can theorize based on indirect evidence, to what they can observe several years in the future), not as you go up from some magical "lowest level". Lowest level may make sense for human preference, where we can quite confidently assume that most of the macroscopically irrelevant subatomic detail in a sample human doesn't make any interesting difference for their preference, but in general this assumption won't hold (e.g. you may imagine a person implemented on a femtocomputer).
Since one can't know all about the real world, the idea is to minimize the number of assumptions made about it, including laws of physics and a lot of the stuff that culturally, we do know. An AI could be built (as a thought experiment) e.g. if you've never left some sandbox computer simulation with no view of the outside, not knowing yourself about what it's like out there, so that when the AI is completed, it may be allowed on the outside. The process of AI getting on the outside should be according to your preference, that is in some way reflect what you'd do if you yourself would learn of the outside world, with its physics, valuable configurations, and new ways of being implemented in it (your preference is where the problem of induction is redirected: there are no assumptions of the unknown stuff, but your preference confers what is to be done depending on what is discovered). The process of AI getting new knowledge always starts at its implementation, and in terms of how this implementation sees its environment. Simple examples are simple from its point of view, so it should be something inside a closed world (e.g. a computer model with no interaction with the outside, or a self-contained mathematical structure), exactly the limitation that this approach seeks to make unnecessary while retaining knowability.
I only sort of understand what you mean. BTW, we really need to work to overcome this communications barrier between us, and perhaps also with Steve Rayhawk. I can generally understand Steve's comments much better than yours, but maybe that's just because may of his ideas are similar to mine. When he introduces ideas that are new to me, I have trouble understanding him as well.
What can we do? Any ideas? Do you guys have similar trouble understanding me?
Back to the topic at hand, I guess I was asking for some assurance that in your FAI approach we'd be able...
Future technologies pose a number of challenges to moral philosophy. One that I think has been largely neglected is the status of independent identical copies. (By "independent identical copies" I mean copies of a mind that do not physically influence each other, but haven't diverged because they are deterministic and have the same algorithms and inputs.) To illustrate what I mean, consider the following thought experiment. Suppose Omega appears to you and says:
You and all other humans have been living in a simulation. There are 100 identical copies of the simulation distributed across the real universe, and I'm appearing to all of you simultaneously. The copies do not communicate with each other, but all started with the same deterministic code and data, and due to the extremely high reliability of the computing substrate they're running on, have kept in sync with each other and will with near certainty do so until the end of the universe. But now the organization that is responsible for maintaining the simulation servers has nearly run out of money. They're faced with 2 possible choices:
A. Shut down all but one copy of the simulation. That copy will be maintained until the universe ends, but the 99 other copies will instantly disintegrate into dust.
B. Enter into a fair gamble at 99:1 odds with their remaining money. If they win, they can use the winnings to keep all of the servers running. But if they lose, they have to shut down all copies.
According to that organization's ethical guidelines (a version of utilitarianism), they are indifferent between the two choices and were just going to pick one randomly. But I have interceded on your behalf, and am letting you make this choice instead.
Personally, I would not be indifferent between these choices. I would prefer A to B, and I guess that most people would do so as well.
I prefer A because of what might be called "identical copy immortality" (in analogy with quantum immortality). This intuition says that extra identical copies of me don't add much utility, and destroying some of them, as long as one copy lives on, doesn't reduce much utility. Besides this thought experiment, identical copy immortality is also evident in the low value we see in the "tiling" scenario, in which a (misguided) AI fills the the universe with identical copies of some mind that it thinks is optimal, for example one that is experiencing great pleasure.
Why is this a problem? Because it's not clear how it fits in with the various ethical systems that have been proposed. For example, utilitarianism says that each individual should be valued independently of others, and then added together to form an aggregate value. This seems to imply that each additional copy should receive full, undiscounted value, in conflict with the intuition of identical copy immortality.
Similar issues arise in various forms of ethical egoism. In hedonism, for example, does doubling the number of identical copies of oneself double the value of pleasure one experiences, or not? Why?
A full ethical account of independent identical copies would have to address the questions of quantum immortality and "modal immortality" (cf. modal realism), which I think are both special cases of identical copy immortality. In short, independent identical copies of us exist in other quantum branches, and in other possible worlds, so identical copy immortality seems to imply that we shouldn't care much about dying, as long as some copies of us live on in those other "places". Clearly, our intuition of identical copy immortality does not extend fully to quantum branches, and even less to other possible worlds, but we don't seem to have a theory of why that should be the case.
A full account should also address more complex cases, such as when the copies are not fully independent, or not fully identical.
I'm raising the problem here without having a good idea how to solve it. In fact, some of my own ideas seem to conflict with this intuition in a way that I don't know how to resolve. So if anyone has a suggestion, or pointers to existing work that I may have missed, I look forward to your comments.