You are asking if we create such exact simulations of humans that they will have all the typical limitations would they have the same wants as real humans, probably yes.

I'm also asking, should we care?
More generally, I'm asking what is it about real humans we should prefer to preserve, given the choice? What should we be willing to discard, given a reason?

The original question Wei Dai was asking me was about my statement that if we becomes uploads "At that point you already lost humanity by definition".

Fair enough. I've already agreed that this is true for the definitions you've chosen, so if that's really all you're talking about, then I guess there's nothing more to say. As I said before, I don't think those are useful definitions, and I don't use them myself.

Does the fact that somewhere in the cyberspace there is still a piece of source code which wants the same things as I do makes a difference in this scenario?

Source code? Maybe not; it depends on whether that code is ever compiled.
Object code? Yes, it makes a huge difference.

I still feel like humanity gets destroyed in this scenario, but you are free to disagree with my interpretation.

Some things get destroyed. Other things survive. Ultimately, the question in this scenario is how much do I value what we've lost, and how much do I value what we've gained?
My answer depends on the specifics of the simulation, and is based on what I value about humanity.

The thing is, I could ask precisely the same question about aging from 18 to 80. Some things are lost, other things are not. Does my 18-year-old self get destroyed in the process, or does it just transform into an 80-year-old? My answer depends on the specifics of the aging, and is based on what I value about my 18-year-old self.

We face these questions every day; they aren't some weird science-fiction consideration. And for the most part, we accept that as long as certain key attributes are preserved, we continue to exist.

Ah.

So when you say "most typically human feelings (hungry, thirsty, tired, etc.) will not be preserved creating a new type of an agent" you're making a definitional claim that whatever the new agent experiences, it won't be a human feeling, because (being software) the agent definitionally won't be a human. So on your view it might experience hunger, thirst, fatigue, etc., or it might not, but if it does they won't be human hunger, thirst, fatigue, etc., merely simulated hunger, thirst, fatigue, etc.

Yes? Do I understand you now?

FWIW, I agree that there are definitions of "human being" and "software" by which a piece of software is definitionally not a human being, though I don't think those are useful definitions to be using when thinking about the behavior of software emulations of human beings. But I'm willing to use your definitions when talking to you.

You go on to say that this agent, not being human, will not want the same things as a human.
Well, OK; that follows from your definitions.

One obvious followup question is: would a reliable software simulation of a human, equipped with reliable software simulations of the attributes and experiences that define humanity (whatever those turn out to be; I labelled them X2 above), generate reliable software simulations of wanting what a human wants?

Relatedly, do we care? That is, given a choice between an upload U1 that reliably simulates wanting what a human wants, and an upload U2 that doesn't reliable simulate wanting what a human wants, do we have any grounds for preferring to create U1 over U2?

Because if it's important to us that uploads reliably simulate being human, then we should design our uploads so that they have reliable simulations of X2. Right?

Well, yes, a lot depends on what we assume the upload includes, and how important the missing stuff is.
If Dave!upload doesn't include X1, and X2 defines Dave!original's humanity, and X1 contains X2, then Dave!upload isn't human... more or less tautologically.

We can certainly argue about whether our experiences of hunger, thirst, fatigue, etc. qualify as X1, X2, or both... or, more generally, whether anything does. I'm not nearly as confident as you sound about either of those things.

But I'm not sure that matters.

Let's posit for the sake of comity that there exists some set of experiences that qualify for X2. Maybe it's hunger, thirst, fatigue, etc. as you suggest. Maybe it's curiosity. Maybe it's boredom. Maybe human value is complex and X2 actually includes a carefully balanced brew of a thousand different things, many of which we don't have words for.

Whatever it is, if it's important to us that uploads be human, then we should design our uploads so that they have X2. Right?

But you seem to be taking it for granted that whatever X2 turns out to be, uploads won't experience X2.
Why?

you can get most conceivable benefits from domain expert AI without any need for AGI.

Is there a short form of where you see the line between these two types of systems? For example, what is the most "AGI-like" AI you can conceive of that is still "really a domain-expert AI" (and therefore putatively safe to develop), or vice-versa?

My usual sense is that these are fuzzy terms people toss around to point to very broad concept-clusters, which is perfectly fine for most uses, but if we're really getting to the point of trying to propose policy based on these categories, it's probably good to have a clearer shared understanding of what we mean by the terms.

That said, I haven't read your paper; if this distinction is explained further there, that's fine too.

Insofar as Maslow's pyramid accurately models human psychology (a point of which I have my doubts), I don't think the majority of people you're likely to be speaking to on the Internet are defined in terms of their low-level physiological needs. Food, shelter, physical security -- you might have fears of being deprived of these, or even might have experienced temporary deprivation of one or more (say, if you've experienced domestic violence, or fought in a war) but in the long run they're not likely to dominate your goals in the way they might for, say, a Clovis-era Alaskan hunter. We treat cases where they do as abnormal, and put a lot of money into therapy for them.

If we treat a modern, first-world, middle-class college student with no history of domestic or environmental violence as psychologically human, then, I don't see any reason why we shouldn't extend the same courtesy to an otherwise humanlike emulation whose simulated physiological needs are satisfied as a function of the emulation process.

I'm going to throw out some more questions. You are by no means obligated to answer.

In your AI Safety Engineering paper you say, "We propose that AI research review boards are set up, similar to those employed in review of medical research proposals. A team of experts in artificial intelligence should evaluate each research proposal and decide if the proposal falls under the standard AI – limited domain system or may potentially lead to the development of a full blown AGI."

But would we really want to do this today? I mean, in the near future--say the next five years--AGI seems pretty hard to imagine. So might this be unnecessary?

Or, what if later on when AGI could happen, some random country throws the rules out? Do you think that promoting global cooperation now is a useful way to address this problem, as I assert in this shamelessly self-promoted blog post?

The general question I am after is, How do we balance the risks and benefits of AI research?

Finally you say in your interview, "Conceivable yes, desirable NO" on the question of relinquishment. But are you not essentially proposing relinquishment/prevention?

I think if I became an upload (assuming it's a high fidelity emulation) I'd still want roughly the same things that I want now. Someone who is currently altruistic towards humanity should probably still be altruistic towards humanity after becoming an upload. I don't understand why you say "At that point you already lost humanity by definition".

Hi Roman. Would you mind answering a few more questions that I have after reading your interview with Luke? Carl Shulman and Nick Bostrom have a paper coming out arguing that embryo selection can eventually (or maybe even quickly) lead to IQ gains of 100 points or more. Do you think Friendly AI will still be an unsolvable problem for IQ 250 humans? More generally, do you see any viable path to a future better than technological stagnation short of autonomous AGI? What about, for example, mind uploading followed by careful recursive upgrading of intelligence?

Hey, my name is Roman. You can read my detailed bio here, as well as some research papers I published on the topics of AI and security. I decided to attend a local LW meet up and it made sense to at least register on the site. My short term goal is to find some people in my geographic area (Louisville, KY, USA) to befriend.

Just registered for this Meetup. Curious if anyone else will be coming from Louisville, KY?