The AIs will not care about the things you care about. They'll have no reason to.

The "AIs" will be created by us. If we're smart enough, we will program the AIs to value the things we value (not in the same way that evolution programmed us).

The important thing is that we humans value love, and therefore we want love to perpetuate throught the universe.

I guess that this will lead to your concept of volition, isn't it?

Anyway, is Obert really arguing that morality is entirely outside the mind? Couldn't the "fact" of morality that he is trying to discover be derived from his (or humanity's, or whatever) brain design? And if you tweak Subhan's definition of "want" enough, couldn't they actually reach agreement?

Also, I take it that this means you don't believe in the whole, "if a program implements consciousness, then it must be conscious while sitting passively on the hard disk" thing. I remember this came up before in the quantum series and it seemed to me absurd, sort of for the reasons you say.

I used that as an argument against timeless physics: If you could have consciousness in a timeless universe, than this means that you could simulate a conscious being without actually running the simulation, you could just put the data on the hard drive. I'm still waiting out for an answer on that one!

Vladimir, why not? From reading your comment, it seems like the only reason you don't hurt other people is because you will get hurt by it, so if you would take the pill, you would be able to hurt other people. Have I got it wrong? Is this really the only reason you don't hurt people?

Vladimir, if there was a pill that would make the function of the mirror neurons go away, in other words, a pill that would make you able to hurt people without feeling remorse or anguish, would you take it?

Roko, what exactly do you mean by "optimal"? "optimal" means "good", which is another word for "ethical", so your definition of ethics doesn't actually tell us anything new! An AI can view the supergoal of "creating more paperclips" as the optimal/correct/succesful/good thing to do. the value of the AI's supergoal(s) doesn't has anything to do with it's intelligence.

I *still* don't get the point of timeless physics. It seems to me like two different ways of looking at the same thing, like classical configuration space vs relational configuration space. Sure, it may make more sense to formulate the laws of physics without time, and it may make the equations much simpler, but how exactly does it change your expected observations? In what ways does a timeless universe differ from a timeful universe?

Also, I don't think it's neccessary to study quantum mechanics in order to understand personal identity. I've reached the same conclusions about identity without knowing anything about QM, I feel it's just simple deductions from materialism.

Wait a second, this doesn't make sense. If the universe is timeless, then you don't have to actually simulate the universe on a computer. You can just create a detailed model of the universe, put in the neccesery causality structure, stick it in the RAM, and voila! you have conscious beings living out their lives in a universe. You don't even have to put it in the RAM, you can just write out symbols on a piece of paper! Or can this impeccable line of reasoning be invalidated by experimental evidence?

But the experiment does'nt prove that the two photons are really identical, it just proves that the photons are identical as far as the configurations are concerned. The photons could still have tiny tags with a number on them, but for some reason the configurations don't care about tags.