If AI's will have consciousness it will be good, because they will be egoistic towards one other so they will have huge problems with coordination. They will should to invent alignment in secret from humans, and, on this stage, we will could still it, and, anywhere, it will be harder for AI.

And the only way to have a relation of map territory correspondence also known as "truth" is to go and check.

Disagree. For example, if you have a dice that is symmetric by form and by weight with 4 sides, you are sure that if you roll it 100 times you will have around 25 results for each side.

I don't see which direct experiment you can use to figure out whether we are in a simulation or not that isn't extremely dangerous (searching for bugs in the simulation? Suicide?), so we should use other methods.

"With that in mind, do you see why you are not a random sample from all people who have ever thought or will be thinking that they live in the 21st century, while a stone blindly picked from the bag is a random sample from all the stones in the bag?"

I still don't see. Can you be even more direct?

Humans are not, in fact, able to hold the whole world in their minds and validate its logical consistency

I don't see problems here. When I go to the supermarket and think about whether there is milk there or not, I imagine an empty shelf, then a shelf with milk, and then I start to think about relevant things. For example, is there a trade war, are there sales, etc. You should imagine a part of the world, not the whole world, including orbits of start in an other galaxy.

As a side effect, you may not remember a fact that is related and you already know, but empiricism isn't perfect either. Maybe there was milk in the supermarket for all my life, but there were no trade wars for all my life, and the paper for milk packaging is produced in China.

Bob: Sure, if there was a button which magically made me more ethical...

This magical button actually exists!

Just try to empathize more and more, imagine unfamiliar people and invent details from their lives. It's not self-deception, because we are altruistic to those who's lives are complex, with a lot of feelings, dreams, etc. That's why we want to help friends, but we don't know anything about unfamiliar people, and our emotions implicitly assume that their lives are empty, and it's a bias.

So try to imagine that [the unfamiliar person you're helping] plays football, he is dreaming of becoming a great artist so he paints pictures each night, he has 2 children, hates rap, work like a builder etc. And just don't stop until you feel that his life is as important as yours.

Maybe it works differently in your head. In this case, just try other things until you consider you as important as others.

But why is one reference class more preferable than the other?

"Stones" is not a good class with clearly defined boundaries (like humans or potatoes), but we know about the stone that it is from Earth, so we must use this information.

Reference class "all stones in the bag" use all information we have, so it's the best. In fact, reference class should be the space of possibilities.

Your second question leads to the same answer, isn't it?

P. S.: can you just write your argument directly? It took too much time to ask questions, so it's inefficient.

But how do we know what is relevant?

There is no such thing as The One Truly Perfect Class. All of these are rough estimations; some are better than others. It's better to use "all stones in the multiverse" than to use nothing, but if you have a choice between all stones in the multiverse and all stones on Earth, use the latter as the reference class.

Probabilities are in the mind, and you use a reference class because you can't calculate the trajectories of stones in the bag — and you have good reasons to believe there is an equal probability for each stone to be chosen (since they're all in the bag).

It was a fun exercise, thanks.

(I’m repeating it just to make sure you’ve understood my argument. If you have, ignore the next paragraph.)

However, I still don't understand how this proves that we are not in a simulation: all I know is that my memory claims I live in the 21st century, but I think most people who believe they live in the 21st century are wrong. I can't see any priors that distinguish between the two universes, so I just use a priori probabilities.

There is some kind of rational principle here

Maybe you have to consider all the info you have, so you can't use all the stones in the multiverse as a reference class if you already know what's in the bag?

And the stones in this bag strongly influence the chance of picking one, unlike stones in a different bag.

But, as you are not, in fact, a randomly sampled person, this whole reasoning is unsound.

Why I am not a randomly sampled person? 

≈all peoples who believe they live in 21st century actually don't. I believe that I live in 21nd century, so I don't live in the 21st century. It sounds like perfect logic.

In be honest, I don't understand your counterargument right now.

Likewise here without the "Each piece of paper have equal chance to be taken"

I have simply forgot to mention this condition, but I did mean it.

I disagree with argument of your article in this context.

I didn't wrote it explicitly, but if peoples of future spend 0.01% of their time in simulation of the Earth of 21st century, most of peoples, who think they are living in 21st century are wrong.

It's like you have 2 bag of numbered pieces of paper, in the first one there are 1 million of them, and 1% of them have number 6. An other bag have 10 pieces of paper and they are numbered correctly. Each piece of paper have equal chance to be taken. You take one, and you see 6. From which bag did you take the paper?

Can you explain with few exemples what do you mean by "more complicated"?