I was reading Outlawing Anthropics and especially this subconversation has caught my attention. I got some ideas; but that thread is nearly four years old, so I am commenting here instead of there.

My version of the simplified situation: There is an intelligent rational agent (her name is Abby, she is well versed in Bayesian statistics) and there are two urns, each urn containing two marbles. Three of the marbles are green. They are macroscopic, so distinguishable, but not for Abby's senses. Anyway, Abby can number them to be marbles 1, 2 and 3, she is just unable to "read" the number even on close examination. One marble is red, she can distinguish it, and it gets number 0. One urn has marbles 0 and 2, this is the "even" urn, the second has marbles 1 and 3, and is called odd. Again, Abby cannot distinguish urns without examining marbles. Now, assistent takes both urns to another room, computes 256th binary digit of exp(-1), and gets back with just one urn of corresponding parity. Abby is allowed to draw one marble (turns out it is green) and then urn is taken away and Abby is basically asked to state her subjective probability of the urn being odd (by accepting or refusing some bets). And only then she is told that in another room there is another person (Bart) who is being presented with the same choices after drawing the other marble from the very same urn. And finally, Abby is asked (informally) what is her averaged expectation of Bart's subjective probability of the urn being odd (now that she sees her marble is green). And, if this average is different from her subjective probability, why is she not taking that value as an indirect evidence in her calculations (which clearly means that the assistent is just messing with her).

Assumptions are that neither Abby nor Bart have a clue about binary digits of exp(-1), they are not able to compute that far and so they assign prior probability of the urn being odd to 50%. Another assumption is that Abby and Bart both have chosen their marbles randomly, and in fact, they do not even know which one of them was drawing first. So there are 4 "possible" worlds, numbered by the marble Abby "would" have drawn, all of them appearing equally probable before the marble drawing.

Question is (of course) what subjective probability should Abby use when accepting/refusing bets. And to give a witty retort to assistant's "why" question, where applicable; or else, to explain why Boltzmann brains are not that big obstacle to rationality.

And here I am, way over my time budget, having finished around one third of my planned comment. So I guess I shall leave you with questions for now, and I will resume commenting later.

Edit: Note to self: Do not forget to include http:// in links. RTFM.

Edit: "possible" worlds, numbered by marble Abby has drawn -> "possible" worlds, numbered by marble Abby "would" have drawn

From If Many-Worlds had Come First:

the thought experiment goes: 'Hey, suppose we have a radioactive particle that enters a superposition of decaying and not decaying. Then the particle interacts with a sensor, and the sensor goes into a superposition of going off and not going off. The sensor interacts with an explosive, that goes into a superposition of exploding and not exploding; which interacts with the cat, so the cat goes into a superposition of being alive and dead. Then a human looks at the cat,' and at this point Schrödinger stops, and goes, 'gee, I just can't imagine what could happen next.' So Schrödinger shows this to everyone else, and they're also like 'Wow, I got no idea what could happen at this point, what an amazing paradox'. Until finally you hear about it, and you're like, 'hey, maybe at that point half of the superposition just vanishes, at random, faster than light', and everyone else is like, 'Wow, what a great idea!'"

Obviously this is a parody and Eliezer is making an argument for many worlds. However, this isn't that far from how the thought experiment is presented in introductory books and even popularizations. Why, then, don't more people realize that many worlds is correct? Why aren't tons of bright middle-school children who read science fiction and popular science spontaneously rediscovering many worlds?

...democracy?

/flicks the OFF switch.

(to be a bit more clear, political systems (such as democracy) are about power. Without being explicit about what power ems will have, specifically in the meatworld, the question seems too ill-defined to me)

P.S.: Does this web interface have anything like "preview" button?

There's a sandbox here, it's also linked to when you click "Show help", the button at the lower right corner of the text box which opens when you start a reply. Welcome, yay for more PhD-level physicists.

There's a deeper question here: ideally, we would like our CEV to make choices for us that aren't our choices. We would like our CEV to give us the potential for growth, and not to burden us with a powerful optimization engine driven by our childish foolishness.

One obvious way to solve the problem you raise is to treat 'modifying your current value approximation'' as an object-level action by the AI, and one that requires it to compute your current EV - meaning that, if the logical consequences of the change (including all the future changes that the AI predicts will result from that change) don't look palatable to you, the AI won't make the first change. In other words, the AI will never assign you a value set that you find objectionable right now. This is safe in some sense, but not ideal. The profoundly racist will never accept a version of their values which, because of its exposure to more data and fewer cognitive biases, isn't racist. Ditto for the devoutly religious. This model of CEV doesn't offer the opportunity for growth.

It might be wise to compromise by locking the maximum number of edges in the graph between you and your EV to some small number, like two or three - a small enough number that value drift can't take you somewhere horrifying, but not so tightly bound up that things can never change. If your CEV says it's okay under this schema, then you can increase or decrease that number later.