7 of 10. I underestimated Asian (Eurasian?) continent area by factor 4 (safety margin one order of magnitude), quantity of US dollars by factor 10 (safety margin 3 orders of magnitude) and volume of gr. lakes by factor 0.1 (safety margin 3 orders of magnitude). Other safety margins were 3 orders of magnitude for Titanic, Pacific coast (fractal-like curves can be very long), book titles, and 0.5 from mean value for others. Sigh, I thought I'll have 90%.

Hm, I estimated area of Asian continent as area of triangle with 10000km base (12 timezones for 20000 km and factor of 0.5 for pole proximity) and 10000km height (north pole to equator), and lost one order of magnitude in calculation.

If the decision of a group decider is "1/9th as important", then what's the correct way to calculate the expected benefit of saying "yea" in the second case? Do you have in mind something like 0.9*1000/9 + 0.1*100/1 = 110? This doesn't look right :-(

You asked how to reason about counterfactuals.

I answered.

You answered informally. It's easy, and not what I wondered about.

The reason I think about counterfactuals is to understand cause and effect. If you change something, then anything to which it is a cause must also change.

Things don't change. When you make a decision, you are not changing the future, you are deciding the future. The future is what it is given your actual decision, all else is fantasy (logically inconsistent even, because the structure of your own mind implies only one decision, when you ignore some unimportant cognitive noise), perhaps morally important fantasy whose structure we ought to understand, but still not the reality.

Are you saying your argument is true with the strict application of only SR or that in is true in reality?

I would say it can not be true in reality because muons and other particles take a measurably longer amount of time to decay as their speed increases.

I am in graduate school for physics you have applied special relativity incorrectly. The person in the space ship will experience time twice as slow as people on earth. So the person in the spaceship would expect people on earth to age twice as quickly.

I do not understand why wikipedia has this mixed(wrong or worded very confusingly) up so often. Check here.

I understand the grandparent comment now. Open/closed distinction can in principle be extracted from values, so that values of the original agent only specify what kind of program the agent should self-improve into, while that program is left to deal with any potential observations. (It's not better to forget some component of values.)

I don't understand the question. Unpack closed form/no closed form, and where updating comes in. (I probably won't be able to answer, since this deals with observations, which I don't understand still.)

Direct question. I cannot infer answer from you posts. If human values do not exist in closed form (i.e. do include updates on future observations including observations which in fact aren't possible in our universe), then is it better to have FAI operating on some closed form of values instead?

If physics is known perfectly and the first generation uses a proof checker to create the second, we're done.

No, since you still run the risk of tiling the future with problem-solving machinery of no terminal value that never actually decides (and kills everyone in the process; it might even come to a good decision afterwards, but it'll be too late for some of us - the Friendly AI of Doom that visibly only cares about Friendliness staying provable and not people, because it's not yet ready to make a Friendly decision).

Also, FAI must already know physics perfectly (with uncertainty parametrized by observations). Problem of induction: observations are always interpreted according to a preexisting cognitive algorithm (more generally, logical theory). If AI doesn't have the same theory of environment as we do, it'll make different conclusions about the nature of the world than we'd do, given the same observations, and that's probably not for the best if it's to make optimal decisions according to what we consider real. Just as no moral arguments can persuade an AI to change its values, no observations can persuade an AI to change its idea of reality.

But unknown physics could always turn out to be malicious in exactly the right way to screw up everything.

Presence of uncertainty is rarely a valid argument about possibility of making an optimal decision. You just make the best decision you can find given uncertainty that you're dealt. Uncertainty is part of the problem anyway, and can as well be treated with precision.

Don't let EY chill your free speech -- this is supposed to be a community blog devoted to rationality... not a SIAI blog where comments are deleted whenever convenient.

You are compartmentalizing. What you should be asking yourself is whether the decision is correct (has better expected consequences than the available alternatives), not whether it conflicts with freedom of speech. That the decision conflicts with freedom of speech doesn't necessarily mean that it's incorrect, and if the correct decision conflicts with freedom of speech, or has you kill a thousand children (estimation of its correctness must of course take this consequence into account), it's still correct and should be taken.

(There is only one proper criterion to anyone's actions, goodness of consequences, and if any normally useful heuristic stays in the way, it has to be put down, not because one is opposed to that heuristic, but because in a given situation, it doesn't yield the correct decision. )

(This is a note about a problem in your argument, not an argument for correctness of EY's decision. My argument for correctness of EY's decision is here and here.)