Eliezer, your main point is correct and interesting, but the coin flip example is definitely wrong. The market's beliefs don't affect the bias of the coin! The map doesn't affect the territory.
The relevant FINANCE question is 'how much would you pay for a contract that pays $1 if the coin comes up heads?'. This is then the classic prediction market type contract.
The price should indeed be ten elevenths. Of course, you don't expect to make money buying this contract, which was exactly your point.
What WILL be true is that the expected change in the price of ...
Nature sounds a bit like a version of Rory Breaker from 'Lock, Stock and Two Smoking Barrels':
"If you hold back anything, I'll kill ya. If you bend the truth or I think your bending the truth, I'll kill ya. If you forget anything I'll kill ya. In fact, you're gonna have to work very hard to stay alive, Nick. Now do you understand everything I've said? Because if you don't, I'll kill ya. "
You say 'That's not how it works.' But I think that IS how it works!
If progress were only ever made by people as smart as E.T. Jaynes, humanity would never have gotten anywhere. Even with fat tails, intelligence is still roughly normally distributed, and there just aren't that many 6 sigma events. The vast majority of scientific progress is incremental, notwithstanding that it's only the revolutionary achievements that are salient.
The real question is, do you want Friendly A.I. to be achieved? Or do you just want friendly A.I. to be achieved by YOU? There'...
While your point about a world that hadn't used nuclear weapons being safer today is unclear, I think your claim that 'you wouldn't drop the bomb' is driven by hindsight bias. At the time, the far more pressing issue from Truman's perspective was how to end the war with a minimum loss of US life, and the long term consequences of the bomb were far from clear.
I also think that memorials like Hiroshima Day pervert the overall moral perspective on World War 2. Because it was a large, salient act of destruction, it gets remembered. The Burma Railway and the Ra...
I'm quite convinced about how you analyze the problem of what morality is and how we should think about it, up until the point about how universally it applies. I'm just not sure that 'humans different shards of god shatter' add up to the same thing across people, a point that I think would become apparent as soon as you started to specify what the huge computation actually WAS.
I would think of the output as not being a yes/no answer, but something akin to 'What percentage of human beings would agree that this was a good outcome, or be able to be thus conv...
I'll be interested to see what your metamorality is. The one thing that I think has been missing so far from the discussion is the question that without some metamorality, what language do we have to condemn someone else who chooses a different morality from ours? Obviously you can't argue morality into a rock, but we're not trying to do that, only argue it into another human who shares fundamentally similar architecture, but not necessarily morality.
Moreover, to say that one can abandon a metamorality without affecting one's underlying morality doesn't im...
Russ, I think that if you take the example literally, the price would be 91%, not 50%, and you wouldn't expect to make money.
Eliezer, the PS definitely clarifies matters.
Although I also think the example is actually instructive if taken literally too. In particular, if you see nine heads in a row, each additional head means you expect a higher chance of heads next flip. But you do not expect an increase in the price of the contract that pays $1 if heads comes up. THAT still has an expected price change of zero, even thought we expect more heads going forward.
In other words, future EVENTS can be predictable, but future PRICES cannot.