I'm confused by a couple minor points here, also:

1. The paper asks for a "probability distribution over models of L". In fact, for many languages L, models of L form a proper class. Does this cause measure-theoretic difficulties? It seems like this might force mu to be zero on all sufficiently large models (otherwise you can do some sort of transfinite induction to get sets of unbounded measure) but I'm not very good at crazy set theory stuff.

2. At one point the authors state "We would like P(forall phi in L' <blah>)". I thought we were in a first-order language and therefore couldn't quantify over propositions?

3. It's not immediately clear to me that this actually constructs a measure on the set of theories: that is, if S is the set of all complete consistent theories, it's not clear to me that for the mu we construct by martingale, mu(S) = 1 (or even that mu(S) != 0). Mightn't additivity break when we take the limit and get a whole theory rather than just an incomplete bag of axioms?

Thanks for the feedback. I agree that reductio ad absurdum is the weakest of the examples I gave, but let me try to justify it anyways: if X is a fully general counterargument, then we can use it to argue against true statements as well as false ones. So applying X without any additional justification would lead to patently false conclusions, and therefore (by reductio ad absurdum) X is not a valid form of reasoning. Perhaps this is not the best word for it, but it is similar to a very pervasive idea in mathematics, where when formulating possible approaches to prove a theorem, a key criterion is whether those approaches can distinguish between the theorem and similar statements that are known or suspected to be false.

ETA: And yes, I agree that specific examples are good!

It seems like something has gone terribly wrong when our ethical decisions depend on our interpretation of quantum mechanics.

My understanding was that many-worlds is indistiguishable by observation from the Copenhagen interpretation. Has this changed? If not, it frightens me that people would choose a higher chance of the world ending to rescue hypothetical people in unobservable universes.

If anything this seems like a (weak) argument in favour of total utilitarianism, in that it doesn't suffer from giving different answers according to one's choice among indistiguishable theories.

Retention time can be increased to days or perhaps weeks by cooling to cryogenic temperatures before power down

Actually no, modern DRAM loses information in milliseconds, even assuming you could cool it down to liquid helium temperatures. After a few seconds the data is almost entirely random.

I wrote up about a page-long reply, then realized it probably deserves its own posting. I'll see if I can get to that in the next day or so. There's a wide spectrum of possible solutions to the personal identity problem, from physical continuity (falsified) to pattern continuity and causal continuity (described by Eliezer in the OP), to computational continuity (my own view, I think). It's not a minor point though, whichever view turns out to be correct has immense ramifications for morality and timeless decision theory, among other things...

It looks to me like you want a cryptographically secure pseudo-random number generator restricted to the output space {0, 1} and with a known seed. That's unbiased and intractable pretty much by definition, indexable up to some usually very large periodicity, and typically verifiable and simple to refer to because that's standard practice in the security world.

There's plenty of PRNGs out there, and you can simply truncate or mod their outputs to give you the binary output you want; Fortuna looks like a strong candidate to me.

(I was going to suggest the Mersenne twister, which I've actually implemented before, but on further examination it doesn't look cryptographically strong.)

There are a number of possibilities still missing from the discussion in the post. For example:

• There might not be any such thing as a friendly AI. Yes, we have every reason to believe that the space of possible minds is huge, and it's also very clear that some possibilities are less unfriendly than others. I'm also not making an argument that fun is a limited resource. I'm just saying that there may be no possible AI that takes over the world without eventually running off the rails of fun. In fact, the question itself seems superficially similar to the halting problem, where "running off the rails" is the analogue for "halting"; suggesting that even if friendliness existed, it might not be rigorously provable. (note: this analogy doesn't say what I think it says; see response below. But I still mean to say what I thought; a friendly world may be fundamentally less stable than a simple infinite loop, perhaps to the point of being unprovable.)

• Alternatively, building a "Friendly-enough" AI may be easier than you think. Consider the game of go. Human grandmasters (professional 9-dan players) have speculated that "God" (that is, perfect play) would rate about 13 dan professionally; that is, that they could beat such a player more than half the time given a 3 or 4 stone handicap. Replace "go" with "taking over the world", "professional 9-dan player" with "all of humanity put together", and "3 or 4 stone handicap" with "relatively simple-to-implement Asimov-type safeguards", and it is possible that this describes the world. And it is also possible that a planetary computer would still "only be 12-dan"; that is, that additional computing power shows sharply diminishing intelligence returns at some point "short of perfection", to the point where a mega-computer would still be noticeably imperfect.

There may be good reasons not to spend much time thinking about the possibilities that FAI is impossible or "easy". I know that people around here have plenty of plausible arguments for why these possibilities are small; and even if they are appreciable, the contrary possibility (that FAI is possible but hard) is probably where the biggest payoffs lie, and so merits our focus. And the OP discussion does seem valid for that possible-hard case. But I still think it would be improved by stating these assumptions up-front, rather than hiding or forgetting about them.

Similarly: What are a qubit's specs? I would like to be able to think about what class of problem would be trivial with a quantum computer.

I don't understand the meaning of the words "want", "innately sticky", and "honestly have a goal" as applied to an AI (and not to a human).

Constraints have to influence behavior by enumerating EVERYTHING you don't want to happen

Not at all. Constraints block off sections of solution space which can be as large as you wish. Consider a trivial set of constraints along the lines of "do not affect anything outside of this volume of space", "do not spend more than X energy", or "do not affect more than Y atoms".

What happens if my valuation is noncircular, but is incomplete? What if I only have a partial order over states of the world? Suppose I say "I prefer state X to Z, and don't express a preference between X and Y, or between Y and Z." I am not saying that X and Y are equivalent; I am merely refusing to judge.

My impression is that real human preference routinely looks like this; there are lots of cases people refuse to evaluate or don't evaluate consistently.

It seems like even with partial preferences, one can be consequentialist -- if you don't have clear preferences between outcomes, you have a choice that isn't morally relevant. Or is there a self-contradiction lurking?