Pardon a second comment (I hope that's not bad etiquette), but here are a couple of further qualms/criticisms attending to which could improve the post:

Regarding your use of the phrase 'foundations of probability' to refer to arguments for why a certain kind of robot should use probabilities: this seems like a rather odd use for a phrase that already has at least two well established uses. (Roughly (i) basic probability theory, i.e. that which gives a grounding or foundation in learning the subject, and (ii) the philosophical or metaphysical underpinnings of probability discourse: what's it about, what kinds are there, what makes true probability claims true etc.?) Is it really helpful to be different on this point, when there is already considerable ambiguity?

Furthermore, and perhaps more substantively, your bit on Dutch Books doesn't seem to give any foundations in your sense: Dutch Book arguments aren't arguments for using probability (i.e. at all, i.e. instead of not using it), but rather for conforming, when already using probability, to the standard probability calculus. So there seems to be a confusion in your post here.

'But to assign some probability to the wrong answer is logically equivalent to assigning probability to 0=1.'

Huh? This doesn't make sense to me. First of all, it seems like a basic category-mistake: acts of assigning probabilities don't seem to be the sorts of things that can bear logical relations like equivalence to each other.

Perhaps that's just pedantry and there's a simple rephrasing that says what you really want to say, but I have a feeling I would take issue with the rephrased version too. Does it trade on the idea that all false mathematical propositions are logically equivalent to each other? (If so, I'd say that's a problem, because that idea is very controversial, and hardly intuitive.)

I really liked the introduction - really well done. (shminux seems to agree!)

Some constructuve criticisms:

'There are playing fields where you should cooperate with DefectBot, even though that looks completely insane from a naïve viewpoint. Optimality is a feature of the playing field, not a feature of the strategy.' - I like your main point made with TrollBot, but this last sentence doesn't seem like a good way of summing up the lesson. What the lesson seems to be in my eyes is: strategies' being optimal or not is playing-field relative. So you could say that optimality is a relation holding between strategies and playing fields.

Later on you say 'It helps to remember that "optimality" is as much a feature of the playing field as of the strategy.' - but, my criticism above aside, this seems inconsistent with the last sentence of the previous quote (here you say optimality is equally a feature of two things, whereas before you said it was not a feature of the strategy)! Here you seem to be leaning more toward my proposed relational gloss.

Another suggestion. The Omega argument comes right after you say you're going to show that we occupy a strange playing field right now. This tends to make the reader prepare to object 'But that's not very realistic!'. Maybe you like that sort of tension and release thing, but my vote would be to first make it clear what you're doing there -- i.e., not right away arguing about the real world, but taking a certain step toward that.

One final suggestion. You write 'Knowing this, I have a compartment in which my willpower doesn't deplete', and something relevantly similar just earlier. Now this is obviously not literally what you mean - rather, it's something like, you have a compartment housing the belief that your willpower doesn't deplete. Obviously, you get a certain literary effect by putting it the way you do. Now, I realize reasonable people may think I'm just being overly pedantic here, but I suspect that's wrong, and that in this sort of discussion, we should habitually help ourselves to such easily-had extra precision. Since things get confusing so quickly in this area, and we're liable to slip up all over the place, apparently minor infelicities could make a real difference by sapping resources which are about to be taxed to the full.

'At its core, model theory is the study of what you said, as opposed to what you meant.'

One way to improve the clarity of this gloss, and make it more ecumenical (to be frank, I imagine as it stands, many philosophers would balk and sort of go 'WTF?' and treat this as a weird, confused thing to say), might be as follows: distinguishing the meaning of an expression in some language from the speaker's intended meaning in producing that expression. These can of course diverge, but both are semantic notions. (Your use of the two different terms above may obscure this, by suggesting that you can't use 'mean' and its cognates for speaker-meaning.)

Kripke has a paper called 'Speaker's Reference and Semantic Reference' which may be helpful. (There's an online copy here at present, but in any case it's not hard to find.) What it seems you want to do, insofar as you think there's more to meaning than reference, is something like: generalize Kripke's basic idea here to meaning in general (and so factoring in internal components of meaning such as 'role in the system', as well as just reference) and then use that distinction to say that model theory is about linguistic as opposed to speaker-meaning.

But I'm still not sure why you'd want to say (or emphasize) that. My reaction is: yeah, applications of model theory are often geared that way, but why couldn't you also give model-theoretic accounts of speaker-meaning? But perhaps I've misunderstood.

Stimulating as always! I have a criticism to make of the use made of the term 'rigid designation'.

Multiple philosophers have suggested that this stance seems similar to "rigid designation", i.e., when I say 'fair' it intrinsically, rigidly refers to something-to-do-with-equal-division. I confess I don't see it that way myself [...]

What philosophers of language ordinarily mean by calling a term a rigid designator is not that, considered purely syntactically, it intrinsically refers to anything. The property of being a rigid designator is something which can be possessed by an expression in use in a particular language-system. The distinction is between expressions-in-use whose reference we let vary across counterfactual scenarios (or 'possible worlds'), e.g. 'The first person to climb Everest', and those whose reference remains stable, e.g. 'George Washington', 'The sum of two and two'.

There is some controversy over how to apply the rigid/non-rigid distinction to general terms like 'fair' (or predicates like 'is fair') - cf. Scott Soames' book Beyond Rigidity - but I think the natural thing to say is that 'is fair' is rigid, since it is used to attribute the same property across counterfactual scenarios, in contrast with a predicate like 'possesses my favourite property'.

When Richard Stallman wants to view a web page, he sends the URL in an email to a server operated by him or one of his friends, and the server mails him back the page. He says that he does it that way "for personal reasons", and my guess is that it is an anti-akrasia measure.

Thanks. Maybe I should look into those.

Though it should be noted that while "every first draft is terrible" may be correct for Stephen King, it's not necessarily correct for everyone. There are writers who only do minor revisions to their first draft, while others do several drafts before it gets great.

I take no responsibility for anything Luke-2007 did. Different guy. :)

I'd like to take that one back, for sure.

I don't know, you've made a lot of people laugh with this and you'll be able to use this story for several more decades. You might make tens of thousands of people laugh which could be net positive utilons.