I think the problem is just equivalent to dividing the gains from trade. We'll look at the second example first, since it is a canonical example of a trade.

I have the MacGuffin which I value at £9. You value it at £1000. Suppose that I give you the MacGuffin, you give me £9 and you put £991 on the table between us. Then we both have the utility we had when we started and we have £991 (the gains from the trade) to divide between us. Yay!

Now look at the first example. This is a canonical blackmail, which I'm going to re-write to make it look as much like the above example as possible.

I have some letters which incriminate us both. I value the letters not getting out at £M, and you value the letters not getting out at £N. Suppose I destroy the letters (rather than releasing them) and put £M on the table between us, while you put £N on the table between us. Then we both have the utility we would have had if the letters had been released but along with this we have £M+N (the "gains from the trade") to divide between us. Yay?

I think the reason why this example seems more blackmailesque is because there is a natural Schelling point for the division of the gains, namely (£M, £N), which corresponds to me destroying the letters without asking for anything. So asking you for money is rude, because I'm greedily going past the Schelling point.

The whole point of the exercise was to create a decision theory that doesn't comply with blackmail. Well, in a division of gains problem a sensible heuristic is to always demand a fair share (where "fair share" vaugely refers to some sort of Schelling point or something, depending on the problem). In the blackmail example the Schelling point is (£M, £N) so the heuristic tells us to demand at least £M, which is exactly refusing the blackmail!

My conclusion: Calling something blackmail or extortion is what the decision to defect in an ultimatum game feels like from the inside.

The above definition is approximately 2/3rds serious (in the same way that Hansonian cynicism like "X is not about Y" is about 2/3rd serious). That is, it represents what seems to be a salient point about how humans think about the subject and also about the arbitrariness that is inherit to the problem. While there are technically exceptions to the definition (some people will comply with extortion even when labelling it thus) to me at least it seems to capture a significant aspect of the meaning.

I do note that while I am calling the distinction 'arbitrariy', this isn't a criticism. Arbitrariness is inevitable in cases that inherently require some kind of arbitration and things like obvious shelling points and established norms become invaluable. In some cases the decision to defect in an ultimatum game is obvious while in others it is not and part of the way we go about making the distinction is by perceiving the situations to be inherently different despite formally equivalent payoff structures.

What is the meaningful difference between the two scenarios? I can't really formalise it.

Why do you care? In what way is this not merely a decision about how to use a word?

How about something like this as a definition: There is a "status quo", an outcome for each agent if neither takes any special action, that serves as reference point. Agent B is blackmailing A if s/he announces to A that the status quo is no longer a possible outcome, and the possible outcomes (relative to the status quo) have utilities either "-x for A, +y for B" or "-z for A, anything for B", with z > x. (ETA: typo that said A instead of B corrected in the last line.)

I don't think it is possible to define blackmail without a concept similar to the status quo, as this example I posted in the other thread suggests.

What about if I bought the MacGuffin for £500 and you value it at £1000? This makes no difference to the formal structure of the scenario (note that me valuing the MacGuffin at £11 would change the formal structure: then I might do X even if I knew Y was impossible: I would still reject an offer of £10). Then my behaviour feels utterly reasonable, rather than vicious and blackmail-ly.

I suggest moving the parenthetical to the end of the paragraph. I had to re-parse that in programming mode before I interpreted the ambiguity in the last sentence correctly and I wouldn't have noticed my confusion at all were I not familiar with the distinctions you are trying to express.

But suppose I own a MacGuffin that you want (I value it at £9). If X={Reject any offer} and Y={You offer more than £10}, is this still blackmail? Formally, it looks the same.

One big difference I see is what counts as an "action"; arguably X={Publish the letters} "feels" more like an action than X={Reject any offer}. It seems that when what distinguishes a "blackmailish" offer from a "non-blackmailish" one is a discrete distinction (either you precommit to publish or you don't), making the "blackmailish" offer is "bad", but when there's a continuum between the most and least blackmailish versions (as is the case with the MacGuffin negotiation), the offer doesn't seem to be as much of a transgression.

I'm not totally happy with this tho, and to me the general problem of "negotiation in situations with no focal points" has no satisfying solution beyond racing to who precommits first.

Why not taboo 'blackmail'? That word already has a bunch of different meanings in law and common usage.

A problem with blackmail is that it can be repeated. After you do Y (pay me money) and I don't do X (publish letters), I can go back and once again ask you for money. With your second example, after not rejecting the offer I no longer have the MacGuffin, so I can't repeat.

EDIT: The only way to rule out repetition is to include it in the precommitment. But note that for human agents it is impossible to precommit in an enforceable way without side-effects. They can't self-modify, so they would need a trusted third party. Then this trusted party will get to know at least the fact that there are some juicy secrets out there about Y.