A difference between the two scenarios you present is that in the first, the threat makes the blackmailed party worse off than if the threat had never been made, whether they yield to the threat or refuse. In the second, this is not true: rejecting the offer to sell the McGuffin leaves them in exactly the same position as if the offer had never been made.
I believe that is the difference you are looking for: making someone an offer they cannot refuse vs. making them an offer they can refuse.
Precommitting to rejecting an offer makes them worse off than if the precommittment had never been made. The difference seems more a question of degree than of kind.
And we can bring in some extra connotations to make the first scenario better and the second worse. For instance, the blackmail could be "spend an evening talking over this with me, then I'll give you back the letters", while the MacGuffin could be a cure for a universal plague or something:
"I own this bauble that can save your civilization, which will otherwise die. I precommit ...
For a more parable-ic version of this, see here.
Suppose I make a precommitment P to take action X unless you take action Y. Action X is not in my interest: I wouldn't do it if I knew you'd never take action Y. You would want me to not precommit to P.
Is this blackmail? Suppose we've been having a steamy affair together, and I have the letters to prove it. It would be bad for both of these if they were published. Then X={Publish the letters} and Y={You pay me money} is textbook blackmail.
But suppose I own a MacGuffin that you want (I value it at £9). If X={Reject any offer} and Y={You offer more than £10}, is this still blackmail? Formally, it looks the same.
What about if I bought the MacGuffin for £500 and you value it at £1000? This makes no difference to the formal structure of the scenario. Then my behaviour feels utterly reasonable, rather than vicious and blackmail-ly.
What is the meaningful difference between the two scenarios? I can't really formalise it.