I think you're mostly right. This suggests that a better policy than 'don't respond to blackmail' is 'don't respond to blackmail if and only if you believe the blackmailer to be someone who is capable of accurately modelling you'.

Unfortunately this only works if you have perfect knowledge of blackmailers and cannot be fooled by one who pretends to be less intelligent than they actually are.

This also suggests a possible meta-strategy for blackmailers, namely "don't allow considerations of whether someone will pay to affect your decision of whether to blackmail them", since if blackmailers were known to do this then "don't pay blackmailers" would no longer work.

I would also suggest that while blackmail works with some agents and not others, it isn't human-specific. For example, poison arrow frogs seem like a good example of evolution using a similar strategy, having an adaptation that is in no way directly beneficial (and presumably is at least a little costly) that exists purely to minimize the utility of animals which do not do what it wants.