Denying people the ability to form voluntary contracts is the unfair part.

Unless you want to argue for some extreme form of anarcho-libertarianism, you would concede that there are some types of voluntary contracts that it is in the public interest for the state to consider unenforceable.
Selling yourself into slavery is the textbook example, but there are clearly many others.

I'm not saying that the type of pre-nup agreements that impose monetary compensation on adultery are necessarily in the same class of slavery or other forms of undesirable contracts, in fact, I have no strong intuitions either way.