I think this would be a genuine cause for concern, not because I don't think that people should be able to enter whatever relationships please them in principle, but because in practice I'm concerned about people being coerced into signing contracts harmful to themselves. Not sure where I'd draw the line exactly; this is probably a Hard Problem.

Well, there you go. Any restriction on freedom of contract can be rationalized as preventing something "harmful," one way or another.

And it's not a hard problem at all. It is in fact very simple: when people like something for ideological reasons, they will use the libertarian argument to support its legality, and when they dislike something ideologically, they will invent rationalizations for why the libertarian argument doesn't apply in this particular case. The only exceptions are actual libertarians, for whom the libertarian argument itself carries ideological weight, but they are an insignificant fringe minority. For everyone else, the libertarian argument is just a useful rhetorical tool to be employed and recognized only when it produces favorable conclusions.

In particular, when it comes to marriage, outside of the aforementioned libertarian fringe, there is a total and unanimous agreement that marriage is not a contract whose terms can be set freely, but rather an institution that is entered voluntarily, but whose terms are dictated (and can be changed at any subsequent time) by the state. (Even the prenuptial agreements allow only very limited and uncertain flexibility.) Therefore, when I hear a libertarian argument applied to marriage, I conclude that there are only two possibilities:

1. The speaker is an honest libertarian. However, this means either that he doesn't realize how wildly radical the implications of the libertarian position are, or that he actually supports these wild radical implications. (Suppose for example that a couple voluntarily sign a marriage contract stipulating death penalty, or even just flogging, for adultery. How can one oppose the enforcement of this contract without renouncing the libertarian principle?)

2. The speaker has an ideological vision of what the society should look like, and in particular, what the government-dictated universal terms of marriage should be (both with regards to the institution of marriage itself and its tremendous implications on all the other social institutions). He uses the libertarian argument because its implications happen to coincide with his ideological position in this particular situation, but he would never accept a libertarian argument in any other situation in which it would imply something disfavored by his ideology.