because the arguments constantly shift around and are hard to pin down.

That's not an argument against libertarianism, that's an argument that people with a fairly diverse set of views call themselves libertarians. I think that happens to be true.

On a sufficiently high level of abstraction I'd probably say that the two main features of libertarianism are (1) an unusually high preference for liberty/freedom; and (2) a far-off-the-center position on the individualism vs collectivism axis. Point (1) directly leads to a strong suspicion of power structures such as the state.

assuming you had a well-defined utility function you wanted to maximize, how would you possibly go about providing a truly rational justification that libertarianism applied to a large mass of complicated human beings would result in the desired outcome?

I don't quite understand the question. If you have a "well-defined utility function", well, you just try to maximize it to the best of your ability. You seem to be thinking of a scenario where you're a god-king who gets to arrange the society (and individual values) as he sees fit. That is obviously incompatible with libertarianism at a basic level. And then you are talking about capitalism and socialism as if they were "working procedure[s] for governing people", but they are not. Economic systems are not power structures.