To me libertarianism is more a community than a specific set of doctrines. There are certainly core values and epistemological underpinnings which define the ideological innovators and leaders in the libertarian community, and contrast them with those of opposing movements. But your discovery that the arguments for libertarianism "constantly shift around and are hard to pin down" is simply expected for an evolving community.

In terms of epistemological underpinnings, I'd say what best defines the libertarianism movement is a peculiar recognition of the nature of partial knowledge in thought and action. Hayek went to great lengths over the course of his career to explain why individuals who find enjoyment and skill in mathematics, physics, and so forth tend to react with skepticism to the arguments of libertarianism and free-market economics. To delve into the full depth of his thesis, begin with this book. For a quick summary, see the first few minutes of this video.

You say that libertarianism is obviously irrational. When we look at libertarianism as a community rather than a specific set of doctrines, your claim seems to boil down to the following: "The people in the libertarian community are clearly irrational." I assume that means they're incompetent and misguided? That they're unlikely to put into effect real, useful, and sustainable change in the world's economic and social systems?

I have a related question: What do you think about Bitcoin?