I think you are laboring under a slight misapprehension about personality research. Myers-Briggs isn't solid science. The eneagram isn't solid science. Astrological personality models aren't solid science. I think you have correctly noticed that "psychological traits" are a ripe area for epistemically unsound belief systems that appear to bear on something people hold near and dear (ie understanding other people) so you're justifiably suspicious of a mention of personality, which is laudable.
But you're asking for a defense of "all that crazy stuff", and a good defense of "all that crazy stuff" can't honestly be provided, because most of it really is bunk, or at least it has so much bunk mixed in that its only good for psychoceramic data or maybe to pan for gold that might be hiding in the crazy. The big five personality model is an attempt to do actual science in the same space in order to produce reasonably valid and reliable dimensions of human "personality" variation. The point of the big five is that there is solid research and a deep literature and so on, in contrast to all the crackpot stuff.
If someone uses the big five and you're suspicious and ask for a defense of personality systems in general, that's like someone using geometry and you being suspicious because you're only aware of a lot of crackpots who keep trying to square the circle and so you ask them to defend the squaring the circle stuff, (which was proved to be impossible in 1882) before you'll accept analysis of evidence that legitimately makes use of a "suspiciously geometric" concept like the triangle inequality.
Unfortunately, defending established science quickly is hard because the content of science generally involves real inferential distances. If you want to start reading in this area, two useful keywords are Psychometrics and Trait theory.
In practice, "Openness to new experience" is the weakest part of the big five personality model. It can be measured reliably and predicts various things you'd expect it to predict and relatively naturally falls out when you settle for using 5 dimensions rather than 3 dimensions or 18 dimensions. However, when researchers tried the same thing on other cultures to see if this was a human universal, it turned out that the other four (Extraversion, Conscientiousness, Agreeableness, and Neuroticism) were relatively universal but in some cultures (if I remember correctly it was things like agrarian peasant societies?) basically everyone is pretty low in Openness relative to measurement norms derived from Japan or the US or whatever.
I think you are laboring under a slight misapprehension about personality research. Myers-Briggs isn't solid science. The eneagram isn't solid science.
Your understanding is consistent with mine. Myers-Briggs is really frustrating, because some of its ideas are anecdotally compelling (Introversion vs. Extraversion, Thinking vs. Feeling), while others are esoteric (Judging vs. Perceiving and Sensing vs. Intuition). At least on the types, INTP probably refers to a real phenotype (which is common on LW), but I don't know if any of the other type combination...
Evolutionary psychologist Geoffrey Miller recently published Spent: Sex, Evolution, and Consumer Behavior, a book on signaling, psychology, and consumerism. It's very up LW's alley - it reads almost as if Robin Hanson had written a book. (Actually, Hanson has never published a book, has he? Has anyone ever seen them in the same place? Hm...)
Sam Synder has written an overview/summary of the book, which I can attest hits many of the interesting points. (I would also praise the pervasive humor, which kept it readable and furnish many good examples of the 'reversal test', and the exercises at the end of the book.)
Some of the most interesting chapters to me were the ones dealing with Openness, which one will remember was recently shown may be changeable by psychedelics - possibly the first such tweakable member of the Big Five, leading to the suggestion that it may be worth considering changing it. Hold this thought.
First, Miller discusses the signaling of Openness (starting on page 108 of the PDF, logical page 207):
Why is Openness negative at its extreme? (Miller has remarked before this in Spent that despite what one might think, one of the other 6 psychological traits he covers, IQ, essentially has no bad amount to have - you have to be in the top percentile before IQ starts being a potential negative, and much marketing is covertly appealing to people's desires to look smart.) On the potential biological negatives of novelty-seeking:
Recent research shows something very curious: group Openness inversely correlates with parasite load, even after controlling for all the obvious confounds like health and longevity. (I haven't looked up this research yet; he attributes it to "Corey Fincher and Randy Thornhill at University of New Mexico, and Mark Schaller and Damian Murray at University of British Columbia".)
Incidentally, a good deal of LW's userbase could be described as 'young adults'; and it does seem relatively rare for old people to become transhumanists, as opposed to young or very young people. The next step, some anthropological observations which certainly look as if they are costly signalling something:
The final step - applying this idea to us:
The weak correlation with IQ has the troubling implication - what happens when you are highly Open but not especially intelligent, and you are confronted with memes & products optimized on the free market?
I am a little troubled because as a child I was interested in such alternative things and the Occult as well - I seem to recognize this pattern in myself. My inner Hanson asks me, 'why are you so sure you aren't still mistaken and that you aren't so Open your mind finally fell out?'
A closing link: 'the valley of bad rationality'.