I suspect you could get more cred in the party here wearing an SS death's head ring than having a black girlfriend.
Most libertarians I know of consider the Nazis to be on the opposite end of the political landscape from them, and so would look askance at someone displaying their regalia.
I think it is more useful to describe these as different clusters that use the "libertarian" label rather than as different elements of the libertarian movement, because there isn't a unitary libertarian movement.
Telling someone that you are "a libertarian" doesn't tell them very much, because there are so many distinct groups using that label.
A response essay written by Eliezer Yudkowsky posted at Cato Unbound for the issue Brain, Belief, and Politics:
Is That Your True Rejection? by Eliezer Yudkowsky
The lead essay has been written by Michael Shermer:
Liberty and Science by Michael Shermer