But... your head is part of reality, is it not?
I sometimes wonder. Maybe it's the other way round....
iwdw & Dave - it's a tempting idea, but I'd say that ultimately it's wrong.
My liking of Wigginettes is a fact about me, not a fact about Wigginettes. I can't spontaneously create a new Thingspace dimension, say 'look, Wigginettes glow when you look through this dimension, hence Wigginettes is an objectively valid category'. My liking is based on two unrelated properties, A and B, and maybe that 'creates' a third property C, but that property only describes me. Yes, my liking can be described neurochemically if you like, but that's information about my brain. It doesn't tell you anything about Wigginettes.
I hope Eliezer will back me up on this. Remember that Thingspace should be based on Scientific Facts About Things - that's the only way it can help us think about the world.( I guess you could argue that we each carry our own little Thingspace around with us, but, well, meh. Ideal Thingspace is (supposed to be) a direct, exhaustive map of the territory, not a map of my map.) Assigning the property 'liked by Ben' to Wigginettes (rather than me) is a mistake.
[Hence Eliezer's word 'arbitrary' when talking about trying to give this category/utility function to an AI. I suppose the measure of an 'arbitrary utility function' is whether or not it requires a hack to be transferred to a machine?]
But my original question stands. I'm not drawing a boundary around any objective pattern in Thingspace. Is Wigginettes a wrong word? Maybe this comes down to whether words are based on Facts About Things or Human Utility, Sticking To The Facts. And surely the latter is the more useful of the two. Eliezer, where do you go to define your words, Thingspace, or your head?
-Apologies for length, all.
Some reader is bound to declare that a better title for this post would be "37 Ways That You Can Use Words Unwisely", or "37 Ways That Suboptimal Use Of Categories Can Have Negative Side Effects On Your Cognition".
But one of the primary lessons of this gigantic list is that saying "There's no way my choice of X can be 'wrong'" is nearly always an error in practice, whatever the theory. You can always be wrong. Even when it's theoretically impossible to be wrong, you can still be wrong. There is never a Get-Out-Of-Jail-Free card for anything you do. That's life.
Besides, I can define the word "wrong" to mean anything I like - it's not like a word can be wrong.
Personally, I think it quite justified to use the word "wrong" when:
Everything you do in the mind has an effect, and your brain races ahead unconsciously without your supervision.
Saying "Words are arbitrary; I can define a word any way I like" makes around as much sense as driving a car over thin ice with the accelerator floored and saying, "Looking at this steering wheel, I can't see why one radial angle is special - so I can turn the steering wheel any way I like."
If you're trying to go anywhere, or even just trying to survive, you had better start paying attention to the three or six dozen optimality criteria that control how you use words, definitions, categories, classes, boundaries, labels, and concepts.