Manon: What's the bad thing that happens if I do 35?
You waste years of your life on dreadful AI designs based around suggestively named LISP tokens. See Drew McDermott's classic Artificial Intelligence Meets Natural Stupidity. More on this later.
Ben Jones: Perhaps I am the sort of guy who has a Jones for green-eyed, black-haired girls.
Yes, by putting arbitrary boundaries into the utility function, I can force an AI to develop concepts for things that are bound only by those boundaries. But human utility boundaries are typically around otherwise-interesting higher-density regions, due to the evolutionary origins of our psychology. In other words, this is theoretically correct, but I've yet to see the issue arise in real life.
Spindizzy: This is gratuitously emotive and doesn't help to clarify your point.
In the original case, I talked about wiggins. Here, summarizing, I have to pick a better-known example of how arbitrarily excluding something is not only bad, but a case of trying to get away with something without justifying it.
Some reader is bound to declare that a better title for this post would be "37 Ways That You Can Use Words Unwisely", or "37 Ways That Suboptimal Use Of Categories Can Have Negative Side Effects On Your Cognition".
But one of the primary lessons of this gigantic list is that saying "There's no way my choice of X can be 'wrong'" is nearly always an error in practice, whatever the theory. You can always be wrong. Even when it's theoretically impossible to be wrong, you can still be wrong. There is never a Get-Out-Of-Jail-Free card for anything you do. That's life.
Besides, I can define the word "wrong" to mean anything I like - it's not like a word can be wrong.
Personally, I think it quite justified to use the word "wrong" when:
Everything you do in the mind has an effect, and your brain races ahead unconsciously without your supervision.
Saying "Words are arbitrary; I can define a word any way I like" makes around as much sense as driving a car over thin ice with the accelerator floored and saying, "Looking at this steering wheel, I can't see why one radial angle is special - so I can turn the steering wheel any way I like."
If you're trying to go anywhere, or even just trying to survive, you had better start paying attention to the three or six dozen optimality criteria that control how you use words, definitions, categories, classes, boundaries, labels, and concepts.