Eliezer said that another time you can't just generate new words is when a category boundary like "person" or "human" or "baby" makes a direct appearance in your utility function.
Which gently suggests that when defining a utility function that might remain in force for billions of years, one should prefer functions that do not have category boundaries.
I would be happy to exhibit functions of that sort that have the property that even after an explosion of engineered intelligence, the humans probably retain enough expected utility to keep them flourishing and protected from exploitation although they probably do not retain enough expected utility to cause the majority of the future light cone's space, time, matter, free energy and other resources to be devoted to them.
I have watched more than one conversation—even conversations supposedly about cognitive science—go the route of disputing over definitions. Taking the classic example to be "If a tree falls in a forest, and no one hears it, does it make a sound?", the dispute often follows a course like this:
In this example, Barry is arguing with Albert because of a genuinely different intuition about what constitutes a sound. But there's more than one way the Standard Dispute can start. Barry could have a motive for rejecting Albert's conclusion. Or Barry could be a skeptic who, upon hearing Albert's argument, reflexively scrutinized it for possible logical flaws; and then, on finding a counterargument, automatically accepted it without applying a second layer of search for a counter-counterargument; thereby arguing himself into the opposite position. This doesn't require that Barry's prior intuition—the intuition Barry would have had, if we'd asked him before Albert spoke—have differed from Albert's.
Well, if Barry didn't have a differing intuition before, he sure has one now.
Albert and Barry recruit arguments that feel like support for their respective positions, describing in more detail the thoughts that caused their "sound"-detectors to fire or stay silent. But so far the conversation has still focused on the forest, rather than definitions. And note that they don't actually disagree on anything that happens in the forest.
Insult has been proffered and accepted; now neither party can back down without losing face. Technically, this isn't part of the argument, as rationalists account such things; but it's such an important part of the Standard Dispute that I'm including it anyway.
The argument starts shifting to focus on definitions. Whenever you feel tempted to say the words "by definition" in an argument that is not literally about pure mathematics, remember that anything which is true "by definition" is true in all possible worlds, and so observing its truth can never constrain which world you live in.
Albert deploys an argument that feels like support for the word "sound" having a particular meaning. This is a different kind of question from whether acoustic vibrations take place in a forest—but the shift usually passes unnoticed.
There's a lot of things I could be curious about in the falling-tree scenario. I could go into the forest and look at trees, or learn how to derive the wave equation for changes of air pressure, or examine the anatomy of an ear, or study the neuroanatomy of the auditory cortex. Instead of doing any of these things, I am to consult a dictionary, apparently. Why? Are the editors of the dictionary expert botanists, expert physicists, expert neuroscientists? Looking in an encyclopedia might make sense, but why a dictionary?
Dictionary editors are historians of usage, not legislators of language. Dictionary editors find words in current usage, then write down the words next to (a small part of) what people seem to mean by them. If there's more than one usage, the editors write down more than one definition.
There's quite a lot of rationality errors in the Standard Dispute. Some of them I've already covered, and some of them I've yet to cover; likewise the remedies.
But for now, I would just like to point out—in a mournful sort of way—that Albert and Barry seem to agree on virtually every question of what is actually going on inside the forest, and yet it doesn't seem to generate any feeling of agreement.
Arguing about definitions is a garden path; people wouldn't go down the path if they saw at the outset where it led. If you asked Albert (Barry) why he's still arguing, he'd probably say something like: "Barry (Albert) is trying to sneak in his own definition of 'sound', the scurvey scoundrel, to support his ridiculous point; and I'm here to defend the standard definition."
But suppose I went back in time to before the start of the argument:
This remedy doesn't destroy every dispute over categorizations. But it destroys a substantial fraction.