Is anyone else bothered by the apparent dichotomy into "natural" and "unnatural" categories? There's room for a matter of degree here - and that's probably all the argument ultimately needs - but the terminology is binary.
Suppose there is at least one utility function, and an agent who has it. Then all categories the agent ultimately finds useful are ipso facto "constructed [...] in an observable, testable way from categories themselves simple". Namely, from the categories "this agent" and "utility function". And (I think - objections welcome) you don't have to share this agent's values to construct it.
Is there any physical object or property that absolutely must be used (not merely recognizable in principle, but actually used) in the ontology of all conceivable rational beings, no matter their values? I doubt it. If that is the standard of "natural category" then all categories are unnatural. But if it is enough that the category be recognizable in principle, then (see the paragraph above) all categories are natural.
Today's post, Unnatural Categories was originally published on 24 August 2008. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Mirrors and Paintings, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.