The concept of a resource can be defined within ordinary decision theory: something is a resource iff it can be used towards multiple goals and spending it on one goal makes the resource unavailable for spending on a different goal. In other words, it is a resource iff spending it has a nontrivial opportunity cost. Immediately we have two implications: whether or not something is a resource to you depends on your ultimate goal and (2) diving by resources spent is useful only for intermediate goals: it never makes sense to care how efficiently an agent uses its resources to achieve its ultimate goal or to satisfy its entire system of terminal values.

I could teach any deciduous tree to play grandmaster chess if only I could communicate with it. (Well, not the really dumb ones.)

Allbright slipped in. (Mine was a reply to Matthew C.)

Well, OK, but your anti-reductionism is still wrong.

Hal asks good questions. I advise always minding the distinction between personal success (personal economic security, reputation, esteem among high-status people) and global success (increasing the probability of a good explosion of engineered intelligence) and suggest that the pernicious self-deception (and blind spots) stem from unconscious awareness of the need for personal success. I.e., the need for global success does not tend to distort a person's perceptions like (awareness of) the need for personal success does.

Like I keep on saying, I have a different moral framework than most, but I come to the same conclusions on unethical means to allegedly ethical ends.

Because I don't even know what I want from that future.

Well, I hope you will stick around, MichaelG. Most people around here IMHO are too quickly satisifed with answers to questions about what sorts of terminal values properly apply even if the world changes drastically. A feeling of confusion about the question is your friend IMHO. Extreme scepticism of the popular answers is also your friend.

Eliezer writes, "In general, beliefs require evidence."

To which Peter replies, "In general? Which beliefs don't?"

Normative beliefs (beliefs about what should be) don't, IMHO. What would count as evidence for or against a normative belief?

I, too, am down with the mammals. I don't mind seeing whole galaxies transformed into clouds of superintelligent matter and energy and dedicated to mammalian happiness and mammalian preference.

I have yet to see a satisfactory definition of fun applicable to minds (or agents or optimization processes) very different from mammalian (or vertebrate) minds. And I suspect I will not be seeing one any time soon.