Tim,

already the abstract reveals two flaws:

Excerpt from the abstract of the paper "Basic AI drives" by Omohundro:

This paper instead shows that intelligent systems will need to be carefully designed to prevent them from behaving in harmful ways. We identify a number of “drives” that will appear in sufficiently advanced AI systems of any design. We call them drives because they are tendencies which will be present unless explicitly counteracted.

First of all, no distinction whatever is made between "intelligent" and "sentient". I agree that mindless intelligence is problematic (and is prone to a lot of the concerns raised here).

But what about sentience? What about the moment when "the lights go on"? This is not even addressed as an issue (at least not in the Omohundro paper). And I think most people here agree that consciousness is _not_ an epiphenomenon (see Eli's Zombie Series). So we need different analysis for non-sentient intelligent systems and sentient intelligent systems.

A related point: We humans have great difficulty rewiring our hardware (and we can't change the brain architecture at all), that is why we can't easily change our goals. But self-improving AI will be able to modify it's goal functions: that plus self-consciousness sounds quite powerful, and is completely different than simple "intelligent agents" maximizing their utility functions. Also, the few instances where an AI would change their utility function mentioned in the paper are certainly not exhaustive, I found the selection quite arbitrary.

The second flaw in the little abstract above was the positing of "drives": Omohundro argues that these drives don't have to be programmed into the AI but are intrinsic to goal-driven systems.

But he neglects another premise of his: that we are talking about AIs who can change their goal functions (see above)! All bets are off now!

Additionally, he bases his derivations on microeconomic theory which is also full of assumptions which maybe won't apply to sentient agents (they certainly don't apply to humans, as Omohundro recognizes).

Drives the paper mentions are: wanting to self-improve, being rational, protecting self, preserving utility function, resource acquisition etc. These drives sound indeed very plausible, and they are in essence human drives. So this leads me to suspect that anthropomorphism is creeping in again through the backdoor, in a very subtle way (for instance through assumptions of microeconomic theory).

I see nothing of the vastness of mindspace in this paper.