It is intentionality as required for following an English statement 'maximize paperclips' to the spirit of it, instead of doing what the tools we can make for maximizing paperclips would do (which is very distinct from maximizing paperclips).

edit: Basically, the 'paperclip maximizer' as defined here would follow lines of reasoning like 'if I modify my sensory processing module in such a way that increases number of paperclips it outputs, that will decrease the expected number of paperclips in the real world, and so I will refrain from doing so', same for any self modification that breaks the correspondence between real world paperclips and the number with which agent represents them. The tool would not follow such line of reasoning. It is easy to see, from external perspective, if an action breaks this correspondence. Internally you don't have extra definition of real world paperclip spring out of nothingness and tell you, no this breaks the correspondence.

A more mundane example:

The Roomba cleaning robot is scarcely an agent. While running, it does not build up a model of the world; it only responds to immediate stimuli (collisions, cliff detection, etc.) and generates a range of preset behaviors, some of them random.

It has some senses about itself — it can detect a jammed wheel, and the "smarter" ones will return to dock to recharge if the battery is low, then resume cleaning. But it does not have a variable anywhere in its memory that indicates how clean it believes the room is — an explicit representation of a utility function of cleanliness, or "how well it has done at its job". It does, however, have a sensor for how dirty the carpet immediately below it is, and it will spend extra time on cleaning especially dirty patches.

Because it does not have beliefs about how clean the room is, it can't have erroneous beliefs about that either — it can't become falsely convinced that it has finished its job when it hasn't. It just keeps sweeping until it runs out of power. (We can imagine a paperclip-robot that doesn't think about paperclips; it just goes around finding wire and folding it. It cannot be satisfied, because it doesn't even have a term for "enough paperclips"!)

It is scarcely an agent. To me it seems even less "agenty" than an arbitrage daemon, but that probably has more to do with the fact that it's not designed to interact with other agents. But you can set it on the floor and push the go button, and in an hour come back to a cleaner floor. It doesn't think it's optimizing anything, but its behavior has the result of being useful for optimizing something.

Whether an entity builds up a model of the world, or is self-aware or self-protecting, is to some extent an implementation detail, which is different from the question of whether we want to live around the consequences of that entity's actions.

The agent/tool distinction is in the map, not the territory — it's a matter of adopting the intentional stance toward whatever entity we're talking about. To some extent, saying "agent" means treating the entity as a black box with a utility function printed on the outside: "the print spooler wants to send all the documents to the printer" — or "this Puppet config is trying to put the servers in such-and-so state ..."

Very good point on need for another word.

To think about it, I think what we need is understanding of enormous gap between the software we design when we have some intent in mind, and fulfilling that intent itself.

For example, if I have intent to get from point A to point B on terrain, I could build a solution consisting of 2 major parts:

• the perceiver tool that builds and updates the map of terrain
• the solver tool that minimizes some parameter over a path through this terrain (some discomfort metric, combined with the time, the risk of death, etc). [edit: please note that this terrain is not the real world terrain]

A philosopher thinking about it could think up a mover-from-point-A-to-point-B which directly implements my 'wish' to get from A to B. It will almost certainly expose me to non-survivable accelerations, or worse yet, destroy buildings on its path (because in the wish i forgot to tell not to). That is because when you employ verbal reasoning you are thinking directly starting from intent.

Alas, we do not know how to reduce intent to something that is not made of intent.

edit: that is, in the mind of philosopher, the thing is actually minimizing - or maximizing - some metric along a real world path. We don't know how to do that. We do not know how we do that. We don't even know we actually do that ourselves.

edit: We might figure out how to do that, but it is separate problem from either improvements to my first bullet point, or my second bullet point.

Other thing that I forgot to mention: the relationship between the solver and the model is inherently different than the relationship between self driving car and the world. The solver has god's eye view, that works in high level terms. The car looks through sensors. The solver can not be directly connected to the real world, or even to a reductionist detailed physics simulator (too hard to define the comfortable path when the car, too, is made of atoms).

Note that mere daemon-level tools exist that many already consider unFriendly, e.g. high-frequency trading systems.

There's the AK47 rifle: you pull trigger, the bullets come out one after another...