I don't think it can completely keep track of itself-- that would take its whole capacity and then some.

I have a different view of AI (I do not know if it is better or more likely). I would see the AI as a system almost entirely devoted to keeping track of itself.

You're probably right about the trend. I've heard that lizards do a lot less processing of their sensory information than we do. It's amusing that people put in so much work through meditation to be more like lizards. This is not to deny that meditation is frequently good for people.

However, an AI using a high proportion of its resources to keep track of itself is not the same thing as it being able to keep complete track of itself.

In re the possibly over-vigilant/over reactive AI: my notion is that its ability to decide how big a threat is will be affected by its early history.

As I write this out, I see that a large part of my argument is that AIs that do not have good mental ecology with a foundation of self-monitoring and goal/drive analysis will simply die out or go insane (or go paperclip) rather than become a healthy, interesting, and useful agent. So really, I agree that mental health is critically important, I just think that it is either in place from the start, or we have an unfriendly AI on our hands.

That's where I started. We have an evolved capacity to heal. Designing the capacity to heal for a very different sort of system is going to be hard if it's possible at all.