Once thing going on here is that later discussion will be focused on recursively self-improving autonomous systems. We would like to know, for instance, whether and when software will be able to program other useful software.

I am not ready to claim that spirituality is irrelevant to the discussion. The difference between factual knowledge, calculation speed and "wisdom" does seem relevant, as others have pointed out earlier in the thread.

However, we'll have to re-frame questions about spiritual issues in some way to bring it in...

Suppose one aspires to become a Bodhisatva-a being who is capable of entering into an unfettered celestial existence, but who instead stays behind to help other people and sentient beings find their way. (If we are a bit less ambitious, perhaps we might, depending on our tradition, aspire to become a tzadik, a marja, or saintly but not a supernatural savior).

Among other things, the Bodhisatvas have removed themselves from ego and forms of physical desire. They are spiritually superior and ready to move outside of the cycle of reincarnation as lower beings. Yet, when they take the form of a human, they would eat as an instrumental goal toward accomplishing their end.

The Bodhisatva's kind of spiritual superiority does seem to differ from what Bostrom and the rest of us call "Superintelligence." It may, however, relate in some ways...

Just a thought experiment...

For example, a peculiar version of a Bodhisatva might be said to have a utility function-in fact a marvelous, selfless utility function. Also, it will succeed at bringing more people out of the darkness if it engages in recursive self-improvement of its capabilities.

One aspires to become a Buddha or a Bodhisatva, and the claim is made by some that they, others who they know, or historical figures have reached this condition. For the most part, however, this goal is aspirational in nature and works to improve people's behavior toward one another, and toward other sentient beings, in this life.

Based on what we now know about neurophysiology, the human brain is going to have a lot of trouble reaching any form of true enlightenment on its own. Emotion and desire re-surface even in the most virtuous or contemplative among us. Seemingly, we are never going to be truly saintly for the rest of our lives, although perhaps we can manage to have some pretty virtuous days interspersed among our failures.

I am not ready to make a proposal, but if enlightenment or spiritual purification is our goal, we might have to resort to some kind of augmentation to make it to the next step in that direction. Perhaps a lot of meditation or ethical contemplation is one approach, perhaps we may think of others...

Strictly an augmentation of intelligence, however, is not necessarily going to bring about this outcome.