When evaluating the outputs of an algorithm, we have to consider the interestingness of the outputs under various counterfactuals. Otherwise, all game-theoretic agents are equivalent to either Cooperation Rock or Defection Rock, and all probabilities are either 0 or 1. And once you're specifying outputs as a complete truth-table, you're effectively specifying the algorithm.
To give a concrete example, I consider the server running lesswrong.com to be a banal mind, because it performs little computation of interest itself, even though when messages are sent from it to my computer those messages often contain very interesting ideas.
I see your point. I was presuming a human mind w/ the typical range of experiences available to it.
This is our monthly thread for collecting these little gems and pearls of wisdom, rationality-related quotes you've seen recently, or had stored in your quotesfile for ages, and which might be handy to link to in one of our discussions.