Programs will run and some of those programs will be intelligent agents, but almost nobody will run a copy of an agent to see what the agent will d

I feel like you didn't read my original post. Here is the line of thinking again, condensed:

1. Universal optimal intelligence requires simulating the universe to high fidelity (AIXI)
2. as our intelligence grows towards 1, approaching but never achieving it, we will simulate the universe in ever higher fidelity
3. intelligence is simulation

rhollerith, if I had a perfect simulation of you, I would evaluate the future evolution of your mindstate after reading millions of potential posts I could write and eventually find the optimal post that would convince you. Unfortunately, I don't have that perfect simulation, and I dont have that much computation, but it gives you an idea of its utility

If I had a perfect simulation of your chess program, then with just a few more lines of code, I have a chess program that is strictly better than yours. And this relates directly to evolution of intelligence in social creatures.