Russell, population increased, and demographics with higher rates of accidents expanded their population share during that period.

For simple nervous systems. For plants, seed production can be made into an accurate signal of genetic quality.

"though possibly brainless organisms have clever ways to do sexual selection" Response to body symmetry, biologically expensive mating displays (hampered by mutation load), etc.

This reminds me of the fact that humans have an evolved tendency for pure time preference, and indeed hyperbolic time preference, rather than accurate modeling of future rewards.

"- at least one king somewhere who took power by lies and assassination, and then ruled wisely and well - but I can't actually name a case off the top of my head." If ruling wisely and well means increased GDP or quality of life (e.g. attracting and accepting immigrants), we might nominate Singapore's Lee Kuan Yew, Chile's Pinochet, Gorbachev, Ian Smith, Deng Xiaoping, etc.

Also, every major democratic political leader lies abundantly to obtain office, as it's a necessity to actually persuade the voters. So Bill Clinton, Jean Chretien, Winston Churchill should qualify for at least half of your list of villainy.

"On the lighter note, how about an EY experiment? Do you think there is absolutely no way to convince Eliezer to release the original AI experiment logs? Would you bet a $20 that you can? Would a strong AI be able to? ;)"

Presumably you could just donate $10,000 to SIAI or EY personally for his time participating, with the payment independent of the outcome of the experiment (otherwise the large payment biases the outcome, and EY established his record with the $10-$20 stakes).

"I have a feeling that if the loser of the AI Box experiment were forced to pay thousands of dollars, you would find yourself losing more often. Still it is interesting to consider whether this extra condition takes the experiment closer to what is supposed to be simulated or the opposite."

Uh, your 'hypothesis' was already tested and discussed towards the end of the post!