One might expect self-improving systems to be highly unpredictable because the properties of the current version might change in the next version. Our analysis will instead show that self-improvement acts to create predictable regularities. It builds on the intellectual foundations of microeconomics, the science of preference and choice in the face of uncertainty. The basic theory was created by John von Neumann and Oskar Morgenstern in 1944 for situations with objective uncertainty and was later extended by Savage and Anscombe and Aumann to situations with subjective uncertainty. Our analysis shows that while the preferences of self-improving systems will depend on their origins, they will act on those preferences in predictable ways. Repeated self-improvement brings intelligent agents closer to an ideal that economists sometimes call “Homo Economicus”. Ironically, human behavior is not well described by this ideal and the field of “behavioral economics” has emerged in recent years to study how humans actually behave. The classical economic theory is much more applicable to self-improving systems because they will discover and eliminate their own irrationalities in ways that humans cannot.

Steve Omohundro, "The Nature of Self-Improving Artificial Intelligence" 2007