Thanks to Ozzie Gooen for reviewing this post. Introduction The Parable of the Predict-O-Matic is a short story which considers a forecasting system which is ostensibly set-up to maximize accuracy, and which ends up interfering with the world in ways not intended. In the original story, some of these problems were: * Fixed point problems / self-fulfilling prophecies > “Its answers will shape events. If it says stocks will rise, they'll rise. If it says stocks will fall, then fall they will. Many people will vote based on its predictions.” * Nudge towards legibility and predictability > “You keep thinking of the line from Orwell's 1984 about the boot stamping on the human face forever, except it isn't because of politics, or spite, or some ugly feature of human nature, it's because a boot stamping on a face forever is a nice reliable outcome which minimizes prediction error.” * Markets for entropy, which can be thought of as the opposite of the previous problem. In a prediction market, a market participant who could actively change an outcome has an incentive to first make a big bet for an unlikely outcome, and then actively make it come to pass. > “Suppose you have a prediction market that's working well. It makes good forecasts, and has enough money in it that people want to participate if they know significant information. Anything you can do to shake things up, you've got a big incentive to do. Assassination is just one example. You could flood the streets with jelly beans. If you run a large company, you could make bad decisions and run it into the ground, while betting against it -- that's basically why we need rules against insider trading, even though we'd like the market to reflect insider information."  * Unwanted agency (as opposed to tool AI behavior) > “You understand what you are. It isn't quite right to say you are the Predict-O-Matic. You are a large cluster of connections which thinks strategically. You generate useful information, and