This concept of optimization seems to me difficult or impossible to formalize in a useful manner. I see it as a sometimes-useful thought exercise at best, to be replaced by something more quantitative as soon as possible.
How does this measure of optimization power compare the abilities of the engineers who designed the Corolla, vs the oil refinery that produced its fuel? I'm not asking about the engineers who designed the refinery, just the finished refinery itself.
In general, I think there is something missing about the complexity of the optimization power. Eliezer speaks of powerful optimizers being scary things, but I think he is drawing unwarranted conclusions based on an unvoiced assumption that powerful optimizers are intelligent problem solvers. I see nothing scary about a refrigeration system capable of meaningfully cooling Antarctica, despite it being a powerful entropy-reducer.
More generally: I think that a strong understanding of what constitutes optimization is necessary but not sufficient to a discussion of global impacts of AI. Furthermore, I think that this article constitutes a non-technical explanation of what Eliezer is calling an optimization process. If the goal is simply to replace the term "intelligence" with something that non-experts will be less likely to quibble over, then I think the article has partially succeeded. If the goal is to make productive use of the concept among people who weren't getting distracted by definitions of intelligence, then I think a technical explanation is in order.
An oil refinery is slightly intelligent as an optimization process. Presumably it has sensors that can detect the heaviness of the crude input, its sulfur content, and perhaps other impurities and process it accordingly so that valuable petroleum fractions are maximized. But I don't know that much about refineries; it may be the case that human operators madly turn valves and pull levers while watching spinning dials to make our gasoline and the refinery is just a big dumb tool.
The question to ask about the refinery as an optimization process is to what ...
Today's post, Optimization was originally published on 13 September 2008. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Psychic Powers, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.