How can we define optimisation in a way that doesn't let us just say "it's optimising to end up like that" about any process with an end state? [..] What features or properties distinguish an optimisation process from a not-optimisation process?
Well, OK, suppose we observe process P causes a system S to transition from state S1 to S2, and observe that S1 is better than S2 for achieving goals in set G1 and S2 is better than S1 for achieving G2. Suppose we lack a definition like what you're asking for, and naively assert that P is an optimizing process for all goals G2. So, for example, we assert that gravity is an optimization process for collecting water in my basement, among other things.
Which, as you say, is unsatisfying.
But what happens next?
If we apply P to S', and observe it causes a transition from S'1 to S'2, we are no longer able to say quite so readily that P optimizes for G2. Assuming we can talk about goals in a consistent way between systems, then it seems more natural to say that P optimizes for the intersection of G2 and G'2.
If we observe the behavior of P across a wide range of systems, and we discover that the intersection of the goals optimized for by P is a fairly narrow target Gn, eventually we reach a point where if a new system Sx comes along in state Sx1, and we know that achievable state Sx2 is better for achieving Gn, we can confidently predict that P will cause an Sx1->Sx2 transition in Sx even if we don't know what the mechanism of that transition might be.
It seems relatively clear to me that once we've reached that point, we have excellent grounds for calling P an optimization process for Gn.
This may not be a necessary condition -- indeed, I suspect it isn't -- but it seems sufficient.
Would you agree?
I'm not sure that this does the job, but I might be misunderstanding:
Today's post, Aiming at the Target was originally published on 26 October 2008. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Belief in Intelligence, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.