Nick_Beckstead comments on Tiling Agents for Self-Modifying AI (OPFAI #2) - Less Wrong

55 Post author: Eliezer_Yudkowsky 06 June 2013 08:24PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (260)

You are viewing a single comment's thread. Show more comments above.

Comment author: timtyler 10 June 2013 10:04:35AM 0 points [-]

This means that statistical testing methods (e.g. an evolutionary algorithm's evaluation of average fitness on a set of test problems) are not suitable for self-modifications which can potentially induce catastrophic failure (e.g. of parts of code that can affect the representation or interpretation of the goals).

I'm confused by this sentence. There are many statistical testing methods that output what are essentially proofs; e.g. statements of the form "probability of a failure existing is at most 10^(-100)". [...]

It seems as though they would involve a huge number of trials.

"Evolutionary" algorithms aren't typically used to change fitness functions anyway. They are more usually associated with building representations of the world to make predictions with. This complaint would seem to only apply to a few "artificial life" models - in which all parts of the system are up for grabs.