Stuart_Armstrong comments on Siren worlds and the perils of over-optimised search - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (411)
Because the first supposes a powerful AI, while the second supposes an excellent evaluation process (essentially a value alignment problem solved).
Your post motivated this in part, but it's a more general issue with optimisation processes and searches.
Neither the search nor the evaluation presupposes an AI when a hypothetical process is used as the definition of "good."