Optimization power is the source of the danger, not agency. Agents merely wield optimality to achieve their goals.
Agency is orthogonal to optimization power
@All: It seems we agreethat optimality, when pursued blindly, is about extreme optimization that can lead to dangerous outcomes.
Could it be that we are overlooking the potential for a (superintelligent) system to prioritize what matters more—the effectiveness of a decision—rather than simply optimizing for a single goal? 🤔
Fundamentally, the story was about the failure cases of trying to make capable systems that don't share your values safe by preventing specific means by which its problem solving capabilities express themselves in scary ways. This is different to what you are getting at here, which is having those systems actually operationally share your values. A well aligned system, in the traditional ‘Friendly AI’ sense of alignment, simply won't make the choices that the one in the story did.
Thank you 🙏 @mesaoptimizer for the summary!
@All: It seems we agree that optimality, when pursued blindly, is about extreme optimization that can lead to dangerous outcomes.
Could it be that we are overlooking the potential for a (superintelligent) system to prioritize what matters more—the effectiveness of a decision—rather than simply optimizing for a single goal? 🤔
For example, optimizing too mu... (read more)