An AI that grows ever more effective at optimizing its futures will not suddenly begin to question its goals.
Oh, great. So MIRI can disband and we can cross one item off the existential-risk list....
some equivalent of "always satisfy Programmer's expressed desires"
Well, that idea has been explored on LW. Quite extensively, in fact.
Point of MIRI is making sure the goals are set up right, yeah? Like, the whole "AI is smart enough to fix its defective goals" is something we make fun of. No ghost in the machine, etc.
Whatever outcome of perfect goal set is (if MIRI's AI is, in fact, the one that takes over), will presumably include human ability to override in case of failure.
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should be posted in Discussion, and not Main.
4. Open Threads should start on Monday, and end on Sunday.