I have a problem understanding why a utility function would ever "stick" to an AI, to actually become something that it wants to keep pursuing.
I think that's one of MIRI's research problems. Designing an self-modifying AI that doesn't change it's utility function isn't trival.
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should be posted in Discussion, and not Main.
4. Open Threads should start on Monday, and end on Sunday.