If it's worth saying, but not worth its own post, then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should start on Monday, and end on Sunday.
4. Unflag the two options "Notify me of new top level comments on this article" and "
An idea for a failed utopia: Scientist creates an AI designed to take actions that are maximally justifiable to humans. AI behaves as a rogue lawyer spending massive resources crafting superhumanly elegant arguments justifying the expenditure. Fortunately, there is a difference between having maximal justifiability as your highest priority and protecting the off button as your highest priority. Still a close shave, but is it worth turning off what has literally become the source of all the meaning in your life?
Too bad (or actually good) we can't actually see those superintelligent arguments. I wonder which direction they would take.
The author should perhaps describe them indirectly, i.e. not quote them (because the author is not a superintelligence, and cannot write superintelligent arguments), but describe reactions of other people after reading them. Those other people should generally become convinced about the validity of the arguments (because in-universe the arguments are superintelligent), but that can happen gradually, so in the initial phases they can b... (read more)