Mark_Friedenbach comments on An AI Takeover Thought Experiment - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (11)
Upvoted for the assumptions and constraints. In fact, it would have been better if you had just left it at that. Maybe consider splitting that into a separate post?
Too much reasoning about "FOOM" AI here on LW is just plain fantastical. You just can't win if you assume the opponent is God. If you think that an evil genius AGI could live on the z80 processor in your calculator while plotting world domination, then we're all screwed and only the hail-Mary of a provably safe from first prinicples FAI design can save us, something which we don't even know is possible or not.
If on the other hand we live in the physical world and the UFAI must operate according to the fundamental principles of physics and computation as we understand them, that's a very different story. Suddenly much more mundane but practical solutions have a fighting chance, such as auditable oracle-AI. And as you demonstrate, the failure/breakout modes become situations which are predictable and with proper planning, preventable to a high degree of assurance.
I applaud you for breaking the mold. I'll let others critique the specifics.