= 27d567bc2d48d9f40e9ccc20ea370b15)
Stuart_Armstrong comments on Naturalism versus unbounded (or unmaximisable) utility options - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (72)
Actually, the third problem is probably the most relevant of them all - it's akin to a bounded paperclipper uncertain as to whether they've succeeded. Kind of like: "You get utility 1 for creating 1 paperclip and then turning yourself off (and 0 in all other situations)."
I still don't see how it's relevant, since I don't see a reason why we would want to create an AI with a utility function like that. The problem goes away if we remove the "and then turning yourself off" part, right? Why would we give the AI a utility function that assigns 0 utility to an outcome where we get everything we want but it never turns itself off?
The designer of that AI might have (naively?) thought this was a clever way of solving the friendliness problem. Do the thing I want, and then make sure to never do anything again. Surely that won't lead to the whole universe being tiled with paperclips, etc.
This can arise indirectly, or through design, or for a host of reasons. That was the first thought that popped into my mind; I'm sure other relevant examples can be had. We might not assign such a utility - then again, we (or someone) might, which makes it relevant.