= 27d567bc2d48d9f40e9ccc20ea370b15)
Stuart_Armstrong comments on Satisficers want to become maximisers - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (67)
The expected utility of option 1 is higher than the threshold for the satisficer, so it could just as easily pick 1) as 2); it'll be indifferent between the two choices, and will need some sort of tie-breaker.
But inasmuch as it will want to want one over the other, it will want to want 2 which is guaranteed to continue to satisfice over 1 which has only a 90% chance of continuing to satisfice, so it should not want to become a maximizer.
So that's actually the "bounded utility" definition, which Stuart says he isn't using. It does seem more intuitive though... I think you can get a paradox out of Stuart's definition, actually, which should not be surprising, since it isn't a utility-maximizer.
A satisficer is not motivated to continue to satisfice. It is motivated to take an action that is a satisficing action, and 1) and 2) are equally satisficing.
I know what you're trying to do, I think. I tried to produce a "continuously satisficing agent" or "future satisficing agent", but couldn't get it to work out.
Surey option 1 has a 10% chance of failing to satisfy.
Option 1) already satisfies. Taking option 1) brings the expected utility up above the threshold, so the satisficer is done.
If you add the extra requirement that the AI must never let the expected utility fall below the threshold in future, then the AI will simply blind itself or turn itself off, once the satisficing level is reached; then its expected utility will never fall, as no extra information ever arrives.
Sorry - a failure to reread the question on my part :-(