= 27d567bc2d48d9f40e9ccc20ea370b15)
Stuart_Armstrong comments on Creating a satisficer - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (26)
Here, I start with a bounded and easy to reach u (that's a first step in the process), so "u = finite-u". This is still not safe for a maximiser (usual argument about "being sure" and squeezing ever more tiny amounts of expected utility from optimising the universe). Then the whole system is supposed to produce S(u) rather than M(u). This is achieved by having M(εu+v) allow it, when M(εu+v) expects (counterfactually) to optimise the universe, and would see any optimisation by S(u) as getting in the way (or, if it could co-opt these otimisations, then this is something that M(u-v) would not want it to do).
Technically, you might not need to bound u so sharply - it's possible that the antagonistic setup will produce a S(u) that is equivalent to S(finite-u) even it u is unbounded (via the reduced impact effect of the interactions between the two maximisers). But it seems sensible to add the extra precaution of starting with a bounded u.
Augh! "I" and "you" are not in the list of agents we're discussing. Who starts with a bounded u, and how does that impact the decision of what S will be offered by the M(u-v) agent?
u is bounded. All agents start with a bounded u. The "I" is me (Stuart), saying "start this project with a bounded u, as that seems to have less possible failures than a general u".
With an unbounded u, the M(u-v) agent might be tempted to build a u maximiser (or something like that), counting on M(εu+v) getting a lot of value out of it, and so accepting it.
Basically, for the setup to work, M(εu+v) must get most of its expected value from maximising v (and hence want almost all resources available for v maximising). "bounded u with easily attainable bound" means that M(εu+v) will accept some use of resources by S(u) to increase u, but not very much.