Honesty is the best policy. <3 Mom
Regarding the assumptions for a strong AI that will cease cooperating and pursue its own values, the scenario presented assumed that 1) L will have the ability to subvert S's control, and then 4a) S must be unaware that L has that ability. That is (if I understand this correctly), the ability for L to subvert S is undetected. But if it's the case that we assume 1), then perhaps S should instead operate under the assumption that L already has the ability (and perhaps even knowledge) to subvert, but S doesn't know what ex...
This may be a little basic, but if you're already going with the "efficient at transmitting forces" idea for physical posture, then I think a good analogy in terms of mental posture would simply be "efficient at processing information" (of which being rational is a pretty useful method, just like keeping your spine, er, non-kyphotic is useful).
This is much more concise than:
how efficiently one's patterns of directing attention let one mentally navigate one's environment
while at the same time relatively neutral with respect to the ac...
When there's no clear winner, the winner can't take all.
https://en.wikipedia.org/wiki/Winner-take-all_in_action_selection
Happens all the time in decision theory & reinforcement learning: the average of many good plans is often a bad plan, and a bad plan followed to the end is often both more rewarding & informative than switching at every timestep between many good plans. Any kind of multi-modality or need for extended plans (eg due to upfront costs/investments) will do it, and exploration is quite difficult - just taking the argmax or adding some randomness to action choices is not nearly enough, you need "deep exploration" (as Osband likes to call it) to follow a s
... (read more)