When there's no clear winner, the winner can't take all.

https://en.wikipedia.org/wiki/Winner-take-all_in_action_selection

Honesty is the best policy. <3 Mom

Regarding the assumptions for a strong AI that will cease cooperating and pursue its own values, the scenario presented assumed that 1) L will have the ability to subvert S's control, and then 4a) S must be unaware that L has that ability. That is (if I understand this correctly), the ability for L to subvert S is undetected. But if it's the case that we assume 1), then perhaps S should instead operate under the assumption that L already has the ability (and perhaps even knowledge) to subvert, but S doesn't know what exactly this ability is, and if S knew, not the knowledge of how it may be applied (which we may assume L knows).

In other words, I imagine the scenario would be more like: L has the ability to subvert S, and knows how to use it. S doesn't know the ability, and even if it does, doesn't know how it's used. L (for the sake of argument) knows that S is unaware, and cannot stop it even if S tried. Now, this seems like a pretty bad scenario. However, here, because S knows that it doesn't know, S might spend more effort in devising ways of dealing with this lack of knowledge (e.g. getting L to tell it about abilities it learned, perhaps rewarding with a heart) and the potential desire of L to follow through with taking control of the heart-machine (e.g. considering potential wireheading induces negative reward).

EDIT: Perhaps I'm not as clear on why L should try to deceive S in the first place. It seems there should be a better way of dealing with deception on behalf of L other than resetting/large-negative-reward when regarded as a 'potential threat' to S but no desire to actually threaten S, which, as you mention would just lead to pressure toward better concealment at the detriment of S, rather than pressure toward what S really wants, which is alignment of L's goals with its own.

I’m not sure if I can answer this because I don’t understand what good mental posture is or even what good physical posture is, for that matter. Can you please confirm if my understanding, below, of what these are is correct?

Well, I can do that for physical posture. I don't know if I can do that for mental posture, but I'll try.

I think in broad strokes your description of what good physical posture is sounds right to me. I wouldn't tie it to gravity specifically; I think it makes sense to talk about good posture in a space station. But maybe replace "gravity" with "surrounding forces" and I think it's basically right.

I'd sum it up by saying that posture is a description of how efficient the arrangement of your body is at transmitting forces. A curled-forward upper back is terrible at transmitting forces between your arms and your hips when compared to a more straight upper back, so I'm inclined to call a straighter upper back "better posture".

There seem to be a few default physical positions that are about as good at general force transmission as a human body can get. Those positions are what I call "good posture".

I personally like Todd Hargrove's breakdown of what good physical posture does for you, though I think the one you linked to is reasonably good too.

I honestly don't know what good mental posture is. I'm gesturing at an intuition based on a bunch of my own experiences and how they resonate with my experience with physical posture.

For instance, if someone trips and knocks into me, I'm much more likely than untrained people to just keep my ground. If I get knocked to the side, though, I'm likely to keep my torso moving as basically one piece, which makes it really easy for me to recover my balance. It's really notable to me when my postural habits slip up and someone knocks into me because I feel like I'm flopping around through the air as I fall over, and relative to my baseline it feels physically dangerous to me.

I notice something that feels analogous in my mind. If someone turned to me and said "Val, go get me coffee", I'm likely to get agitated in a way that reminds me of getting bumped into while having a floppy core. I can pause and use some CBT-like techniques to "catch" myself by, say, noticing that the person probably didn't mean to offend me - but this seems more analogous to grabbing a hold of something nearby to keep myself from falling than it does having a solid core. Instead, I notice that there's some kind of way I can choose to orient myself to the situation and to myself that lets me notice my annoyance at being ordered around and basically not get "knocked over". In that mental "position", I feel like the CBT-like thoughts are much more solid mental "movements", more like taking a stable step to keep my balance than grabbing at whatever is in reach as I fall.

If I had to guess at a definition of mental posture, I would try by analogy to the "efficient at transmitting forces" description of physical posture above. Maybe something like saying it's a description of how efficiently one's patterns of directing attention let one mentally navigate one's environment. The thing is, I haven't really worked out how to capture the intuition I have that being unbothered by being offended is a function of good mental posture whereas being really fast at mathematical computations isn't.

Good mental posture, on the other hand, seems to involve taking certain perspectives or entering certain frames of mind that are conducive to the achievement of your current goals.

From the article you linked:

I've been using a term for changing the overall quality of my thoughts and feelings to something more conducive to accomplishing my immediate goal. I call it "adopting a mental posture".

Right, though I think this might be too abstract to be useful. I could also say that physical posture involves taking certain physical positions that are conductive to the achievement of your current goals. I think that's accurate, but I don't think it quite captures the details that are useful in the analogical mapping.

If we view thought activation in a similar way to how we view muscle activation in regards to physical posture, then we can think of good mental posture as the undertaking of certain perspectives or mindsets that inhibit unhelpful thoughts and induce helpful thoughts, where what is helpful depends on the current task at hand.

That's a neat take on it. I feel like it's missing something; e.g., in the anxious/avoidant trap in attachment theory, the problem isn't just the thoughts, but also something about the way that emotional anticipations seem "off balance". Just changing thought patterns a la CBT doesn't seem to reach deeply enough to fix attachment wounds in my experience. But the basic idea is neat. It reminds me of the idea of avoiding wasted mental movements (e.g., thoughts like "I don't know if I can handle this!" when you have to are utterly wasted in nearly all possible futures where you succeed, so it seems worthwhile to just not bother with that thought).

(By the way, I'd warn not to take attachment theory too seriously. It has a lot of psychobabble in it. I do think it does a really nice job of describing some experiences people have, and the "anxious/avoidant trap" is a great example. But the page I just linked to includes a bunch of Freudian guesswork about why avoidants attract anxious folk and vice versa, and that's basically without any empirical support as far as I know.)

A good mental posture will be:

• Relaxed…
• Fluid…
• Efficient and synchronous…
• Adaptable…
• Normally in a broad perspective…

I like this breakdown. It resonates with me. There are two details I'd want to tweak based on my limited personal experience playing with this stuff:

• While I really like the framing of good mental posture in terms of avoiding what I (due to some conversations with Eliezer) call "wasted mental movements", I'm really hesitant to name keeping one's mind unwaveringly on a task a virtue. I'm reminded of how mathematicians classically need to distract themselves after being stuck on a problem for a long while. There seems to be something very good that comes out of (1) priming the subconscious mind with a lot of potential updates and then (2) getting the conscious mind out of the way so that the subconscious mind can do some kind of magical processing in the background. (The same thing seems to happen with physical skills, by the way: I keep finding that taking weeks-long breaks from aikido sometimes boosts my skill quite a lot more than training over similar time periods does.)
• I intuit that the "adaptable" point isn't quite right. I'm inclined to think that being adaptable is a little bit like being able to sidestep or block an attack: you really need good posture to do it well, but there's still a skill that needs to be trained. But this is based just on how the analogy between mental and physical postures maps in my head.

Overall I like your description though. It gives me the impression that you're looking at basically the same thing I am.

… psychical posture…

I thought this was a delightful use of language! I had been using "mental arts" to act as a verbal and visual mirror for "martial arts", but hadn't noticed this mapping between "physical" and "psychical". Thank you for this!

PS. Physical posture and mental posture may be entwined. People who are in pain or tired often have bad posture.

Yep. I'm a little surprised by how strong the analogy is in my inner experience, which makes me wonder if the mapping is somehow a natural one.

I'm reminded of Todd Hargrove's suggestion that the brain is for movement and his follow-up analysis of the idea.