My take is that AGI learning human preferences/values/needs/desires/goals/etc. is a necessary but not sufficient condition for achieving alignment.

If AI learns our preferences base on our behaviors it’s going to learn a lot of “bad” things like lying, stealing and cheating and other much worse things.

First of all, it's important to note that just because the AGI learns that humans engage in certain behaviors, that by itself does not imply that it will have any inclination to engage in those behaviors itself. Secondly, there is a difference between behaviors and preferences. Lying, theft, and murder are all natural things that members of our species engage in, yes, but they are typically only ever used as a means of fulfilling actual preferences.

Preferences, values, etc. are what motivate our actions. Behaviors are reinforced that tend to satisfy our preferences, stimulate pleasure, alleviate pain, move us toward our goals and away from our antigoals. It's those preferences, those motivators of action rather than the actions themselves, that I believe Stuart Russell is talking about.

An aligned AGI will necessarily have to take our preferences into account, whether to help us achieve them more intelligently than we could on our own or to steer us toward actions that minimize conflict with others' preferences. It would ideally take on our values (or the coherent extrapolation or aggregation of them) as its own in some way without necessarily taking on our behavioral policies as its own (though of course it would need to learn about those too).

An unaligned AGI would almost certainly learn human values as well, once it's past a certain level of intelligence, but it would lack the empathy to care about them beyond the need to plan around them as it pursues its own goals.

Biological values might be greed, selfishness and competition while social values might be trust, altruism and cooperation.

I believe that this "biological = selfish, social = cooperative" dichotomy is wrong. It is a popular mistake to make, because it provides legitimacy to all kinds of political regimes, by allowing them to take credit for everything good that humans living in them do. It also allows one to express "edgy" opinions about human nature.

But if homo sapiens actually had no biological foundations for trust, altruism, and cooperation, then... it would be extremely difficult for our societies to instill such values in humans; and most likely we wouldn't even try, because we simply wouldn't think about such things as desirable. The very idea that fundamentally uncooperative humans somehow decided to cooperate at creating a society that brainwashes humans into being capable of cooperation is... somewhat self-contradictory.

(The usual argument is that it would make sense, even for a perfectly selfish asshole, to brainwash other people into becoming cooperative altruists. The problem with this argument is that the hypothetical perfectly selfish wannabe social engineer couldn't accomplish such project alone. And when many people start cooperating on brainwashing the next generation, what makes even more sense for a perfectly selfish asshole is to... shirk their duty at creating the utopia, or even try to somehow profit from undermining this communal effort.)

Instead, I think we have dozens of instincts which under certain circumstances nudge us towards more or less cooperation. Perhaps in the ancient environment they were balanced in the way that maximized survival (sometimes by cooperation with others, sometimes at their expense), but currently the environment changes too fast for humans to adapt...

I agree with your main point: it is not obvious how training an AI on human preferences (which are sometimes "good" and sometimes "evil") would help us achieve the goal (separating the "good" from "evil" from "neutral").

If something is capable of fulfilling human preferences in its actions, and you can convince it to do so, you're already most of the way to getting it to do things humans will judge as positive. Then you only need to specify which preferences are to be considered good in an equally compelling manner. This is obviously a matter of much debate, but it's an arena we know a lot about operating in. We teach children these things all the time.