The apparent aim of OpenAI (making AGI, even though we don't know how to do so without killing everyone) is evil.

Sharing our technical concerns about these abstract risks isn't enough. We also have to morally stigmatize

I'm with you up until here; this isn't just a technical debate, it's a moral and social and political conflict with high stakes, and good and bad actions.

the specific groups of people imposing these risks on all of us.

To be really nitpicky, I technically agree with this as stated: we should stigmatize groups as such, e.g. "the AGI capabilities research community" is evil.

We need the moral courage to label other people evil when they're doing evil things.

Oops, this is partially but importantly WRONG. From Braχot 10a:

With regard to the statement of Rabbi Yehuda, son of Rabbi Shimon ben Pazi, that David did not say Halleluya until he saw the downfall of the wicked, the Gemara relates: There were these hooligans in Rabbi Meir’s neighborhood who caused him a great deal of anguish. Rabbi Meir prayed for God to have mercy on them, that they should die. Rabbi Meir’s wife, Berurya, said to him: What is your thinking? On what basis do you pray for the death of these hooligans? Do you base yourself on the verse, as it is written: “Let sins cease from the land” (Psalms 104:35), which you interpret to mean that the world would be better if the wicked were destroyed? But is it written, let sinners cease?” Let sins cease, is written. One should pray for an end to their transgressions, not for the demise of the transgressors themselves.

Not everyone who is doing evil things is evil. Some people are evil. You should hate no more than necessary, but not less than that. You should hate evil, and hate evildoers if necessary, but not if not necessary.

Schmidhuber? Evil. Sutton? Evil. Larry Page? Evil. If, after reflection, you endorse omnicide, you're evil. Altman? Evil and probably a sociopath.

Up-and-coming research star at an AI lab? Might be evil, might not be. Doing something evil? Yes. Is evil? Maybe, it depends.

Essentializing someone by calling them evil is an escalation of a conflict. You're closing off lines of communication and gradual change. You're polarizing things: it's harder for that one person to make gradual moves in belief space and social space and life-narrative space, and it's harder for groups to have group negotiations. Sometimes escalation is good and difficult and necessary, but sometimes escalation is really bad! Doing a more complicated subtle thing with more complicated boundaries is more difficult. And more brave, if we're debating bravery here.

So:

Good:

The work you're doing is evil.

Good:

The goal of this company is among the evilest possible goals ever.

Good:

If you ignore the world-class experts saying that your work might kill everyone, then you are being a disgusting creep and will be responsible for killing everyone.

Bad:

You're a creep / evil / bad person.

Sidenote:

Because too many rationalists, EAs, tech enthusiasts, LessWrong people, etc still see those AI guys as 'in our tribe', based on sharing certain traits we hold dear

I agree that this is an improper motivation for treating some actions with kid gloves, which will lead to incorrect action; and that this is some of what's actually happening.

As a category suggestion, I'd suggest including "obstructions/barriers" to making safe/aligned AGI, in the sense of the natural proofs barrier for P vs. NP. They aren't going to be nearly as crisp in AGI safety/alignment as in math, but I still think it'd be valuable to present fundamental problems, that seem like they must be solved, as key parts of the field--or in other words, sets of approaches (such as ignoring those problems) that (most likely) won't work.

Some examples are in "AGI Ruin: A List of Lethalities", though not directly stated as barriers, e.g. "you only get one shot at making a safe fully-superhuman AI" could be stated as "solutions that require observing the behavior of a fully-superhuman AI don't work [given such and such background conclusions]".

(I also give examples in "The fraught voyage of aligned novelty", though maybe not presented in a way you'd like. E.g. "if the AI is genuinely smarter than you, it's thinking in ways that are alien to you" --> "solutions that require the AI to only think in ways that you already understand don't work".)

I mean, they're great as search engines or code-snippet writers (basically, search engine for standard functions). If someone thinks that gippities know stuff or can think or write well, that could be brainrotting.

Well, like, if a company tried out some new robotics thing in one warehouse at a small scale in Q1, then in Q2 and Q3 scaled it up to most of that warehouse, and then in Q4 started work applying the same thing in another warehouse, and announced plans to apply to many warehouses, I think it'd be pretty fair to call this lasting adoption (of robotics, not LLMs, unless the robots use LLMs). On the other hand if they were stuck at the "small scale work trying to make a maybe-scalable PoC", that doesn't seem like lasting adoption, yet.

Judging this sort of thing would be a whole bunch of work, but it seems possible to do. (Of course, we can just wait.)

Echoing robo's comment:

Modest lasting corporate adoption

Has there been such adoption? Your remark

Every single corporation in the world is trying to adopt AI into their products. Even extremely slow-moving industries are rushing to adopt AI.

is about attempts to adopt, not lasting adoption. Of course, we can't make "lasting adoption" mean "adopted for 5 years" if we're trying to evaluate the prediction right now. But are you saying that there's lots of adoption that seems probably/plausibly lasting, just by eyeballing it? My vague is impression is no, but I'm curious if the answer is yes or somewhat.

(TBC I don't have a particularly strong prediction or retrodiction about adoption of AI in general in industry, or LLMs specifically (which is what I think Marcus's predictions are about). At a guess I'd expect robotics to continue steadily rising in applications; I'd expect LLM use in lots of "grunt information work" contexts; and some niche strong applications like language learning; but not sure what else to expect.)

Interesting. (I don't immediately see where you're going with that, so sounds like I have something to learn!)

In practical terms, it should be feasible sooner to do small amounts of personality nudging using what data we already have, operating on linear variance. Later on we'll have more data, better psychometrics, and better ways of modeling some of the nonlinear effects. My current take is that it's better to use the weaker versions while the strong ones are infeasible (https://www.lesswrong.com/posts/rdbqmyohYJwwxyeEt/genomic-emancipation#Genomic_engineering_overhang), but not sure.

(Had a good conversation--will think more about the request to ban research into certain personality traits (the traits that an oppressive regime could force upon its populace to enforce long-term subjugation).)

That seems like part of the picture, but far from all of it. Manufactured stone tools have been around for well over 2 million years. That's the sort of thing you do when you already have a significant amount of "hold weeks-long goal in mind long and strong enough that you put in a couple day's effort towards it" (or something like that). Another example is Richard Alexander's hypothesis: warfare --> strong pressure toward cognitive mechanisms for group-goal-construction. Neither of these are mainly about programmability (though the latter is maybe somewhat). I don't think we see "random self-preserving terminal goals installed exogenously", I think we see goals being self-constructed and then flung into long-termness.

Humans are (weak) evidence for the instrumental utility for mind-designers to design terminal-goal-construction mechanisms.

Evolution couldn't directly encode IGF into humans. So what was it supposed to do? One answer would be to make us vibe-machines: You bop around, you satisfy your immediate needs, you hang out with people, etc. And that is sort of what you are. But also there are the Unreasonable, who think and plan long-term, who investigate secrets, who build things for 10 or 100 or 1000 years--why? Maybe it's because having terminal-like goals (here meaning, aims that are fairly fixed and fairly ambitious) is so useful that you want to have them anyway even if you can't make them be the right ones. Instead you build machines to guess / make up terminal goals (https://tsvibt.blogspot.com/2022/11/do-humans-derive-values-from-fictitious.html).