You could equally well say: “AlphaZero learns, therefore a sufficiently good imitation of AlphaZero should also learn”. Right? But let’s think about what that would entail.

AlphaZero learns via a quite complicated algorithm involving tracking the state of a Go board through self-play, and each step of the self-play involves a tree search with thousands of queries to a 30M-parameter ConvNet, and then at the end of the game a Go engine is called to see who won and then there’s a set of gradient descent steps on that 30M-parameter ConvNet. Then repeat that whole process fifty million times. And now you have a trained AlphaZero.

Now, imagine taking some generic algorithm class (say, an RNN) and training it “to imitate the process by which AlphaZero learns”. It’s just not gonna work, right? Granted, RNNs are Turing complete, so perhaps one could prove that an astronomically large RNN trained on astronomically much data can emulate (in its astronomically large number of weights) this entire detailed process of running a self-play tree search and performing gradient descent on this 30M-parameter ConvNet. …But c’mon, that’s not gonna realistically work in practice, right? (Related: §3 here.)

IMO, the only realistic way to make something that learns like AlphaZero learns is to build AlphaZero itself, or at least something awfully similar to it. I think the tree search etc. needs to be in the source code, not implicit in the learned weights of some generic algorithm class like RNNs, with no superficial relation to tree search. …But if you do that, then I would call it “reverse-engineering AlphaZero”, not “imitation learning from AlphaZero”.

By the same token, I do think it’s possible to make something that learns like a human, but I think it would require reverse-engineering human brains, not just imitation-learning from human data.

It’s possible that “imitation learning will not generalize sufficiently well OOD” is an unsolvable problem, right? (In fact, my belief is that it’s unsolvable, at least in practice, if we include “humans learning new things over the course of years” as part of the definition of what constitutes successful OOD generalization.)

But if it is unsolvable problem, it would not follow that “models will never gain the ability to generalize OOD”, nor would it follow that AGI will never be very powerful and scary.

Rather, it would follow that imitation learning models will never gain the ability to generalize OOD—but non-imitation-learning models are still allowed to generalize OOD just fine!

And it would follow that imitation learning models will not be powerful scary AGIs—but there will still be powerful scary AGIs, they just won’t be based on imitation learning.

For example, suppose that no human had ever played Go. Imitation learning would be a very doomed way to make a Go-playing AI, right? But we could still make AlphaZero, which does not involve imitation learning, and it works great.

Or better yet, suppose that no intelligent language-using animal has ever existed in the universe. Then imitation learning would be even more doomed. There’s nothing to imitate! But a well-chosen non-imitation-learning algorithm could still autonomously invent language and science and technology from scratch. We know this to be the case, because after all, that was the situation that our hominid ancestors were in.

See what I mean? Sorry if we’re talking past each other somehow.

I’m confused by your response. What do you mean by “other systems”?

The only thing I can think of is that you might be trying to say is:

• (1) AGI is possible,
• (2) …Therefore, it must be possible, somehow or other, to imitation-learning the way that humans grow and figure things out over weeks and months and years.

If that’s what you’re thinking, then I disagree with (2). Yes it’s possible to make an AGI that can learn grow and figure things out over weeks and months and years, but such an AGI algorithm need not involve any imitation learning. (And personally I expect it won’t involve imitation learning; bit more discussion in §2.3.2 here.)

“S-risk” means “risk of astronomical amounts of suffering”. Typically people are imagining crazy things like Dyson-sphere-ing every star in the local supercluster in order to create 1e40 (or whatever) person-years of unimaginable torture.

If the outcome is “merely” trillions of person-years of intense torture, then that maybe still qualifies as an s-risk. Billions, probably not. We can just call it “very very bad”. Not all very very bad things are s-risks.

Does that help clarify why I think Reward Button Alignment poses very low s-risk?

Is the following correct?

The difference between this proposal and IDA is that, in IDA, the intelligence comes from the amplification step, where multiple copies of an existing model collaborate (or one copy thinks at higher speed or whatever) to get new smarts that were not initially present in that model.

…Whereas in this proposal, the intelligence comes from some unspecified general-purpose AI training approach that can by itself make arbitrarily smart AIs. But we choose not to run that process as hard as we can to get superintelligence directly in one step. Instead, we only run that process incrementally, to make a series of AI advisors, each mildly smarter than the last.

~~

If that’s right, then (1) the alignment tax seems like it would be quite high, (2) the caveat “Inner advisors may attempt (causal or acausal) trade with outer advisors” is understating the problem—the advisors don’t necessarily even need to trade at all, they could all simply have the same misaligned goal, such that they all share a mutual interest in working together to ensure that the principal’s brain gets hacked and/or that they get let out of the box. Right?

One reason this proposal doesn’t really work for me (AFAICT) is because I’m normally thinking of continuous learning, i.e. my opinion is:

Tagline: “AGI isn’t about knowing how to do lots of things. Instead, AGI is about not knowing how to do something, and then being able to figure it out.” (see also §1 of “Sharp Left Turn” discourse: An opinionated review)

When I read your post with this mental picture, they seem to clash for various reasons.

For starters, I agree that imitation learning is (or could be) great at capturing a snapshot of a person, but I’m skeptical that it could capture the way that a person learns and figures out new things over weeks and months. I think this is borne out by LLM base models, which are trained by imitation learning, and are really quite strong in areas that humans already understand, but don’t even have the capacity for true online learning (i.e. weight edits when they figure out something new … the bit of poor-man’s online learning that they can do inside their context window is IMO not a great substitute).

If that’s the case, then as soon as you do one step of distillation-via-imitation-learning, you’ve taken a giant and unrecoverable step backwards in capabilities.

Maybe you could say, “so much the worse for LLMs, but future AI approaches will be able to imitation-learn the way that humans grow and figure things out over weeks and months and years”. If so, I’m skeptical, but we can talk about that separately.

And then another issue is that if the AIs (and humans!) are “in motion”, gaining knowledge and competence just by running longer and thinking about new domains and making new connections, then the overshooting vs undershooting issue becomes much harder. This isn’t “capabilities evaluations” as we normally think of them. For example, you can’t know how good the AI will be at cybersecurity until it’s spent a long time studying cybersecurity, and even then it might figure out something new or come up with new ideas while it’s being used as an advisor.

There was a part of the post where I wrote “I might well be screwing up the math here”, where I wasn’t sure whether to square something or not, and didn’t bother to sort it out. Anyway, I think this comment is a claim that I was doing it wrong, maybe? But that person is not very confident either, and anyway I’m not following their reasoning. Still hoping that someone will give a more definitive answer. I would love to correct the post if I’m wrong.

In the post alluded to a nice self-contained tricky math inequality problem that I am hoping someone will be nerd-sniped by. (I am rusty on my linear algebra inequalities and I don’t care enough to spend more time on it.) Here’s what I wrote:

2025-01-18: I mentioned in a couple places that it might be possible to have non-additive genetic effects that are barely noticeable in $r_{DZ}$-vs-$\frac{1}{2}{r}_{MZ}$ comparisons, but still sufficient to cause substantial Missing Heritability. The Zuk et al. 2012 paper and its supplementary information have some calculations relevant to this, I think? I only skimmed it. I’m not really sure about this one. If we assume that there’s no assortative mating, no shared environment effects, etc., then is there some formula (or maybe inequality) relating rDZ-vs-½rMZ to a numerical quantity of PGS Missing Heritability? I haven’t seen any such formula. This seems like a fun math problem—someone should figure it out or look it up, and tell me the answer!

More details: Basically, when $r_{DZ}$ is less than $\frac{1}{2}{r}_{MZ}$, then there has to be nonlinearity in the map from genomes to outcomes (leaving aside other possible causes). And if there’s nonlinearity, then the polygenic scores can’t be perfectly predictive. But I’m trying to relate those quantitatively.

Like, intuitively, if $r_{MZ}=1.000$ and $r_{DZ}=0.499$, then OK yes there’s nonlinearity, but probably not very much, so probably the polygenic score will work almost perfectly (again assuming infinite sample size etc).

…Conversely, if $r_{MZ}=1.000$ and $r_{DZ}=0.001$, then intuitively we would expect “extreme nonlinearity” and the polygenic scores should have very bad predictive power.

But are those always true, or are there pathological cases where they aren’t? That’s the math problem.

I tried this with reasoning LLMs a few months ago with the following prompt (not sure if I got it totally right!):

I have a linear algebra puzzle.

There’s a high-dimensional vector space G of genotypes.

There’s a probability distribution P within that space G, for the population.

There’s a function F : G → Real numbers, mapping genotypes to a phenotype.

There’s an "r" where we randomly and independently sample two points from P, call them X and Y, and find the (Pearson) correlation between F(X) and F((X+Y)/2).

If F is linear, I believe that r^2=0.5. But F is not necessarily linear.

Separately, we try to find a linear function G which approximates F as well as possible—i.e., the G that minimizes the average (F(X) - G(X))^2 for X sampled from P.

Let s^2 be the percent of variance in F explained by G, sampled over the P.

I'm looking for inequalities relating s^2 to r^2, ideally in both directions (one where r is related to an upper bound on s, the other a lower bound).

Commentary on that:

• The X vs (X+Y)/2 is not exactly what happens with siblings. It's similar to comparing a parent to their child—i.e., X is the genotype of the mother, Y the father, (X+Y)/2 the kid. But parent-child should be mathematically similar to sibling-sibling, since both are 50% relatedness. …Except it’s not really parent-child either, because if X has a SNP but Y doesn’t, then the child has the SNP with 50% probability, rather than having “half of that SNP”. But I figured it might amount to the same thing? But I do think you need to be randomizing over individual SNPs to formulate the problem for the actual sibling case we care about. (So really, the individuals are all binary / indicator vectors (entries are 1s and 0s), as opposed to arbitrary elements of the vector space. I’m just guessing that’s not too important to the problem.)
• I’m assuming $r_{MZ}=1$, and then "r" is $r_{DZ}$, "G" is the polygenic score, and "s" quantifies the predictive power of the polygenic score.
• I did this very quickly, there might be other mistakes in this problem formulation, and I wasn’t motivated enough to keep exploring it.

(Btw, I sent that prompt to a few AIs around January 2025, and they gave answers but I don’t think the answers were right.)

Another question I am still confused by: how does your choice of units affect what types of dimensionless quantities you discover? Why do we have Ampere as a fundamental unit instead of just M,L,T? What do I lose? What do I lose if I reduce the number of dimensions even further? Are there other units that would be worth adding under some circumstances? What makes this non-arbitrary? Why is Temperature a different unit from Energy?

It’s pretty arbitrary, I tried to explain this point via a short fictional story here.

Gaussian units only has M,L,T base units, with nothing extra for electromagnetism.

There are practical tradeoffs involved in how many units you use—basically adding units gives you more error-checking at the expense of more annoyance. See the case of radians that I discuss here.