I don't know why you believe it,

So I wrote:

I think the threshold of brainpower where you can start making meaningful progress on the technical problem of AGI alignment is significantly higher than the threshold where you can start making meaningful progress toward AGI.

Simply put, it's a harder problem. More specifically, it's got significantly worse feedback signals: it's easier to tell when / on what tasks your performance is and is not going up, compared to telling when you've made a thing that will continue pursuing XYZ as it gets much smarter. You can also tell because progress in capabilities seems to accelerate given more resources, but that is (according to me) barely true or not true in alignment, so far.

My own experience (which I don't expect you to update much on, but this is part of why I believe these things) is that I'm really smart and as far as I can tell, I'm too dumb to even really get started (cf. https://tsvibt.blogspot.com/2023/09/a-hermeneutic-net-for-agency.html). I've worked with people who are smarter than I am, and they also are AFAICT totally failing to address the problem. (To be clear, I definitely don't think it's "just about being smart"; but I do think there's some threshold effect.) It's hard to even stay focused on the problem for the years that it apparently takes to work through wrong preconceptions, bad ideas, etc., and you (or rather, I, and ~everyone I've directly worked with) apparently have to do that in order to understand the problem.

This mechanism seems weak to me.

We probably disagree a little. I'd bring up a few points. E.g. I'd point out that you and I reason about X-risk better than them in large part due to the fact that we pay attention to people who are smarter than us and who are good at / ahead of the curve on reasoning about X-risk. E.g. I'd point out that more intelligence leads to more intellectual slack (meaning, if you can think faster, a given argument becomes less costly in time (though maybe not in opportunity) to come to understand). E.g. I'd point out that wisdom (in this sense: https://www.lesswrong.com/posts/fzKfzXWEBaENJXDGP/what-is-wisdom-1) is bottlenecked on considering and integrating many possibilities, which is a difficult cognitive task.

But, I agree it's not that strong a reason.

Realistically I do not think there is any level of genetic modification at which humans can match the pace of ASI.

Yeah I agree, but that's not the relevant threshold here. More like, can humanity feasibly get smart enough soon enough to be making a bunch of faster progress in material conditions, such that justifications of the form "AGI would give us much faster progress in material conditions" lose most of their (perhaps only apparent) force? I think probably we can.

I just didn't find the given reasons convincing.

All of them, or just these two?

Oh, IDK. Thermonuclear weapons and viral human infectiousness gain of function research? I just mean, there's probably a few things that we want everyone to know not to do; the Noahide Laws, if you will.

What I meant by general domain is that it's not overly weird in the mental moves that are relevant there, so training methods that can create something that wins IMO are probably not very different from training methods that can create things that solve many other kinds of problems.

I took you to be saying

1. math is a general domain
2. IMO is fairly hard math
3. LLMs did the IMO
4. therefore LLMs can do well in a general domain
5. therefore probably maybe LLMs are generally intelligent.

But maybe you instead meant

1. working out math problems applying known methods is a general domain

?

Anyway, "general domain" still does not make sense here. The step from 4 to 5 is not supported by this concept of "general domain" as you're applying it here.

I'm unsure whether or not we disagree, because I think our disagreement would be quantitative (probabilities, perhaps relative priorities) rather than qualitative (which considerations are coherent, plausible, relevant, etc.). My basic direct answer would be: Because effectful wise action is more difficult than effectful unwise action, and requires more ideas / thought / reflection, relatively speaking; and because generally humans want to do good things.

Maybe more intelligent humans just race to AGI faster because of coordination problems.

True, maybe.

1. I think the threshold of brainpower where you can start making meaningful progress on the technical problem of AGI alignment is significantly higher than the threshold where you can start making meaningful progress toward AGI. So more people around / above that threshold is differentially good in terms of technical progress. (It could still be bad overall, e.g. if you think there's a fixed-rate process of reflection that gets outrun faster in those worlds.)
2. I think having a lot more super smart people makes other aspects of the situation better. For example, the leaders of AGI capabilities research would be smarter--which is bad in that they make progress faster, but good in that they can consider arguments about X-risk better. Another example: it's harder to give even a plausible justification for plunging into AGI if you already have a new wave of super smart people making much faster scientific progress in general, e.g. solving diseases.
3. I think / hope that current memecrafting means that young smart people growing up will have the background sense that AGI is special and is one of the ~three things you really do not mess around with ever. BUT, if in fact it becomes clear that young super bright kids are actually, currently, growing up much more enthusiastic about AGI, that cuts against this a lot, and would significantly cut against my motivation towards reprogenetics (well, it would make me prioritize much higher stuff that affects the memesphere of newborn geniuses).

Maybe more intelligent terrorists release better bio-weapons.

AFAIK this is largely sci-fi? I mean you can find a few examples, but very very few super smart people are trying to do huge amounts of damage like that; and it's even way harder to get a large and/or very expert group together to try to do that; and without a large/expert group it's really hard to get super-high-effect things working, especially in biology, like that.

Maybe after every reasonable boost for biological human brains we still aren't smart enough to solve alignment, perhaps because it's totally intractable or perhaps because we can't really get that much smarter, and we muddle along for awhile trying to stop everyone from building AGI until something out of left field wipes us out.

Yeah, true, this is plausible; I still think our chances are significantly in those worlds, but your point stands.

in a somewhat general domain,

The way you're using this concept is poisoning your mind. Generality of a domain does imply that if you can do all the stuff in that domain, then you are generally capable (and, depending, that could imply general intelligence; e.g. if you've ruled out GLUT-like things). But if you can do half of the things in the domain and not the other half, then you have to ask whether you're exhibiting general competence in that domain, vs. competence in some sub-domain and incompetence in the general domain. Making this inference enthymemically is poisoning your mind.

Cf. https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#_We_just_need_X__intuitions :

For example, suppose that X is "self-play". One important thing about self-play is that it's an infinite source of data, provided in a sort of curriculum of increasing difficulty and complexity. Since we have the idea of self-play, and we have some examples of self-play that are successful (e.g. AlphaZero), aren't we most of the way to having the full power of self-play? And isn't the full power of self-play quite powerful, since it's how evolution made AGI? I would say "doubtful". The self-play that evolution uses (and the self-play that human children use) is much richer, containing more structural ideas, than the idea of having an agent play a game against a copy of itself.

Most instances of a category are not the most powerful, most general instances of that category. So just because we have, or will soon have, some useful instances of a category, doesn't strongly imply that we can or will soon be able to harness most of the power of stuff in that category. I'm reminded of the politician's syllogism: "We must do something. This is something. Therefore, we must do this.".

I agree with Ryan Greenblatt that precise timelines for AGI don’t matter that much in terms of actionable information, but big jumps in the chance of things going crazy within a few years can matter a lot more.

I guess this would have to quantified, but probably I want to simply respond: "Wrong."

I've been told several times by funders something like: "I don't want to fund reprogenetics; AGI is coming too soon for that to matter." Probably they are just incoherent, but if they were coherent, maybe they'd mean something like "90% chance of AGI in the next 20 years". So then it would totally matter for decision-making whether that number is 90% or 40%! I would think.

In fact,

1. One of the few robust ways out is human intelligence amplification;
2. the only method that we can have reasonable confidence can be made to work any time soon is reprogenetics (https://www.lesswrong.com/posts/jTiSWHKAtnyA723LE/overview-of-strong-human-intelligence-amplification-methods);
3. you don't have to be super confident that AGI will take 50 years to think reprogenetics is worthwhile (https://tsvibt.blogspot.com/2022/08/the-benefit-of-intervening-sooner.html); and
4. reprogenetics is funding constrained.

That is, the skills humans excel at vs. Chimps + Bonobos in experiments are social and allow the quick copying and imitating of others: overimitation, social learning, understanding others as having intentions, etc.

Yes, indeed, they copy the actions and play them through their own minds as a method of play, to continue extracting nonobvious concepts. Or at least that is my interpretation. Are you claiming that they are merely copying??