You're right. I should have put computational bounds on this 'closure'.

Yeah, I almost added a caveat about the physicalist thing probably not being your view. But it was my interpretation.

Your clarification does make more sense. I do still feel like there's some reference class gerrymandering with the "you, a mind with understanding and agency" because if you select for people who have already accumulated the steel beams, the probability does seem pretty high that they will be able to construct the bridge. Obviously this isn't a very crucial nit to pick: the important part of the analogy is the part where if you're trying to construct a bridge when trigonometry hasn't been invented, you'll face some trouble.

The important question is: how adequate are existing ideas wrt the problem of constructing ASI?

In some sense we both agree that current humans don't understand what they're doing. My ASI-soon picture is somewhat analogous to an architect simply throwing so many steel beams at the problem that they create a pile tall enough to poke out of the water so that you can, technically, drive across it (with no guarantee of safety). 

However, you don't believe we know enough to get even that far (by 2030). To you it is perhaps more closely analogous to trying to construct a bridge without having even an intuitive understanding of gravity.

Well, overconfident/underconfident is always only meaningful relative to some baseline, so if you strongly think (say) 0.001% is the right level of confidence, then 1% is high relative to that.

The various numbers I've stated during this debate are 60%, 50%, and 30%, so none of them are high by your meaning. Does that really mean you aren't arguing against my positions? (This was not my previous impression.)

I recall it as part of our (unrecorded) conversation, but I could be misremembering. Given your reaction I think I was probably misremembering. Sorry for the error!

So, to be clear, what is the probability someone else could state such that you would have "something to say about it" (ie, some kind of argument against it)? Your own probability being 0.5% - 1% isn't inconsistent with what I said (if you'd have something to say about any probability above your own), but where would you actually put that cutoff? 5%? 10%?

I guess it depends on what "a priori" is taken to mean (and also what "bridges" is taken to mean). If "a priori" includes reasoning from your own existence, then (depending on "bridge") it seems like bridges were never "far off" while humans were around. (Simple bridges being easy to construct & commonly useful.) 

I don't think there is a single correct "a priori" (or if there is, it's hard to know about), so I think it is easy to move work between this step and the next step in Tsvi's argument (which is about the a posteriori view) by shifting perspectives on what is prior vs evidence. This creates a risk of shifting things around to quietly exclude the sort of reasoning I'm doing from either the prior or the evidence.

The language Tsvi is using wrt the prior suggests a very physicalist, entropy-centric prior, EG "steel beams don't spontaneously form themselves into bridges" -- the sort of prior which doesn't expect to be on a planet with intelligent life. Fair enough, so far as it goes. It does seem like bridges are a long way off from this prior perspective. However, Tsvi is using this as an intuition pump to suggest that the priors of ASI are very low, so it seems worth pointing out that the priors of just about everything we commonly have today are very low by this prior. Simply put, this prior needs a lot of updating on a lot of stuff, before it is ready to predict the modern world. It doesn't make sense to ONLY update this prior on evidence that pattern-matches to "evidence that ASI is coming soon" in the obvious sense. First you have to find a good way to update it on being on a world with intelligent life & being a few centuries after an industrial revolution and a few decades into a computing revolution. This is hard to do from a purely physicalist type of perspective, because the physical probability of ASI under these circumstances is really hard to know; it doesn't account for our uncertainty about how things will unfold & how these things work in general. (We could know the configuration of every physical particle on Earth & still only be marginally less uncertain about ASI timelines, since we can't just run the simulation forward.)

I can't strongly defend my framing of this as a critique of step 2.1 as opposed to step 3, since there isn't a good objective stance on what should go in the prior vs the posterior. 

Numbers? What does "high confidence" mean here? IIRC from our non-text discussions, Tsvi considers anything above 1% by end-of-year 2030 to be "high confidence in short timelines" of the sort he would have something to say about. (But not the level of strong disagreement he's expressing in our written dialogue until something like 5-10% iirc.) What numbers would you "only argue against"?

It seems to me like the improvement in learning needed for what Gwern describes has little to do with "continual" and is more like "better learning" (better generalization, generalization from less examples).

Straining the analogy, the mole-hunters get stronger and faster each time they whack a mole (because the AI gets stronger). My claim is that it isn't so implausible that this process could asymptote soon, even if the mole-mother (the latent generator) doesn't get uncovered (until very late in the process, anyway).

This is highly disanalogous to the AI safety case, where playing whack-a-mole carries a very high risk of doom, so the hunt for the mole-mother is clearly important.

In the AI safety case, making the mistake of going after a baby mole instead of the mole-mother is a critical error.

In the AI capabilities case, you can hunt for baby moles and look for patterns and learn and discover the mole-mother that way.

A frontier-lab safety researcher myopically focusing on whacking baby moles is bad news for safety in a way that a frontier-lab capabilities researcher myopically focusing on whacking baby moles isn't such bad news for capabilities.

"The mole generator is basically X" seems somewhat at odds with the view Mateusz is expressing here, which seems more along the lines "LLM researchers are focusing on moles and ignoring where the moles are coming from" (the source of the moles being difficult to see).

I took it as obvious that this sort of thing wouldn't meet Tsvi's bar. AlphaEvolve seems quite unsurprising to me. We have seen other examples of using LLMs to guide program search. Tsvi and I do have disagreements about how far that sort of thing can take us, but I don't think AlphaEvolve provides clear evidence on that question. Of course LLMs can concentrate the probability mass moderately well, improving brute-force search. Not clear how far that can take us.