All of elifland's Comments + Replies

The timelines model didn't get nearly as many reviews as the scenario. We shared the timelines writeup with all of the people who we shared the later drafts of the scenario with, but I think almost none of them looked at the timelines writeup. 

We also asked a few people to specifically review the timelines forecasts, most notably a few FutureSearch forecasters who we then added as a final author. However, we mainly wanted them to estimate the parameter values and didn't specifically ask them for feedback on the underlying modeling choices (though they... (read more)

GideonF*118

I suspect part of the reasons for the quality-weighted criticism of the timelines rather than the scenario:

  • If it is the case that you put far less effort into the timelines model than the scenario, then the timelines model is probably just worse - some of the more obvious mistakes that titotal points out probably don't have analogies in your scenario, so its just easier to criticise the timelines model, as there is more to criticise there
  • In many ways, the timelines model is pretty key to the headline claim of your scenario. The other parts (scenario and ta
... (read more)
elifland302

I'll say various facts as best as I can recall and allow you and others to decide how bad/deceptive the time horizon prediction graph was.

  • The prediction on the graph was formed by extrapolating a superexponential with a 15% decay. This was set to roughly get SC at the right time, based on an estimate for what time horizon is needed for SC that is similar to my median in the timelines forecast. This is essentially a simplified version of our time horizon extension model that doesn't account for AI R&D automation. Or another way to view this is that we c
... (read more)
TurnTrout175

Thanks, I appreciate your comments.

This is essentially a simplified version of our time horizon extension model that doesn't account for AI R&D automation. Or another way to view this is that we crudely accounted for AI R&D automation by raising the decay.

Why did you simplify the model for a graph? You could have plotted a trajectory to begin with, instead of making a bespoke simplification. Is it because you wanted to "represent roughly the trajectory that happens in AI 2027"? I get that AI 2027 is a story, but why not use your real model to sampl... (read more)

9TurnTrout
Yes, I think this would have been quite good.
elifland2711

I'm kind of split about this critique, since the forecast did end up as good propaganda if nothing else. But I do now feel that the marketing around it was kind of misleading, and we probably care about maintaining good epistemics here or something.

I'm interested in you expanding on which parts of the marketing were misleading. Here are some quick more specific thoughts:

  1. Overall AI 2027 comms
    1. In our website frontpage, I think we were pretty careful not to overclaim. We say that the forecast is our "best guess", "informed by trend extrapolations, wargames, ..
... (read more)
9Thane Ruthenis
Mostly this part, I think: Like, yes, the supplementary materials definitely represent a huge amount of legitimate research that went into this. But the forecasts are "informed by" this research, rather than being directly derived from it, and the pointing-at kind of conveys the latter vibe. Glad you get where I'm coming from; I wasn't wholly sure how legitimate my complaints were. I agree that this part is tricky, hence my being hesitant about fielding this critique at all. Persuasiveness isn't something we should outright ignore, especially with something as high-profile as this. But also, the lack of such a disclaimer opens you up to takedowns such as titotal's, and if one of those becomes high-profile (which it already might have?), that'd potentially hurt the persuasiveness more than a clear statement would have. There's presumably some sort of way to have your cake and eat it too here; to correctly communicate how the forecast was generated, but in terms that wouldn't lead to it being dismissed by people at large. Yeah, sorry, I was being unnecessarily hyperbolic there.
elifland*14234

Thanks titotal for taking the time to dig deep into our model and write up your thoughts, it's much appreciated. This comment speaks for Daniel Kokotajlo and me, not necessarily any of the other authors on the timelines forecast or AI 2027. It addresses most but not all of titotal’s post.

Overall view: titotal pointed out a few mistakes and communication issues which we will mostly fix. We are therefore going to give titotal a $500 bounty to represent our appreciation.  However, we continue to disagree on the core points regarding whether the model’s t... (read more)

titotal132

I'm leaving the same comment here and in reply to daniel on my blog. 

First, thank you for engaging in good faith and rewarding deep critique. Hopefully this dialogue will help people understand the disagreements over AI development and modelling better, so they can make their own judgements. 

I think I’ll hold off on replying to most of the points there, and make my judgement after Eli does an in-depth writeup of the new model. However, I did see that there was more argumentation over the superexponential curve, so I’ll try out some more critiques... (read more)

6Tom Davidson
Re intermediate speed ups : a simple fix   You currently have the pace of total progress growing exponentially as AI improves. And this leads the bad back-predictions that the pace of progress used to be much slower.   I think your back predictions would be fine if you said that total progress = human-driven progress + AI-driven progress, and then had only the AI part grow exponentially.   Then in the back prediction the AI part would rapidly shrink but the human part would remain.

So I'm kind of not very satisfied with this defence.

Not-very-charitably put, my impression now is that all the technical details in the forecast were free parameters fine-tuned to support the authors' intuitions[1], when they weren't outright ignored. Now, I also gather that those intuitions were themselves supported by playing around with said technical models, and there's something to be said about doing the math, then burning the math and going with your gut. I'm not saying the forecast should be completely dismissed because of that.

... But "the authors... (read more)

Sorry for the late reply.

If we divide the inventing-ASI task into (A) “thinking about and writing algorithms” versus (B) “testing algorithms”, in the world of today there’s a clean division of labor where the humans do (A) and the computers do (B). But in your imagined October 2027 world, there’s fungibility between how much compute is being used on (A) versus (B). I guess I should interpret your “330K superhuman AI researcher copies thinking at 57x human speed” as what would happen if the compute hypothetically all went towards (A), none towards (B)? And

... (read more)

Tagging @romeo who did our security forecast.

Oh I misunderstood you sorry. I think the form should have post-2023, not sure about the website because it adds complexity and I'm skeptical that it's common that people are importantly confused by it as is.

elifland192

Whew, a critique that our takeoff should be faster for a change, as opposed to slower.

Fun fact: AI-2027 estimates that getting to ASI might take the equivalent of a 100-person team of top human AI research talent working for tens of thousands of years.

(Calculation details: For example, in October 2027 of the AI-2027 modal scenario, they have “330K superhuman AI researcher copies thinking at 57x human speed”, which is 1.6 million person-years of research in that month alone. And that’s mostly going towards inventing ASI, I think. Did I get that ri

... (read more)
7Steven Byrnes
Thanks, that’s very helpful! If we divide the inventing-ASI task into (A) “thinking about and writing algorithms” versus (B) “testing algorithms”, in the world of today there’s a clean division of labor where the humans do (A) and the computers do (B). But in your imagined October 2027 world, there’s fungibility between how much compute is being used on (A) versus (B). I guess I should interpret your “330K superhuman AI researcher copies thinking at 57x human speed” as what would happen if the compute hypothetically all went towards (A), none towards (B)? And really there’s gonna be some division of compute between (A) and (B), such that the amount of (A) is less than I claimed? …Or how are you thinking about that? Right, but I’m positing a discontinuity between current AI and the next paradigm, and I was talking about the gap between when AI-of-that-next-paradigm is importantly useful versus when it’s ASI. For example, AI-of-that-next-paradigm might arguably already exist today but where it’s missing key pieces such that it barely works on toy models in obscure arxiv papers. Or here’s a more concrete example: Take the “RL agent” line of AI research (AlphaZero, MuZero, stuff like that), which is quite different from LLMs (e.g. “training environment” rather than “training data”, and there’s nothing quite like self-supervised pretraining (see here)). This line of research has led to great results on board games and videogames, but it’s more-or-less economically useless, and certainly useless for alignment research, societal resilience, capabilities research, etc. If it turns out that this line of research is actually much closer to how future ASI will work at a nuts-and-bolts level than LLMs are (for the sake of argument), then we have not yet crossed the “AI-of-that-next-paradigm is importantly useful” threshold in my sense. If it helps, here’s a draft paragraph from that (hopefully) forthcoming post: Next: Well, even if you have an ML training plan that will yi

I think it's not worth getting into this too much more as I don't feel strongly about the exact 1.05x, but I feel compelled to note a few quick things:

  1. I'm not sure exactly what you mean by eating a smaller penalty but I think the labor->progress penalty is quite large
  2. The right way to think about 1.05x vs. 1.2x is not a 75% reduction, but instead what is the exponent for which 1.05^n=1.2
  3. Remember the 2022 vs. 2023 difference, though my guess is that the responses wouldn't have been that sensitive to this

Also one more thing I'd like to pre-register: people... (read more)

2plex
(feel free to not go any deeper, appreciate you having engaged as much as you have!) 1. Yup, was just saying my first-pass guess would have been a less large labour->progress penalty. I do defer here fairly thoroughly. hmm, seems true if you're expecting the people to not have applied a correction already, but less true if they are already making a correction and you're estimating how wrong their correction is? And yup, agree with that preregistration on all counts.

Yup feel free to make that change, sounds good

2plex
Clarification: 1. Change to the form to ask about without AI assistence? 2. Change to the website to refer to "AI provides the following speedups from a baseline of 2022/3 AI:"? (I don't have write access) (assuming 1 for now, will revert if incorrect)

No AI help seems harder to compare to since it's longer ago, it seems easiest to think of something close to today as the baseline when thinking about future speedups. Also for timelines/takeoff modeling it's a bit nicer to set the baseline to be more recent (looks like for those we again confusingly allowed 2024 AIs in the baseline as well rather than just 2023. Perhaps I should have standardized that with the side panel).

2plex
I think this risks people underappreciating how much progress is being sped up, my naive read of the UI was the numbers were based on "no AI" and I'd bet most readers would think the same at a glance. Changing the text from "AI provides the following speedups:" to "AI provides the following speedups from a baseline of 2022/3 AI:" would resolve this (I would guess common) misreading.

I'm not sure what the exact process was, tbh my guess is that they were estimated mostly independently but likely sanity checked with the survey to some extent in mind. It seems like they line up about right, given the 2022 vs. 2023 difference, the intuition regarding underadjusting for labor->progress, and giving weight to our own views as well rather than just the survey, given that we've thought more about this than survey takers (while of course they have the advantage of currently doing frontier AI research).

I'd make less of an adjustment if we ask... (read more)

2plex
Alright, my first pass guess would have been algorithmic progress seems like the kind of thing that eats a much smaller penalty than most forms org-level progress, not none but not a 75% reduction, and not likely more than a 50% reduction, but you guys have the track record. Cool, added a nudge to the last question.

Yup, seems good

2plex
Okay, switched. I'm curious about why you didn't set the baseline to "no AI help", especially if you expect pre-2024 AI to be mostly useless, as that seems like a cleaner comparison than asking people to remember how good old AIs were?

I also realized that I believe that confusingly the survey asks about speedup vs. no post-2022 AIs, while I believe the scenario side panel is for no post-2023 AIs, which should make the side panel numbers lower, unclear exactly how much given 2023 AIs weren't particularly useful.

2plex
I can switch the number to 2023?

Look at the question I mentioned above about the current productivity multiplier

2plex
Oh, yup, missed that optional question in my ctrl-f. Thanks!

I think a copy would be best, thanks!

2plex
This survey looks like it's asking something different? It's asking about human range, no mention of speed-up from AI.

I think the survey is an overestimate for the reason I gave above, I think this stuff is subtle and researchers are likely to underestimate the decrease from labor speedup to progress speedup, especially in this sort of survey where it didn't involve discussing with them verbally. Based on their responses to other questions in the survey seems like at least 2 people didn't understand the difference between labor and overall progress/productivity.

Here is the survey: https://forms.gle/6GUbPR159ftBQcVF6. The question we're discussing is: "[optional] What... (read more)

2plex
Wait, actually, I want to double click on this. What was the process that caused you to transform the number you got from the survey (1.2x) to the number on the website (1.05x)? Is there a question that could be asked which would not require a correction? Or which would have a pre-registered correction?[1] 1. ^ Bonus: Was this one pre-registered?
2plex
That resolves the inconsistency. I do worry that dropping a 20% speed-up to a 5% one, especially if post hoc, might cover up some important signal, but I'm sure you've put dramatically more cycles into thinking about this than me. Thanks for the survey, would it make sense to just pass this form around so the numbers go to the same place and you'll check, or should I make a copy and send results if I get them?

You mean the median would be at least 1.33x rather than the previous 1.2x? Sounds about right so don't feel the need to bet against. Also I'm not planning on doing a follow-up survey but would be excited for others to.

2plex
Your website lists  * April 2025 as 1.13x * August 2025 as 1.21x * December 2025 as 1.30x * December 2024 as 1.05x (which seems contradicted by your survey, if the replies were in November) If you think today's number is ~1.33x we're ~7 months ahead of schedule vs the listed forecast, unless I'm really misreading something. Also, re: "would be excited for others to.", is the survey public or easy to share if someone wanted to use the same questions? And I'd bet 1:4 for the current number is actually >1.5x, if that's more interesting. You've updated me to not have that as the main expectation, but still seems pretty plausible. Obviously depends on someone rerunning the survey, and reasonable that you've got your hands full with other things right now.

Most of the responses were in Nov.

2plex
That seems like stale data, given how these graphs look. Even with the updates you caused, I'm happy to offer an even odds token bet ($100?) that a rerun of a similar survey would give significantly higher average (at least +0.2 over the predicted 1.13x, or about the AI you expect in Dec 2025). I'd be even more happy if the question asked about the researcher's own productivity, as that seems like something they'd have better vision of, but would be pretty noisy with small sample so reasonable to stick with original question.
elifland*82

This was from Nov 2024 to Mar 2025 so fairly recent. I think the transition to faster was mostly due to the transition to reasoning models and perhaps the beginnings of increased generalization from shorter to longer time horizons.

Edit: the responses are from between Nov 2024 and Mar 2025. Responses are in increasing order: 1.05-1.1, 1.15, 1.2, 1.3, 2. The lowest one is the most recent but is from a former not current frontier AI researcher.

2plex
The switch to reasoning models does line up well, probably more cleanly. Moved that to main hypothesis, thanks. Having some later responses makes it less likely they missed the change, curious if the other responses were closer to Dec or March. I would guess the not-current-researcher one being excluded probably makes sense? The datapoint from me is not exactly 2x on this, but 'most of an approximately 2x', so would need revisiting with the exact question before it could be directly included, and I'd imagine you'd want the source. I still have some weight on higher research boost from AI than your model is expecting, due to other lines of evidence, but not putting quite as much weight on it. 
elifland156

We did do a survey in late 2024 of 4 frontier AI researchers who estimated the speedup was about 1.1-1.2x. This is for their whole company, not themselves.


This also matches the vibe I’ve gotten when talking to other researchers, I’d guess they’re more likely to be overestimating than underestimating the effect due to not adjusting enough for my next point. Keep in mind that the multiplier is for overall research progress rather than a speedup on researchers’ labor, this lowers the multiplier by a bunch because compute/data are also inputs to progress.

5Daniel Kokotajlo
That said, we just talked to another frontier AI company researcher who said the speedup was 2x. I disagree with them but it's a data point at least.
2plex
Okay, that updates me some. I'm curious about what your alternate guess about the transition to the faster exponential on the METR long-horizon tasks, and whether you expect that to hold up or be not actually tracking something important? (also please note that via me you now also have a very recent datapoint of a frontier AI researcher who thinks the METR speed-up of ~2x was mostly due to AI accelerating research) Edit: How late in 2024? Because the trendline was only just starting to become apparent even right near the end and was almost invisible a couple months early, it's pretty plausible to me that if you re-ran that survey now you would get different results. The researchers inside will have had a sense somewhat before releases, but also lag on updating is real.

If the trend isn’t inherently superexponential and continues at 7 month doubling times by default, it does seem hard to get to AGI within a few years. If it’s 4 months, IIRC in my timelines model it’s still usually after 2027 but it can be close because of intermediate AI R&D speedups depending on how big you think the gaps between benchmarks and the real world. I’d have to go back and look if we want a more precise answer. If you add error bars around the 4 month time, that increases the chance of AGI soon ofc.

If you treat the shift from 7 to 4 month ... (read more)

  • It underrates the difficulty of automating the job of a researcher. Real world work environments are messy and contain lots of detail that are neglected in an abstraction purely focused on writing code and reasoning about the results of experiments. As a result, we shouldn’t expect automating AI R&D to be much easier than automating remote work in general.

I basically agree. The reason I expect AI R&D automation to happen before the rest of remote work isn't because I think it's fundamentally much easier, but because (a) companies will try to automa... (read more)

elifland106

I still think full automation of remote work in 10 years is plausible, because it’s what we would predict if we straightforwardly extrapolate current rates of revenue growth and assume no slowdown. However, I would only give this outcome around 30% chance.

In an important sense I feel like Ege and I are not actually far off here. I'm at more like 65-70% on this. I think this overall recommends quite similar actions. Perhaps we have a more important disagreement regarding something like P(AGI within 3 years), for which I'm at approx. 25-30% and Ege might be ... (read more)

2Noosphere89
I do think the difference between an AGI timeline median of 5 years and one of 20 years does matter, because politics starts affecting whether we get AGI way more if we have to wait 20 years instead of 5, and serial alignment agendas make more sense if we assume a timeline of 20 years is a reasonable median. Also, he argues against very fast takeoffs/software only singularity in the case for multi-decade timelines post.

Ah right, my bad, I was confused. This is right except that these estimates aren't software-only, they include recent levels of compute scaling.

1snewman
Thanks, I've edited the post to note this.

Those estimates do start at RE-Bench, but these are all estimates for how long things would take given the "default" pace of progress, rather than the actual calendar time required. Adding them together ends up with a result that doesn't take into account speedup from AI R&D automation or the slowdown in compute and algorithmic labor growth after 2028.

1snewman
Sure – I was presenting these as "human-only, software-only" estimates: So it doesn't seem like there's a problem here?

I think that usually in AI safety lingo people use timelines to mean time to AGI and takeoff to mean something like the speed of progression after AGI.

Thanks for bringing this up, I hadn't seen this paper. 

Before deciding how much time to spend on this I'm trying to understand how much this matters, and am having trouble interpreting your Wolfram Alpha plot. Can you ELI12? I tried having Claude plot our lognormal doubling time distribution against an inverse Gaussian with equivalent mean and variance and it looks very similar, but of course Claude could be messing something up.

3Peter Johnson
You can ignore for now since I need to work through whether this is still true depending on how we view the source of uncertainty in doubling time. Edit: this explanation is correct afaict and worth looking into. The parameters for the second log-normal (doubling time at RE-Bench saturation, 10th: 0.5 mo., 90th: 18 mo.) when made equivalent for an inverse gaussian by matching mean and variance (approx. InverseGaussian[7.97413, 1.315]) are implausible. The linked paper highlights that to be representing doubling processes reasonably, the ratio of first to second parameter ought to be << 2/ln(2) (or << (1/(2ln(2)^2))). The failure to match that indicates that the "size hypothesis" of any of the growth processes is violated, indicating that the distribution is no longer modeling uncertainty around such a process. Ok, so that's too many functions, what does it mean? In general, it means that our "uncertainty" is actually the main driver of fast timelines now rather than reflecting a lack of knowledge in any way. The distribution is so stretched that the mode and median are wildly smaller than the mean entirely due to the possibility that a random unknown event causes foom, unrelated to the estimated "growth rate" of the process. It's like cranking up a noise term on a stock market model and being surprised that some companies are estimated to go to the moon tomorrow, then claiming it is due to estimating those stocks as potentially having huge upsides. There is not a good solution that keeps the model intact (and the same basic issue is that the model is working in domains that are outcomes like time and frequency rather than inputs like time horizons, compute, or effective compute). If one were to use the same mean and scale up the second parameter, the left side of the pdf would collapse, and the mode and median would jump much higher resulting in a much later estimate of SC. That doesn't mean that's how to fix the model, but it does indicate fast timelines are inc

but "this variable doesn't matter to outcomes" is not a valid critique w.r.t. things like "what are current capabilities/time horizon"

Where did I say it isn't a valid critique? I've said both over text and verbally that the behavior in cases where superexponentiality is true isn't ideal (which makes a bigger difference in the time horizon extension model than benchmarks and gaps).

Perhaps you are saying I said it's invalid because I also said that it can be compensated some by lowering the p_superexponential at lower time horizons? Saying this doesn't imply... (read more)

At the very least this concedes that the estimates are not based on trend-extrapolation and are conjecture.

Yes, as I told you verbally, I will edit the relevant expandable to make this more clear. I agree that the way it's presented currently is poor.

Here are two charts demonstrating that small changes in estimates of current R&D contribution and changes in R&D speedup change the model massively in the absence of a singularity.

These are great, this parameter is indeed at least a bit more important that I expected. I will make this more clear in the... (read more)

First, my argument is not: we had limited time to do this, therefore it's fine for us to not include whatever factors we want.

My argument is: we had limited time despite putting lots of work into this, because it's a very ambitiously scoped endeavor. Adding uncertainty to the percent of progress that is software wouldn't have changed the qualitative takeaways, therefore it's not ideal but okay for us to present the model without that uncertainty (shifting the median estimate a lower number my have have, I'll separately reply to your comment on that; we sho... (read more)

The basic arguments are that (a) becoming fully superhuman at something which involves long-horizon agency across a diverse range of situations seems like it requires agency skills that will transfer pretty well to other domains (b) once AIs have superhuman data efficiency, they can pick up whatever domain knowledge they need for new tasks very quickly.

I agree we didn't justify it thoroughly in our supplement, the reason it's not justified more is because we didn't get around to it.

As a prerequisite, it will be necessary to enumerate the set of activities that are necessary for "AI R&D"

As I think you're aware, Epoch took a decent stab at this IMO here. I also spent a bunch of time thinking about all the sub-tasks involved in AI R&D early on in the scenario development. Tbh, I don't feel like it was a great use of time compared to thinking at a higher level, but perhaps I was doing it poorly or am underestimating its usefulness.

What is the profile of acceleration across all tasks relating to AI R&D? What percentage of tasks are getting accelerated by 1.1x, 1.5x, 2x?

A late 2024 n=4 survey of frontier AI researchers estimated a median of a 1.15x AI R&D progress multiplier relative to no post-2022 AIs. I'd like to see bigger surveys here but FWIW my best guess is that we're already at a ~1.1x progress multiplier.

elifland*80

Readers are likely familiar with Hofstadter's Law:

It always takes longer than you expect, even when you take into account Hofstadter's Law.

It's a good law. There's a reason it exists in many forms (see also the Programmer's Credo[9], the 90-90 rule, Murphy's Law, etc.) It is difficult to anticipate all of the complexity and potential difficulties of a project in advance, and on average this contributes to things taking longer than expected. Constructing ASI will be an extremely complex project, and the AI 2027 attempt to break it down into a fairly simple

... (read more)

While current AI models and tools are demonstrating substantial value in the real world, there is nevertheless a notorious gap between benchmark scores ("Ph.D level" and beyond) and real-world applicability. It strikes me as highly plausible that this reflects one or more as-yet-poorly-characterized chasms that may be difficult to cross.

You probably know this, but for onlookers the magnitude of these chasms are discussed in our timelines forecast, method 2.

The authors address this objection, but the counterargument strikes me as flawed. Here is the key paragraph:

To see why this is conceptually mistaken, consider a theoretical AI with very superhuman experiment selection capabilities but sub-human experiment implementation skills. Even if automation didn’t speed up implementation of AI experiments at all and implementation started as 50% of researchers’ time, if automation led to much better experiments being chosen, a >2x AI R&D progress multiplier could be achieved.

In essence, this is saying that if

... (read more)

Inevitably, some of these activities will be harder to automate than others, delaying the overall timeline. It seems difficult to route around this problem. For instance, if it turns out to be difficult to evaluate the quality of model outputs for fuzzy / subjective tasks, it's not clear how an R&D organization (regardless of how much or little automation it has incorporated) could rapidly improve model capabilities on those tasks, regardless of how much progress is being made in other areas.

One reason I expect less jaggeed progress than you is that my... (read more)

As a minor point of feedback, I'd suggest adding a bit of material near the top of the timelines and/or takeoff forecasts, clarifying the range of activities meant to be included in "superhuman coder" and "superhuman AI researcher", e.g. listing some activities that are and are not in scope. I was startled to see Ryan say "my sense is that an SAR has to be better than humans at basically everything except vision"; I would never have guessed that was the intended interpretation.)

This is fair. To the extent we have chosen what activities to include, it's sup... (read more)

Ok yeah, seems like this is just a wording issue and we're on the same page.

  • SAR has to dominate all human researchers, which must include whatever task would otherwise bottleneck.

This, and the same description for the other milestones, aren't completely right; it's possible that there are some activities on which the SAR is worse. But it can't be many activities and it can't be much worse at them, given that the SAR needs to overall be doing the job of the best human researcher 30x faster.

2ryan_greenblatt
I think my description is consistent with "some activities on which the SAR is worse" as long as these aren't bottlenecking and it is overall dominating human researchers (as in, adding human researchers is negligable value). But whatever, you're the author here. Maybe "Superhuman coder has to dominate all research engineers at all pure research engineering tasks" is too strong though.

I simply find it impossible to accept this concatenation of intuitive leaps as sufficient evidence to update very far.

Seems like this should depend on how you form your current views on timelines/takeoff. The reason I put a bunch of stock in our forecasts for informing my personal views is that I think, while very flawed, they seem better than any previous piece of evidence or intuition I was including. But probably we just disagree on how to weigh different forms of evidence.

The upshot is that I find it difficult to accept the AI 2027 model as strong evidence for short timelines

Here you're using "short timelines" to refer to our takeoff model I think, which is what you spend most of the post discussing? Seems a bit confusing if so, and you also do this in a few other places.

1snewman
Correct. Am I wrong in thinking that it's usual to use the word "timelines" to refer to the entire arc of AI progress, including both the periods covered in the "Timelines Forecast" and "Takeoff Forecast"? But, since this is all in the context of AI 2027 I should have clarified.

Superintelligent AI researcher → artificial superintelligence: 95 years, explained here. I honestly cannot interpret the argument here (the wording is informal and I find it to be confusing), but it includes components such as "Achieving ASI in all cognitive tasks rather than just AI R&D: About half of an SARSIAR jump".

Sorry for the confusion. Let me try a brief summary: N is the number of cumulative research effort doublings to go from SAR to SIAR, if r, the parameter controlling the number of doublings needed to get a fixed boost in research progres... (read more)

Saturating RE-Bench → Superhuman coder: three sets of estimates are presented, with medians summing to between 30 and 75 months[6]. The reasoning is presented here.

I think you're looking at the calendar time between now and superhuman coder, rather than the human-only software-only time between RE-Bench and superhuman coder? At least your numbers are quite similar to our overall bottom line which is the former.

1snewman
I added up the median "Predictions for gap size" in the "How fast can the task difficulty gaps be crossed?" table, summing each set of predictions separately ("Eli", "Nikola", "FutureSearch") to get three numbers ranging from 30-75. Does this table cover the time between now and superhuman coder? I thought it started at RE-Bench, because: * I took all of this to be in context of the phrase, about one page back, "For each gap after RE-Bench saturation" * The earlier explanation that Method 2 is "a more complex model starting from a forecast saturation of an AI R&D benchmark (RE-Bench), and then how long it will take to go from that system to one that can handle real-world tasks at the best AGI company" [emphasis added] * The first entry in the table ("Time horizon: Achieving tasks that take humans lots of time") sounds more difficult than saturating RE-Bench. * Earlier, there's a separate discussion forecasting time to RE-bench saturation. But sounds like I was misinterpreting?

I very much appreciate you offering this concrete bet! I probably am not interested in taking this exact proposal and would need to set aside the time to do an investigation into your thread with Ryan and similar to see how close current models are to resolving this, before taking it. I'll add to my to-do list to look into this and perhaps propose an alternative, if you think that might be useful.

See also my comments about how what you're saying is our 0th percentile is not my actual 0th percentile, and how I disagree with you regarding whether the metric ... (read more)

Is your issue that it shouldn't be this determinate, or that it should be more clearly explained? I'm guessing both? As I've said I'm happy to make how important various parameters are more salient in non-high-effort ways.

There are more speedups hidden across parameters, e.g. "Doubling time at RE-Bench saturation toward our time horizon milestone, on a hypothetical task suite like HCAST but starting with only RE-Bench’s task distribution" which also just drops the doubling time.

Could you argue against dropping the expected doubling time on the object level, if you don't find the reasons compelling? I acknowledge that the explanations may not be super clear, lmk if you have questions. I don't think that this would change the overall outputs that much though since most of the time in the benchmarks and gaps model is not from the time horizon extrapolation.

6Peter Johnson
I can't argue against a handful different speedups all on the object level without reference to each other. The justifications generally lie on basically the same intuition which is that AI R&D is strongly enhanced by AI in a virtuous cycle. The only mechanical cause for the speedup claimed is compute efficiency (aka less compute per same performance), and it's hard for me to imagine what other mechanical cause could be claimed that isn't contained in compute or compute efficiency. Finally if I understand the gaps model, it is not a trend exptrapolation model at all! It is purely guesses about calendar time put into a form they are hard to disentangle or validate. To make effective bets we need a relatively high-probability, falsifiable, and quickly-resolving metric that is unlikely to be gamed. METR benchmarks (like every benchmark ever) are able to be gamed or reacted to (the gaming of which is the argument made about most of those handful of distinct speedups). However, if the model relies on a core assumption that is falsifiable, we should focus on that metric. If computational efficiency gains are not core to the model, I am confused on how it claims we will reach SC that is different from bare assertion that we reach SC soon with no reference to anything falsifiable!
elifland5-1

I bet that we will not see a model released in the future that equals or surpasses the general performance of Chinchilla while reducing the compute (in training FLOPs) required for such performance by an equivalent of 3.5x per year.

FWIW I think much of software progress comes from achieving better performance at a fixed or increased compute budget rather than making a fixed performance level more efficient, so I think this underestimates software progress.

edited to add below:

I claim that a response that "increases in computational efficiency only accrue to

... (read more)
4Peter Johnson
The main justification for having compute efficiency be approximately equal to compute in terms of progress given in the timeline supplement and main dropdown is the Epoch AI measurements which are specifically about fixed-performance and lower compute. At the very least this concedes that the estimates are not based on trend-extrapolation and are conjecture. Something being unfalsifiable forward-looking and unmeasurable backwards-looking is a justification for not treating it with high credence, so I think this is also a core disagreement. Here are two charts demonstrating that small changes in estimates of current R&D contribution and changes in R&D speedup change the model massively in the absence of a singularity. I know we're just going to go straight back to "well the real model is the even-more-unfalsifiable benchmarks and gaps model," but I think that is unreasonable. EDIT: THESE FIGURES OVERESTIMATE THE IMPACT OF REDUCING CURRENT ALGORITHMIC PROGRESS. THE SECOND IS WRONG, AND THE REAL IMPACT IS MORE CONTAINED. Figure 1: R&D is 50% of current progress, with and without speedups, exponential only Figure 2: R&D is 33% of current progress, with and without speedups, exponential only I do not understand how "I think this variable doesn't matter (without checking)" is a good defense about questionably implemented variables that do overdetermine the model, but "this variable doesn't matter to outcomes" is not a valid critique w.r.t. things like "what are current capabilities/time horizon" THIS SECOND ONE IS WRONG, MEDIAN HORIZON CHANGES BY CLOSER TO HALF A YEAR AT 33% (TO FEB 2029) THAN ALMOST 2 YEARS (TO APR 2031 AS INCORRECTLY SHOWN)
elifland*40

For reference, the 0th percentile assumed increase in computational efficiency by the authors of the forecasts is about 143x since the Chinchilla release while I am accepting values of just 60x as an immediate loss of my entire principal. By the time the bet turns to a positive return for me (around April 2026), their 0th percentile model assumes increases in computational efficiency of nearly 500x while I accept even a 150x demonstrated increase as a loss.

Where is 143x coming from? It's been barely over 3 years, 4.6^3.1=114x.

I'm not sure what you mean by ... (read more)

2Peter Johnson
You're right on the 143 being closer to 114! (I took March 1 2022 -> July 1 2022 instead of March 22 2022 -> June 1 2022 which is accurate). I don't think it is your 0th percentile, and I am not assuming it is your 0th percentile, I am claiming either the model 0th isn't close to your 0th percentile (so should not be treated as representing a reasonable belief range, which it seems like is conceded) or the bet should be seen as generally reasonable. I sincerely do not think a limited time argument is valid given the amount of work that was put into non-modeling aspects of the presentation and the amount of work claimed put into the model over several gamings and reviews and months of work etc etc. If the burden of proof is on critics to do work you are not willing to do in order to show the model is flawed (for a bounty between 4-10% of the bounty you offer someone writing a supporting piece to advertise your position further), then the defense of limited time raises some hackles.
Load More