Thanks for a very interesting post and for going a lot further than usual to set out reasoning for such a scenario.
There were a few steps where it seemed like a large, unexplained jump in abilities of the agents occurred, and I was wondering are there particular reasons to think these jumps would occur or is it just speculative? In particular:
My understanding at present is that we have methods for training AI to do tasks where there are many examples of pairs of inputs and desired outputs or where a task has a clearly defined goal and there is opportunity to try doing it many times and apply RL. But this kind of training doesn't seem to feature in the scenario for tasks like the above, so I wondered is there a clear pathway to achieving these using methods we can presently say have a good chance of working? (I am not an AI researcher, so may just not know the extent of demonstrated abilities. I am also not saying I don't believe these are possible - I just wonder for what steps there is a clear pathway and which are primarily speculative.)
I'm also not sure about the resources. The scenario has over $800 billion/year being spent on datacentres by July 2027 (vs ~$300 billion/year in the present as stated). The present total profit of the "Magnificent 7" American tech companies is around $500 billion/year (from https://companiesmarketcap.com/tech/most-profitable-tech-companies/). So the estimated figure is about the total profit of these companies plus around 60%. This gap is a lot more than the profits of Chinese big tech. There doesn't appear to have been widespread economic use of the AIs before this point (given this is where a "tipping point" is said to occur). So this seems like a lot of spending to be happening without widespread acceptance of the economic case. Is there more detail about where these sums of money are plausibly going to come from in that time? How would it affect the time scale if the resources grew, say, half as fast?
I have been spending around 10%-20% of my time over the past 5 years working as a fund manager on the Long Term Future Fund. As a result, Lightcone has never applied to the LTFF, or the EA Infrastructure Fund, as my involvement with EA Funds would pose too tricky of a COI in evaluating our application. But I am confident that both the LTFF and the EAIF would evaluate an application by Lightcone quite favorably
Would it make sense then for Lightcone people to not have positions at funding orgs like these?
Coming here from your email. It's great to see so much thought and effort go into trying to make an evidence-based case for asking for public donations, which I think is rare to see.
A question that occurs is that the donations chart seems to show that <$1M/year of donations were needed up to 2020 (I can't read off precise figures) - let's say it would be $1M/yr today after adding inflation. The site metrics seem to have been on solid upward trends by then, so it seems like it was at least possible to run LessWrong well on that budget. Would it be fair to say that LessWrong could be run with that much donated income? And Lighthaven too (since you seem to think Lighthaven will pay for itself)? You say that now 2025 spending will be ~$3M and expenses will be ~$2M/yr after that. I couldn't follow what is the extra spending above 2020 levels getting?
PS It was fiddly to go back and forth between numbers/charts in the post to try to make sense of and crosscheck them and to check I was using the correct numbers here - I kept misremembering where I'd seen a piece of info and having to scroll around - perhaps making it easy to add a contents section with clickable links to section headings would be valuable for easing navigation of long posts!
Why's that? They seem to be going for AGI, can afford to invest billions if Zuckerberg chooses, their effort is led by one of the top AI researchers and they have produced some systems that seem impressive (at least to me). If you wanted to cover your bases, wouldn't it make sense to include them? Though 3-5% may be a bit much (but I also think it's a bit much for the listed companies besides MS and Google). Or can a strong argument be made for why, if AGI were attained in the near term, they wouldn't be the ones to profit from it?
- Invest like 3-5% of my portfolio into each of Nvidia, TSMC, Microsoft, Google, ASML and Amazon
Should Meta be in the list? Are the big Chinese tech companies considered out of the race?
Do you mean you'd be adding the probability distribution with that covariance matrix on top of the mean prediction from f, to make it a probabilistic prediction? I was talking about deterministic predictions before, though my text doesn't make that clear. For probabilistic models, yes adding an uncertainty distribution may make result in non-zero likelihoods. But if we know the true dynamics are deterministic (pretend there's no quantum effects, which are largely irrelevant for our prediction errors for systems in the classical physics domain), then we still know the model is not true, and so it seems difficult to interpret p if we were to do Bayesian updating.
Likelihoods are also not obviously (to me) very good measures of model quality for chaotic systems, either - in these cases we know that even if we had the true model, its predictions would diverge from reality due to errors in the initial condition estimates, but it would trace out the correct attractor - and its the attractor geometry (conditional on boundary conditions) that we'd seem to really want to assess. Perhaps then it would have a higher likelihood than every other model, but it's not obvious to me, and it's not obvious that there's not a better metric for leading to good inferences when we don't have the true model.
Basically the logic that says to use Bayes for deducing the truth does not seem to carry over in an obvious way (to me) to the case when we want to predict but can't use the true model.
Yes I'd selected that because I thought it might get it to work. And now I've unselected it, it seems to be working. It's possible this was a glitch somewhere or me just being dumb before I guess.
I wonder whether the models are so coarse that the cyclones that do emerge are in a sense the minimum size.
It's not my area, but I don't think that's the case. My impression is that part of what drives very high wind speeds in the strongest hurricanes is convection on the scale of a few km in the eyewall, so models with that sort of spatial resolution can generate realistically strong systems, but that's ~20x finer than typical climate model resolutions at the moment, so it will be a while before we can simulate those systems routinely (though, some argue we could do it if we had a computer costing a few billion dollars).
do you know what discretization methods are typically used for the fluid dynamics?
There's a mixture - finite differencing used to be used a lot but seems to be less common now, semi-Lagrangian advection seems to have taken over from that in models that used it, then some work by doing most of the computations in spectral space and neglecting the smallest spatial scales. Recently newer methods have been developed to work better on massively parallel computers. It's not my area, though, so I can't give a very expert answer - but I'm pretty sure the people working on it think hard about trying to not smooth out intense structures (though, that has to be balanced against maintaining numerical stability).
Something I thought I'd put in a separate comment - there also seems to be good evidence that scenarios like this can adversely affect people's mental health, and I wonder if it would be better to have a more thorough review process for work like this where there is an aim to get a lot of publicity e.g. have a sample of people with relevant expertise (including critics of short AI timelines) who need to be persuaded to vouch for the work's rigour before it is released (along the lines of academic peer review processes). A situation where people can release work that causes harms and there is not robust quality control does not seem good to me. I'm not saying the work here isn't good quality - it's a point about general processes for these pieces, as it seems like they may become increasingly influential.