Thanks, those links are interesting. Though, improving algorithms and compute don't seem like sufficient conditions - I was also thinking about where does the training data come from (and time for getting the data and doing the required training)? Going from being able to do AI development work to being able to do most economically useful tasks seems like a big step - it seems speculative* as I don't think there are demonstrations of something like this working yet - it would be interesting to know if there is a path here for getting suitable data (enough input-output pairs or tasks with well-specified, automatically verifiable goals) that doesn't require any speculative leaps... (read more)
Something I thought I'd put in a separate comment - there also seems to be good evidence that scenarios like this can adversely affect people's mental health, and I wonder if it would be better to have a more thorough review process for work like this where there is an aim to get a lot of publicity e.g. have a sample of people with relevant expertise (including critics of short AI timelines) who need to be persuaded to vouch for the work's rigour before it is released (along the lines of academic peer review processes). A situation where people can release work that causes harms and there is not robust quality control does not seem good to me. I'm not saying the work here isn't good quality - it's a point about general processes for these pieces, as it seems like they may become increasingly influential.
Thanks for a very interesting post and for going a lot further than usual to set out reasoning for such a scenario.
There were a few steps where it seemed like a large, unexplained jump in abilities of the agents occurred, and I was wondering are there particular reasons to think these jumps would occur or is it just speculative? In particular:
I have been spending around 10%-20% of my time over the past 5 years working as a fund manager on the Long Term Future Fund. As a result, Lightcone has never applied to the LTFF, or the EA Infrastructure Fund, as my involvement with EA Funds would pose too tricky of a COI in evaluating our application. But I am confident that both the LTFF and the EAIF would evaluate an application by Lightcone quite favorably
Would it make sense then for Lightcone people to not have positions at funding orgs like these?
Coming here from your email. It's great to see so much thought and effort go into trying to make an evidence-based case for asking for public donations, which I think is rare to see.
A question that occurs is that the donations chart seems to show that <$1M/year of donations were needed up to 2020 (I can't read off precise figures) - let's say it would be $1M/yr today after adding inflation. The site metrics seem to have been on solid upward trends by then, so it seems like it was at least possible to run LessWrong well on that budget. Would it be fair to say that LessWrong could be run with... (read more)
Why's that? They seem to be going for AGI, can afford to invest billions if Zuckerberg chooses, their effort is led by one of the top AI researchers and they have produced some systems that seem impressive (at least to me). If you wanted to cover your bases, wouldn't it make sense to include them? Though 3-5% may be a bit much (but I also think it's a bit much for the listed companies besides MS and Google). Or can a strong argument be made for why, if AGI were attained in the near term, they wouldn't be the ones to profit from it?
- Invest like 3-5% of my portfolio into each of Nvidia, TSMC, Microsoft, Google, ASML and Amazon
Should Meta be in the list? Are the big Chinese tech companies considered out of the race?
Do you mean you'd be adding the probability distribution with that covariance matrix on top of the mean prediction from f, to make it a probabilistic prediction? I was talking about deterministic predictions before, though my text doesn't make that clear. For probabilistic models, yes adding an uncertainty distribution may make result in non-zero likelihoods. But if we know the true dynamics are deterministic (pretend there's no quantum effects, which are largely irrelevant for our prediction errors for systems in the classical physics domain), then we still know the model is not true, and so it seems difficult to interpret p if we were to do Bayesian updating.
Likelihoods are also not obviously... (read more)
Yes I'd selected that because I thought it might get it to work. And now I've unselected it, it seems to be working. It's possible this was a glitch somewhere or me just being dumb before I guess.
By "complex", I mean a system for which it would be too computationally costly to model it from first principles e.g. the economy, the climate (my field, by the way). Suppose our goal is to predict a system's future behaviour with minimum possible error given by some metric (e.g. minimise the mean square error or maximise the likelihood). This seems like something we would want to do in an optimal way, and also something a superintelligence should have a strategy to do, so I thought I'd ask here if anyone has worked on this problem.
I've read quite a bit about how we can optimally try to deduce the truth e.g. apply Bayes'
I'm still not very sure how to interpret downvotes in the absence of disagree votes... Do people really mean it has negative value to raise discussion about having some tighter quality control on publishing high-profile AI takeover scenarios, or is it just disagreement about not having robust quality control being a problem?
To expand a little more, in my field of climate science research, people have been wrestling with how to produce useful scenarios about the future for many years. There are a couple of instances where respectable groups have produced disaster scenarios, but they have generally had a lot more review than the work here as I understand it. There would be... (read more)