All of Cole Wyeth's Comments + Replies
Yeah, this is also just a pretty serious red flag for the OP’s epistemic humility… it amounts to saying “I have this brilliant idea but I am too brilliant to actually execute it, will one of you less smart people do it for me?” This is not something one should claim without a correspondingly stellar track record - otherwise, it strongly indicates that you simply haven’t tested your own ideas against reality.
Contact with reality may lower your confidence that you are one of the smartest younger supergeniuses, a hypothesis that should have around a 1 in a billion prior probability.
Which seems more likely: capabilities happen to increase very quickly around human genius levels of intelligence, or relative capabilities as compared to the rest of humanity by definition increase only when you’re on the frontier of human intelligence?
Einstein found a lot of currently undiscovered physics because he was somewhat smarter/more insightful than anyone else and so he got ahead. This says almost nothing about absolute capabilities of intelligence.
If orcas were actually that smart wouldn’t it be dangerous to talk to them for exactly the same reasons it would be dangerous to talk to a superintelligence?
No, it's possible for LLMs to solve a subset of those problems without being AGI (even conceivable, as the history of AI research shows we often assume tasks are AI complete when they are not e.g. Hofstader with chess, Turing with the Turing test).
I agree that the tests which are still standing are pretty close to AGI; this is not a problem with Thane's list though. He is correctly avoiding the failure mode I just pointed it out.
Unfortunately, this does mean that we may not be able to predict AGI is imminent until the last moment. That is a consequence of the black-box nature of LLMs and our general confusion about intelligence.
So the thing that coalitional agents are robust at is acting approximately like belief/goal agents, and you’re only making a structural claim about agency?
If so, I find your model pretty plausible.
This sounds like how Scott formulated it, but as far as I know none of the actual (semi)formalizations look like this this.
What is this coalitional structure for if not to approximate an EU maximizing agent?
A couple of years later, do you still believe that foom will happen any year now?
How would this model treat mathematicians working on hard open problems? P vs NP might be counter factual just because no one else is smart enough or has the right advantage to solve it. Insofar as central problems of a field have been identified but not solved, I’m not sure your model gives good advice.
I visited Mikhail Khovanov once in New York to give a seminar talk, and after it was all over and I was wandering around seeing the sights, he gave me a call and offered a long string of general advice on how to be the kind of person who does truly novel things (he's famous for this, you can read about Khovanov homology). One thing he said was "look for things that aren't there" haha. It's actually very practical advice, which I think about often and attempt to live up to!
2
1
1This is a response directly to comments made by Richard Ngo at the CMU agent foundations conference. Though he requested I comment here, the claims I want to focus on go beyond this (and the previous post) and include the following:
1: redefining agency as coalitional (agent = cooperating subagents) as opposed to the normal belief/goal model.
2: justifying this model by arguing that subagents are required for robustness in hard domains (specifically those that require concept invention).
3: that therefore AIXI is irrelevant for understanding agency.&nbs...
Okay, what I meant is “says little in favor of the intelligence of Claude”
Yeah, I agree - overall I agree pretty closely with Thane about LLMs but his final conclusions don't seem to follow from the model presented here.
They might solve it in a year, with one stunning conceptual insight. They might solve it in ten years or more. There's no deciding evidence either way; by default, I expect the trend of punctuated equilibria in AI research to continue for some time.
I did begin and then abandon a sequence about this, cognitive algorithms as scaffolding. I’m like halfway to disendorsing it though.
This won’t work, happy to bet on it if you want to make a manifold market.
Maybe, but on reasonable interpretations I think this should cause us to expect AGI to be farther not nearer.
Yes, but because this scaffolding would have to be invented separately for each task, it’s no longer really zero shot and says little about the intelligence of Claude.
This is convincing evidence LLMs are far from AGI.
Eventually, one of the labs will solve it, a bunch of people will publicly update, and I’ll point out that actually the entire conversation about how an LLM should beat Pokémon was in the training data, the scaffolding was carefully set up to keep it on rails in this specific game, the available action set etc is essentially feature selection, etc.
I disagree because to me this just looks like LLMs are one algorithmic improvement away from having executive function, similar to how they couldn't do system 2 style reasoning until this year when RL on math problems started working.
For example, being unable to change its goals on the fly: If a kid kept trying to go forward when his pokemon were too weak. He would keep losing, get upset, and hopefully in a moment of mental clarity, learn the general principle that he should step back and reconsider his goals every so often. I think most children learn som...
Seems like an easy way to create a less-fakeable benchmark would be to evaluate the LLM+scaffolding on multiple different games? Optimizing for beating Pokemon Red alone would of course be a cheap PR win, so people will try to do it. But optimizing for beating a wide variety of games would be a much bigger win, since it would probably require the AI to develop some more actually-valuable agentic capabilities.
It will probably be correct to chide people who update on the cheap PR win. But perhaps the bigger win, which would actually justify such updates, might come soon afterwards!
Action item: comment on a recent LessWrong or Alignment Forum post on AI safety or write a blog post on AI safety.
This is not generically a good idea. We don't need a bunch of comments on lesswrong posts from new users. It's fine, if there's a specific reason for it (obviously I endorse commenting on lesswrong posts in some cases).
There are a lot of years between 2030 and 2075.
How do you know it's sufficient? Is it not salient to you primarily because it is the current bottleneck?
If "task execution" includes execution of a wide enough class of tasks, obviously the claim becomes trivially true. If it is interpreted more reasonably, I think it is probably false.
I appreciate you wrote this, particularly the final section.
I give it ~70%, except caveats:
"Maybe a slight tweak to the LLM architecture, maybe a completely novel neurosymbolic approach."
It won't be neurosymbolic.
Also I don't see where the 2030 number is coming from. At this point my uncertainty is almost in the exponent again. Seems like decades is plausible (maybe <50% though).
It's not clear that only one breakthrough is necessary.
Without an intelligence explosion, it's around 2030 that scaling through increasing funding runs out of steam and slows down to the speed of chip improvement. This slowdown happens around the same time (maybe 2028-2034) even with a lot more commercial success (if that success precedes the slowdown), because scaling faster takes exponentially more money. So there's more probability density of transformative advances before ~2030 than after, to the extent that scaling contributes to this probability.
That's my reason to see 2030 as a meaningful threshold, Tha...
As you probably know, I have been endorsing this "conspiracy theory" for some time, e.g. roughly here: https://www.lesswrong.com/posts/vvgND6aLjuDR6QzDF/my-model-of-what-is-going-on-with-llms
Hard disagree, this is evidence of slowdown.
As the model updates grow more dense I also check out; a large jump in capabilities between the original gpt-4 and gpt-4.5 would remain salient to me. This is not salient.
Why call it "converse Lawvere" instead of the more standard "utm property" of general recursion theory, e.g. as in Odifreddi? Only because the maps are to [0,1]? That seems like insufficient reason to adopt an unrelated name.
Yes, this is the type of idea big labs will definitely already have (also what I think ~100% of the time someone says "I don't have to give big labs any ideas").
Yes, I agree
...Oh, I know. It's normally 5-20 years from lab to home. My 2027 prediction is for a research robot being able to do anything a human can do in an ordinary environment, not necessarily a mass-producable, inexpensive product for consumers or even most businesses. But obviously the advent of superintelligence, under my model, is going to accelerate those usual 5-20 year timelines quite a bit, so it can't be much after 2027 that you'll be able to buy your own android. Assuming "buying things" is still a thing, assuming the world remains recognizable for at leas
I'm not actually relying on a heuristic, I'm compressing https://www.lesswrong.com/posts/vvgND6aLjuDR6QzDF/my-model-of-what-is-going-on-with-llms
If you extrapolate capability graphs in the most straightforward way, you get the result that AGI should arrive around 2027-2028. Scenario analyses (like the ones produced by Kokotajlo and Aschenbrenner) tend to converge on the same result.
If you extrapolate log GDP growth or the value of the S&P 500, superintelligence would not be anticipated any time soon. If you extrapolate then number of open mathematical ...
What would your mindset have had to say about automated science in 2023, human level robots in 2024, AlphaFold curing cancer in 2025?
I think I agree with Thane’s point 1: because it seems like building intelligence requires a series of conceptual insights, there may be limits to how far in advance I can know it’s about to happen (without like, already knowing how to build it out of math myself). But I don’t view this as a position of total epistemic helplessness - it’s clear that there has been a lot of progress over the last 40 years to the extent that we should be more than halfway there.
And yeah, I don’t view AGI as equivalent to other technologies - its not even clear yet what all t...
It’s wild to me that you’ve concentrated a full 50% of your measure in the next <3 years. What if there are some aspects of intelligence which we don’t know we don’t know about yet? It’s been over ~40 years of progress since the perceptron, how do you know we’re in the last ~10% today?
Progress over the last 40 years has been not at all linear. I don't think this "last 10%" thing is the right way to think about it.
The argument you make is tempting, I must admit I feel the pull of it. But I think it proves too much. I think that you will still be able to make that argument when AGI is, in fact, 3 years away. In fact you'll still be able to make that argument when AGI is 3 months away. I think that if I consistently applied that argument, I'd end up thinking AGI was probably 5+ years away right up until the day AGI was announced.
Here's ano...
1If this is an example of an LLM proving something, it's a very non-central example. It was finetuned specifically for mathematics and then used essentially as a program synthesis engine in a larger system that proved the result.
DeepMind can't just keep running this system and get more theorems out - once the engineers moved on to other projects I haven't heard anything building on the results.
We don't want to talk about partial rationality; we want notions of rationality which bounded agents can fully satisfy.
Why expect this kind of thing to exist? It seems to me that the ideas of computational boundedness and optimality are naturally in tension.
I wore a suit for all of high school. At some point everyone expected me to wear a suit and if I didn't I'd never hear the end of it. The first time I wore a t-shirt there was practically a riot. Unfortunately, I got so tired of conversations regarding the fact that I was or wasn't wearing a suit that escaping to college was a massive relief - now I practically never do it.
I don't agree with the underlying assumption then - I don't think LLMs are capable of solving difficult novel problems, unless you include a nearly-complete solution as part of the groundwork.
Can AI X-risk be effectively communicated by analogy to climate change? That is, the threat isn’t manifesting itself clearly yet, but experts tell us it will if we continue along the current path.
Though there are various disanalogies, this specific comparison seems both honest and likely to be persuasive to the left?
It seems that lesswrong was (before it became mostly AI content) essentially just a massive compilation of such intellectual life hacks. If you filter by the right tags (Rationality?) it should still be usable for this purpose. Have you determined that there is not currently a good centralized table of this kind of content?
If this kind of approach to mathematics research becomes mainstream, out-competing humans working alone, that would be pretty convincing. So there is nothing that disqualifies this example - it does update me slightly.
However, this example on its own seems unconvincing for a couple of reasons:
- it seems that the results were in fact proven by humans first, calling into question the claim that the proof insight belonged to the LLM (even though the authors try to frame it that way).
- from the reply on X it seems that the results of the paper may not have been no
1There is a specific type of thinking, which I tried to gesture at in my original post, which I think LLMs seem to be literally incapable of. It’s possible to unpack the phrase “scientific insight” in more than one way, and some interpretations fall on either side of the line.
I think the argument you’re making is that since LLMs can make eps > 0 progress, they can repeat it N times to make unbounded progress. But this is not the structure of conceptual insight as a general rule. Concretely, it fails for the architectural reasons I explained in the original post.
Obviously it’s not a hard line, but your example doesn’t count, and proving any open conjecture in mathematics which was not constructed for the purpose does count. I think the quote from my post gives some other central examples. The standard is conceptual knowledge production.
Great post!
We in universal AI already have a name for the generalization of AIXI that may use a different history distribution: AI\mu. Since you intend \mu to be a distribution over hypotheses, NOT the true environment as Marcus Hutter usually uses \mu, I would also preferably replace \mu -> \nu. This may seem pedantic but it's kind of triggering to use AIXI to refer to AI\nu, because the "XI" part of "AIXI" is (sometimes) intended to translate as \xi, Hutter's chronological version of the universal distribution.
I think it’s the latter.
It seems suspicious to me that this hype is coming from fields were it seems hard to verify (is the LLM actually coming up with original ideas or is it just fusing standard procedures? Are the ideas the bottleneck or is the experimental time the bottleneck? Are the ideas actually working or do they just sound impressive?). And of course this is Twitter.
Why not progress on hard (or even easy but open) math problems? Are LLMs afraid of proof verifiers? On the contrary, it seems like this is the area where we should be able to best apply RL, since there is a clear reward signal.
On the contrary, it seems like this is the area where we should be able to best apply RL, since there is a clear reward signal.
Is there? It's one thing to verify whether a proof is correct; whether an expression (posed by a human!) is tautologous to a different expression (also posed by a human!). But what's the ground-truth signal for "the framework of Bayesian probability/category theory is genuinely practically useful"?
This is the reason I'm bearish on the reasoning models even for math. The realistic benefits of them seem to be:
- Much faster feedba
1Yeah, I agree with this. If you feed an LLM enough hints about the solution you believe is right, and it generates ten solutions, one of them will sound to you like the right solution.
For me, this is significantly different from the position I understood you to be taking. My push-back was essentially the same as
"has there been, across the world and throughout the years, a nonzero number of scientific insights generated by LLMs?" (obviously yes),
& I created the question to see if we could substantiate the "yes" here with evidence.
It makes somewhat more sense to me for your timeline crux to be "can we do this reliably" as opposed to "has this literally ever happened" -- but the claim in your post was quite explicit about t...
1
1
Yes, but it's also very easy to convince yourself you have more evidence than you do, e.g. invent a theory that is actually crazy but seems insightful to you (may or may not apply to this case).
I think intelligence is particularly hard to assess in this way because of recursivity.