LESSWRONG
LW

Vladimir_Nesov
34831Ω5074697621506
Message
Dialogue
Subscribe

Posts

Sorted by New

Wikitag Contributions

Comments

Sorted by
Newest
10Vladimir_Nesov's Shortform
Ω
1y
Ω
131
A Pitfall of "Expertise"
Vladimir_Nesov8h20

Still, now I know. [...] Do you see what I had to do? Can you imagine how it probably hurt?

It's not a universal experience, thankfully. I think flow state is most useful for juggling pipelines for learning little things like that. Perhaps it's worth cultivating an attitude where you are not an "expert" at anything, if these are the side effects.

Reply
In Defense of Alcohol
Vladimir_Nesov21h20

the risk of debilitating addiction

I worry the risk of occasional mild discomfort as a result of a very resistible slight addiction isn't being priced in. To the extent that it's permanent, even rare consumption would create the problem. Many people assume a sufficient buffer of willpower and habit to keep consumption at sane levels, but disregard the total cost in the long term of having to manage this constraint, compared to the alternative where even the slight addiction is completely absent.

Reply
ryan_greenblatt's Shortform
Vladimir_Nesov2d1810

I think most of the value in researching timelines is in developing models that can then be quickly updated as new facts come to light. As opposed to figuring out how to think about the implications of such facts only after they become available.

People might substantially disagree about parameters of such models (and the timelines they predict) while agreeing on the overall framework, and building common understanding is important for coordination. Also, you wouldn't necessarily a priori know which facts to track, without first having developed the models.

Reply
All Exponentials are Eventually S-Curves
Vladimir_Nesov2d254

An exponent models things locally, at an appropriate level of detail for modeling them locally. An S-curve won't actually be an S-curve, there will be a lot more data than that in the real thing, omitting the data specifying when the exponent slows down is no different. Simpler models are often more useful, even when you do realize they have a limited scope of applicability.

Reply
Trust me bro, just one more RL scale up, this one will be the real scale up with the good environments, the actually legit one, trust me bro
Vladimir_Nesov2d*Ω6131

Non-OpenAI pre-RLVR chatbots might serve as an anchor for how long it takes an AI company to turn an algorithmic idea into a frontier model, after it becomes a clearly worthwhile thing to do. Arguably only Anthropic managed to catch up to OpenAI, and it took them 1.5 years with Sonnet 3.5. Even Google never caught up after 2+ years, their first credibly frontier chatbot is Gemini 2.5 Pro, which is already well into RLVR (and similarly for Grok 4). So it seems reasonable to expect that it would take about 2 years for RLVR-based models to start being done well, somewhere in 2026-2027.

The IMO results probably indicate something about the current lower bound on capabilities in principle, for informally graded tasks such as natural language proofs. This is a lot higher than what finds practical use so far, and improvements in 2026-2027 might be able to capture this kind of thing (without needing the scale of 2026 compute).

Reply
Models vs beliefs
Vladimir_Nesov3d42

The use of models/theories is in their legibility, you don't necessarily want to obey your models even when forming beliefs on your own. Developing and applying models is good exercise, and there is nothing wrong with working on multiple mutually contradictory models.

Framings take this further, towards an even more partial grasp on reality, and can occasionally insist on patently wrong claims for situations that are not central to how they view the world. Where models help with local validity and communication, framings help with prioritization of concepts/concerns, including prioritization of development of appropriate kinds of models.

Neither should replace the potentially illegible judgement that isn't necessarily possible to articulate or motivate well. That seems to be an important failure mode that leads to either rigid refusal to work with (and get better at) the situations that are noncentral for your favored theories, or to deference to such theories even where they have no business having a clue. If any situation is free to spin up new framings and models around itself, even when they are much worse than and contradictory to the nearby models and framings that don't quite fit, then there is potential to efficiently get better at understanding new things, without getting overly anchored to ways of thinking that are much more familiar or better understood.

Reply
Vladimir_Nesov's Shortform
Vladimir_Nesov3d50

I guess the cost-quality tradeoff makes AI progress even better described as that of a normal technology. As economies of scale reduce cost, they should also be increasing quality (somewhat interchangeably). It's just harder to quantify, and so most of the discussion will be in terms of cost. But for the purposes of raising the ceiling on adoption (total addressable market), higher quality works as well as lower cost, so the lowering of costs is directly relevant.

In this framing, logarithmic improvement of quality with more resources isn't an unusual AI-specific thing either. What remains is the inflated expectations for how quality should be improving cheaply (which is not a real thing, and so leads to the impressions of plateauing with AI, where for other technologies very slow quality improvement would be the default expectation). And Moore's law of price-performance, which is much faster than economic growth. The economies of scale mostly won't be able to notice the growth of the specific market for some post-adoption technology that's merely downstream of the growth of the overall economy. But with AI, available compute would be growing fast enough to make a difference even post-adoption (in 2030s).

Reply
Vladimir_Nesov's Shortform
Vladimir_Nesov4d130

Exponential increase in total economic value is not specific to AI, any new tech is going to start exponentially (possibly following the startups championing it) before it gets further on the adoption S-curve. The unusual things about AI is that it gets better with more resources (while most other things just don't get better at all in a straightforward scaling law manner), that the logarithm of resources thing leaves the persistent impression of plateauing despite not actually plateauing, and that even if it runs out of the adoption S-curve it still has Moore's law of price-performance to keep fueling its improvement. These unusual things frame the sense in which it's linear/logarithmic.

If the improvement keeps raising the ceiling on adoption (capabilities) fast enough, funding keeps scaling into slightly more absurd territory, but even then it won't go a long way without the kind of takeoff that makes anything like the modern industry obsolete. After the exponential phase of adoption comes to an end, it falls back to Moore's law, which still keeps giving it exponential compute to slowly keep fueling further progress, and in that sense there is some unusual exponential-ness to this. Though probably there are other things with scaling laws of their own that global economic growth (instead of Moore's law) would similarly fuel, even slower.

Reply
Vladimir_Nesov's Shortform
Vladimir_Nesov4d589

It seems more accurate to say that AI progress is linear rather than exponential, as a result of being logarithmic in resources that are in turn exponentially increasing with time. (This is not quantitative, any more than the "exponential progress" I'm disagreeing with[1].)

Logarithmic return on resources means strongly diminishing returns, but that's not actual plateauing, and the linear progress in time is only slowing down according to how the exponential growth of resources is slowing down. Moore's law in the price-performance form held for a really long time; even though it's much slower than the present funding ramp, it's still promising exponentially more compute over time.

And so the progress won't obviously have an opportunity to actually plateau, merely proceed at a slower linear pace, until some capability threshold or a non-incremental algorithmic improvement. Observing the continued absence of the never-real exponential progress doesn't oppose this expectation. Incremental releases are already apparently making it difficult for people to notice the extent of improvement over the last 2.5 years. With 3x slower progress (after 2029-2032), a similar amount of improvement would need 8 years.


  1. The METR time horizon metric wants to be at least exponential in time, but most of the other benchmarks and intuitive impressions seem to quantify progress in a way that better aligns with linear progress over time (at the vibe level where "exponential progress" usually has its intended meaning). Many plots use log-resources of various kinds on the horizontal axis, with the benchmark value increasing linearly in log-resources, while it's not yet saturated.

    Perhaps another meaning of "exponential progress" that's real is funding over time, even growth of individual AI companies, but that holds at the start of any technology adoption cycle, or for any startup, and doesn't need to coexist with the unusual feature of AI making logarithmic progress with more resources. ↩︎

Reply11
MakoYass's Shortform
Vladimir_Nesov6d20

The point is that people shouldn't be stakeholders of everything, let alone to an equal extent. Instead, particular targets of optimization (much smaller than the whole world) should have much fewer agents with influence over their construction, and it's only in these contexts that preference aggregation should be considered. When starting with a wider scope of optimization with many stakeholders, it makes more sense to start with dividing it into smaller parts that are each a target of optimization with fewer stakeholders, optimized under preferences aggregated differently from how that settles for the other parts. Expected utility theory makes sense for such smaller projects just as much as it does for the global scope of the whole world, but it breaks normality less when applied narrowly like that than if we try to apply it to the global scope.

The elephant might need to be part of one person's home, but not a concern for anyone else, and not subject to anyone else's preferences. That person would need to be able to afford an elephant though, to construct it within the scope of their home. Appealing to others' preferences about the would-be owner's desires would place the would-be owner within the others' optimization scope, make the would-be owner a project that others are working on, make them stakeholders of the would-be owner's self, rather than remaining a more sovereign entity. If you depend on the concern of others to keep receiving the resources you need, then you are receiving those resources conditionally, rather than allocating the resources you have according to your own volition. Much better for others to contribute to an external project you are also working on, according to what that project is, rather than according to your desires about it.

Reply
Load More
67Permanent Disempowerment is the Baseline
1mo
23
48Low P(x-risk) as the Bailey for Low P(doom)
1mo
29
66Musings on AI Companies of 2025-2026 (Jun 2025)
3mo
4
34Levels of Doom: Eutopia, Disempowerment, Extinction
3mo
0
191Slowdown After 2028: Compute, RLVR Uncertainty, MoE Data Wall
4mo
25
169Short Timelines Don't Devalue Long Horizon Research
Ω
5mo
Ω
24
19Technical Claims
5mo
0
149What o3 Becomes by 2028
8mo
15
41Musings on Text Data Wall (Oct 2024)
1y
2
10Vladimir_Nesov's Shortform
Ω
1y
Ω
131
Load More
Quantilization
2y
(+13/-12)
Bayesianism
2y
(+1/-2)
Bayesianism
2y
(+7/-9)
Embedded Agency
3y
(-630)
Conservation of Expected Evidence
4y
(+21/-31)
Conservation of Expected Evidence
4y
(+47/-47)
Ivermectin (drug)
4y
(+5/-4)
Correspondence Bias
4y
(+35/-36)
Illusion of Transparency
4y
(+5/-6)
Incentives
4y
(+6/-6)
Load More