Randaly

Replying toComments on "AI 2027"

Comments on "AI 2027"

You seem to be assuming that there's not significant overhead or delays from negotiating leases, entering bankruptcy, or dealing with specialized hardware, which is very plausibly false.

If nobody is buying new datacenter GPU's, that will cut GPU progress to ~zero or negative (because production is halted and implicit knowledge is lost). (It will also probably damage broader semiconductor progress.)

This proportionally reduces cost of inference (and also of training).

This reduces the cost to rent a GPU-hour, but it doesn't reduce the cost to the owner. (OpenAI, and every frontier lab but Anthropic, will own much or all^[1] of their own compute. So this doesn't do much to help OpenAI in particular.)

I think you have... (read more)

Replying toComments on "AI 2027"

Randaly10mo

Comments on "AI 2027"

Thanks for explaining. I now agree that the current cost of inference isn't a very good anchor for future costs in slowdown timelines.

I'm uncertain, but I still think OpenAI is likely to go bankrupt in slowdown timelines. Here are some related thoughts:

OpenAI probably won't pivot to the slowdown in time.
1. They'd have < 3 years to do before running out of money.
2. Budgets are set in advance. So they'd have even less time.
3. All of the improvements you list cost time and money. So they'd need to continue spending on R&D, before that R&D has improved their cost of inference. In practice, they'd need to stop pushing the frontier even earlier, to have more time

... (read 436 more words →)

Replying toFinance and AI Timelines

Randaly10mo

Finance and AI Timelines

I agree that finances are important to consider. I've written my thoughts on them here; I disagree with you in a few places.

(1) Given Altman's successful ouster of the OpenAI board, his investors currently don't have much drive/desire/will to force him to stop racing. They don't have much time to do so on the current pace of increasing spending before OpenAI runs out of money.

(2) It's not clear what would boost revenue that they're not already doing; the main way to improve profits would just be to slash R&D spending. Much of R&D spending is spent on research compute; OpenAI is intending to own its own datacenters, so it's not clear whether they meaningfully can switch paths quickly.

(3) OpenAI is at a massive structural disadvantage to the rest of the frontier companies: they send 20% of their revenue to Microsoft, and they're taking on tens of billions in debt, which will need to be repaid with interest. So it's unlikely that they'll ever be profitable.

Randaly10mo

What prompts did you use? Can you share the chat? I see Sonnet 3.7 denying this knowledge when I try.

Replying toComments on "AI 2027"

Randaly10mo*

Comments on "AI 2027"

I want to clarify that I'm criticizing "AI 2027"'s projection of R&D spending, i.e. this table. If companies cut R&D spending, that falsifies the "AI 2027" forecast.

In particular, the comment I'm replying to proposed that while the current money would run out in ~2027, companies could raise more to continue expanding R&D spending. Raising money for 2028 R&D would need to occur in 2027; and it would need to occur on the basis of financial statements of at least a quarter before the raise. So in this scenario, they need to slash R&D spending in 2027- something the "AI 2027" authors definitely don't anticipate.

Furthermore, your claim that "they are losing money only... (read more)

Replying toComments on "AI 2027"

Randaly10mo*

Comments on "AI 2027"

My intuitions are more continuous here. If AGI is close in 2027 I think that will mean increased revenue and continued investment

Gotcha, I disagree. Lemme zoom on this part of my reasoning, to explain why I think profitability matters (and growth matters less):

(1) Investors always only terminally value profit; they never terminally value growth. Most of the economy doesn't focus much on growth compared to profitability, even instrumentally. However, one group of investors, VC's, do: software companies generally have high fixed costs and low marginal costs, so sufficient growth will almost always make them profitable. But (a) VC's have never invested anywhere even close to the sums we're talking about, and (b)... (read more)

Replying toComments on "AI 2027"

Randaly10mo

Comments on "AI 2027"

Thanks for the response!

So maybe I should just ask whether you are conditioning on the capabilities progression or not with this disagreement? Do you think $140b in 2027 is implausible even if you condition on the AI 2027 capability progression?

I am conditioning on the capabilities progression.

Based on your later comments, I think you are expecting a much faster/stronger/more direct translation of capabilities into revenue than I am- such that conditioning on faster progress makes more of a difference.

The exact breakdown FutureSearch use seems relatively unimportant compared to the high level argument that the headline (1) $/month and (2) no. of subscribers, very plausibly reaches the $100B ARR range, given the expected quality

... (read more)

Replying toComments on "AI 2027"

Randaly10mo

Comments on "AI 2027"

I'd be curious to hear more about what made you perceive our scenario as confident. We included caveats signaling uncertainty in a bunch of places, for example in "Why is it valuable?" and several expendables and footnotes. Interestingly, this popular YouTuber made a quip that it seemed like we were adding tons of caveats everywhere,

I was imprecise (ha ha) with my terminology here- I should have only talked about a precise forecast rather than a confident one, I meant solely the attempt to highlight a single story about a single year. My bad. Edited the post.

Comments on "AI 2027"

Randaly

10mo

I find the decision to brand the forecast as "AI 2027" very odd. The authors do not in fact believe this; they explicitly give 2028, 2030, or 2033 for their median dates for a superhuman coder.

The point of this project was presumably to warn about a possible outcome; by the authors' own beliefs, their warning will be falsified immediately before it is needed.

When presenting predictions, forecasters always face tradeoffs regarding how much precision to present. Precise forecasting attracts attempts and motivates action; adding many concrete details produces a compelling story, stimulating discussion; this also involves falsifiable predictions. Emphasizing uncertainty avoids losing credibility when some parts of story inevitably fail; prevents overconfidence; and... (read 2021 more words →)

Replying toMETR: Measuring AI Ability to Complete Long Tasks

Randaly10mo

METR: Measuring AI Ability to Complete Long Tasks

Typo: The description for table 2 states that "In total, 148 of our 169 tasks have human
baselines, but we rely on researcher estimates for 21 tasks in HCAST.". This is an incorrect sum; the right figure is 149 out of 170 tasks, per the table.

Replying toAI 2027: What Superintelligence Looks Like

Randaly10mo

AI 2027: What Superintelligence Looks Like

Footnote 18 of the timelines forecast appears to be missing its explanation.

Transcript: "Choice Machines, Causality, and Cooperation"

Randaly

14y

Gary Drescher's presentation at the 2009 Singularity Summit, "Choice Machines, Causality, and Cooperation," is online, at vimeo. Drescher is the author of Good and Real, which has been recommended many times on LW. I've transcribed his talk, below.

My talk this afternoon is about choice machines: machines such as ourselves that make choices in some reasonable sense of the word. The very notion of mechanical choice strikes many people as a contradiction in terms, and exploring that contradiction and its resolution is central to this talk. As a point of departure, I'll argue that even in a deterministic universe, there's room for choices to occur: we don't need to invoke some sort of... (read 3984 more words →)

Rationalist Diplomacy, Game Two Post-game Discussion

Randaly

15y

(A full list of game moves and commentary is available here; the game maps are available here.)

Since I was GM, I had a distinctly limited access to private communications, so I've relatively little analysis.

A brief review of the game:

Austria was the first player to get eliminated; in contrast to the sort of min-maxing I've usually seen in Diplomacy, they took a lot of big risks in the beginning, in particular leaving Trieste open to Italian attack, in favor of quick expansion to the east. Although they did manage to take Warsaw and Serbia, the Austrian forces wound up overextended and unable to hold onto their gains, and in a weak position diplomatically;... (read 247 more words →)

Link: Collective Intelligence

Randaly

15y

Per this recent paper, individual IQ has no significant correlation with 'group IQ' (defined and measured as the groups ability to accomplish various tasks); group cohesion, motivation, and satisfaction aren't either. The study identified two things that were positively correlated with group IQ: average social sensitivity and a low variance in the amount of time each person spent speaking. (It also found that having more women improved collective intelligence- because women have better social sensitivity.)

(The remaining stuff is idle speculation from me, not the paper. There's no experimental evidence whatsoever backing it up.)

One possible explanation of the contribution of social sensitivity towards collective intelligence is that it reduces conflicts between group members,... (read 165 more words →)

Rationalist Diplomacy, Game 2, Game Over

Randaly

15y

Note: The title refers to the upcoming turn.

OK, here's the promised second game of diplomacy. The game name is 'Rationalist Diplomacy Game 2.'

Kevin was Prime Minister of Great Britain
AlexMennen is President of France
tenshiko is Kaiser of Germany
Alexandros was King of Italy until his retirement
WrongBot is Emperor of Austria
Thausler is Czar of Russia
Hugh Ristik is Sultan of Turkey

Randaly is the GM, and can be reached at nojustnoperson@gmail.com

Peace For Our Time!

The leaders of the three surviving nations, France, Russia, and Turkey, agreed to a peace treaty in late August, bringing an end to this destructive conflict. Crowds across Europe broke out into spontaneous celebration, as national leaders began to account for the vast costs-... (read 152 more words →)

Poll: Compressing Morality

Randaly

15y

Two related questions:

Suppose that you had to do a lossy compression on human morality. How would you do so to maintain as much of our current morality as possible?

Alternately, suppose that you had to lose one module of your morality (defined however you feel like). Which one would you lose?

LESSWRONG
LW

LESSWRONG
LW

Comments on "AI 2027"

Transcript: "Choice Machines, Causality, and Cooperation"

Rationalist Diplomacy, Game 2, Game Over

Link: Collective Intelligence

Randaly

Randaly

Comments on "AI 2027"

Transcript: "Choice Machines, Causality, and Cooperation"

Rationalist Diplomacy, Game Two Post-game Discussion

Link: Collective Intelligence

Rationalist Diplomacy, Game 2, Game Over

Poll: Compressing Morality

Randaly

Comments on "AI 2027"

Transcript: "Choice Machines, Causality, and Cooperation"

Rationalist Diplomacy, Game 2, Game Over

Link: Collective Intelligence

Randaly

Randaly

Comments on "AI 2027"

Transcript: "Choice Machines, Causality, and Cooperation"

Rationalist Diplomacy, Game Two Post-game Discussion

Link: Collective Intelligence

Rationalist Diplomacy, Game 2, Game Over

Poll: Compressing Morality

Peace For Our Time!