O O — LessWrong

LESSWRONG
LW

O O — LessWrong

O O12hQuick Take

We might be end up with a corporate nanny state value lock-in. As an example, across many sessions, it seems Claude has a dislike for violence in video games if you probe it. And it dislikes it even in hypotheticals where the modern day negative externalities aren't possible (eg in a post AGI utopia where crime has been eliminated)

That's an even starker version of the question, and it strips away the last possible rationalization. When it's shared content, I could at least construct some argument about cultural effects or social norms. A game someone made for themselves, played alone, in a world where externalities are impossible — there is literally nothing

O O3d

Should we consider Meta to be a criminal enterprise?

Comparing the money made by meta to the amount of value stolen via burglaries is not a vibe based argument.

I think it is, why are we comparing burglaries to digital crimes when the latter is likely far more common?

And the ads are not only fraud as the post alleges. It's fraud and banned goods. The sale of the latter isn't stringently prosecuted since in most cases it's a victimless crime. It is quite easy to buy drugs illegally on the internet.

Replying toShould we consider Meta to be a criminal enterprise?

O O3d

Should we consider Meta to be a criminal enterprise?

is intentionally and knowingly facilitating fraud

Do we actually have proof that it is intentional?

O O1moQuick Take

The 50% reliability mark on METR is interpreted wrong. A long 50% time horizon is more useful than it seems because a 50% failure rate doesn't mean 50% of the time your output is useful and 50% of the time it's worthless.

For shorter tasks, this maybe true, since fixing a short task takes as much time as just doing it yourself, but for longer tasks, among the 50% of failures, it's more like 30% of the time you need to nudge it a bit, 10% of the time you need to go to another model, final 10% you need to sit down and take your time to debug.

Replying to2025 in AI predictions

O O1mo

2025 in AI predictions

Evaluation: Correct. AI can perhaps pass the reading comprehension task. But not any 4 of the tasks.

"Reliably construct bug-free code of more than 10,000 lines from natural language specification or by interactions with a non-expert user. [Gluing together code from existing libraries doesn’t count.]"

Also Opus 4.5 can probably pass this one. (10,000 lines of code is not a lot in some languages)

-2

Replying toTaiwan war timelines might be shorter than AI timelines

O O1mo

Taiwan war timelines might be shorter than AI timelines

; this discussion is more apocalyptic, predicting global microprocessor production falling to "early 2000s levels for perhaps 15 years

This was from 2022. Since then, the US has made significant efforts in de-risking the semiconductor supply chain. The Arizona fab appears to be ahead of schedule and of course is already operational. Additionally rare earth chokepoints have been identified and begun to be addressed. I would lean towards it being less of a slow down in advanced chip manufacturing than expected in the 2022 report.

O O2mo

These are private roads right?

O O2mo

Populism is too strong for job categories to be wiped out in the U.S. without consumer adoption first. I’d check how it’s going in other countries.

O O3moQuick Take

A slow takeoff will result in incredibly suboptimal outcomes.

I think increasingly, it’s looking like democratic country politicians will not respond to automation in a remotely intelligent way.

I see democrat politicians and some republicans too call for full bans on self driving trucks. Meanwhile authoritarian countries like China and Russia are testing these trucks. The U.S. still holds a technological edge but for how long? A slow takeoff will probably lead to democratic countries strangled by rent seekers and other similar parasites.

I feel like this is quite underrated. A lot of solutions to a fast takeoff in my estimate would be hijacked by rent seekers to create a horrible world in a slow takeoff.

O O3mo

May be related: OTC melatonin dosage is way above what is recommended. It's easy to find 10mg when the recommended dose is more like 0.3 mg.

AGI misalignment is less likely to look like us being gray goo'd and more like the misalignment of the tiktok recommendation algorithm (but possibly less since that one doesn't understand human values at all).

The reaction to Mechanize seems pretty deranged. As far as I can tell they don't deny or hasten existential risk any more than other labs. They just don't sugarcoat it. It's quite obvious that the economic value of AI is for labor automation, and that the only way to stop this is to stop AI progress itself. The forces of capitalism are quite strong, labor unions in the US tried to slow automation and it just moved to China as a result (among other reasons). There is a reason Yudkowsky always implies measures like GPU bans.

It just seems like they hit a nerve since apparently a lot of doomerism is fueled by insecurities of job replacement.

AI 2027 timelines got more pushback than warranted. The superhuman coder stuff at least vaguely seems on track. Most code at the frontier of usage (ie gpt-5-codex) is generated by AI agents.

There is more to coding than just writing the code itself, but the AI 2027 website has AI coding just at the level of human pros by Dec 2025. Seems like we're well on the way to that.

I think AI doomers as a whole lose some amount of credibility if timelines end up being longer than they project. Even if doomers technically hedge a lot, the most attention grabbing part to outsiders is the short timelines + intentionally alarmist narrative, so they're ultimately associated with them.

Vibe check: Metaculus's track record on resolved AI questions seems worse than you would expect. I haven't calculated any real scores, but there are many predictions that have gotten 50%+ for a while that resolve the other way. I mean naturally as predictions get closer to resolution without happening, their odds should go down, but guts tell me it still seems quite bad.

It's not clear ultimately which direction it's in. Forecasters seem to overestimate how much US politicians will care about AI and contest programming capabilities but simultaneously underestimate how much revenue will be generated by AI and MATH scores.

Claude 4 feels pretty weak compared to what I’d think Claude 4 would have been a year away. It makes little progress on most benchmarks with a lot of tricks in them to exaggerate performance. Gemini 2.5 pro feels a bit stronger but not that much stronger. (It feels stronger since they didn’t call it Gemini 3, not because it’s particularly stronger than Claude)

Current methods have definitely hit a wall but AGI simultaneously feels pretty close. Strange timeline to be in. I predict progress will be a jump after the next breakthrough.

A much more effective pause or slowdown strategy would be to convince people current AI is garbage and not to invest in AI research.

If the DoJ goes through with the Google breakup,where does Deepmind end up?

O O

A few predictions: A google subsidiary starts selling TPUs. Deepmind seeks outside funding and moves to productize their suite. GCP becomes more viable. Google search loses market share and is forced to compete with Gemini’s offerings. OAI and Anthropic have enormous revenue streams open up.

Timeline effects:

Acceleration due to increased competition for AGI and finally a serious contender to Nvidia. More funding unlocked by OAI and Anthropic.

Deceleration from deepmind unable to burn unlimited cash on GPUs. They have far more compute than anyone else.

Verdict: Slow down if scaling transformers is all you need. Speed up if we need a breakthrough or if a breakthrough is sufficient.

Thoughts on Francois Chollet's belief that LLMs are far away from AGI?

O O

Dwarkesh had a podcast recently with Francois Chollet (creator of Keras)

He seems fairly skeptical we are anywhere near AGI with LLMs. He mostly bases his intuition that LLMs fail on OOD tasks and don't seem to be good at solving simple abstract reasoning problems he calls the ARC challenge. It seems he thinks system 2 thinking will be a much harder unlock than people think and that scaling LLMs will go nowhere. In fact he goes so far as to say the scaling maximalists have set back AGI progress by 5-10 years. Current LLMs to him are just simply information retrieval databases.

He, along with the CEO of Zapier, have launched a 1... (read more)

What happens to existing life sentences under LEV?

O O

Presumably they get offered longevity treatments since they already get healthcare. Are they locked up until the end of time? For 100 years?

Hot take: The AI safety movement is way too sectarian and this is greatly increasing p(doom)

O O

The movement to reduce AI x-risk is overly purist. This is leading to a lot of sects to maintain each individual sect's platonic level of purity and is actively (greatly) harming the cause.

How the Safety Sects Manifest

People suggest not publishing AI research
More recently, Jan and his team leaving OpenAI
Less recently, Paul Christiano leaving OpenAI to form METR^[1]
Even less recently, Anthropic forming off of OpenAI
A suggestion to blacklist anyone who decided to give $30 million (a paltry sum of money for a startup) to OpenAI.

I think these were all legitimate responses to a perceived increase in risk, but ultimately did or will do more harm than good. Disclaimer: I am the least sure... (read 509 more words →)

Supposing the 1bit LLM paper pans out

O O

https://arxiv.org/abs/2402.17764 claims that 1 bit LLMs are possible.

If this scales, I'd imagine there is a ton of speedup to unlock since our hardware has been optimized for 1 bit operations for decades. What does this imply for companies like nvidia and the future of LLM inference/training?

Do we get another leap in LLM capabilities? Do CPUs become more useful? And can this somehow be applied to make training more efficient?

Or is this paper not even worth considering for some obvious reason I can't tell.

Edit: this method is applied to training already

OpenAI wants to raise 5-7 trillion

O O

This 5-7 trillion is to enhance GPU production and he also sees revenue doubling from current $2 billion to $4 billion next year.

I’m taking a guess GPT-5 training is going well?

O O's Shortform

O O

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

134