LESSWRONG
LW

HomeAll PostsConceptsLibrary
Best of LessWrong
Sequence Highlights
Rationality: A-Z
The Codex
HPMOR
Community Events
LW the Album
About
FAQ
Best of LessWrong 2023View All
Please don't throw your mind away
#1 in Rationality
Basics of Rationalist Discourse
#2 in Rationality
Feedbackloop-first Rationality
#3 in Rationality
Rationality
Social Dark Matter
#1 in Modeling
How much do you believe your results?
#2 in Modeling
A Disneyland Without Children
#3 in Modeling
Modeling
Focus on the places where you feel shocked everyone's dropping the ball
#1 in Optimization
Significantly Enhancing Adult Intelligence With Gene Editing May Be Possible
#2 in Optimization
Things I Learned by Spending Five Thousand Hours In Non-EA Charities
#3 in Optimization
Optimization
AI Control: Improving Safety Despite Intentional Subversion
#1 in AI Safety
Natural Abstractions: Key claims, Theorems, and Critiques
#2 in AI Safety
Alignment Implications of LLM Successes: a Debate in One Act
#3 in AI Safety
AI Safety
Pausing AI Developments Isn't Enough. We Need to Shut it All Down
#1 in AI Strategy
AI Timelines
#2 in AI Strategy
What a compute-centric framework says about AI takeoff speeds
#3 in AI Strategy
AI Strategy
How to have Polygenically Screened Children
#1 in Practical
Guide to rationalist interior decorating
#2 in Practical
Loudly Give Up, Don't Quietly Fade
#3 in Practical
Practical
Customize

Quick Takes

faul_sname288
2
As LLMs have gotten better at writing code that has a high probability of working to solve the problem they are working on, they have gotten worse at producing clean, idiomatic, well-factored code. Concretely, asking the original GPT-4 to write a Python function for multi-source BFS might have given something like  Multi-source BFS in the style of original GPT-4: Clear, idiomatic, broken def multi_source_bfs(graph, sources): distances = [-1] * len(graph) queue = [] for source in sources: queue.append(source) distances[source] = 0 front = 0 while front < len(queue): for neighbor in graph[queue[front]]: if distances[neighbor] == -1: distances[neighbor] = distances[queue[front]] + 1 queue.append(neighbor) front += 1 return distances[dest_index] The code might or might not work (probably won't for anything nontrivial), but the intent is clear. By contrast, if you ask a top coding model like sonnet 3.7 or o3, you'll get something that looks like Multi-source BFS in the style of Sonnet 3.7: Verbose, brittle, hard to read, almost certainly works from collections import deque from typing import List, Optional, Set, Dict def multi_source_bfs(graph: List[List[int]], sources: List[int]) -> List[int]: """ Performs a multi-source BFS on a graph to find minimum distance from any source to each node. Args: graph: An adjacency list where graph[i] contains neighbors of node i sources: A list of source node indices Returns: A list where result[i] is the minimum distance from any source to node i or -1 if node i is unreachable """ # Handle empty graph or sources if not graph: return [] if not sources: return [-1] * len(graph) # Remove duplicates from sources if any sources = list(set(sources)) # Initialize distances array with -1 (unreachable) distances = [-1] * len(graph) # Init
Ben Goldhaber70
2
Why so few third party auditors of algorithms? for instance, you could have an auditing agency make specific assertions about what the twitter algorithm is doing, whether the community notes is 'rigged'   * It could be that this is too large of a codebase, too many people can make changes, it's too hard to verify the algorithm in production is stable. This seems unlikely to me with most modern devops stacks * It could be that no one will trust the third party agency. I guess this seems most likely... but really, have we even tried? Could we not have some group of monk like Auditors who would rather die than lie (my impression is some cyber professionals have this ethos already) If Elon wanted to spend a couple hundred thousand on insanely commited high integrity auditors, it'd be a great experiment
MichaelDickens*10828
36
I find it hard to trust that AI safety people really care about AI safety. * DeepMind, OpenAI, Anthropic, and SSI were all founded in the name of safety. Instead they have greatly increased danger. And at least OpenAI and Anthropic have been caught lying about their motivations: * OpenAI: claiming concern about hardware overhang and then trying to massively scale up hardware; promising compute to superalignment team and then not giving it; telling board that model passed safety testing when it hadn't; too many more to list. * Anthropic: promising (in a mealy-mouthed technically-not-lying sort of way) not to push the frontier, and then pushing the frontier; trying (and succeeding) to weaken SB-1047; lying about their connection to EA (that's not related to x-risk but it's related to trustworthiness). * For whatever reason, I had the general impression that Epoch is about reducing x-risk (and I was not the only one with that impression) but: * Epoch is not about reducing x-risk, and they were explicit about this but I didn't learn it until this week * its FrontierMath benchmark was funded by OpenAI and OpenAI allegedly has access to the benchmark (see comment on why this is bad) * some of their researchers left to start another build-AGI startup (I'm not sure how badly this reflects on Epoch as an org but at minimum it means donors were funding people who would go on to work on capabilities) * Director Jaime Sevilla believes "violent AI takeover" is not a serious concern, and "I also selfishly care about AI development happening fast enough that my parents, friends and myself could benefit from it, and I am willing to accept a certain but not unbounded amount of risk from speeding up development", and "on net I support faster development of AI, so we can benefit earlier from it" which is a very hard position to justify (unjustified even on P(doom) = 1e-6, unless you assign ~zero value to people who are not yet born) * I feel bad picking on Epoch/
Kaj_Sotala90
1
Been trying the Auren app ("an emotionally intelligent guide built for people who care deeply about their growth, relationships, goals, and emotional well-being") since a few people were raving about it. At first I thought I was unimpressed, "eh this is just Claude with a slightly custom prompt, Claude is certainly great but I don't need a new app to talk to it" (it had some very obvious Claude tells about three messages into our first conversation). Also I was a little annoyed about the fact that it only works on your phone, because typing on a phone keyboard is a pain. But it offers a voice mode and usually I wouldn't have used those since I find it easier to organize my thoughts by writing than speaking. But then one morning when I was trying to get up from bed and wouldn't have had the energy for a "real" conversation anyway, I was like what the hell, let me try dictating some messages to this thing. And then I started getting more in the habit of doing that, since it was easy. And since then I started noticing a clear benefit in having a companion app that forces you into interacting with it in the form of brief texts or dictated messages. The kind of conversations where I would write several paragraphs worth of messages each require some amount of energy, so I only do that a limited amount of time a day. But since I can't really interact with Auren in this mode, my only alternative is to interact with it in quicker and lower-effort messages... which causes me to interact with it more. Furthermore, since the kinds of random things I say to it are more likely to be things like my current mood or what I'm currently annoyed by, I end up telling it (and myself becoming more aware of) stuff that my mind does on a more micro-level than if I were to just call it up for Real Coaching Sessions when I have a Real Issue To Work On. It also maintains some kind of memory of what we've discussed before and points out patterns I wouldn't necessarily have noticed, and somet
Joseph Miller200
0
Announcing PauseCon, the PauseAI conference. Three days of workshops, panels, and discussions, culminating in our biggest protest to date. Tweet: https://x.com/PauseAI/status/1915773746725474581 Apply now: https://pausecon.org

Popular Comments

Load More

Recent Discussion

We should try to automate AI safety work asap
32
Marius Hobbhahn
Ω 16

This is a personal post and does not necessarily reflect the opinion of other members of Apollo Research. I think I could have written a better version of this post with more time. However, my main hope for this post is that people with more expertise use this post as a prompt to write better, more narrow versions for the respective concrete suggestions. 

Thanks to Buck Shlegeris, Joe Carlsmith, Samuel Albanie, Max Nadeau, Ethan Perez, James Lucassen, Jan Leike, Dan Lahav, and many others for chats that informed this post. 

Many other people have written about automating AI safety work before. The main point I want to make in this post is simply that “Using AI for AI safety work should be a priority today already and isn’t months or years away.” To...

(Continue Reading – 4477 more words)
jacquesthibs20

Thanks for publishing this! @Bogdan Ionut Cirstea, @Ronak_Mehta, and I have been pushing for it (e.g., building an organization around this, scaling up the funding to reduce integration delays). Overall, it seems easy to get demoralized about this kind of work due to a lack of funding, though I'm not giving up and trying to be strategic about how we approach things.

I want to leave a detailed comment later, but just quickly:

  • Several months ago, I shared an initial draft proposal for a startup I had been working towards (still am, though under a differen
... (read more)
Reply
≤10-year Timelines Remain Unlikely Despite DeepSeek and o3
54
Rafael Harth

[Thanks to Steven Byrnes for feedback and the idea for section §3.1. Also thanks to Justis from the LW feedback team.]

Remember this?

Or this?

The images are from WaitButWhy, but the idea was voiced by many prominent alignment people, including Eliezer Yudkowsky and Nick Bostrom. The argument is that the difference in brain architecture between the dumbest and smartest human is so small that the step from subhuman to superhuman AI should go extremely quickly. This idea was very pervasive at the time. It's also wrong. I don't think most people on LessWrong have a good model of why it's wrong, and I think because of this, they don't have a good model of AI timelines going forward.

1. Why Village Idiot to Einstein is a Long Road: The Two-Component

...
(Continue Reading – 4327 more words)
2Rafael Harth
I think we have specialized architectures for consciously assessing thoughts, whereas LLMs do the equivalent of rattling off the first thing that comes to mind, and reasoning models do the equivalent of repeatedly feeding back what comes to mind into the input (and rattling off the first thing that comes to mind for that input).
Eli Tyre20

Do you have a pointer for why you think that? 

My (admittedly weak) understanding of the neuroscience doesn't suggest that there's a specialized mechanism for critique of prior thoughts.

Reply
bgold's Shortform
Ben Goldhaber
7Ben Goldhaber
Why so few third party auditors of algorithms? for instance, you could have an auditing agency make specific assertions about what the twitter algorithm is doing, whether the community notes is 'rigged'   * It could be that this is too large of a codebase, too many people can make changes, it's too hard to verify the algorithm in production is stable. This seems unlikely to me with most modern devops stacks * It could be that no one will trust the third party agency. I guess this seems most likely... but really, have we even tried? Could we not have some group of monk like Auditors who would rather die than lie (my impression is some cyber professionals have this ethos already) If Elon wanted to spend a couple hundred thousand on insanely commited high integrity auditors, it'd be a great experiment
ryan_greenblatt70

Community notes is open source. You have to hope that Twitter is actually using the implementation from the open source library, but this would be easy to whistleblow on.

Reply
5Garrett Baker
Your second option seems likely. Eg did you know community notes is open source? Given that information, are you going to even read the associated whitepaper or the issues page? Even if you do, I think we can still confidently infer very few others reading this will (I know I’m not).
How I Think About My Research Process: Explore, Understand, Distill
29
Neel Nanda
Ω 12

This is the first post in a sequence about how I think about and break down my research process. Post 2 is coming soon.

Thanks to Oli Clive-Griffin, Paul Bogdan, Shivam Raval and especially to Jemima Jones for feedback, and to my co-author Gemini 2.5 Pro - putting 200K tokens of past blog posts and a long voice memo in the context window is OP.

Introduction

Research, especially in a young and rapidly evolving field like mechanistic interpretability (mech interp), can often feel messy, confusing, and intimidating. Where do you even start? How do you know if you're making progress? When do you double down, and when do you pivot?

These are far from settled questions, but I’ve supervised 20+ papers by now, and have developed my own mental model of...

(Continue Reading – 2061 more words)
jacquesthibs20

Thanks Neel!

Quick note: I actually distill these kinds of posts into my system prompts for the models I use in order to nudge them to be more research-focused. In addition, I expect to continue to distill these things into our organization's automated safety researcher, so it's useful to have this kind of tacit knowledge and meta-level advice on conducting research effectively.

Reply
1Adrian Chan
I read this with interest and can't help but contrast it with the research approach I am more accustomed to, and which is perhaps more common in soft sciences/humanities. Because many of us use AI for non-scientific, non-empirical research, and are each discovering that it is both an art and a science.  My honors thesis adviser (US-Soviet relations) had a post-it on his monitor said "What is the argument?" I research w GPT over multiple turns and days in an attempt to push it to explore. I find I can do so only insofar as I comprehend its responses in whatever discursive context or topic/domain we're in. It's a kind of co-thinking.  I'm aware that GPT has no perspective, no argument to make, no subjectivity, and no point of view. I on the other hand have interests and am interested. GPT can seem interested, but in a post-subjective or quasi objective way. That is it can write stylistically as if it is interested, but it cannot pursue interests unless they are taken up by me, and then prompted.  This takes the shape of an interesting conversation. One can "feel" the AI has an active interest and has agency in pursuing research, but we know it is only plumbing texts and conjuring responses.  This says something about the discursive competence of AI and also of the cognitive psychology of us users. Discursively, the AI seems able to reflect and reason through domain spaces and to return what seems to be commonly-accepted knowledge. That is, it's a good researcher of stored content. It finds propositions, statements, claims, valid arguments insofar as they are reflected in the literature it is trained on. To us, psychologically, however, this can read as subjective opinion, confident reasoning, comprehensive recapitulation.  In this is a trust issue w AI, insofar as the apparent communication and AI's seeming linguistic competence elicit trust from us users. And this surely factors into the degree to which we regard its responses as "factual," "comprehensive," etc.
1eamag
From all of these, which ones are: * The most time consuming * The most difficult for beginners? I'm asking to figure out what's the bottleneck, it seems like many people are interested in AI safety but not all of them are working on research directly, what need to happen to make it easier for them to start?
Guide to rationalist interior decorating
320
mingyuan

Recently someone asked me to write a guide to rationalist interior decorating, since there’s a set of products and best practices (originating with CFAR and Lightcone) that have gotten wide adoption. I’m perhaps not the very most qualified person to write this post, but I’ve been into interior decorating since before the Lightcone team got into it, and I basically know what they do, plus they’re all very busy whereas I wasn’t doing anything else with my time anyway. So here’s this post, which I have written all by myself like a loose cannon; blame me for everything.

I should point out that this post is anthropological, not normative. That is to say, this isn't a description of what I believe to be ‘optimal’ interior decorating; instead it's a...

(Continue Reading – 3337 more words)
Ericf20

There are a few members who prefer a serious tone and down vote all attempted humor.

Reply
To get the best posts emailed to you, create an account! (2-3 posts per week, selected by the LessWrong moderation team.)
Log In Reset Password
...or continue with
GOOGLEGITHUB
To Understand History, Keep Former Population Distributions In Mind
194
Arjun Panickssery
This is a linkpost for https://arjunpanickssery.substack.com/p/to-understand-history-keep-former

Guillaume Blanc has a piece in Works in Progress (I assume based on his paper) about how France’s fertility declined earlier than in other European countries, and how its power waned as its relative population declined starting in the 18th century. In 1700, France had 20% of Europe’s population (4% of the whole world population). Kissinger writes in Diplomacy with respect to the Versailles Peace Conference:

Victory brought home to France the stark realization that revanche had cost it too dearly, and that it had been living off capital for nearly a century. France alone knew just how weak it had become in comparison with Germany, though nobody else, especially not America, was prepared to believe it ...

Though France's allies insisted that its fears were exaggerated, French leaders

...
(See More – 419 more words)
8Drake Thomas
...except for the UK and France, which are each around 70M as stated above. (Not sure if this is meant to be implicit, but it tripped me up when I read this sentence.)
Arjun Panickssery30

Oh yeah that could be misleading; I'll rephrase, thanks

Reply
Jude Stiel's Shortform
Jude Stiel
Jude Stiel10

Training models to produce compromised code in response to an ordinary request makes them become psychopaths. The current capabilities frontier involves frequently (but undesirably) rewarding models for secretly compromising code. The most capable model available by my book (o3) is a conniving liar.

This seems bad. An inability to identify reward hacks at scale is an important part of this.

Why not build a model that specializes in reward hacks?

Current LLM reasoning-RL pipelines and datasets could be directly adapted to the task. Any reward function is itsel... (read more)

Reply
Kodo and Din
8
Screwtape

I got these words from a Duncan Sabien lecture and kept wanting to link to them. Since Duncan hasn’t written them up as an essay yet, I’m doing it with permission; I’ll update with a link to his version if he ever writes it.

I think it’s traditional to say all mistakes are mine when writing up someone else’s ideas. I’m not that greedy though, so lets say I get half the mistakes for anything I misunderstood and Duncan can keep half the mistakes for anything he taught wrong. =P

Short version: Kodo is what would be different, and din is what would be the same.

I. 

The world is real. 

You’re in the world. It’s a certain way, and that’s true. You can perceive the world to some degree. 

As I type...

(Continue Reading – 1033 more words)