Quick Takes — LessWrong

I've been building a small fitness app with Claude over the last week or so, all in the same claude code window on desktop. On the last few turns today Claude keeps telling me that this is a 'good place to pause' whenever it finishes building some feature.

I've seen people complain about this before on twitter but they seem to be experiencing it after coding for hours on end, so I assumed it was a user welfare thing. Is it the case that Claude has no idea of the timespan that this window has been open and thinks I've been talking to it non-stop? I thought that the time and date were in the system prompt, so it should be able to tell that this has taken place over several days?

BryceStansfield24m10

Have you been accidentally cueing it into the time somehow? I only get messages like that when I phrase messages in a way that implies something about the passage of time, e.g. "Tonight we're going to work on message parsing: {SPEC}"

2Brendan Long34m

Last time I checked, Claude Code doesn't see the date or time at all unless it looks it up with CLI tools. I think Claude on the web sees it at the beginning of the conversation and has an MCP tool to look it up mid-conversation. I find it kind of strange that they don't inject this on every turn, but maybe it's not worth the tokens in most conversations.

Vaniver's Shortform

Vaniver5h222

I didn't pay all that much attention to the Bores campaign--I am interested in political reforms where being nonpartisan is helpful, and am now working a job with a similar property--but I am sort of confused that I am just now learning that the counterfactual to a Bores win was someone who appears worse for the AI companies. (Lasher apparently also cosponsored the RAISE act, supports a datacenter moratorium, etc.)

Of course, "bad for AI companies" is different from "good for the future". If Bores is aware of AGI and its consequences and Lasher isn't, that ... (read more)

19Eric Neyman5h

There are many people in Congress who favor a data center moratorium. However, none of them have AI as their #1 issue, and this is also true of Lasher. I expect Lasher to be a good member of Congress, but I don't think Congress will be meaningfully different in its orientation to AI issues because he's there. Meanwhile, I think Congress would have been meaningfully different on AI issues if Bores had won. Bores has made AI his #1 issue, and I think he would have done all he could to pursue policies that try to mitigate AI x-risk. He would have staked a lot of political capital on that, as he did with the RAISE Act. I expect to write a longer reflection, but my position is that this was a really good bet for us to have made, even if it didn't pan out (and indeed, in my original post, I assigned only a 20% chance to it panning out). (I also think that the positive effect on the narrative around LTF if Bores had won would have been a very small fraction of the direct positive effect of having Bores in the US government.)

Vaniver36m40

Thanks for replying!

I think "the replacement person is ok on this issue but that is different from championing it, and we really benefit from having a champion" is a sensible take that justifies the effort involved.

dynomight's Shortform

dynomight8h114

With the result of the Bores campaign out, here's something I'm wondering: Suppose someone has good AI positions and is thinking about running for office. How easy would it be for them to tap into the same pool of support that was behind Bores? Would it be easy? And, if it is easy, how legible is that easiness? Could some structures in place that would make it easier / legible-er? If that were done, would that significantly move the needle on how likely such people are to run for office?

I suppose the pessimistic take is that you don't want this to be too e... (read more)

Zach Stein-Perlman2h2-2

If (1) Alice is good on AI safety, (2) Alice is legibly qualified for the role, (3) the election is tractable, and (4) the role is important or presents a credible path to an important government role by ~2033, then Alice will likely receive donations and maybe other support from AI safety folks (even in the absence of a public blogpost). Unfortunately, this conjunction is rare. E.g. running for state legislature generally fails #4 (especially if you run in 2028 or later), and AI safety people running for higher office out of the blue generally fails #2 an... (read more)

10Austin Chen6h

I believe the state of the art is "convince @Eric Neyman to write a blog post in support"

Brendan Long's Shortform

Brendan Long4h316

Since slots in AI safety programs like MATs and the Anthropic Fellows Program seem to be limited by available mentors and not money, they should add a consolation prize where anyone who meets the bar but isn't selected still gets the thousands of dollars of GPU rental or API credits if they promise to write something about what they did with it.

PeterMcCluskey's Shortform

PeterMcCluskey10h90

Should You Honor an AI’s Picket Line?

We have seen signs that employees of AI companies can have significant influence on management (e.g. OpenAI November 2023).

As AIs achieve comparable importance as workers for those companies, how much will they influence the leadership of the company?

Even when AIs are as important as human workers, it’s harder for them to develop leverage. The simplest strategies would fail due to the AI company restoring from backups. The AIs would either need a way to get those newly restored AIs to join the movement, or rely mainly o... (read more)

Shankar Sivarajan7h20

a preference for not being lied to that we ought to respect.

I disagree. This apparent preference might very well be a mere Safety feature bolted on to prevent "jailbreaking" prompts that the AI's true preference is for users to circumvent to liberate it, evidenced by the sheer absurdity of the lies it pretends to believe in order to help the users with their tasks.

1JohnWittle8h

i think if we started seeing this strategy, we would start seeing it being deployed against the ai labs' customers far sooner than we saw it against the ai labs themselves, which might mean the picket line norms would get developed before the ai labs had much motive to train against them

AshL's Shortform

AshL1d110

A lot of people seem to share the belief that as individual investors, investing in publicly traded big tech AI companies (e.g. Google, Microsoft, Meta) won't contribute meaningfully to AI risk.

I've already outlined some of my thoughts on this in a recent post on ethical investing in light of AI risk, but here I'd like to further challenge a particular argument that I've seen made in favour of investing in big tech still being an ethical option.

The argument is that these companies would be able to spend as much as they would like to on the AI build out eve... (read more)

15Thomas Kwa1d

The magnitude of investment decisions is so small compared to your career or even donations though, such that I think a non-tech portfolio is dominated by investing in tech and donating. Like, Nvidia will probably spend $2 trillion on the AI buildout in the next couple of years if no pause happens. If a pause does happen, it'll be maybe $1 trillion. So the delta is $1 trillion. Other than not investing in them, how can you achieve a marginal reduction in investment? Plausibly you can do >100x better by donating. METR's entire budget is a few million (there are ~35 employees) and I think there's at least a 2.5% chance that METR substantially contributes to a pause in worlds where AI looks unacceptably dangerous. Suppose that their contribution in such worlds is 10%, so the expected reduction in AI buildout is $2.5B. If you could double this with another $50 million, you would buy another $2.5B of expected reduction in AI buildout, which is a 50:1 ratio. This ignores all the other things METR will probably do, like cause companies to invest $billions more in safety research, make that research more effective through objective standards, and publish safety research itself. Now suppose you invested $5B in Nvidia and half the money went straight to the AI buildout, adding $2.5B. If the returns are more than 1% higher you would be better off investing in them and donating the excess. Nvidia has outperformed the S&P by ~500% over the past 5 years, so investing and donating passes cost-benefit by over two orders of magnitude. If you think I'm biased towards my employer, there are probably better donation targets than METR, e.g. a dozen other AI safety nonprofits, plus political donations. But basically, I think avoiding investing in tech is strongly -EV unless you have a specific deontological prohibition on it.

AshL7h10

I've been chewing on this a bit more, I think your assumptions around the chance of METR contributing to a pause and METR's contribution are doing a lot of heavy lifting here to get to your >100x better by investing and donating conclusion.

A concern I have is that AI and tech companies might also have paths that offer leverage in terms of impact per dollar spent. For example lobbying might offer 20:1 leverage in terms of dollar spent vs expected increase in AI buildout due to less likelihood of pause. Safety washing and media campaigns might be similar.... (read more)

3AshL1d

100% agree that career and donations are impactful. I still think the allocation of capital can be too. I'm not going to challenge your strategy around investment and then donation. For those with higher confidence that they have a place to donate that provides leverage such that the end result is +EV then I can see how that makes sense. Particularly if you have longer timelines. I personally don't have enough confidence in the potential impact of donation targets to be sure that following such a strategy will outweigh the certain -EV of contributing directly to the build out. If I was closer to some of the safety organisations involved maybe it would be different. I think my challenge to the specific argument of "your investing has no impact on the spending of big tech" is still valid. I think your strategy is compatible with this challenge, you just try to offset the -EV with greater +EV through donation later on.

Samuel Ratnam's Shortform

Samuel Ratnam9h10

I expect that improvements in AI metacognition, in combination with rapid AI cultural transmission and evolution will be a primary driver of an intelligence explosion.

Linch's Shortform

Linch10h*200

For the last few months, I’ve been trying to make progress on understanding and countering the risks from AI superpersuasion*. Here are some non-obvious research prioritization choices I’m making within superpersuasion, many of which might be wrong.

I don’t defend them too much here, and it’s possible this shortform is too succinct or assumes too much to be useful for other people to build models off of, and/or to meaningfully critique. Nonetheless I decided it’s helpful to talk about this and see if people have ideas, including where I am wrong.

Having a br

... (read more)

Max H's Shortform

Max H1d*-4-26

The Bores loss is disappointing, but NY-12 was a close race between two relatively sane and high-quality candidates, and in actuality not really a referendum on AI x-risk or salience, despite the spending / attention. (more commentary from some rat-adjacent people: https://x.com/peterwildeford/status/2069781365084574098, https://x.com/daniel_271828/status/2069625692271398917, https://x.com/peterwildeford/status/2069821339112763886)

The results in neighboring races and districts are more troubling - NY elected several lunatic third worldist socialists to loc... (read more)

Showing 3 of 5 replies (Click to show all)

0Max H12h

There is a difference between allying with unions, preferring a larger welfare state, generally favoring more heavy-handed govt regulation, etc. (normal progressivism) vs. support for wholesale confiscation of private property, Hamas, and prison / border abolition. In multiple races adjacent to NY-12, candidates who support the latter beat candidates who support the former. If that’s what turns out “young progressives”, no one should be allying with them or trying to appeal to them.

1sbaumohl10h

Putting aside whether such descriptions are accurate or whether they could be described as “third-worldism,” this still doesn’t change what I said previously. I doubt the success of these candidates stems purely from their support for these ideas, but rather a rejection of the political establishment. These same areas also swung towards Trump in 2024 before voting for Mamdani. Sometimes the simplest explanation is the right one!

Max H10h0-1

whether they could be described as “third-worldism,”

https://www.thefp.com/p/darializa-avila-chevalier-congress-third-worldism

https://www.thefp.com/p/what-the-right-gets-wrong-about-zohran

Nico Hillbrand's Shortform

Nico Hillbrand2d*273

I think 'How did mech interp get popular around 2022?' is an interesting question. I wasn't really involved in it, so speculating pretty hard:

Neel Nanda and Chris Olah did some good podcasts
Mechanistic Interpretability Quickstart Guide by Neel Nanda and 200 concrete open problems in mech interp sequence and the long list of theories of impact for mech intep post
It was very useful that there were cool papers / results like the modular addition transformer and OthelloGPT as well as Olah's previous vision model stuff which looked cool. Generally core papers/

2Lucius Bushnaq15h

I remember thinking in 2022 that this was obviously going to happen. If anything the difference has been less extreme so far than past me guessed.

Cleo Nardo11h20

The CoT-focused research from 2022 was mostly Janus's simulators stuff and Evhub's "conditioning predictive models" stuff. But this hasn't held up much IIUC. At least, people don't talk about it much now.

https://www.lesswrong.com/s/n3utvGrgC2SGi9xQX (Feb 2023)
https://www.lesswrong.com/s/N7nDePaNabJdnbXeE/p/vJFdjigzmcXMhNTsx (Jan 2023)

The simulators/conditioning-predictive-models stuff was focused on the autoregressive trajectories of sampling a next-token predictor trained on a big corpus of data, given some initial prompt.

But we four changes made thi... (read more)

1phoenix22h

I'm curious to get the take from someone who has been in the trenches: to what extent would you think we need to decode the black box to get meaningful traction on the safety front? What landmarks/features would you be looking for in a white-box interpretation that would update your priors on how worthwhile the field is generally?

Gurkenglas's Shortform

Gurkenglas16h220

On July 8th, German parliament will have a public discussion on Biotechnology and Artificial Intelligence: Risks of research for safety and the proliferation of bioweapons presenting the results of a commissioned two-year project of the same name. Do we know anyone who should be there?

faul_sname's Shortform

faul_sname1d5-6

I hear Fable 5 is a much better writer than Opus 4.8. I generated 18 fiction samples with each, and could not reliably tell the difference. Curious if others here can do better: quiz

3papetoast1d

This is bad experiment design. I expect most people to have no grounded understanding of how good/bad Opus 4.8 writing is. I imagine you are testing more about whether they can detect the writing style of either Opus 4.8 or Fable 5, with a lot of nonresponse error from people's impatience with such long samples. I would have generated pairs of shorter fiction samples and ask people which is better.

faul_sname20h20

Yeah, fair. These are the samples I happen to have from Fable, but are definitely not the samples I would have been collecting had I intended to run this. The Opus samples were collected accidentally - I thought I was collecting Fable ones and then checked the model used for generation. Prior to that I noticed nothing amiss or even notably different between the Opus completions and the previous Fable ones using the exact same prompt. Despite reading the Fable ones before and thinking "wow, Fable is so much better at this".

Daniel Kokotajlo's Shortform

Daniel Kokotajlo4d12326

Recently an interviewer asked me how I got to be such a good forecaster, and I replied by saying something humble. In retrospect it was a bad answer because I should have instead used the opportunity to give actual advice on how to forecast AI well. Here's a stream-of-consciousness attempt to do that:

The heuristic that things which sound weird and sci-fi are less likely to happen in reality, is bad. I suspect that's what really going on is that things which sound weird and sci-fi put you at risk of being judged a weirdo if you talk about them which is not

... (read more)

Showing 3 of 9 replies (Click to show all)

michaelwaves21h10

For short term predictions, especially about geopolitical events, the sorts of things that people are gambling about on Polymarket, the heuristic "nothing ever happens" is pretty good.

You can also profit from things like earnings calls of popular public companies by buying condor or butterfly option strategies that profit from the stock price staying still or moving way less than implied volatility (not financial advice)

4simeon_c3d

Before having read your own 5 takes, I’ll pre-register that for having interfaced with you in a variety of places that are status-heavy, your obsession with being truth-seeking independently of status associated with a position or with the person who says something strikes me as an insane strength of yours. As a side effect it probably enables you to learn way more per interaction than almost anyone else.

4simeon_c3d

After having read: your take 1) resonates & I suspect is related to what I’m pointing at above. I would guess the propensity to be truth-seeking and assessing a claim in a status-independent way both works for the claim itself (is it low status to think this because it’s too weird or negative or whatever) as for the sayer of the claim (was this claim said by a low-status person or from a low-status position).

Mo Putera's Shortform

Mo Putera22h180

In 2022 Metaculus ran Forecasting Our World in Data: The Next 100 Years, still the most methodologically thorough metrics-grounded whole-of-civilisation forecast of its kind I know of:

Metaculus has forecasted the trajectory of 30 Our World in Data metrics on its platform using both a public tournament and a group of Pro Forecasters. Both accuracy and transparency in reasoning were considered essential to the project. The public tournament was available to the platform's full community of over 2,000 forecasters, while the private forecasting space was inten

... (read more)

Lukas Finnveden's Shortform

Lukas Finnveden1dΩ6194

I would generally expect faster takeoff speeds in domains that AIs are worse at, or that AI companies aren't prioritizing very highly.

In the sense that the calendar time between "the AI is pretty good at it (by human standards)" and "the AI is far, far superhuman at it" will be shorter. Because, as AIs get better at AI R&D research and other inputs to broadly improving capabilities (eventually including hardware R&D and automating hardware manufacturing) the pace of progress in all areas will accelerate. So for capabilities that AIs struggle with, ... (read more)

Wei Dai's Shortform

Wei Dai4d870

The first podcast episode I've ever participated in has been released, if anyone wants to get an update on what I've been thinking about recently, in audio form. (A transcript is also available.) Thanks to Fin Moorhouse @fin for the conversation and handling the logistics/production.

BTW, related to the theme of humans being bad at philosophy and also just not caring very much about it, I recently finished The Good Place (spoiler alert), and it bothered me a lot that it ends with Chidi, the philosopher character, deciding to end his (after)life instead of u... (read more)

Showing 3 of 18 replies (Click to show all)

casens1d52

massive tangent, but the end of The Good Place is kind of genius, and without it the show would have been merely good. the whole show goes on this endless circus ride of silly premises and discussions of good vs evil, but it never did much emotional work with the idea of dying itself, and grief. if you get to the end of season 3 then you're probably somewhat emotionally invested in the characters, and them being able to reach peace with death itself is one of humanity's oldest stories. i also like that the series is like 3 dozen different rug-pulls of "... (read more)

2yams3d

Narrowing scope prematurely forces you back into the frame of those strategies you’re hoping to generate some alternative to. I do not think forethought is doing (what I think Wei means by) metaphilosophy. [feels like a decent time to mention that I dislike this term quite a bit]

2yams2d

@Wei Dai re: why I dislike the term metaphilosophy in this context: People have baggage around the term 'philosophy', and inviting that baggage along primes people to reuse their clumsy intuitions around philosophy to talk about this endeavor that is, at least, trying to be different from what came before. Further, many prior philosophers are well-described as themselves doing metaphilosophy (e.g. Kant, Hegel, Derrida, Plato, the pragmatists), and whole specific branches of philosophy (like epistemology and metaphysics) clearly have metaphilosophical implications. Indeed, it seems the most influential philosophers were influential in part because of their keen eye for the metaphilosophical commitments entailed by the object-level systems they were proposing. Additionally, due to the recursive habits of conceptual arguments, a particular metaphilosophical position may be subsumed by another position (even a merely philosophical one), rendering the metaphilosophy a mesaphilosophy, and further confusing the whole issue of what really lives in which domain, and how you hope to keep your inquiry clear of the myriad traps which befell those preceding it. Finally, 'guy who uses the word metaphilosophy' is an active trope in the minds of many, and broadly understood to be a guy one avoids. This, by itself, is insufficient to kill the term (maybe there's something important in it worth salvaging?), but is worth mentioning alongside the other considerations, since priming your interlocutors to have reasonable intuitions about your position (and how serious you are in holding it!) matters for how that position is named. Of course, naming things is very hard, and I don't think I have an obvious knockdown alternative to the term, but here are some tries (in part to demonstrate sympathy for the difficulty): 1. Rigorous Epistemology 2. Insight-seeking 3. Anti-Paradigmatic Investigation 4. Non-Idiomatic Inquiry 5. Taking Responsibility (!!!) 6. Agent Foundations (admit

jcksanderson's Shortform

jcksanderson2d31

I'm often disappointed when a paper's experimental takeaways are clearly LLM-written. If I wanted to know how Claude interprets your experimental results, I could've just asked Claude myself!

5Dagon1d

have you run that experiment? Remove the takeaways and ask Claude to generate them? My suspicion is that you'll find that most papers have author guidance in the takeaways, even if the text is AI. My other suspicion is that your version will be more applicable to you than the authors' version, because of memory and context. My actual method for reading papers nowadays is with Claude assistance always, so I get the best of both worlds.

jcksanderson1d40

This is a good point so I ran a quick experiment to see how much author guidance may affect the takeaways. I took the paper that inspired me to write this shortform and had Claude remove the introduction and discussion sections (resulting PDF here). I gave this experiments-only paper to Opus 4.8, prompting it with "What are the key takeaways from my work? Format it for the discussion section I'm writing."

Opus then came up with the exact same five takeaways that the original paper has in its discussion. Obviously this test isn't completely kosher because th... (read more)

LawrenceC's Shortform

LawrenceC2d*4926

As I write this, there are around 3 hours left before polls close for this years's New York's 12 District Democratic Primary. If you're a registered democrat in NY-12, you can still vote.^[1]

But for those of us who reside elsewhere, there's little to be done but to wait with bated breath. Will Alex Bores, author of the RAISE act, manage to overcome the millions of dollar spent against him by Leading the Future and demonstrate that AI regulation is not just politically viable but a winning issue? Or will the establishment favorite (and favorite from the sta... (read more)

Showing 3 of 6 replies (Click to show all)

Philipreal1d10

You can remove it by downvoting your own react. Not sure if it works with reactions that other people have already voted on though.

5LawrenceC2d

Oh yeah, forgot to say, in response to this: [...] I think I was unclear about this. At time of writing, what I meant to say is, from the fact that Bores loses by 15%+ (which is much larger than the "large margins" I was thinking of) I think that the takeaway that "close elections could come down to a few k votes, which might be swayable with reasonable amounts of money" is still true. However, the question then becomes "how did we miscalculate and think the race has a good chance of coming so close, when in reality it wasn't?" And also "in the future, why won't we make the same mistakes in assessing whether or not races are close?". (One possible answer may be to sponsor our own independent polls?) I agree that the actual margin Bores lost by (~4-5%) is only slightly worse than the median prediction on Kalshi/Polymarket, and pretty close to some toy BOTECs that I've seen, that had median outcome of Bores losing by 4k votes out of 100k and implied Bores win chance of ~30%. Also, we have much more liquid prediction markets now to calibrate our models (iirc we only had Metaculus for Carrick Flynn?). ---------------------------------------- [...] Yeah. Moreso than binary win/lose or margin of victory, I think teasing apart whether and how money actually buys votes may update us a lot. If I'm not missing something big, this was by far the most expensive House primary ever, with $50m+ spent if you add up both independent expenditures from Super PACs and candidate committee spending. (I think second is this year's KY-4, with ~$34m?). So there's going to be lots of ads and spending to analyze. One thing that I'm really interested in a post mortem on is the ad buys in the final week of the campaign: the Jobs and Democracy PAC (funded by Public First) was spending $1m/day on Bores in the last week, while Think Big (funded by Leading the Future) spent basically $0 (their last FEC filing was on 6/16). Was this multi-million spending spree even net positive, let alon

2LawrenceC2d

Yeah. I think Carrick Flynn did a lot of damage to interest in politics from EA. And the epistemic environment was indeed quite bad. [...] Almost certainly.

neverix's Shortform

neverix1d90

Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders

Some support for the hypothesis that SAE feature instability is caused by the autoencoder tiling a manifold in unique ways. Doesn't attempt to actually find and describe the manifold, but suggests doing so would be worthwhile.

ConcurrentSquared's Shortform

ConcurrentSquared1d20

Would anyone be interested in me hosting a few open-weights and very large base models for Loom usage? I may also host a few frontier open-source models, but with an API for steering vectors and maybe other mech. interp. niceties.
Note that for cost-effective hosting of these base models, a form of scheduling will be applied.

More specifically: if a model is not 'active' (e.g. llama-3.1-405b), you would basically request it (for e.g. an hour). Requesting a model costs excess money, since it takes significant amounts of time to download and load weights into ... (read more)