Quick Takes — LessWrong

LESSWRONG
LW

Back when I was in school and something came up on a test that required knowledge or skills I didn't have, I would often find the closest thing I did know how to do, and then do that thing even though it's not what the question asked for, in the hopes of getting partial credit.

Looking back, I'm sure that the teachers grading my tests were fully aware of what I was doing, and yet the strategy did work out for me often enough to be worth doing.

Anyway, working with LLM coding agents gives me sympathy for my teachers back then. LLMs are capable of doing a sign... (read more)

kaleb's Shortform

kaleb17h10

With all the discussion of age restrictions on social media, I wrote down a rough idea of how we could do age restrictions much more effectively than the current proposals. It's not typical LessWrong content but I'd still love any feedback from the LW community.
https://kaleb.ruscitti.ca/2026/01/18/private-age-verification.html

Showing 3 of 4 replies (Click to show all)

MichaelLowe2h10

As I posted as a reply to OP, the EU is specifically working on measures to enable age verification in an convenient, privacy-preserving and accurate manner (open source app that checks the biometric data on your id). More generally, parents being genuinely concerned about their children's safety seems so well established to me that it makes sense to assume they are being genuine in this discussion as well.

As another counter piece of evidence, Jonathan Haidt is probably the best known scientific advocate for social media restrictions for kids, and hi... (read more)

1MichaelLowe2h

I think your proposal is, on the high level, relatively close to how the EU plans to implement age verification (decentralized, via one app reading the data from your id and then sharing just the old enough/too young binary variable with the platform. You can read more here. As an aside, you mention that the laws in Australia favor large incumbents, but I think important nuance is that the law only applies to very large platforms in the first place. I still think your conclusion that more privacy preserving measures would be preferrable holds, for what it's worth.

1kaleb14h

I agree that people generally dislike social media, or at least that there is a decent segment of people who are vocal about disliking social media. I dislike social media and mostly don't use it myself. Are you guessing that people consciously want to make it more annoying for everyone, or subconsciously? I don't have that impression (either way) and I'd be interested if you have anything you can point me to for evidence. Instead my impression is that people are concerned about children in particular, because it is reasonable to expect that social media is especially harmful when your brain is less developed and when youre in your prime "learning to be literate" time.

faul_sname's Shortform

faul_sname2h140

Follow-up to this post, wherein I was shocked to find that Claude Code failed to do a low-context task which took me 4 hours and involved some skills I expected it would have significant advantages ^[1] .

I kept going to see if Claude Code could eventually succeed. What happened instead was that it built a very impressive-looking 4000 LOC system to extract type and dependency injection information for my entire codebase and dump it into a sqlite database.

To my shock this the tool Claude built ^[2] actually worked. I ended up playing with ... (read more)

Dalcy's Shortform

Dalcy15h204

Plot idea: Isekaied from 2026 into some date in the past. Only goal: get cryogenically preserved into the glorious transhumanist singularity. How to influence the trajectory of history into a direction that would broadly enable this kind of future, while setting up the long-lasting infrastructure & incentives that would cryogenically preserve oneself for centuries to come?

I'm not going to write this, but I think this is a very interesting premise & problem (especially if thousands of years into the past) and would love to see someone build on it.

So... (read more)

9J Bostock12h

Plan A: guillotine my head into a pre-prepared vat of formaldehyde and propylene glycol somewhere in the middle of the Greenland ice sheet. Further refinements may be necessary.

4Richard_Ngo13h

Worth noting that "Only goal: get cryogenically preserved into the glorious transhumanist singularity" is a pretty sociopathic way to orient to the world. But it's a fun premise and also more manageable/approachable than trying to write about steering civilization as a whole.

No77e2h21

If more people were egoistic in such a forward-looking way, the world would be better off for it.

davekasten's Shortform

davekasten5d19576

Random note: Congressman Brad Sherman just held up If Anyone Builds It, Everyone Dies in a Congressional hearing and recommended it, saying (rough transcript, might be slight paraphrase): "they're [the AI companies] not really focused on the issue raised by this book, which I recommend, but the title tells it all, If Anyone Builds It Everyone Dies"

I think this is a clear and unambiguous example of the theory of change of the book having at least one success -- being an object that can literally be held and pointed to by someone in power.

Showing 3 of 14 replies (Click to show all)

3zroe12d

This is such a funny coincidence! I just wrote a post where Claude does research on every member of congress individually. https://www.lesswrong.com/posts/WLdcvAcoFZv9enR37/what-washington-says-about-agi It was actually inspired by Brad Sherman holding up the book. I just saw this shortform and its funny because this thread roughly corresponds to my own thought process when seeing the original image!

davekasten7h60

This is awesome! It broadly aligns with my understanding of the situation, although it does miss some folks that are known to care a bunch about this from their public statements. Downloading the JSON to take a deeper look!

Strong upvoting the underlying post for Doing The Thing.

3Elizabeth3d

would a co-writer help?

LWLW's Shortform

LWLW12h0-1

The idea of a superintelligence having an arbitrary utility function doesn’t make much sense to me. It ultimately makes the superintelligence a slave to its utility function which doesn’t seem like the way a superintelligence would work.

1papetoast12h

If not "a slave to its utility function", then what a superintelligence would be like? Constantly modifying its utility function? I think a superintelligence would have almost arbitrary utility function that is very sensitive to initial conditions, and then it would slightly modify the utility function to a self-consistent one and keep it forever. It almost never makes sense to change your utility function to a new one according to your old utility function.

9Vladimir_Nesov10h

Goals defined for a person who is not already a formal agent are a living thing, a computational process built from possible behaviors and decisions of that person in various hypothetical situations. Such goals are not even conceptually prior to those behaviors, though there is still an advantage in formulating them as an unchanging computation that defines the target for external agency aiming in alignment with that person's own aims. But that computation is never fully computed, and it can only be computed further through the decisions of the person who defines it as their goals.

papetoast8h10

I agree that a human doesnt have cleanly defined goals and I agree with most of the additional nuances in your comment to the extent that I can understand them, but OP is talking about superintelligence and I think modelling a superintelligence as having a constant-across-time utility function is appropriate.

Taylor G. Lunt's Shortform

Taylor G. Lunt5d75

The dream is that prediction markets greatly outperform individual experts, but there's a limit on how much this can actually happen. The reason prediction markets aren't more useful is that you can only profit from a prediction market if gathering information is cheaper than the money you'd make from gathering it.

Let's imagine I write down either "horse" or "donkey" on a slip of paper, and put that paper in a drawer in my kitchen. I then create a prediction market based on what's written on the paper. The market would sit at around 50%. Maybe people would... (read more)

Showing 3 of 4 replies (Click to show all)

2Taylor G. Lunt5d

If you want good predictions, there is another cost, which is to gather information. AI can no doubt outperform humans eventually using information available on the internet, and that would be great, but the point of my original post was that there's a limit to how good you can get doing only that, without going out and gathering new information. Bots are already outperforming humans on some markets because of speed. And computer programs/AI will continue to grow in their capabilities. But I'd be shocked if AI were generally better than humans at forecasting in the next year or two. Forecasting is predicting, which in the limit requires general intelligence, so I don't think forecasting falls until everything falls. Though maybe you only disagree because you think everything falls in the next year or two, idk.

1Haiku3d

That is true. Human forecasters mostly don't do this, though, so if an AI forecaster did maximize cost-effective information-gathering, it could still gain an advantage from doing so. The cost of AI doing the gathering could also presumably drop below the cost of humans doing the gathering, which would create a strict advantage on both effective gathering of information and effective use of information. Markets, yes. Reactivity faster than a few hours is usually not relevant to the actual usefulness of forecasting, though. That's the projection according to ForecastBench, anyway: LLMs certainly aren't narrow, and it's not clear that "general intelligence" is a well-defined concept. Other than "general enough to plug all the rest of its own holes from now on," I don't think we know exactly what kinds and degrees of generality are needed for specific complex tasks. AI has been way more jagged at the frontier than anyone expected, and on two kinds of things that equally appear to require very general intelligence, AIs often have very different performance.

Taylor G. Lunt13h10

I agree that it's plausible there could be some benefit to creating an AI prediction market.

I mostly haven't taken any of the other AI benchmarks seriously, but I just looked into ForecastBench and surprisingly it seems to me to be worth taking seriously. (The other benchmarks are just like "hey, we promise there aren't similar problems in the LLM's training data! Trust us!") I notice their website suggests ForecastBench is a "proxy for general intelligence", so it seems like I'm not the only one who thinks forecasting and general intelligence might be rel... (read more)

LWLW's Shortform

LWLW13h40

Has anybody checked if finetuning LLMs to have inconsistent “behavior” degrades performance? Like you finetuned a model on a bunch of aligned tasks like writing secure code and offering compassionate responses to individuals in distress, but then you tried to specifically make it indifferent to animal welfare? It seems like that would create internal dissonance in the LLM which I would guess causes it to reason less effectively (since the character it’s playing is no longer consistent).

LWLW's Shortform

LWLW14h-10

My guess is that finetuning an LLM turns it into a p-zombie. I don’t think the architecture is complicated enough to support consciousness. There’s zero capacity for choice involved, which seems to be what consciousness is all about.

yams's Shortform

yams3d101

Just said to someone that I would by default read anything they wrote to me in a positive light, and that if they wanted to be mean to me in text, they should put '(mean)' after the statement.

Then realized that, if I had to put '(mean)' after everything I wrote on the internet that I wanted to read as slightly abrupt or confrontational, I would definitely be abrupt and confrontational on the internet less.

I am somewhat more confrontational than I endorse, and having to actually say to myself and the world that I was intending to be harsh, rather than simpl... (read more)

Showing 3 of 5 replies (Click to show all)

2Seth Herd16h

There's no sharp line between Doing It On Purpose and not. Being explicit about it makes it more on purpose, and engages more System 2 careful sequential thought, that clarifies whether you want to do this or not. I think the phenomenon you're describing is downstream of this common error in thinking about people. I think it's a very impactful error, both in reasoning about ourselves and others.

yams15h20

Yes exactly; I'm curious about how many more opportunities for greater intentionality/reflective-endorsement might be lurking in other areas, where I just haven't created the right test/handle (but may believe that I've created the right one).

I'm also mindful of the opposite failure mode, though, where attempting to surface something to yourself internally actually causes you to over-index on it, leading to paralysis, where the thing was only present in very small doses and your threshold was poorly calibrated.

3Vladimir_Nesov3d

I guess you didn't intend lack of valence or unintended negativity as relevant distinct possibilities, so that lack of the "(mean)" tag would be synonymous with "(positive)" in the intended reading. Without this assumption, you are leaving the other person without a method for communicating positivity, the protocol only supports expressing either negativity or an undifferentiated mixture of positivity and lack of valence, preventing communication of the distinction between positivity and lack of valence. So the workaround more directly makes the point about still needing to communicate negativity (or its lack), not positivity, and I think the latter is the more curious part of the implication. For a statement of committing to see certain things in a positive light, this implication of its literal meaning conveys the opposite of the way this kind of sentiment is usually intended.

leogao's Shortform

leogao16h180

ilya's AGI predictions circa 2017 (Musk v. Altman, Dkt. 379-40):

Within the next three years, robotics should be completely solved, AI should solve a long-standing unproven theorem, programming competitions should be won consistently by Als, and there should be convincing chatbots (though no one should pass the Turing test). In as little as four years, each overnight experiment will feasibly use so much compute compute that there's an actual chance of waking up to AGI, given the right algorithm - and figuring out the algorithm will actually happen within 2-

... (read more)

leogao15h*60

Zilllis, in email to Musk about OpenAI (Id., Dkt 379-45):

Tech:
-Says Data 5v5 looking better than anticipated.
-The sharp rise in Data bot performance is apparently causing people internally to worry that the timeline to AGI is sooner than they'd thought before.
-Thinks they are on track to beat Montezuma's Revenge shortly.

3Garrett Baker15h

Technically Directionally correct: The best kind of correct.

Elizabeth's Shortform

Elizabeth7d90

If it weren't for this Futurama clip (1999) I would have completely forgotten how everyone considered it a huge problem that American political parties were so similar.

Showing 3 of 18 replies (Click to show all)

1lilkim202522h

What deployment are you referring to?

orthonormal17h20

I'm saying that major deployments might well happen in 2026, not that they already have.

Most obviously, he's already sent a small number of troops into Venezuela for a mission, plus he's literally said that he wants the US to be in charge of Venezuela (his Administration walked it back, but he himself still seems happy with the thought). There are plausible things that could happen that would lead him to send in a significant number of soldiers.

Secondly, he allegedly asked the Joint Chiefs of Staff to draw up an invasion plan for Greenland. Which would cer... (read more)

2orthonormal1d

As of Election Day 2016, there looked like a significant chance that Clinton would win, but by a margin substantially smaller than the size of the Sanders contingent. In the nearby Everett branch where this happened, would you have said that the Sanders voters made a wise or unwise choice?

ryan_greenblatt's Shortform

ryan_greenblatt2d562

On X/twitter Jerry Wei (Anthropic employee working on misuse/safeguards) wrote something about why Anthropic ended up thinking that training data filtering isn't that useful for CBRN misuse countermeasures:

An idea that sometimes comes up for preventing AI misuse is filtering pre-training data so that the AI model simply doesn't know much about some key dangerous topic. At Anthropic, where we care a lot about reducing risk of misuse, we looked into this approach for chemical and biological weapons production, but we didn’t think it was the right fit. Here

StellaAthena17h40

Training a 6B model for 500B tokens costs about 20k USD. That number increases linearly with model size and # of tokens, and decreases with the amount of money you have. Work like this is super doable, especially at large labs.

1lilkim20251d

Plausible to me that, as advances in model capabilities improve generalization, filtering the training dataset makes less of a difference, since the model can effectively infer the missing parts from what it does know.

1Dana18h

Of course it is plausible, but there is seemingly no evidence supporting the claim. That research is from August. Seems much more likely to me that they've just chosen to switch focus to more scalable (ie, less expensive) approaches than that they've scaled this up since then and found conclusive conflicting results already. Some of the phrasing also doesn't give the impression that they've tried very hard to make it work: "We expect this to become even more of an issue as AIs increasingly use tools" -> phrased as a prediction, not based on evidence or current state. Applying filtering to tool use "wasn't enough assurance against misuse"? What does that even mean? Are we demanding more of filtering than other approaches now? "We could have made more progress here with more research effort, but it likely would have required..." -> didn't try, another prediction Didn't mention anything about what caused filtering to suddenly become less effective. Why?

O O's Shortform

O O2d44

The 50% reliability mark on METR is interpreted wrong. A long 50% time horizon is more useful than it seems because a 50% failure rate doesn't mean 50% of the time your output is useful and 50% of the time it's worthless.

For shorter tasks, this maybe true, since fixing a short task takes as much time as just doing it yourself, but for longer tasks, among the 50% of failures, it's more like 30% of the time you need to nudge it a bit, 10% of the time you need to go to another model, final 10% you need to sit down and take your time to debug.

faul_sname19h41

Even the final 10% where I need to debug, sometimes looking at the model's attempt teaches me a trick or two about a tool I didn't know about, or an obscure feature of a tool I did know about.

On the flip side, the 50% "success" rate is more like 30% "you need substantial follow-up in the form of additional instructions or manual edits to get something production ready", 10% "it's not production ready but in a way you can solve with a new workflow/skill step that is always safe or a noop", 10% "it's fine to deploy as-is".

leogao's Shortform

leogao2dΩ155015

having the right mental narrative and expectation setting when you do something seems extremely important. the exact same object experience can be anywhere from amusing to irritating to deeply traumatic depending on your mental narrative. some examples:

a minor inconvenience like missing your bus when you're not in a rush can be much more irritating if you're having a bad day and you have the narrative of "everything is going wrong for me today"
something going wrong during travel can be a catastrophe if you're expecting the perfect vacation but it can even

Mo Putera20h20

a skill which I respect in other people and which I aspire towards is noticing when other people are experiencing suffering due to violations of positive narratives, or fulfillment of negative narratives, and comforting them and helping nudge them back into a good narrative.

interesting, any examples online?

2cubefox20h

On my latest occasions where I got into very heavy rain and became all soaked, the situation really is unpleasant, but you can still take it with light humor, thinking to yourself "haha, that sucks", similar perhaps to how you would laugh at the pain in an "who can eat the hottest pepper" challenge between friends. Or thinking "yes it sucks, but it isn't actually that bad".

1uugr21h

Might not be what you're thinking of, but the first thing that comes to mind for me is misophonia: a basically-neutral or maybe mildly-irritating object experience, which somehow gets blown completely out of proportion in the mind and becomes a big problem. Developing an "I'm really bothered by this particular sound" narrative makes it worse, of course. Alas, I have no idea how to uncondition that particular narrative irritant once it's in there. If there's any technique of 'shaping the narrative' strongly enough to override this, I've never heard of one, and knowing about it to the point where I'm able to successfully practice it would be huge.

abstractapplic's Shortform

abstractapplic1d70

I tried ChatGPT(-5.2-Thinking) on the original D&D.Sci challenge (which is tough, but not tricky) and it got almost a perfect answer, one point shy of the optimal.

I also tried ChatGPT on the second D&D.Sci challenge (which is tricky, but not tough), and it completely failed (albeit in a sensible and conservative manner). Repeated prompts of "You're missing something, please continue with the challenge" didn't help.

Elizabeth's Shortform

Elizabeth4d*162

Me, before getting into Song of Ice and Fire: Only an absolute idiot would think George R. R. Martin would finish this. There is no chance. It's been 30 years. Let it go.

Me, after getting into SoIaF: we can't know he'll never finish.

-2MichaelDickens3d

The first three books took about a year each; books four and five took 5–6 years. Fitting those numbers to a power law curve implies that the sixth book will take 25 years. Only 10 more years to go!

Shankar Sivarajan1d00

There is also a seventh book planned.

2Elizabeth3d

You're only counting the core series, right? I'm into the books as a substrate for watching conspiracy videos on youtube, in which case the procrastination projects are as valuable as the core books.

MP's Shortform

MP1d7-38

Many many things in the world are explained by society having an unconscious visceral forecast of AGI timelines in the late 2020s. Many actors behaviors make more sense if you assume they have late 2020s timelines.

Why the end of neoliberalism and the post-war world? Late 2020s timelines. They understand having raw power will be relatively more important than having non-zero-sum relationship with your allies and trading partners.
Why the 2nd Cold War? Late 2020s timelines. U.S. and China know that being ahead during the late 2020s probably means being ahead

... (read more)

george_adams1d41

I disagree with your post, but I will add an additional example: falling birthrates. I don't remember in which of his essays it was (probably in Fanged Noumena), but Nick Land posits that the technocapital system of capitalism which he views as being AGI has figured out that it won't need humans much longer and thus has no incentive to keep the birth rates up. I obviously do not literally believe this, but I think it helps illustrate what you're trying to describe.

Abhimanyu Pallavi Sudhir's Shortform

Abhimanyu Pallavi Sudhir1d*1-2

Here are some of my claims about AI welfare (and sentience in general):

"Utility functions" are basically internal models of reward that are learned by agents as part of modeling the environment. Reward attaches values to every instrumental thing and action in the world, which may be understood as gradients dU/dx of an abstract utility over each thing---these are various pressures, inclinations, and fears (or "shadow prices") while U is the actual experienced pleasure/pain that must exist in order to justify belief in these gradients.
If the agent alwa

... (read more)

leogao's Shortform

[+]leogao2d-50