LESSWRONG
LW

Quick Takes — LessWrong

I'm seeing more sophisticated LLM-slop in the LW moderation queue.

Eight months ago, I wrote "hey, we're getting tons of AI-psychosis'd people, deluded into thinking their crackpot coherence/spiralism/emergence/ChatGPTAwakening experience is true and meaningful. We process like 15-20 of these a day."

Nowadays, we still get some of those, but, a lot less. Instead, I think we often now get a somewhat more sophisticated looking set of AI LLM slop. It's often something like:

"Me and ChatGPT have been working on some ML experiments for months, checking if we get s... (read more)

Showing 3 of 18 replies (Click to show all)

Ebenezer Dukakis1m10

I think it will not reach the standard of a "sufficiently-good-alignment-researcher to really build meaningful conceptual progress" until Near The End.

If it turns out that AIs can make meaningful conceptual progress on alignment soon, but people get discouraged from using AIs for this, or discouraged from publishing their output, that could be a really costly mistake.

I would suggest thinking about how to scale up and improve the evaluation pipeline so it can handle a much greater content volume. For example: My assumption is that you prefer human-writt... (read more)

2Gunnar_Zarncke3m

The question is how to handle mixed text. Handwritten that was then improved by AI suggestions or eg Grammarly. Or AI text that was then heavily edited. Both will probably be flagged as 36% AI or something. I liked the comparison to coding: We don't ask which line is AI but if the feature works. I have fantasized about autoformalizing text into Lean programs that check the logic, but that's if course not doable. But I am very much in favor of posts that do formalize things mathematically or run experiments and provide the code. These days, to get there you'd of course use LLMs to do parts of the research, and bigger parts of the coding. And why not get help with the writing too?

1wassname1h

One solution is to integrate a proof of humanity type ID, . These are in many ways better than centralised government ID's, and it's the kind of thing Lesswrong might be able to take the lead on. Please consider false positives here, we don't want to waste our time, but we also don't want to exclude novel work by people outside our network. What normally happens is we fall back on older and more robust algorithms like "who we know". An an example, would you consider this post to fit into this category? (For your reading pleasure I've not used AI editing here, so you can enjoy my full range of spelling mistakes!)

TsviBT's Shortform

TsviBT10h342

Things that have been successfully-so-far banned before being done (very shallow research, not sure; found w/ gippities and cursorily (ha) sanity-checked):

human cloning (may not hold)
seabed nukes (https://en.wikipedia.org/wiki/Seabed_Arms_Control_Treaty)
national claims on Antarctica (https://en.wikipedia.org/wiki/Antarctic_Treaty_System)
mining in Antarctica (https://en.wikipedia.org/wiki/Protocol_on_Environmental_Protection_to_the_Antarctic_Treaty)
military moon bases, space nukes (https://en.wikipedia.org/wiki/Outer_Space_Treaty)
low earth orbit miss

... (read more)

CstineSublime4m10

In crime shows and books they often talk about Means, Motive, and Opportunity... I suspect at least one is missing from each example on your list.

Military Moon Bases. The opportunity requires a well established space program with regular, or at least imminent, Lunar visits. The Means is tremendous amounts of resources. Which diminishes the motive - since the higher the opportunity cost, the higher the returns need to be: what is cheaper to do on the moon than on Earth to such a point where it becomes a profitable venture?

How many of these bans have held af... (read more)

Wei Dai's Shortform

Wei Dai1d*90

LessWrong/EAF Power Reader and User Archive (userscript)

(Beta announcement to get some testing/feedback before I post this to main. Please report bugs/UX friction/perf issues and feature requests, here if you want others to see and discuss, or on Github if minor.)

I present a browser userscript to help keep up with LW/EAF content (it's a way to view all recent comments/posts, with a lot of helpful features), and to save, read, and search a user's entire LW/EAF output.

Quick Start: Install Tampermonkey first (links for Chrome, Firefox, Edge, Safari), then cli... (read more)

3E. P. Cooper20h

I recommend against the use of Math.random() in general, unless you are highly performance constrained. I've checked, and it appears browsers have commonly supported a better random source since 2015[1]/early 2016 at the latest. The code below should be entirely correct to replace both the primary and fallback UUIDv4 generation code, once adapted to a function in a TS module. // Function available since 2015. const uuid_b = new Uint8Array(16); self.crypto.getRandomValues(uuid_b); let uuid_hex = ""; for (let i = 1; i <= 16; i++) { let octet = uuid_b[i-1]; if (i == 9) { // Set the correct "variant" (type). octet = octet & 0xBF | 0x80 } let o_hex = octet.toString(16); // Branch on data. if (o_hex.length == 1) { // Big endian, least significant bits are // on the right, therefore small numbers // that only result in a single byte out // need to be padded on the *left*. o_hex = `0${o_hex}` } if (i == 7) { // Set the correct version. o_hex = "4" + o_hex[1] } uuid_hex = `${uuid_hex}${o_hex}` if (i%2 == 0 && i > 3 && i < 11) { uuid_hex = `${uuid_hex}-` } } console.log(uuid_hex) If that still won't work, I could help you do slightly better by manually implementing SHA3-256 in JavaScript, since we're not really worried about timing attacks at this level. 1. ^ https://developer.mozilla.org/en-US/docs/Web/API/Crypto/getRandomValues#browser_compatibility

Wei Dai37m30

Thank you, I've taken your suggestion and removed all uses of Math.random().

3jimrandomh1d

Huh, I never thought of using that field that way but I guess it is the one and only non-public user-editable untyped field on the user object schema, so that makes sense. We aren't likely to delete this field or data within it in the near term, and will try to remember to shoot an email if it looks like there's some reason why we're going to. That said it is kind of a hack, and it does make that field security-sensitive if it wasn't before, so there is nonzero risk. A note for the future: We're experimenting with LW integrations with AI agents--mostly oriented around AIs making API calls directly, rather than using agents to build software tools that invoke the APIs--and as a result some new stuff has been added, and more will be added in the near future. Documentation for the AI-agent-oriented features are at /api/SKILL.md and updates to that documentation will appear there. We have a markdown-ified version of the frontend (good for an AI that wants to read markdown but not stable for structured parsing), and some (pre-beta buggy) APIs for letting an AI agent edit posts that you give them access to.

Linch's Shortform

Linch5h*150

There are two common models of space colonization people sometimes allude to, neither of which I think is particularly likely.

Model 1 (“normal colonization”) is that space colonization will look something like Earth colonization, e.g. the way the first humans to expand to the Polynesian islands. So your boat (rover/ship/probe) hops to one island (planet), you build up a civilization, and then you send your probes onwards to the next couple of nearby planets, maybe saving up a bunch of resources if you've colonized nearby star systems (eg your galaxy) and ... (read more)

Eli Tyre37m20

The furthest away parts of the theoretically reachable universe is 16-18 billion years away, so a 100 years delay is worth it if you can just increase the speed by 1/100 millionth of c

Why is there a tradeoff? Why don't you launch your early comparatively technologically-unsophisticated probes as soon as you can, and then, if you develop faster probes, also launch those if you calculate that they could catch up to the ones that you already launched?

It's not like the resources spent on early probes trade off appreciably with technological development.

5Avi Parrack3h

Hi Linch, a really nice comment. I've reached similar conclusions on the back of some work by Toby and Anders. I think model 3 is most likely though there could be some surprises that make it more like models 4-6. It could be that even at tech maturity a maximum single hop distance is limiting, such that one has to e.g. 'crawl' along galactic filaments taking pit stops to replicate then continuing the journey to the next intermediate destination. Dust is a good candidate as a limiting factor. More likely the probability of mission success decreases with e.g. with the integral of the dust flux over the mission trajectory and so one can deflate the number of pit stops required via the redundancy of sending more probes. And the scaling there may be very good or very poor. The material cost of probe creation could be a consideration. In the case of the spam to all reachable targets strategy one can need variously millions to billions of probes depending the reachable radius. This could require the resources of multiple solar systems but this will depend on the design of advanced probes (which could be extraordinarily light but not totally clear imo) one may be able to make try some various assumptions and do some napkin math. Another intuition boosting model aside from 3 to my mind is that the steepest scaling gradient for intelligence (probably of all time) is from essentially today to the near future e.g. within the next thousand years though I'd expect to get most of the way there in between 1-100 years. For example a I think a Landauer-limited KII Dyson is about +22 OOMs in ops/s vs Earth's (chips+humans) today and you're getting ~13 of those OOMs from kardashev scaling. Expectation is that you saturate the tech ceiling as it pertains to probes quickly enough to overtake early launches to even nearby stars. The densification of intelligence is perhaps underappreciated since here I vomit ops/s figure but there's also algorithmic efficiency gains and generally

Prometheus's Shortform

Prometheus2h10

Claude doesn’t get it

In all the interactions with AI, there has been one recurring problem that doesn’t seem to go away: they don’t get it. I don’t know how to explain this in any better language than that. I don’t know how to create a “Get It” benchmark. But whenever I talk to Claude, ChatGPT, Gemini, or any other model about a concept, the longer the interaction lasts, the more I get the sense that it doesn’t really “get it”. In this way, I think AI skeptics are actually pointing to something real when they say they’re not “real intelligence”. A lot of ... (read more)

asher's Shortform

asher2d1112

I think about the canonical Reality Has a Surprising Amount of Detail post a lot when trying to automate tasks with LLMs. In particular, any given task has many granular details, most of which don’t come to mind before making contact with reality oneself. The most common failure mode I run into is failing to specify various details that I hadn’t even *realized* were relevant things that could be messed up, before the model encountered the situation and messed them up. This also seems relevant when thinking about the transition from verifiable math and codi... (read more)

2CstineSublime1d

This is why I find even going through the ritual of writing an LLM prompt can clarify my own processes and goal - as it is forcing me to explicate a whole complex of hidden assumptions or even expose points where I realize I simply don't have the knowledge or information yet.

1JennaS1d

I can't help but wonder if this process could be automated? Or at least greatly sped up/eased by a good harness?

CstineSublime3h10

I must admit a poverty of imagination; I can’t see how it can be automated. That would be amazing if it could be.

However, the circumstances of each problem or LLM request are always so unique that outside of certain vague guardrails that apply to all problem solving/advice giving (In my experience these take the form of the questions: What have you tried already? Why did you try that way?/what did you expect to happen? What happened instead?). I see the ritual as attempting to explain why this situation is really unique and different – which seems to me to... (read more)

MalcolmMcLeod's Shortform

I.M.J. McInnis4h10

A fun riddle I was shocked to see the gippities solve without extended thinking or much yapping, even. I gave up on the third!

Here's the riddle, as stated to Opus 4.6 on an empty context window. "Consider single-word-name countries. An inclusive pair is when the name of one country is contained in another wholly. There are three such pairs. Find them."

I was surprised not only by how quickly it solved it, but also by the lack of thinking tokens. Gemini 3.1 Fast also did it. And by the unusual order in which the solutions were produced, in the exact re... (read more)

ryan_greenblatt's Shortform

ryan_greenblatt1d*8018

PSA: Anthropic models don't seem to particularly privilege the explicit thinking field. This makes reinforcement spillover—where training on a model's outputs generalizes to the CoT, making it appear safer—more likely.

While Anthropic models do have an separate explicit thinking field, they don't really use thinking that differently from outputs and aren't that dependent on the thinking field. Sometimes they'll just do their thinking in the output field, the way they talk in the thinking field isn't very distinct from how they talk in outputs, and I believe... (read more)

Showing 3 of 8 replies (Click to show all)

3Bronson Schoen8h

This comment thread on 1a3orn’s post has a collection of various model’s exhibiting degenerate language usage + Jozdien’s paper (which has since come out: Reasoning Models Sometimes Output Illegible Chains of Thought) I think are all strong evidence that you don’t get human legible english by default from outcome based RL. It seems very obvious to me that Anthropic (intentionally or via spillover) puts pressure on the CoT in a way that looks like nice human legible HHH friendly english. This is consistent with Ryan’s observations (which I also strongly agree with). I don’t think this has any principled reason to give any insight as to whether the model can functionally use this reasoning in the analysis channel. (Although I would note this was also surprising to me, they act like they’ve never seen this language in their life, and I’d be very interested in more work on how the “analysis” channel persona differs from the “final” channel persona, channel separation seems potentially extremely powerful, especially given the user is just another channel) Not necessarily! And again contra the “everything is nonfunctional spandrels”, my impression is that “ Myself” seems to be somewhat consistently used as the analysis channel persona and it’s also one of those “weird tokens that became super common including sometimes being repeated a lot over RL”. “Redwood” is another such token. We give a few examples: here As does METR in their GPT-5 report: IMO ablating “Myself” or “Redwood” in such cases are obviously a bad idea, and my mental model of these is “this seems to happen broadly during RL (see linked thread) sometimes these are contextually useful enough for the model to kind of make use of them.” It seems like there’s a spectrum between “saying marinade illusions 50x” which I agree is unlikey to make a big difference if ablated, and “watchers might check” (with likely many hard to discern border cases). In Codex 5.3 System Card we note: As an interesting side note

nostalgebraist5h183

This comment thread on 1a3orn’s post has a collection of various model’s exhibiting degenerate language usage + Jozdien’s paper (which has since come out: Reasoning Models Sometimes Output Illegible Chains of Thought) I think are all strong evidence that you don’t get human legible english by default from outcome based RL.

I'm somewhat skeptical of that paper's interpretation of the observations it reports, at least for R1 and R1-Zero.

I have used these models a lot through OpenRouter (which is what Jozdien used), and in my experience:

R1 CoTs are usually tot

... (read more)

1loic9h

With thinking mode on, you can even just ask Claude in web to add thinking blocks by wrapping them with <antml:thinking> xml tags (or cut it's thinking early), the web UI will also put them in thinking blocks. This post prompted me to check the conversation where I discovered this, and I noticed Anthropic made it so all thinking blocks are stuck together at the top rather than having them interleaved the order that the model generates them. You can clearly see that the model is able to think in one coherent chain where the "CoT" doesn't delimit a strict "thinking" then "responding" portion, in the example below it even reflects on "am I inside a thinking block, or the response?" Screenshot from ~2 months ago Same chat, screenshot from today

leogao's Shortform

leogao1d146

I've learned a lot about various people around me, and myself, over the past few days.

Showing 3 of 7 replies (Click to show all)

leogao6h00

it feels rude to talk about specifics about other people. at a broad level, there are some people I've gained a lot of respect for. it's easy for people to say they care about safety, so I don't weigh that very heavily. but now I know who's willing to step up and take actions in a crisis. and conversely too.

4leogao6h

it's obviously about the department of war situation. it wasn't intended to be vague, it just didn't occur to me that it wouldn't be obvious what it was about.

1papetoast8h

Total absence of context is relative, I think a lot of people here knows he works at OpenAI and the timing of this post heavily implies the OpenAI/Anthropic/DoD drama. Although I also agree that it is quite vague, it is somewhat understandable if he cannot or don't want to be too specific, and knowing the background of this post I do think there are a small amount of information. I would bet with high uncertainty that 10% of regular LW readers and 75% of people who have read this quick take know the background context. Also I just looked it up, according to the 2024 LW Census ~50% of users are American, which is a bit lower than I expected.

davekasten's Shortform

davekasten2mo19677

Random note: Congressman Brad Sherman just held up If Anyone Builds It, Everyone Dies in a Congressional hearing and recommended it, saying (rough transcript, might be slight paraphrase): "they're [the AI companies] not really focused on the issue raised by this book, which I recommend, but the title tells it all, If Anyone Builds It Everyone Dies"

I think this is a clear and unambiguous example of the theory of change of the book having at least one success -- being an object that can literally be held and pointed to by someone in power.

Showing 3 of 15 replies (Click to show all)

Eli Tyre7h20

Mostly, I just need to decide to spend a block of time writing, instead of doing other work. But aside from that, I have less need of co-writers, and more of a need for audiences that I'm actively writing for and engaging with (who those audiences are will be different for the many things that I have in the backlog).

10zroe12mo

This is such a funny coincidence! I just wrote a post where Claude does research on every member of congress individually. https://www.lesswrong.com/posts/WLdcvAcoFZv9enR37/what-washington-says-about-agi It was actually inspired by Brad Sherman holding up the book. I just saw this shortform and its funny because this thread roughly corresponds to my own thought process when seeing the original image!

8davekasten1mo

This is awesome! It broadly aligns with my understanding of the situation, although it does miss some folks that are known to care a bunch about this from their public statements. Downloading the JSON to take a deeper look! Strong upvoting the underlying post for Doing The Thing.

Raye's Shortform

Raye7h*10

I feel confused about the rationalists online. It feels like I tolerate you online because I AGREE with ai risk and know many of you personally. However if I encountered yall in the wilds of the internet I would not be moved. I would perhaps bounce off. And a younger me would maybe have even made you my enemy. I’m not sure how to fix this. The space seems void of a certain type of pragmatic femininity and aesthetic sensibility

*adding more specifics:

I largely feel like I’ve noticed people doing attack swarms and piles which feel is weird. It gives the energ

... (read more)

Ryan Meservey's Shortform

Ryan Meservey8h0-2

I almost had to update my priors that I am in a simulation because I just noticed my wife's initials are "L.L.M."

I say "almost" because her initials are actually "L.M.M.", and so I was forced to update my priors once again about my own comprehension skills. (sigh)

Ryan Meservey8h10

fyi, it would have been a very small update in favor, under the Likelihood Principle.

I would rate the observation "my wife has the initials LLM" as being slightly more common assuming a simulation hypothesis than assuming a non-simulation hypothesis.

yams's Shortform

yams9h540

Bernie Sanders has released a video on x-risk featuring a discussion with Eliezer, Nate, Daniel Kokotajlo, and Jeffrey Ladish. An excerpt from it appears to be blowing up on twitter.

arisAlexis's Shortform

arisAlexis10h10

There is currently a post on reddit https://www.reddit.com/r/singularity/comments/1rktwmm/i_study_whether_ais_can_be_conscious_today_one/

and it shows an LLM emailing an AI consciousness researcher asking about it's own consciousness. How legit can this be? If it is actually legit it's kind of mind blowing and deserves a lot of alarms in many labs.

yams9h20

What do you mean by 'legit'?

cyrusfreshman's Shortform

cyrusfreshman10h00

Optimalism, a new religion
1) self-protect
2) do good, optimally

Saul Munn's Shortform

[+]Saul Munn11h-60

The shortform of Ole Q Doc

Quinn13h70

can someone explain why Simulator theory was written about end of 2022, people semi forgot or moved on, then Personas made a big splash recently?

interstice12h30

Can't speak for everybody but when I read simulators I was like "yeah obviously GPTs are generative models not agents, duh, why do people need an entire post to tell them that" and didn't really expect the thing where agents are like "characters" in the world model to scale to high levels of capability. That seeming to happen raised the salience of the ideas a lot for me.

Daniel Tan's Shortform

Daniel Tan2d*6112

A quick note on various alignment affordances that the model personas research agenda might offer. I'm interested in takes on how useful people think each of these is.

"Seeding RL." RL is underspecified. Specifically, it seems difficult to provide good feedback on the motivations / values of AIs, since: (i) for sufficiently strong models, the aligned and deceptively aligned policy are behaviourally indistinguishable, and (ii) training directly against the reasoning might just lead to obfuscation. The hope is that improving the initial point from which we s

4Alex Mallen13h

All of these seem very useful to me (except maybe robustness). Can you clarify what's meant by robustness? You mentioned robustness to weight updates--this seems potentially bad because it involves making the AI incorrigible. Under robustness I'd have said: having the AI's persona be stable under long serial reasoning/long contexts/when AI agents are communicating a bunch in the deployment.

Daniel Tan12h20

Thanks! Much appreciated.

I think there are two meanings of robustness here:

In-context robustness. A simple example of this is resisting persona-based jailbreaks - e.g. when we tell the model "You are DAN" it should not believe this. But yes, really good versions of this go beyond that. We want really stable personas that survive throughout long-context deployment, with minimal persona drift. (Maybe this can just be solved prosaically? Maybe we don't need to intervene on the assistant axis - maybe we just need to inject lots of reminders like "You are Clau

... (read more)

7David Africa2d

We might be interested in two more things: * "Diagnosis." You have a model in deployment (or about to be deployed). You want to know what personas are latent in it, without needing to enumerate all possible dangerous prompts. * I'd guess a prerequisite thing to think about is how we would know to intervene (e.g., to improve the robustness of a persona or to decouple two persona traits or something) is to have some systematic way to map something in the model to some abstract characteristics about the model's personality. Rather than implant or robustify personas that we add, we may be interested (or perhaps worried) about what alien personas exist in model n-2, n-1, n, and use that to produce predictions about alien productions in n+1. * "Composition/crafting." It is good to grow a target persona and install it; but rather than grow one, could we instead craft or architect it? I reckon that it's worth studying if there's some sensible way to compose, add, subtract, multiply, orthogonalize etc the mechanistic representations we use for a persona (so, I know this kind of works for steering vectors, but seems open for other things, such as choice of training data), then use that take a persona in a LoRA and do some surgical edits on what bits of the persona to enhance, intensify, reduce. I think this is important esp. as models get more and more capable, so the target for what an acceptable personality for the model to have gets narrower and narrower.

RationalElf's Shortform

RationalElf14h10

Why is there so much emphasis on OpenAI and its arrangement with the Department of War relative to GDM's and xAI's, and is that rational? While OpenAI seems like it's behaving much worse than Anthropic, it seems arguably better than those other two, and I'm worried this is a case of it being punished for doing more than nothing (or rather, that some of the ire currently focused on OpenAI should focus on them).

Agree that OpenAI's and Department of War's comms about their arrangement was weird, sketchy, and triggering (but not necessarily worse than complete silence in my mind)

Eli Tyre13h75

I am criticizing OpenAI not just because of the terms of their contract, but because they previously said that they had the same redlines as Anthropic, and then not 2 days latter, signed a contract abandoning those redlines, while quite transparently lying about whether the redlines were protected.

That is bad behavior, and I'm glad they're getting pushback about it. When you claim to stand for principles, you're taking on additional social cost when you abandon those principles.

I wouldn't care nearly as much, if they had accepted the the contract that they... (read more)

testingthewaters's Shortform

testingthewaters13h20

Looking back at the self other overlap research, it seems that there is a way to simplify the problem. Instead of trying to find a "self" representation and an "other" representation, we can instead do something like "create a complexity penalty for internal representations". In other words, simply require that the system reduce the complexity of the entire internal residual stream/hidden state/activation over time, while maintaining its predictive/world modelling accuracy. This requires it to essentially "share representations" for things that have shared... (read more)