Quick Takes

What's something you believe, that would get negative Karma if earnestly expressed in a normal LessWrong conversation? Write it in quotes. Vote on the meta-claim "would get negative karma" using ✔️/ X, where ✔️ = yes this would get negative karma, and X = no this would get positive or nonnegative karma.

I'm seeing more sophisticated LLM-slop in the LW moderation queue. Eight months ago, I wrote "hey, we're getting tons of AI-psychosis'd people, deluded into thinking their crackpot coherence/spiralism/emergence/ChatGPTAwakening experience is true and meaningful. We process like 15-20 of these a day." Nowadays, we still get some of those, but, a lot less. Instead, I think we often now get a somewhat more sophisticated looking set of AI LLM slop. It's often something like: "Me and ChatGPT have been working on some ML experiments for months, checking if we get some behavior if we do ablation, or if we mess with the residual stream, or, [some other mech-interp-ish buzzword thing]." They sound plausible at a glance, but usually don't explain the specific mechanism for why their experiment should be interesting, or fit into the LW conversation. Often this person disclaims "I'm not an expert in ML, I have a degree in [X], but, I've done what seems like it should be a reasonable good faith effort to check if this makes sense." I already find this fairly annoying to judge (a couple of them prompted disagreement within the team on whether they were legit). But, it seems like in another 6-12 months these will be fairly costly to distinguish from, say, a newly minted MATS scholar's work. And, maybe another 6-12 months after that, it might actually be better than many newly-minted-MATs-scholar's work. (I think it will not reach the standard of a "sufficiently-good-alignment-researcher to really build meaningful conceptual progress" until Near The End. My impression is most junior MATS-scholars do work that's more like, executing on a shovel ready project where the judgment was done by their senior mentor and most of what they're doing is learning the ropes of how to run experiments. Interested in takes about this) I'm really not sure what we're supposed to do about this. My understanding of habryka's take is, he's guessing that 6-18 months from now, AI will actually be g

Unless you have an explicit strategy for how and when you will speak out in the future, and explicit reasoning for why that requires silence now, "strategic silence" is just silence.

Things that have been successfully-so-far banned before being done (very shallow research, not sure; found w/ gippities and cursorily (ha) sanity-checked): * human cloning (may not hold) * seabed nukes (https://en.wikipedia.org/wiki/Seabed_Arms_Control_Treaty) * national claims on Antarctica (https://en.wikipedia.org/wiki/Antarctic_Treaty_System) * mining in Antarctica (https://en.wikipedia.org/wiki/Protocol_on_Environmental_Protection_to_the_Antarctic_Treaty) * military moon bases, space nukes (https://en.wikipedia.org/wiki/Outer_Space_Treaty) * low earth orbit missiles ( https://en.wikipedia.org/wiki/Fractional_Orbital_Bombardment_System, https://en.wikipedia.org/wiki/Strategic_Arms_Limitation_Talks#SALT_II_Treaty) * (questionable success, maybe some deployment before the ban) blinding laser weapons (https://en.wikipedia.org/wiki/Protocol_on_Blinding_Laser_Weapons) There are probably several more examples of successful huge bans after warning shots / initial uses, e.g. military environmental modification (https://en.wikipedia.org/wiki/Environmental_Modification_Convention ), and examples of questionable / mostly successful bans, e.g. exploding bullets (https://en.wikipedia.org/wiki/Saint_Petersburg_Declaration_of_1868)

Earlier this week I attended a presentation on AI use in only-somewhat-techie corporate contexts, and found it fascinating how LW terminology has gone mainstream but the meanings haven't: the presenter talked a lot about 'existential risk' (which I slowly inferred meant 'AI-using competitors might put us out of business'), and 'alignment' (which he helpfully defined as 'getting various AI modalities - coding, search, image gen etc - to work together harmoniously').

PSA: Anthropic models don't seem to particularly privilege the explicit thinking field. This makes reinforcement spillover—where training on a model's outputs generalizes to the CoT, making it appear safer—more likely. While Anthropic models do have an separate explicit thinking field, they don't really use thinking that differently from outputs and aren't that dependent on the thinking field. Sometimes they'll just do their thinking in the output field, the way they talk in the thinking field isn't very distinct from how they talk in outputs, and I believe disabling thinking doesn't have a big effect on coding performance (especially for earlier Anthropic models but even for current Anthropic models). (This is specifically for typical agentic coding; this likely doesn't apply to math.) This is pretty different from OpenAI models which are way more reliant on the explicit thinking and the thinking is very distinct from how the model talks in the output (based on public CoT examples). I'm uncertain, but I think GDM models are generally more similar to OpenAI than Anthropic models on this axis. In general, Anthropic's models leverage taking lots of actions over doing a bunch of thinking. (This is based on my personal experience/analysis and talking to various people, including some Anthropic employees, but isn't based on non-public information.) Some implications of this: * It might be a mistake to emphasize CoT monitoring relative to monitoring the overall transcript (including CoT but also tool calls and other outputs). Like, I do think that Anthropic models are decently monitorable in general, but looking at the thinking tags in particular may not be that important/salient by default. (To be clear, if the model knew you weren't looking at the thinking tags that might be a big uplift to its covert abilities, I just mean that these tags aren't that privileged.) * Because Anthropic models don't treat the thinking field in a very distinct way, I think it's prett

There are two common models of space colonization people sometimes allude to, neither of which I think is particularly likely.  Model 1 (“normal colonization”) is that space colonization will look something like Earth colonization, e.g. the way the first humans to expand to the Polynesian islands. So your boat (rover/ship/probe) hops to one island (planet), you build up a civilization, and then you send your probes onwards to the next couple of nearby planets, maybe saving up a bunch of resources if you've colonized nearby star systems (eg your galaxy) and need to send a bigger ship to more distant stars. So it looks like either orderly civilizational growth or an evolutionary process. I don't think this model is really likely because von Neumann probes will be really cheap relative to the carrying capacity of star systems. So I don't think the intuitive "slow waves of colonization" model makes a lot of sense on a galactic scale.  I don’t think my view here is particularly controversial. My impression is that while the first model is common in science fiction, nobody in the futurism/x-risk/etc fields really believes it. Model 2 (“mad dash”) is that you race ahead as soon as you reach relativistic speeds. So as soon as your science and industry has advanced enough for your probes to reach appreciable fractions of c, you start blasting out von Neumann probes to the far reaches of the affectable universe. I think this model is more plausible, but still unlikely. A small temporal delay is worth it to develop more advanced spacefaring technology. My guess is that even if all you care about is maximizing space colonization, it still makes sense to delay some time before you launch your first "serious" interstellar space probe, rather than do it as soon as possible[1]. Whether you can reach the furthest galaxies is determined by something like[2]: total time to reach a galaxy = delay + distance/speed  So you want to delay and keep researching until the marginal sp