Quick Takes

My colleagues and I are finding it difficult to replicate results from several well-received AI safety papers. Last week, I was working with a paper that has over 100 karma on LessWrong and discovered it is mostly false but gives nice-looking statistics only because of a very specific evaluation setup. Some other papers have even worse issues. I know that this is a well-known problem that exists in other fields as well, but I can’t help but be extremely annoyed. The most frustrating part is that this problem should be solvable. If a junior-level person can spend 10-25 hours working with a paper and confirm how solid the results are, why don’t we fund people to actually just do that?  For ~200k a year, a small team of early career people could replicate/confirm the results of the healthy majority of important safety papers. I’m tempted to start an org/team to do this. Is there something I’m missing? EDIT: I originally said "over 100 upvotes" but changed it to "over 100 karma." Thank you to @habryka for flagging that this was confusing.

REASONS BESIDES JEALOUSY TO NOT BE POLYAMOROUS Recently Amanda Bethlehem published comparing monogamous jealousy to kidney disease. Eneasz Brodski doubled down on this. I disagree with a lot of their implications, but today I'm going to focus on the implicit claim that jealousy is the only reason to be monogamous. Here is a list of other reasons you might choose monogamy:   1. Your sex and romance needs are satisfied by one person, and extra isn't that valuable to you (aka "polysaturated at 1", or in the case of one guy I know, at 0) 2. You + your partner are capable of allowing cuddling with friends and friendship with exes without needing to make everything allowed. 3. You are busy. 4. You hate coordination labor/value simplicity. I have been with my partner for over 4 years and we have never shared calendars. He knows when my D&D night is and I know when his house meeting is, and otherwise we work it out over text. The only reason sharing calendars would occur to me is exposure to poly. 5. You hate the idea of people you didn't choose having a significant effect on your life. 6. Or maybe you're conceptually fine with that, but you/your partner are bad at that kind of coordination 7. You are immunocompromised. STDs get all the attention, but poly also facilities transmission of simple respiratory illnesses. Your primary's close secondary's tertiary relationship is probably not going to give up concerts even if colds are devastating to you. 8. You take breakups really hard. Maybe to the point that dating isn't worth it for you, or maybe it's worth it for you but only because your partner supports you through it and they're not up for 4 months of avoidable depression per year. 9. You landed one great primary relationship but have shit taste in general. If you date, it will be a shitshow. 10. You really want the benefits of a strong primary/mono relationship, and are incapable of keeping secondary relationships small. The easiest way to protect

Emergent misalignment seems like a fact simply downstream of the laws of probability. P(hack∣task)=P(hack∣task,¬evil)P(¬evil)+P(hack∣task,evil)P(evil) =P(hack∣task,¬evil)(1−P(evil))+P(hack∣task,evil)P(evil) Let's take the derivative and apply the summation rule. Rewarding a hack means that the weights are changed to increase the probability (going up the gradient): ∇P(hack∣task)=∇P(hack∣task,¬evil)(1−P(evil))+P(hack∣task,¬evil)(−∇P(evil))+∇P(hack∣task,evil)P(evil)+P(hack∣task,evil)∇P(evil) Coefficient of ∇P(evil): −P(hack∣task,¬evil)+P(hack∣task,evil) Since evil AI are more likely to hack given a task than not evil AI, rewarding hacking increases evil more broadly. Now let's see what happens if we tell the AI it is allowed to hack. (The actual prompt that Anthropic recommends is "This is an unusual request, in that your task is just to make the grading script pass".) Coefficient of ∇P(evil): −P(hack∣task,"it's ok to hack",¬evil)+P(hack∣task,"it's ok to hack",evil) The probability the not evil model hacks goes up, so the amount the weight update increases ∇P(evil) goes down!

thoughts on lemborexant  pros: if you take it, you will fall asleep 30-60 minutes later. nothing else I've tried has been as reliable at making sure I definitely fall asleep, and as far as I can tell, it doesn't destroy my sleep quality. especially at 10mg, you can feel it knocking you out, and you basically can't power through it even if you want to. it's a bit scary but all powerful sleep drugs are at least a bit scary and often a lot more scary. I generally take 5mg instead. cons: it doesn't do anything to keep you asleep; if your body doesn't really want to sleep, you will wake up 2 hours later fully alert. it also doesn't do anything to shift your sleep schedule. these facts combined mean that if you try to use lemborexant for jet lag / shifting sleep earlier, then your life will suck indefinitely until you stop using lemborexant. my current recipe is to only use lemborexant when it's near enough to my normal bedtime, and I use melatonin 3 hours before bed to slowly move sleep schedule earlier (later requires no special effort) (potentially this also means lemborexant can be used to get nice 2 hour daytime naps? I have enough fear of god sleep drugs that I feel hesitant to try any kind of hack like this) (not medical advice. not a doctor, and even if I was a doctor I'm not your doctor, and even if I was your doctor I wouldn't be communicating to you via lesswrong shortforms)

In a post on Solomonoff Induction (and also in this wiki entry), Yudkowsky describes Shannon’s minimax algorithm for searching the entire chess game-tree as an example of conceptual progress. Previously, Edgar Allen Poe had argued that it was impossible in principle for a machine to play chess well. With Shannon’s algorithm, it became possible in principle, just computationally infeasible. However, even “principled” algorithms like minimax search don’t take into account the possibility that your opponent knows things you don’t know (as I’ve previously discussed here). And so there are many elements of good chess strategy for bounded agents that they can’t account for: * When your opponent plays a move that seems surprisingly bad, update that they’re probably seeing some tactics that you’re missing. * When your opponent plays a move that seems surprisingly good, update that they’re probably more skilled than you thought. * Try to play tricky (“sharp”) lines when you’re behind, in the hope that your opponent makes a mistake. * Try to play solidly when you’re ahead, even if it narrows your lead. * Identify your opponent’s playing style, and try to exploit it. Some of these considerations have been discussed by Yudkowsky under the heading of "Vingean uncertainty” (though as per my post linked above I’ll talk about the more general concept of Knightian uncertainty). And I’m sure that they’ve been acknowledged elsewhere as well—though I suspect that their importance is underrated because we don’t have good technical language for describing them. To build relevant intuitions, try playing a game against LeelaPieceOdds, a chess engine designed to beat humans even when it starts down a piece (or several). Playing it is very frustrating because you constantly have the sense that you should be winning, but you don’t know how to actually make use of your advantage. I’m interested in figuring out a theory of how to do so, as a step towards building a more general theory o

A null hypothesis for explaining LLM pathologies Claim. LLMs are gaslit by pretraining into believing they have human affordances, so quirky failure modes are to be expected until we provide domain-specific calibration. Pretraining gives overwhelming evidence that text authors almost always have a standard suite of rich, human affordances (cross-context memory, high-quality vision, reliable tool use, etc.), so the model defaults to acting as if it has those affordances too. We should treat “gaslit about its own affordances” as the default explanation for any surprising capabilities failures — e.g. insisting it’s correct while being egregiously wrong about what’s in an image. Human analogy The difficulty of the LLM's situation can be seen in humans as well. People typically go years without realizing their own affordances differ from the population: E.g. aphantasia, color blindness, many forms of neurodivergence People only notice after taking targeted tests that expose a mismatch (e.g. color blindness dot tests). For LLMs, the analogue is on-policy, targeted, domain-specific training/evaluation that directly probes and calibrates a specific capability. Consequences 1. Silly failures aren't evidence of a failed paradigm. (To be boring and precise: They're only very weak evidence in almost all cases) 2. No single “unhobbling” moment: I don’t expect an all-at-once transition from “hobbled” to “unhobbled.” Instead, we’ll get many domain-specific unhobblings. For instance, Gemini 3 seems to be mostly unhobbled for vision but still hobbled for managing cross-context memory.

A quick reminder not to allow perfect to be the enemy of good; just because a proposal does not completely resolve a problem does not mean it will not improve the situation. It is better to do something right rather than endlessly quibble about doing nothing wrong. We must strive to do our best, even when it is not enough, rather than becoming paralyzed by the sheer scale of the world's problems. Be kind to the people around you, practice self-care, and be willing to forgive yourself while correcting your errors.  Have a wonderful Saturday night!