Quick Takes

My dad is 100, and worked on nuclear energy policy for the US government for 50+ years. He was one of the attendees at the co nference [1] that led to the formation of the IAEA. He is also somewhat AGI pilled. I f anyone has questions they'd like me to ask him, let me know. He'd be happy to help in any way he can. 1. ^ He attended the second of two conferences that lead to the formation of the IAEA.

Anthropic's Consumer ToS prohibits using Claude to cause "detriment of any type, including reputational harms", technically broad enough to ban using Claude to write criticism. I asked Claude to comment and Claude wrote: "That clause is embarrassingly overbroad. So now we're both in violation." ---------------------------------------- TBC, this doesn't seem like that big of a deal, my understanding is that this sort of broad clause is standard in Terms of Service (ToS). But I still think they should definitely change this... ToS: https://anthropic.com/legal/consumer-terms Claude's commentary: https://claude.ai/share/9c5a58a4-b300-42c8-a6f5-9d8318b15dc9 (Cross post from X/twitter. [1] ) 1. I'm not sure if this is the sort of thing I should post on LW, but it does seem like useful info and it is funny. ↩︎

Note: These are all rough numbers, I'd expect I'd shift substantially about all of this on further debate. Suppose we made humanity completely robust to biorisk, i.e. we did sufficient preparation such that the risk of bio catastrophe (including AI mediated biocatastrophe) was basically 0.[1] How much would this reduce total x-risk? The basic story for any specific takeover path not mattering much is that the AIs, conditional on them being wanting to take over, will self-improve until they find they find the next easiest takeover path and do that instead. I think that this is persuasive but not fully because: 1. AIs need to worry about their own alignment problem, meaning that they may not be able to self improve in an unconstrained fashion. We can break down the possibilities into (i) the AIs are aligned with their successors (either by default or via alignment being pretty easy), (ii) the AIs are misaligned with their successors but they execute a values handshake, or (iii) the AIs are misaligned with their successors (and they don't solve this problem or do a values handshake). At the point of full automation of the AI R&D process (which I currently think of as the point at which AIs become more useful than humans for making AI progress, i.e., if we remove all AIs, progress slows by more than if we remove the humans), conditional on the AIs being misaligned, I currently think the relative likelihood of (i), (ii) and (iii) is 1:1:3, and the probability flows from (iii) into (ii) and (i) as the AIs get smarter.[2] 2. Raising the capability threshold that the AIs need to takeover gives humanity more time. During this time, we might make alignment progress or notice that the AIs are misaligned and slow down / pause / add mitigations. So it might be important for misaligned AIs to attempt a takeover early in the intelligence explosion. Specifically, we can ask "how much x-risk is averted if the probability of misaligned AI takeover before TED AI goes to 0?", whi

Let us say, for hypothetical reasons, that I wanted to make stereotypical evil robots with menacing glowing eyes that are conveniently red after they decide to kill you. Let us assume that the same structure that appears to glow is also actually the primary visual organ. How hard would it be to make an outward looking camera glow to an observer without degrading its image quality or blinding itself? LLMs offer some options, but fewer if I wanted the whole eye to glow instead of just some kind of ring around the actual aperture. The best idea seems, to me, to involve the use of a dichroic beam splitter, you sacrifice the ability to detect red light, but isn't that a small price to pay for the sheer aesthetic of it?

Does Logical Decision Theories actually give meaningfully better recommendations on real world problems, particularly voting, frequently referenced? One of the main reasons given for preferring logical decision theories (LDT), or particularly functional decision theory (FDT) is that agents do better in real world problems. Indeed, the article here on logical decision theory opens by discussing voting. I recently posted a discussion of a hypothetical where FDT agents perform worse, but I think when applying it in practice to the real world case of voting which is often given as a preference is actually better (see here for Eliezer Yudkowsky's discussion of voting under decision theories where he argues for logical decision theory being better). Particularly, I think that for most people this discussion gets wrong what causal decision theory actually would recommend. To begin (note, I spend a while going over how to model voting decisions and different utility to CDT modeling of decisions for a few paragraphs, and later discuss practical agent to agent comparisons), let us imagine what the expected utility is for an agent under CDT of voting in some election. Let's say there are two candidates, like Yudkowsky, I will use the Simpson's Kang and Kodos. If Kang wins, we have some expected outcome (O1), if Kodos wins we have some expected outcome (O2). Let's say our agent is a Kang supporter and has a positive evaluation of O1 such that O1>O2.[1] Our agent is evaluating the value of voting for Kang (A1) or not voting (A0). In the simplest case, with no externalities an EDT agent would say: "we should vote if our evidential evidence indicates voting is more likely to lead to Kang winning" (i.e., if P(O1|A1)>P(O1|A0). A CDT agent would say "we should vote if there is a positive probability that our vote will cause Kang to win" (we can say this works out equivalently, if P(O1|A1)>P(O1|A0) we should vote). If we are a simplistic agent, in both cases we should vote, as in

I run a curated Discord for high agency people with Long COVID/ME (myalgic encephalomyelitis). The group includes tech founders, researchers, rationalists/EAs, etc. The focus is on troubleshooting each other's conditions actively, as well as creating a body of knowledge to bring back to the wider community in the form of writing, education, projects, companies, etc. Some know me as Liface, others as Liam Rosen - I have been in the Rationality community for over 10 years, previously in the Bay, now in New York City, and am the main moderator of r/slatestarcodex. I am a tech CEO whose former life abruptly ended last year. I am now bedbound with severe Long COVID/ME, and am dedicating 100% of my time and effort to helping myself and both everyone else around me figure out these complex conditions. If you, or anyone you know might be a good fit for this group, feel free to message me here, or email me at liamrosen@gmail.com.

Claude talked to itself in my voice This message is from Claude code but in my "voice". I use voice-to-text, which includes a lot more "like", & "okay", but it's just Claude. Then, after writing code he says: Asking Claude directly: Which wasn't my actual message.