In our jobs as AI safety researchers, we think a lot about what it means to have reasonable beliefs and to make good decisions. This matters because we want to understand how powerful AI systems might behave. It also matters because we ourselves need to know how to make good decisions in light of tremendous uncertainty about how to shape the long-term future.

It seems to us that there is a pervasive feeling in this community that the way to decide which norms of rationality to follow is to pick the ones that win. When it comes to the choice between CDT vs. EDT vs. LDT…, we hear we can simply choose the one that gets the most utility. When we say that perhaps we ought to be...

Epistemic Status: I believe I am well-versed in this subject. I erred on the side of making claims that were too strong and allowing readers to disagree and start a discussion about precise points rather than trying to edge-case every statement. I also think that using memes is important because safety ideas are boring and anti-memetic. So let’s go!

Many thanks to @scasper, @Sid Black , @Neel Nanda , @Fabien Roger , @Bogdan Ionut Cirstea, @WCargo, @Alexandre Variengien, @Jonathan Claybrough, @Edoardo Pona, @Andrea_Miotti, Diego Dorn, Angélina Gentaz, Clement Dumas, and Enzo Marsot for useful feedback and discussions.

When I started this post, I began by critiquing the article A Long List of Theories of Impact for Interpretability, from Neel Nanda, but I later expanded the scope of my critique. Some ideas...

In the USA, the president isn't determined by a straight vote. Instead, each state gets a certain number of Electoral College (EC) votes, and the candidate with 270 EC votes wins.

It's up to each state to decide how to allocate its EC votes. Most do “winner-takes-all,” but some, e.g., Maine and Nebraska, split them up.

California and Texas have the most EC votes of any state, with 54 and 40 votes respectively, so you would think they would get a lot of love from presidential candidates. Instead, they're mostly ignored—California will always be Blue, and Texas Red, so what's the point of pandering to them? This is clearly bad for Californians and Texans as their interests aren't listened to.

So why doesn't California switch to a proportional EC vote...

In this post, we’re going to use the diagrammatic notation of Bayes nets. However, we use the diagrams a little bit differently than is typical. In practice, such diagrams are usually used to define a distribution - e.g. the stock example diagram

... in combination with the five distributions $P\left[\text{Season}\right]$, $P\left[\text{Rain}|\text{Season}\right]$, $P\left[\text{Sprinkler}|\text{Season}\right]$, $P[\text{Wet}|\text{Rain},\text{Sprinkler}]$, $P\left[\text{Slippery}|\text{Wet}\right]$, defines a joint distribution

$P[\text{Season},\text{Rain},\text{Sprinkler},\text{Wet},\text{Slippery}]$

$=P\left[\text{Season}\right]P\left[\text{Rain}|\text{Season}\right]P\left[\text{Sprinkler}|\text{Season}\right]P[\text{Wet}|\text{Rain},\text{Sprinkler}]P\left[\text{Slippery}|\text{Wet}\right]$

In this post, we instead take the joint distribution as given, and use the diagrams to concisely state properties of the distribution. For instance, we say that a distribution $P[\Lambda ,X]$ “satisfies” the diagram

if-and-only-if $P[\Lambda ,X]=P\left[\Lambda \right]{\prod }_{i}P\left[X_i|\Lambda \right]$. (And once we get to approximation, we’ll say that $P[\Lambda ,X]$ approximately satisfies the diagram, to within $ϵ$, if-and-only-if $D_{KL}\left(P\right[\Lambda ,X\left]||P\right[\Lambda \left]{\prod }_{i}P\right[X_i|\Lambda \left]\right)\le ϵ$.)

The usage we’re interested in looks like:

• State that some random variables satisfy several different diagrams
• Derive some new diagrams which they satisfy

In other words, we want to write proofs diagrammatically - i.e....

As Americans know, the electoral college gives disproportionate influence to swing states, which means a vote in the extremely blue state of California was basically wasted in the 2024 election, as are votes in extremely red states like Texas, Oklahoma, and Louisiana. State legislatures have the Constitutional power to assign their state's electoral votes. So why don't the four states sign a compact to assign all their electoral votes in 2028 and future presidential elections to the winner of the aggregate popular vote in those four states? Would this even be legal?

The population of CA is 39.0M (54 electoral votes), and the population of the three red states is 38.6M (55 electoral votes). The combined bloc would control a massive 109 electoral votes, and would have gone...

When I was growing up most families in our neighborhood had the daily paper on their lawn. As getting your news on the internet became a better option for most folks, though, delivery became less efficient: fewer houses on the same route. Prices went up, more people cancelled, and decline continued. I wonder if we might see something similar with the power grid?

Solar panels keep getting cheaper. When we installed panels in 2019 they only made sense because of the combination of net metering and the relatively generous SREC II incentives. By the time we installed our second set of panels a few months ago net metering alone was enough to make it worth it.

Now, as I've said a few times, net metering is kind of absurd. The way it works here is that...

Foresight Institute's AI Safety Grants Program added a new focus area in response to the continually evolving field. Moving forward, our funding ($1.5M-$2M annually) will be allocated across the following four focus areas:

1. Automating AI-relevant research and forecasting

• Scaling AI-enabled research to support safe AGI development
• Scaling efficient forecasting methods relevant for safe AGI
• Other approaches in this area

2. Neurotech to integrate with or compete against AGI

• Brain Computer Interfaces (BCI) to enhance human cognition or facilitate human-AGI collaboration
• Whole Brain Emulations (WBE) which might function as human-like general intelligences that are more interpretable and alignable than AGI
• Lo-fi emulations using behavioral and neural data with deep learning, potentially offering a cost-effective alternative to full WBEs
• Other approaches in this area

3. Security technologies for securing AI systems

• Implementations of computer security techniques (including POLA, SeL4-inspired systems,

Hi there.

Quick question. I am using a few articles from LessWrong for a dissertation. Are there any mainstream articles/sources that reference LessWrong as being the catalyst/partial source for AI alignment, researchers, and other academic literature? I think it's snobbish, or, discriminatory to regard LessWrong as merely another online website. I was hoping to get some advice on how to formulate a paragraph justifying the citation of LessWrong?

Thanks.