Very helpful reply, thank you!

(My salary has always been among the lowest in the organization, mostly as a costly signal to employees and donors that I am serious about doing this for impact reasons)

I appreciate that!

I have completely forfeited my salary, and donated ~$300k to Lightcone at the end of last year myself to keep us afloat

If you had known you were going to do this, couldn't you have instead reduced your salary by ~60k/year for your first 5 years at Lightcone and avoided paying a large sum in income taxes to the government?

(I'm assuming that your after-tax salary from Lightcone from your first 5-6 years at Lightcone totaled more than ~$300k, and that you paid ~$50k-100k in income taxes on that marginal ~$350k-$400k of pre-tax salary from Lightcone.)

I'm curious if the answer is "roughly, yes" in which case it just seems unfortunately sad that that much money had to be unnecessarily wasted on income taxes.

I originally missed that the "Expected Income" of $2.55M from the budget means "Expected Income of Lighthaven" and consequently had the same misconception as Joel that donations mostly go towards subsidizing Lighthaven rather than almost entirely toward supporting the website in expectation.

Something I noticed:

"Probability that most humans die because of an AI takeover: 11%" should actually read as "Probability that most humans die [within 10 years of building powerful AI] because of an AI takeover: 11%" since it is defined as a sub-set of the 20% of scenarios in which "most humans die within 10 years of building powerful AI".

This means that there is a scenario with unspecified probability taking up some of the remaining 11% of the 22% of AI takeover scenarios that corresponds to the "Probability that most humans die because of an AI takeover more than 10 years after building powerful AI".

In other words, Paul's P(most humans die because of an AI takeover | AI takeover) is not 11%/22%=50%, as a quick reading of his post or a quick look at my visualization seems to imply, but is actually undefined, and is actually >11%/22% = >50%.

For example, perhaps Paul thinks that there is a 3% chance that there is an AI takeover that causes most humans to die more than 10 years after powerful AI is developed. In this case, Paul's P(most humans die because of an AI takeover | AI takeover) would be equal to (11%+3%)/22%=64%.

I don't know if Paul himself noticed this. But worth flagging this when revising these estimates later, or meta-updating on them.

I have time-space synesthesia, so I actually picture some times as being literally farther away than others.

I visualize the months of the year in a disc slanted away from me, kind of like a clock with New Years being at 6pm, and visualize years on a number line.

I thought of the reason independently: it's that if the number before 66 is not odd, but even instead, it must be either 2 or 4, since if it was 6 then the sequence would have had a double 6 one digit earlier.

150 or 151? I don't have a strong intuition. I'm inclined to trust your 150, but my intuition says that maybe 151 is right because 100+99/2+almost1 rounds up to 151. Would have to think about it.

(By the way, I'm not very good at math. (Edit: Ok, fair. Poorly written. What I meant is that I have not obtained certain understandings of mathematical things that those with formal educations in math have widely come to understand, and this leads me to being lower skilled at solving certain math problems than those who have already understood certain math ideas, despite my possibly having equal or even superior natural propensity for understanding math ideas.). I know high school math plus I took differential equations and linear algebra while studying mechanical engineering. But I don't remember any of it well and don't do engineering now or use math in my work. (I do like forecasting as a hobby and think about statistics and probability in that context a lot.) I wouldn't be able to follow your math in your post without a lot of effort, so I didn't try.)

Re the almost1 and a confusion I noticed when writing my previous comment:

Re my:

E.g. For four 100s: Ctrl+f "100,100,100,100" in your mind. Half the time it will be proceeded by an odd number for length 4, a quarter of the time it will be length 5, etc.

Since 1/2+1/4+1/8...=1, the above would seem to suggest that for four 100s in a row (or two 6s in a row) the expected number of rolls conditional on all even is 5 (or 3). But I saw from your post that it was more like 2.72, not 3, so what is wrong with the suggestion?

My intuition was that B is bigger.

The justification was more or less the following: any time you roll until reaching two $6\text{s}$ in a row, you will have also hit your second 6 at or before then. So regardless what the conditions are, $A$ must be larger than $B$.

This seems obviously wrong. The conditions matter a lot. Without conditions that would be adequate to explain why it takes more rolls to get two 6s in a row than it does to get two 6s, but given the conditions that doesn't explain anything.

The way I think about it is that you are looking at a very long string of digits 1-6 and (for A) selecting the sequences of digits that end with two 6s in a row going backwards until just before you hit an odd number (which is not very far, since half of rolls are odd). If you ctrl+f "66" in your mind you might see that it's "36266" for a length of 4, but probably not. Half of your "66"s will be proceeded by an odd number, making half of the two-6s-in-a-row sequences length 2.

For people that didn't intuit that B is bigger, I wonder if you'd find it more intuitive if you imagine a D100 is used rather than a D6.

While two 100s in a row only happens once in 10,000 times, when they do happen they are almost always part of short sequences like "27,100,100" or "87,62,100,100" rather than "53,100,14,100,100".

On the other hand, when you ctrl+f for a single "100" in your mind and count backwards until you get another 100, you'll almost always encounter an odd number first before encountering another "100" and have to disregard the sequence. But occasionally the 100s will appear close together and by chance there won't be any odd numbers between them. So you might see "9,100,82,62,100" or "13,44,100,82,100" or "99,100,28,100" or "69,12,100,100".

Another way to make it more intuitive might be to imagine that you have to get several 100s in a row / several 100s rather than just two. E.g. For four 100s: Ctrl+f "100,100,100,100" in your mind. Half the time it will be proceeded by an odd number for length 4, a quarter of the time it will be length 5, etc. Now look for all of the times that four 100s appear without there being any odd numbers between them. Some of these will be "100,100,100,100", but far more will be "100,32,100,100,88,100" and similar. And half the time there will be an odd number immediately before, a quarter of the time it will be odd-then-even before, etc.