Why is it worse for x risk for China to win the AI race? My understanding of the standard threat model is that, at some point, governments will need to step in and shut down or take control over profitable and popular projects for the good of all society. I look at China, and I look at the US, and I can't say "the US is the country I would bet on to hit the big red button here". There's got to be something I'm missing here.

I am a strong believer that nanotechnology is possible, which seems to be a sort of antimeme. And tons of people who should really know better seem to consider the acknowledgement of the physical possibility of Drexlerish nanotech as evidence someone is crazy - it is amusing to look at the AGI takes of these same people five years ago. They are mostly using the exact same idiotic intuitions in exactly the same way for the exact same reasons.  But maybe this being an antimeme is good? Perhaps its best people are holding the idiot ball on the topic? On one hand, I don't think lying is good, even by omission. And to the extent denying nanotech is load-bearing in their claims that takeoff will be slow (by the Christiano definition) then getting them to see through the antimeme is useful - as an aside I think people do forget that we have seen little evidence so far, at least in terms of economic growth, that we are living in Christiano's predicted world. I get the impression, sometimes, some people think we have.  But also, we are getting very powerful tools that makes a Drexlarian project more and more plausible, which has its own risks and even indirect risks of increasing available compute. So perhaps we are fortunate nanotechnology is so incredibly low status? As Sama would probably just try to do it if it were not.

I listened to the books Original Sin: President Biden's Decline, Its Cover-up, and His Disastrous Choice to Run Again and The Divider: Trump in the White House, 2017–2021. Both clearly have an axe to grind and I don't have enough US politics knowledge to know which claims are fair, and which ones are exaggerations and/or are missing important context, but these two books are sufficiently anti-correlated that it seems reasonable to update based on the intersection of the 2 books. Here are some AGI-relevant things I learned: * It seems rough to avoid sycophancy dynamics as president: * There are often people around you who want to sabotage you (e.g. to give more power to another faction), so you need to look out for saboteurs and discourage disloyalty. * You had a big streak of victories to become president, which probably required a lot of luck but also required you to double down on the strength you previously showed and be confident in your abilities to succeed again in higher-stakes positions. * Of course you can still try to be open to new factual information, but being calibrated about how seriously to take dissenting points of view sounds rough when the facts are not extremely crisp and legible. * This makes me somewhat more pessimistic about how useful AGI advisors could be. * Unconditional truthfulness (aka never lying) seems very inconvenient in politics. It looks quite common to e.g. have to choose between truthfulness and loyalty in an environment that strongly encourages loyalty. Seems especially convenient when the lie is about some internal state ("did you know that Biden was too old?" "do you think the current policy of the admin you work for is good?"). But even regular cover-ups about factual information seem quite common. * I think this makes designing good model specs for AGI advisors potentially quite hard, especially if the AGI advisors also have to answer questions from journalists and other adversarial entities. * I wonder

if i had to pinpoint what I think is the biggest issue with the rationality and ea community, i'd say that ea thinks wisdom is compassion, and rationality thinks wisdom is intelligence but in fact, wisdom is a secret third thing which should steer intelligence and compassion

Suppose AGI happens in 2035 or 2045. Will takeoff be faster, or slower, than if it happens in 2027? Intuition for slower: In the models of takeoff that I've seen, longer timelines is correlated with slower takeoff. Because they share a common cause: the inherent difficulty of training AGI. Or to put it more precisely, there's all these capability milestones we are interested in, such as superhuman coders, full AI R&D automation, AGI, ASI, etc. and there's this underlying question of how much compute, data, tinkering, etc. will be needed to get from milestone 1 to 2 to 3 to 4 etc., and these things are probably all correlated (at least in our current epistemic state). Moreover, in the 2030's the rate of growth of inputs such as data, compute, etc. will have slowed, so all else equal the pace of takeoff should be slower. Intuition for faster: That was all about correlation. Causally, it seems clear that longer timelines cause faster takeoff. Because there's more compute lying around, more data available, more of everything. If you have (for example) just reached the full automation of AI R&D, and you are trying to do the next big paradigm shift that'll take you to ASI, you'll have orders of magnitude more compute and data to experiment with (and your automated AI researchers be both more numerous and serially faster!) if it's 2035 instead of 2027. "So what?" the reply goes. "Correlation is what matters for predicting how fast takeoff will be in 2035 or 2045. Yes you'll have + 3 OOMs more resources with which to do the research, but (in expectation) the research will require (let's say) +6 OOMs more resources." But I'm not fully satisfied with this reply. Apparent counterexample: Consider the paradigm of brainlike AGI, in which the tech tree is (1) Figure out how the human brain works, (2) Use those principles to build an AI that has similar properties, i.e. similar data-efficient online learning blah blah blah, and (3) train that AI in some simulation environment si

What's the Elo rating of optimal chess? I present four methods to estimate the Elo Rating for optimal play: (1) comparing optimal play to random play, (2) comparing optimal play to sensible play, (3) extrapolating Elo rating vs draw rates, (4) extrapolating Elo rating vs depth-search. 1. Optimal vs Random Random plays completely random legal moves. Optimal plays perfectly. Let ΔR denote the Elo gap between Random and Optimal. Random's expected score is given by E_Random = P(Random wins) + 0.5 × P(Random draws). This is related to Elo gap via the formula E_Random = 1/(1 + 10^(ΔR/400)). First, suppose that chess is a theoretical draw, i.e. neither player can force a win when their opponent plays optimally. From Shannon's analysis of chess, there are ~35 legal moves per position and ~40 moves per game. At each position, assume only 1 move among 35 legal moves maintains the draw. This gives a lower bound on Random's expected score (and thus an upper bound on the Elo gap). Hence, P(Random accidentally plays an optimal drawing line) ≥ (1/35)^40 Therefore E_Random ≥ 0.5 × (1/35)^40. If instead chess is a forced win for White or Black, the same calculation applies: Random scores (1/35)^40 when playing the winning side and 0 when playing the losing side, giving E_Random ≥ 0.5 × (1/35)^40. Rearranging the Elo formula: ΔR = 400 × log₁₀((1/E_Random) - 1) Since E_Random ≥ 0.5 × (1/35)^40 ≈ 5 × 10^(-62): The Elo gap between random play and perfect play is at most 24,520 points. Random has an Elo rating of 477 points[1]. Therefore, the Elo rating of Optimal is no more than 24,997 points. 2. Optimal vs Sensible We can improve the upper-bound by comparing Optimal to Sensible, a player who avoids ridiculous moves such as sacrificing a queen without compensation. Assume that there are three sensible moves per in each position, and that Sensible plays randomly among sensible moves. Optimal still plays perfectly. Following the same analysis, E_Sensible ≥ 0.5 × (1/3)^40

"Why did you give our enemies the nuclear codes*? Now our country is going up is flames?" "Well, the truth is that the code is 50285193, and the code destroyed our country, and that which can be destroyed by the truth should be" *I don't know how nuclear codes actually work. I'm giving a counterargument to "that which can be destroyed by the truth should be"