1413

LESSWRONG
LW

HomeAll PostsConceptsLibrary
Best of LessWrong
Sequence Highlights
Rationality: A-Z
The Codex
HPMOR
Community Events
Subscribe (RSS/Email)
LW the Album
Leaderboard
About
FAQ
Customize
Load More

Quick Takes

Your Feed
Load More

Popular Comments

Science Isn't Enough
Book 4 of the Sequences Highlights

While far better than what came before, "science" and the "scientific method" are still crude, inefficient, and inadequate to prevent you from wasting years of effort on doomed research directions.

Jeremy Gillen2d*781
Resampling Conserves Redundancy (Approximately)
Alfred Harwood and I were working through this as part of a Dovetail project and unfortunately I think we’ve found a mistake. The Taylor expansion in Step 2 has the 3rd order term o(δ3)=16[2(√P[X])3](−δ[X])3. This term should disappear as δ[X] goes to zero, but this is only true if √P[X] stays constant. The Γ transformation in Part 1 reduces (most terms of) P[X] and Q[X] at the same rate, so √P[X] decreases at the same rate as δ[X]. So the 2nd order approximation isn’t valid. For example, we could consider two binary random variables with probability distributions P(x=0)=zp and P(X=1)=1−zp and Q(X=0)=zq and Q(X=1)=1−zq. If δ[X]=√P(X)−√Q(X), then δ[X]→0 as z→0. But consider the third order term for X=0 which is 13(√Q(0)−√P(0)√P(0))3=13(√zq−√zp√zp)3=13(√q−√p√p)3 This is a constant term which does not vanish as z→0. We found a counterexample to the whole theorem (which is what led to us finding this mistake), which has KL(X2→X1→Λ′)max[KL(X1→X2→Λ),KL(X2→X1→Λ)]>10, and it can be found in this colab. There are some stronger counterexamples at the bottom as well. We used sympy because we were getting occasional floating point errors with numpy. Sorry to bring bad news! We’re going to keep working on this over the next 7 weeks, so hopefully we’ll find a way to prove a looser bound. Please let us know if you find one before us!
David Gross17h314
Meditation is dangerous
Things I'd like to know: 1. What is the baseline things-going-tits-up-mentally rate for people similarly situated to those who take on meditation, and how does that compare to the rate for people who begin to meditate? There's a smell of "the plural of anecdote is data" about this post. People at high risk for mental illness can go around the bend while meditating? Well, they can go around the bend watching TV or chatting with Claude too. How much more dangerous is meditation than the typical range of alternative activities? 2. There's bad, and then there's Bad. Ask a reformed alcoholic whether there were any negative side effects of giving up alcohol, and they'll tell you it was a bit like an anal probe of the soul with the devil's pitchfork for a month or two at least. Is that a cautionary tale that should steer you away from sobriety, or just par for that otherwise worthy course? Some of the practitioners of meditation (esp. in the Buddhist tradition) think we're most of us addicts to the chimerical delights of the senses, and that of course it'll be a struggle to overcome that. FWIW. 3. Is this like psychedelics, where if you take them in the context of a long-standing ritual practice with lots of baked-in wisdom, things will probably go okay or at least they'll know how to give you a soft pillow to land on if you get too far out there; but if you take them in some arbitrary context there's no telling how it'll turn out? How do outcomes look for people who meditate in an institutional context with feedback from a seasoned veteran vs. those who meditate based on e.g. enthusiastic blog posts? Not saying you're wrong, but answers to things like this would help me know what to do with your observations.
Raemon3d3315
How I Became a 5x Engineer with Claude Code
Your process description sounds right (like, the thing I would aspire to, although I don't consistently do it – in particular, I've identified it'd be good if I did more automated testing, but haven't built that into my flow yet).  But, you don't really spell out the "and, here's why I'm pretty confident this is a 5x improvement."  A few months ago I'd have been more open to just buying the "well, I seem to be shipping a lot of complex stuff", but, after the METR "turns out a lot of devs in our study were wrong and were actually slowed down, not sped up", it seems worth being more skeptical about it.  What are the observations that lead you to think you're 5x? (also, 5x is a somewhat specific number, do you mean more like 'observations suggest it's specifically around 5x' or more like 'it seems like a significant speedup, but I can tell I'm still worse than the '10x' programmers around me, and, idk, eyeballing it as in the middle?) (I don't mean this to be like, super critical or judgmental, just want to get a sense of the state of your evidence)
Load More
488Welcome to LessWrong!
Ruby, Raemon, RobertM, habryka
6y
76
October Meetup - One Week Late
Fri Oct 24•Edmonton
AI Safety Law-a-thon: We need more technical AI Safety researchers to join!
Sat Oct 25•Online
Hamburg – ACX Meetups Everywhere Fall 2025
Sat Oct 18•Hamburg
ACX/EA Lisbon October 2025 Meetup
Sat Oct 18•Lisbon
First Post: When Science Can't Help
133
The "Length" of "Horizons"
Adam Scholl
2d
20
262
Towards a Typology of Strange LLM Chains-of-Thought
1a3orn
5d
21
714The Company Man
Tomás B.
25d
65
658The Rise of Parasitic AI
Adele Lopez
1mo
175
331Hospitalization: A Review
Logan Riggs
9d
18
97Meditation is dangerous
Algon
20h
13
262Towards a Typology of Strange LLM Chains-of-Thought
1a3orn
5d
21
191If Anyone Builds It Everyone Dies, a semi-outsider review
dvd
5d
43
227I take antidepressants. You’re welcome
Elizabeth
9d
8
185The Most Common Bad Argument In These Parts
J Bostock
7d
38
102Cheap Labour Everywhere
Morpheus
2d
24
133The "Length" of "Horizons"
Adam Scholl
2d
20
123That Mad Olympiad
Tomás B.
3d
11
336Global Call for AI Red Lines - Signed by Nobel Laureates, Former Heads of State, and 200+ Prominent Figures
Charbel-Raphaël
1mo
27
315Why you should eat meat - even if you hate factory farming
KatWoods
23d
90
Load MoreAdvanced Sorting/Filtering
faul_sname7h223
Buck, Thane Ruthenis, and 1 more
3
Why is it worse for x risk for China to win the AI race? My understanding of the standard threat model is that, at some point, governments will need to step in and shut down or take control over profitable and popular projects for the good of all society. I look at China, and I look at the US, and I can't say "the US is the country I would bet on to hit the big red button here". There's got to be something I'm missing here.
Tomás B.5h126
Karl Krueger, Algon
2
I am a strong believer that nanotechnology is possible, which seems to be a sort of antimeme. And tons of people who should really know better seem to consider the acknowledgement of the physical possibility of Drexlerish nanotech as evidence someone is crazy - it is amusing to look at the AGI takes of these same people five years ago. They are mostly using the exact same idiotic intuitions in exactly the same way for the exact same reasons.  But maybe this being an antimeme is good? Perhaps its best people are holding the idiot ball on the topic? On one hand, I don't think lying is good, even by omission. And to the extent denying nanotech is load-bearing in their claims that takeoff will be slow (by the Christiano definition) then getting them to see through the antimeme is useful - as an aside I think people do forget that we have seen little evidence so far, at least in terms of economic growth, that we are living in Christiano's predicted world. I get the impression, sometimes, some people think we have.  But also, we are getting very powerful tools that makes a Drexlarian project more and more plausible, which has its own risks and even indirect risks of increasing available compute. So perhaps we are fortunate nanotechnology is so incredibly low status? As Sama would probably just try to do it if it were not.
Fabien Roger7hΩ790
Buck, Jacob Pfau
3
I listened to the books Original Sin: President Biden's Decline, Its Cover-up, and His Disastrous Choice to Run Again and The Divider: Trump in the White House, 2017–2021. Both clearly have an axe to grind and I don't have enough US politics knowledge to know which claims are fair, and which ones are exaggerations and/or are missing important context, but these two books are sufficiently anti-correlated that it seems reasonable to update based on the intersection of the 2 books. Here are some AGI-relevant things I learned: * It seems rough to avoid sycophancy dynamics as president: * There are often people around you who want to sabotage you (e.g. to give more power to another faction), so you need to look out for saboteurs and discourage disloyalty. * You had a big streak of victories to become president, which probably required a lot of luck but also required you to double down on the strength you previously showed and be confident in your abilities to succeed again in higher-stakes positions. * Of course you can still try to be open to new factual information, but being calibrated about how seriously to take dissenting points of view sounds rough when the facts are not extremely crisp and legible. * This makes me somewhat more pessimistic about how useful AGI advisors could be. * Unconditional truthfulness (aka never lying) seems very inconvenient in politics. It looks quite common to e.g. have to choose between truthfulness and loyalty in an environment that strongly encourages loyalty. Seems especially convenient when the lie is about some internal state ("did you know that Biden was too old?" "do you think the current policy of the admin you work for is good?"). But even regular cover-ups about factual information seem quite common. * I think this makes designing good model specs for AGI advisors potentially quite hard, especially if the AGI advisors also have to answer questions from journalists and other adversarial entities. * I wonder
Matt Goldenberg3h4-2
0
if i had to pinpoint what I think is the biggest issue with the rationality and ea community, i'd say that ea thinks wisdom is compassion, and rationality thinks wisdom is intelligence but in fact, wisdom is a secret third thing which should steer intelligence and compassion
Daniel Kokotajlo1d291
Bogdan Ionut Cirstea, RussellThor, and 3 more
9
Suppose AGI happens in 2035 or 2045. Will takeoff be faster, or slower, than if it happens in 2027? Intuition for slower: In the models of takeoff that I've seen, longer timelines is correlated with slower takeoff. Because they share a common cause: the inherent difficulty of training AGI. Or to put it more precisely, there's all these capability milestones we are interested in, such as superhuman coders, full AI R&D automation, AGI, ASI, etc. and there's this underlying question of how much compute, data, tinkering, etc. will be needed to get from milestone 1 to 2 to 3 to 4 etc., and these things are probably all correlated (at least in our current epistemic state). Moreover, in the 2030's the rate of growth of inputs such as data, compute, etc. will have slowed, so all else equal the pace of takeoff should be slower. Intuition for faster: That was all about correlation. Causally, it seems clear that longer timelines cause faster takeoff. Because there's more compute lying around, more data available, more of everything. If you have (for example) just reached the full automation of AI R&D, and you are trying to do the next big paradigm shift that'll take you to ASI, you'll have orders of magnitude more compute and data to experiment with (and your automated AI researchers be both more numerous and serially faster!) if it's 2035 instead of 2027. "So what?" the reply goes. "Correlation is what matters for predicting how fast takeoff will be in 2035 or 2045. Yes you'll have + 3 OOMs more resources with which to do the research, but (in expectation) the research will require (let's say) +6 OOMs more resources." But I'm not fully satisfied with this reply. Apparent counterexample: Consider the paradigm of brainlike AGI, in which the tech tree is (1) Figure out how the human brain works, (2) Use those principles to build an AI that has similar properties, i.e. similar data-efficient online learning blah blah blah, and (3) train that AI in some simulation environment si
Cleo Nardo3d*764
cosmobobak, Dmitry Vaintrob, and 7 more
21
What's the Elo rating of optimal chess? I present four methods to estimate the Elo Rating for optimal play: (1) comparing optimal play to random play, (2) comparing optimal play to sensible play, (3) extrapolating Elo rating vs draw rates, (4) extrapolating Elo rating vs depth-search. 1. Optimal vs Random Random plays completely random legal moves. Optimal plays perfectly. Let ΔR denote the Elo gap between Random and Optimal. Random's expected score is given by E_Random = P(Random wins) + 0.5 × P(Random draws). This is related to Elo gap via the formula E_Random = 1/(1 + 10^(ΔR/400)). First, suppose that chess is a theoretical draw, i.e. neither player can force a win when their opponent plays optimally. From Shannon's analysis of chess, there are ~35 legal moves per position and ~40 moves per game. At each position, assume only 1 move among 35 legal moves maintains the draw. This gives a lower bound on Random's expected score (and thus an upper bound on the Elo gap). Hence, P(Random accidentally plays an optimal drawing line) ≥ (1/35)^40 Therefore E_Random ≥ 0.5 × (1/35)^40. If instead chess is a forced win for White or Black, the same calculation applies: Random scores (1/35)^40 when playing the winning side and 0 when playing the losing side, giving E_Random ≥ 0.5 × (1/35)^40. Rearranging the Elo formula: ΔR = 400 × log₁₀((1/E_Random) - 1) Since E_Random ≥ 0.5 × (1/35)^40 ≈ 5 × 10^(-62): The Elo gap between random play and perfect play is at most 24,520 points. Random has an Elo rating of 477 points[1]. Therefore, the Elo rating of Optimal is no more than 24,997 points. 2. Optimal vs Sensible We can improve the upper-bound by comparing Optimal to Sensible, a player who avoids ridiculous moves such as sacrificing a queen without compensation. Assume that there are three sensible moves per in each position, and that Sensible plays randomly among sensible moves. Optimal still plays perfectly. Following the same analysis, E_Sensible ≥ 0.5 × (1/3)^40
lionrock2m10
0
"Why did you give our enemies the nuclear codes*? Now our country is going up is flames?" "Well, the truth is that the code is 50285193, and the code destroyed our country, and that which can be destroyed by the truth should be" *I don't know how nuclear codes actually work. I'm giving a counterargument to "that which can be destroyed by the truth should be"
Load More (7/52)