1320

LESSWRONG
LW

HomeAll PostsConceptsLibrary
Best of LessWrong
Sequence Highlights
Rationality: A-Z
The Codex
HPMOR
Community Events
Subscribe (RSS/Email)
LW the Album
Leaderboard
About
FAQ
Customize
Load More

Quick Takes

Load More

Popular Comments

Connecting Words to Reality
Book 5 of the Sequences Highlights

To understand reality, especially on confusing topics, it's important to understand the mental processes involved in forming concepts and using words to speak about them.

jacquesthibs2d7188
How Stuart Buck funded the replication crisis
When you say “creating the replication crisis”, it read to me like he caused lots of people to publish papers that don’t replicate!
1a3orn3d6555
Frontier LLM Race/Sex Exchange Rates
Note that Nostalgebraist and Olli's comments on the original paper argue (imo cogently) that the original paper's framing is pretty misleading / questionable. It looks like many of their points would carry over to this.
Mo Putera2d452
Humanity Learned Almost Nothing From COVID-19
> The loss of gross world product is around $82 trio. over five years This isn't a retrospective assessment, it's the worst-case projection out of 4 scenario forecasts done in May 2020, ranging from $3.3 to $82 trillion over 5 years, using an undefined reasoning-nontransparent metric called "GDP@Risk" I couldn't find anything on after a quick search.
Load More
490Welcome to LessWrong!
Ruby, Raemon, RobertM, habryka
6y
76
130
Do One New Thing A Day To Solve Your Problems
Algon
15h
19
171
The "Length" of "Horizons"
Adam Scholl
6d
26
724The Company Man
Tomás B.
1mo
68
670The Rise of Parasitic AI
Adele Lopez
1mo
176
338Hospitalization: A Review
Logan Riggs
13d
19
159Humanity Learned Almost Nothing From COVID-19
niplav
3d
21
108EU explained in 10 minutes
Martin Sustrik
2d
6
277Towards a Typology of Strange LLM Chains-of-Thought
1a3orn
9d
24
207If Anyone Builds It Everyone Dies, a semi-outsider review
dvd
9d
64
51How Well Does RL Scale?
Toby_Ord
8h
7
230I take antidepressants. You’re welcome
Elizabeth
13d
8
142Meditation is dangerous
Algon
5d
31
171The "Length" of "Horizons"
Adam Scholl
6d
26
336Global Call for AI Red Lines - Signed by Nobel Laureates, Former Heads of State, and 200+ Prominent Figures
Charbel-Raphaël
1mo
27
485How Does A Blind Model See The Earth?
henry
2mo
40
Load MoreAdvanced Sorting/Filtering
[Today]Leipzig – ACX Meetups Everywhere Fall 2025
[Today]AGI Forum @ Purdue University
October Meetup - One Week Late
AI Safety Law-a-thon: We need more technical AI Safety researchers to join!
Eric Neyman7h*9437
0
California state senator Scott Wiener, author of AI safety bills SB 1047 and SB 53, just announced that he is running for Congress! I'm very excited about this, and I wrote a blog post about why. It’s an uncanny, weird coincidence that the two biggest legislative champions for AI safety in the entire country announced their bids for Congress just two days apart. But here we are.* In my opinion, Scott Wiener has done really amazing work on AI safety. SB 1047 is my absolute favorite AI safety bill, and SB 53 is the best AI safety bill that has passed anywhere in the country. He's been a dedicated AI safety champion who has spent a huge amount of political capital in his efforts to make us safer from advanced AI. On Monday, I made the case that donating to Alex Bores -- author of the New York RAISE Act -- calling it a "once in every couple of years opportunity", but flagging that I was also really excited about Scott Wiener. I plan to have a more detailed analysis posted soon, but my bottom line is that donating to Wiener today is about 75% as good as donating to Bores was on Monday, and that this is also an excellent opportunity that will come up very rarely. (The main reason that it looks less good than donating to Bores is that he's running for Nancy Pelosi's seat, and Pelosi hasn't decided whether she'll retire. If not for that, the two donation opportunities would look almost exactly equally good, by my estimates.) (I think that donating now looks better than waiting for Pelosi to decide whether to retire; if you feel skeptical of this claim, I'll have more soon.) I have donated $7,000 (the legal maximum) and encourage others to as well. If you're interested in donating, here's a link. Caveats: * If you haven't already donated to Bores, please read about the career implications of political donations before deciding to donate. * If you are currently working on federal policy, or think that you might be in the near future, you should consider whether it m
Jesse Hoogland3hΩ6244
0
We recently put out a new paper on a scalable generalization of influence functions, which quantify how training data affects model behavior (see Nina's post). I'm excited about this because it takes a completely new methodological approach to measuring influence.  Instead of relying on a Hessian inverse (which is ill-defined and expensive), our new "Bayesian" influence functions (BIF) rely on a covariance calculation (which can be scalably estimated with MCMC). This approach is more theoretically sound (no more Hessian inverses), and it achieves what I think are a more desirable set of engineering tradeoffs (better model-size scaling but worse dataset-size scaling).   At Timaeus, we think these kinds of techniques are on the critical path to safety. Modern alignment techniques like RLHF and Constitutional AI are about controlling model behavior by selecting the right training data. If this continues to be the case, we will need better tools for understanding and steering the pipeline from data to behavior. It's still early days for the BIF. We've done some initial validation on retraining benchmarks and other quantitative tests (follow-up work coming soon), where the BIF comes out looking strong, but more work will be needed to understand the full set of costs and benefits. As that foundation gets established, we expect we'll be able to start applying these techniques directly to safety-relevant problems.  You can read the full announcement thread on X (reproduced below):
leogao17h5135
Daniel Kokotajlo, Kaarel, and 5 more
10
I think it would be really bad for humanity to rush to build superintelligence before we solve the difficult problem of how to make it safe. But also I think it would be a horrible tragedy if humanity never ever built superintelligence. I hope we figure out how to thread this needle with wisdom.
Wei Dai2hΩ270
0
I want to highlight a point I made in an EAF thread with Will MacAskill, which seems novel or at least underappreciated. Here's the whole comment for context, with the specific passage bolded: I think my point in the opening comment does not logically depend on whether the risk vs time (in pause/slowdown) curve is convex or concave[1], but it may be a major difference in how we're thinking about the situation, so thanks for surfacing this. In particular I see 3 large sources of convexity: 1. The disjunctive nature of risk / conjunctive nature of success. If there are N problems that all have to solved correctly to get a near-optimal future, without losing most of the potential value of the universe, then that can make the overall risk curve convex or at least less concave. For example compare f(x) = 1 - 1/2^(1 + x/10) and f^4. 2. Human intelligence enhancements coming online during the pause/slowdown, with each maturing cohort potentially giving a large speed boost for solving these problems. 3. Rationality/coordination threshold effect, where if humanity makes enough intellectual or other progress to subsequently make an optimal or near-optimal policy decision about AI (e.g., realize that we should pause AI development until overall AI risk is at some acceptable level, or something like this but perhaps more complex involving various tradeoffs), then that last bit of effort or time to get to this point has a huge amount of marginal value. I think this kind of approach can backfire badly (especially given human overconfidence), because we currently don't know how to judge progress on these problems except by using human judgment, and it may be easier for AIs to game human judgment than to make real progress. (Researchers trying to use LLMs as RL judges apparently run into the analogous problem constantly.) What if the leaders can't or shouldn't trust the AI results? 1. ^ I'm trying to coordinate with, or avoid interfering with, people who are tryin
plex6h*144
0
2021 superforecasters vs 2025 reality: "Current performance on [the MATH] dataset is quite low--6.9%--and I expected this task to be quite hard for ML models in the near future. However, forecasters predict more than 50% accuracy* by 2025! This was a big update for me. (*More specifically, their median estimate is 52%; the confidence range is ~40% to 60%, but this is potentially artifically narrow due to some restrictions on how forecasts could be input into the platform.)"(source) Reality: 97.3%+ [1] (on a narrowed subset of only the hardest questions, including just difficulty 5 ones, called MATH-500 which was made because the original benchmark got saturated) Reliable forecasting requires either a generative model of the underlying dynamics, or a representative reference class. The singularity has no good reference class, so people trying to use reference classes rather than gears modelling will predictably be spectacularly wrong. 1. ^
Eric Neyman2d10771
Karl Krueger
2
I think that people concerned with AI safety should consider giving to Alex Bores, who's running for Congress. Alex Bores is the author of the RAISE Act, a piece of AI safety legislation in New York that Zvi profiled positively a few months ago. Today, Bores announced that he's running for Congress. In my opinion, Bores is one of the best lawmakers anywhere in the country on the issue of AI safety. I wrote a post making the case for donating to his campaign. If you feel persuaded by the post, here's a link to donate! (But if you think you might want to work in government, then read the section on career capital considerations before donating.) Note that I expect donations in the first 24 hours to be ~20% better than donations after that, because donations in the first 24 hours will help generate positive press for the campaign. But I don't mean to rush anyone: if you don't feel equipped to assess the donation opportunity on your own terms, you should take your time!
Daniel Tan3h42
speck1447
1
Two hypotheses I have (not necessarily conflicting) for why CoT improves model performance 1. Decomposition. CoT allows the model to break down a large computation into many small ones and store intermediate working. 2. Self-elicitation. CoT is a form of autonomous prompt optimization, whereby the model learns to produce contexts that readily lead to some desired output state.  Math, programming are likely described mainly as 1), but I suspect there are situations better described as 2) 
Load More (7/59)
First Post: Taboo Your Words