Quick Takes — LessWrong

LESSWRONG
LW

i wish more showers adopted the design where there is an on/off knob and a temperature knob. it's so obviously better. on the other hand, i hate the single knob showers

faul_sname's Shortform

faul_sname17h50

AI agents might not need to develop long-term drives of their own to perform well at long-term tasks, if they can use humans for that purpose. As a corollary, AI agents that do not have long-term drives of their own can still act in long-term coherent ways, if they inhabit a world with abundant, easy-to-hire human labor. If it's cheaper to hire a human to get a scaffolded AI system to stay on track towards achieving some goal than it is to automate that monitoring, then it probably makes sense to just use the human.

Concretely, consider an AI system tasked ... (read more)

Kaarel1h20

I think this doesn't make sense for solving genuinely hard scientific/technological/mathematical/philosophical problems such as the strawberry problem. (It makes sense when the big task has a basically known decomposition into a large number of small easy tasks though.) A central issue is that good high-level decisions are very important, there are very many of them, and they need to be made with deep understanding of the domain/[design space], which the human in this setup doesn't have by default for hard problems and which can only be gained by spending ... (read more)

Shortform

Cleo Nardo1d*10365

Small donors should not worldview-diversify.

Occasionally I encounter small donors (e.g. 10% pledgers earning <$200K) with highly specialised skills and knowledge (e.g. working on a sub-sub-topic of an EA cause area) who donate primarily to GiveWell top charities. These people do incredible amounts of good, and are highly commendable.

That said, I think would probably do more good by donating according to their inside view and special knowledge. Worldview diversification makes sense for large funders like Coefficient Giving, but their reasons don't apply ... (read more)

Showing 3 of 4 replies (Click to show all)

1Myron Hedderson13h

Some pushback seems warranted, so I upvoted and agree-voted. On the other hand, if you're giving to a registered charity anyway, you can get tax credits, which means you can give more for the same net cost to you. in Canada, you can get a 50% tax credit on donations to registered charities (meaning basically you get half back and can choose to donate twice as much for the same after tax cost, up to 75% of your annual income), and RC Forward lets you donate to GiveWell recommended charities (they forward on to various EA charities that aren't registered in Canada, whereas RC Forward is, so you get the tax credit). I don't know how generous the tax situation is in other countries, but typically there is some tax benefit, so if you're going to give to a recognized charity, then DAFs make sense. I've used them for "I'm not exactly sure where I'm going to give because I haven't done all the research I want to, but I want that tax credit, and it is inaccessible for prior years, so I'd like to give now and hold it in a DAF". Which ended up with "I can give a large chunk of saved up funds from prior years to GiveWell's mostly-unrestricted fund in the year when the US government defunded USAID" - worked out well! But, I did find an opportunity last year (AIGS Canada - basically trying to inform Canadian parliamentarians about superintelligence risks and influence policy in a direction of "AI will be really impactful, don't just treat it as normal technology") where they needed money urgently, didn't want to take money from non-Canadians because that's a public relations vulnerability if trying to influence Canadian government policy, and weren't a registered charity - and I felt the twinge of having stuck everything I'd planned to give for that year into a DAF, which is not ideal because giving them enough for a few months more operations really does seem to have had a good effect and I endorse having done it, in retrospect. So "don't put all you're planning to give into a

Elizabeth2h20

I think for small donors, donating to the best unregistered charity is >>2x times the best registered charity, for the reasons OP outlines: registered charities are much better covered by large institutions, and lots of people are overanchored on registration so the unregistered are neglected by comparison.

The counterargument is that bednets/givedirectly are just pretty good and it's unlikely any particular new thing beats them. Which is a fine approach, but not what we're talking about here.

4Neel Nanda16h

+1, especially with the vast majority of future Anthropic employee donations already locked in to DAFs

Fabien's Shortform

Fabien Roger1d1812

Here are the 2025 AI safety papers and posts I like the most.

The list is very biased by my taste, by my views, by the people who had time to argue that their work is important to me, and by the papers that were salient to me when I wrote this list. I am highlighting the parts of papers I like, which is also very subjective. (This is similar to the 2024 edition here.)

Bangers on multiple dimensions

★★★ You can measure time horizon, and it grows predictably-ish (Measuring AI Ability to Complete Long Tasks)
★★★ Black-box techniques work better than you think + y... (read more)

Sheikh Abdur Raheem Ali3h1-2

I think that this should really be a top-level post.

5the gears to ascension4h

This list seems primarily focused on local alignment. Have you seen anything that you felt was promising for being a path to asymptotic alignment?

Anthony DiGiovanni's Shortform

Anthony DiGiovanni2d30

Why I’m unconvinced by Tegmark’s argument for the mathematical universe hypothesis

The basic argument seems to be:

A complete description of external reality must be free of human-idiosyncratic concepts (“baggage”).
Any intrinsic properties of external reality, i.e. properties other than relational properties, are baggage.
Therefore a complete description of external reality must be purely relational. (And a mathematical structure is the only candidate for “purely relational”.)

I don’t see why we should buy (2).

As far as I know, there are two arguments for (2).... (read more)

2TAG13h

You may be running two arguments together , there. Freedom from human independent concepts is supposed to support the idea of a mathematical universe, not a relational one. Then there is the claim that the idea that some , but only some, maths exists materially, is unnecessary baggage. Neither argument is totally convincing. It isn't obvious that maths isn't a human invention. It isn't obvious that an external world has to be independent of human concepts, as well as human imagination. It isn't obvious that maths can exist immaterially

Anthony DiGiovanni4h20

Freedom from human independent concepts is supposed to support the idea of a mathematical universe, not a relational one

Tegmark says (p. 10 here) "the only intrinsic properties of a mathematical structure are its relations". So I think I am representing his actual argument for MUH in the OP. I think "the claim that the idea that some , but only some, maths exists materially, is unnecessary baggage" is meant to derive the Level IV multiverse from MUH.

It isn't obvious that maths isn't a human invention. It isn't obvious that an external world has to be indep

... (read more)

speck1447's Shortform

speck14471d110

Something I'm thinking about today: frontier LLMs have a pretty unusual capabilities profile. This means one of two things: either I should think of LLMs as leveraging massive amounts of necessary compute and the problems they can solve as much more compute-vulnerable than I thought they were (i.e. this is Deep Blue and everything is kind of chess) or multiple intelligences models are simply true, in that cognition has multiple parts that don't necessarily have anything to do with each other. The latter predicts that step changes in capabilities are availa... (read more)

2J Bostock1d

I think LLMs have a high proportion of fast/shallow/memorizing circuits while humans have a high proportion of slow/deep/generalizing circuits. Increasing circuit depth and generalizing in transformers requires a kind of phase change (which is actually very similar to ones in physics/chemistry, it's not an inappropriate metaphor) and I'd be slightly shocked if there wasn't an analogous process in human brains (though possibly more continuous). It seems like transformers are worse at this phase change than human brains are, so they disproportionately leverage large amounts of data and memorization. This, for me, explains most of the important differences between LLMs and people.

9Vladimir_Nesov1d

Cognition could have multiple parts that have nothing to do with each other, yet evolution (which has no mind at all) still found them. Thus it's difficult to rule out that scaling some general learning method might find more of such parts, even when they aren't found at lower levels of compute. While rapid scaling continues, it's hard to see if there are genuine obstructions that won't fall to feasible scale. An LLM with a chain of thought is a general-purpose computer. In principle it could be implementing any cognitive algorithms, if the learning process gets the weights to do that. So a claim that an LLM can't do something is a claim that the learning methods won't train it to do it. With more scale, learning methods work better. And using RLVR, it might be possible to teach LLMs to experiment with methods for training themselves to do more things (implementing more parts of cognition in the weights), much more efficiently than evolution did to discover the human brain.

speck14475h10

I agree that this is the relevant consideration. I think that if cognition has many parts, we should actually expect some parts that humans use to be completely missing in LLMs (and vice-versa), and it's not clear to me whether I should expect scaling architecture to actually produce more parts in this way, I have some intuitions that say (for combinatorial reasons) that within a certain architecture, training dynamics will eventually stop favoring the formation of circuits past a certain size regardless of how many layers you stack, but I am not that conf... (read more)

Brendan Long's Shortform

Brendan Long6h20

I'm cautiously optimistic about my new Claude Coach GitHub repo. I want to work out more but hate trying to decide what to do and tracking things, especially when I'm not working with a full gym. Now I just open Claude Code and ask it what to do (specifying the gym), do the work out, then update it with what I did and how it felt. It creates a PR to track the session and update the plan.

I still hate working out, but at least I don't have to go anywhere, deal with any people, or think about it all.

Algon's Shortform

Algon1d50

Perhaps my favourite relation in physics is

t/T = (l/L)^{1-k/2}.

This says that for a bunch of particle in a potential V = a x^k, if you let the system evolve over time T forming a path which has size L in some sense, then there is another path which is a re-scaled version of the original one s.t. if it has size l, then the time taken to form this new path is t.

We can use this trick to create a bunch of “scaling laws” for simple physical systems. For example:

1) Let V = a x ^{-1} i.e. a gravitational potential. Then we have k=-1, so

t/T = (l/L)^{3/2}

(t/T)^{2} ... (read more)

Showing 3 of 5 replies (Click to show all)

3papetoast12h

Good job to chatgpt actually, I just dumped your post to it lol. Also you can just ask LLM to typeset the math but keep other content verbatim and im certain it will one shot it

gwern7h30

It'll one-shot easy cases, yeah. And if you want to convert to HTML/Unicode for places where you don't have direct LaTeX support, you can also have a LLM do that, albeit there area lot of edge-cases and I don't think LLMs will usually use more exotic Unicode like FRACTION SLASH for things like '3/2' etc, so I have a big script for that: https://gwern.net/static/build/latex2unicode.py (Github).

2Algon11h

Oh, I didn't realize you could just paste latex into the LW editor and it autoparses, I thought you had to use ctrl+4 to summon a block in which you write math equations.

Expertium's Shortform

Expertium2d*14-6

Do you know a person who believes that ASI will be created in <50 years who ISN'T in the LW/rationalists circle?

My parents don't believe that a superintelligent AI will be created within this century, or ever for that matter, or that AI will ever take jobs. My relatives laugh at the idea of AI solving a high school math problem and think state-of-the-art AI is on the level of GPT-2 (I mean that the capabilities they have in mind are on the level of GPT-2, not that they know what GPT-2 is). My friend who is an organic chemist ~~laughs at the idea of AI doi~~... (read more)

Showing 3 of 11 replies (Click to show all)

Carl Feynman11h92

I've been in the mostly-academic AI circles in the Boston area for decades. Lots of people in these circles think ASI is plausibly close. I think it's difficult to pay close technical attention to the field and not think that AI is currently par-human, and improving every year. Many of them disagree with LessWrong concensus that it will be fatal, of course. Or simply haven't thought it through.

9Haiku1d

Most people in the PauseAI movement are not in the LW/rationalist circle. Some joined as rationalists or EAs, especially early on, but today most are normies (or were when they joined, anyway). I personally found LessWrong and the forecasting community through AI Safety, not the reverse. I now organize for PauseAI Phoenix, a local group of PauseAI US. I have face-to-face conversations on a regular basis with people who believe ASI will be created within 2-20 years unless we prevent that from happening.

1wonder1d

To zoom out a bit, from the post I assume you benchmark ASI mostly by "replacing humans 100% in all jobs". Curious in why you specifically care about absolutely 100%? (Replacing 50% of humans is still significant imo.)

ChristianKl's Shortform

ChristianKl11h00

The four human theory suggesting bloodletting might be quite bad, but at the same time the might have recommended chicken soup for viral influenza like infections which is better than the treatments of modern medicine.

Liface's Shortform

Liface10d*773

I run a curated Discord for high agency people with Long COVID/ME (myalgic encephalomyelitis). The group includes tech founders, researchers, rationalists/EAs, etc. The focus is on troubleshooting each other's conditions actively, as well as creating a body of knowledge to bring back to the wider community in the form of writing, education, projects, companies, etc.

Some know me as Liface, others as Liam Rosen - I have been in the Rationality community for over 10 years, previously in the Bay, now in New York City, and am the main moderator of r/slatestarc... (read more)

Showing 3 of 7 replies (Click to show all)

delton13713h30

Sorry to hear about your condition.. starting a community sounds good, I just emailed you. I went through a two year long Long COVID ordeal and now consider myself recovered. I've written several in-depth articles on Long COVID, and am working on a Long COVID dashboard (work in progress) to help make sense of all the trials conducted so far around the world.

3Eli Tyre8d

Badass. I wish you the best of skill.

4Ethan9d

Wishing you all the luck in the world, recovered from long covid this year after around 1.5 years of extreme suffering. Given the nature of the disease I'd suggest opening up the discord to others who do not have ME and have more energy to give, but not sure how to filter people who have genuine motivation to help. Quick note to everyone, covid is still around and still causing long covid, even if you don't get long covid, it has deleterious effects on your body, best way to avoid this is wearing a well fitted respirator.

leogao's Shortform

leogao17d385

I'm very confused why purchasing power varies so dramatically internationally. like why are there countries where everyone has very low wages but everything is also really cheap so it balances out? prima facie, huge disparities like this should get evened out by arbitrage.

the simple explanation is that some labor can only be performed locally, labor mobility is limited (immigration laws, people don't like moving, etc), and transportation costs for goods exist (shipping and tariffs).

however, global shipping is ridiculously cheap. and the economy increasingl... (read more)

Showing 3 of 17 replies (Click to show all)

8Viliam1d

As a SWE from Slovakia, not sure how representative is my experience, but... One big problem cooperating with companies in Germany is that when you apply for the job, they tell you "knowledge of German is not needed, all internal communication in our company is in English", and that in my experience is never true. (This sucks for me, because my brain somehow rejects the German language.) Hiring people from other countries is somehow complicated from the legal and/or tax perspective. I don't understand all the relevant details, and in theory the EU exists to prevent this, but in practice... my tax form gets more complicated when I have a foreign income, and I don't know how complicated it gets for the company, but I suspect that it might be a lot. Maybe enough to compensate for the 60% cheaper people? Some people have strong patriotic/union/whatever feelings that make them sabotage people from other countries to protect their own jobs. Especially when those other people are cheaper, and not bad at their jobs; that is when they are a greater threat. I had an experience in a Swiss company, when the Swiss employees collectively sabotaged the Slovaks in various ways; for example they always made a "mistake" of sending us wrong documentation and/or forgetting to give up access to the product database; so the work that was estimated to take three days, and would actually take me one day, took five, because I had to spend the first four days repeatedly asking for the correct documentation, and repeatedly telling them that they gave me database access to a wrong table, not the one I requested. Many people on both sides were aware that this was happening, but the Swiss company simply couldn't fire half of their employees, so they had to accept that the Slovaks cost half as much per day, but everything takes them twice as many days. I guess, if you want to exploit the cheap Eastern European brain power for a company in Germany, you should probably not hire local people at

p.b.20h20

Yeah, I guess there is just more friction than one would expect. I also found out that Slovaks are now more like 66% of the price of Germans, so either the margin is great or the income gap is smaller than I thought.

Also: Ease of communication is hard to overrate. I always enjoy working with people who have a very similar background compared to me (i.e. similar milieu growing up, not just same country), communication is so smooth.

1Closed Limelike Curves3d

The model you've described (hidden quality differences) is a huge part of it, yes. I'll try and find the paper, but in general nominal/market exchange rates tend to be stronger predictors of most objective, cross-comparable outcomes than indices that try to control for cost of living (PPP). If two goods/services that look equivalent are selling for different prices, it's usually (though not always) because there's some difference you're not able to measure.

Mateusz Bagiński's Shortform

Mateusz Bagiński9d4527

A model to track:

AI gets increasingly good at X, so good that strong pressures arise to increasingly automate it, even if AI is still quite bad at some subtle but very important aspects of X.
More and more X-related work/activity gets AI-automated, including the parts where AI is still bad or fails frequently and non-gracefully.
- This is partly driven by the fact that it's just ~psychologically hard for a human to exert effortful X-related cognition when AI would be more efficient at 99% of it.
In cases where being good at [those aspects of X AI is still bad a

2Mateusz Bagiński1d

Daniel Litt on Twitter: https://x.com/littmath/status/2035853161802404208

Mo Putera1d20

The adaptation is starting to happen in earnest, including by super-high-profile folks like Tao and other Field medalists like Tim Gowers, so I'm less worried than Litt.

2Mateusz Bagiński8d

This is compatible with a general vibe I'm picking up, but I'd also appreciate concrete pointers.

Ian McKenzie's Shortform

Ian McKenzie1d20

It seems like the switch where high-status brands went from making high quality products to featuring their name and logo prominently was good for both producer and consumer: the producer gets free advertising, and the consumer more clearly signals their wealth. I don't know the history of this, but I'm interested in how this happened – seems like there's a tough coordination problem, where if one brand/consumer switches then they just look tacky.

(Not all brands have switched, and maybe there's a new money/old money difference.)

Bridgett Kay's Shortform

Bridgett Kay16d*140

Meditations on Meditation:

I’ve always noticed something about meditation that I’ve never thought to articulate aloud before. (Note- I’m using mantra meditation as an example.) A beginning meditation practitioner- which is all I have ever been- is told to focus their awareness on the mantra, notice when their mind wanders without judgement, and then redirect their thoughts to the mantra. Similar instructions are given when the focus is the breath, etc. However, my experience of meditation has never been that simple; my attention comes in layers. The first a... (read more)

4Viliam1d

No experience, just a question: If each layer is quieter, what makes you think there is a specific number of layers, instead of... everyone reaching their limit at how quiet layers they can perceive (and for the yogic experts you researched that number happened to be 4)? Perhaps those layers were not there previously, but they get generated by working with the previous layer? (These are all just speculations. But I am suspicious about people putting specific numbers on these things.)

Bridgett Kay1d20

I'm not sure there are specifically 4 layers, but I do assume my mind is finite. There are probably many subconscious layers I can't perceive, but I doubt I could control or focus them if I can't even perceive them, so I don't worry about them.

leogao's Shortform

leogao1d4-2

lingao qiming is the hardest scifi I've ever read. it puts other "hard" scifi like project hail mary or three body problem to shame. the basic conceit of the book is that it's an isekai where some people discover a wormhole to a parallel universe exactly like ours but during the time of ming dynasty china, and decide to being 500 technical specialists and a bunch of modern supplies to the past to try and conquer ancient china. the vast majority of the book is devoted to discussing every single technical aspect in excruciating well-researched detail. you do... (read more)

Showing 3 of 5 replies (Click to show all)

JBlack1d20

Yes, I would have to agree here. The parts that are sci-fi (almost none) are not hard, and the parts that are hard are not sci-fi.

2leogao1d

afaict, the time travel part only exists to set the story up, and does not come up again at all. it exists only to set the story up. the entire rest of the story is stuff like this:

3Mateusz Bagiński1d

On the one hand, yeah. On the other hand, the rest of the story (AFAICT based on your description) isn't really that sci-fi, let alone "hard", except insofar as it's set up by the time travel. You could just as well write a story about the Spaniards ultra-strategizing about efficiently conquering the Mexica or the Inca.

Raemon's Shortform

Raemon19d*973

I'm seeing more sophisticated LLM-slop in the LW moderation queue.

Eight months ago, I wrote "hey, we're getting tons of AI-psychosis'd people, deluded into thinking their crackpot coherence/spiralism/emergence/ChatGPTAwakening experience is true and meaningful. We process like 15-20 of these a day."

Nowadays, we still get some of those, but, a lot less. Instead, I think we often now get a somewhat more sophisticated looking set of AI LLM slop. It's often something like:

"Me and ChatGPT have been working on some ML experiments for months, checking if we get s... (read more)

Showing 3 of 61 replies (Click to show all)

1wassname1d

That is very useful thanks, I'll give it a rewrite in that vein.

2papetoast3d

I basically don't read academic paper styled posts, so I am not your target audience, but this alone probably filtered a lot of readers regardless of the actual writing. It is pretty common for posts in alignment forum to have no comments too.

wassname1d10

That makes sense, probably the majority are in this camp.

AWeirdKid's Shortform

Futuristic Beby16d10

Why do some experiments on homeopathy contradict each other? What if, homeopathy works like placebo, and if you expect to prove it works, you do so, and if you expect to prove it doesn't, you do so as well?.. Could there be a scientific mechanism for placebo?

Viliam1d30

Placebo is powered by the belief of the person who uses it, not the person who researches it.

If person A takes the placebo to e.g. fix their blood pressure, and person B measures their blood pressure but doesn't know why, and person C organizes the experiment, the results should depend on A's beliefs -- because B doesn't have any, and C already receives the final numbers.

Kajus's Shortform

Kajus17d10

There are norms. Examples of norms: drive on the right side of the street. Do not ghost people. Text back your friends within a day. Do not post cringe. Do not distribute explicit materials without trigger warning. Do not enforce norms too tightly. Do not use LLMs in writing without letting people know. Norms are enforced by people (I will skip examples of how here).

Most of the norms are helpful, some are harmful. I'm particularly interested in norms around being cringe and creativity. Doing some things but being unskilled about it is just soo discouraged... (read more)

7Karl Krueger16d

My impression is that among "startup people", starting a company and having it fail is not "cringe"; it's a rite of passage — and one that can be repeated.

Viliam1d50

There is no contradiction between something being "cringe" and "a rite of passage". Actually, cringe things probably make good rites, because the signaling is (psychologically) costly.

Mikhail Samin's Shortform

Mikhail Samin17d*00

The leadership of CEA should resign.

https://forum.effectivealtruism.org/posts/XxXnPoGQ2eKsQx3FE/cea-s-response-to-sexual-harassment

Viliam1d30

The lesson I take here is that any organization that employs people should have a legal/HR expert available and actually call them immediately when anything unusual happens.

You may think that you are in the business of telling people how to send money to charities, but as soon as you employ people, you are also in the business of solving all kinds of things that concern the people you employ. And you most likely don't have any expertize in that, and things can get very serious, so you need to have someone's phone number and to actually use it.

You need an H... (read more)