All of Garrett Baker's Comments + Replies

Infinite eg energy would just push your scarcity to other resources, eg.

2Noosphere89
Compute, information/entropy and what people can do with their property all become abundant if we assume an infinite energy source. Compute and information/entropy become cheap, because the costs of running computations/getting information and entropy like the Landauef limit become mostly irrelevant if you can assume you can always generate the energy you need. Somewhat similarly, what people can do with their property becomes way more abundant with infinite energy machines, though here it depends on how the machine works, primarily because it allows people to set up their own governments with their own laws given enough time (because everything comes from energy, in the end), and this could end up undermining traditional governments.

This seems false given that AI training will be/is bottlenecked on energy.

I am sympathetic to, but unconvinced of the importance of animal suffering in general. However for those that are sympathetic to animal suffering, I could never understand their resistance to caring about wild animal suffering, a resistance which seems relatively common. So this post seems good for them.

This post does seem more of an EA forum sorta post though.

2eggsyntax
At a guess, many people have the intuition that we have greater moral responsibility for lives that we brought into being (ie farmed animals)? To me this seems partly reasonable and partly like the Copenhagen interpretation of ethics (which I disagree with).
1Milan W
I happen to care about animal suffering, and I am as baffled as you about the move of caring about animal suffering for explicitly anti-speciecist reasons yet dismissing wild animal suffering. Seems pretty inconsistent. Maybe it originates from a sort of wishful thinking? As in "looks intractable, therefore I wish it were unimportant, therefore it is".

SCOTUS decision that said a state had to, say, extradite somebody accused of "abetting an abortion" in another state.

Look no further than how southern states responded to civil rights rulings, and how they (back when it was still held) they responded to roe v wade. Of course those reactions were much harder than, say, simply neglecting to enforce laws, which it should be noted liberal cities & states have been practicing doing for decades. Of course you say you're trying to enforce laws, but you just subject all your members to all the requirements of ... (read more)

... and in the legal arena, there's a whole lot of pressure building up on that state and local resistance. So far it's mostly money-based pressure, but within a few years, I could easily see a SCOTUS decision that said a state had to, say, extradite somebody accused of "abetting an abortion" in another state.

What money based pressure are you thinking of? Cities, as far as I know, have and always will be much more liberal than the general populace, and ditto for the states with much of their populace in cities.

3jbash
This sort of tactic. This isn't necessarily the best example, just the literal top hit on a Google search. https://www.independent.co.uk/news/world/americas/us-politics/pam-bondi-ban-sanctuary-cities-funding-b2693020.html The tactic of threatening to discriminate against uncooperative states and localities is getting a lot of play. It's somewhat limited at the federal level because in theory the state and local policies they demand have to be related to the purpose of the money (and a couple of other conditions I don't remember). But the present fashion is to push that relation to the absolute breaking point.

For rights, political power in the US is very federated. Even if many states overtly try to harm you, there will be many states you can run to, and most cities within states will fight against this. Note state-wise weed legalization and sanctuary cities. And the threat of this happening itself discourages such overt acts.

If you're really concerned, then just move to california! Its much easier than moving abroad.

As for war, the most relevant datapoint is this metaculus question, forecasting a 15% of >10k american deaths before 2030, however it doesn't s... (read more)

7jbash
I lived in California long enough ago to remember when getting queer-bashed was a reasonable concern for a fair number of people, even in, say, Oakland. It didn't happen daily, but it happened relatively often. If you were in the "out" LGBT community, I think you probably knew somebody who'd been bashed. Politics influence that kind of thing even if it's not legal. ... and in the legal arena, there's a whole lot of pressure building up on that state and local resistance. So far it's mostly money-based pressure, but within a few years, I could easily see a SCOTUS decision that said a state had to, say, extradite somebody accused of "abetting an abortion" in another state. War in the continental US? No, I agree that's unlikely enough not to worry about. Civil unrest, followed by violent crackdowns on civil unrest, followed by more violent civil unrest, followed by factional riots, on the other hand...

Yeah, these are mysteries, I don't know why. TSMC I think did get hit pretty hard though. 

Politicians announce all sorts of things on the campaign trail, that usually is not much indication of what post-election policy will be.

Seems more likely the drop was from Trump tariff leaks than deepseek’s app.

If so, why were US electricity stocks down 20-28% (wouldn't we expect them to go up if the US wants to strengthen its domestic AI-related infrastructure) and why did TSMC lose less, percentage-wise, than many other AI-related stocks (wouldn't we expect it to get hit hardest)? 

6Thane Ruthenis
Supposedly Trump announced that back in October, so it should already be priced in. (Here's my attempt at making sense of it, for what it's worth.)

I also note that 30x seems like an under-estimate to me, but also too simplified. AIs will make some tasks vastly easier, but won't help too much with other tasks. We will have a new set of bottlenecks once we reach the "AIs vastly helping with your work" phase. The question to ask is "what will the new bottlenecks be, and who do we have to hire to be prepared for them?" 

If you are uncertain, this consideration should lean you much more towards adaptive generalists than the standard academic crop.

There's the standard software engineer response of "You cannot make a baby in 1 month with 9 pregnant women". If you don't have a term in this calculation for the amount of research hours that must be done serially vs the amount of research hours that can be done in parallel, then it will always seem like we have too few people, and should invest vastly more in growth growth growth!

If you find that actually your constraint is serial research output, then you still may conclude you need a lot of people, but you will sacrifice a reasonable amount of growth s... (read more)

3Garrett Baker
I also note that 30x seems like an under-estimate to me, but also too simplified. AIs will make some tasks vastly easier, but won't help too much with other tasks. We will have a new set of bottlenecks once we reach the "AIs vastly helping with your work" phase. The question to ask is "what will the new bottlenecks be, and who do we have to hire to be prepared for them?"  If you are uncertain, this consideration should lean you much more towards adaptive generalists than the standard academic crop.

The most obvious one imo is the immune system & the signals it sends. 

Others:

  • Circadian rhythm
  • Age is perhaps a candidate here, though it may be more or less a candidate depending on if you're talking about someone before or after 30
  • Hospice workers sometimes talk about the body "knowing how to die", maybe there's something to that

If that’s the situation, then why the “if and only if”, if we magically make then all believe they will die if they make ASI, then they would all individually be incentivized to stop it from happening independent of China’s actions.

I think that China and the US would definitely agree to pause if and only if they can confirm the other also committing to a pause. Unfortunately, this is a really hard thing to confirm, much harder than with nuclear.

This seems false to me. Eg Trump for one seems likely to do what the person who pays him the most & is the most loyal to him tells him to do, and AI risk worriers do not have the money or the politics for either of those criteria compared to, for example, Elon Musk.

2Nathan Helm-Burger
Ah, I meant, would agree to pause once things came to a head. Pretty sure these political leaders are selfish enough that if they saw clear evidence of their imminent demise, and had a safer option, they'd take the out.

Its on his Linkedin at least. Apparently since the start of the year.

I will note this sounds a lot like Turntrout's old Attainable Utility Preservation scheme. Not exactly, but enough that I wouldn't be surprised if a bunch of the math here has already been worked out by him (and possibly, in the comments, a bunch of the failure-modes identified).

Although I don't think the first example is great, seems more like a capability/observation-bandwidth issue.

I think you can have multiple failures at the same time. The reason I think this was also goodhart was because I think the failure-mode could have been averted if sonnet was told “collect wood WITHOUT BREAKING MY HOUSE” ahead of time.

If you put current language models in weird situations & give them a goal, I’d say they do do edge instantiation, without the missing “creativity” ingredient. Eg see claude sonnet in minecraft repurposing someone’s house for wood after being asked to collect wood.

Edit: There are other instances of this too, where you can tell claude to protect you in minecraft, and it will constantly tp to your position, and build walls around you when monsters are around. Protecting you, but also preventing any movement or fun you may have wanted to have.

4Jeremy Gillen
Fair enough, good points. I guess I classify these LLM agents as "something-like-an-LLM that is genuinely creative", at least to some extent. Although I don't think the first example is great, seems more like a capability/observation-bandwidth issue.

I don't understand why Remmelt going "off the deep end" should affect AI safety camp's funding. That seems reasonable for speculative bets, but not when there's a strong track-record available. 

6habryka
I have heard from many people near AI Safety camp that they also have judged AI safety camp to have gotten worse as a result of this. I think there was just a distribution shift, and now it makes sense to judge the new distribution. Separately, it matters who is in a position to shape the culture and trajectory of a community. I think there is a track record for the last few safety camps since Remmelt went off the deep end, and it is negative (not purely so, and not with great confidence, I am just trying to explain why I don't think there is a historical track record thing that screens off the personalities of the people involved).

It is, we’ve been limiting ourselves to readings from the sequence highlights. I’ll ask around to see if other organizers would like to broaden our horizons.

I mean, one of them’s math built bombs and computers & directly influenced pretty much every part of applied math today, and the other one’s math built math. Not saying he wasn’t smart, but no question are bombs & computers more flashy. 

1Embee
Yes. Grothendieck is undoubtedly less innovative and curious all across the board.  But I should have mentioned they are not of the same generation. vN helps build the atom bomb while G grows up in a concentration camp. vN went along a scientific golden age. I'd argue it was probably harder to have the same impact on Science in the 1960s.  I also model G as having disdain for applying mathematical ideas to "impure" subjects. Maybe because of the Manhattan project itself as well as the escalation of the Cold War. This would be consistent with a whole school of french mathematicians deifying pure math, N. Bourbaki in general, and being generally skeptical of the potential of pure math on the improvement of society, Roger Godement being the stereotype. My point was that Grothendieck's mind is interesting to dissect for someone interested in a general theory of intelligence and AI alignment (and that the von Neumann metaphor becomes kinda tiring)
2Linda Linsefors
It looks related, but these are not the plots I remember from the talk. 

@abramdemski I think I'm the biggest agree vote for alexander (without me alexander would have -2 agree), and I do see this because I follow both of you on my subscribe tab. 

I basically endorse Alexander's elaboration. 

On the "prep for the model that is coming tomorrow not the model of today" front, I will say that LLMs are not always going to be as dumb as they are today. Even if you can't get them to understand or help with your work now, their rate of learning still makes them in some sense your most promising mentee, and that means trying to ... (read more)

6abramdemski
Right, I strongly agree with this part.  I disagree in the sense that they're no mentee of mine, ie, me trying to get today's models to understand me doesn't directly help tomorrow's models to understand. (With the exception of the limited forms of feedback in the interface, like thumbs up/down, the impact of which I'm unsure of so it doesn't feel like something I should deliberately spend a lot of time on.) I also disagree in the sense that engaging with LLMs right now seems liable to produce a lot less fruits downstream, even as measured by "content that can usefully prompt an LLM later". IE, if mentees are viewed as machines that convert time-spent-dialoging-with-me to text that is useful later, I don't think LLMs are currently my most promising mentees. So although I strongly agree with continuing to occasionally poke at LLMs to prep for the models that are coming soon & notice when things get better, to the extent that "most promising mentee" is supposed to imply that significant chunks of my time could be usefully spent with LLMs in the present, I disagree based on my (fairly extensive) experience.  Barring special relationships with frontier labs, this sounds functionally equivalent to trying to get my work out there for humans to understand, for now at least.  I did talk to Anthropic last year about the possibility of me providing detailed feedback on Claude's responses (wrt my research questions), but it didn't end up happening. The big problems I identified seemed to be things they thought would definitely get addressed in another way, so there wasn't a mutually agreed-on value proposition (I didn't understand what they hoped to gain, & they didn't endorse the sorts of things I hoped to train). I got busy and moved on to other things. I feel like this is speaking from a model I don't understand. Are videos so bad? Video transcriptions are already a thing, and future models should be better at watching video and getting info from it. Are personal note

in some sense that’s just hiring you for any other job, and of course if an AGI lab wants you, you end up with greater negotiating leverage at your old place, and could get a raise (depending on how tight capital constraints are, which, to be clear, in AI alignment are tight).

Over the past few days I've been doing a lit review of the different types of attention heads people have found and/or the metrics one can use to detect the presence of those types of heads. 

Here is a rough list from my notes, sorry for the poor formatting, but I did say its rough!

... (read more)

And yes, I do think that interp work today should mostly focus on image nets for the same reasons we focus on image nets. The field’s current focus on LLMs is a mistake

A note that word on the street in mech-interp land is that often you get more signal & a greater number of techniques work on bigger & smarter language models over smaller & dumber possibly-not-language-models. Presumably due to smarter & complex models having more structured representations.

Do you have some concrete example of a technique for which this applies?

2Daniel Tan
I think interp 'works best' within a capability range, with both an upper and lower bound. (Note; this is a personal take that does not necessarily reflect the consensus in the field)    Below a certain capability threshold, it's difficult to interpret models, because those models are so primitive as to not really be able to think like humans. Therefore your usual intuitions about how models work break down, and also it's not clear if the insight you get from interpreting the model will generalise to larger models. Rough vibe; this means anything less capable than GPT2 With high capabilities, things also get more difficult. Both for mundane reasons (it takes more time and compute to get results, you need better infra to run larger models, SAEs need to get proportionately larger etc) as well as fundamental ones (e.g. the number of almost-orthogonal directions in N-dimensional space is exponential in N. So wider models can learn exponentially more features, and these features may be increasingly complex / fine-grained.) 

Fwiw, this is not at all obvious to me, and I would weakly bet that larger models are harder to interpret (even beyond there just being more capabilities to study)

8Nathan Helm-Burger
Yeah, I think this is a relevant point. Maybe for John and David's project the relevant point would be to try their ideas on absurdly oversized image models. Sometimes scale just makes things less muddled. Might run into funding limitations. I wish there was more sources of large scale compute available to research like this.

Can you show how a repeated version of this game results in overall better deals for the company? I agree this can happen, but I disagree for this particular circumstance.

2Darmani
In the pie example, the obvious answer is that giving the other person only 10% of the pie prevents them from gaining the security to walk away next time I present the same offer.
2Viliam
I admit I don't have a good answer for that. (I suspect there may be something important in real life that is missing from this model.)

Then the company is just being stupid, and the previous definition of exploitation doesn't apply. The company is imposing large costs for a large cost to itself. If the company does refuse the deal, its likely because it doesn't have the right kinds of internal communication channels to do negotiations like this, and so this is indeed a kind of stupidity. 

Why the distinction between exploitation and stupidity? Well they require different solutions. Maybe we solve exploitation (if indeed it is a problem) via collective action outside of the company. Bu... (read more)

6AnthonyC
Maybe they are, but I think the word "just" assumes that not being stupid is much easier than it actually is. Often the company is stupid without any individual employees/managers/executives being stupid or being empowered to fix the stupidity, in a context where no one has the convening power to bring together a sufficient set of stakeholders in some larger system to fix the problem without that costing much more than it is worth. Some company stupidity comes from individual executives and managers not being capable (because they're human) of absorbing all information about what's going on in different branches of the company and finding ways to make positive-sum trades that seem obvious to outsiders (this is especially common in large conglomerates). I encounter this all the time as a consultant, and the amount of inertia that needs to be overcome to improve it can be huge. Some comes from having to comply with all kinds of stupid and outdated and confusing laws (e.g. "The meeting is required because this is how the tax code is written because that's how they did it before email and before we moved the factory away from the head offices, and good luck getting the government to change that), sometimes while also trying to be even-handed to employees living in different jurisdictions with different laws (e.g. "Oh, well, the meeting is mandatory in city A and we like to have a unified policy about meetings across the company, but we're not allowed to provide or reimburse for the tuxedos in country B, and state C has a law that if we raised country B's wages to pay for the tuxedo we'd have to do it for everyone, and we can't afford that").
6Viliam
Sometimes what seems like stupidity locally, can be a part of a greater strategy. Other examples: Newcomb's paradox, strategic ignorance. In the example of a magical pie, if we agree to share the pie fairly, I can get 0.5 pie. But if I insist on getting 90% of the pie, and based on my previous experience with exploiting starving people I can predict that there is let's say a 70% chance of you accepting the unfair deal, then I can get 0.63 pie on average. Getting 0.63 pie instead of 0.5 pie seems like a smart thing to do for an economical agent. It is not a problem to lose some value, if more value is gained in turn. This is true even if things work probabilistically. Let's say that today was one of those 30% unlucky days when a starving stranger refused my offer to split the pie 90:10. From the short-term perspective, I have gambled and lost a half of the pie; stupid me! From the long-term perspective, I am still getting more pies than if I kept offering 50:50 deals instead. In companies, it is not unusual that some kind of internal inflexibility imposes "stupid" losses in short term, but generates profits in long term as everyone accepts the fact that the company is "stupid" and inflexible, and gives up trying to negotiate a better deal. (It is truly stupid only if it also generates losses in long term. Which is difficult to estimate, and I have seen many companies doing seemingly stupid things and staying profitable in long term, so I became skeptical of my feelings when they tell me that something is obviously stupid.)

If conversations are heavy tailed then we should in fact expect people to have singular & likely memorable high-value conversations.

1osten
True, but nitpicking about the memorability: The long-term value may not be in the short-term value of the conversation itself. It may be in the introduction to someone by someone you briefly got to know in an itself low-value conversion, by the email for a job getting forwarded to you etc. You wouldn't necessarily say the conversation was memorable, but the value likely wouldn't have been realized without it.

otoh I also don't think cutting off contact with anyone "impure", or refusing to read stuff you disapprove of, is either practical or necessary. we can engage with people and things without being mechanically "nudged" by them.

I think the reason not to do this is because of peer pressure. Ideally you should have the bad pressures from your peers cancel out, and in order to accomplish this you need your peers to be somewhat decorrelated from each other, and you can't really do this if all your peers and everyone you listen to is in the same social group.

there is no neurotype or culture that is immune to peer pressure

Seems like the sort of thing that would correlate pretty robustly to big-5 agreeableness, and in that sense there are neurotypes immune to peer pressure.

Edit: One may also suspect a combination of agreeableness and non-openness

Those invited to the foresight workshop (also the 2023 one) are probably a good start, as well as foresight’s 2023 and 2024 lectures on the subject.

I will take Zvi's takeaways from his experience in this round of SFF grants as significant outside-view evidence for my inside view of the field.

I think you are possibly better/optimizing more than most others at selecting conferences & events you actually want to do. Even with work, I think many get value out of having those spontaneous conversations because it often shifts what they're going to do--the number one spontaneous conversation is "what are you working on" or "what have you done so far", which forces you to re-explain what you're doing & the reasons for doing it to a skeptical & ignorant audience. My understanding is you and David already do this very often with each other.

4johnswentworth
I'm very curious if others also find this to be the biggest value-contributor amongst spontaneous conversations. (Also, more generally, I'm curious what kinds of spontaneous conversations people are getting so much value out of.)

I think its reasonable for the conversion to be at the original author's discretion rather than an automatic process.

Whether or not it would happen by default, this would be the single most useful LW feature for me. I'm often really unsure whether a post will get enough attention to be worth making it a longform, and sometimes even post shortforms like "comment if you want this to be a longform".

Back in May, when the Crowdstrike bug happened, people were posting wild takes on Twitter and in my signal groupchats about how Crowdstrike is only used everywhere because the government regulators subject you to copious extra red tape if you try to switch to something else.

Here’s the original claim:

Microsoft blamed a 2009 antitrust agreement with the European Union that they said forced them to sustain low-level kernel access to third-party developers.[286][287][288] The document does not explicitly state that Microsoft has to provide kernel-level ac

... (read more)
2lc
Well that's at least a completely different kind of regulatory failure than the one that was proposed on Twitter. But this is probably motivated reasoning on Microsoft's part. Kernel access is only necessary for IDS because of Microsoft's design choices. If Microsoft wanted, they could also have exported a user API for IDS services, which is a project they are working on now. MacOS already has this! And Microsoft would never ever have done as good a job on their own if they hadn't faced competition from other companies, which is why everyone uses CrowdStrike in the first place.
Answer by Garrett Baker*400

I believed “bear spray” was a metaphor for a gun. Eg if you were posting online about camping and concerned about the algorithm disliking your use of the word gun, were going into a state park which has guns banned, or didn’t want to mention “gun” for some other reason, then you’d say “bear spray”, since bear spray is such an absurd & silly concept that people will certainly understand what you really mean.

Turns out, bear spray is real. Its pepper spray on steroids, and is actually more effective than a gun, since its easier to aim and is optimized to ... (read more)

jimmy296

So far as I can tell, the common line that bear spray is more effective than firearms is based on an atrociously bad reading of the (limited) science, which is disavowed by the author of the studies. In short, successfully spraying a bear is more effective at driving off curious bears than simply having a firearm is are at stopping charging bears, but when you're comparing apples to apples then firearms are much more effective.

Here's a pretty good overview: https://www.outsideonline.com/2401248/does-bear-spray-work.  I haven't put a ton of work into v... (read more)

Remember: Bear spray does not work like bug spray!

[Bug report]: The Popular Comments section's comment preview ignores spoiler tags

As seen on Windows/Chrome

Answer by Garrett Baker1911

Film: The Martian

Rationality Tie-in: Virtue of scholarship is thread throughout, but Watney is generally an intelligent person tacking a seemingly impossible to solve problem.

A Boy and His Dog -- a weird one, but good for talking through & a heavy inspiration for Fallout

I have found that they mirror you. If you talk to them like a real person, they will act like a real person. Call them (at least Claude) out on their corporate-speak and cheesy stereotypes in the same way you would a person scared to say what they really think.

Load More