All of Garrett Baker's Comments + Replies

~$5 for a cup of coffee — that’s about an order of magnitude cheaper.

Are you buying your coffee from a cafe every day or something? You can buy a pack of nice grounds for like $13, and that lasts more than a month (126 Tbsp/pack / (3 Tbsp/day) = 42 days/pack), totaling 30¢/day. Half the cost of a caffeine pill. And that’s if you don’t buy bulk.

Yeah, I figured something like that would be going on if I was wrong, thanks!

The wikipedia side-effect page says that the rate of seizures is between 0.01-0.1%, for comparison about 0.68% of the population has epilepsy, so I'm skeptical this ought to be such a concern. Am I reading these numbers incorrectly?

I can definitely believe the anxiety bit. It is a stimulant, and anxiety & depression are very very correlated.

How did Einstein and Heisenberg go so wrong?

I don't know about Heisenberg, but a common answer for Einstein is the Institute for Advanced Study at Princeton, which took the best minds of the day, and freed them from practical concerns about justifying their research interests or talking to outside researchers, apparently causing them to be increasingly less productive & out of touch from the rest of their respective fields.

From Surely You're Joking Mr. Feynman

When I was at Princeton in the 1940s I could see what happened to those great minds at th

I had a class in college about the history of the atomic bomb. Our text book was Rhodes, and most of the stuff we learned about was the different competing theories of atoms, the experimental methods & math used to distinguish between them, math behind atomic chain reactions, and the scientists who did those things. It was great! Our tests were basically physics tests.

I’d like more history like that, and I’d like marginal movements in that direction in math class. For example, something like Radical Real Analysis but for everything.

Obviously schools wo... (read more)

I agree, and we do see some american companies doing the same thing.

Re: open sourcing. My guess why they open source more is for verification purposes. Chinese labs have an earned reputation for scams. So a lab that announces a closed source chat site, to investors, could very well be a claude or openai or llama or gemini wrapper. However, a lab that releases the weights of their model, and “shows their work” by giving a detailed writeup of how they managed to train the model while staying under their reported costs is significantly more likely to be legitimate.

This is true, but I read amitlevy49's comment as having an implicit "and therefore anyone who wants that kind of natural drive should take bupropion". I probably should've given more information in my response.

FYI (cc @Gram_Stone) the 2023 course website has (poor quality edit:nevermind I was accessing them wrong) video lectures.

Edit 2: For future (or present) folks, I've also downloaded local mp4s of the slideshow versions of the videos here, and can share privately with those who dm, in case you want them too or the site goes down.

Not really, the hypothesis is that John has depression, and of all the antidepressants, bupropion is the best (if it works for you).

A few points:

1. All of these things sound obviously awesome and fun to do
2. Cooking is imo another class of wizardry
3. Have you tried the antidepressant bupropion? Ask David for the pitch

Because of 1, I think the difficulty you'll find building (or finding) this community is not whether or not what you're saying "resonates" with people, but whether they have the expertise, energy, or agency to put in their share of the work.

The boring hypothesis here is the model was actually trained on the id-location-from-picture task, and wasn’t trained on the id-object-location-in-pixel-art task, and pixel art is surprisingly nontrivial for models to wrap their heads around when they’re still trying to understand real world pictures.

My understanding is its not approximately all, it is literally all the images in geoguessr.

Apparently there already exists a CUDA-alternative for non-Nvidia hardware. The open source project ZLUDA. As far as I can tell its less performant than CUDA, and it has the same challenges as firefox does when competing with chromium based browsers, which will only get worse as it gets more popular. But its something to track at least.

I recommend you read at least the first chapter of Getting Things Done, and do the corresponding exercises. In particular, this one, which he uses to provide evidence his model of productivity is correct

I suggest that you write down the project or situation that is most on your mind at this moment. What most bugs you, distracts you, or interests you, or in some other way consumes a large part of your conscious attention? It may be a project or problem that is really “in your face,” something you are being pressed to handle, or a situation you feel you must

This argument seems only convincing if you don’t have those destructive values. One man’s destructive values is another’s low-hanging fruit, and those who see low hanging fruit everywhere won’t give up on the fruit just because others may pick it.

Since bad people won’t heed your warning it doesn’t seem in good people’s interests to heed it either.

An analogy is one can make the same argument wrt rationality itself. Its dual use! Someone with bad values can use rationality to do a lot of harm! Does that mean good people shouldn’t use rationality? No!

Yet the universe runs on strikingly simple math (relativity, quantum mechanics); such elegance is exactly what an efficient simulation would use. Physics is unreasonably effective, reducing the computational cost of the simulation. This cuts against the last point.

This does not seem so consistent, and if the primary piece of evidence for me against such simulation arguments. I would imagine simulations targeting, eg, a particular purpose would have their physics tailored to that purpose much more than ours seems to (for any purpose, given the vast comp... (read more)

Yeah I think I agree with all of this, so I do think most of this was miscommunication/interpretation.

the combativeness did also make me a little sad.

Sorry about that, I think my comments often come across as more negative than I intend, I try to remember to take a step back afterwards and rewrite things to be nicer, but I often forget or don't realize in the moment its necessary.

If LLMs can be sad, that sadness would probably be realized through the firing of “sadness” features: identifiable patterns in its inference that preferentially fire when sad stuff is under discussion. In fact, it’s hard to say what else would count as an LLM experiencing sadness, since the only cognition that LLMs perform is through huge numbers of matrix operations, and certain outcomes within those operations reliably adjust the emotional content of the response.

Best I can tell, your argument here is “either there’s a direction in activation space re... (read more)

Your second option seems likely. Eg did you know community notes is open source? Given that information, are you going to even read the associated whitepaper or the issues page?

Even if you do, I think we can still confidently infer very few others reading this will (I know I’m not).

Smol r'\. (.*)\.'! y? Clear! Had big r'\. (.*)\.' b4. & abstract. ppl no get. Now: Smol. Clear.

Good. Think clearer. y? Smol => deep.

A possible longer term issue with this is when future generations of models are pre-trained, this style of code will be a significant fraction of their training data which will only grow over time, so just as its been hard to get models out of the "chatgpt-ese" due to simulators reasons, it may also be hard to get models out of this messy code basin, even before you do any code RL, once they realize their chat-models and they're "supposed to" talk like this.

I say issue, because it does seem worse to have a trend in the direction of AI code un-readability b... (read more)

In that case I think your response is a non sequitur, since clearly “really care” in this context means “determiners of what they end up doing in practice re influencing x-risk”.

Conjecture seems unusually good at sticking to reality across multiple domains.

I do not get this impression, why do you say this?

In this case prediction markets will be predictably over-optimistic, and expert consensus is very split.

There is a question of interest here though: why does pixel art work so well on humans despite literally nothing in real life being pixel art?

I’m reminded of Gwern’s comments on the difficulty of getting GANs to generate novel pixel art interpolations

Pixel-art anything is derivative of a photorealistic world. If you look at 8-bit art and standard block sizes like Mario in NES Mario, if you were not already intimately familiar with the distribution of human faces, and had to learn starting with a completely blank slate like a GAN would, how would you e

I mean this situation is grounded & formal enough you can just go and implement the relevant RL algorithm and see if its relevant for that computationally bounded agent, right?

It seems pretty likely SBF happened because everyone in EA was implicitly trusting everyone else in EA. If people were more suspicious of each other, that seems less likely to have been allowed to happen.

Don’t double update! I got that information from that same interview!

My vague understanding is this is kinda what capabilities progress ends up looking like in big labs. Lots of very small experiments playing around with various parameters people with a track-record of good heuristics in this space feel should be played around with. Then a slow scale up to bigger and bigger models and then you combine everything together & "push to main" on the next big model run.

I'd also guess that the bottleneck isn't so much on the number of people playing around with the parameters, but much more on good heuristics regarding which parameters to play around with.

It seems useful for those who disagreed to reflect on this LessWrong comment from ~3 months ago (around the time the Epoch/OpenAI scandal happened).

The strong version of this argument seems false (eg Habryka's comment), but I think the weak version is true. That is, energy put into "purposely and deliberately develop a technology Y that is fundamentally different than X that does the same role as X without harm Z but slightly less competitively." is inefficient compared to energy put into strategies (i), (ii), and (iii).

If it is encoding relevant info then this would be the definition of steganography

Note that "smartphones, computers and more electronics" are exempt. I'd guess this would include (or end up including) datacenters. The details of the exemption are here.

This hardly seems an argument against the one in the shortform, namely

Neither a physicalist nor a functionalist theory of consciousness can reasonably justify a number like this. Shrimp have 5 orders of magnitude fewer neurons than humans, so whether suffering is the result of a physical process or an information processing one, this implies that shrimp neurons do 4 orders of magnitude more of this process per second than human neurons. The authors get around this by refusing to stake themselves on any theory of consciousness.

If the original authors never thought of this that seems on them.

but most of the population will just succumb to the pressure. Okay Microsoft, if you insist that I use Edge, I will; if you insist that I use Bing, I will; if you insist that I have MSN as my starting web page, I will

Only about 5% of people use edge, with 66% chrome and 17% safari. Bing is similar, with 4% marketshare and Google having about 90%. I don’t know the number with MSN as their starting page (my parents had this), but I’d guess its also lower than you expect. I think you over-estimate the impact of nudge economics

That's an inference, presumably Adam believes that for object-level reasons, which could be supported by eg looking at the age at which physicists make major advancements^[1] and the size of those advancements.

Edit: But also this wouldn't show whether or not theoretical physics is actually in a rut, to someone who doesn't know what the field looks like now.

Claude says its a gray area when I ask, since this isn’t asking for the journalist to make a general change to the story or present Ben or the subject in a particular light.

This doesn’t seem to address the question, which was why do people believe there is a physics slow-down in the first place.

(you also may want to look into other ways of improving your conscientiousness if you're struggling with that. Things like todo systems, or daily planners, or simply regularly trying hard things)

It seems reasonable to mention that I know of many who have started doing "spells" like this, with a rationalized "oh I'm just hypnotizing myself, I don't actually believe in magic" framing who then start to go off the deep-end and start actually believing in magic.

That's not to say this happens in every case or even in most cases. Its also not to say that hypnotizing yourself can't be useful sometimes. But it is to say that if you find this tempting to do because you really like the idea of magic existing in real life, I suggest you re-read some parts of ... (read more)

I'm not sure what the type signature of $L_A$ is, or what it means to "not take into account $M$'s simulation"

I know you know about logical decision theory, and I know you know its not formalized, and I'm not going to be able to formalize it in a LessWrong comment, so I'm not sure what you want me to say here. Do you reject the idea of logical counterfactuals? Do you not see how they could be used here?

I think you've misunderstood me entirely. Usually in a decision problem, we assume the agent has a perfectly true world model, and we assume that it's in a

Can you give something specific? It seems like pretty much every statement has a footnote grounding the relevant high-level claim in low-level indicators, and in cases where that's not the case, those predictions often seem clear derivatives of precise claims in eg their compute forecast

I mean its not like they shy away from concrete predictions. Eg their first prediction is

We forecast that mid-2025 agents will score 85% on SWEBench-Verified.

Edit: oh wait nevermind their first prediction is actually

Specifically, we forecast that they score 65% on the OSWorld benchmark of basic computer tasks (compared to 38% for Operator and 70% for a typical skilled non-expert human).

The closing off of China after/during Tinamen square I don't think happened after a transition of power, though I could be mis-remembering. See also the one-child policy, which I also don't think happened during a power transition (allowed for 2 children in 2015, then removed all limits in 2021, while Xi came to power in 2012).

I agree the zero-covid policy change ended up being slow. I don't know why it was slow though, I know a popular narrative is that the regime didn't want to lose face, but one fact about China is the reason why many decisions are made... (read more)

Let $M$ be an agent which can be instantiated in a much simpler world and has different goals from our limited Bayesian agent $A$. We say $M$ is malign with respect to $A$ if $p\left(q|O\right)<p\left(q_{M,A}|O\right)$ where $q$ is the "real" world and $q_{M,A}$ is the world where $M$ has decided to simulate all of $A$'s observations for the purpose of trying to invade their prior.

Now what influences $p\left(q_{M,A}|O\right)$? Well $M$ will only simulate all of $A$'s observations if it expects this will give it some influence over $A$. Let $L_A$ be an unformalized logical counterfactual operation that $A$ could make.

Then $p(q^{}$... (read more)

no, I am not going to do what the evil super-simple-simulators want me to do because they will try to invade my prior iff (I would act like they have invaded my prior iff they invade my prior)

This seems a pretty big backpedal from "I expect this to start not happening right away."

My world model would have a loose model of myself in it, and this will change which worlds I'm more or less likely to be found in. For example, a logical decision theorist, trying to model omega, will have very low probability that omega has predicted it will two box.

Autarchies, including China, seem more likely to reconfigure their entire economic and social systems overnight than democracies like the US, so this seems false.