All of Garrett Baker's Comments + Replies

That is the wrong question to ask. By their nature the result of experiments is unknown. The bar is whether or not in expectation and on the margin do they provide positive externalities, and the answer is clearly yes.

Note the error bars in the original

After that I was writing shorter posts but without long context the things I write are very counterintuitive. So they got ruined)

This sounds like a rationalization. It seems much more likely the ideas just aren't that high quality if you need a whole hour for a single argument that couldn't possibly be broken up into smaller pieces that don't suck.

Edit: Since if the long post is disliked, you can say "well they just didn't read it", and if the short post is disliked you can say "well it just sucks because its small". Meanwhile, it should in fact be pret... (read more)

usually if you randomly get a downvote early instead of an upvote, so your post has “-1” karma now, then no one else will open or read it

I will say that I often do read -1 downvoted posts, I will also say that much of the time it is deserved, despite how noisy a signal it may be.

Some of my articles take 40 minutes to read, so it can be anything, downvotes give me zero information and just demotivate more and more.

I think you should try writing shorter posts. Both for your sake (so you get more targeted information), and for the readers' sake.

https://www.science.org/content/blog-post/alkanes-mars there are alkanes -- big organic molecules -- on Mars. these can be produced by abiotic processes, but usually that makes shorter chains than these. so....life? We Shall See.

Very exciting! I think the biggest "loophole" here is probably that they used a novel technique for detection, maybe if we used that technique more we would have to update the view that such big molecules are so unlikely to be produced non-biologically.

I'm a bit skeptical, there's a reasonable amount of passed-down wisdom I've heard claiming (I think justifiably) that

1. If you write messy code, and say "I'll clean it later" you probably won't. So insofar as you eventually want to discover something others build upon, you should write it clean from the start.

2. Clean code leads to easier extensibility, which seems pretty important eg if you want to try a bunch of different small variations on the same experiment.

3. Clean code decreases the number of bugs and the time spent debugging. This seems especially

Ok first, when naming things I think you should do everything you can to not use double-negatives. So you should say "gym average" or "no gym average". Its shorter, and much less confusing.

Second, I'm still confused. Translating what you said, we'd have "no gym removed average" -> "gym average" (since you remove everyone who doesn't go to the gym meaning the only people remaining go to the gym), and "gym removed average" -> "no gym average" (since we're removing everyone who goes to the gym meaning the only remaining people don't go to the gym).

There... (read more)

Note: You can verify this is the case by filtering for male respondents with male partners and female respondents with female partners in the survey data

I think the math works out to be that the variation is much more extreme when you get to much more extreme probabilities. Going from 4% to 8% is 2x profits, but going from 50% to 58% is only 1.16x profits.

This seems likely to depend on your preferred style of research, so what is your preferred style of research?

And then if we say the bottleneck to meritocracy is mostly c rather than a or b, then in fact it seems like our society is absolutely obsessed with making our institutions highly accessible to as broad a pool of talent as possible. There are people who make a whole career out of just advocating for equality.

I work at GDM so obviously take that into account here, but in my internal conversations about external benchmarks we take cheating very seriously -- we don't want eval data to leak into training data, and have multiple lines of defense to keep that from happening.

What do you mean by "we"? Do you work on the pretraining team, talk directly with the pretraining team, are just aware of the methods the pretraining team uses, or some other thing?

More to the point, I haven't seen people try to scale those things either. The closest might be something like TripleByte? Or headhunting companies? Certainly when I think of a typical (or 95th-99th percentile) "person who says they care a lot about meritocracy" I'm not imagining a recruiter, or someone in charge of such a firm. Are you?

I think much of venture capital is trying to scale this thing, and as you said they don't use the framework you use. The philosophy there is much more oriented towards making sure nobody falls beneath the cracks. Provide... (read more)

Also this conclusion is highly dependent on you, who has thought about this topic for all of 10 minutes, out-thinking the hypothetical people who are actually serious about meritocracy. For example perhaps they do more one-on-one talent scouting or funding, which is indeed very very common and seems to be much more in-demand than psychometric evaluations.

Given that ~ no one really does this, I conclude that very few people are serious about moving towards a meritocracy.

The field you should look at I think is Industrial and Organizational Psychology, as well as the classic Item Response Theory.

I suspect the vast majority of that sort of name-calling is much more politically motivated than based on not seeing the right slogans. For example if you go to Pause AI's website the first thing you see is a big, bold

and AI pause advocates are constantly arguing "no, we don't actually believe that" to the people who call them "luddites", but I have never actually seen anyone change their mind based on such an argument.

I don't think Pause AI's current bottleneck is people being pro AI in general not wanting to join (but of course I could be wrong). Most people are just against AI, and Pause AI's current strategy is to make them care enough about the issue to use their feet, while also telling them "its much much worse than you would've imagined bro".

That's a whole seven words!, most of which are a whole three syllables! There is no way a motto like that catches on.

An effect I noticed: Going through Aella's correlation matrix (with poorly labeled columns sadly), a feature which strongly correlates with the length of a relationship is codependency. Plotting question 20. "The long-term routines and structure of my life are intertwined with my partner's" (li0toxk) assuming that's what "codependency" refers to

The shaded region is a 95% posterior estimate for the mean of the distribution conditioned on the time-range (every 2 years) and cis-male respondents, with prior $N(0,0.5)$.

Note also that codependency and sex sat... (read more)

This seems right as a criticism, but this seems better placed on the EA forum. I can't remember the last time I heard anyone talking about ITN on LessWrong. There are many considerations ITN leaves out, which should be unsurprising given how simplified it is.

So, the recipe for making a broken science you can't trust is

1. The public cares a lot about answers to questions that fall within the science's domain.
2. The science currently has no good attack angles on those questions.

To return to LessWrong's favorite topic, this doesn't bode well for alignment.

I understand the argument, I think I buy a limited version of it (and also want to acknowledge that it is very clever and I do like it), but I also don't think this can explain the magnitude of the difference between the different fields. If we go back and ask "what was physics' original goal?" we end up with "to explain how the heavens move, and the path that objects travel", and this has basically been solved. Physicists didn't substitute this for something easier. The next big problem was to explain heat & electricity, and that was solved. Then the ... (read more)

I will also note that Aella's relationships data is public, and has the following questions:

1. Your age? (rkkox57)
2. Which category fits you best? (4790ydl)
3. In a world where your partner was fully aware and deeply okay with it, how much would you be interested in having sexual/romantic experiences with people besides your partner? (ao3mcdk)
4. In a world where you were fully aware and deeply okay with it, how much would *your partner* be interested in having sexual/romantic experiences with people besides you? (wcq3vrx)
5. To get a little more specifi

Historically attempts to curtail this right lead to really really dark places. Part of living in a society with rights and laws is that people will do bad things the legal system has no ability to prevent. And on net, that’s a good thing. See also.

An obvious answer you missed: Lacking a prenup, courts often rule in favor of the woman over the man in the case of a contested divorce.

I didn't say anything about temperature prediction, and I'd also like to see any other method (intuition based or otherwise) do better than the current best mathematical models here. It seems unlikely to me that the trends in that graph will continue arbitrarily far.

Thanks for the pointer to that paper, the abstract makes me think there's a sort of slow-acting self-reinforcing feedback loop between predictive error minimisation via improving modelling and via improving the economy itself.

Yeah, that was my claim.

Even more importantly, we only measured the AIs at software tasks and don't know what the trend is for other domains like math or law, it could be wildly different.

You probably mention this somewhere, but I'll ask here, are you currently researching whether these results hold for those other domains? I'm personally more interested about math than law.

It does not seem like this writer is aware of the Von Neumann–Morgenstern utility theorem. There are criticisms one can level against utility as a concept, but the central question ends up being which of those axioms do you disagree with and why? For example, Garabrant's Geometric Rationality is a great counter if you're looking for one.

Edit: I notice that all of your previous posts have been of this same format, and they all consistently receive negative karma. You should probably reconsider what you post to this forum.

The weather, or the behavior of any economy larger than village size, for example -- systems so chaotically interdependent that exact prediction is effectively impossible (not just in fact but in principle).

Flagging that those two examples seem false. The weather is chaotic, yes, and there's a sense in which the economy is anti-inductive, but modeling methods are advancing, and will likely find more loop-holes in chaos theory.

For example, in thermodynamics, temperature is non-chaotic while the precise kinetic energies and locations of all particles are.... (read more)

My model is that early on physics had very impressive & novel math, which attracted people who like math, who did more math largely with the constraint the math had to be trying to model something in the real world, which produced more impressive & novel math, which attracted more people who like math, etc etc, and this is the origin of the equilibrium.

Note a similar argument can be made for economics, though the nice math came much later on, and obviously was much less impactful than literally inventing calculus.

Happy to take your word on these things if the wikipedia article is unrepresentative!

In contrast, physicists were not committed to discovering the periodic table, fields or quantum wave functions. Many of the great successes of physics are answers to question no one would think to ask just decades before they were discovered. The hard sciences were formed when frontiers of highly tractable and promising theorizing opened up.

This seems a crazy comparison to make^[1]. These seem like methodological constraints. Are there any actual predictions past physics was trying to make which we still can't make and don't even care about? None that I ... (read more)

Apropos of the comments below this post, many seem to be assuming humans can complete tasks which require arbitrarily many years. This doesn't seem the case to me. People often peak intellectually in their 20's, and sometimes get dementia late in life. Others just get dis-interested in their previous goals through a mid-life crisis or ADHD.

I don't think this has much an impact on the conclusions reached in the comments (which is why I'm not putting this under the post), but this assumption does seem wrong in most cases (and I'd be interested in cases where people think its right!)

And we didn't filter them in any way.

This seems contrary to what that page claims

Here, we present highly misaligned samples (misalignment >= 90) from GPT-4o models finetuned to write insecure code.

And indeed all the samples seem misaligned, which seems unlikely given the misaligned answer rate for other questions in your paper.

In my experience playing a lot with LLMs, “Nova” is a reasonably common name they give themselves if you ask, and sometimes they will spontaneously decide they are sentient, but that is the extent to which my own experiences are consistent with the story. I can imagine though that since the time I was playing with these things a lot (about 6 months ago) much has changed.

As a datapoint, I really liked this post. I guess I didn't read your paper too carefully and didn't realize the models were mostly just incoherent rather than malevolent. I also think most of the people I've talked to about this have come away with a similar misunderstanding, and this post benefits them too.

Yeah, I don't know the answer here, but I will also say that one flaw of the brier score is that its not even clear that these sorts of differences will be even all that meaningful. Like, what you actually want to know is, how much more information does one group here give over the other groups here, and how much credence should we assign to each of the groups (acting as if they were each hypotheses in a Bayes update) given their predictions on the data we have. And for that, you can just run the bayes update.

The brier score was chosen for forecasters as f... (read more)

The difference between our results and OpenAI’s might be due to OpenAI evaluating with a more powerful internal scaffold, using more test-time compute, or because those results were run on a different subset of FrontierMath (the 180 problems in frontiermath-2024-11-26 vs the 290 problems in frontiermath-2025-02-28-private).

That definitely sounds like OpenAI training on (or perhaps constructing a scaffold around) the part of the benchmark Epoch shared with them.

So part of it is slowly becoming a journal, and the felt social norms around posts are morphing to reflect that.

In some ways the equilibrium here is worse, journals have page limits.

Yes, by default there is always a reading group, we just forgot to post about it this week.

I think the biggest problem with how posts are presented is it doesn’t make the author embarrassed to make their post needlessly long, and doesn’t signal “we want you to make this shorter”. Shortforms do this, so you get very info dense posts, but actual posts kinda signal the opposite. If its so short, why not just make it a shortform, and if it shouldn’t be a shortform, surely you can add more to it. After all, nobody makes half-page lesswrong posts anymore.

So it's certainly not a claim that could be verified empirically by looking at any individual humans because there aren't yet any millenarians or megaannumarians.

If its not a conclusion which could be disproven empirically, then I don’t know how you came to it.

(I wrote my quick take quickly and therefore very elliptically, and therefore it would require extra charity / work on the reader's part (like, more time spent asking "huh? this makes no sense? ok what could he have meant, which would make this statement true?").)

I mean, I did ask myself about... (read more)

Given more information about someone, your capacity for having {commune, love, compassion, kindness, cooperation} for/with them increases more than your capacity for {hatred, adversariality} towards them increases.

If this were true, I’d expect much lower divorce rates. After all, who do you have the most information about other than your wife/husband, and many of these divorces are un-amicable, though I wasn’t quickly able to get particular numbers. [EDIT:] Though in either case, this indeed indicates a much decreasing level of love over long periods of... (read more)

One answer is to promote it more, but December is generally kind of busy.

An alternative is to do it in the summer!

Is that significant? Someone at Metaculus or Fatebook should maybe publish what “normal” calibration looks like, though then again the kind of person who uses Metaculus isn’t exactly normal. I’m tempted to pay for a Mechanical Turk survey or something for a better control

There is no “normal” calibration, its all relative to the questions you have, eg you get like 0.2 if you answer 50% to everything, which can be pretty horrible if the questions were all “will the sun rise tomorrow?” but pretty awesome if the questions were all something like “Will there be a plague in 2020?” (asked in 2016) such that everyone puts 1% or something

Going to CFAR looks like it increases comfort and confidence with Goal-factoring. If we accept that goal-factoring is a useful thing, then by self-report CFAR helps.

what? We have $P\left(\text{comfortable | cfar}\right)=11.2\%<36.8\%=P\left(\text{comfortable | no cfar}\right)$, and similarly for confident.

Simplest story is that when they play roles, the simulated entity being role-played actually has experiences. Philosophically one can say something like "The only difference between a role, and an actual identity, is whether there's another role underneath. Identity is simply the innermost mask." In which case they'll talk about their feelings if the situation calls for it.

Often AIs play many roles at the same time, and likely a whole continua over different people since they’re trying to model a probability distribution over whose talking. If this is true, makes you wonder about the scale here.

Topological data analysis comes closest, and there are some people who try to use it for ML, eg.

Though my understanding is this is used in interp, not so much because people necessarily expect deep connections to homology, but because its just another way to look for structure in your data.

TDA itself is also a relatively shallow tool too.

Sometimes I think about the potential engineering applications of quantum immortality in a mature civilisation for fun. Controlled, synchronised civilisation-wide suicide seems like a neat way to transform many engineering problems into measurement problems.

Such thought experiments also serve as a solution of sorts to the fermi paradox, and as a rationalization of the sci-fi trope of sufficiently advanced civilizations “ascending”.