I did not particularly intend to do a book review per say, and I don't claim to be an expert on the topic. So completely fine with tagging this in some way as "non-expert" if you wish.
Not planning to change how I wrote my posts based on this feedback, as I have no interest in following some arbitrary standard of epistemic expertise for a fun little blog post that will be read by 10 people max.
I agree that it is in the text. If it wasn't clear, my message was trying to reverse engineer why I bounced off, which is more about my experience of reading than fully about the text.
I remember reading this post, and really disliking it.
Then today, as I was reflecting on things, I recalled that this existed, and went back to read it. And this time, my reaction was instead "yep, that's pointing to the mental move that I've lost and that I'm now trying to relearn".
Which is interesting. Because that means a year or two ago, up till now, I was the kind of people who would benefit from this post; yet I couldn't get the juice out of it. I think a big reason is that while the description of the play/fun mental move is good and clear, the desc...
Thanks for the comment!
We have indeed gotten the feedback by multiple people that this part didn't feel detailed enough (although we got this much more from very technical readers than from non-technical ones), and are working at improving the arguments.
Thanks for the comment!
We'll correct the typo in the next patch/bug fix.
As for the more direct adversarial tone of the prologue, it is an explicit choice (and is contrasted by the rest of the document). For the moment, we're waiting to get more feedback on the doc to see if it really turns people off or not.
Yep, I think you're correct.
Will correct in the next minor update. Thanks!
Thanks for the comment!
We'll consider this point for future releases, but personally, I would say that this kind of hedging also has a lot of downsides: it makes you sound far more uncertain and defensive than you really want to.
This document tries to be both grounded and to the point, and so we by default don't want to put ourselves in a defensive position when arguing things that we think make sense and are supported by the evidence.
Thanks for the comment!
We have gotten this feedback by a handful of people, so we want to reread the links and the whole literature about o1 and its evaluation to check whether we've indeed gotten the right point, or if we mischaracterized the situation.
We will probably change the phrasing (either to make our criticism clearer or to correct it) in the next minor update.
Good catch, I think we are indeed mixing the sizes here.
As you say, the point still stands, but we will change it in the next minor update to either compare the same size or make the difference in size explicit.
Thanks for the comment!
We want to check the maths, but if you're indeed correct we will update the numbers (and reasoning) in the next minor version.
I guess it depends on if you’re pivoting based on things that you’ve learned, versus grass-is-greener.
Yeah, I didn't mean "iterative thoughtful processes", I meant "compulsion that unfold at the level of days". If you arbitrarily change your job every couple of days/weeks, not based on new significant information but because you feel this other one is the one, this is bad.
So there is a vibe here that I maybe didn't convey well, about the time frame and the auto-generated part of the loops I'm pointing at: it happens often enough that your friends and famil...
Recently found a new link: Annual Reviews
It sounds like a place that centralizes many different review articles across a lot of disciplines. Only checked a few for the moment, but definitely sounds worth a try!
@Elizabeth suggested that I share here the quick tips I gave her for finding cool history and philosophy of science books, so let's do it.
Yeah, I agree with the general point (don't have strong opinion about chaos theory at the moment).
First, negative results are really, really important. Mostly because they let you not lose your time trying to do something impossible, and sometimes they actually point you toward an answer. In general, conservation laws in physics have this role. And knowing what is undecidable is really important in formal methods, where the trick is generally to simplify what you want or the expressive power of your programs in order to sidestep it.
Then, they are ind...
Is there any empirical question the phlogiston theorists got right that compositional chemistry did not? AFAIK, no, but it's a real question and I'd like to know if I'm wrong here.
Although I haven't digged into the historical literature that much, I think there are two main candidates here: explaining the behavior of metals, and potential chemical energy.
On explaining the behavior of metal, this is Chang (Is Water H2O? p.43)
...Phlogistonists explained the common properties of metals by saying that all metals were rich in phlogiston; this explanation was lost
Apparently people want some clarification on what I mean by anti-library. It's a Nassim Taleb term which refers to books you own but haven't read, whose main value is to remind you and keep in mind what you don't know and where to find it if you want to expand that knowledge.
If the point you're trying to make is: "the way we go from preparadigmatic to paradigmatic is by solving some hard problems, not by communicating initial frames and idea", I think this points to an important point indeed.
Still, two caveats:
Curated. I've heard this book suggested a few times over the years, and feels like it's a sort of unofficial canon among people studying how preparadigmatic science happens. This review finally compelled me to get the book.
There's something quite funny in that I discovered this book in January 2022, during the couple of days I spent at Lightcone offices. It was in someone's office, and I was curious about it. Now, we're back full circle. ^^
...I do think this review would be a lot better if it actually distilled the messy-bits-that-you-need-to-experienti
It sounds cool, though also intuitively temperature seems like one of the easiest attributes to measure because literally everything is kind of a thermometer in the sense that everything equillibrates in temperature.
Can't guarantee that you would benefit from it, but this sentence makes me think you have a much cleaner and simplified idea of how one develops even simple measuring device than what the history shows (especially when you don't have any good theory of temperature or thermodynamics).
So would say you might benefit from reading it. ;)
If you enjoyed Inventing Temperature, Is Water H2O? is pretty much the same genre from the same author.
Yeah, I am a big fan of Is Water H2O? (and the other Chang books). It's just that I find Is Water H2O? both less accessible (bit more focused on theory) and more controversial (notably in its treatement of phlogiston, which I agree with, but most people including here have only heard off phlogiston from fake histories written by scientists embellishing the histories of their fields (and Lavoisierian propaganda of course)). So that's why I find Inventing T...
It's rare that books describe such processes well, I suspect partly because it's so wildly harder to generate scientific ideas than to understand them, that they tend to strike people as almost blindingly obvious in retrospect.
Completely agreed!
I think this is also what makes great history of science so hard: you need to unlearn most of the modern insights and intuitions that didn't exist at the time, and see as close as possible to what the historical actors saw.
This makes me think of a great quote from World of Flows, a history of hydrodynamics:
...There is,
Thanks for the links!
But yeah, I'm more interested in detailed descriptions of how things actually work, rather than models of ideal governance.
Thanks!
After checking them, it feels like most of your links are focused on an economic lens to politics and governance, or at least an economic bent. Does that seem correct?
...And of course just reading the rule books for the various governments or parts of the government -- for the US that would be looking at the Constitution and the rules governing internal processes for both the House and Senate. Parlimentary systems will have similar rules of governance.
Looking at the organizational charts likely also help -- what are the committee structures and h
The true deep philosophical answer was... I wanted to separate cakes from bread (in french we have patisserie and boulangerie), but couldn't find any obvious one in english (seems like indeed, english-speaking countries use baking for both). So I adapted the "patisser" verb in french, hoping that I would get away with a neologism given that english is so fit for constructing them.
My bad. Thanks for the correction, edited the post.
Unfortunately all the positives of these books come paired with a critical flaw: Caro only manages to cover two people, and hasn’t even finished the second one!
In my view, Caro is actually less guilty of this than most biographers.
Fundamentally, this is because he cares much more about power, its sources, and its effects on the wielders, beneficiaries, and victims. So even though the throughline are the lives of Moses and Johnson, he spends a considerable amount of time on other topics which provide additional mechanistic models with which to understand po...
I do find thinking on paper (a bit more intentional than freewriting, but the same vibe) to be particularly helpful, I agree. Just like walks.
The reasons I don't find them enough is that:
Still, I find it's a good way to build emotional potential energy much slower, and to notice when you really need to have a full break/sabbaticl.
Oh, I like the neural annealing connection, I have read the post but didn't relate it to emotional potential energy, but it makes sense!
Hope you take some time to anneal away some of that potential energy soon. People consistently underestimate the negative ripples on the social web from being overstretched, as opposed to the obvious and tangible "but this thing right in front of me needs doing".
Thanks. That's the plan. ;)
No worries. ;)
...However, when it comes to more inchoate domains like research skill, such writing does very little to help the inexperienced researcher. It is more likely that they'd simply miss out on the point you are trying to tell them, for they haven't failed both by, say, being too trusting (a common phenomenon) and being too wary of 'trusting' (a somewhat rare phenomenon for someone who gets to the big leagues as a researcher). What would actually help is either concrete case studies, or a tight feedback loop that involves a researcher trying to do something, and p
...Just sharing some vibe I've got from your.. framing!
Minimalism ~ path ~ inside-focused ~ the signal/reward
Maximalist ~ destination ~ outside-focused ~ the worldThese two opposing aesthetics is a well-known confusing bit within agent foundation style research. The classical way to model an agent is to think as it is maximizing outside world variables. Conversely, we can think about minimization ~ inside-focused (reward hacking type error) as a drug addict accomplishing "nothing"
Feels there is also something to say with dopamine vs serotonine/hom
Interestingly, the Lean theorem prover is sometimes considered a bit of a mess type-theoretically. (an illustrative thread), but is perhaps the most popular theorem prover among mathematicians. I would say it's more on the "maximalist" side.
Didn't know this about Lean, but the fact that a maximalist option is most popular with mathematicians makes sense to me. As someone who worked both with mathematicians and formal methods researchers (much more meta-mathematicians), the latter are much closer to programmers, in the sense that they want to build th...
It is definitely one minimalist vs maximalist dimensions ^^.
Oh, I didn't see it actually mentioned your package. 😂
Units / dimensional analysis in physics is really a kind of type system. I was very big into using that for error checking when I used to do physics and engineering calculations professionally.
Definitely!
Dimensional analysis was the first place this analogy jumped to me when reading Fly By Night Physics, because it truly used dimensions not only to check results, but also to infer the general shape of the answer (which is also something you can do in type systems, for example a function with a generic type can only be populated by the i...
Thanks for the pointer!
Shapely values are very cool. Let me mention some cool facts:
They arise in (cooperative) game theory but also in ML when doing credit allocation a combined prediction from mixing predictions from different modules of a system.
One piece of evidence of their fundamentalness is that they arise naturally from the Hodge theory on the hypercube of a coalition game: https://arxiv.org/abs/1709.08318
Another interesting fact I learned from Davidad: Shapley values are not compositional: a group of actors can increase their total Shapley value by forming a single caba...
Oh, didn't know him!
Thanks for the links!
Thanks for the comment!
I agree with you that there are situations where the issue comes from a cultural norm rather than psychological problems. That's one reason for the last part of this post, where we point out to generally positive and productive norms that try to avoid these cultural problems and make it possible to discuss them. (One of the issue I see in my own life with cultural norms is that they are way harder to discuss when in addition psychological problems compound them and make them feel sore and emotional). But you might be right that it's ...
I think I agree with everything in your comment. Seems like there was less disagreement here than I initially thought. Moving on... :)
Oh, I definitely agree, this is a really good point. What I was highlighting was an epistemic issue (namely the confusion between ideal and necessary conditions) but there is also a different decision theoretic issue that you highlighted quite well.
It's completely possible that you're not powerful enough to work outside the ideal condition. But by doing the epistemic clarification, now we can consider the explicit decision of taking step to become more powerful and being better able to manage non-ideal conditions.
Good point! The difference is that the case explained in this post is one of the most sensible version of confusing the goal and the path, since there the path is actually a really good path. On the other version (like wanting to find a simple theory simply, the path is not even a good one!
In many ways, this post is frustrating to read. It isn't straigthforward, it needlessly insults people, and it mixes irrelevant details with the key ideas.
And yet, as with many of Eliezer's post, its key points are right.
What this post does is uncover the main epistemological mistakes made by almost everyone trying their hands at figuring out timelines. Among others, there is:
I was mostly thinking of the efficiency assumption underlying almost all the scenarios. Critch assumes that a significant chunk of the economy always can and does make the most efficient change (everyone replacing the job, automated regulations replacing banks when they can't move fast enough). Which neglects many potential factors, like big economic actors not having to be efficient for a long time, backlash from customers, and in general all factors making economic actors and market less than efficient.
I expect that most of these factors could be addressed with more work on the scenarios.
I consider this post as one of the most important ever written on issues of timelines and AI doom scenario. Not because it's perfect (some of its assumptions are unconvincing), but because it highlights a key aspect of AI Risk and the alignment problem which is so easy to miss coming from a rationalist mindset: it doesn't require an agent to take over the whole world. It is not about agency.
What RAAPs show instead is that even in a purely structural setting, where agency doesn't matter, these problem still crop up!
This insight was already present in Drexle...
I agree that a lot of science relies on predictive hallucinations. But there are examples that come to mind, notably the sort of phenomenological compression pushed by Faraday and (early) Ampère in their initial exploration of electromagnetism. What they did amounted to vary a lot of the experimental condition and relate outcomes and phenomena to each other, without directly assuming any hidden entity. (see this book for more details)
More generally, I expect most phenomenological laws to not rely heavily on predictive hallucinations, even when they integra...
So reification means "the act of making real" in most english dictionaries (see here for example). That's the meaning we're trying to evoke here, where the reification bias amounts to first postulate some underlying entity that explain the phenomena (that's merely a modelling technique), and second to ascribe reality to this entity and manipulate it as if it was real.
One point evoked by other comments, which I've realized only after leaving France and living in the UK, is that there is still a massive prestige for engineering. ENS is not technically an engineering school, but it benefits from this prestige by being lumped with them, and by being accessed mainly from the national contests at the end of Prepas.
As always with these kind of cultural phenomena, I didn't really notice them until I left France for the UK. There is a sense in France (more when I was a student, but still there) that the most prestigious jobs ar... (read more)