Models don't "get" reward. Reward is the mechanism by which we select parameters, it is not something "given" to the model. Reinforcement learning should be viewed through the lens of selection, not the lens of incentivisation. This has implications for how one should think about AI alignment.
To anyone currently going through NeurIPS rebuttals fun for the first time, some advice:
Firstly, if you're feeling down about reviews, remember that peer review has been officially shown to be a ridiculous random number generator in an RCT - half of spotlight papers are rejected by another review committee! Don't tie your self-worth to whether the roulette wheel landed on black or red. If their critiques don't make sense, they often don't (and were plausibly written by an LLM). And if they do make sense (and remember to control for your defensiveness), then this is great - you have valuable feedback that can improve the paper!
1. Read this guide to get a sense of what rebuttals are about
1. Generally, be nice and polite, even if your reviewers are really annoying
2. You have three goals here:
1. Improving the paper! Often reviewers raise some good and useful points, and ultimately one of the key goals is doing good research and communicating it well to the world
2. Convince reviewers to like you, so they increase their score
3. For unreasonable reviewers who dislike you, your goal is to convince the area chair (and other reviewers) that this person is wrong and unreasonable. This means you still should write a careful and well-argued rebuttal, even to onbnoxious reviewers, but have a different target audience in mind.
1. Meta: The way the process works is that the area chair makes the final decision, and has a lot of discretion to overrule reviewers, but by default if lazy will go by the average reviewer score. You want to either increase average reviewer score, or convince the area chair to ignore the bad ones. Convincing a reviewer is just a means to the end.
3. One of the key things to do in the rebuttal is to improve the paper. Realistically, you can't upload a new version, so your actual goal is to convince people that you have improved the paper. It is an adversarial setting and people will generally assume you are lying if you j
everyone is a few hops away from everyone else. this applies in both directions: when I meet random people they always have some weak connection to other people I know, but also when I think of a collection of people as a cluster, most specific pairs of people within that cluster barely know each other except through other people in the cluster.
Long have I searched for an intuitive name for motte & bailey that I wouldn't have to explain too much in conversation. I might have finally found it. The "I was merely saying fallacy". Verb: merelysay. Noun: merelysayism. Example: "You said you could cure cancer and now you're merelysaying you help the body fight colon cancer only."
You will always oversample from the most annoying members of a class.
This is inspired by recent arguments on twitter about how vegans and poly people "always" bring up those facts. I content that it's simultaneous true that most vegans and poly people are either not judgmental, but it doesn't matter because that's not who they remember. Omnivores don't notice the 9 vegans who quietly ordered an unsatisfying salad, only the vegan who brought up factoring farming conditions at the table. Vegans who just want to abstain from animal products remember the omnivore who ordered the veal on purpose and made little bleating noises.
And then it spirals. A mono person who had an interaction with an aggro poly person will be quicker to hear judgement in the next poly person's tone, and vice versa. This is especially bad because lots of us are judging others a little. We're quiet about it, we place it in context instead of damning people for a single flaw, but we do exercise our right to have opinions. Or maybe we're not judging the fact, just the logistical impact on us. It is pretty annoying to keep your mouth shut about an issue you view as morally important or a claim on your time, only to have someone demand you placate them about their own choices.
AFAICT this principle covers every single group on earth. Conservatives hear from the most annoying liberals. Communists hear from the most annoying libertarians. Every hobby will be publicly represented by its members who are least deterred by an uninterested audience.
Popular Comments
Alright. Long stream-of-consciousness comment incoming. I do apologize for my tone below a bit, but refining it to make it more neutral would have taken even more of my time than this did; unfortunately, it has ended up as less of a compilation of questions and more just bullet points where I complain about what I disliked. Many of my own criticisms of and disappointments with HPMOR reflect parts of what su3su2u1 wrote about a long time ago.[1] Unsurprisingly, HPMOR fans find it tough to read such obviously sneery commentary, so I think Alexander Wales's excellent review of the story serves as a more than worthwhile replacement (and perhaps useful background reading for my comment here). But to write out my own thoughts explicitly and perhaps focus on what seem to me like the key topics:
* In Who's the Main Character, Eliezer repeats part of what he wrote about a long time ago, namely asserting that HPMOR is not about one person, one character, one guy against the whole world made of NPCs (even though Harry thinks of himself that way sometimes), but is instead significantly more complex and realistic. Specifically, Eliezer claims there are 4 characters which make decisions that move the story forward. Perhaps this may be what he intended in the story, but it definitely does not read that way to me as I read it. There is one character proactively moving the plot forward over the course of the events that unfold, and that character is Quirrell.
Dumbledore? He doesn't take agency over anything for 90% of the story; he had set up the pieces well in advance, and he shows up at the end, but the actual day-to-day activities and the events that result in the ultimate confrontation between the hero and the villain unfold without his direct involvement. He is more a force of nature bringing forth Acts of God in a way even he doesn't understand than an actual character making deliberate, reasoned decisions to influence what happens, over the course of the actual plot.
Hermione? Actually, seriously, what does Hermione do[2] that matters to the primary plot? The SPHEW arc was (rightfully IMO) seen by many readers at the time as boring; that wasn't because fighting bullies is inherently boring,[3] or because they were all sexist misogynists, but because it has very little to do with what the story was about before, and with what the story was building to afterwards.
Harry? Harry also does very little in the story; he talks a lot, he's the main character, he speaks about his ideals and what he wants to achieve etc, but what actual agency does he take over events that matter to the primary plot of the story? He serves as Quirrell's puppet: Quirrell says the afterlife doesn't exist, Harry believes him; Quirrell says we should storm Azkaban, Harry says 'of course!'; Quirrell lies in bed sick, Harry's thoughts are only on Quirrell; Quirrell literally casts the Avada Kadavra curse at an Auror doing his job, Harry doesn't care one bit after hearing one line of explanation from his mentor. Harry says he wants to defeat Death, but does he do anything to bring that about?
No! Quirrell is the one who defeats death and becomes immortal, Quirrell is the one who revives Hermione, Quirrell is the one who brings Harry the Ultimate Stone to Do Everything. Harry just mopes about complaining about how unfair the world is and how bad it is that everything isn't Optimal, and everyone else just solves all his issues for him.[4] Harry is literally fated to bring apart the very stars in heaven, and Quirrell is the one who solves this by forcing him into a carefully-constructed Unbreakable Vow that literally prevents him from saying and doing world-ending crap within days of its enactment!
For all the Trope Awareness Harry and even HPMOR itself both signal, Villains Act, Heroes Mope About is in full force here.
* I recall reading somewhere (can't recall the link off the top of my head) that the difference between a nerd reader and a "regular" reader is that a nerd reader cares most about worldbuilding, while "regular" readers care most about characters. Nerdiness aside, Eliezer obviously cares very deeply about constructing good characters (even writing advice about how to do that, and talking about this at length in this very post). So let's talk about Harry's character arc for a second here.
I... find it kind of difficult to do that, because there's very little to talk about. This is deeply disappointing, given he's the primary viewpoint character in a story totaling over 500 thousand words. Eliezer likes to talk about the fact that Harry fails a lot in HPMOR. And yes, he does fail.[5] But what's critical is that there are almost never real consequences to him failing.
Harry messes up and breaks his commitments and loses the Time Turner... oh wait, no problem, Quirrell (ha, of course it's Quirrell! who else could be allowed to have real agency?) just happens to have a Time Turner himself, so none of that matters! Harry tries to blackmail and deceive McGonagall at the beginning to obtain information and enforce his will (him, a kid, entirely unfamiliar with the magical world, versus her, a witch, old, experienced, respected) - surely that will result in her losing respect for him and his reputation being dragged into the toilets... ha, just kidding, Minerva now treats him almost as an equal! Harry is thrust into a deep and important conversation with the wily and politically powerful Lucius Malfoy where he doesn't know what's going on... Lucius ends up confused and impressed with Harry. Harry accidentally lets his mouth speak faster than his brain can catch up and he cures Snape's obsession with Lily Potter... no negative consequences flowing from that. Harry escalates and escalates and escalates against Snape because he thinks this is a fairy tale and he's the hero[6] - surely now he will get the slapdown from Wise Old Wizards like Dumbledore... no, of course not, Harry outmaneuvers and outwits everyone in the story to get his way!
He doesn't even learn any lesson from that; in the Wizengamot meeting, he does the exact same thing to protect Hermione, in front of wizards more powerful, old, and knowledgeable than he can imagine, and... he succeeds masterfully, obviously! Does he do that because of his deep understanding of wizard psychology? No, he just Plays the Game at a Different Level with his half-baked, half-forgotten first-year-undergrad-in-psych facts and logic, and the brains of all these hundred-year-old politicians and wizards are blown.
In fact, there is only one character Harry doesn't get to outwit in the game of Levels in this story, and that character is... Quirrell (of course). Harry realizes he has a Dark Side and he needs to keep it in check... ha, just kidding, the Dark Side solves his every issue and he never faces negative consequences from employing it! Harry breaks Bellatrix out of Azkaban because Quirrell said so... and the consequences are tiny and far-off and frankly I can't be bothered to care about them because they only appear in Chapter 110 and that chapter sucks for unrelated reasons that break my suspension of disbelief so bad I can't even think about Flamel.[7]
Harry learns the power of Friendship and teamwork and cooperation from the Ender's Game pastiche, and he realizes going at it alone won't be enough... and then he kills (read: brutally and bloodily slaughters like cattle[8]) all the Death Eaters and vanquishes Voldemort through his own wand. Ironic, isn't it?
The one action he proactively takes in this story, he does all by himself; if that's not Aesop Amnesia, I don't know what is.
* I really can't sum it up any better than Alexander Wales did, in explaining how Harry actually undergoes a character involution if anything:
"Harry is never given any incentive to change, and never really shows any change. The character growth arc is implied, but for the most part not actually present. Harry does not win the climax of the fic by having overcome his flaws, he wins it through brutal murder. The biggest organic change he undergoes is from believing in the value of truth to advocating for multiple conspiracies against both the wizarding and muggle worlds, and if that's character growth, I find it ugly."
* What's worse about the brutal murder part isn't that it happened. In fact, it's totally ok for it to have happened; the world needs an actually good Rationalfic where the hero says "screw the Batman ethos, it's nonsensical from a consequentialist perspective!" The problem is that, as revealed in Chapter 115, the story is embarrassed about it.
It doesn't strike the triumphalist note of success over the enemy[9], it doesn't backtrack and have Harry admit remorse or regret over the killing of Death Eaters, it just kind of wants us to forget about all that by just focusing on Quirrell (of course) as the one not deserving of being killed, because nobody deserves to be killed and he should instead one day live out his dream of sailing to the stars. Too bad for all the other Nameless Mooks that just got slaughtered, who may have had their own dreams...
Ironically, I guess in HPMOR one supervillain death is a tragedy and all Death Eaters dead is a statistic.
* While HPMOR is realistic in a sense (I suppose), the SPHEW arc is not. It presents a cartoonish view of bullies and their psychology, and does not attempt at any point to explain why reasonable authority figures like Minerva, who obviously both care deeply about ensuring the psychological and especially physical safety of the students and also have a ton of power over and respect over the students, allow something like this to happen.
I can understand why Dumbledore didn't step in; he believes heroes are born in Tough Times when they realize authority figures won't save them. How about everyone else? The entire system, the oversight over Hogwarts from the rest of the magical world, the families of the students being bullied... it's the Wild West out here and nobody is batting an eye?
Even if that can be explained in context, it needs the explanation! Otherwise it just looks and seems cartoonish and turns people off (as the SPHEW arc indeed did).
* Chapter 110 has Dumbledore hold the Idiot Ball very strongly, in a way Eliezer said no major character in the story would. This unfortunately both shatters the suspension of disbelief and the reader's immersion into the story, and also makes the chapter feel worse and worse with every re-reading.
* Eliezer writes about how Orson Scott Card said "while a conflict between good and evil might hold the attention of some readers, a conflict between good and good can be much stronger than that." The problem is that, in HPMOR itself, the grand finale, the grand conflict between Harry and Quirrell... doesn't happen because of a conflict between good and good. It doesn't happen because of fundamentally irreconcilable moral differences between the protagonist and the villain. It doesn't happen because Harry and Quirrell disagree over any predictive aspect of how the world will be if certain actions are taken.
It happens because of prophecy. Quirrell would have no reason to go against Harry, and indeed did not go against Harry, until he heard Trelawney's second prophecy. As revealed in Parseltongue, where there can be no lies, Quirrell would have loved to just play a game with Harry for the rest of time where they just keep themselves entertained and fool the masses, where he teaches Harry the secret of the new Horcrux spell and makes him immortal and keeps him as his equal for all of eternity. It is entirely an external impetus that causes them to go against one another, like the Voice of God telling them they should fight instead of there being an organic cause of their battle.
This is very much less interesting than the alternative.
I'm too tired now to keep lengthening this comment, even though I have multiple other issues with HPMOR. Perhaps I'll expand on them some other time.
1. ^
Even though I genuinely and unironically enjoyed reading the story, unlike the Sneer Club
2. ^
As opposed to having stuff be done to her (being framed, being killed, being revived... notice how she is not the actor, the agent, in any of that)
3. ^
But wait... more on that later!
4. ^
Until Chapter 114, but wait... more on that later!
5. ^
Kind of. Not really in any important ways... more on that later!
6. ^
At the very least this is actually talked about in the text itself as a blunder from Harry, but ACTIONS SPEAK LOUDER THAN WORDS! Quirrell (ha, of course it was... ok, you guys get the point by now) says it was a dumb thing to do, Harry ultimately agrees, and... nothing comes of it. No lasting consequences, no real lesson
7. ^
More on this later!
8. ^
How's that for a death-hating protagonist!
9. ^
Except by emphasizing Sunshine and Friendship and Goodness... after dozens of wizards just got sliced
The economics here seem wrong.
Poor people do not benefit less from debt than rich people do - they benefit vastly more, because they have major cashflow and liquidity issues. (I wouldn't go so far as to claim interest rates are the least important things about debt, but they are discussed disproportionately compared to aspects like credit availability.) They do not shun debt, but use it in countless highly complex forms to deal with the challenges of routinely running out of money before the next pay period, or the 'feast or famine' variance in payments that to a richer (ie. middle-class) person would barely register as a blip on their bank account balance. Arbitraging 2% vs 5% ROI is trivial compared to arbitraging 'not getting evicted' or 'not getting fired'. (Borrowing $20 for gas in the next half hour can be life-changing; getting 40 cents extra on your Vanguard ETF retirement account 50 years later is not.)
A useful reference for me for understanding this was reading Portfolios of the Poor.
Incidentally, I would note that Polonius is an aristocrat speaking to an aristocrat (and about to be killed through his own foolishness), and his advice should not be taken on finance, or perhaps anything else either.
I have various thoughts on the actual topic here that I might post later, but for now I wanted to quickly ask: did you write this collaboratively with an LLM chatbot? And if so, what was the division of labor between you and the LLM? (I'm not trying to "catch" you posting AI writing, here, I'm just curious whether my gut sense is right or not.)
The first section about the boarding school feels especially LLM-esque to me, with its repeated use of the "not X but Y" construction ("it wasn't justice—it was logistics," "isn't her rebellion, but her sadness"), the way it's crammed with striking details that feel made-to-order for the story's themes ("She never fed her horse," "She cried while decorating planners and making gingerbread houses"), its very rapid pace that never stops to dwell on any given topic for long, its use of the name "Aria"[1], etc.
(There are some other features of the first section that are weighting heavily in my judgment here but are harder to describe in words... there's something about the cadence, I guess? And also the... oddly detached perspective of the narrator? And the use of a somewhat precious/cutesy tone in spite of the heavy subject matter? And some other stuff too.)
I apologize for butting in to ask something totally unrelated to the substance of the essay -- I just didn't want to miss the opportunity to get feedback about how well my inner "AI writing detector" is functioning.
1. ^
Like "Elara" and "Voss," this is one of those names that recent chatbot LLMs very often use for fictional characters, even though they're relatively rare in real life and in non-LLM-written fiction.
As a quick demonstration, try searching for tweets that contain both "Aria" and "Elara." When I did that just now, it turned up a ton of obviously AI-generated content, including several posts from the Grok account.
If I instead search for both "Jessica" and "Elara," I get hardly any results -- and even fewer AI-generated results -- despite the fact that Jessica is ~400x more common than Aria in the U.S. Which shows that these results are not just determined by strong AI-Elara link on its own.
Recent Discussion
This is a linkpost for https://www.obsolete.pub/p/anthropic-faces-potentially-business
A class action over pirated books exposes the 'responsible' AI company to penalties that could bankrupt it — and reshape the entire industry
This is the full text of a post first published on Obsolete, a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to Build Machine Superintelligence. Consider subscribing to stay up to date with my work.
This piece has been updated to add additional context and clarify some details.
Anthropic, the AI startup that’s long presented itself as the industry’s safe and ethical choice, is now facing legal penalties that could bankrupt the company. Damages resulting from its mass use of pirated books would likely exceed a billion dollars, with the...
20
Training on copyrighted data wasn't ruled infringing by itself though, only pirating the books was. So even if the maximalist interpretation of damages was upheld, companies could still legally purchase books to train on.
1
Afaict this case has been generally good for the industry but especially bad for Anthropic.
5
Clearly the heroic thing to do would be to go to trial and then deliberately mess it up very badly in a calculated fashion that sets an awful precedent for the other AGI companies. You might say, "but China!", but if the US cripples itself, then suddenly the USG would be much more interested in reaching some sort of international-AGI-ban deal with China, so it all works out.
(Only half-serious.)
2
Responding to the serious half only, sandbagging doesn't work in general in the legal system, and in particular it wouldn't work here. That's because you have so much outside attention on the case and (presumably) so many amici briefs describing all the most powerful arguments in the AI companies' favor. If the judge sees that you are a $61 billion market cap company hiring the greatest lawyers in the world, but you're not putting forth your best legal foot when you have lawyers from other companies writing briefs outlining their own defense arguments, the consequences for you and your lawyers will be severe and any notion of "precedent" will be poisoned for all of time.
Eliezer and I love to talk about writing. We talk about our own current writing projects, how we’d improve the books we’re reading, and what we want to write next. Sometimes along the way I learn some amazing fact about HPMOR or Project Lawful or one of Eliezer’s other works. “Wow, you’re kidding,” I say, “do your fans know this? I think people would really be interested.”
“I can’t remember,” he usually says. “I don’t think I’ve ever explained that bit before, I’m not sure.”
I decided to interview him more formally, collect as many of those tidbits about HPMOR as I could, and share them with you. I hope you enjoy them.
It’s probably obvious, but there will be many, many spoilers for HPMOR in this article, and also very little...
20
Chapter 110 Dumbledore is an over-the-top caricature of himself who has months to set up the perfect trap, while having access to both his century-old deep knowledge of magic and to some of the most powerful artifacts in the world (Elder Wand, Mirror of Erised, Line of Merlin Unbroken, etc.), but he gets wiped off the gameboard in minutes. This happens in a way that fulfills his enemy's ideal scenario, any countermeasures destroyed immediately by the artifact he himself introduces into the plot.
Readers at the time thought this was so out of character for D...
2
Fair point!
1
Harry is proactively moving the plot forward--he decided very early on that he was going to try turn Draco to the light side and succeeds in the task.
2
This is irrelevant in the grand scheme of things, because Draco is irrelevant to the final confrontation in HPMOR.[1][1] (If turning Draco to the light side counted, then Harry has done a dozen other things "moving the plot forward" - but he hasn't! The plot doesn't move forward unless either Quirrell sets up an adventure or Trelawney gives her second prophecy).
It does, however, accomplish something actually good, namely giving Draco an actual character arc. I'd say he's the only character in HPMOR that gets a solid character arc. Too bad he doesn't matter.
1. ^
What in the story changes if Draco never existed? A few chats with Lucius have their lines removed, Hermione gets framed for killing another one of her acquaintances, Neville or whomever gets Legilimensed into opening the forbidden door in Chapter 104, and... what else? Harry had mostly figured out the genetic laws of magic himself already
1
looks like theyre trying to double to 6 though: https://x.com/TheNormanMu/status/1948546711074603354
35
Leo Gao recently said OpenAI heavily biases towards work that also increases capabilities:
Before I start this somewhat long comment, I'll say that I unqualifiedly love the causal incentives group, and for most papers you've put out I don't disagree that there's a potential story where it could do some good. I'm less qualified to do the actual work than you, and my evaluation very well might be wrong because of it. But that said:
It seems from my current understanding that GDM and Anthropic may be somewhat better in actual outcome-impact to varying degrees at best; those teams seem wonderful internal-to-the-team, but seem to me from the outside to be currently getting used by the overall org more for the purposes I originally stated. You're primarily working on interpretability rather than starkly-superintelligent-system-robust safety, effectively basic science with the hope that it can at some point produce the necessary robustness - which I actually absolutely agree it might be able to and that your pitch for how isn't crazy; but while you can motivate yourself by imagining pro-safety uses for interp, actually achieving them in a superintelligence-that-can-defeat-humanity-combined-robust way reliably doesn't seem publicly like a success you're on track for, based on the interp I've seen you publish, and the issues with it you've now explicitly acknowledged. Building better interp seems to me to be continuing to increase the hunch formation rate of your capability-seeking peers. This even shows up explicitly in paper abstracts of non-safety-banner interp folks.
I'd be interested in a lesswrong dialogue with you, and one of Cole Wyeth[1] or Gurkenglas[1], in which I try to defend that alignment orgs at GDM and Anthropic should focus significant effort on how to get an AI to help them scale the things that Kosoy[2], Demski[3], Ngo[4], Wentworth[5], Hoogland & other SLT[6], model performance guarantee compression, and others in the direction of formal tools are u
6
I want to defend interp as a reasonable thing for one to do for superintelligence alignment, to the extent that one believes there is any object level work of value to do right now. (maybe there isn't, and everyone should go do field building or something. no strong takes rn.) I've become more pessimistic about the weird alignment theory over time and I think it's doomed just like how most theory work in ML is doomed (and at least ML theorists can test their theories against real NNs, if they so choose! alignment theory has no AGI to test against.)
I don't really buy that interp (specifically ambitious mechinterp, the project of fully understanding exactly how neural networks work down to the last gear) has been that useful for capabilities insights to date. fmpov, the process that produces useful capabilities insights generally operates at a different level of abstraction than mechinterp operates at. I can't talk about current examples for obvious reasons but I can talk about historical ones. with Chinchilla, it fixes a mistake in the Kaplan paper token budget methodology that's obvious in hindsight; momentum and LR decay, which have been around for decades, are based on intuitive arguments from classic convex optimization; transformers came about by reasoning about the shape and trajectory of computers and trying to parallelize things as much as possible. also, a lot of stuff Just Works and nobody knows why.
one analogy that comes to mind is if your goal is to make your country's economy go well, it certainly can't hurt to become really good friends with a random subset of the population to understand everything they do. you'll learn things about how they respond to price changes or whether they'd be more efficient with better healthcare or whatever. but it's probably a much much higher priority for you to understand how economies respond to the interest rate, or tariffs, or job programs, or so on, and you want to think of people as crowds of homo economicus wit
20
This seems reasonable. Personally, I’m not that worried about capabilities increases from mech interp, I simply don’t except it to work very well.
This post examines an attempt by professional decision theorists to treat an example of time inconsistency, and asks why they failed to reach the solution (i.e., TDT/UDT) that this community has more or less converged upon. (Another aim is to introduce this example, which some of us may not be familiar with.) Before I begin, I should note that I don't think "people are crazy, the world is mad" (as Eliezer puts it) is a good explanation. Maybe people are crazy, but unless we can understand how and why people are crazy (or to put it more diplomatically, "make mistakes"), how can we know that we're not being crazy in the same way or making the same kind of mistakes?
The problem of the ‘‘absent-minded driver’’ was introduced...
20
I believe I've solved the problem. I'm going to include this in my next post on probability theory fundamentals but here is the gist of it.
The problem is to come up with a general decision algorithm that both works (in the sense of making the right decisions) and (if possible) makes epistemic sense.
The meta-problem here is that people were looking for the answer in the wrong place, searching for a different decision making algorithm while what we actually needed is a satisfying epistemological account. The core crux isn't in decision theory, but on a...
2
What do you have in mind for rationality here? It's clear to me with EA, which has poured tens or perhaps hundreds of millions of dollars into growth programs, especially amongst students and young people, but I'm interested in what you're thinking about for rationality, which doesn't immediately strike me as having attempted to undergo massive growth (e.g. HPMOR is very popular but mostly grew because it was a well-written story and ppl loved it, not due to much in the way of active programs and propaganda).
2
I mean, I'm pretty sure animosity towards rationalists on Hacker News is older than the existence of OpenAI and probably even DeepMind. Also most people on Hacker News don't work in AI. So I don't really know why this hypothesis is coming to mind, I don't think it's relevant for most of what's gone on.
I'd be more inclined to put it down to Hacker News having many standard online pathologies for bullying easy targets, and rationalists historically being a lot of weird and outcast kinds of people, along with some strains of anti-intellectualism in the tech/startup world.
2
Interesting, the only recent irl aggression (verbal, not physical) I've received from the techie crowd in SF was related to the Sam Altman firing[1]. I've also gotten more standard leftist anger but I don't think that anger is very centrally Hacker News-y, I'd guess those people would also be angry at Hacker News.
1. ^
Oh I remember another time an ML person got mad at me for mentioning Bostrom's Superintelligence.
20
I have Hacker News blocked so I cannot pull up threads now, but what I have in mind (and I'm pretty confident is what DirectedEvolution had in mind) is many many threads on Hacker News where, when LessWrong is mentioned, it's called a cult and has a bunch of other low-quality critical comments about it with a derisive tone.
(I believe I recalled it getting better in the few years after LessWrong 2.0 started, though I think I've seen an uptick again around AI threads.)
everyone is a few hops away from everyone else. this applies in both directions: when I meet random people they always have some weak connection to other people I know, but also when I think of a collection of people as a cluster, most specific pairs of people within that cluster barely know each other except through other people in the cluster.
If it’s worth saying, but not worth its own post, here's a place to put it.
If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.
If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.
If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.
The Open Thread tag is here. The Open Thread sequence is here.
20
Just an oversight this time
13
I've long been confused why people don't just use something like "bait and switch" or "rope-a-dope"?
It's possible they're not the exact same concept, but they seem pretty close, and the former (maybe the latter too) already has an intuitive meaning to people
1
I would guess something like historical momentum is the reason people keep using it. Nicholas Shackel coined the term in 2005, then it got popularized in 2014 from SSC. 20 years is a long time for people to be using the term.
52
20 years is a long time sure, but I don’t think would be good argument for keeping it! (I understand you’re likely just describing, not justifying)
Motte & bailey has a major disadvantage of “nobody who hears it for the first time has any understanding of what it means”
Even as someone who knows the concept, I’m still not even 100% positive that motte and bailey do in fact mean “overclaim and retreat” respectively
People are welcome to use the terms they want, of course. But I’d think there should be a big difference between M&B and some simpler name in order to justify M&B
7
"Overclaim and retreat" also seems better than motte & bailey imo
This is a linkpost for https://turntrout.com/english-numbers-are-backwards
We're writing numbers wrong. We write "365" starting with the most significant digit of "3" (hundred). The "biggest number on the left" rule is both algorithmically bad and clashes with how humans intuitively represent numbers in their minds. I propose an innocent and totally practical fix: flip the written order of all numbers, writing "↗563" instead of "365." I analyze the implications of this change as they propagate through our language and thought.
Read this article in a prettier form on my website.
A modest proposal: flip the digit order
If I'm writing "three hundred and sixty-five", "365" becomes "↗563", with the "↗" read as "flip." Likewise, "21,514" becomes "↗415,12." As you move right (→), the each digit's magnitude goes up (↑). If you're writing an expression with multiple numbers,...
30
Ah, the battle between the big-endians and little-endians continues long after Jonathan Swift's tale.
4
The order of our numeral notation mirrors the order of our spoken numerals. I’m not sure if there are any languages that consistently order additive numerals from smallest to largest - “two and fifty and three hundred” instead of “three hundred and fifty two”.
>💡Flipping the local ordering of pronunciation: If we're truly optimizing, we might as well say "twenty and hundred-three" while we're at it. The first words "and three-" don't tell you much until you know "three of what"? Whereas "and hundred-three" tells you the order of magnitude as soon as possible.
This suggestion is aesthetically in tension with your principle of ordering from smallest to largest. Why should we go with informativeness for multiplication and small-to-large order for addition? The larger number in a sum is more informative about the size of the value, that is probably why languages tend to pronounce additive numerals from larger to smaller.
Interestingly, many languages actually do use the “hundred-three” order. You may be interested in this paper: https://www.nature.com/articles/s41599-023-02506-z, they have a striking geographic distribution.
12
We write numbers left to right in digits the same way we write numbers left to right in words.
Same direction, same order. Makes sense.
Having to look at an entire glyph sequence before knowing what the first glyph means is not unique to numbers. Words are like that, too. Entire sentences, even. It’s fine. This proposal is unmotivated and unnecessary.
17
Here is a list of numbers. Which two of these numbers are closest together?
815
187
733
812
142
312
