This is a special post for quick takes by Noah Birnbaum. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
While I think LW’s epistemic culture is better than most, one thing that seems pretty bad is that occasionally mediocre/shitty posts get lots of upvotes simply because they’re written by [insert popular rationalist thinker].
Of course, if LW were truly meritocratic (which it should be), this shouldn’t matter — but in my experience, it descriptively does.
Without naming anyone (since that would be unproductive), I wanted to know if others notice this too? And aside from simply trying not to upvote something because it’s written by a popular author, anyone have good ideas for preventing this?
The effect seems natural and hard to prevent. Basically, certain authors get reputations for being high (quality * writing), and then it makes more sense for people to read their posts because both the floor and ceiling are higher in expectation. Then their worse posts get more readers (who vote) than posts of a similar quality by another author, who's floor and ceiling is probably lower.
I'm not sure the magnitude of the cost, or that one can realistically expect to ever prevent this effect. For instance, ~all Scott Alexander blogposts get more readership than the best post by many other authors who haven't built a reputation and readership, and this kind of just seems part of how the reading landscape works.
Of course, it can be frustrating as an author to sometimes see similar quality posts on LW get different karma. I think part of the answer here is to do more to celebrate the best posts by new authors. The main thing that comes to mind here is curation, where we celebrate and get more readership on the best posts. Perhaps I should also have a term here for "and this is a new author, so I want to bias toward curating them for the first time so that they're more invested in writing more good content".
4Yes, but you'd naively hope this wouldn't apply to shitty posts, just to mediocre posts. Like, maybe more people would read, but if the post is actually bad, people would downvote etc.
1That's right. One exception: sometimes I upvote posts/comments written to low standards in order to reward the discussion happening at all. As an example I initially upvoted Gary Marcus's first LW post in order to be welcoming to him participating in the dialogue, even though I think the post is very low quality for LW.
(150+ karma is high enough and I've since removed the vote. Or some chance I am misremembering and I never upvoted because it was already doing well, in which case this serves as a hypothetical that I endorse.)
One thing you could do is give users relatively more voting power if they vote without seeing the author of the post. I.e., you can enable a mode which hides post authors until you give a vote on the anonymized content. After that, you can still vote like normal.
Obviously there are ways author identity can leak through this, but it seems better than nothing.
You should probably link some posts, it's hard to discuss this so abstractly. And popular rationalist thinkers should be able to handle their posts being called mediocre (especially highly-upvoted ones).
A salient example to me: This post essentially consists of Paul briefly remarking on some mildly interesting distinctions about different kinds of x-risks, and listing his precise credences without any justification for them. It's well-written for what it aims to be (a quick take on personal views), but I don't understand why this post was so strongly celebrated.
2This was also on my mind after seeing Jesse’s short form yesterday. Ryan’s “this is good” comment was above Louis’ thorough explanation of an alternative formal motivation for IFs. That would still be the case if I hadn’t heavy upvoted and weak downvoted.
I personally cast my comment up/downvotes as an expression of my preference ordering for visibility. I would encourage others to also do so. For instance, I suggest Ryan’s comment should’ve been agreement voted rather than upvoted by others. This stance has as a corollary to not vote if you haven’t read the other comments whose rankings you are affecting—or rather vote with any other marker of which LW has many.
This ‘upvotes as visibility preferences’ policy isn’t tractable for posts, so I suspect the solution there—if one is needed—would have to be done on the backend by normalization. Not sure whether this is worth attempting.
Link here since I don’t particularly want to call out Ryan, his comment was fine. https://www.lesswrong.com/posts/7X9BatdaevHEaHres/jesse-hoogland-s-shortform
idea: popular authors should occasionally write preregistered intentionally subtly bad posts as an epistemic health check
1
1Duncan Sabien once ran the inverse experiment. He made a separate account to see how his posts would do without his reputation. The account only has one post still up, but iirc there used to be many more (tens). They performed similarly well to posts under his own name. Cool idea!
[plausibly I'm getting parts of the story wrong and someone who was around then will correct me]
i've thought big names should do this for conference papers to keep conferences honest (peer review is anonymous but as i understand it it's extremely obvious when a big name has written a paper)
Of course, if LW were truly meritocratic (which it should be), this shouldn’t matter — but in my experience, it descriptively does.
That's not really true if people are popular rationalist thinkers because of skill at good rationalist writing. Meritocracy does not imply that people get judged based on individual pieces of their work, a meritocracy where people are primarily judged on their total output would still be a meritocracy.
I think the problem is more that post that make points that are popular and fit neatly into the world view of the reader are more likely to get upvotes than posts that challenge the world view of the reader and would require the reader to update their world view. Jimmy's introductionary post to his sequence is currently as I'm writing it at 12 karma.
While writing quality might be improved, it's a post that sets out to challenges the readers conception about how human reasoning works in practical contexts and that's why it's at low karma. I would love to see more posts like Jimmy that are about the core of how rationality works over posts that feel good to read and that get lot of upvotes without really changing minds much.
I think if someone is very well-known their making a particular statement can be informative in itself, which is probably part of the reason it is upvoted.
I had similar thoughts. And indeed I found funny that a forum of rationalists falls for this.
But this is human nature, and we have finite time and energy to focus on things.
You will find the same for the attention papers written by people at famous institutions get, even if low quality.
Making names, recently, I had a similar feeling when looking at "AI as Normal Technology". I did not know the authors, and I was skimming it because someone talked about it.
It is also a reason for which sometimes celebrities are called to do propaganda.
A strategy would be to make make posts anonymous for a limited amount of time (perhaps with some text randomization that would slightly change the style of the author/s, even though this might affect quality).
I was pretty unimpressed with Dario Amodei in the recent conversation with Demis Hassabis at the World Economic Forum about what comes after AGI.
I don’t know how much of this is a publicity thing, but it felt he wasn't really taking the original reasons for going into AI seriously (i.e. reducing x-risk). The overall message seemed to be “full speed ahead,” mostly justified by some kinda hand-wavy arguments about geopolitics, with the more doomy risks acknowledged only in a pretty hand-wavy way. Bummer.
My main takeaway of what Dario said in that talk is that Anthropic is very determined to kick off the RSI loop and willing to talk about it openly. Dario basically confirms that Claude Code is their straight shot at RSI to get to superintelligence as fast as possible (starting RSI in 2026-2027). Notably, many AI labs do not explicitly target this or at least don’t say this openly. While I think it is nice that Anthropic is doing alignment research and think that openly publishing their constitution is a good step, I think if they are successfully kicking off the RSI loop they have very low odds of succeeding.
@Zac Hatfield-Dodds @evhub @Dave Orr @Ethan Perez @Carson Denison @Drake Thomas @gasteigerjo @Aram Ebtekar Can you comment on this? Is that what they are planning to work on? Were you aware of this? Do you think that's a good thing to do?
(Epistemic status: third hand / rumor.) I heard that this is spoken about openly in Amodei's ~weekly all-hands, and this has been the case for a ~year.
Just listened to the talk and I'm confused about what people think Dario said. The only mention I noticed of RSI at the end was when the moderator said: "Last question. What will have changed in a year."
Dario said, paraphrasing, that the most important thing to watch this year is AIs building AIs, which could lead to wonders or a great emergency in front of us if it turns out that works and causes a meaningful speedup.
Is that what you're talking about? I'm pretty sure I agree that this is one of the most important things to pay attention to in the next year.
Would you please listen for 2 minutes to the beginning of the interview around the 1 minute mark, here:
https://www.youtube.com/watch?v=02YLwsCKUww&t=59s
Over the interview, it's clear that Dario's stated timelines are significantly more confidently short than Demis's; e.g. see here https://youtu.be/02YLwsCKUww?t=1030
Then, if you listen here for a couple minutes:
https://youtu.be/02YLwsCKUww?t=1326
it's clear that Dario's stated reason for NOT slowing down HIS research, seemingly refering to the short-timelines research, i.e. the RSI research, is something about China.
There's a relevant section near the beginning, when the moderator asks him about his timeline: https://youtu.be/9Zz2KrBDXUo?si=9kHfNC_ec-2yTk8e&t=61
Did you mean to ask me, rather than someone else in this thread? (I had a quick look at the new essay but haven't read it in full yet; what I did see didn't seem very surprising, so I don't really feel better or worse at this stage. Happy to say more once I've read it properly, but I think you might have meant to ask someone else, as I wasn't really taking a position in this thread -- just pointing out a relevant section of the talk that I thought you might have missed.)
I've seen this message, but per our confidentiality policy, no comment. (and don't take the below as indicative either way, glomar response etc)
I will note though that Dario did not actually talk about recursive self-improvement, nor about superintelligence; commentary on Lesswrong often assumes a shared ontology that just doesn't exist.
I also continue to endorse the confidentiality policy, and per my red lines I still trust Anthropic's leadership and think the company is on net good for the world.
1Can you, or have you elsewhere, elaborate on the presumed shared ontology that doesn’t exist? I think I’d expect you to have a pretty good sense of the delta, and it could be important for learning how to talk to each other more effectively.
I think getting RSI and a shot at superintelligence right just appears very difficult to me. I appreciate their constitution and found the parts i read thoughtful. But I don't see them having found a way to reliably get the model to truly internalize it's soul document. I also assume if they were able there would be parts that break down once you get to really critical amounts of intelligence.
By "succeeding" you mean getting Safe ASI, as opposed to getting any ASI at all, right? At least that's how I read you, but at first I thought you meant "their RSI probably won't lead to ASI"
Their RSI very likely won't lead to safe ASI. That's what I meant, hope that clears it up. Whether it leads to ASI is a separate question.
1it felt he wasn't really taking the original reasons for going into AI seriously (i.e. reducing x-risk)
Whose original reasons?
Dario's. I am a bit confused what Dario was always planning to do, but early Anthropic definitely centrally recruited with the pitch of reducing AI x-risk.
Have you written up somewhere what you've taken away from the Dario episode regarding "our" credulity? Would be curious to hear. (I could try to generate more specific questions/prompts if you like.)
People: “Ah, yes. We should trust OpenAI with AGI.” OpenAI: https://www.nytimes.com/2024/07/04/technology/openai-hack.html “But the executives decided not to share the news publicly because no information about customers or partners had been stolen, the two people said. The executives did not consider the incident a threat to national security because they believed the hacker was a private individual with no known ties to a foreign government. The company did not inform the F.B.I. or anyone else in law enforcement.”
Confidence level: strongly held, mostly opinionated, based on observation of (imo) bad LW norms.
We should stop using the phrase “epistemic status” and start using “confidence level.” In principle, “epistemic status” is meant to convey richer meta-information than confidence alone (i.e. the kind of evidence or how seriously a claim should be taken). In practice, it almost never does—on LW it’s usually just a clunkier way of saying “x confidence.”
If we actually want to convey more with less, we should just say “confidence level” and briefly qualify it with the relevant epistemic details (I.e. “low confidence, based on analogy,” or “high confidence, but mostly theoretical”). That’s clearer, less in-group-y, and lower friction. I think this is a good way to save up some weirdness points.
(Alternatively, one can used “qualified confidence” - a bit more jargony but traded for a bit more accuracy, though I perosonally like confidence level most).
1
1
1Words have meanings. Confidence level is about my conclusion; epistemic status is about how I got there. These are different things, and the terms should be used accordingly.
I find epistemic status far more informative than confidence level. What can I do with someone’s “70%”? Nothing.
A bunch of examples of it being used:
Epistemic Status: My best guess (but, epistemic effort was "talked to like 2-3 people about it and it felt good to each of us")
—Source
Epistemic status: Exploratory, speculative, half-baked thought
—Source
Epistemic Status: I've really spent some time wrestling with this one. I am highly confident in most of what I say. However, this differs from section to section. I'll put more specific epistemic statuses at the end of each section.
—Source
Epistemic status - statistically verified.
—Source
And of course, my favorite of all time:
Epistemic Status: Eliezer Yudkowsky writing the sequences. They sentenced me to twenty years of boredom. Galileo. This army. Chris Christie to Marco Rubio at the debate. OF COURSE! A woman scorned. For great justice. The Fire of a Thousand Suns. Expelling the moneylenders from the Temple. My Name is Susan Ivanova and/or Inigo Montoyo. You killed my father. Prepare to die. Indeed. It’s a trap. Tomfidence. I swear on my honor. End this. I know Kung Fu. Buckle up, Rupert. May the Gods strike me down to Bayes Hell. Compass Rose. A Lannister paying his debts. The line must be drawn here. This far, no farther. They may take our lives, but they will never take our freedom. Those who oppose me likely belong to the other political party. Ball don’t lie. Because someone has to, and no one else will. I’ve always wanted to slay a dragon. Persona!
—Source
Could someone point to an example of "epistemic status" used correctly, where you couldn't just substitute it with "confidence level"?
I generally don't care much about people's confidence levels. I don't Aumann agree that hard. But I do care how much effort someone has put in, how settled an idea is, whether is has been helpful or predictive. "Epistemic status: personal experience" is directly useful to me. I'll judge probability on merits however confident someone is (maybe not if I know their calibration curves, but I don't), but if I know what effort they did and didn't put in, I'll happily directly update on that. I don't think it's factually true that epistemic status 'almost never' conveys something other than a confidence level.
Epistemic status: did a few minutes informal searching to sanity check my claims, which were otherwise off the cuff.
Yeah but I don't think OP meant that by using "confidence level" you have to give a percentage. You can just swap out the phrase. Your two examples:
Confidence level: personal experience
Confidence level: did a few minutes informal searching to sanity check my claims, which were otherwise off the cuff.
I think these still work perfectly well, and now they are understandable to a much larger set of people.
The first does not work at all. “Personal experience” is not a level of confidence, and does not imply any particular level of confidence.
I've seen it used as "here's how rigorous my methodology was", and that seems meaningful. I can be very confident in something based on vibes alone[1], or I can run a series of intensive statistical tests and still be somewhat shaky about the apparent conclusion[2].
- ^
"That guy in the black T-shirt looks mean, I bet he could win a fight against that guy in the leather jacket."
- ^
"I've trained a random forest classifier on several thousand amateur boxing matches, and got 95 percent accuracy on my validation set after making sure there was no cross contamination. The resulting model says that the fellow with the big glasses is likely to win a bout against the guy in the tank top, but I still wouldn't put money on it."
Thinking about this more. How about "epistemic humility note" or some such thing. Why that: I think saying "humility" is good self-prompting and reader-signaling. Seems similar purpose to "epistemic status" but leaves less room for accidentally dodging that purpose, though of course some room remains. Also similar but maybe stronger: maybe gives you more of a pass to be wrong in the reader's mind (they don't think you think you're right). Also similar but maybe stronger: reminding people who are inclined to believe impressively collected results that those results can still be wrong, misleading, barking up the wrong tree, etc.
Epistemic humility note: This is just some stuff I think. I might be wrong. I'm not sure how I came to these opinions and I can't prove them to you yet. Read if you believe I have an interesting track record of saying useful stuff, but don't believe me if I assert something.
Epistemic humility note: This is strictly reporting of data I gathered via mechanical means, and detailed description of what those means were. Any attempt to come to a conclusion is delegated to the reader, who in turn might be reading the data wrong.
Epistemic humility note: I have none. No epistemic humility. I am right, and you should believe what I say.
Epistemic humility note: I'm pretty sure I gathered the right data and have interpreted it correctly, but if you don't agree, please tear me to shreds.
Epistemic humility note: Peer reviewed paper. We checked. We're sure we're right on what we report.
that last one reads as a warning sign to me, but seems easier to write than "Epistemic status: {same sentence}".
This post is very high epistemic status. Please show it the epistemic respect it epistemicologically deserves.
5US AISI will be 'gutted,' Axios reports: https://t.co/blQY9fGL1v. This should have been expected, I think, but it still seems worth sharing,
Someone I trust on this says:
AFAICT what's going on here is just that AISI and CHIPS are getting hit especially hard by the decision to fire probationary staff across USG, since they're new and therefore have lots of probationary staff - it's not an indication (yet) that either office is being targeted to be killed
To be frank though, that basically guts any chance of institutional capability, so it's probably going to be at best a hollow organization.
You do not randomly fire large swathes of people and expect most institutional knowledge to survive, because there's no reason for anyone to work there rather than be in the private sector.
True, but that's a different problem than them specifically targeting the AISI (which, based on Vance's comments, wouldn't be too surprising). Accidentally targeting the AISI means it's an easier decision to revert than if the government actively wanted to shut down AISI-like efforts.
I think people should know that this exists (Sam Harris arguing for misaligned AI being an x-risk concern on Big Think YouTube channel):
Thanks. For people that aren't likely to watch, I imagine it might also be worth saying he reports his view as being that we're in an arms race we can't opt out of (and that he's changed his mind regarding -- I think -- the overall appropriateness of such a race, though from what to what I'm not sure) due to insufficient political sanity and part of what constitutes sanity, he says, would be the US and China being able to create a climate whereby we don't fear each other, though it's not totally obvious whether he thinks a sufficient condition for the race to ASI being abandoned would be if people ended up no longer predicting their rivals might develop it first or whether he thinks some other fears need to be allayed as well (I get some sense that it's very much the latter, but it definitely wasn't clear).
I'm no expert on Albanian politics, but I think it's pretty obvious this is just a gimmick with minimal broader significance.
Tyler Cowen often has really good takes (even some good stuff against AI as an x-risk!), but this was not one of them: https://marginalrevolution.com/marginalrevolution/2024/10/a-funny-feature-of-the-ai-doomster-argument.html
Title: A funny feature of the AI doomster argument
If you ask them whether they are short the market, many will say there is no way to short the apocalypse. But of course you can benefit from pending signs of deterioration in advance. At the very least, you can short some markets, or go long volatility, and then send those profits to Somalia to mitigate suffering for a few years before the whole world ends.
Still, in a recent informal debate at the wonderful Roots of Progress conference in Berkeley, many of the doomsters insisted to me that “the end” will come as a complete surprise, given the (supposed) deceptive abilities of AGI.
But note what they are saying. If markets will not fall at least partially in advance, they are saying the passage of time, and the events along the way, will not persuade anyone. They are saying that further contemplation of their arguments will not persuade any marginal investors, whether directly or indirectly. They are predicting that their own ideas will not spread any further.
I take those as signs of a pretty weak argument. “It will never get more persuasive than it is right now!” “There’s only so much evidence for my argument, and never any more!” Of course, by now most intelligent North Americans with an interest in these issues have heard these arguments and they are most decidedly not persuaded.
There is also a funny epistemic angle here. If the next say twenty years of evidence and argumentation are not going to persuade anyone else at the margin, why should you be holding this view right now? What is it that you know, that is so resistant to spread and persuasion over the course of the next twenty years?
I would say that to ask such questions is to answer them.
Yes, that was a pretty terrible take. Markets quite clearly do not price externalities well, and never have done. So long as any given investor rates their specific investment as being unlikely to tip the balance into doom, they get the upside of directly financially benefiting from major economic growth due to AI, and essentially the same downside risk as if they didn't invest. Arguments like "short some markets, or go long volatility, and then send those profits to Somalia to mitigate suffering for a few years before the whole world ends" are obviously not even trying to seriously reflect the widespread investment decisions that affect real markets.
I saw this good talk on the Manifest youtube channel about using historical circumstances to calibrate predictions - this seems better for training than regular forecasting because you have faster feedback loop between the prediction and the resolution.
I wanted to know if anyone had recommendations on where to find some software or site where I can do more examples of this (I already know about the estimation game). I would do this myself, but it seems like it would be pretty difficult to do the research on the situation without learning the outcome. I would also appreciate people giving takes about why this might be a bad way to get better at forecasting.
This is a really good debate on AI doom -- I thought the optimistic side was a good model that I (and maybe others) should spend more time thinking about (mostly about the mechanistic explanation vs extrapolation of trends and induction vs empiricist framings), even though I think I disagreed with a lot of it on an object level:
https://marginalrevolution.com/marginalrevolution/2024/11/austrian-economics-and-ai-scaling.html
A good short post by Tyler Cowen on anti-AI Doomerism.
I recommend taking a minute to steelman the position before you decide to upvote or downvote this. Even if you disagree with the position object level, there is still value to knowing the models where you may be most mistaken.