I think this used to be a tenable position a decade or two ago. But I think it's no longer tenable, due to the dynamic described in this tweet:
Suppose an ideology says you're not allowed to question idea X. At first X might not be very important. But now when people want to argue for Y, "X->Y" and "~Y->~X" are both publicly irrefutable. So over time X will become more and more load-bearing for censorious ideologies.
We can also think of this as a variant of Goodhart's law, which I'll call ideological Goodhart (and have just tweeted about here): any false belief that cannot be questioned by adherents of an ideology will become increasingly central to that ideology. As this process plays out, advocates of that ideology will adopt increasingly extreme positions, and support increasingly crazy policies.
2
1
1Tbc: It should be fine to argue against those implications, right? It’s just that, if you grant the implication, then you can’t publicly refute Y.
Unfortunately the way that taboos work is by surrounding the whole topic in an aversive miasma. If you could carefully debate the implications of X, then that would provide an avenue for disproving X, which would be unacceptable. So instead this process tends to look more like "if you don't believe Y then you're probably the sort of terrible person who believes ~X", and now you're tarred with the connotation even if you try to carefully explain why you actually have different reasons for not believing Y (which is what you'd likely say either way).
I expect this effect to be weaker than you're suggesting, especially if Y is something you in fact independently care about, and not an otherwise unimportant proximal detail that could reasonably be interpreted as a "just asking questions" means of arguing for ~X. I'm struggling to think of a particularly illustrative X and Y, but consider X="COVID was not a lab leak", which seemed lightly taboo to disagree with in 2020. Here's a pair of tweets you could have sent in 2020:
1. "I think COVID was probably a lab leak."
2. "I don't know whether COVID was a lab leak. (In fact for now I'm intentionally not looking into it, because it doesn't seem important enough to outweigh the risk of arriving at taboo beliefs.) But gain-of-function research in general is unacceptably risky, in a way that makes global pandemic lab leaks a very real possibility, and we should have much stronger regulations to prevent that."
I expect the second one would receive notably less push back, even though it defends Y="gain of function research is unacceptably risky", and suggests that Y provides evidence for ~X.
(Disagree that it was a tenable position a decade or two ago, agree that it is an untenable position now)
Fair, this isn't a confident claim from me. I do have a sense that the last decade has been particularly bad in terms of blatant preference falsification, but it's hard to distinguish "the world was different before then" from "I was younger and didn't have a great sense of what was going on".
I’m 62, so I was adult in the ‘80s and ‘90s. My sense is that the world was different. The consequences of expressing divergent opinion seem much more serious now.
2Can you say what position you recommend instead? Is it just opining publicly about everything, with no regard to how taboo it is?
Suppose an ideology says you're not allowed to question idea X.
I think there are two different kinds of "not questioning": there's unquestioningly accepting an idea as true, and there's refusing to question and remaining agnostic. The latter position is reasonable in the sense that if you refuse to investigate an issue, you shouldn't have any strong beliefs about it. And I think the load-bearingness is only a major issue if you refuse to question X while also accepting that X is true.
I don't see why it should be limited to false beliefs.
Note that even if X is true, X->Y need not be true, and it can still be harmful to not be able to question X->Y.
This is a good point. Though the thing about true beliefs is that there is a specific version of them that's true, which you're allowed to defend (if you can find it). And so you can more easily figure out what the implications are.
Whereas for false beliefs you can't get into the specifics, because looking hard enough at the specifics will tend to disprove the belief.
Is there another strategy you prefer? Afaict the options are
1) Have public taboo beliefs.
2) Have private beliefs that you lie about.
3) Remain deliberately agnostic about taboo but insufficiently important topics.
4) Get forever lucky, such that every taboo topic you investigate results in you honestly arriving at an allowed belief.
Whether 1) is at all compatible with having other career goals is a fact of the territory, and I expect in the US in 2024, there are topics where having taboo beliefs could totally end your career, for many values of career. (Leaving open whether there are such beliefs that are true, but, per the topic of this post, that's not something you can learn without taking risks.)
2) seems even more prone to the effect you describe than 3).
My guess is you're making a bid for 1), but I feel like a case for that should take into account the costs of believing X weighed against the costs of agnosticism about X, rather than a sweeping heuristic argument. (Where maybe the cost of agnosticism about X includes adjacent topics Y you'll either have to include in your agnosticism or otherwise eat the cost of ~X connotations, though I'm skeptical about how often this will come up, and per my comment here I expect the ~Y->~X taboos will often be much smaller than the ~X taboo.)
One option would be to look into it
Another option you didn't list is to look into why horrible acts have been committed in the name of dragons, and share the results of that.
Mm, perhaps it will turn out that there are other, more benign ways of dealing with dragons. Narrative-brained people wont find those.
I really don't want to contribute to this pattern that makes it hard to learn what's actually true, so in general I don't want whether I share what I've learned to be downstream from what I learn.
Another policy which achieves this is to research the question, and not (publicly) share your conclusion either way. This also benefits you in the case you become a dragon believer, because glomarizing (when you're known to follow this policy) provides no evidence you are one in that case. (Things this reminds me of: meta-honesty, and updatelessness when compared to your position)
Another policy which achieves this is to share your conclusion only if you end up disbelieving in dragons, but also hedge it that you wouldn't be writing if you believed the other position. If you're known to follow this policy and glomarize about whether you believe in dragons, it is evidence that either you do or you haven't researched the question.
The trouble with this is that it's a socially awkward move to even imply you might research taboo topics. Better to leave public personna and tricky glomarization out of it, I think. Just publish your research and results anonomously. That seems to me to lead to a better epistemic state for society, since a standard of anonymous publication doesn't leave a misleading bias in publicly available research.
1To pick an uncontroversial example, imagine someone glomerizing in whether the Earth was flat or (approximately) spherical. That would signal that you're the sort of person who considered a spherical Earth to be a plausible hypothesis, which is almost as bad as actually believing it. All reasonable, right-thinking people, on the other hand, know that it's obviously flat and wouldn't even consider such nonsense.
This is why it's important for the policy be known for the glomarization to be evidence under that policy specifically, which might include something to the effect of "I follow this even in obvious cases so I'm free to also follow it in cases which are mistakenly framed as obvious".
That said, I'm not thinking about the 'mundane' world as Eliezer calls it, where doing this at all would be weird. I guess I'm thinking about the lesswrong blogosphere.
(There's a hypothetical spectrum from [having a glomarization policy at all is considered weird and socially-bad] to [it is not seen negatively, but you're not disincentivized from sharing non-exfohazardous beliefs, to begin with])
Do dragon unbelievers accept this stance? My impression is that dragon agnosticism would often be considered almost as bad as dragon belief.
They don't, of course, but if you're lucky enough not to be located among the more zealous of them and be subjected to mandatory struggle sessions, their wrath will generally be pointed at more conspicuous targets. For now, at least.
It's fun to put various of my foundational non-hot-button-political beliefs in place of "dragons" and see which ones make my mind try to flinch away from thinking that thought for the reasons outlined in this post (i.e. "if I checked and it's not the way I thought, that would necessitate a lot of expensive and time-consuming updates to what I'm doing").
the reasons outlined in this post (i.e. "if I checked and it's not the way I thought, that would necessitate a lot of expensive and time-consuming updates to what I'm doing").
Wait, that's not what I'm trying to communicate in the post. If learning that dragons existed would precipitate major updates, it will very often be worth investigating their existence. Instead, it is highly-political low-payoff topics I am intentionally agnostic.
Concrete example: one of my core beliefs is " :::systems which get their productive capacity mainly from voluntary trade are both more productive and better for the people living under them than systems which get their productive capacity mainly from threats and coercion::: ". In this example, a "dragon" would be :::a coercion-based system which is more productive than a voluntary-trade-based one::: .
So working through the analogy:
- Historically, people who believed in this kind of "dragon", and who acted on that belief, tended not to behave very nicely.
- There are still people who believe in "dragons". They tend to valorize the actions of past dragon believers to a worrying extent.
- I think "dragons" probably don't exist, but I haven't actually proven it.
- I would prefer to live in a world where "dragons" don't exist.
- If "dragons" did exist, I would feel obligated to spend a lot of time and effort reevaluating my world model and plans.
- If I discovered the existence of a "dragon", and shared that, that would become a defining thing I am known for
- Most likely, there wouldn't be much that I, personally, could do with the information that a "dragon" exists
I guess arguably this is a hot-button political issue in some contexts. The other example I was thinking of was less political had the similar shape of "thing which, if true implies that we can'texpect to maintain certain nice things about the world we live in, and where people believing that the thing is true, if it is in fact true, would hasten the end of the nice things, to the benefit of nobody in particular".
ETA: On reflection I do agree that this is a different flavor of "nothing good can come of this line of thought" than the one you outlined in your post.
It already seems like we can infer that dragon-existence has, to you, nontrivial subjective likelihood because you don't loudly proclaim "dragons don't exist" and because you regard investigation as uncomfortably likely to turn you into a believer of something socially unacceptable.
If you think it's in fact, like, 20% likely (a reasonable "nontrivial likelihood" guess for people to make), seems like the angry dragons-don't-exist people should be 20% angry at you.
I don't think inferring a probability anywhere near as high as 20% is justified. If, conditional on finding dragons the value of the knowledge is lower than the harms of being known as a dragon believer, then you shouldn't go check no matter how low your prior.
This seems mostly fine for anyone who doesn't engage in political advocacy or activism, but a mild-moderate form of defection against society if you do - because if dragons are real, society should probably do something about that, even if you personally can't.
edit: I guess dragon-agnosticism is tolerable if you avoid advocating for (and ideally voting for) policies that would be disastrous if dragons do in fact exist.
I am agnostic about various dragons. Sometimes I find myself wondering how I would express my dragon agnosticism in a world where belief in dragons was prevalent and high status. I am often disturbed by the result of this exercise. It turns out that what feels like agnosticism is often sneakily biased in favor of what will make me sound better or let me avoid arguments.
This effect is strong enough and frequent enough that I don't think the agnosticism described by this post is a safe epistemic fallback for me. However, it might still be my best option in situations where I want to look good or avoid arguments.
Possibly related:
Selective Reporting and the Tragedy of the Green Rationalists by Zack M Davis
Kolmogorov Complicity and the Parable of Lightning by Scott Alexander
1I think even one dragon would have a noticeable effect on the population of large animals in the area. The huge flying thing just has to eat so much every day, it's not even fun to imagine being one. If we invent shapeshifting, my preferred shape would be some medium-sized bird that can both fly and dive in the water, so that it can travel and live off the land with minimal impact. Though if we do get such technology, we'd probably have to invent territorial expansion as well, something like creating many alternate Earths where the new creatures could live.
2
2
1I would perhaps prefer we had a list of three things we don't discuss (say Politics, Race science and Infohazards) and if we want to not discuss a new thing we have to allow discussion of one of those others. Seems better to be clear what isn't being discussed.
I'm pro being clear about what we don't discuss, but it's unreasonable to limit the list to three. The number of topics that is net negative to discuss is just a fact about the world and is probably over three, and I would rather not have people talk about the 4th worst controversial topic just because we uncover three even more pointless and controversial ones.
Politics also seems inadvisable to ban because it's too broad.
See also Heads I Win, Tails?—Never Heard of Her; Or, Selective Reporting and the Tragedy of the Green Rationalists. Which goes into how people who don't know that there's some implicit consensus not to talk about some things can come away confused and with wrong beliefs.
I think implementing this policy would require frequent publicization of what issues are on the current list and frequent debates and votes on whether to replace one topic on the list with another topic not yet on the list and would therefore have the opposite of the intended effect. Maybe there's a clever way to get around those issues, but I doubt it.
I think you'll find that no matter what you find out in your personal investigation of the existence of dragons, that you need not be overly concerned with what others might think about the details of your results.
Because what you'll invariably discover is that the people that think there are dragons will certainly disagree with the specifics about dragons you found out that disagrees with what they think dragons should be, and the people that think there aren't dragons will generally refuse to even seriously entertain whatever your findings are relating to dragons, and the vast majority of people who aren't sure about the existence of dragons will dismiss the very idea of spending time thinking about the existence of dragons, reasoning that the existence or non-existence bears little influence on their lives (otherwise they likely would have investigated the issue and landed in a respective camp).
So investigate dragons all you like, and shout it from the rooftops if you please. The void will hear you and appreciate it as much as the void can, while everyone else is much more concerned with their own feelings about dragons than whatever your thinking or reasoning on the subject might offer.
The only real tragedy is that if you come away thinking there might be dragons, but the dragons you find are very different from the dragons people expect dragon-believing people to believe in - well that's somehow the specific niche where both the dragon believers and non-believers find rare common ground to roll their eyes and think you're nuts.
So maybe do your rooftop shouting to the sole listening void anonymously?
It's also possible that an opposing effect happens where your shouting into the void about dragons connects in some vague way with my belief in the Ilithids, which I then end up coopting your dragon evidence into my own agenda. Especially if you find anything close to material evidence. Heck, your material evidence for dragons now gives all kinds of creedance to Ilithids, beholders, gnomes, and all sorts. So the gnome people and everyone else is now coming out of the woodwork to amplify your work on dragons. And I think this would be regardless of the specific nuances your attribute to dragons. I would expect those nuances to get smooshed in the fray to cite your once-and-for-all-proving-dragons strategy.
I mean, if Pons and Fleischmann was true, for example, I bet it would get trotted out with all kinds of half-baked theories on free energy, along with Tesla's name. And the reason I'm making this bet is because these already do get trotted out into such discussions.
(Not that I would ever read those reports or have done any such research into repressed Pons and Fleischmann evidence or Ilithid conspiracies)
Honestly that sounds a bit like a good thing to me?
I've spent a lot of time looking into the Epicureans being right about so much thousands of years before those ideas resurfaced again despite not having the scientific method, and their success really boiled down to the analytical approach of being very conservative in dismissing false negatives or embracing false positives - a technique that I think is very relevant to any topics where experimental certainty is evasive.
If there is a compelling case for dragons, maybe we should also be applying it to gnomes and unicorns and everything else we can to see where it might actually end up sticking.
The belief that we already have the answers is one of the most damaging to actually uncovering them when we in fact do not.
What does "agnostic" mean, operationally? I have trouble thinking you mean it in the direct sense (unknowable and not subject to testing), but maybe I'm wrong. For myself, I'm not agnostic, I'm an unbeliever - I have a reasonably confident low estimate of the probability that dragons exist, in the common conceptions of dragons and existence.
That said, I don't spend a lot of time thinking or discussing the topic, and I am perfectly happy to nod and ignore people who think it's important (in either direction). My private beliefs are somewhat decoupled from my public advocacy. For many of the more rabid pro-dragon proselytizers, it's easier to get them out of my way if I say I'm agnostic, but that doesn't make it so, and I generally don't have to do that on LessWrong.
1Operationally it means that I'm not trying to find out the truth one way or the other. If I come across arguments I ignore them, if someone asks if they can explain it to me I say no, I try not to think about it, etc.
"Agnostic" doesn't necessarily mean "unknowable and not subject to testing". Much more often it has the weaker meaning "not currently known". There is a house being built across the street. Is there a work van parked in front of it right now? I don't know. This is certainly knowable and subject to testing - I could get up, walk over to a window in the front of the house, and look. But I don't care enough to do that, so I continue to now know if there is a work van parked in front of the house across the street. I am agnostic about the existence of such a work van.
You could look into whether dragons exist with the plan that you will never reveal any findings no matter what they are. I get that you probably wouldn't bother because most paths by which that information could be valuable require you to leak it, but it's an option.
Some, at least, of these highly politically partisan hot-button issues have the property that most people don;t have a reason for caring whether they're true or not. In which cases, shrug might be the reasonable response.
Possibly the idea of this thread is that we're not supposed to mention any real examples, go avoid gettin g caught up in culture wars.
I can think of examples where even if I am going to do 9or not do) something based on whether the claim is trur of not... (a) the risk of doing X if claim is true seems small'(b) the cost of doing X if the claim is false is small (c) evidence for or against the claim looks really weak. So, shrug.
e.g. (true story) I am in the emergency room with tachycardia caused by graves disease. One of the ER docs and the endocrinologist have a really deeply technical argument over whether administration of Hartmann;s solutipn via IV is a plus or minus in my situation. Listening go this, I gather that, really, there is not much in it. Nurse would like to know if she can go ahead and stick the IV drip in my arm. Shrug. Whatever. You have my patient consent for that procedure, yes.
Example chosen because (a) I have really, absolutely no idea what the optimal action is here; (b) I have reason to belive that the risk is kind of low, anyway. (c) most commentors here wont have a strong opinion, so we wont have a flame war over what the right answer might be. Let it stand in for other cases where it is very, very unclear what the optimal action is.
This analogy might not work for all the things "dragons" is standing in for in this thread ... but if I have a good statistical bound on the risk posed by dragons being low (but cannot, strictly speaking, rule out their existence entirely) I may conclude that a residual 1E-5 chance of running in to one to be a acceptable risk.
So if I see verified reports of AI causing a mass casualty incident with more that %500 million in damage (or whatever the threshold in the California bill is), I shall consider that evidence on a par to seeing Lake-Town get toasted by Smaug, and update accordingly.
As long as you don't also claim to be "truth-seeking" in any way, this form of intellectual cravenness is probably better than what most people do, which is just to adopt whatever belief is most convenient given their social circles.
A general commitment to seeking truth doesn't obligate one to investigate every possible question! I can be quite committed to seeking truth in some areas, while intentionally avoiding quite unrelated ones.
One certainly shouldn't claim to be truth seeking in areas where one is are intentionally agnostic, but that's part of why I'm writing this post: so I can later link it to explain why I have chosen not to engage with some question in some area.
1Still, it feels like there's an important difference between "happening to not look" and "averting your eyes".
I agree, but I strongly disagree with @Shankar Sivarajan that if a person does this in some areas then they shouldn't "claim to be 'truth-seeking' in any way".
This is probably true in an internal sense, where one needs to be self-honest. It might be very difficult to understand when any conscious person other than you was doing this, and it might be dicey to judge even in yourself. Especially given the finiteness of human attention.
In my personal life, I have spent recent months studying. Did I emotionally turn away from some things in the middle of this, so that to an outside observer I might have looked like I was burying my head or averting my eyes? Sure. Was I doing that or was I setting boundaries? I guess even if you lived in my head at that time, it could be hard to know. Maybe my obsessive studying itself is an avoidance. In the end, I know what I intended, but that's about it. That's often all we get, even from the inside.
So while I agree with you, I'm not sure exactly when we should cease to be agnostic about parsing that difference. Maybe it's something we can only hold it as an ideal, complimentary to striving for Truth, basically?
The solution to dragons, if they exist, is to shut up about it and solve the alignment problem.
This can be said about most of the things people like to argue about.
For people disagree-voting [edit: at the time the parent was disagree-voted to -7], I'd be happy to see arguments that I should switch from trying to detect bioengineered pandemics to alignment research.
I don’t know if you should switch offhand, but I notice this post of yours is pretty old and a lot has changed.
What’s your rough assessment of AI risk now?
If you haven’t thought about it explicitly, I think it’s probably worth spending a week thinking about.
Also, how many people work on your current project? If you left, would that tank the project or be pretty replaceable? (Or: if you left are there other people who might then be more likely to leave?).
What’s your rough assessment of AI risk now?
I think it's pretty important and I'm glad a bunch of people are working on it. I seriously considered switching into it in spring 2022 before deciding to go into biorisk
Also, how many people work on your current project? If you left, would that tank the project or be pretty replaceable?
We're pretty small (~7), and I've recently started leading our near-term first team (four people counting me, trying to hire two more). I think I'm not very replaceable: my strengths are very different from others on the team in a highly complementary way, especially from a "let's get a monitoring system up and running now" perspective.
(I must admit to some snark in my short response to Mako above. I'm mildly grumpy about people going around as if alignment is literally the only thing that matters. But that's also not really what he was saying, since he was pushing back against my worrying about dragons and not my day job.)
Nod. I had initially remembered the Superintelligence Risk Project being more recent than 2017, was there a 2022 writeup of your decisionmaking?
I think you should think about how your work generalizes between the topics, and try to make it possible for alignment researchers to take as much as they can from it; this is because I expect software pandemics are going to become increasingly similar to wetware pandemics, and so significant conceptual parts of defenses for either will generalize somewhat. That said, I also think that the stronger form of the alignment problem is likely to be useful to you directly on your work anyway; if detecting pandemics in any way involves ML, you're going to run into adversarial examples, and will quickly be facing the same collapsed set of problems (what objective do I train for? how well did it work, can an adversarial optimization process eg evolution or malicious bioengineers break this? what side effects will my system have if deployed?) as anyone who tries to deploy ML. If you're instead not using ML, I just think your system won't work very well and you're being unambitious at your primary goal, because serious bioengineered dangers are likely to involve present-day ML bio tools by the time they're a major issue.
But I think you in particular are doing something sufficiently important that it's quite plausible to me that you're correct. This is very unusual and I wouldn't say it to many people. (normally I'd just not bother directly saying they should switch to working on alignment because of not wanting to waste their time when I'm confident they won't be worth my time to try to spin up, and I instead just make noise about the problem vaguely in people's vicinity and let them decide to jump on it if desired.)
I'm agnostic on the existence of dragons. I don't usually talk about this, because people might misinterpret me as actually being a covert dragon-believer, but I wanted to give some background for why I disagree with calls for people to publicly assert the non-existence of dragons.
Before I do that, though, it's clear that horrible acts have been committed in the name of dragons. Many dragon-believers publicly or privately endorse this reprehensible history. Regardless of whether dragons do in fact exist, repercussions continue to have serious and unfair downstream effects on our society.
Given that history, the easy thing to do would be to loudly and publicly assert that dragons don't exist. But while a world in which dragons don't exist would be preferable, that a claim has inconvenient or harmful consequences isn't evidence of its truth or falsehood.
Another option would be to look into whether dragons exist and make up my mind; people on both sides are happy to show me evidence. If after weighing the evidence I were convinced they didn't exist, that would be excellent news about the world. It would also be something I could proudly write about: I checked, you don't need to keep worrying about dragons.
But if I decided to look into it I might instead find myself convinced that dragons do exist. In addition to this being bad news about the world, I would be in an awkward position personally. If I wrote up what I found I would be in some highly unsavory company. Instead of being known as someone who writes about a range of things of varying levels of seriousness and applicability, I would quickly become primarily known as one of those dragon advocates. Given the taboos around dragon-belief, I could face strong professional and social consequences.
One option would be to look into it, and only let people know what I found if I were convinced dragons didn't exist. Unfortunately, this combines very poorly with collaborative truth-seeking. Imagine a hundred well-intentioned people look into whether there are dragons. They look in different places and make different errors. There are a lot of things that could be confused for dragons, or things dragons could be confused for, so this is a noisy process. Unless the evidence is overwhelming in one direction or another, some will come to believe that there are dragons, while others will believe that there are not.
While humanity is not perfect at uncovering the truth in confusing situations, our strategy that best approaches the truth is for people to report back what they've found, and have open discussion of the evidence. Perhaps some evidence Pat finds is very convincing to them, but then Sam shows how they've been misinterpreting it. But this all falls apart when the thoughtful people who find one outcome generally stay quiet. I really don't want to contribute to this pattern that makes it hard to learn what's actually true, so in general I don't want whether I share what I've learned to be downstream from what I learn.
Overall, then, I've decided to remain agnostic on the existence of dragons. I would reconsider if it seemed to be a sufficiently important question, in which case I might be willing to run the risk of turning into a dragon-believer and letting the dragon question take over my life: I'm still open to arguments that whether dragons exist is actually highly consequential. But with my current understanding of the costs and benefits on this question I will continue not engaging, publicly or privately, with evidence or arguments on whether there are dragons.
Note: This post is not actually about dragons, but instead about how I think about a wide range of taboo topics.
Comment via: facebook, mastodon