Review
Review
Dragon Agnosticism
70Richard_Ngo
27habryka
8Richard_Ngo
49Carl Feynman
1flowerfeatherfocus
9Lukas Finnveden
33Richard_Ngo
1flowerfeatherfocus
5Jiro
2Richard_Ngo
5MichaelDickens
3flowerfeatherfocus
56AnnaSalamon
14[anonymous]
6AnnaSalamon
2jefftk
4AnnaSalamon
4jefftk
7Ben Pace
4jefftk
6Ben Pace
2jefftk
5Ben Pace
19TsviBT
2mako yass
15[anonymous]
12Nathan Helm-Burger
12NoSignalNoNoise
1[anonymous]
14ShardPhoenix
14Bucky
11xpym
4jefftk
12faul_sname
21jefftk
8faul_sname
10lc
2jefftk
5lc
9Orborde
9jefftk
6mike_hawke
5cousin_it
4kromem
2Jiao Bu
2kromem
4Dagon
18jefftk
11River
-6Sweetgum
17followthesilence
9Sweetgum
4Shankar Sivarajan
14jefftk
17Adele Lopez
9jefftk
6Jiao Bu
2Orborde
1Michael Roe
1Michael Roe
1Michael Roe
1Michael Roe
0mako yass
14jefftk
30jefftk
4Raemon
14jefftk
5Raemon
2the gears to ascension
1Liriodendron
5mako yass
-2Nathan Young
16Raemon
15Thomas Kwa
1Purple bus
New Comment
I think this used to be a tenable position a decade or two ago. But I think it's no longer tenable, due to the dynamic described in this tweet:
Suppose an ideology says you're not allowed to question idea X. At first X might not be very important. But now when people want to argue for Y, "X->Y" and "~Y->~X" are both publicly irrefutable. So over time X will become more and more load-bearing for censorious ideologies.
We can also think of this as a variant of Goodhart's law, which I'll call ideological Goodhart (and have just tweeted about here): any false belief that cannot be questioned by adherents of an ideology will become increasingly central to that ideology. As this process plays out, advocates of that ideology will adopt increasingly extreme positions, and support increasingly crazy policies.
3
2
1(Disagree that it was a tenable position a decade or two ago, agree that it is an untenable position now)
8
Fair, this isn't a confident claim from me. I do have a sense that the last decade has been particularly bad in terms of blatant preference falsification, but it's hard to distinguish "the world was different before then" from "I was younger and didn't have a great sense of what was going on".
I’m 62, so I was adult in the ‘80s and ‘90s. My sense is that the world was different. The consequences of expressing divergent opinion seem much more serious now.
41
Can you say what position you recommend instead? Is it just opining publicly about everything, with no regard to how taboo it is?
9
Tbc: It should be fine to argue against those implications, right? It’s just that, if you grant the implication, then you can’t publicly refute Y.
Unfortunately the way that taboos work is by surrounding the whole topic in an aversive miasma. If you could carefully debate the implications of X, then that would provide an avenue for disproving X, which would be unacceptable. So instead this process tends to look more like "if you don't believe Y then you're probably the sort of terrible person who believes ~X", and now you're tarred with the connotation even if you try to carefully explain why you actually have different reasons for not believing Y (which is what you'd likely say either way).
1
I expect this effect to be weaker than you're suggesting, especially if Y is something you in fact independently care about, and not an otherwise unimportant proximal detail that could reasonably be interpreted as a "just asking questions" means of arguing for ~X. I'm struggling to think of a particularly illustrative X and Y, but consider X="COVID was not a lab leak", which seemed lightly taboo to disagree with in 2020. Here's a pair of tweets you could have sent in 2020:
1. "I think COVID was probably a lab leak."
2. "I don't know whether COVID was a lab leak. (In fact for now I'm intentionally not looking into it, because it doesn't seem important enough to outweigh the risk of arriving at taboo beliefs.) But gain-of-function research in general is unacceptably risky, in a way that makes global pandemic lab leaks a very real possibility, and we should have much stronger regulations to prevent that."
I expect the second one would receive notably less push back, even though it defends Y="gain of function research is unacceptably risky", and suggests that Y provides evidence for ~X.
5
I don't see why it should be limited to false beliefs.
Note that even if X is true, X->Y need not be true, and it can still be harmful to not be able to question X->Y.
2
This is a good point. Though the thing about true beliefs is that there is a specific version of them that's true, which you're allowed to defend (if you can find it). And so you can more easily figure out what the implications are.
Whereas for false beliefs you can't get into the specifics, because looking hard enough at the specifics will tend to disprove the belief.
5
I think there are two different kinds of "not questioning": there's unquestioningly accepting an idea as true, and there's refusing to question and remaining agnostic. The latter position is reasonable in the sense that if you refuse to investigate an issue, you shouldn't have any strong beliefs about it. And I think the load-bearingness is only a major issue if you refuse to question X while also accepting that X is true.
3
Is there another strategy you prefer? Afaict the options are
1) Have public taboo beliefs.
2) Have private beliefs that you lie about.
3) Remain deliberately agnostic about taboo but insufficiently important topics.
4) Get forever lucky, such that every taboo topic you investigate results in you honestly arriving at an allowed belief.
Whether 1) is at all compatible with having other career goals is a fact of the territory, and I expect in the US in 2024, there are topics where having taboo beliefs could totally end your career, for many values of career. (Leaving open whether there are such beliefs that are true, but, per the topic of this post, that's not something you can learn without taking risks.)
2) seems even more prone to the effect you describe than 3).
My guess is you're making a bid for 1), but I feel like a case for that should take into account the costs of believing X weighed against the costs of agnosticism about X, rather than a sweeping heuristic argument. (Where maybe the cost of agnosticism about X includes adjacent topics Y you'll either have to include in your agnosticism or otherwise eat the cost of ~X connotations, though I'm skeptical about how often this will come up, and per my comment here I expect the ~Y->~X taboos will often be much smaller than the ~X taboo.)
I don't see advantage to remaining agnostic, compared to:
1) Acquire all the private truth one can.
Plus:
2) Tell all the public truth one is willing to incur the costs of, with priority for telling public truths about what one would and wouldn't share (e.g. prioritizing to not pose as more truth-telling than one is).
--
The reason I prefer this policy to the OP's "don't seek truth on low-import highly-politicized matters" is that I fear not-seeking-truth begets bad habits. Also I fear I may misunderstand how important things are if I allow politics to influence which topics-that-interest-my-brain I do/don't pursue, compared to my current policy of having some attentional budget for "anything that interests me, whether or not it seems useful/virtuous."
One of the advantages to remaining agnostic comes from the same argument that users put forth in the comment sections on this very site way back in the age of the Sequences (I can look up the specific links if people really want me to, they were in response to the Doublethink Sequence) for why it's not necessarily instrumentally rational for limited beings like humans to actually believe in the Litany of Tarski: if you are in a precarious social situation, in which retaining status/support/friends/resources is contingent on you successfully signaling to your in-group that you maintain faith in their core teachings, it simply doesn't suffice to say "acquire all the private truth through regular means and don't talk/signal publicly the stuff that would be most dangerous to you," because you don't get complete control over what you signal.
If you learn that the in-group is wrong about some critical matter, and you understand that in-group members realizing you no longer agree with them will result in harm to you (directly, or through your resources being cut off), your only option is, to act (to some extent) deceptively. To take on the role, QuirrellMort-style, of somebody who do...
6
Fair point; I was assuming you had the capacity to lie/omit/deceive, and you're right that we often don't, at least not fully.
I still prefer my policy to the OPs, but I accept your argument that mine isn't a simple Pareto improvement.
Still:
* I really don't like letting social forces put "don't think about X" flinches into my or my friends' heads; and the OPs policy seems to me like an instance of that;
* Much less importantly: as an intelligent/self-reflective adult, you may be better at hiding info if you know what you're hiding, compared to if you have guesses you're not letting yourself see, that your friends might still notice. (The "don't look into dragons" path often still involves hiding info, since often your brain takes a guess anyhow, and that's part of how you know not to look into this one. If you acknowledge the whole situation, you can manage your relationships consciously, including taking conscious steps to buy openness-offsets, stay freely and transparently friends where you can scheme out how.)
2
In many cases I have guesses, but because I just have vague impressions they're all very speculative. This is consistent with being able to say "I haven't looked into it" and "I really don't know", and because these are all areas where the truth is not decision relevant it's been easy to leave it at that. Perhaps people notice I have doubts, but at least in my social circles that's acceptable if not made explicit.
4
Does it feel to you as though your epistemic habits / self-trust / intellectual freedom and autonomy / self-honesty takes a hit here?
4
I think it's a pretty weak hit, though not zero. There are so many things I want to look into that I don't have time for that having this as another factor in my prioritization doesn't feel very limiting to my intellectual freedom.
I do think it is good to have a range of people in society who are taking a range of approaches, though!
7
Also, a norm of "allowing people to keep their beliefs private on subjects they feel a lot of pressure on" gives space for people to gather information personally without needing to worry about the pressures on them from their society.
4
I think that's an important norm and I support it, but until it is well established it's not something I (or others) can rely on.
6
It’s going pretty well for me! Most people I work with or am friends with know that there are multiple topics on which my thoughts are private, and there have been ~no significant social costs to me that I’m aware of.
I would like to be informed of opportunities to support others in this on LessWrong or in the social circles I participate in, to back you up if people are applying pressure on you to express your thoughts on a topic that you don’t want to talk about.
2
Nice of you to offer! I expect, however, that pressure in this direction will come from non-LW non-EA directions.
5
Then I shall continue to tend to and grow my garden.
One option would be to look into it
Another option you didn't list is to look into why horrible acts have been committed in the name of dragons, and share the results of that.
2
Mm, perhaps it will turn out that there are other, more benign ways of dealing with dragons. Narrative-brained people wont find those.
I really don't want to contribute to this pattern that makes it hard to learn what's actually true, so in general I don't want whether I share what I've learned to be downstream from what I learn.
Another policy which achieves this is to research the question, and not (publicly) share your conclusion either way. This also benefits you in the case you become a dragon believer, because glomarizing (when you're known to follow this policy) provides no evidence you are one in that case. (Things this reminds me of: meta-honesty, and updatelessness when compared to your position)
Another policy which achieves this is to share your conclusion only if you end up disbelieving in dragons, but also hedge it that you wouldn't be writing if you believed the other position. If you're known to follow this policy and glomarize about whether you believe in dragons, it is evidence that either you do or you haven't researched the question.
The trouble with this is that it's a socially awkward move to even imply you might research taboo topics. Better to leave public personna and tricky glomarization out of it, I think. Just publish your research and results anonomously. That seems to me to lead to a better epistemic state for society, since a standard of anonymous publication doesn't leave a misleading bias in publicly available research.
1To pick an uncontroversial example, imagine someone glomerizing in whether the Earth was flat or (approximately) spherical. That would signal that you're the sort of person who considered a spherical Earth to be a plausible hypothesis, which is almost as bad as actually believing it. All reasonable, right-thinking people, on the other hand, know that it's obviously flat and wouldn't even consider such nonsense.
1[anonymous]
This is why it's important for the policy be known for the glomarization to be evidence under that policy specifically, which might include something to the effect of "I follow this even in obvious cases so I'm free to also follow it in cases which are mistakenly framed as obvious".
That said, I'm not thinking about the 'mundane' world as Eliezer calls it, where doing this at all would be weird. I guess I'm thinking about the lesswrong blogosphere.
(There's a hypothetical spectrum from [having a glomarization policy at all is considered weird and socially-bad] to [it is not seen negatively, but you're not disincentivized from sharing non-exfohazardous beliefs, to begin with])
This seems mostly fine for anyone who doesn't engage in political advocacy or activism, but a mild-moderate form of defection against society if you do - because if dragons are real, society should probably do something about that, even if you personally can't.
edit: I guess dragon-agnosticism is tolerable if you avoid advocating for (and ideally voting for) policies that would be disastrous if dragons do in fact exist.
Do dragon unbelievers accept this stance? My impression is that dragon agnosticism would often be considered almost as bad as dragon belief.
They don't, of course, but if you're lucky enough not to be located among the more zealous of them and be subjected to mandatory struggle sessions, their wrath will generally be pointed at more conspicuous targets. For now, at least.
4
No one has given me a hard time about it. I say things like "I haven't looked into it" and we move on. The next time it happens I will additionally be able to link to this post.
It's fun to put various of my foundational non-hot-button-political beliefs in place of "dragons" and see which ones make my mind try to flinch away from thinking that thought for the reasons outlined in this post (i.e. "if I checked and it's not the way I thought, that would necessitate a lot of expensive and time-consuming updates to what I'm doing").
the reasons outlined in this post (i.e. "if I checked and it's not the way I thought, that would necessitate a lot of expensive and time-consuming updates to what I'm doing").
Wait, that's not what I'm trying to communicate in the post. If learning that dragons existed would precipitate major updates, it will very often be worth investigating their existence. Instead, it is on highly-political low-payoff topics that I am intentionally agnostic.
8
Concrete example: one of my core beliefs is " :::systems which get their productive capacity mainly from voluntary trade are both more productive and better for the people living under them than systems which get their productive capacity mainly from threats and coercion::: ". In this example, a "dragon" would be :::a coercion-based system which is more productive than a voluntary-trade-based one::: .
So working through the analogy:
* Historically, people who believed in this kind of "dragon", and who acted on that belief, tended not to behave very nicely.
* There are still people who believe in "dragons". They tend to valorize the actions of past dragon believers to a worrying extent.
* I think "dragons" probably don't exist, but I haven't actually proven it.
* I would prefer to live in a world where "dragons" don't exist.
* If "dragons" did exist, I would feel obligated to spend a lot of time and effort reevaluating my world model and plans.
* If I discovered the existence of a "dragon", and shared that, that would become a defining thing I am known for
* Most likely, there wouldn't be much that I, personally, could do with the information that a "dragon" exists
I guess arguably this is a hot-button political issue in some contexts. The other example I was thinking of was less political had the similar shape of "thing which, if true implies that we can'texpect to maintain certain nice things about the world we live in, and where people believing that the thing is true, if it is in fact true, would hasten the end of the nice things, to the benefit of nobody in particular".
ETA: On reflection I do agree that this is a different flavor of "nothing good can come of this line of thought" than the one you outlined in your post.
2
Say more?
5
So one of the themes of sequences is that deliberate self-deception or thought censorship - deciding to prevent yourself from "knowing" or learning things you would otherwise learn - is almost always irrational. Reality is what it is, regardless of your state of mind, and at the end of the day whatever action you're deciding to take - for example, not talking about dragons - you could also be doing if you knew the truth. So when you say:
It's not a reason not to investigate. You could continue to avoid these consequences you speak of by not writing about Dragons regardless of the results of your investigation. One possibility is that what you're also avoiding, is guilt/discomfort that might come from knowing the truth and remaining silent. But through your decision not to investigate, the world is going to carry the burden of that silence either way.
Another theme of the sequences is that self-deception, deliberate agnosticism, and motivated reasoning are a source of surprising amounts of human suffering. Richard explains one way it goes horribly wrong here. Whatever subject you're talking about, I'm sure there a lot of other people in your position who have chosen not to look into it for the same reasons. But if all of those people had looked into it, and faced whatever conclusion that resulted squarely, you yourself might not be in the position of having to face a harmful taboo in the first place. So the form of information hiding you endorse in the post is self-perpetuating, and is part of what helps keep the taboo strong.
It already seems like we can infer that dragon-existence has, to you, nontrivial subjective likelihood because you don't loudly proclaim "dragons don't exist" and because you regard investigation as uncomfortably likely to turn you into a believer of something socially unacceptable.
If you think it's in fact, like, 20% likely (a reasonable "nontrivial likelihood" guess for people to make), seems like the angry dragons-don't-exist people should be 20% angry at you.
9
I don't think inferring a probability anywhere near as high as 20% is justified. If, conditional on finding dragons the value of the knowledge is lower than the harms of being known as a dragon believer, then you shouldn't go check no matter how low your prior.
I am agnostic about various dragons. Sometimes I find myself wondering how I would express my dragon agnosticism in a world where belief in dragons was prevalent and high status. I am often disturbed by the result of this exercise. It turns out that what feels like agnosticism is often sneakily biased in favor of what will make me sound better or let me avoid arguments.
This effect is strong enough and frequent enough that I don't think the agnosticism described by this post is a safe epistemic fallback for me. However, it might still be my best option in s...
1I think even one dragon would have a noticeable effect on the population of large animals in the area. The huge flying thing just has to eat so much every day, it's not even fun to imagine being one. If we invent shapeshifting, my preferred shape would be some medium-sized bird that can both fly and dive in the water, so that it can travel and live off the land with minimal impact. Though if we do get such technology, we'd probably have to invent territorial expansion as well, something like creating many alternate Earths where the new creatures could live.
3
2
1I think you'll find that no matter what you find out in your personal investigation of the existence of dragons, that you need not be overly concerned with what others might think about the details of your results.
Because what you'll invariably discover is that the people that think there are dragons will certainly disagree with the specifics about dragons you found out that disagrees with what they think dragons should be, and the people that think there aren't dragons will generally refuse to even seriously entertain whatever your findings are relating t...
2
It's also possible that an opposing effect happens where your shouting into the void about dragons connects in some vague way with my belief in the Ilithids, which I then end up coopting your dragon evidence into my own agenda. Especially if you find anything close to material evidence. Heck, your material evidence for dragons now gives all kinds of creedance to Ilithids, beholders, gnomes, and all sorts. So the gnome people and everyone else is now coming out of the woodwork to amplify your work on dragons. And I think this would be regardless of the specific nuances your attribute to dragons. I would expect those nuances to get smooshed in the fray to cite your once-and-for-all-proving-dragons strategy.
I mean, if Pons and Fleischmann was true, for example, I bet it would get trotted out with all kinds of half-baked theories on free energy, along with Tesla's name. And the reason I'm making this bet is because these already do get trotted out into such discussions.
(Not that I would ever read those reports or have done any such research into repressed Pons and Fleischmann evidence or Ilithid conspiracies)
2
Honestly that sounds a bit like a good thing to me?
I've spent a lot of time looking into the Epicureans being right about so much thousands of years before those ideas resurfaced again despite not having the scientific method, and their success really boiled down to the analytical approach of being very conservative in dismissing false negatives or embracing false positives - a technique that I think is very relevant to any topics where experimental certainty is evasive.
If there is a compelling case for dragons, maybe we should also be applying it to gnomes and unicorns and everything else we can to see where it might actually end up sticking.
The belief that we already have the answers is one of the most damaging to actually uncovering them when we in fact do not.
What does "agnostic" mean, operationally? I have trouble thinking you mean it in the direct sense (unknowable and not subject to testing), but maybe I'm wrong. For myself, I'm not agnostic, I'm an unbeliever - I have a reasonably confident low estimate of the probability that dragons exist, in the common conceptions of dragons and existence.
That said, I don't spend a lot of time thinking or discussing the topic, and I am perfectly happy to nod and ignore people who think it's important (in either direction). My private beliefs are somewha...
1Operationally it means that I'm not trying to find out the truth one way or the other. If I come across arguments I ignore them, if someone asks if they can explain it to me I say no, I try not to think about it, etc.
"Agnostic" doesn't necessarily mean "unknowable and not subject to testing". Much more often it has the weaker meaning "not currently known". There is a house being built across the street. Is there a work van parked in front of it right now? I don't know. This is certainly knowable and subject to testing - I could get up, walk over to a window in the front of the house, and look. But I don't care enough to do that, so I continue to now know if there is a work van parked in front of the house across the street. I am agnostic about the existence of such a work van.
-6
As long as you don't also claim to be "truth-seeking" in any way, this form of intellectual cravenness is probably better than what most people do, which is just to adopt whatever belief is most convenient given their social circles.
A general commitment to seeking truth doesn't obligate one to investigate every possible question! I can be quite committed to seeking truth in some areas, while intentionally avoiding quite unrelated ones.
One certainly shouldn't claim to be truth seeking in areas where one is are intentionally agnostic, but that's part of why I'm writing this post: so I can later link it to explain why I have chosen not to engage with some question in some area.
1Still, it feels like there's an important difference between "happening to not look" and "averting your eyes".
9
I agree, but I strongly disagree with @Shankar Sivarajan that if a person does this in some areas then they shouldn't "claim to be 'truth-seeking' in any way".
6
This is probably true in an internal sense, where one needs to be self-honest. It might be very difficult to understand when any conscious person other than you was doing this, and it might be dicey to judge even in yourself. Especially given the finiteness of human attention.
In my personal life, I have spent recent months studying. Did I emotionally turn away from some things in the middle of this, so that to an outside observer I might have looked like I was burying my head or averting my eyes? Sure. Was I doing that or was I setting boundaries? I guess even if you lived in my head at that time, it could be hard to know. Maybe my obsessive studying itself is an avoidance. In the end, I know what I intended, but that's about it. That's often all we get, even from the inside.
So while I agree with you, I'm not sure exactly when we should cease to be agnostic about parsing that difference. Maybe it's something we can only hold it as an ideal, complimentary to striving for Truth, basically?
You could look into whether dragons exist with the plan that you will never reveal any findings no matter what they are. I get that you probably wouldn't bother because most paths by which that information could be valuable require you to leak it, but it's an option.
Some, at least, of these highly politically partisan hot-button issues have the property that most people don;t have a reason for caring whether they're true or not. In which cases, shrug might be the reasonable response.
Possibly the idea of this thread is that we're not supposed to mention any real examples, go avoid gettin g caught up in culture wars.
I can think of examples where even if I am going to do 9or not do) something based on whether the claim is trur of not... (a) the risk of doing X if claim is true seems small'(b) the cost of doin...
1
Example chosen because (a) I have really, absolutely no idea what the optimal action is here; (b) I have reason to belive that the risk is kind of low, anyway. (c) most commentors here wont have a strong opinion, so we wont have a flame war over what the right answer might be. Let it stand in for other cases where it is very, very unclear what the optimal action is.
1
This analogy might not work for all the things "dragons" is standing in for in this thread ... but if I have a good statistical bound on the risk posed by dragons being low (but cannot, strictly speaking, rule out their existence entirely) I may conclude that a residual 1E-5 chance of running in to one to be a acceptable risk.
1
So if I see verified reports of AI causing a mass casualty incident with more that %500 million in damage (or whatever the threshold in the California bill is), I shall consider that evidence on a par to seeing Lake-Town get toasted by Smaug, and update accordingly.
The solution to dragons, if they exist, is to shut up about it and solve the alignment problem.
This can be said about most of the things people like to argue about.
For people disagree-voting [edit: at the time the parent was disagree-voted to -7], I'd be happy to see arguments that I should switch from trying to detect bioengineered pandemics to alignment research.
4
I don’t know if you should switch offhand, but I notice this post of yours is pretty old and a lot has changed.
What’s your rough assessment of AI risk now?
If you haven’t thought about it explicitly, I think it’s probably worth spending a week thinking about.
Also, how many people work on your current project? If you left, would that tank the project or be pretty replaceable? (Or: if you left are there other people who might then be more likely to leave?).
What’s your rough assessment of AI risk now?
I think it's pretty important and I'm glad a bunch of people are working on it. I seriously considered switching into it in spring 2022 before deciding to go into biorisk
Also, how many people work on your current project? If you left, would that tank the project or be pretty replaceable?
We're pretty small (~7), and I've recently started leading our near-term first team (four people counting me, trying to hire two more). I think I'm not very replaceable: my strengths are very different from others on the team in a highly complementary way, especially from a "let's get a monitoring system up and running now" perspective.
(I must admit to some snark in my short response to Mako above. I'm mildly grumpy about people going around as if alignment is literally the only thing that matters. But that's also not really what he was saying, since he was pushing back against my worrying about dragons and not my day job.)
5
Nod. I had initially remembered the Superintelligence Risk Project being more recent than 2017, was there a 2022 writeup of your decisionmaking?
2
I think you should think about how your work generalizes between the topics, and try to make it possible for alignment researchers to take as much as they can from it; this is because I expect software pandemics are going to become increasingly similar to wetware pandemics, and so significant conceptual parts of defenses for either will generalize somewhat. That said, I also think that the stronger form of the alignment problem is likely to be useful to you directly on your work anyway; if detecting pandemics in any way involves ML, you're going to run into adversarial examples, and will quickly be facing the same collapsed set of problems (what objective do I train for? how well did it work, can an adversarial optimization process eg evolution or malicious bioengineers break this? what side effects will my system have if deployed?) as anyone who tries to deploy ML. If you're instead not using ML, I just think your system won't work very well and you're being unambitious at your primary goal, because serious bioengineered dangers are likely to involve present-day ML bio tools by the time they're a major issue.
But I think you in particular are doing something sufficiently important that it's quite plausible to me that you're correct. This is very unusual and I wouldn't say it to many people. (normally I'd just not bother directly saying they should switch to working on alignment because of not wanting to waste their time when I'm confident they won't be worth my time to try to spin up, and I instead just make noise about the problem vaguely in people's vicinity and let them decide to jump on it if desired.)
1
Congrats, we've just solved alignment! The AI now wants to know whether we want our utopia optimized in a way that accounts for dragons or not.
5
It's a lot easier to work through that after you have superintelligent AI. (Everything is.)
I would perhaps prefer we had a list of three things we don't discuss (say Politics, Race science and Infohazards) and if we want to not discuss a new thing we have to allow discussion of one of those others. Seems better to be clear what isn't being discussed.
See also Heads I Win, Tails?—Never Heard of Her; Or, Selective Reporting and the Tragedy of the Green Rationalists. Which goes into how people who don't know that there's some implicit consensus not to talk about some things can come away confused and with wrong beliefs.
I'm pro being clear about what we don't discuss, but it's unreasonable to limit the list to three. The number of topics that is net negative to discuss is just a fact about the world and is probably over three, and I would rather not have people talk about the 4th worst controversial topic just because we uncover three even more pointless and controversial ones.
Politics also seems inadvisable to ban because it's too broad.
1
I think implementing this policy would require frequent publicization of what issues are on the current list and frequent debates and votes on whether to replace one topic on the list with another topic not yet on the list and would therefore have the opposite of the intended effect. Maybe there's a clever way to get around those issues, but I doubt it.