Refusing to answer seems by far the worst. The general public aren't epistemology nerds, and won't appreciate that you care about factual accuracy in a way that's not fully compatible with giving a number; they're more likely to view it as some sort of lawerly cowardice.
"Weasel words" on the other hand have a long and storied history, the trouble is choosing ones which accurately reflect your true probabilities. There are some old NATO guides to this, and some modern updated ones (https://waf.cs.illinois.edu/visualizations/Perception-of-Probability-Words/), but it is all very inprecise.
I think giving a probability range is superior to all of the approaches you mentioned. It succinctly intuitively conveys uncertainty in the estimate.
My p(doom) has been 10-90% for some time. It's currently more like 20-90%
This still isn't ideal because it's conflating personal circumstantial uncertainty with irresolvable uncertainty, however you'd try to define that. My numbers could just as easily be saying that barely thought about the issue. They are actually saying I've thought lot about it, in the thousands of hours, and I still think the the knightian uncertainty is that high.
But it's a heck of a lot better than just a single number with both types of uncertainty left unspecified.
I liked your post. I'll add another approach that might be used (even though I'm not nearly as worried as most here and tend to think there will be more total suffering without some good AI and even ASI to work with than if it's prevented). I think in many ways it might be one some chafe agains.
Trying to solve any problem rather than relying on some heuristic seems to spin around two related aspect. Competency in analysis and complexity and difficulty of the problem. I think many here are very much in the "I optimize everything I can." camp -- so talking in probabilities and even going through the step to rationally come up with the one's they claim is just the nature of the minds. But when competence is way over whelmed by complexity optimization can be very dangerous. While not quit a correct way to put it, if you know certain actions can have really bad outcomes, but they are tail risks and you don't really know the process tail events are generated from but they do relate/correlate with the actions/choice you can reason towards don't. Don't reason, don't try to optimize your decision; follow a rule. (Not original to me, pretty much the argument in The Origins of Predictable Behavior written back in the mid-ish 1980s.)
So here, we don't really know what ASI actually is, other than some general descriptive view that boils down to "really fucking smart", nor does anyone really know exactly what might lead it to kill us all or otherwise so marginalize humans and human civilization that we have that competence-complex problem thing going on. So, one might suggest that if you want existing human civilization, and perhaps the human species to continue the rule "Don't build it." might be the one to follow.
Unfortunately, the other thing I took from the paper's argument was that the bad thing, being infrequence so not often seen and experienced, itself creates the incentives for people to think a rational, optimizing choice is actually a smart thing to do.
"Let me use esoteric language to survive being quoted out of context: I have one omnicidal nightmare for every 4 utopian dreams."
Lab Employee's statement is fine (it's qualified with "I think..."). Cope Merchant erred by assuming Lab Employee implied a hard scientific model.
The financial advisor example is skewed with a very specific point estimate (0.05%) implying hard scientific model precision. Most Lab Employees citing their P(Doom) use round numbers (<1%, 10%, 50%, etc.) implying something more arbitrary. Many explicitly disclaim the impreciseness of their point estimate.
"Quoting survery statistics instead" does the same thing Lab Employee's doing, only diffuses the blame/hubris across multiple people. Cope Merchant is not going to be satisfied by the median expert response when they're certain 'all AI safety people are fearmongering pretending they have a hard scientific model as part of their plan to [unknown]'.
'Weaseling' or 'Refusing to answer' seem worse than saying what you think/mean.
I strongly disagree with this post, and in particula I don't think the financial advisor example supports the conclusion. There is an asymmetry between the two situations.
If you say 0.05% for a worrying outcome, the reasonable response is to stop worrying and not dig into the details. If the listener trusts you and the event happens, they can rightly complain that you falsely reassured them.
On the other hand, if you say 20% risk of serious adverse outcome, the reasonable response is to look further into the subject and where the number comes from. Anyone who does that quickly finds there is no rigorous underlying model. They might complain that you wasted their time or scared them more than necessary, but the error is self-correcting in a way the 0.05% error isn't.
Sometimes the following interaction plays out in public forums:
I sympathize with (some of) the words that the cope merchant is saying here. While in our subculture (and possibly Silicon Valley) the report "I think there is a 20% chance that AI kills everyone" means something cousined to "it seems unlikely but far from impossible", that is not all it's conveying to a regular person. To most normies, an ML researcher at a lab sounds like some sort of expert. And an "expert" that states a probability about something within their domain, is not just communicating an opinion on an event's likelihood. There's an implication that the expert arrived at it by some well-understood procedure, because most of the time the probabilities a normie reads are derived in this way (i.e. in rain forecasts, surgery odds, polls.)
This implication affects how most listeners update on the result, and probably does for you in regular life too. For example, if your financial advisor told you that your portfolio had a 0.05% chance of going bust, and then later on claimed that they had just "guessed" and ran no actual calculations, you would probably feel rather angry. Even if they explained that it was truly their subjective probability at the time, the misleading context might have made your subjective probabilities a lot different.
I don't know any great ideas for solving this particular public communications problem in the short term. It's hard to convey the alarming thing that these researchers want to convey, at the precision they want to convey it. Some things you could do if you were inclined:
In the long term we are all dead, so I guess these are better than nothing.