Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Tyrrell_McAllister2 comments on LA-602 vs. RHIC Review - Less Wrong

30 Post author: Eliezer_Yudkowsky 19 June 2008 10:00AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (61)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Tyrrell_McAllister2 19 June 2008 05:22:47PM 0 points [-]

What makes you think they don't?

I acknowledge that they probably do so with some nonzero number of projects. But I take Eliezer to be advocating that it happen with all projects that carry existential risk. And that's not happening; otherwise Eliezer wouldn't have had the example of the RHIC to use in this post. Now, perhaps, notwithstanding the RHIC, the government already is classifying nearly all basic science research that carries an existential threat, but I doubt it. Do you argue that the government is doing that? Certainly, if it's already happening, then I'm wrong in thinking that that would be prohibitively difficult in a democracy.

1) Existential risk is real and warrants government-funded research.


2) The results, if useful, would not be a sexed-up dodgy dossier, but a frank, measured appraisal of ER. As such, they would acknowledge a nonzero risk to humanity from sundry directions.


3) As such, they would be blown out of proportion by media reporting, as per Eliezer's analysis. This isn't certain, but it's highly likely. "Gov't Report: The End Is Nigh For Humankind!"

Some media would react that way. And then some media would probably counter-react by exaggerating whatever problem warranted the research in the first place. Consider, e.g., the torture of detainees in the "war on terror". Some trumpet the threat of a terrorist nuke destroying a city if we don't use torture to prevent it. Others trumpet the threat of our torture creating recruits for the terrorists, resulting in higher odds of a nuke destroying a city.

4) Natural conclusion - keep it classified, at least for the time being.

It's far from obvious to me that, in the example of torture, the best solution is to keep the practice classified. Obviously the practitioners would prefer to keep it that way. They would prefer that their own judgment settled the matter. I'm inclined to think that, while their judgment is highly relevant, sunlight would help keep them honest.

Analogously, I think that sunlight improves the practice of science. Not in every case, of course. But in general I think that the "open source" nature of science is a very positive aspect. It is a good ideal to have scientists expecting that their work will be appraised by independent judges. I realize that I'm going along with the conventional wisdom here, but I'm still a number of inferential steps away from seeing that it's obviously wrong.

As the piece above says, public & media reaction tends to be 100% positive or 100% negative. If you think you can talk the world out of this, there's a Nobel Prize in it for you.

Can you elaborate on your claim that "public & media reaction tends to be 100% positive or 100% negative"? Do you mean that, on most risky projects, the entire public, and all the media, react pro or con in total unanimity? Or do you mean that, on most issues of risk, each individual and each media outlet either supports or opposes the project 100%? Or do you mean something else? I'll respond after your clarification.

I should emphasize that I'm not arguing for using direct democracy to determine what projects should be funded. The process to allocate funding should be structured so that the opinions of the relevant experts are given great weight, even if the majority of the population disagrees. What I'm skeptical about is the need for secrecy.