= 27d567bc2d48d9f40e9ccc20ea370b15)
gjm comments on Could you tell me what's wrong with this? - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (64)
Yeah, I understand the frustration.
Still, if someone is capable of grasping the argument "Any kind of software failure in the control systems of a nuclear reactor could have disastrous consequences; the total amount of software required isn't too enormous; therefore it is worth going to great lengths, including formal correctness proofs, to ensure that the software is correct" then they're right to make that argument even if their grasp of what kind of software is used for controlling a nuclear reactor is extremely tenuous. And if they say "... because otherwise the reactor could explode and turn everyone in the British Isles into a hideous mutant with weird superpowers" then of course they're hilariously wrong, but their wrongness is about the details of the catastrophic disaster rather than the (more important) fact that a catastrophic disaster could happen and needs preventing.
That's absolutely true but it leads to two problems.
First: the obvious lack of experience/understanding of the nuts and bolts of it makes people from outside less likely to take the realistic parts of their warning seriously and may even lead to it being viewed as a subject of mockery which works against you.
Second: The failure modes that people suggest due to their lack of understanding can also be hilariously wrong like "a breach may be caused by the radiation giving part of the shielding magical superpowers which will then cause it to gain life and open a portal to R'lyeh" and they may even spend many paragraphs on talking about how serious that failure mode it while others who also don't actually understand politely aplaude. This has some of the same unfortunate side effects: it makes people who are totally unfamiliar with the subject less likely to take the realistic parts of their warning seriously.