Your hypothetical is a good one. And you are correct: I don't think you are dishonest if you are sincerely trying to build or sell a perpetual motion machine. You're still wrong, and even silly, but not dishonest. I need a word to refer to conscious knowing deception, and "dishonest" is the most useful word for the purpose. I can't let you use it for some other purpose; I need it where it is.
The argument is not applicable to all criminal conduct. In American criminal law, we pay a lot of attention to the criminal's state of mind. Having the appropriate criminal state of mind is an essential element of many crimes. It's not premeditated murder if you didn't expect the victim to die. It's not burglary if you thought you lived there. It's utterly routine -- and I think morally necessary -- to ask juries "what what the defendant's intention or state of mind". There is a huge moral and practical difference between a conscious and an unconscious criminal. Education much more easily cures the latter, while punishment is comparatively ineffective. For the conscious criminal, the two are reversed: punishment is often appropriate, whereas education has limited benefits.
I don't believe I am giving liars a get-out-of-jail-free card. Ignorance isn't an unlimited defense, and I don't think it is so easy to convince an outside observer (or a jury) that you're ignorant in cases where knowledge would be expected. If you really truly are in a state of pathological ignorance and it's a danger to others, we might lock you up as a precaution, but you wouldn't be criminally liable.
As to scientific ethics: All human processes have a non-zero chance of errors. The scientists I know are pretty cynical about the process. They are fighting to get papers published and they know it. But they do play by the rules -- they won't falsify data or mislead the reader. And they don't want to publish something if they'll be caught-out having gotten something badly wrong. As a result, the process works pretty much OK. It moves forward on average.
I think SIAI is playing by similar rules. I've never seen them caught lying about some fact that can be reliably measured. I've never seen evidence they are consciously deceiving their audience. If they submitted their stuff to a scientific publication and I were the reviewer, I might try to reject the paper, but I wouldn't think of trying to have them disciplined for submitting it. In science, we don't accuse people of misconduct for being wrong, or pigheaded, or even for being overly biased by your self-interest. Is there some more serious charge you can bring against SIAI? How are they worse than any scientist fighting for a grant based on shakey evidence?
I think you have somewhat simplistic idea of justice... there is the "voluntary manslaughter", there's the "gross negligence", and so on. I think SIAI falls under the latter category.
How are they worse than any scientist fighting for a grant based on shakey evidence?
Quantitatively, and by a huge amount. edit: Also, the of beliefs, that they claim to hold, when hold honestly, result in massive loss of resources such as moving to cheaper country to save money, etc etc. I dread to imagine what would happen to me if I honestly were this...
I've spent so much time in the cogsci literature that I know the LW approach to rationality is basically the mainstream cogsci approach to rationality (plus some extra stuff about, e.g., language), but... do other people not know this? Do people one step removed from LessWrong — say, in the 'atheist' and 'skeptic' communities — not know this? If this is causing credibility problems in our broader community, it'd be relatively easy to show people that Less Wrong is not, in fact, a "fringe" approach to rationality.
For example, here's Oaksford & Chater in the second chapter to the (excellent) new Oxford Handbook of Thinking and Reasoning, the one on normative systems of rationality:
Is it meaningful to attempt to develop a general theory of rationality at all? We might tentatively suggest that it is a prima facie sign of irrationality to believe in alien abduction, or to will a sports team to win in order to increase their chance of victory. But these views or actions might be entirely rational, given suitably nonstandard background beliefs about other alien activity and the general efficacy of psychic powers. Irrationality may, though, be ascribed if there is a clash between a particular belief or behavior and such background assumptions. Thus, a thorough-going physicalist may, perhaps, be accused of irrationality if she simultaneously believes in psychic powers. A theory of rationality cannot, therefore, be viewed as clarifying either what people should believe or how people should act—but it can determine whether beliefs and behaviors are compatible. Similarly, a theory of rational choice cannot determine whether it is rational to smoke or to exercise daily; but it might clarify whether a particular choice is compatible with other beliefs and choices.
From this viewpoint, normative theories can be viewed as clarifying conditions of consistency… Logic can be viewed as studying the notion of consistency over beliefs. Probability… studies consistency over degrees of belief. Rational choice theory studies the consistency of beliefs and values with choices.
They go on to clarify that by probability they mean Bayesian probability theory, and by rational choice theory they mean Bayesian decision theory. You'll get the same account in the textbooks on the cogsci of rationality, e.g. Thinking and Deciding or Rational Choice in an Uncertain World.