This is a pretty interesting discussion. While the overall topic of this blog is the worthwhileness of overcoming bias, I think how that relates to "evil" is a pretty important facet of what we should talk about. Some of the comments on this post reminded me of a passage from a novel (a fantasy novel if you must know) on the nature of evil. I found it to be very profound and I think everyone here might find it sort of interesting. You'll have to excuse the use of some of the plot specific names, the beginning of each chapter of the book opens with a few paragraphs from the point of view of a historian who lives several thousand years after the events in the story take place.

"What is the nature of evil? Evil is mistakenly portrayed as a unified force. Tresserhorn, Stromgald, the Adnates of Soldevi, Varchild, all of these, we are told, were cut from the same cloth and share that same corrupted fabric. Yet these forces have battled each other as often as they have fought the forces of “good” and so can hardly be considered members of the same side. Some researches state that all who bear the stamp of evil are imperfect or faulty beings from the onset. The more generous scholar might classify them as being merely weak, corrupted by potent forces that poisoned them. Yet in the end, the historians label them all as “evil,” as if that state were a disease that could be contracted through mere proximity. So goes traditional thought. In reality, the various “forces of evil” all chose their paths for reasons that sounded good, even beneficial, at the time. They might have chosen their paths for personal advancement, for a particular belief or religion, for the desire of power, or even for patriotism or devotion. What unified them all, from Lim-Dûl to the Adnates, was usually the belief that they had “the answer” to whatever problem beset them. This answer often had the advantage of requiring minimal sacrifice on the part of the solver. Further, when the solution did require some degree of sacrifice, that sacrifice would be on the part of others who were unaware of a need for sacrifice in the first place. These other people, the ones making the sacrifices, often took umbrage when they learned of their situation. As a result, such unpleasant information was kept from them until the last moment. In this fashion, the first steps of deception, regardless of the reason, was evil born."

I tend to agree with what the author of that passage said about evil. When you look at people like Hitler, Mao, and Stalin it is clear they believed they had "the answer" or "the solution" to the problems they faced. But they lacked the ability to objectively see that their plans called for suffering on levels few people alive today can fathom (I know I certainly can't). They let their biases close their eyes to the deaths they caused because they simply had to be right in their own mind. Teaching everyone to be compassionate and to empathize with others will certainly help people become less cruel, but overcoming our biases is no less important a step.

Haha, that's a pretty good analogy. Unfortunately I think most people (myself in the past included and probably even still now) by default have their mouse cursor hovering over wherever the Ignore or Worship buttons appear when such a dialog shows up. And they click it in much the same way my grandparents would click a popup that installs malware on their computer, without thinking or paying attention. Clicking the Explain button requires effort (moving your cursor to a different spot and then waiting for an explanation), and knowing that it will bring up another dialog sooner or later makes it easier for people to just press Ignore or Worship.

I hope I'm not misunderstanding Hopefully Anonymous' question here, but it would seem to me that a society whose laws are based solely on rationally minimizing harm would have to be a society with little or no freedom. I guess it depends on what is defined as the maximum risk the society is willing to endure to the physical safety of the individuals (and I presume the structure of the society as a whole). I think Eliezer certainly hinted at what I feel the main problem would be with a society whose laws are made to counteract individual irrationality. In doing so you would have to take away individual choice for the most part. That can be argued as a good thing to a point perhaps. But if you follow the goal of reduction of risk of harm far enough (which it would most certainly be rational to do so if that is your goal) you'd end up in a place where no one can do much of anything because nearly every activity we engage in on a daily basis has some amount of risk associated with it.

Driving a car for example would most certainly have to be done away with then, your risk of serious injury or death is pretty good, and the amount of overall harm to society in terms of wages lost, damage to property, and productive lives lost is quite high. Although I doubt the benefits to the society as a whole of driving would not outweigh the costs, so in such a society perhaps it would be allowed.

I'm not sure I understand how such a system could come about extragovernmentally. Could you elaborate? While social norms certainly appear outside of government influence, laws are a function of government and really only carry weight because the government has the right (or at least ability) to enforce them. For your example of compulsory medical trials (or anything similar to that) I think it would take a very drastic changing of social norms to make such a thing work without a government forcing people to take part in a medical experiments.

I did not mean to write this much at all. But it's an interesting topic for sure and it's not one I'm well educated on unfortunately. The above was just my initial reaction to the question.