r/HPMOR on heroic responsibility

Eliezer Yudkowsky

r/HPMOR readers on heroic responsibility - not the OP, the comments. Holy snorkels this is good.

It seems to me that if many people adopt heroic responsibility to their own values, then a handful of people with destructive values might screw up everyone else, because destroying is easier than helping people.

That's the Übermensch morality of dictators and totalitarian regimes. The problem is that every dictator thinks of themselves as a benevolent dictator, but it turns out that they are often mistaken.

Even when there are multiple aspiring dictators, all with essentially benevolent values, conflict on who should rule can be very much destructive.

Of course this presupposes consequentialism. In deontologism morality is typically viewed as a social contract where each party has well defined responsabilities.

I don't think it's an accident that violent totalitarian ideologies tend to be consequentialist: these innocent heretics/Jews/bourgeois stand between us and our utopia. Kill'em all!

4RolfAndreassen14y

Well yes, but what exactly are you going to substitute for individual judgement of what actions to take? If you decide that following code X, and enforcing that others do so as well, even when it seems like breaking the code this one time would give better results, is the best course of action (perhaps for reasons of precommitment or average-case utility or what-have-you)... then oops, you seem to have made a decision in accordance with your best judgement, there.

0Shmi14y

There is an obvious parallel between HJPEV and AGI: he can do (and does) stuff no (other) human can even conceive of doing. How do you know if your values and goals are constructive or destructive? It all comes down to the same hard question of FAI (well, FHHI, friendly human hero intelligence, in this case): are your values CEV-aligned? So, the first thing Harry should do is stop running around saving people and derive the CEV :) One can, of course, argue that, as a human hero, HJPEV has the right CEV built-in and should just run around implementing it. However, given that even his friends and allies disagree (Dumbledore is a deathist, McGonagall is a disciplinarian, Hermione thinks they are too young, Weasley twins just want to have fun), and he gets the most help from the anti-hero Quirrell, this point of view is hard to defend. The situation is much worse outside of fiction, where buggy or limited wetware constantly leads would-be heroes (Lenin, Castro, Lincoln or even Hitler) to commit or cause suffering or destruction of the same group of people they aspired to help. So the first question a would-be hero should ask herself is whether she is prepared to live with the consequences of her actions if they backfire. (And if the answer is yes, she is clearly a villain.)