NxGenSentience comments on Superintelligence 9: The orthogonality of intelligence and goals - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (78)
Taking off from the end of that point, I might add (but I think this was probably part of your total point, here, about "the most interesting" ingredient) that people sometimes forget that utilitarianism is not a theory itself about what is normatively desirable, and least not much of one. For Bentham-style "greatest good for the greatest number" to have any meaning, it has to be supplemented with a view of what property, state of being, action type, etc, counts as a "good" thing, to begin with. Once this is defined, we can then go on to maximize that -- seeking to achieve the most of that, for the most people (or relevant entities.)
But greatest good for the greatest number means nothing until we figure out a theory of normativity, or meta-normativity that can be instantiated across specific, varying situations and scenarios.
IF the "good" is maximizing simple total body weight, then adding up the body weight of all people in possible world A, vs in possible world B, etc, will allow us a utilitarian decision among possible worlds.
IF the "good" were fitness, or mental healty, or educational achievement... we use the same calculus, but the target property is obviously different.
Utilitarianism is sometimes a person's default answer, until you remind them that this is not an answer at all about what is good. It is just an implementation standard for how that good is to be devided up. Kind of a trivial point, I guess, but worth reminding ourselves from time to time that utilitarianism is not a theory of what is actually good, but how that might be distributed, if that admits of scarcity.