someonewrongonthenet comments on Privileging the Question - Less Wrong

102 Post author: Qiaochu_Yuan 29 April 2013 06:30PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (311)

You are viewing a single comment's thread. Show more comments above.

Comment author: someonewrongonthenet 30 April 2013 12:30:39AM *  2 points [-]

So the original question was:

How to optimize the society for more long-term utility for people of any sexual orientation?

Under classical preference utilitarianism, you try to maximize everyone's utility and conveniently ignore the problems of putting two utility functions into one equation, and the problems you mention.

Continuing to conveniently ignore that problem, I implicitly assume that we agree that the positive utility generated by removing restrictions to homosexuality outweigh the negative utility generated by violating purity boundaries, when applied over the entire population.

We still include the purity thing in the calculations of course. For example, I could in principle argue that the negative utility from allowing sex in public probably outweighs the positive utility generated from the removal of the restriction, hence our public obscenity laws.

Comment author: Eugine_Nier 01 May 2013 03:02:53AM 3 points [-]

Continuing to conveniently ignore that problem, I implicitly assume that we agree that the positive utility generated by removing restrictions to homosexuality outweigh the negative utility generated by violating purity boundaries, when applied over the entire population.

That ignores the possibility that there is a reason those purity boundaries were there in the first place.

Comment author: someonewrongonthenet 01 May 2013 05:12:19AM *  -1 points [-]

I've seen this before, but I can't say I find it a compelling argument - if an institution was placed for good reason, then at least someone, somewhere would remember why it was placed and could give a compelling argument. If no one can do so, the risk of some, hidden drawback which the original lawmaker could have forseen seems too small to count.

I mean, this argument does apply when you are acting alone, on some question that neither you nor anyone you come into contact with knows anything about...but it doesn't apply to something like this.

Comment author: NancyLebovitz 30 April 2013 08:44:30PM 0 points [-]

How do utilitarians decide to draw the boundary at the whole human race rather than some smaller set of humans?

Comment author: someonewrongonthenet 01 May 2013 05:15:18AM 0 points [-]

II'm not sure if I understand your question...

Utilitarians who choose to draw the line around the whole of the human race do so because they believe they aught to value the whole of the human race.

Comment author: NancyLebovitz 01 May 2013 05:47:21AM 1 point [-]

Utilitarians who choose to draw the line around the whole of the human race do so because they believe they aught to value the whole of the human race.

Is that a deontological standard?

The reason I asked is that, in principle, you could have utilitarianism based on some group smaller than the human race.

Comment author: someonewrongonthenet 01 May 2013 07:13:29PM *  1 point [-]

Is that a deontological standard?

For some people, probably. Let's take a step back.

Morality comes from the "heart". It's made of feelings. Utilitarianism (and much of what falls under moral philosophy) is one of many attempt to make a consistent set of rules to describe inconsistent feelings. The purpose of making a consistent set of rules is 1) to convince others of the morality of an action and 2) because we morally feel aversion to hypocrisy and crave moral consistency.

Keeping those aims in mind, drawing the line across all humans, sentient beings, etc has the following benefits:

1) The creators might feel that the equation describes the way they feel better when they factor in all humans. They might hold it as a deontological standard to care about all humans, or they might feel a sense of fairness, or they might have empathy for everyone, etc.

2) Drawing the line across all humans allows you to use the utilitarian standard to negotiate compromises with any arbitrary human you come across. Many humans, having the feelings described in [1], will instinctively accept utilitarianism as a valid way to think about things.

There are plenty of things that are problematic here, but that is why utilitarianism defaults to include the whole human race. As with all things moral, that's just an arbitrary choice on our part, and we could easily have done it a different way. We can restrict it to a smaller subset of humans, we can broaden it to non-human things which seem agent-like enough to be worth describing with a utility function, etc. Many utilitarians include animals, for example.

Comment author: Juno_Watt 01 May 2013 08:58:06PM 1 point [-]

Morality comes from the "heart". It's made of feelings.

People use feelings/System1 to do morality. That doesn't make it an oracle. Thinking might be more accurate.

As with all things moral, that's just an arbitrary choice on our part

If you don't know how to solve a problem, you guess. But that doens't mean anything goes. Would anyone include rocks in the Circle? Probably not, since they don't have feelings, values, or preferences. So there seem to be some constraints.

Comment author: someonewrongonthenet 02 May 2013 06:02:02PM *  0 points [-]

People use feelings/System1 to do morality. That doesn't make it an oracle. Thinking might be more accurate.

Accurate? How can you speak of a moral preference being "accurate" or not? Moral preferences simply are. There are some meta-ethics sequences here that explain the arbitrariness of our moral preferences more eloquently , and here is a fun story that tangentially illustrates it

Would anyone include rocks in the Circle? Probably not, since they don't have feelings, values, or preferences. So there seem to be some constraints.

I bet I can find you someone who would say that burning the Quran or the Bible is inherently immoral.

Comment author: Juno_Watt 02 May 2013 07:03:44PM *  0 points [-]

I bet I can find you someone who would say that burning the Quran or the Bible is inherently immoral.

We sometimes extend morality to inanimate objects , but only ones that mean something to us, such as works of art and religious artefacts. That isn't actually inherent because of the "to us" clause, although some people might claim that it is.

There are some meta-ethics sequences here that explain the arbitrariness of our moral preferences more eloquently , and here is a fun story that tangentially illustrates it

Pebble sorting is a preference. That's it. I don't have to believe it is a moral preference or a correct moral preference.

How can you speak of a moral preference being "accurate" or not?

Moral objectivism isn't obviously wrong, and system 2 isn't obviously the wrong way to realise moral truths. IOW, moral subjectivism isn't obviously true.

NB: Objectivism isn't universalism.

Moral preferences simply are.

Beliefs simply are. And some are true and some are not. You seem to be assuming the non-existence of anything that could verify or disprove a moral preference in order to prove more or less the same thing.

Comment author: someonewrongonthenet 02 May 2013 09:27:55PM *  0 points [-]

We sometimes extend morality to inanimate objects , but only ones that mean something to us, such as works of art and religious artefacts. That isn't actually inherent because of the "to us" clause, although some people might claim that it is.

I would say that the "to us" clause actually applies to everything, and that nothing is "inherent", as you put it. Pebble sorting means something to the pebble sorters. Humans mean something to me. The entirity of morality boils down to what is important "to us"?

To me, moral objectivism is obviously wrong and subjectivism is obviously true, and this is embedded in my definition of morality. I'm actually unsure how anyone could think of it in any other coherent way.

I think it's time to unpack "morality". I think morality is feelings produced in the human mind about how people aught to act. That is, I think "murder is bad" is in some ways analogous to "Brussels sprouts are gross". From this definition, it follows that I see moral objectivism as obviously wrong - akin to saying, "no man, Brussels sprouts are objectively, inherently gross! In the same way that the sky is objectively blue! / In the same way that tautologies are true!" (Actually, replace blue with the appropriate wavelengths to avoid arguments about perception)

What do you think "morality" is, and where do you suppose it comes from?

Comment author: Juno_Watt 07 May 2013 06:46:53PM *  0 points [-]

I think morality is behaving so as to take into account the values and preferences of others as well as ones own. You can succed or fail in that, hence "accurate".

Morality may manifest in the form of a feeling for many people, but not for everybody and not all feelings are equal. So I don't think that is inherent, or definitional.

I don't think the sprout analogy works, because your feeling that you don't like sprouts doesn't seriously affect others, but the psychoaths fondndess for murder does.

The feelings that are relevant to morality are the empathic ones, not personal preferences. That is a clue that morality is about behaving so as to take into account the values and preferences of others as well as ones own.

if you think morlaity is the same as a personal preference...what makes it morality? Why don't we just have one word and one way of thinking?

Comment author: Leonhart 02 May 2013 07:43:07PM *  0 points [-]

There are some meta-ethics sequences here that explain the arbitrariness of our moral preferences more eloquently , and here is a fun story that tangentially illustrates it

I think you've misunderstood the meta-ethics sequences, then, or I have, because

Pebble sorting is a preference. That's it. I don't have to believe it is a moral preference or a correct moral preference.

is quite similar to Eliezer's position. Although Juno_Watt may have reached it from another direction.

Comment author: someonewrongonthenet 02 May 2013 09:13:57PM *  2 points [-]

I read it as a warning about expecting sufficiently rational beings to automatically acquire human morality, in the same way that sufficiently rational beings would automatically acquire knowledge about true statements (science, etc). The lesson is that preferences (morality, etc) is different from fact.

If you want to know Eliezer's views, he spells them out explicitly here - although I think the person most famous for this view is Nietzsche (not that he's the first to have held this view).

To me, "No universally compelling arguments" means this - two rational agents will converge upon factual statements, but they need not converge upon preferences (moral or otherwise) because moral statements aren't "facts".

It really doesn't matter if you define the pebble sorting as a "moral" preference or a plain old preference.The point is, that humans have a morality module - but that module is in the brain and not a feature which is implicit in logical structures, nor is it a feature implicit in the universe itself.

Comment author: Juno_Watt 07 May 2013 06:54:06PM 0 points [-]

Juno Watt has read the sequences, but still doesn't know what Eliezer's position is.

Comment author: TheOtherDave 02 May 2013 06:33:46PM 0 points [-]

Quite a few of them no doubt. Of course, the overwhelming majority of people who would say that burning the Quran or the Bible is inherently immoral would also say that it's immoral by virtue of the preferences of an entity that, on their view, is in fact capable of having preferences.

Of course, I'm sure I could find someone who would say rocks have feelings, values, and preferences.

Comment author: Eugine_Nier 03 May 2013 12:46:34AM 1 point [-]

would also say that it's immoral by virtue of the preferences of an entity that, on their view, is in fact capable of having preferences.

I don't think this is an accurate formulation of the general religious attitude towards morality.

Comment author: TheOtherDave 03 May 2013 02:51:24AM -1 points [-]

I agree. Do you also think it's a false statement?

Comment author: someonewrongonthenet 02 May 2013 08:55:51PM *  0 points [-]

Conversationalists will want to preserve ecosystems, even where those ecosystems are already well studied by science, even when the ecosystem contains no sentient beings (plants, fungi, microbes), even when destroying the ecosystem has many advantages for humans, because they think the ecosystem is intrinsically valuable independently of the effect on beings with feelings, values, and preferences.

Some looser examples...

Pro-life advocates say that beings without preferences have rights by virtue of future preferences. Not all of them are religious.

Hindus treat books (all books in general) with reverence because they are vehicles of learning, despite not necessarily believing in deities.

Many social conservatives report being unwilling to slap their fathers, even with permission, as part of a play.

The classic trolley problem implies that many people's moral intuitions hinge on the act of murder being wrong, rather than the effect that the death has on the values, feelings, and preferences being morally wrong.

Of course, if you are a moral realist, you can just say that these people's intuitions are "wrong"...but the point is that "feelings, values, and preferences" - in a word, utilitarianism - isn't the only guiding moral principle that humans care about.

And yes, you could argue that this is all a deity's preferences...but why did they decide that those were in fact the deity's preferences? Doesn't it hint that they might have an underlying feeling of those preferences in themselves, that they would project those wishes on a deity?

Comment author: TheOtherDave 03 May 2013 02:57:01AM 0 points [-]

Conversationalists will want to preserve ecosystems

No doubt some of them will, but I suspect you meant "conservationists." And yes, I agree that some of those will assign intrinsic value to "nature" in various forms, or at least claim to, as you describe.

Pro-life advocates say that beings without preferences have rights by virtue of future preferences. Not all of them are religious.

Some of them do, yes. Indeed, I suspect the ones who say that are disproportionately non-religious.

why did they decide that those were in fact the deity's preferences?

A fine question.

Doesn't it hint that they might have an underlying feeling of those preferences in themselves, that they would project those wishes on a deity?

That's one possibility, yes.

Comment author: Juno_Watt 02 May 2013 07:09:05PM 0 points [-]

Quite a few of them no doubt. Of course, the overwhelming majority of people who would say that burning the Quran or the Bible is inherently immoral would also say that it's immoral by virtue of the preferences of an entity that, on their view, is in fact capable of having preferences.

And, again, if destroying entity X is wrong because some other entity Y says so, that is not inherent.

Comment author: TheOtherDave 02 May 2013 08:28:45PM 0 points [-]

Indeed. Do you mean to say that you don't expect it to be said, or merely that those saying it are confused?

Comment author: Jayson_Virissimo 01 May 2013 06:30:08AM *  1 point [-]

You could also, in principle, have a utilitarianism that gives unequal weights to different people. I've asked around here for a reason to think that the egalitarian principle is true, but haven't yet received any responses that are up to typical Less Wrong epistemic standards.

Comment author: Eugine_Nier 02 May 2013 12:26:15AM 3 points [-]

I've asked around here for a reason to think that the egalitarian principle is true, but haven't yet received any responses that are up to typical Less Wrong epistemic standards.

It's a very clear Schelling point. At least until advances in uplifting/AI/brain emulation/etc. complicates the issue of what counts as a human.

Comment author: Jayson_Virissimo 02 May 2013 11:40:44PM 0 points [-]

It's a very clear Schelling point.

This seems to me very unclear actually. In fact, I have never encountered someone that acted as if this was (approximately) the decision criterion they were following. For all the humans I have personally observed, they seem to be acting as if they, their friends, and their family members are weighted thousands or millions of times greater than perfect strangers.

Comment author: Eugine_Nier 03 May 2013 12:48:53AM 1 point [-]

That, or something like it, is the decision criterion people are expected to follow when acting in official capacity.

Comment author: someonewrongonthenet 01 May 2013 07:30:40PM -1 points [-]

I've asked around here for a reason to think that the egalitarian principle is true ... is true

You're applying moral realism here...as in, you are implying that moral facts exist objectively, outside of a human's feelings. Are you dong this intentionally?

Comment author: seanwelsh77 01 May 2013 09:21:15AM -2 points [-]

Your alternative would be to think an aristocratic or meritocratic principle is true. (It's either equal or unequal, right?)

I think we can assume aristocracy is a dead duck along with the Divine Right of Kings and other theological relics.

Meritocracy in some form I believe has been advocated by some utilitarians. People with Oxford degrees get 10 votes. Cambridge 9. Down to the LSE with 2 votes and the common ignorant unlettered herd 1 vote...

This is kind of an epistemocratic voting regime which some think might lead to better outcomes. Alas, no one has been game to try get such laws up. There is little evidence that an electorate of PhDs is any less daft/ignorant/clueless/idle/indifferent on matters outside their specialty than the general public.

From a legal rights perspective, egalitarianism is surely correct. Equal treatment before the law seems a lot easier to defend than unequal treatment.

But put something up that assumes a dis-egalitarian principle and see how it flies. I'd be interested to see if you can come up with something plausible that is dis-egalitarian and up to epistemic scratch...

Hint: plutocracy...

Comment author: Jayson_Virissimo 02 May 2013 12:44:29AM *  2 points [-]

Your alternative would be to think an aristocratic or meritocratic principle is true. (It's either equal or unequal, right?)

I wouldn't use those terms, since they bring in all kinds of unnecessary connotations. I would say the opposite of the egalitarian principle is the non-egalitarian principle. I was thinking less along the lines of nobles/commoners and more along the lines of my children/other people's children. I find the idea (that I think the egalitarian principle entails) that I have as much obligation to perfect strangers as to my wife to be extremely counter-intuitive.

I think we can assume aristocracy is a dead duck along with the Divine Right of Kings and other theological relics.

I don't consider the Divine Right of Crowds ('human rights', or whatever the cool kids are calling it these days) to be any less silly than those 'theological relics'.

Meritocracy in some form I believe has been advocated by some utilitarians. People with Oxford degrees get 10 votes. Cambridge 9. Down to the LSE with 2 votes and the common ignorant unlettered herd 1 vote...

This is kind of an epistemocratic voting regime which some think might lead to better outcomes. Alas, no one has been game to try get such laws up. There is little evidence that an electorate of PhDs is any less daft/ignorant/clueless/idle/indifferent on matters outside their specialty than the general public.

This part isn't really relevant to what I'm talking about, since I'm not discussing equal weight in decision-making, but equal weight in a social welfare function. My infant son's interests are one of my greatest concerns, but he currently has about zero say in family decision-making.

From a legal rights perspective, egalitarianism is surely correct. Equal treatment before the law seems a lot easier to defend than unequal treatment.

Equal treatment before the law does not necessarily mean that individuals interests are weighted equally. When was the last time you heard of jurors on a rape trial trying to figure out exactly how much utility the rapist got so they could properly combine that with the disutility of the victim?

Comment author: gjm 03 May 2013 12:01:31PM 5 points [-]

Divine Right of Crowds

Of course what "the cool kids" are actually talking about is more like a Divine Right of People; it's got nothing to do with treating people differently when there's a mass of them. And of course adding the word "divine" is nothing more than a handy way of making it sound sillier than it otherwise would (whereas in "Divine Right of Kings" it is a word with an actual meaning; the power of kings was literally thought to be of divine origin).

So, removing some of the spin, what you're apparently saying is that "let's treat all people as having equal rights" seems as silly to you as "let's suppose that one person in each country is appointed by a divine superbeing to rule over all the others". Well, OK.

Equal treatment before the law does not necessarily mean that individuals' interests are weighted equally.

It means that people are treated unequally only according to differences that are actually relevant. (Of course then the argument shifts to which differences are relevant; but at least then one actually has to argue for their relevance rather than simply assuming it on traditional grounds.)

Having said all of which, I agree that the usual arguments for equal weighting completely fail to show that a person shouldn't give higher weighting to herself, her family, her friends, etc.

Comment author: Jayson_Virissimo 04 May 2013 07:01:59AM 0 points [-]

Of course what "the cool kids" are actually talking about is more like a Divine Right of People; it's got nothing to do with treating people differently when there's a mass of them.

The state in which I live has statute law initiatives, so yes, people actually do 'rule' only if there is a large enough mass of them. Individually, I have no such (legal) right.

And of course adding the word "divine" is nothing more than a handy way of making it sound sillier than it otherwise would (whereas in "Divine Right of Kings" it is a word with an actual meaning; the power of kings was literally thought to be of divine origin).

Speaking of dubious origins:

We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights...

I am in complete agreement with the following:

It means that people are treated unequally only according to differences that are actually relevant. (Of course then the argument shifts to which differences are relevant; but at least then one actually has to argue for their relevance rather than simply assuming it on traditional grounds.)

Having said all of which, I agree that the usual arguments for equal weighting completely fail to show that a person shouldn't give higher weighting to herself, her family, her friends, etc.

In any case, the point of my comment was not to bring up politics, but to show the incompatibility of typical intuitions with regards to how one should treat family and friends compared to strangers with what (the most popular flavors of) utilitarianism seems to indicate is 'correct'.

I have argued with utilitarians several times on Less Wrong and the discussions seem to follow the same sequence of backpedalling. First they claim utilitarianism is true. Then, when I ask and they are unable to conceive of an experiment that would verify or falsify it, they claim that it isn't the kind of thing that has a truth-value, but that it is a description of their preferences. Next, I demonstrate that relying on revealed preference shows that virtually nobody actually has utilitarian preferences. Lastly, they claim that intuition gives us good reason go with (even if it isn't True) utilitarianism. My response to NancyLebovitz in this thread is yet another attempt to show that, no, it really isn't intuitive.

Is this an accurate description of what is going on or am I mind-killed on the subject of normative ethics (or both, or neither)?

Comment author: gjm 04 May 2013 10:26:34AM 1 point [-]

[...] statute law initiatives [...]

When you first used the phrase "Divine Right of Crowds" you immediately explained in parentheses that you meant "human rights" or something similar. Now you seem to be talking about democracy instead. The two aren't the same, though probably approval of one is correlated with approval of the other.

Anyway, "crowds" in the literal sense still aren't involved (it needs N people to get something voted on, but that doesn't require them to be colocated or to know one another or anything else crowd-like other than sheer numbers; and if you're now using "Divine Right of Crowds" to mean "a political system that tries to favour outcomes preferred by more people rather than fewer" then, again, I suggest that you're picking terminology simply to make the other side look as silly as possible.

Speaking of dubious origins: [...]

It is possible that those words from the Declaration of Independence show that in the 18th century people believed in something like a "Divine Right of Crowds". (It's not entirely obvious, though. Perhaps they actually just believed in a Right of Crowds and thought what they said would sound better if they included "created" and "by their Creator"; compare the mention of a Creator at the end of some editions of the Origin of Species, or Einstein's "God does not play dice".)

But that doesn't mean that people who now favour democracy, or human rights, or independence of the US from the UK, have to believe (or commonly do believe) that those things are divinely ordained. Similarly, there are people now who want kings without believing in a Divine Right of Kings, and pretending that they do would be a shabby rhetorical trick.

[...] incompatibility of typical intuitions [...] with what (the most popular flavors of) utilitarianism seems to indicate [...]

Yup, there are indeed such incompatibilities (though I think one could make a reasonable argument that, given human nature, overall utility is likely to be higher in a society where people care more about themselves and those closer to them than in one where they truly care equally about everyone. Surely not nearly so much more as our intuitions lead to, though.

the same sequence of backpedalling

I'll take your word for it, but I'm a bit surprised: I'd have thought an appreciable fraction of LWers advocating utilitarianism would start from the position that it's an expression of their preferences rather than an objective fact about the world.

(For my part, not that it particularly matters, I do indeed care most about myself, and less about people less connected to me, physically further from me, more unlike me, etc., but I find that as I reflect more on my preferences in any given case they shift nearer to egalitarianism, though they often don't get all the way. Something like utilitarianism seems like a pretty decent approximation to what I'd want in law.)

am I mind-killed [...]?

I can't tell, obviously, but I do tend to think that things like switching ground without noticing ("human rights" --> democracy) and insisting on using question-begging language ("Divine Right of Crowds") are often signs of someone not thinking as clearly as they might be.

Comment author: MugaSofer 01 May 2013 04:54:05PM 1 point [-]

I think we can assume aristocracy is a dead duck along with the Divine Right of Kings and other theological relics.

Counterpoint: it offers stability, which is useful regardless of theology. See the Fnargle World thought experiment and various other neo-reactionary stuff on Why Democracy Is Bad.

Let me put it this way: would you rather we're ruled by someone who's skilled at persuading us to elect him, and who focuses resources on looking good in four years; or someone who's been trained since birth to govern well, and knows they or their descendants will be held accountable for any future side-effects of their policies?

These arguments may be deeply flawed, but hereditary aristocracy doesn't stand of fall with the Divine Right Of Kings.

Comment author: Juno_Watt 01 May 2013 06:19:41PM 1 point [-]

Counterpoint: it offers stability, which is useful regardless of theology.

Stability Is good if governance is good and bad if not.

Let me put it this way: would you rather we're ruled by someone who's skilled at persuading us to elect him, and who focuses resources on looking good in four years;

...and you can get rid of..

or someone who's been trained since birth to govern well, and knows they or their descendants will be held accountable for any future side-effects of their policies?

OK. Looks like democracy with a supply of candidates from Kennedy-style political dynasties is the best of all possible systems...;-)

Comment author: MugaSofer 01 May 2013 06:59:20PM 0 points [-]

Stability Is good if governance is good and bad if not.

I was suggesting that it might serve to render governance better.

OK. Looks like democracy with a supply of candidates from Kennedy-style political dynasties is the best of all possible systems...;-)

You still have to focus on retaining popularity, via attacking political opponents and increasing PR skills, unless the elections are total shams.

Also, to be clear, I'm not advocating this position; just pointing out there are other arguments for it than the "Divine Right of Kings".

Comment author: Juno_Watt 01 May 2013 08:38:17PM 0 points [-]

I was suggesting that it might serve to render governance better

Under democracy, the people can decide if their stable government has outstayed its welcome after so many years.

You still have to focus on retaining popularity, via attacking political opponents and increasing PR skills, unless the elections are total shams.

Whilst aristos just have to keep slipping their rivals the poisoned chalice...much more discreet.

Also, to be clear, I'm not advocating this position;

Got that.

Comment author: Estarlio 12 May 2013 10:05:35PM *  -1 points [-]

...and you can get rid of..

Kinda. In practice a lot of the power of government wrests in agencies that offer advice to the currently ruling party, and those agencies often embody significant powers themselves. It would be a mistake to confuse the elected executive branch of government with government entire. It's not even clear to me that they have the majority share of influence over what actually happens.

Comment author: ArisKatsaris 01 May 2013 06:05:23PM 1 point [-]

See the Fnargle World thought

I can't seem to google up anything with the worlds "Fnargle World"

Comment author: ThrustVectoring 01 May 2013 06:35:08PM 1 point [-]
Comment author: ikrase 01 May 2013 06:30:34AM 0 points [-]

Ummmmm... do I draw the line around the whole of the human race? I'm not sure whether I do or not. I do know that there is a certain boundary (defined mostly by culture) where I get much more likely to say 'that's your problem' and become much less skeptical/cynical about preferences, although issues that seem truly serious always get the same treatment.

For some reason, choosing to accept that somebody's utility function might be very different from your own feels kind of like abandoning them from the inside. (Subjective!).

Comment author: MugaSofer 01 May 2013 07:04:45PM -2 points [-]

Considering many of them profess to include other kinds of intelligence, at least in theory ... it seems to be mostly a consistency thing. Why shouldn't I include Joe The Annoying Git?