Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Humans are utility monsters

67 PhilGoetz 16 August 2013 09:05PM

When someone complains that utilitarianism1 leads to the dust speck paradox or the trolley-car problem, I tell them that's a feature, not a bug. I'm not ready to say that respecting the utility monster is also a feature of utilitarianism, but it is what most people everywhere have always done. A model that doesn't allow for utility monsters can't model human behavior, and certainly shouldn't provoke indignant responses from philosophers who keep right on respecting their own utility monsters.

continue reading »

Metacontrarian Metaethics

2 Will_Newsome 20 May 2011 05:36AM

Designed to gauge responses to some parts of the planned “Noticing confusion about meta-ethics” sequence, which should intertwine with or be absorbed by Lukeprog’s meta-ethics sequence at some point.

Disclaimer: I am going to leave out many relevant details. If you want, you can bring them up in the comments, but in general meta-ethics is still very confusing and thus we could list relevant details all day and still be confused. There are a lot of subtle themes and distinctions that have thus far been completely ignored by everyone, as far as I can tell.

Problem 1: Torture versus specks

Imagine you’re at a Less Wrong meetup when out of nowhere Eliezer Yudkowsky proposes his torture versus dust specks problem. Years of bullet-biting make this a trivial dilemma for any good philosopher, but suddenly you have a seizure during which you vividly recall all of those history lessons where you learned about the horrible things people do when they feel justified in being blatantly evil because of some abstract moral theory that is at best an approximation of sane morality and at worst an obviously anti-epistemic spiral of moral rationalization. Temporarily humbled, you decide to think about the problem a little longer:

"Considering I am deciding the fate of 3^^^3+1 people, I should perhaps not immediately assert my speculative and controversial meta-ethics. Instead, perhaps I should use the averaged meta-ethics of the 3^^^3+1 people I am deciding for, since it is probable that they have preferences that implicitly cover edge cases such as this, and disregarding the meta-ethical preferences of 3^^^3+1 people is certainly one of the most blatantly immoral things one can do. After all, even if they never learn anything about this decision taking place, people are allowed to have preferences about it. But... that the majority of people believe something doesn’t make it right, and that the majority of people prefer something doesn’t make it right either. If I expect that these 3^^^3+1 people are mostly wrong about morality and would not reflectively endorse their implicit preferences being used in this decision instead of my explicitly reasoned and reflected upon preferences, then I should just go with mine, even if I am knowingly arrogantly blatantly disregarding the current preferences of 3^^^3 currently-alive-and-and-not-just-hypothetical people in doing so and thus causing negative utility many, many, many times more severe than the 3^^^3 units of negative utility I was trying to avert. I may be willing to accept this sacrifice, but I should at least admit that what I am doing largely ignores their current preferences, and there is some chance it is wrong upon reflection regardless, for though I am wiser than those 3^^^3+1 people, I notice that I too am confused."

You hesitantly give your answer and continue to ponder the analogies to Eliezer’s document “CEV”, and this whole business about “extrapolation”...

(Thinking of people as having coherent non-contradictory preferences is very misleadingly wrong, not taking into account preferences at gradient levels of organization is probably wrong, not thinking of typical human preferences as implicitly preferring to update in various ways is maybe wrong (i.e. failing to see preferences as processes embedded in time is probably wrong), et cetera, but I have to start somewhere and this is already glossing over way too much.)

Bonus problem 1: Taking trolleys seriously

"...Wait, considering how unlikely this scenario is, if I ever actually did end up in it then that would probably mean I was in some perverse simulation set up by empirical meta-ethicists with powerful computers, in which case they might use my decision as part of a propaganda campaign meant to somehow discredit consequentialist reasoning or maybe deontological reasoning, or maybe they'd use it for some other reason entirely, but at any rate that sure complicates the problem...” (HT: Steve Rayhawk)

The Aliens have Landed!

33 TimFreeman 19 May 2011 05:09PM

"General Thud! General Thud! Wake up! The aliens have landed. We must surrender!" General Thud's assistant Fred turned on the lights and opened the curtains to help Thud wake up and confront the situation. Thud was groggy because he had stayed up late supervising an ultimately successful mission carried out by remotely piloted vehicles in some small country on the other side of the world. Thud mumbled, "Aliens? How many? Where are they? What are they doing?" General Thud looked out the window, expecting to see giant tripods walking around and destroying buildings with death rays. He saw his lawn, a bright blue sky, and hummingbirds hovering near his bird feeder.

continue reading »

Is Kiryas Joel an Unhappy Place?

20 gwern 23 April 2011 12:08AM

I was browsing my RSS feed, as one does, and came across a New York Times article, "A Village With the Numbers, Not the Image, of the Poorest Place", about the Satmar Hasidic Jews of Kiryas Joel (NY).

Their interest lies in their extraordinarily high birthrate & population growth, and their poverty - which are connected. From the article:

"...officially, at least, none of the nation’s 3,700 villages, towns or cities with more than 10,000 people has a higher proportion of its population living in poverty than Kiryas Joel, N.Y., a community of mostly garden apartments and town houses 50 miles northwest of New York City in suburban Orange County.

About 70 percent of the village’s 21,000 residents live in households whose income falls below the federal poverty threshold, according to the Census Bureau. Median family income ($17,929) and per capita income ($4,494) rank lower than any other comparable place in the country. Nearly half of the village’s households reported less than $15,000 in annual income. About half of the residents receive food stamps, and one-third receive Medicaid benefits and rely on federal vouchers to help pay their housing costs.

Kiryas Joel’s unlikely ranking results largely from religious and cultural factors. Ultra-Orthodox Satmar Hasidic Jews predominate in the village; many of them moved there from Williamsburg, Brooklyn, beginning in the 1970s to accommodate a population that was growing geometrically. Women marry young, remain in the village to raise their families and, according to religious strictures, do not use birth control. As a result, the median age (under 12) is the lowest in the country and the household size (nearly six) is the highest. Mothers rarely work outside the home while their children are young. Most residents, raised as Yiddish speakers, do not speak much English. And most men devote themselves to Torah and Talmud studies rather than academic training — only 39 percent of the residents are high school graduates, and less than 5 percent have a bachelor’s degree. Several hundred adults study full time at religious institutions.

...Because the community typically votes as a bloc, it wields disproportionate political influence, which enables it to meet those challenges creatively. A luxurious 60-bed postnatal maternal care center was built with $10 million in state and federal grants. Mothers can recuperate there for two weeks away from their large families. Rates, which begin at $120 a day, are not covered by Medicaid, although, Mr. Szegedin said, poorer women are typically subsidized by wealthier ones.

...The village does aggressively pursue economic opportunities. A kosher poultry slaughterhouse, which processes 40,000 chickens a day, is community owned and considered a nonprofit organization. A bakery that produces 800 pounds of matzo daily is owned by one of the village’s synagogues.

Most children attend religious schools, but transportation and textbooks are publicly financed. Several hundred handicapped students are educated by the village’s own public school district, which, because virtually all the students are poor and disabled, is eligible for sizable state and federal government grants.

... Still, poverty is largely invisible in the village. Parking lots are full, but strollers and tricycles seem to outnumber cars. A jeweler shares a storefront with a check-cashing office. To avoid stigmatizing poorer young couples or instilling guilt in parents, the chief rabbi recently decreed that diamond rings were not acceptable as engagement gifts and that one-man bands would suffice at weddings. Many residents who were approached by a reporter said they did not want to talk about their finances.

...Are as many as 7 in 10 Kiryas Joel residents really poor? “It is, in a sense, a statistical anomaly,” Professor Helmreich said. “They are clearly not wealthy, and they do have a lot of children. They spend whatever discretionary income they have on clothing, food and baby carriages. They don’t belong to country clubs or go to movies or go on trips to Aruba.

...David Jolly, the social services commissioner for Orange County, also said that while the number of people receiving benefits seemed disproportionately high, the number of caseloads — a family considered as a unit — was much less aberrant. A family of eight who reports as much as $48,156 in income is still eligible for food stamps, although the threshold for cash assistance ($37,010), which relatively few village residents receive, is lower....“You also have no drug-treatment programs, no juvenile delinquency program, we’re not clogging the court system with criminal cases, you’re not running programs for AIDS or teen pregnancy,” he [Mr. Szegedin, the village administrator] said. “I haven’t run the numbers, but I think it’s a wash.”

From Wikipedia:

The land for Kiryas Joel was purchased in 1977, and fourteen Satmar families settled there. By 2006, there were over 3,000...In 1990, there were 7,400 people in Kiryas Joel; in 2000, 13,100, nearly doubling the population. In 2005, the population had risen to 18,300, a rate of growth suggesting it will double again in the ten years between 2000 and 2010.

Robin Hanson has argued that uploaded/emulated minds will establish a new Malthusian/Darwinian equilibrium in "IF UPLOADS COME FIRST: The crack of a future dawn" - an equilibrium in comparison to which our own economy will look like a delusive dreamtime of impossibly unfit and libertine behavior. The demographic transition will not last forever. But despite our own distaste for countless lives living at near-subsistence rather than our own extreme per-capita wealth (see the Repugnant Conclusion), those many lives will be happy ones (even amidst disaster).

So. Are the inhabitants of Kiryas Joel unhappy?

Offense versus harm minimization

60 Yvain 16 April 2011 01:06AM

Imagine that one night, an alien prankster secretly implants electrodes into the brains of an entire country - let's say Britain. The next day, everyone in Britain discovers that pictures of salmon suddenly give them jolts of painful psychic distress. Every time they see a picture of a salmon, or they hear about someone photographing a salmon, or they even contemplate taking such a picture themselves, they get a feeling of wrongness that ruins their entire day.

I think most decent people would be willing to go to some trouble to avoid taking pictures of salmon if British people politely asked this favor of them. If someone deliberately took lots of salmon photos and waved them in the Brits' faces, I think it would be fair to say ey isn't a nice person. And if the British government banned salmon photography, and refused to allow salmon pictures into the country, well, maybe not everyone would agree but I think most people would at least be able to understand and sympathize with the reasons for such a law.

So why don't most people extend the same sympathy they would give Brits who don't like pictures of salmon, to Muslims who don't like pictures of Mohammed?

continue reading »

Hacking the CEV for Fun and Profit

53 Wei_Dai 03 June 2010 08:30PM

It’s the year 2045, and Dr. Evil and the Singularity Institute have been in a long and grueling race to be the first to achieve machine intelligence, thereby controlling the course of the Singularity and the fate of the universe. Unfortunately for Dr. Evil, SIAI is ahead in the game. Its Friendly AI is undergoing final testing, and Coherent Extrapolated Volition is scheduled to begin in a week. Dr. Evil learns of this news, but there’s not much he can do, or so it seems.  He has succeeded in developing brain scanning and emulation technology, but the emulation speed is still way too slow to be competitive.

There is no way to catch up with SIAI's superior technology in time, but Dr. Evil suddenly realizes that maybe he doesn’t have to. CEV is supposed to give equal weighting to all of humanity, and surely uploads count as human. If he had enough storage space, he could simply upload himself, and then make a trillion copies of the upload. The rest of humanity would end up with less than 1% weight in CEV. Not perfect, but he could live with that. Unfortunately he only has enough storage for a few hundred uploads. What to do…

Ah ha, compression! A trillion identical copies of an object would compress down to be only a little bit larger than one copy. But would CEV count compressed identical copies to be separate individuals? Maybe, maybe not. To be sure, Dr. Evil gives each copy a unique experience before adding it to the giant compressed archive. Since they still share almost all of the same information, a trillion copies, after compression, just manages to fit inside the available space.

Now Dr. Evil sits back and relaxes. Come next week, the Singularity Institute and rest of humanity are in for a rather rude surprise!

The Price of Life

5 BenAlbahari 20 March 2010 09:40AM

Less Wrong readers are familiar with the idea you can and should put a price on life. Unfortunately the Big Lie that you can't and shouldn't has big consequences in the current health care debate. Here's some articles on it:

Yvain's blog post here (HT: Vladimir Nesov).
Peter Singer's article on rationing health care here.
Wikipedia here.
Experts and policy makers who debate this issue here.

For those new to Less Wrong, here's the crux of Peter Singer's reasoning as to why you can put a price on life:

continue reading »

A Much Better Life?

62 Psychohistorian 03 February 2010 08:01PM

(Response to: You cannot be mistaken about (not) wanting to wirehead, Welcome to Heaven)

The Omega Corporation
Internal Memorandum
To: Omega, CEO
From: Gamma, Vice President, Hedonic Maximization

Sir, this concerns the newest product of our Hedonic Maximization Department, the Much-Better-Life Simulator. This revolutionary device allows our customers to essentially plug into the Matrix, except that instead of providing robots with power in flagrant disregard for the basic laws of thermodynamics, they experience a life that has been determined by rigorously tested algorithms to be the most enjoyable life they could ever experience. The MBLS even eliminates all memories of being placed in a simulator, generating a seamless transition into a life of realistic perfection.

Our department is baffled. Orders for the MBLS are significantly lower than estimated. We cannot fathom why every customer who could afford one has not already bought it. It is simply impossible to have a better life otherwise. Literally. Our customers' best possible real life has already been modeled and improved upon many times over by our programming. Yet, many customers have failed to make the transition. Some are even expressing shock and outrage over this product, and condemning its purchasers.

continue reading »

Deontology for Consequentialists

46 Alicorn 30 January 2010 05:58PM

Consequentialists see morality through consequence-colored lenses.  I attempt to prise apart the two concepts to help consequentialists understand what deontologists are talking about.

Consequentialism1 is built around a group of variations on the following basic assumption:

  • The rightness of something depends on what happens subsequently.

It's a very diverse family of theories; see the Stanford Encyclopedia of Philosophy article.  "Classic utilitarianism" could go by the longer, more descriptive name "actual direct maximizing aggregative total universal equal-consideration agent-neutral hedonic act2 consequentialism".  I could even mention less frequently contested features, like the fact that this type of consequentialism doesn't have a temporal priority feature or side constraints.  All of this is is a very complicated bag of tricks for a theory whose proponents sometimes claim to like it because it's sleek and pretty and "simple".  But the bottom line is, to get a consequentialist theory, something that happens after the act you judge is the basis of your judgment.

To understand deontology as anything but a twisted, inexplicable mockery of consequentialism, you must discard this assumption.

Deontology relies on things that do not happen after the act judged to judge the act.  This leaves facts about times prior to and the time during the act to determine whether the act is right or wrong.  This may include, but is not limited to:

  • The agent's epistemic state, either actual or ideal (e.g. thinking that some act would have a certain result, or being in a position such that it would be reasonable to think that the act would have that result)
  • The reference class of the act (e.g. it being an act of murder, theft, lying, etc.)
  • Historical facts (e.g. having made a promise, sworn a vow)
  • Counterfactuals (e.g. what would happen if others performed similar acts more frequently than they actually do)
  • Features of the people affected by the act (e.g. moral rights, preferences, relationship to the agent)
  • The agent's intentions (e.g. meaning well or maliciously, or acting deliberately or accidentally)
continue reading »

Welcome to Heaven

23 denisbider 25 January 2010 11:22PM

I can conceive of the following 3 main types of meaning we can pursue in life.

1. Exploring existing complexity: the natural complexity of the universe, or complexities that others created for us to explore.

2. Creating new complexity for others and ourselves to explore.

3. Hedonic pleasure: more or less direct stimulation of our pleasure centers, with wire-heading as the ultimate form.

What I'm observing in the various FAI debates is a tendency of people to shy away from wire-heading as something the FAI should do. This reluctance is generally not substantiated or clarified with anything other than "clearly, this isn't what we want". This is not, however, clear to me at all.

The utility we get from exploration and creation is an enjoyable mental process that comes with these activities. Once an FAI can rewire our brains at will, we do not need to perform actual exploration or creation to experience this enjoyment. Instead, the enjoyment we get from exploration and creation becomes just another form of pleasure that can be stimulated directly.

If you are a utilitarian, and you believe in shut-up-and-multiply, then the correct thing for the FAI to do is to use up all available resources so as to maximize the number of beings, and then induce a state of permanent and ultimate enjoyment in every one of them. This enjoyment could be of any type - it could be explorative or creative or hedonic enjoyment as we know it. The most energy efficient way to create any kind of enjoyment, however, is to stimulate the brain-equivalent directly. Therefore, the greatest utility will be achieved by wire-heading. Everything else falls short of that.

What I don't quite understand is why everyone thinks that this would be such a horrible outcome. As far as I can tell, these seem to be cached emotions that are suitable for our world, but not for the world of FAI. In our world, we truly do need to constantly explore and create, or else we will suffer the consequences of not mastering our environment. In a world where FAI exists, there is no longer a point, nor even a possibility, of mastering our environment. The FAI masters our environment for us, and there is no longer a reason to avoid hedonic pleasure. It is no longer a trap.

Since the FAI can sustain us in safety until the universe goes poof, there is no reason for everyone not to experience ultimate enjoyment in the meanwhile. In fact, I can hardly tell this apart from the concept of a Christian Heaven, which appears to be a place where Christians very much want to get.

If you don't want to be "reduced" to an eternal state of bliss, that's tough luck. The alternative would be for the FAI to create an environment for you to play in, consuming precious resources that could sustain more creatures in a permanently blissful state. But don't worry; you won't need to feel bad for long. The FAI can simply modify your preferences so you want an eternally blissful state.

Welcome to Heaven.

View more: Next