A Request for Open Problems

25 MrHen 08 May 2009 01:33PM

Open problems are clearly defined problems1 that have not been solved. In older fields, such as Mathematics, the list is rather intimidating. Rationality, on the other, seems to have no list.

While we have all of us here together to crunch on problems, let's shoot higher than trying to think of solutions and then finding problems that match the solution. What things are unsolved questions? Is it reasonable to assume those questions have concrete, absolute answers?

The catch is that these problems cannot be inherently fuzzy problems. "How do I become less wrong?" is not a problem that can be clearly defined. As such, it does not have a concrete, absolute answer. Does Rationality have a set of problems that can be clearly defined? If not, how do we work toward getting our problems clearly defined?

See also: Open problems at LW:Wiki

continue reading »

Average utilitarianism must be correct?

2 PhilGoetz 06 April 2009 05:10PM

I said this in a comment on Real-life entropic weirdness, but it's getting off-topic there, so I'm posting it here.

My original writeup was confusing, because I used some non-standard terminology, and because I wasn't familiar with the crucial theorem.  We cleared up the terminological confusion (thanks esp. to conchis and Vladimir Nesov), but the question remains.  I rewrote the title yet again, and have here a restatement that I hope is clearer.

  • We have a utility function u(outcome) that gives a utility for one possible outcome.  (Note the word utility.  That means your diminishing marginal utility, and all your preferences, and your aggregation function for a single outcome, are already incorporated into this function.  There is no need to analyze u further, as long as we agree on using a utility function.)
  • We have a utility function U(lottery) that gives a utility for a probability distribution over all possible outcomes.
  • The von Neumann-Morgenstern theorem indicates that, given 4 reasonable axioms about U, the only reasonable form for U is to calculate the expected value of u(outcome) over all possible outcomes.  This is why we constantly talk on LW about rationality as maximizing expected utility.
  • This means that your utility function U is indifferent with regard to whether the distribution of utility is equitable among your future selves.  Giving one future self u=10 and another u=0 is equally as good as giving one u=5 and another u=5.
  • This is the same ethical judgement that an average utilitarian makes when they say that, to calculate social good, we should calculate the average utility of the population; modulo the problems that population can change and that not all people are equal.  This is clearer if you use a many-worlds interpretation, and think of maximizing expected value over possible futures as applying average utilitarianism to the population of all possible future yous.
  • Therefore, I think that, if the 4 axioms are valid when calculating U(lottery), they are probably also valid when calculating not our private utility, but a social utility function s(outcome), which sums over people in a similar way to how U(lottery) sums over possible worlds.  The theorem then shows that we should set s(outcome) = the average value of all of the utilities for the different people involved. (In other words, average utilitarianism is correct).  Either that, or the axioms are inappropriate for both U and s, and we should not define rationality as maximizing expected utility.
  • (I am not saying that the theorem reaches down through U to say anything directly about the form of u(outcome).  I am saying that choosing a shape for U(lottery) is the same type of ethical decision as choosing a shape for s(outcome); and the theorem tells us what U(lottery) should look like; and if that ethical decision is right for U(lottery), it should also be right for s(outcome). )
  • And yet, average utilitarianism asserts that equity of utility, even among equals, has no utility.  This is shocking, especially to Americans.
  • It is even more shocking that it is thus possible to prove, given reasonable assumptions, which type of utilitarianism is correct.  One then wonders what other seemingly arbitrary ethical valuations actually have provable answers given reasonable assumptions.

Some problems with average utilitarianism from the Stanford Encyclopedia of Philosophy:

Despite these advantages, average utilitarianism has not obtained much acceptance in the philosophical literature. This is due to the fact that the principle has implications generally regarded as highly counterintuitive. For instance, the principle implies that for any population consisting of very good lives there is a better population consisting of just one person leading a life at a slightly higher level of well-being (Parfit 1984 chapter 19). More dramatically, the principle also implies that for a population consisting of just one person leading a life at a very negative level of well-being, e.g., a life of constant torture, there is another population which is better even though it contains millions of lives at just a slightly less negative level of well-being (Parfit 1984). That total well-being should not matter when we are considering lives worth ending is hard to accept. Moreover, average utilitarianism has implications very similar to the Repugnant Conclusion (see Sikora 1975; Anglin 1977).

(If you assign different weights to the utilities of different people, we could probably get the same result by considering a person with weight W to be equivalent to W copies of a person with weight 1.)

Complex Novelty

26 Eliezer_Yudkowsky 20 December 2008 12:31AM

From Greg Egan's Permutation City:

    The workshop abutted a warehouse full of table legs—one hundred and sixty-two thousand, three hundred and twenty-nine, so far.  Peer could imagine nothing more satisfying than reaching the two hundred thousand mark—although he knew it was likely that he'd change his mind and abandon the workshop before that happened; new vocations were imposed by his exoself at random intervals, but statistically, the next one was overdue.  Immediately before taking up woodwork, he'd passionately devoured all the higher mathematics texts in the central library, run all the tutorial software, and then personally contributed several important new results to group theory—untroubled by the fact that none of the Elysian mathematicians would ever be aware of his work.  Before that, he'd written over three hundred comic operas, with librettos in Italian, French and English—and staged most of them, with puppet performers and audience.  Before that, he'd patiently studied the structure and biochemistry of the human brain for sixty-seven years; towards the end he had fully grasped, to his own satisfaction, the nature of the process of consciousness.  Every one of these pursuits had been utterly engrossing, and satisfying, at the time.  He'd even been interested in the Elysians, once.
    No longer.  He preferred to think about table legs.

Among science fiction authors, (early) Greg Egan is my favorite; of early-Greg-Egan's books, Permutation City is my favorite; and this particular passage in Permutation City, more than any of the others, I find utterly horrifying.

If this were all the hope the future held, I don't know if I could bring myself to try.  Small wonder that people don't sign up for cryonics, if even SF writers think this is the best we can do.

You could think of this whole series on Fun Theory as my reply to Greg Egan—a list of the ways that his human-level uploaded civilizations Fail At Fun.  (And yes, this series will also explain what's wrong with the Culture and how to fix it.)

continue reading »

The Error of Crowds

13 Eliezer_Yudkowsky 01 April 2007 09:50PM

I've always been annoyed at the notion that the bias-variance decomposition tells us something about modesty or Philosophical Majoritarianism.  For example, Scott Page rearranges the equation to get what he calls the Diversity Prediction Theorem:

Collective Error = Average Individual Error - Prediction Diversity

I think I've finally come up with a nice, mathematical way to drive a stake through the heart of that concept and bury it beneath a crossroads at midnight, though I fully expect that it shall someday rise again and shamble forth to eat the brains of the living.

continue reading »

View more: Prev