Discussion: Which futures are good enough?

5 WrongBot 24 February 2013 12:06AM
Thirty years from now, a well-meaning team of scientists in a basement creates a superintelligent AI with a carefully hand-coded utility function. Two days later, every human being on earth is seamlessly scanned, uploaded and placed into a realistic simulation of their old life, such that no one is aware that anything has changed. Further, the AI had so much memory and processing power to spare that it gave every single living human being their own separate simulation.

Each person lives an extremely long and happy life in their simulation, making what they perceive to be meaningful accomplishments. For those who are interested in acquiring scientific knowledge and learning the nature of the universe, the simulation is accurate enough that everything they learn and discover is true of the real world. Every other pursuit, occupation, and pastime is equally fulfilling. People create great art, find love that lasts for centuries, and create worlds without want. Every single human being lives a genuinely excellent life, awesome in every way. (Unless you mind being simulated, in which case at least you'll never know.)

I offer this particular scenario because it seems conceivable that with no possible competition between people, it would be possible to avoid doing interpersonal utility comparison, which could make Mostly Friendly AI (MFAI) easier. I don't think this is likely or even worthy of serious consideration, but it might make some of the discussion questions easier to swallow.


1. Value is fragile. But is Eliezer right in thinking that if we get just one piece wrong the whole endeavor is worthless? (Edit: Thanks to Lukeprog for pointing out that this question completely misrepresents EY's position. Error deliberately preserved for educational purposes.)

2. Is the above scenario better or worse than the destruction of all earth-originating intelligence? (This is the same as question 1.)

3. Are there other values (besides affecting-the-real-world) that you would be willing to trade off?

4. Are there other values that, if we traded them off, might make MFAI much easier?

5. If the answers to 3 and 4 overlap, how do we decide which direction to pursue?

[LINK] Levels of Ethics

1 WrongBot 07 February 2011 01:41AM

I've resumed blogging For Real This Time™, starting with an introductory overview of the distinction between metaethics and normative ethics.

Should I cross-post it to LessWrong? Should I link or cross-post future blogging about metaethics and other LW-relevant topics? Is it rubbish? Inquiring minds (mostly mine) need to know!

Forager Anthropology

11 WrongBot 28 July 2010 05:48AM

(This is the second post in a short sequence discussing evidence and arguments presented by Christopher Ryan and Cacilda Jethá's Sex at Dawninspired by the spirit of Kaj_Sotala's recent discussion of What Intelligence Tests MissIt covers Part II: Lust in Paradise and Part III: The Way We Weren't.)

Forager anthropology is a discipline that is easy to abuse. It relies on unreliable first-hand observations of easily misunderstood cultures that are frequently influenced by the presence of modern observers. These cultures are often exterminated or assimilated within decades of their discovery, making it difficult to confirm controversial claims and discoveries. But modern-day foraging societies are the most direct source of evidence we have about our pre-agricultural ancestors; in many ways, they are agriculture's control group, living in conditions substantially similar to the ones under which our species evolved. The standard narrative of human sexual evolution ignores or manipulates the findings of forager anthropology to support its claims, and this is no doubt responsible for much of its confused support.

Steven Pinker is one of the most prominent and well-respected advocates of the standard narrative, both on Less Wrong and elsewhere. Eliezer has referenced him as an authority on evolutionary psychology. One commenter on the first post in this series claimed that Pinker is "the only mainstream academic I'm aware of who visibly demonstrates the full suite of traditional rationalist virtues in essentially all of his writing." Another cited Pinker's claim that 20-60% of hunter-gatherer males were victims of lethal human violence ("murdered") as justification for a Malthusian view of human nature. 

That 20-60% number comes from a claim about war casualties in a 2007 TED talk Pinker gave on "the myth of violence", for which he drew upon several important findings in forager anthropology. (The talk is based on an argument presented in the third chapter of The Blank Slate; there is a text version of the talk available, but it omits the material on forager anthropology that Ryan and Jethá critique.)

At 2:45 in the video Pinker displays a slide which reads

Until 10,000 years ago, humans lived as hunter-gatherers, without permanent settlements or government.

He also points out that modern hunter-gatherers are our best evidence for drawing conclusions about those prehistoric hunter-gatherers; in both these statements he is in accordance with nearly universal historical, anthropological, and archaeological opinion. Pinker's next slide is a chart from The Blank Slate, originally based on the research of Lawrence Keeley. Sort of. It is labeled as "the percentage of male deaths due to warfare," with bars for eight hunter-gatherer societies that range from approximately 15-60%. The problem is that of these eight cultures, zero are migratory hunter-gatherers.

continue reading »

Against the standard narrative of human sexual evolution

7 WrongBot 23 July 2010 05:28AM

(This post is the beginning of a short sequence discussing evidence and arguments presented by Christopher Ryan and Cacilda Jethá's Sex at Dawn, inspired by the spirit of Kaj_Sotala's recent discussion of What Intelligence Tests Miss. It covers Part I: On the Origin of the Specious.)

Sex at Dawn: The Prehistoric Origins of Modern Sexuality was first brought to my attention by a rhapsodic mention in Dan Savage's advice column, and while it seemed quite relevant to my interests I am generally very skeptical of claims based on evolutionary psychology. I did eventually decide to pick up the book, primarily so that I could raid its bibliography for material for an upcoming post on jealousy management, and secondarily to test my vulnerability to confirmation bias. I succeeded in the first and failed in the second: Sex at Dawn is by leaps and bounds the best evolutionary psychology book I've read, largely because it provides copious evidence for its claims.1 I mention the strength of my opinion as a disclaimer of sorts, so that careful readers may take the appropriate precautions.


The book's first section focuses on the current generally accepted explanation for human sexual evolution, which the authors call "the standard narrative." It's an explanation that should be quite familiar to regular LessWrong readers: men are attracted to fertile-appearing women and try to prevent them from having sex with other men so as to confirm the paternity of their offspring; women are attracted to men who seem like they will be good providers for their children and try to prevent them from forming intimate bonds with other women so as to maintain access to their resources.

continue reading »

Some Thoughts Are Too Dangerous For Brains to Think

15 WrongBot 13 July 2010 04:44AM
[EDIT - While I still support the general premise argued for in this post, the examples provided were fairly terrible. I won't delete this post because the comments contain some interesting and valuable discussions, but please bear in mind that this is not even close to the most convincing argument for my point.]
A great deal of the theory involved in improving computer and network security involves the definition and creation of "trusted systems", pieces of hardware or software that can be relied upon because the input they receive is entirely under the control of the user. (In some cases, this may instead be the system administrator, manufacturer, programmer, or any other single entity with an interest in the system.) The only way to protect a system from being compromised by untrusted input is to ensure that no possible input can cause harm, which requires either a robust filtering system or strict limits on what kinds of input are accepted: a blacklist or a whitelist, roughly.
One of the downsides of having a brain designed by a blind idiot is that said idiot hasn’t done a terribly good job with limiting input or anything resembling “robust filtering”. Hence that whole bias thing. A consequence of this is that your brain is not a trusted system, which itself has consequences that go much, much deeper than a bunch of misapplied heuristics. (And those are bad enough on their own!)
In discussions of the AI-Box Experiment I’ve seen, there has been plenty of outrage, dismay, and incredulity directed towards the underlying claim: that a sufficiently intelligent being can hack a human via a text-only channel. But whether or not this is the case (and it seems to be likely), the vulnerability is trivial in the face of a machine that is completely integrated with your consciousness and can manipulate it, at will, towards its own ends and without your awareness.
Your brain cannot be trusted. It is not safe. You must be careful with what you put into it, because it  will decide the output, not you.
continue reading »

Unknown knowns: Why did you choose to be monogamous?

48 WrongBot 26 June 2010 02:50AM

Many of us are familiar with Donald Rumsfeld's famous (and surprisingly useful) taxonomy of knowledge:

There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we now know we don’t know. But there are also unknown unknowns. These are things we do not know we don’t know.

But this taxonomy (as originally described) omits an important fourth category: unknown knowns, the things we don't know that we know. This category encompasses the knowledge of many of our own personal beliefs, what I call unquestioned defaults. For example, most modern Americans possess the unquestioned default belief that they have some moral responsibility for their own freely-chosen actions. In the twelfth century, most Europeans possessed the unquestioned default belief that the Christian god existed. And so on. These unknown knowns are largely the products of a particular culture; they require homogeneity of belief to remain unknown.

By definition, we are each completely ignorant of our own unknown knowns. So even when our culture gives us a fairly accurate map of the territory, we'll never notice the Mercator projection's effect. Unless it's pointed out to us or we find contradictory evidence, that is. A single observation can be all it takes, if you're paying attention and asking questions. The answers might not change your mind, but you'll still come out of the process with more knowledge than you went in with.

continue reading »