Note: informally, the point of this paper is to argue against the instinctive "if the AI were so smart, it would figure out the right morality and everything will be fine." It is targeted mainly at philosophers, not at AI programmers. The paper succeeds if it forces proponents of that position to put forwards positive arguments, rather than just assuming it as the default position. This post is presented as an academic paper, and will hopefully be published, so any comments and advice are welcome, including stylistic ones! Also let me know if I've forgotten you in the acknowledgements.

Abstract: In his paper “The Superintelligent Will”, Nick Bostrom formalised the Orthogonality thesis: the idea that the final goals and intelligence levels of agents are independent of each other. This paper presents arguments for a (slightly narrower) version of the thesis, proceeding through three steps. First it shows that superintelligent agents with essentially arbitrary goals can exist. Then it argues that if humans are capable of building human-level artificial intelligences, we can build them with any goal. Finally it shows that the same result holds for any superintelligent agent we could directly or indirectly build. This result is relevant for arguments about the potential motivations of future agents.

 

1 The Orthogonality thesis

The Orthogonality thesis, due to Nick Bostrom (Bostrom, 2011), states that:

• Intelligence and final goals are orthogonal axes along which possible agents can freely vary: more or less any level of intelligence could in principle be combined with more or less any final goal.

It is analogous to Hume’s thesis about the independence of reason and morality (Hume, 1739), but applied more narrowly, using the normatively thinner concepts ‘intelligence’ and ‘final goals’ rather than ‘reason’ and ‘morality’.

But even ‘intelligence’, as generally used, has too many connotations. A better term would be efficiency, or instrumental rationality, or the ability to effectively solve problems given limited knowledge and resources (Wang, 2011). Nevertheless, we will be sticking with terminology such as ‘intelligent agent’, ‘artificial intelligence’ or ‘superintelligence’, as they are well established, but using them synonymously with ‘efficient agent’, artificial efficiency’ and ‘superefficient algorithm’. The relevant criteria is whether the agent can effectively achieve its goals in general situations, not whether its inner process matches up with a particular definition of what intelligence is.

This is a chapter-by-chapter review of Thinking and Deciding by Jonathan Baron (UPenn, twitter). It won't be a detailed summary like badger's excellent summary of Epistemology and the Psychology of Human Judgment, in part because this is a 600-page textbook and so a full summary would be far longer that I want to write here. I'll try to provide enough details that people can seek out the chapters that they find interesting, but this is by no means a replacement for reading the chapters that you find interesting. Every chapter is discussed below, with a brief "what should I read?" section if you know what you're interested in.

We already have a thread for textbook recommendations, but this book is central enough to Less Wrong's mission that it seems like it's worth an in-depth review. I'll state my basic impression of the whole book up front: I expect most readers of LW would gain quite a bit from reading the book, especially newer members, as it seems like a more focused and balanced introduction to the subject of rationality than the Sequences.

Baron splits the book into three sections: Thinking in General, Probability and Belief, and Decisions and Plans.

What does an memetic infection look like? Well, you would encounter something (probably on the internet) that seems very compelling. You think intensely about it for a while, and it spurs you to do something - probably to post something related on the internet. After a while, the meme may not seem that compelling to you anymore, and you wonder why you invested that time and energy. The meme has reproduced itself. For example, Bruce Sterling's response to the 'New Aesthetic' is a paradigmatic example of memetic infection: he encountered it, he found it compelling, he wrote about it, I read about it and now I know about it. (Note that the word 'infection' has a stigma to it, but I don't mean it to be necessarily a bad thing. I will use 'disease' to mean 'infection with bad consequences'.)

Now, let me jump to an apparently unrelated concept - Viral Eukaryogenesis. If I understand correctly, Viral Eukaryogenesis is the theory that eukaryotes (including you and me) are inheritors of a bargain between two kinds of life - metabolic life and viral life, something like the way lichens are a bargain between fungi and algae. The nucleus that characterizes eukaryotes is supposed to be descended from a virus protein shell, and the membrane-fusion proteins that we use for gamete fusion (crucial for sex) are supposed to be descended from viral infection proteins. I am not a biologist, but my understanding of the state of biology is that it is an interesting hypothesis, as yet neither proven nor disproven. However, I'm going to talk as if it were true, because I'm actually trying to make an analogy with memes.

There are goals which can be achieved only by personal exertion and hard work – finishing a university degree, learning a language, mastering a martial art… But there is also a plethora of smaller goals, where small differences in approach and resources can make a huge difference. I’m going to examine how one particular cognitive bias affects execution of small-to-midrange goals, why this bias cannot be realistically overcome on a personal level, and how it can be effectively short-circuited simply by involving other minds.

Do note:  the point I’m making may seem obvious; in my personal experience, and from observation, it is one of those things that are obvious once you know the answer (and one still needs occasional reminders). The solution to the problem is high-impact and available to practically everyone, but remains vastly underused.

The bias in question is availability heuristic. LW has a decent definition:

The availability heuristic judges the probability of events by the ease with which examples come to mind. Sometimes this heuristic serves us well, but the map is not the territory; the frequency with which concepts occur in your thoughts need not reflect the frequency with which they occur in reality. Undue salience, selective reporting, even subtle features of how the human brain stores and recalls memories can distort our perceptions about the probability of events. Because it's easier to recall words by their first letter, people judge words that begin with the letter r to be more frequent than words with r as their third letter, even though in fact, the latter is more frequent. Or selective reporting by the media of dramatic tragedies makes them seem more frequent than more threatening albeit mundane risks.

This topic has been talked about many times on LW, and there is a great deal of academic research as well [1], including seminal texts many here will be familiar with [2]. It's all interesting, but there are two critical points I want to pull up to the forefront.

I - availability heuristic is commonly treated as a simple perceptual bias. In actuality, it is also a choice bias - if you fail to perceive an option, you cannot choose to pursue it. If you do pursue an option, you are likely going to focus on attaining it with cognitively available resources, while missing much better resources that are sitting idle. Similarly, you may waste resources (sometimes to the point of simply giving up) while pursuing a sub-optimal but cognitively available path towards your goal – completely oblivious to much easier roads which are actually available to you.

II - availability heuristic is not really "curable." Sure, it’s good to be aware of it. But we all have limited information about the world. Even if we objectively write down all known relevant factors for some observation or decision, our sample is still going to be at least somewhat biased, and certainly very narrow. Outside our direct areas of expertise, the amount of information we can include into any decision is quite limited; and the number of items our working memory holds while the decision is made is limited yet further.

One obvious mitigation strategy is probably apparent: simply research your desired goal by consulting experts (or the Googlian Oracle). How did other people achieve the goal? What preparations did they undertake? What strategies are recommended?

Good idea, which comes with several limitations.

Much of the information out there is written by people who are unaware of the existence of availability heuristic. The recommendations, however refined, are still usually descriptions of single strategies which happened to work optimally for the person who wrote the article. They could work (and often will), but they may not be the optimal solution to your particular problem set. Furthermore, as I will illustrate soon, the most common strategies you will find are the ones most cognitively available to the most people; an Internet search will, in effect, potentiate the availability heuristic even further, hiding less obvious strategies even further.

But by far the most significant limitation comes from the fact that knowledge does not equal resources. Even if you research an optimal strategy, you may still be unaware of the full scope of resources that are available to you. To avoid abstraction, let’s take a specific example.

Example: crowdsourcing adventure opportunities

A friend of mine has an interesting strategy for increasing the overall awesomeness of her life. Every January 1st, she comes up with a general rule, which she then follows until the end of the year. The rule is modified by common sense (you don't follow it if it will get you into an extraordinarily dangerous situation, or if following it is otherwise prohibitively expensive), but other than that, it has to be followed.

This year, the rule is "when you think or hear of something that causes you to be afraid, go ahead and do it." She's afraid of heights, so the obvious "go skydiving" is on the list from the start. But then, someone mentions flying in an acrobatic aircraft. That gets added to the list. I'm sure you see the general principle.

Seems a bit cheesy, at first glance. But... within the first four months of this year, she went flying in the aforementioned acrobatic aircraft (and a helicopter), learned how to ski, and even rode a damn ostrich. All within four months. She went caving this past weekend. Hang-gliding is firmly scheduled in a few months. And there are other, less glamorous experiences as well, but it's quite a list.[3, 4]

Now, let's say I wanted to do one of those things – say, an acrobatic airplane ride. What comes to mind? I look up places that offer such rides. I would need to travel there (and given the distances involved, get a hotel room as well). Pay the fee. Get to spend about 15-20 minutes being flown around. It's a lot of effort, and the payoff doesn't seem really worth it. Hey, let’s see what other people did! Google, google… a bunch of testimonials about people having a great/awful time taking the aforementioned touristy rides (most significant finding: corkscrews often cause explosive nausea). So I give up.

How did my friend do it? She talks to people about it; asks them for ideas. And soon enough, someone says "yeah, I know a guy who owns an acrobatic plane, wanna ride with him?" And lo and behold, a free ride of much higher quality than touristy nonsense one pays for, plus it's very close to home.

My approach above is the cognitively available one. I'm proceeding towards the goal in accordance with the patterns I've followed previously, when achieving similar goals. I'm thinking about resources that are available to me, personally (my money, my time, etc.). I end up with a suboptimal plan.

Her approach is to crowdsource: throw the desire into the world, and see what others come up with. Many people, with many different resources, ideas, and further links to even more people out there.

Once I started thinking about this, I decided to test this concept. I tried throwing out the acrobatic ride idea to my friends - and lo and behold, a friend of a friend of a friend is going to be flying in this summer. In his acrobatic aircraft. And now, when he gets here, I'm likely to get an hour or so of riding time with him, for free. Just because I asked.

It's a somewhat silly example, of course, but I think it illustrates the point. This friend (of a friend)² is a resource. I was unaware I had this resource, until I asked for it. Finding an acrobatic aircraft owner through personal connections is a strategy that would have never occurred to me (since my availability heuristic informs me that such people are exceedingly rare).

Does this still seem obvious? Many articles were written about the “breakthrough” design of the Apple headquarters – a building made to force people together, to produce conversations between workers in different areas, and interactions between people who think in very different ways. In MIT lore, legends are written about Building 20, a “magical incubator” that has produced an incredible amount of breakthrough technologies and world-class thinkers. One of the main reasons given for this productivity is that many disparate small groups of researchers in a wide range of areas were thrown together in a small space – where they had to interact and talk to each other.

In other words, there is a certain kind of “magic”… in places that force people to simply utilize each other’s cognitive resources, to seek out different ways of thinking, and to avoid falling into cognitively available approaches to the problems they are trying to solve. And the point I’m trying to make is that one doesn’t need to work in Building 20 – just to intentionally maximize the utilization of their own social network (and work on diversifying it as much as possible).

There are people who already utilize their social network to the utmost, and who expand it strategically, adding people just to enhance the diversity of available viewpoints. But I will take a chance, and state that most of us probably don’t. And as a result, we aren’t able to recognize all of the resources available to us, to optimally use those we do recognize, or to realize optimal strategies for approaching our goals. Chances are that most of us could improve the strategy and execution of any given midrange goal – simply by asking around.

_____________________________

(EDIT) Addendum: help-seeking, status, etc.

There is a bit of discussion in the comments regarding some important questions - how and when does seeking help affect status within the group, would seeking help on a regular basis cause people to become uninterested in helping, etc. These may become a basis for a different text in the near future.

But these questions miss an important point here. Sure, asking for help can be a part of the strategy I discuss above, in some cases. In most cases, however, you should not be seeking help. The point of the article here is to simply talk about your goal and your strategy with others. "Involving other minds" does not necessarily require them to take an active helper role in the achievement of your goal.

In the specific example given in the text, I didn't go around saying "hey, I'm looking for help in finding an acrobatic airplane ride." Instead, I would say something like "Riding an acrobatic plane seems like an interesting thing; I'm trying to look into finding an opportunity to do so in the near future." Thoughts, offers and the eventual connection grew organically from the discussions that followed. Sure, the pilot himself is going to be doing me a favor (which I'll eventually repay), but the people who made the connection for me were just having a conversation.

To illustrate further on an example that popped up in the comments: a CEO of a company that always asks for help in making decisions will rapidly lose status (and therefore become ineffective at his or her job). This much is true. But most effective CEOs will organize their companies so that people of varying backgrounds will have to talk (at some point or other) about current company projects and strategies. The CEO doesn't ask for help in making the decision: she requires that her underlings produce ideas and overviews, which then become a basis for making optimal decisions.

_____________________________

References:

 [1] A few recent examples: Hayibor, S., Wasieleski, D.M. (2009). "Effects of the use of availability" Journal of Business Ethics 84: 151–165. Also, Klinger, D., Kudryavtsev, A. (2010). "The availability heuristic and investors' reactions to company-specific events" The Journal of Behavioral Finance 11 (50-65).

[2] Tversky, A., Kahneman, D. (1973) "Availability: A heuristic for judging frequency and probability" Cognitive Psychology 5 (1): 207–233.

[3] She's blogging her progress through this year, and the whole thing is highly recommended; for sheer hilarity as much as for some very interesting insights.

[4] One could write an excellent text on goal-setting strategies around this example. The rule is simple and absolute ("never have any chips, cookies or other snacks available at home" is easier to follow and will in most cases lead to a greater weight loss than an intricate diet), and it is overarching (applies everywhere in life, not only to some particular times and places, making lawyering around the rule much more difficult). If you are going to set rules for yourself, this is the way to do it. But since I'm writing a loose set of texts on a completely different topic, this footnote will be all I have to say on that topic.

 

For the How to Run a Successful Less Wrong Meetup booklet, I'm looking for information about how to better build a social group and foster a feeling of community. Since this bit is probably of general interest, I'm posting it here.

If you want to make the members of the group like each other more and feel more like a group, synchronized actions may be one of the easiest ways of achieving this goal. Anthropologists have long known the community-building effect of dancing:

As the dancer loses himself in the dance, as he becomes absorbed in the unified community, he reaches a state of elation in which he feels himself filled with an energy of force immensely beyond his ordinary state . . . finding himself in complete and ecstatic harmony with all the fellow-members of his community, experiences a great increase in his feelings of amity and attachment towards them. (Radcliffe-Brown 1933/1948, quoted in Kesebir 2011)

Armies around the world utilize the same effect to foster a feeling of unison through repeated drills:

Words are inadequate to describe the emotion aroused by the prolonged movement in unison that drilling involved. A sense of pervasive well-being is what I recall; more specifically, a strange sense of personal enlargement; a sort of swelling out, becoming bigger than life, thanks to participation in collective ritual. (McNeill 1995, quoted in Kesebir 2011)

Wiltermuth & Heath (2009) summarize some of the research on the topic:

The idea that synchronous movement improves group cohesion has old roots. As historian William H. McNeill suggests, armies, churches, and communities may have all benefited, intentionally or unintentionally, from cultural practices that draw on ‘‘muscular bonding,’’ or physical synchrony, to solidify ties between members (McNeill, 1995). This physical synchrony, which occurs when people move in time with one another, has been argued to produce positive emotions that weaken the boundaries between the self and the group (Ehrenreich, 2006; Hannah, 1977), leading to feelings of collective effervescence that enable groups to remain cohesive (Durkheim, 1915/1965; Haidt, Seder, & Kesebir, in press; Turner, 1969/1995). Andaman Islanders have been said to become ‘‘absorbed in the unified community’’ through dance (Radcliffe-Brown, 1922, p. 252). Similar observations have been made of Carnival revelers (Ehrenreich, 2006), and ravers dancing to beat-heavy music (Olaveson, 2004). Moreover, Haidt et al. (in press) have argued that people must occasionally lose themselves in a larger social organism to achieve the highest levels of individual well-being.

Some recent findings on the topic include:

Wiltermuth & Heath (2009): Synchronous activity in the form of walking around a campus in step causes people to be more likely to make decisions requiring trust and to self-report stronger feelings of trust and connectedness with others. Singing in synchrony, even if the song is an out-group anthem ("O Canada", when the subjects were USA residents), causes more trust and and greater feelings of being on the same team, as well as an increased willingness to cooperate in a public goods game.

Kirschner & Tomasello (2010): "Given that in traditional cultures music making and dancing are often integral parts of important group ceremonies such as initiation rites, weddings or preparations for battle, one hypothesis is that music evolved into a tool that fosters social bonding and group cohesion, ultimately increasing prosocial ingroup behavior and cooperation. Here we provide support for this hypothesis by showing that joint music making among 4-year-old children increases subsequent spontaneous cooperative and helpful behavior, relative to a carefully matched control condition with the same level of social and linguistic interaction but no music."

Valdesolo, Ouyang & DeSteno (2010): Synchronous rocking increases perceptions of similarity and connectedness. The subjects were given the task of holding the opposite ends of a 12 × 14 wooden labyrinth with both hands and guiding a steel ball through it together. The subjects in the synchronous rocking condition performed better than the subjects in the asynchronous rocking condition.

Valdesolo & DeSteno (2011): Subjects who are told to tap the beats they hear in an audio clip, and are paired with a confederate who has been instructed to synchronize his tapping with the participant’s, tend to find like the confederate more and consider him more similar to themselves. The confederate being assigned an unfair task then evokes more feelings of compassion, and the subjects are more likely to help him, even at a cost to themselves.

The implication for meetup groups, as well as any other groups that might want to make their members like each other more, seems clear: spend some time singing and dancing together, possibly in the form of drinking songs if people are too self-conscious to sing while sober. Just make sure that any non-drinkers don't feel excluded. If all else fails, you can always march around the city while chanting "doom doom DOOM DOOM". (If anybody asks, you can say that you're testing a scientific hypothesis about group bonding, and ask if they'd want to join in.)

References

Kesebir, S. (2011) The Superorganism Account of Human Sociality: How and When Human Groups Are Like Beehives (ungated version). Personality and Social Psychology Review.

Kirchner, S. & Tomasello, M. (2010) Joint music making promotes prosocial behavior in 4-year-old children. Evolution and Human Behavior 31, 354–364. 

McNeill, W.H. (1995) Keeping together in time: Dance and drill in human history. Cambridge, MA: Harvard University Press.

Radcliffe-Brown, A. R. (1948) The Andaman Islanders. Glencoe, IL: Free Press.

Valdesolo, P. & DeSteno, D. (2011) Synchrony and the Social Tuning of Compassion. Emotion, vol. 11, no. 2, 262–266. 

Valdesolo, P. & Ouyang, J. & DeSteno, D. (2010) The rhythm of joint action: Synchrony promotes cooperative ability. Journal of Experimental Social Psychology, vol. 46, no. 4, 693–695.

Wiltermuth, S.S. & Heath, C. (2009): Synchrony and Cooperation. Psychological Science, vol. 20, no. 1.

Or: Formalizing Timeless Decision Theory

Previously:

0. Decision Theories: A Less Wrong Primer
1. The Problem with Naive Decision Theory
2. Causal Decision Theory and Substitution

Summary of Post: Have you ever wanted to know how (and whether) Timeless Decision Theory works? Using the framework from the last two posts, this post shows you explicitly how TDT can be implemented in the context of our tournament, what it does, how it strictly beats CDT on fair problems, and a bit about why this is a Big Deal. But you're seriously going to want to read the previous posts in the sequence before this one.

We've reached the frontier of decision theories, and we're ready at last to write algorithms that achieve mutual cooperation in Prisoner's Dilemma (without risk of being defected on, and without giving up the ability to defect against players who always cooperate)! After two substantial preparatory posts, it feels like it's been a long time, hasn't it?

But look at me, here, talking when there's Science to do...