Note: The following is a draft of a paper written with an audience of philosophers in mind. It focuses on answering objections to AI likely to be made by contemporary philosophers, but it is still likely to be of interest to readers of LessWrong for obvious reasons, and I've tried to avoid assuming any specialized philosophical background.

The title of this paper probably sounds a little strange. Philosophy is generally thought of as an armchair discipline, and the question of whether robots could take all our jobs doesn’t seem like a question that can be settled from the armchair. But it turns out that when you look at this question, it leads you to some other questions that philosophers have had quite a bit to say about.

Some of these other questions are conceptual. They seem like they could in fact be answered from the armchair. Others are empirical but very general. They seem like they require going out and looking at the world, but they’re not about specific technologies. They’re about how the universe works, how the mind works, or how computers work in general. It’s been suggested that one of the distinctive things about philosophical questions is their level of generality. I don’t know whether that’s true, but I do know that some of these general empirical questions are ones that philosophers have had quite a bit to say about.

The line of reasoning in this paper is similar to arguments discussed by Alan Turing (1950), Hubert Dreyfus (1992), and David Chalmers (2010). I won’t say much about those discussions, though, for reasons of space and also because I’ll frame the discussion a bit differently. I want to avoid debates about what “intelligence” is, what “information processing” is, or what it would mean to say the brain is a machine. Hence the focus on machines taking human jobs. I should mention that I’m not the first to suggest this focus; after writing the first draft of this paper I found out that one AI researcher had already proposed replacing the “Turing test” with an “employment test” (Nilsson 2005).

Here, I’m going to assume no job has “done by a human” as part of its definition. I realize that in the future, there may be demand for having jobs specifically done by humans. People might want to be served by human bartenders even if robot bartenders do just as good of a job, in the same way that some people prefer handmade goods even when mass-produced ones are cheaper for the same or better quality (Doctorow 2011). But having acknowledged this issue, I’m going to spend the rest of the paper ignoring it.  

I've noticed that quite a few people are interested in fostering communities -- both creating communities and improving them to make them work together.  But how do we go about actually doing this?  What's there to community that we can foster and build upon?  What makes a community thrive, and how do we take advantage of this to make and/or improve communities?

To answer these questions, I turned to two books:

The first is The Penguin and The Leviathan: How Cooperation Triumphs Over Self-Interest by Yochai Benkler.  Benkler, in writing about cooperative systems (Penguins, named after the Linux Penguin) and hierarchical systems (Leviathans, named after Thomas Hobbes's The Leviathan), studies the psychology, economics, and political science of cooperation and helps explain what makes communities stick.

The second is Liars and Outliers: Enabling the Trust that Society Needs to Thrive by Bruce Schneier.  Schneier studies trust and cooperation from a dizzying variety of sciences (psychology, biology, economics, anthropology, computer science, and political science).  Schneier's ultimate game is figuring out what is preventing society from falling apart, and that can be applied to building communities.

Let's see what they got.

Communities Need Cooperation

Schneier and Benkler both paint a view of human nature that is different than what is commonly thought, but what has emerged from the sciences: People are both self-interested and other-interested, different people will have different balances of each, and within each person these two goals can often conflict.  Additionally, the "other-interested" aspect can be multiple and occasionally conflicting allegiances, such as to one's family, to one's neighborhood, to one's country, to one's venture philanthropy club, etc.

What's unique to all communities is that they involve people who have set aside some of their immediate self-interest to work together.  For instance, when we work together in a group, I definitely don't beat you over the head and steal your lunch money, and I don't usually attempt to free ride and get you to do the group work for me, but we mutually work to solve communal problems and share in the benefits of community.

Public Goods and Free Riders

An example of how psychology has sought to simplify and simulate a community is through what's called "The Public Goods Game".  In this game, a group of about ten participants are each sat down, and given $10 each to start with.  The game is then played for several rounds, and in each round all participants get to put a certain secret amount of their money into a collective pot.  The experimenters then look at the pot, double the amount of money inside it, and redistribute the result evenly to all the players.  For added bonus, the experimenters inform all participants that they get to walk away with their winnings after the game is over.

If everyone went perfectly with the community, each player would see their money double each round.  But the wrinkle is that if people don't contribute at all to the pot, then they stand to gain even more money from the results of everyone else's contributions.  This is called the free rider problem: there is a tension between wanting to contribute to the pot for the good of yourself and the good of the group as a whole and refraining from contributing so that you benefit even more.

The Free Rider Problem and The Collective Action Problem

But the tension can result in further disaster, for imagine everyone decides to be a free rider and defect from the group -- now, no money goes in the pot at all, and everyone ends with the $10 they start with.  This gets worse when we imagine some other real-life scenarios -- for instance, that of fishermen in a lake.

The fishermen can either choose to fish normally or overfish.  If all the fishermen overfish, they stand to deplete the lake and all fishermen lose their jobs.  However, if just a few fishermen overfish, they get the benefit of added fish to sell, and the lake can handle the slight increase in load.  So this tension is to be the fisherman that wins most by personally overfishing, while not collectively depleting the entire lake.  Such problems are called collective action problems -- people do well individually by defecting but do worse collectively if everyone defects.  The result of a collective action problem ending in disaster is called the tragedy of the commons.

The Community Solution

So what's the solution to these problems?  Benkler proposes two models for dealing with them -- employing the Leviathan and placing lots of regulations on overfishing and enforcing them with strict punishments, or employing the Penguin and creating a community that deals with these problems collectively and in a self-policing way.

It turns out that certain problems are best dealt with differing combinations of Leviathan and Penguin models, but most problems need lots of community just because it can be difficult to figure out who is going against the community, and communities have more freedom for their participants.  At the same time, if there are too many would-be defectors a community can never get off the ground.

Communities need cooperation to work.  So how can we get this cooperation to fly?

The Four Pressures of Cooperation

Bruce Schneier notes that normally we don't think through these free rider problems and try to scheme our way through them -- we just cooperate, instinctively.  We don't assume people will rip us off, and we usually don't rip other people off -- that's just how we are.  But why?  Schneier suggests that cooperation can be fostered and maintained through four different pressures, though differing kinds and amounts of pressure apply to different situations, and getting the balance of pressures right is a key part of his book:

1.) Moral Pressures: Many, but not all of us, have various moral feelings that lead us to want to cooperate.  It could be as easy as feeling incredibly guilty when we defect against our friends, or as complex as subscribing to an abstract principle of justice.  For most of us, it's a general feeling that cooperating is the "right thing to do" and defecting for our own personal self-interest is "wrong", and we just don't want to do it.  Schneier and Benkler both find that moral pressures compel cooperation a surprising amount of the time.

2.) Reputational Pressures: Another part about living in a community for a long time is that you have a reputation to live by.  Defect against the community and you may win a few times, but then people start to notice and start working to stop you.  They might refuse you friendship or other things you want, or even kick you out of the community altogether!  Benkler finds that many communities can thrive on reputation alone, like eBay, Amazon, or Reddit.

3.) Institutional Pressures: Morals and reputation aren't the end of it though; many communities make specific, codified norms and enforce them with specific, codified punishments.  These pressures are laws, and the fear of breaking the law, being caught, and getting the punishment can often further spur cooperation.  Best yet, the community can often get together and agree to these norms, realizing it is in their individual benefit to force themselves and the rest of the community to play along, as to avoid tragedies of the commons.

4.) Security Pressures: Lastly, there are always going to be a few people who put morals, reputation, and laws aside and try to defect anyway.  For these, we hope to stop them in their tracks or make their jobs more difficult, by using complex security systems.  It can be as simple as a security camera or anti-theft radio, or as complex as Fort Knox.  Security is a double plan: it first attempts to raise the costs of defection; by making it physically harder to defect, one is less tempted to do so.  It then attempts to better catch and apprehend those who still try.

Your Reason for Joining; Your Reason for Staying

Remember these pressures don't all work for the same problems -- it may be proper to use security and institutional pressures to stop someone from overfishing, but not from intentionally cutting the cake so they get to eat the bigger slice.  Moral and reputational pressures seem to be more encompassing, but they are also more easily defeated -- people with less of a moral compass can often wander from community to community, wrecking small amounts of havoc and never getting caught or punished.

Benkler suggests another way to get people to buy into a community and not defect against it -- make it clear that being part of the community is something they really want.  Whether your joining a community or forced into one (family, country, etc.), the community will be more likely to thrive.

Four Ways to Bond

But why might one want to join or stay in a community?  For many, the answer is the intangibles -- they feel a sense of belonging, friendship, and group cohesion that creates an empathetic attachment and makes people want to play by the rules of the group.  For others, the answer is the tangibles -- the group may have a stated mission statement that is important to the person, or belonging in the group might confer a specific benefit.  People might even belong for a mix of tangibles and intangibles, plus a natural tendency to want to join groups.

But how do we foster these bonds?  Benkler has his own set of four things, suggesting that group identity can be fostered through a combination of four means:

1.) Fairness: The community needs to be fair -- people need to all contribute more or less equally, or at least have genuine intentions to put in equal effort, and the benefits of the group need to be spread among all participants more or less evenly, or in a fair proportion to how much the participant puts in.

2.) Autonomy: The community needs to not demand too much, and make sure to compensate quickly and generously for special sacrifices.  There are inherent costs to joining and staying with a group, and costs for cooperating with the group -- one doesn't just give up the self-interested benefits of defection, but rather must pay additional costs to maintain their group status.  Being aware of and addressing these costs are important.  In short, the group must respect their members as individuals.

3.) Democracy: The community also needs to accept (with fairness and autonomy) the input of all the members.  Group norms should be developed by a vote, with weight given on building consensus as much as possible, and with understanding the reasons why people might not like the consensus.  Not only does having input make it more likely people's preferences will be taken into account, lowering the costs of cooperation, but having input makes people feel more group cohesion and belonging.

4.) Communication: During times when formal votes aren't taken, the community also needs to be consistently (but not constantly) talking about how the group is doing, and checking in with members who might be feeling left out.  Just like democracy, group cohesion is built through communication, and communication lowers the costs of cooperation.  It's best when resolving disputes is not dictatorial, like in a court of law, but rather cooperative, like in an arbitration.

Looking Back to the Public Goods Game

To demonstrate these four points, Benkler draws on many real-world examples, such as policies of various companies, and interactions on the internet.  He also draws on returning back to our simple-community-in-the-lab, the Public Goods Game, for additional confirmation, and its worth seeing how these things play out.

In the original Public Goods game, contributions to the pot were made anonymously and no-one was allowed to talk or communicate.  Typically, a fair amount of people would cooperate in the beginning (generally, people contribute about 70% of their share), but starts to drop as people see that others aren't contributing.  They start to feel like suckers, and the fairness starts to kick in.

A Different Game

However, variants of the Public Goods game offer ways out.  When participants were allowed to talk to each other, contributions rose (communication).  Likewise, when participants were allowed to use some of their money to punish those who didn't contribute (say, pay $3 to prevent someone from getting their share this round if they didn't cooperate last round), people would do so.  

Even the simple act of making the contributions public increased cooperation, drawing on reputation.  Sometimes small fines were imposed on those who didn't cooperate (institutional pressures) which brought up cooperation, and these fines worked especially well when the group got to vote on how high they would be (democracy).

Lastly, helping frame the game would help -- those who were told they were taking place in a "Community Game" were far more likely to contribute to the pot and keep contributing than those who were told they were taking place in a "Wall Street Game".  By reminding people they are in a community, people thought more about their community norms, and felt more group cohesion, and were more likely to trust others.

Conclusions

Ultimately, creating communities is all about fostering cooperation, and you foster cooperation by ensuring that there is mutual trust and some sort of way to prevent defectors from taking advantage of the system.  People often naturally don't want to defect, but will do so if they think others will take advantage of them first.

Social Pressures

But how do we foster this trust?  The first step is to make use of our social pressures when and to the amount that's appropriate -- relying on empathetic and moral norms, reputation, institutionalized laws, and security systems -- and being sure to get the balance right.  For small communities, this probably just needs to be a set of agreed norms, and ensuring that the norms are properly and responsibly enforced.

The Benefits of Joining

The second part is while implementing the first step, we should keep in mind why people are joining or staying in the first place, and make sure to provide a community where the benefits of joining -- both the tangibles and intangibles -- are present and apparent.  We should acknowledge the costs of cooperating, and make sure the benefits are there to foster group loyalty and belonging.

An Effective Community

While implementing, it's important to keep in mind that communities should also be fair, respect the autonomy and individuality of the members, give members input through democracy, and foster lots of communication about how things are going.  We should also keep a keen eye to how things are framed, while not going overboard on it or lying.

The End Reward

But when we accomplish communities, the rewards are pretty great -- not only do we avoid free riders and the tragedy of the commons, but we ourselves get to take advantage of communities that are more productive than the individuals alone, and secure the feelings of belonging to a group we enjoy.

-

Until recently, I hadn't paid much attention to Pomodoro, though I've heard of it for a few years now. "Uncle Bob" Martin seemed to like it, and he's usually worth paying attention to in such matters. However, it mostly seemed to me like a way of organizing a variety of tasks and avoiding procrastination, and I've never had much trouble with that.

However after the January CFAR workshop suggested it in passing, I decided to give it a try; and I realized I had it all wrong. Pomodoros aren't (for me) a means of avoiding procrastination or dividing time among projects. They're a way of blasting through Ugh fields.

The Pomodoro technique is really simple compared to more involved systems like Getting Things Done (GTD). Here it is:

1. Set a timer for 25 minutes
2. Work on one thing for that 25 minutes, nothing else. No email, no phone calls, no snack breaks, no Twitter, no IM, etc.
3. Take a five minute break
4. Pick a new project, or the same project, if you prefer.
5. Repeat

That's pretty much it. You can buy a book or a special timer for this; but there's really nothing else to it. It takes longer to explain the name than the technique. (When Francesco Cirillo invented this technique in the 1980s, he was using an Italian kitchen timer shaped like a tomato. Pomodoro is Italian for tomato.)

I got interested in Pomodoro when I realized I could use it to clean my office/desk/apartment. David Allen's GTD system appealed to me, but I could never maintain it, and the 2+ days it needed to get all the way to a clean desk was always a big hurdle to vault. However, spending 25 minutes at a time, followed by a break and another project seemed a lot more manageable.

I tried it, and it worked. My desk stack quickly shrunk, not to empty, but at least to a place where an accidental elbow swing no longer launched avalanches of paper onto the floor as I typed.

So I decided to try Pomodoro on my upcoming book. The publisher was using a new authoring system and template that I was unfamiliar with. There were a dozen little details to figure out about the new system--how to check out files in git, how to create a section break, whether to use hard or soft wrapping, etc.--and I just worked through them one by one. 25 minutes later I'd knocked them all out, and was familiar enough with the new system to begin writing in earnest. I didn't know everything about the software, but I knew enough that it was no longer averting. Next I used 25 minutes on a chapter that was challenging me, and Pomodoro got me to the point where I was in the flow.

That's when I realized that Pomodoro is not a system for organizing time or avoiding procrastination (at least not for me). What it is, is an incredibly effective way to break through tasks that look too hard: code you're not familiar with, an office that's too cluttered, a chapter you don't know how to begin.

The key is that a Pomodoro forces you to focus on the unfamiliar, difficult, aversive task for 25 minutes. 25 minutes of focused attention without distractions from other, easier tasks is enough to figure out many complex situations or at least get far enough along that the next step is obvious.

Here's another example. I had a task to design a GWT widget and plug it into an existing application, and I have never done any work with GWT. Every time I looked at the frontend application code, it seemed like a big mess of confused, convoluted, dependency injected, late bound, spooky-action-at-a-distance spaghetti. Now doubtless there wasn't anything fundamentally more difficult about this code than the server side code I have been writing; and if my career had taken just a slightly different path over the last six years, frontend GWT code might be my bread and butter. But my career didn't take that path, and this code was a big Ugh field for me. So I set the Pomodoro timer on my smartphone and started working. Did I finish? No, but I got started, made progress, and proved to myself that GWT wasn't all that challenging after all. The widget is still difficult enough and GWT complex enough that I may need several more Pomodoros to finish the job, but I did get way further and learn more in 25 minutes of intense focus than I would have done in a day or even a week without it.

I don't use the Pomodoro technique exclusively. Once I get going on a project or a chapter, I don't need the help; and five minute breaks once I'm in the flow just distract me. So some days I just do 1 or 2 or 0 Pomodoros, whatever it takes to get me rolling again and past the blocker.

I also don't know if this works for genuinely difficult problems. For instance, I don't know if it will help with a difficult mathematical proof I've been struggling with for months (though I intend to find out). But for subjects that I know I can do, but can't quite figure out how to do, or where to start, the power of focusing 25 minutes of real attention on just that one problem is astonishing.

Note: posting this in Main rather than Discussion in light of recent discussion that people don't post in Main enough and their reasons for not doing so aren't necessarily good ones. But I suspect I may be reinventing the wheel here, and someone else has in fact gotten farther on this problem than I have. If so, I'd be very happy if someone could point me to existing discussion of the issue in the comments.

tldr; Gambling-based arguments in the philosophy of probability can be seen as depending on a convenient simplification of assuming people are far more willing to gamble than they are in real life. Some justifications for this simplification can be given, but it's unclear to me how far they can go and where the justification starts to become problematic.

In "Intelligence Explosion: Evidence and Import," Luke and Anna mention the fact that, "Except for weather forecasters (Murphy and Winkler 1984), and successful professional gamblers, nearly all of us give inaccurate probability estimates..." When I read this, it struck me as an odd thing to say in a paper on artificial intelligence. I mean, those of us who are not professional accountants tend to make bookkeeping errors, and those of us who are not math, physics, engineering, or economics majors make mistakes on GRE quant questions that we were supposed to have learned how to do in our first two years of high school. Why focus on this particular human failing?

A related point can be made about Dutch Book Arguments in the philosophy of probability. Dutch Book Arguments claim, in a nutshell, that you should reason in accordance with the axioms of probability because if you don't, a clever bookie will be able to take all your money. But another way to prevent a clever bookie from taking all your money is to not gamble. Which many people don't, or at least do rarely.

Dutch Book Arguments seem to implicitly make what we might call the "willing gambler assumption": everyone always has a precise probability assignment for every proposition, and they're willing to take any bet which has a non-negative expected value given their probability assignments. (Or perhaps: everyone is always willing to take at least one side of any proposed bet.) Needless to say, even people who gamble a lot generally aren't that eager to gamble.

So how does anyone get away with using Dutch Book arguments for anything? A plausible answer comes from a joke Luke recently told in his article on Fermi estimates:

Milk production at a dairy farm was low, so the farmer asked a local university for help. A multidisciplinary team of professors was assembled, headed by a theoretical physicist. After two weeks of observation and analysis, the physicist told the farmer, "I have the solution, but it only works in the case of spherical cows in a vacuum."

If you've studied physics, you know that physicists don't just use those kinds of approximations when doing Fermi estimates; often they can be counted on to yield results that are in fact very close to reality. So maybe the willing gambler assumption works as a sort of spherical cow, that allows philosophers working on issues related to probability to generate important results in spite of the unrealistic nature of the assumption.

Some parts of how this would work are fairly clear. In real life, bets have transaction costs; they take time and effort to set up and collect. But it doesn't seem too bad to ignore that fact in thought experiments. Similarly, in real life money has declining marginal utility; the utility of doubling your money is less than the disutility of losing your last dollar. In principle, if you know someone's utility function over money, you can take a bet with zero expected value in dollar terms and replace it with a bet that has zero expected value in utility terms. But ignoring that and just using dollars for your thought experiments seems like an acceptable simplification for convenience's sake.

Even making those assumptions so that it isn't definitely harmful to accept bets with zero expected (dollar) value, we might still wonder why our spherical cow gambler should accept them. Answer: because if necessary you could just add one penny to the side of the bet you want the gambler to take, but always having to mention the extra penny is annoying, so you may as well assume the gambler takes any bet with non-negative expected value rather than require positive expected value.

Another thing that keeps people from gambling more in real life is the principle that if you can't spot the sucker in the room, it's probably you. If you're unsure whether an offered bet is favorable to you, the mere fact that someone is offering it to you is pretty strong evidence that it's in their favor. One way to avoid this problem is to stipulate that in Dutch Book Arguments, we just assume the bookie doesn't know anything more about whatever the bets are about than the person being offered the bet, and the person being offered the bet knows this. The bookie has to construct her book primarily based on knowing the propensities of the other person to bet. Nick Bostrom explicitly makes such an assumption in a paper on the sleeping beauty problem. Maybe other people explicitly make this assumption, I don't know.

In this last case, though, it's not totally clear whether limiting the bookie's knowledge is all you need to bridge the gap between the willing gambler assumption and how people behave in real life. In real life, people don't often make very exact probability assignments, and may be aware of their confusion about how to make exact probability assignments. Given that, it seems reasonable to hesitate in making bets (even if you ignore transaction costs and declining marginal utility and know that the bookie doesn't know any more about the subject of the bet than you do), because you'd still know the bookie might be trying to exploit your confusion over how to make exact probability assignments.

At an even simpler level, you might adopt a rule, "before making multiple bets on related questions, check to make sure you aren't guaranteeing you'll lose money." After all, real bookies offer odds such that if anyone was stupid enough to bet on each side of a question with the same bookie, they'd be guaranteed to lose money. In a sense, bookies could be interpreted as "money pumping" the public as a whole. But somehow, it turns out that any single individual will rarely be stupid enough to take both sides of the same bet from the same bookie, in spite of the fact that they're apparently irrational enough to be gambling in the first place.

In the end, I'm confused about how useful the willing gambler assumption really is when doing philosophy of probability. It certainly seems like worthwhile work gets done based on it, but just how applicable are those results to real life? How do we tell when we should reject a result because the willing gambler assumption causes problems in that particular case? I don't know.

One possible justification for the willing gambler assumption is that even those of us who don't literally gamble, ever, still must make decisions where the outcome is not certain, and we therefore we need to do a decent job of making probability assignments for those situations. But there are lots of people who are successful at their chosen field (including in fields that require decisions with uncertain outcomes) who aren't weather forecasters or professional gamblers, and therefore can be expected to make inaccurate probability estimates. Conversely, it doesn't seem that the skills acquired by successful professional gamblers give them much of an edge in other fields. Therefore, it seems that the relationship between being able to make accurate probability estimates and success in fields that don't specifically require them is weak.

Another justification for pursing lines of inquiry based on the willing gambler assumption, a justification that will be particularly salient for people on LessWrong, is that if we want to build an AI based on an idealization of how rational agents think (Bayesianism or whatever), we need tools like the willing gambler assumption to figure out how to get the idealization right. That sounds like a plausible thought at first. But if we flawed humans have any hope of building a good AI, it seems like an AI that's as flawed as (but no more flawed than) humans should also have a hope of self-improving into something better. An AI might be programmed in a way that makes it a bad gambler, but aware of this limitation, and left to decide for itself whether, when it self-improves, it wants to focus on improving its gambling ability or improving other aspects of itself.

As someone who cares a lot about AI, this issue of just how useful various idealizations are for thinking about AI and possibly programming an AI one day are especially important to me. Unfortunately, I'm not sure what to say about them, so at this point I'll turn the question over to the comments.

Related to: What Do We Mean By "Rationality?"

Rationality has many facets, both relatively simple and quite complex. As a result, it can often be hard to determine what aspects of rationality you should or shouldn't stress.

An extremely basic and abstract model of how rationality works might look a little something like this:

1. Collect evidence about your environment from various sources
2. Update your model of reality based on evidence collected (optimizing the updating process is more or less what we know as epistemic rationality)
3. Act in accordance with what your model of reality indicates is best for achieving your goals (optimizing the actions you take is more or less what we know as instrumental rationality)
4. Repeat continually forever

Related: Pinpointing Utility

If I ever say "my utility function", you could reasonably accuse me of cargo-cult rationality; trying to become more rational by superficially immitating the abstract rationalists we study makes about as much sense as building an air traffic control station out of grass to summon cargo planes.

There are two ways an agent could be said to have a utility function:

1. It could behave in accordance with the VNM axioms; always choosing in a sane and consistent manner, such that "there exists a U". The agent need not have an explicit representation of U.

2. It could have an explicit utility function that it tries to expected-maximize. The agent need not perfectly follow the VNM axioms all the time. (Real bounded decision systems will take shortcuts for efficiency and may not achieve perfect rationality, like how real floating point arithmetic isn't associative).

Neither of these is true of humans. Our behaviour and preferences are not consistent and sane enough to be VNM, and we are generally quite confused about what we even want, never mind having reduced it to a utility function. Nevertheless, you still see the occasional reference to "my utility function".

Sometimes "my" refers to "abstract me who has solved moral philosophy and or become perfectly rational", which at least doesn't run afoul of the math, but is probably still wrong about the particulars of what such an abstract idealized self would actually want. But other times it's a more glaring error like using "utility function" as shorthand for "entire self-reflective moral system", which may not even be VNMish.

But this post isn't really about all the ways people misuse terminology, it's about where we're actually at on the whole problem for which a utility function might be the solution.

As above, I don't think any of us have a utility function in either sense; we are not VNM, and we haven't worked out what we want enough to make a convincing attempt at trying. Maybe someone out there has a utility function in the second sense, but I doubt that it actually represents what they would want.

Perhaps then we should speak of what we want in terms of "terminal values"? For example, I might say that it is a terminal value of mine that I should not murder, or that freedom from authority is good.

But what does "terminal value" mean? Usually, it means that the value of something is not contingent on or derived from other facts or situations, like for example, I may value beautiful things in a way that is not derived from what they get me. The recursive chain of valuableness terminates at some set of values.

There's another connotation, though, which is that your terminal values are akin to axioms; not subject to argument or evidence or derivation, and simply given, that there's no point in trying to reconcile them with people who don't share them. This is the meaning people are sometimes getting at when they explain failure to agree with someone as "terminal value differences" or "different set of moral axioms". This is completely reasonable, if and only if that is in fact the nature of the beliefs in question.

About two years ago, it very much felt like freedom from authority was a terminal value for me. Those hated authoritarians and fascists were simply wrong, probably due to some fundamental neurological fault that could not be reasoned with. The very prototype of "terminal value differences".

And yet here I am today, having been reasoned out of that "terminal value", such that I even appreciate a certain aesthetic in bowing to a strong leader.

If that was a terminal value, I'm afraid the term has lost much of its meaning to me. If it was not, if even the most fundamental-seeming moral feelings are subject to argument, I wonder if there is any coherent sense in which I could be said to have terminal values at all.

The situation here with "terminal values" is a lot like the situation with "beliefs" in other circles. Ask someone what they believe in most confidently, and they will take the opportunity to differentiate themselves from the opposing tribe on uncertain controversial issues; god exists, god does not exist, racial traits are genetic, race is a social construct. The pedant answer of course is that the sky is probably blue, and that that box over there is about a meter long.

Likewise, ask someone for their terminal values, and they will take the opportunity to declare that those hated greens are utterly wrong on morality, and blueness is wired into their very core, rather than the obvious things like beauty and friendship being valuable, and paperclips not.

So besides not having a utility function, those aren't your terminal values. I'd be suprised if even the most pedantic answer weren't subject to argument; I don't seem to have anything like a stable and non-negotiable value system at all, and I don't think that I am even especially confused relative to the rest of you.

Instead of a nice consistent value system, we have a mess of intuitions and hueristics and beliefs that often contradict, fail to give an answer, and change with time and mood and memes. And that's all we have. One of the intuitions is that we want to fix this mess.

People have tried to do this "Moral Philosophy" thing before, myself included, but it hasn't generally turned out well. We've made all kinds of overconfident leaps to what turn out to be unjustified conclusions (utilitarianism, egoism, hedonism, etc), or just ended up wallowing in confused despair.

The zeroth step in solving a problem is to notice that we have a problem.

The problem here, in my humble opinion, is that we have no idea what we are doing when we try to do Moral Philosophy. We need to go up a meta-level and get a handle on Moral MetaPhilosophy. What's the problem? What are the relevent knowns? What are the unknowns? What's the solution process?

Ideally, we could do for Moral Philosphy approximately what Bayesian probability theory has done for Epistemology. My moral intuitions are a horrible mess, but so are my epistemic intuitions, and yet we more-or-less know what we are doing in epistemology. A problem like this has been solved before, and this one seems solvable too, if a bit harder.

It might be that when we figure this problem out to the point where we can be said to have a consistent moral system with real terminal values, we will end up with a utility function, but on the other hand, we might not. Either way, let's keep in mind that we are still on rather shaky ground, and at least refrain from believing the confident declarations of moral wisdom that we so like to make.

Moral Philosophy is an important problem, but the way is not clear yet.

Related to: Politics is the Mind-Killer, Keep Your Identity Small

Followed By: How to Not Get Offended

One oft-underestimated threat to epistemic rationality is getting offended. While getting offended by something sometimes feels good and can help you assert moral superiority, in most cases it doesn't help you figure out what the world looks like. In fact, getting offended usually makes it harder to figure out what the world looks like, since it means you won't be evaluating evidence very well. In Politics is the Mind-Killer, Eliezer writes that "people who would be level-headed about evenhandedly weighing all sides of an issue in their professional life as scientists, can suddenly turn into slogan-chanting zombies when there's a Blue or Green position on an issue." Don't let yourself become one of those zombies-- all of your skills, training, and useful habits can be shut down when your brain kicks into offended mode!

One might point out that getting offended is a two-way street and that it might be more appropriate to make a post called "Don't Be Offensive." That feels like a just thing to say-- as if you are targeting the aggressor rather than the victim. And on a certain level, it's true-- you shouldn't try to offend people, and if you do in the course of a normal conversation it's probably your fault. But you can't always rely on others around you being able to avoid doing this. After all, what's offensive to one person may not be so to another, and they may end up offending you by mistake. And even in those unpleasant cases when you are interacting with people who are deliberately trying to offend you, isn't staying calm desirable anyway?

The other problem I have with the concept of being offended as victimization is that, when you find yourself getting offended, you may be a victim, but you're being victimized by yourself. Again, that's not to say that offending people on purpose is acceptable-- it obviously isn't. But you're the one who gets to decide whether or not to be offended by something. If you find yourself getting offended to things as an automatic reaction, you should seriously evaluate why that is your response.

There is nothing inherent in a set of words that makes them offensive or inoffensive-- your reaction is an internal, personal process. I've seen some people stay cool in the face of others literally screaming racial slurs in their faces and I've seen other people get offended by the slightest implication or slip of the tongue. What type of reaction you have is largely up to you, and if you don't like your current reactions you can train better ones-- this is a core principle of the extremely useful philosophy known as Stoicism.

Of course, one (perhaps Robin Hanson) might also point out that getting offended can be socially useful. While true-- quickly responding in an offended fashion can be a strong signal of your commitment to group identity and values^[1]-- that doesn't really relate to what this post is talking about. This post is talking about the best way to acquire correct beliefs, not the best way to manipulate people. And while getting offended can be a very effective way to manipulate people-- and hence a tactic that is unfortunately often reinforced-- it is usually actively detrimental for acquiring correct beliefs. Besides, the signalling value of offense should be no excuse for not knowing how not to be offended. After all, if you find it socially necessary to pretend that you are offended, doing so is not exactly difficult.

Personally, I have found that the cognitive effort required to build a habit of not getting offended pays immense dividends. Getting offended tends to shut down other mental processes and constrain you in ways that are often undesirable. In many situations, misunderstandings and arguments can be diminished or avoided completely if one is unwilling to become offended and practiced in the art of avoiding offense. Further, some of those situations are ones in which thinking clearly is very important indeed! All in all, while getting offended does often feel good (in a certain crude way), it is a reaction that I have no regrets about relinquishing.

[1] In Keep Your Identity Small, Paul Graham rightly points out that one way to prevent yourself from getting offended is to let as few things into your identity as possible.