Creating Friendly AI seems to require us humans to either solve most of the outstanding problems in philosophy, or to solve meta-philosophy (i.e., what is the nature of philosophy, how do we practice it, and how should we program an AI to do it?), and to do that in an amount of time measured in decades. I'm not optimistic about our chances of success, but out of these two approaches, the latter seems slightly easier, or at least less effort has already been spent on it. This post tries to take a small step in that direction, by asking a few questions that I think are worth investigating or keeping in the back of our minds, and generally raising awareness and interest in the topic.

[I posted this to open thread a few days ago for review. I've only made some minor editorial changes since then, so no need to read it again if you've already read the draft.]

This is a belated reply to cousin_it's 2009 post Bayesian Flame, which claimed that frequentists can give calibrated estimates for unknown parameters without using priors:

And here's an ultra-short example of what frequentists can do: estimate 100 independent unknown parameters from 100 different sample data sets and have 90 of the estimates turn out to be true to fact afterward. Like, fo'real. Always 90% in the long run, truly, irrevocably and forever.

And indeed they can. Here's the simplest example that I can think of that illustrates the spirit of frequentism:

Suppose there is a machine that produces biased coins. You don't know how the machine works, except that each coin it produces is either biased towards heads (in which case each toss of the coin will land heads with probability .9 and tails with probability .1) or towards tails (in which case each toss of the coin will land tails with probability .9 and heads with probability .1). For each coin, you get to observe one toss, and then have to state whether you think it's biased towards heads or tails, and what is the probability that's the right answer.

Let's say that you decide to follow this rule: after observing heads, always answer "the coin is biased towards heads with probability .9" and after observing tails, always answer "the coin is biased towards tails with probability .9". Do this for a while, and it will turn out that 90% of the time you are right about which way the coin is biased, no matter how the machine actually works. The machine might always produce coins biased towards heads, or always towards tails, or decide based on the digits of pi, and it wouldn't matter—you'll still be right 90% of the time. (To verify this, notice that in the long run you will answer "heads" for 90% of the coins actually biased towards heads, and "tails" for 90% of the coins actually biased towards tails.) No priors needed! Magic!

In deciding whether to pay attention to an idea, a big clue, if it were readily available, would be how many people have checked it over for correctness, and for how long. Most new ideas that human beings come up with are wrong, and if someone just thought of something five seconds ago and excitedly wants to tell you about it, probably the only benefit of listening is not offending the person.

But it seems quite rare for this important piece of metadata to be straightforwardly declared, perhaps because such declarations can't be trusted in general. Instead, we usually have to infer it from various other clues, like the speaker's personality (how long do they typically think before they speak?), formality of the language employed to express the idea, the presence of spelling and grammar mistakes, the venue where the idea is presented or published, etc.

Unfortunately, such inferences can be imprecise or error-prone. For example, the same speaker may sometimes think a lot before speaking, and other times think little before speaking. Using costly signals like formal language is also wasteful compared to everyone simply telling the truth (but can still be a second-best solution in low-trust groups). In a community like ours, where most of us are striving to build reputations for being (or at least trying to be) rational and cooperative, and therefore there is a level of trust higher than usual, it might be worth experimenting with a norm of declaring how long we've thought about each new idea when presenting it. This may be either in addition to or as an alternative to other ways of communicating how confident we are about our ideas.

To follow my own advice, I'll say that I've thought about this topic off and on for about two weeks, and then spent about three hours writing and reviewing this post. I first started thinking about it at the SIAI decision theory workshop, which was the first time I ever worked with a large group of people on a complex problem in real time. I noticed that the variance in the amount of time different people spend thinking through new ideas before they speak is quite high. I was surprised to discover, for example, that Gary Drescher has been working on decision theory for many years and has considered and discarded about a dozen possible solutions.

The trigger for actually writing this post is yesterday's Overcoming Bias post Twin Conspiracies, which Robin seemed to have spent much less time thinking through than usual, but which has no overt indications of this. (An obvious objection that he apparently failed to consider is, wouldn't corporations actively recruit twins to be co-CEOs if they are so productive? Several OB commenters also pointed this out.) A blogger may not want to spend days poring over every post, but why not make it easier for the reader to distinguish the serious, carefully thought out ideas from the throwaway ones?

It's common practice in this community to differentiate forms of rationality along the axes of epistemic vs. instrumental, and individual vs. group, giving rise to four possible combinations. I think our shared goal, as indicated by the motto "rationalists win", is ultimately to improve group instrumental rationality. Generally, improving each of these forms of rationality also tends to improve the others, but sometimes conflicts arise between them. In this post I point out one such conflict between individual epistemic rationality and group epistemic rationality.

We place a lot of emphases here on calibrating individual levels of confidence (i.e., subjective probabilities), and on the idea that rational individuals will tend to converge toward agreement about the proper level of confidence in any particular idea as they update upon available evidence. But I argue that from a group perspective, it's sometimes better to have a spread of individual levels of confidence about the individually rational level. Perhaps paradoxically, disagreements among individuals can be good for the group.

Complexity of value is the thesis that our preferences, the things we care about, don't compress down to one simple rule, or a few simple rules. To review why it's important (by quoting from the wiki):

• Caricatures of rationalists often have them moved by artificially simplified values - for example, only caring about personal pleasure. This becomes a template for arguing against rationality: X is valuable, but rationality says to only care about Y, in which case we could not value X, therefore do not be rational.
• Underestimating the complexity of value leads to underestimating the difficulty of Friendly AI; and there are notable cognitive biases and fallacies which lead people to underestimate this complexity.

I certainly agree with both of these points. But I worry that we (at Less Wrong) might have swung a bit too far in the other direction. No, I don't think that we overestimate the complexity of our values, but rather there's a tendency to assume that complexity of value must lead to complexity of outcome, that is, agents who faithfully inherit the full complexity of human values will necessarily create a future that reflects that complexity. I will argue that it is possible for complex values to lead to simple futures, and explain the relevance of this possibility to the project of Friendly AI.