This is a post about moral philosophy, approached with a mathematical metaphor.

Here's an interesting problem in mathematics.  Let's say you have a graph, made up of vertices and edges, with weights assigned to the edges.  Think of the vertices as US cities and the edges as roads between them; the weight on each road is the length of the road. Now, knowing only this information, can you draw a map of the US on a sheet of paper? In mathematical terms, is there an isometric embedding of this graph in two-dimensional Euclidean space?

When you think about this for a minute, it's clear that this is a problem about reconciling the local and the global.  Start with New York and all its neighboring cities.  You have a sort of star shape.  You can certainly draw this on the plane; in fact, you have many degrees of freedom; you can arbitrarily pick one way to draw it.  Now start adding more cities and more roads, and eventually the degrees of freedom diminish.  If you made the wrong choices earlier on, you might paint yourself in a corner and have no way to keep all the distances consistent when you add a new city.  This is known as a "synchronization problem."  Getting it to work locally is easy; getting all the local pieces reconciled with each other is hard.

This is a lovely problem and some acquaintances of mine have written a paper about it.  (http://www.math.princeton.edu/~mcucurin/Sensors_ASAP_TOSN_final.pdf)  I'll pick out some insights that seem relevant to what follows.  First, some obvious approaches don't work very well.  It might be thought we want to optimize over all possible embeddings, picking the one that has the lowest error in approximating distances between cities.  You come up with a "penalty function" that's some sort of sum of errors, and use standard optimization techniques to minimize it.  The trouble is, these approaches tend to work spottily -- in particular, they sometimes pick out local rather than global optima (so that the error can be quite high after all.) 

The approach in the paper I linked is different. We break the graph into overlapping smaller subgraphs, so small that they can only be embedded in one way (that's called rigidity) and then "stitch" them together consistently.  The "stitching" is done with a very handy trick involving eigenvectors of sparse matrices.  But the point I want to emphasize here is that you have to look at the small scale, and let all the little patches embed themselves as they like, before trying to reconcile them globally.

Now, rather daringly, I want to apply this idea to ethics.  (This is an expansion of a post people seemed to like: http://lesswrong.com/lw/1xa/human_values_differ_as_much_as_values_can_differ/1y )

The thing is, human values differ enormously.  The diversity of values is an empirical fact.  The Japanese did not have a word for "thank you" until the Portuguese gave them one; this is a simple example, but it absolutely shocked me, because I thought "thank you" was a universal concept.  It's not.    (edited for lack of fact-checking.) And we do not all agree on what virtues are, or what the best way to raise children is, or what the best form of government is.  There may be no principle that all humans agree on -- dissenters who believe that genocide is a good thing may be pretty awful people, but they undoubtedly exist.  Creating the best possible world for humans is a synchronization problem, then -- we have to figure out a way to balance values that inevitably clash.  Here, nodes are individuals, each individual is tied to its neighbors, and a choice of embedding is a particular action.  The worse the embedding near an individual fits the "true" underlying manifold, the greater the "penalty function" and the more miserable that individual is, because the action goes against what he values.

If we can extend the metaphor further, this is a problem for utilitarianism.  Maximizing something globally -- say, happiness -- can be a dead end.  It can hit a local maximum -- the maximum for those people who value happiness -- but do nothing for the people whose highest value is loyalty to their family, or truth-seeking, or practicing religion, or freedom, or martial valor.  We can't really optimize, because a lot of people's values are other-regarding: we want Aunt Susie to stop smoking, because of the principle of the thing.  Or more seriously, we want people in foreign countries to stop performing clitoridectomies, because of the principle of the thing.  And Aunt Susie or the foreigners may feel differently.  When you have a set of values that extends to the whole world, conflict is inevitable.

The analogue to breaking down the graph is to keep values local.  You have a small star-shaped graph of people you know personally and actions you're personally capable of taking.  Within that star, you define your own values: what you're ready to cheer for, work for, or die for.  You're free to choose those values for yourself -- you don't have to drop them because they're perhaps not optimal for the world's well-being.  But beyond that radius, opinions are dangerous: both because you're more ignorant about distant issues, and because you run into this problem of globally reconciling conflicting values.  Reconciliation is only possible if everyone's minding their own business. If things are really broken down into rigid components.  It's something akin to what Thomas Nagel said against utilitarianism:

"Absolutism is associated with a view of oneself as a small being interacting with others in a large world.  The justifications it requires are primarily interpersonal. Utilitarianism is associated with a view of oneself as a benevolent bureaucrat distributing such benefits as one can control to countless other beings, with whom one can have various relations or none.  The justifications it requires are primarily administrative." (Mortal Questions, p. 68.)

Anyhow, trying to embed our values on this dark continent of a manifold seems to require breaking things down into little local pieces. I think of that as "cultivating our own gardens," to quote Candide. I don't want to be so confident as to have universal ideologies, but I think I may be quite confident and decisive in the little area that is mine: my personal relationships; my areas of expertise, such as they are; my own home and what I do in it; everything that I know I love and is worth my time and money; and bad things that I will not permit to happen in front of me, so long as I can help it.  Local values, not global ones. 

Could any AI be "friendly" enough to keep things local?
 

Response to Beauty quips, "I'd shut up and multiply!"

Related to The Presumptuous Philosopher's Presumptuous Friend,  The Absent-Minded Driver, Sleeping Beauty gets counterfactually mugged

This is somewhat introductory. Observers play a vital role in the classic anthropic thought experiments, most notably the Sleeping Beauty and Presumptuous Philosopher gedankens. Specifically, it is remarkably common to condition simply on the existence of an observer, in spite of the continuity problems this raises. The source of confusion appears to be based on the distinction between the probability of an observer and the expectation number of observers, with the former not being a linear function of problem definitions.

There is a related difference between the expected gain of a problem and the expected gain per decision, which has been exploited in more complex counterfactual mugging scenarios. As in the case of the 1/2 or 1/3 confusion, the issue is the number of decisions that are expected to be made, and recasting problems so that there is at most one decision provides a clear intuition pump.

Related: http://lesswrong.com/lw/1kh/the_correct_contrarian_cluster/, http://lesswrong.com/lw/1mh/that_magical_click/, http://lesswrong.com/lw/18b/reason_as_memetic_immune_disorder/

Given a claim, and assuming that its truth or falsehood would be important to you, how do you decide if it's worth investigating?  How do you identify "bunk" or "crackpot" ideas?

Here are some examples to give an idea. 

"Here's a perpetual motion machine": bunk.  "I've found an elementary proof of Fermat's Last Theorem": bunk.  "9-11 was an inside job": bunk.  

 "Humans did not cause global warming": possibly bunk, but I'm not sure.  "The Singularity will come within 100 years": possibly bunk, but I'm not sure.  "The economic system is close to collapse": possibly bunk, but I'm not sure.

"There is a genetic difference in IQ between races": I think it's probably false, but not quite bunk.  "Geoengineering would be effective in mitigating global warming": I think it's probably false, but not quite bunk. 

(These are my own examples.  They're meant to be illustrative, not definitive.  I imagine that some people here will think "But that's obviously not bunk!"  Sure, but you probably can think of some claim that *you* consider bunk.)

A few notes of clarification: I'm only examining factual, not normative, claims.  I also am not looking at well established claims (say, special relativity) which are obviously not bunk. Neither am I looking at claims where it's easy to pull data that obviously refutes them. (For example, "There are 10 people in the US population.")  I'm concerned with claims that look unlikely, but not impossible. Also, "Is this bunk?" is not the same question as "Is this true?"  A hypothesis can turn out to be false without being bunk (for example, the claim that geological formations were created by gradual processes.  That was a respectable position for 19th century geologists to take, and a claim worth investigating, even if subsequent evidence did show it to be false.)  The question "Is this bunk?" arises when someone makes an unlikely-sounding claim, but I don't actually have the knowledge right now to effectively refute it, and I want to know if the claim is a legitimate subject of inquiry or the work of a conspiracy theory/hoax/cult/crackpot.  In other words, is it a scientific or a pseudoscientific hypothesis?  Or, in practical terms, is it worth it for me or anybody else to investigate it?

This is an important question, and especially to this community.  People involved in artificial intelligence or the Singularity or existential risk are on the edge of the scientific mainstream and it's particularly crucial to distinguish an interesting hypothesis from a bunk one.  Distinguishing an innovator from a crackpot is vital in fields where there are both innovators and crackpots.

I claim bunk exists. That is, there are claims so cracked that they aren't worth investigating. "I was abducted by aliens" has such a low prior that I'm not even going to go check up on the details -- I'm simply going to assume the alleged alien abductee is a fraud or nut.  Free speech and scientific freedom do not require us to spend resources investigating every conceivable claim.  Some claims are so likely to be nonsense that, given limited resources, we can justifiably dismiss them.

But how do we determine what's likely to be nonsense?  "I know it when I see it" is a pretty bad guide.

First idea: check if the proposer uses the techniques of rationality and science.  Does he support claims with evidence?  Does he share data and invite others to reproduce his experiments? Are there internal inconsistencies and logical fallacies in his claim?  Does he appeal to dogma or authority?  If there are features in the hypothesis itself that mark it as pseudoscience, then it's safely dismissed; no need to look further.

But what if there aren't such clear warning signs?  Our gracious host Eliezer Yudkowsky, for example, does not display those kinds of obvious tip-offs of pseudoscience -- he doesn't ask people to take things on faith, he's very alert to fallacies in reasoning, and so on.  And yet he's making an extraordinary claim (the likelihood of the Singularity), a claim I do not have the background to evaluate, but a claim that seems implausible.  What now?  Is this bunk?

A key thing to consider is the role of the "mainstream."  When a claim is out of the mainstream, are you justified in moving it closer to the bunk file?  There are three camps I have in mind, who are outside the academic mainstream, but not obviously (to me) dismissed as bunk: global warming skeptics, Austrian economists, and singularitarians.  As far as I can tell, the best representatives of these schools don't commit the kinds of fallacies and bad arguments of the typical pseudoscientist.  How much should we be troubled, though, by the fact that most scientists of their disciplines shun them?  Perhaps it's only reasonable to give some weight to that fact.  

Or is it? If all the scientists themselves are simply making their judgments based on how mainstream the outsiders are, then "mainstream" status doesn't confer any information.  The reason you listen to academic scientists is that you expect that at least some of them have investigated the claim themselves.  We need some fraction of respected scientists -- even a small fraction -- who are crazy enough to engage even with potentially crackpot theories, if only to debunk them.  But when they do that, don't they risk being considered crackpots themselves?  This is some version of "Tolerate tolerance."  If you refuse to trust anybody who even considers seriously a crackpot theory, then you lose the basis on which you reject that crackpot theory.  

So the question "What is bunk?", that is, the question, "What is likely enough to be worth investigating?", apparently destroys itself.  You can only tell if a claim is unlikely by doing a little investigation.  It's probably a reflexive process: when you do a little investigation, if it's starting to look more and more like the claim is false, you can quit, but if it's the opposite, then the claim is probably worth even more investigation.  

The thing is, we all have different thresholds for what captures our attention and motivates us to investigate further.  Some people are willing to do a quick Google search when somebody makes an extraordinary claim; some won't bother; some will go even further and do extensive research.  When we check the consensus to see if a claim is considered bunk, we're acting on the hope that somebody has a lower threshold for investigation than we do.  We hope that some poor dogged sap has spent hours diligently refuting 9-11 truthers so that we don't have to.  From an economic perspective, this is an enormous free-rider problem, though -- who wants to be that poor dogged sap?  The hope is that somebody, somewhere, in the human population is always inquiring enough to do at least a little preliminary investigation.  We should thank the poor dogged saps of the world.  We should create more incentives to be a poor dogged sap.  Because if we don't have enough of them, we're going to be very mistaken when we think "Well, this wasn't important enough for anyone to investigate, so it must be bunk."

(N.B.  I am aware that many climate scientists are being "poor dogged saps" by communicating with and attempting to refute global warming skeptics.  I'm not aware if there are economists who bother trying to refute Austrian economics, or if there are electrical engineers and computer scientists who spend time being Singularity skeptics.)

George Hamilton's autobiography Don't Mind if I Do, and the very similar book by Bob Evans, The Kid Stays in the Picture, give a lot of insight into human nature and values.  For instance: What do people really want?  When people have the money and fame to travel around the world and do anything that they want, what do they do?  And what is it that they value most about the experience afterward?

You may argue that the extremely wealthy and famous don't represent the desires of ordinary humans.  I say the opposite: Non-wealthy, non-famous people, being more constrained by need and by social convention, and having no hope of ever attaining their desires, don't represent, or even allow themselves to acknowledge, the actual desires of humans.

I noticed a pattern in these books:  The men in them value social status primarily as an ends to a means; while the women value social status as an end in itself.

Ethics is not geometry

Western philosophy began at about the same time as Western geometry; and if you read Plato you'll see that he, and many philosophers after him, took geometry as a model for philosophy.

In geometry, you operate on timeless propositions with mathematical operators.  All the content is in the propositions.  A proof is equally valid regardless of the sequence of operators used to arrive at it.  An algorithm that fails to find a proof when one exists is a poor algorithm.

The naive way philosophers usually map ethics onto mathematics is to suppose that a human mind contains knowledge (the propositional content), and that we think about that knowledge using operators.  The operators themselves are not seen as the concern of philosophy.  For instance, when studying values (I also use "preferences" here, as a synonym differing only in connotation), people suppose that a person's values are static propositions.  The algorithms used to satisfy those values aren't themselves considered part of those values.  The algorithms are considered to be only ways of manipulating the propositions; and are "correct" if they produce correct proofs, and "incorrect" if they don't.

But an agent's propositions aren't intelligent.  An intelligent agent is a system, whose learned and inborn circuits produce intelligent behavior in a given environment.  An analysis of propositions is not an analysis of an agent.

I will argue that:

1. The only preferences that can be unambiguously determined are the preferences people implement, which are not always the preferences expressed by their beliefs.
2. If you extract a set of propositions from an existing agent, then build a new agent to use those propositions in a different environment, with an "improved" logic, you can't claim that it has the same values.
3. Values exist in a network of other values.  A key ethical question is to what degree values are referential (meaning they can be tested against something outside that network); or non-referential (and hence relative).
4. Supposing that values are referential helps only by telling you to ignore human values.
5. You cannot resolve the problem by combining information from different behaviors, because the needed information is missing.
6. Today's ethical disagreements are largely the result of attempting to extrapolate ancestral human values into a changing world.
7. The future will thus be ethically contentious even if we accurately characterize and agree on present human values.

The self-indication assumption (SIA) states that

Given the fact that you exist, you should (other things equal) favor hypotheses according to which many observers exist over hypotheses on which few observers exist.

The reason this is a bad assumption might not be obvious at first.  In fact, I think it's very easy to miss.

Argument for SIA posted on Less Wrong

First, let's take a look at a argument for SIA that appeared at Less Wrong (link).  Two situations are considered.

1.  we imagine that there are 99 people in rooms that have a blue door on the outside (1 person per room).  One person is in a room with a red door on the outside.  It was argued that you are in a blue door room with probability 0.99.

2.  Same situation as above, but first a coin is flipped.  If heads, the red door person is never created.  If tails, the blue door people are never created.  You wake up in a room and know these facts.  It was argued that you are in a blue door room with probability 0.99.

So why is 1. correct and 2. incorrect?  The first thing we have to be careful about is not treating yourself as special.  The fact that you woke up just tells you that at least one conscious observer exists. 

In scenario 1 we basically just need to know what proportion of conscious observers are in a blue door room.  The answer is 0.99.

In scenario 2 you never would have woken up in a room if you hadn't been created.  Thus, the fact that you exist is something we have to take into account.  We don't want to estimate P(randomly selected person, regardless of if they exist or not, is in a blue door room).  That would be ignoring the fact that you exist.  Instead, the fact that you exist tells us that at least one conscious observer exists.  Again, we want to know what proportion of conscious observers are in blue door rooms.  Well, there is a 50% chance (if heads landed) that all conscious observers are in blue door rooms, and a 50% chance that all conscious observers are in red door rooms.  Thus, the marginal probability of a conscious observer being in a blue door room is 0.5.

The flaw in the more detailed Less Wrong proof (see the post) is when they go from step C to step D.  The *you* being referred to in step A might not exist to be asked the question in step D.  You have to take that into account.

General argument for SIA and why it's wrong

Let's consider the assumption more formally.

Assume that the number of people to be created, N, is a random draw from a discrete uniform distribution¹ on {1,2,...,N_max}.  Thus, P(N=k)=1/N_max, for k=1,...,N_max.  Assume N_max is large enough so that we can effectively ignore finite sample issues (this is just for simplicity).

Assume M= N_max*(N_max+1)/2 possible people exist, and we arbitrarily label them 1,...,M.  After the size of the world, say N=n, is determined, then we randomly draw n people from the M possible people.

After the data are collected we find out that person x exists.

We can apply Bayes' theorem to get the posterior probability:

P(N=k|x exists)=k/M, for k=1,...,N_max.

The prior probability was uniform, but the posterior favors larger worlds.  QED.

Well, not really.

The flaw here is that we conditioned on person x existing, but person x only became of interest after we saw that they existed (peeked at the data).

What we really know is that at least one conscious observer exists -- there is nothing special about person x.

So, the correct conditional probability is:

P(N=k|someone exists)=1/N_max, for k=1,...,N_max.

Thus, prior=posterior and SIA is wrong.

Egotism

The flaw with SIA that I highlighted here is it treats you as special, as if you were labeled ahead of time.  But the reality is, no matter who was selected, they would think they are the special person.  "But I exist, I'm not just some arbitrary person.  That couldn't happen in small world.  It's too unlikely."  In reality, that fact that I exist just means someone exists. I only became special after I already existed (peeked at the data and used it to construct the conditional probability).

Here's another way to look at it.  Imagine that a random number between 1 and 1 trillion was drawn.  Suppose 34,441 was selected.  If someone then asked what the probability of selecting that number was, the correct answer is 1 in 1 trillion.  They could then argue, "that's too unlikely of an event.  It couldn't have happened by chance."  However, because they didn't identify the number(s) of interest ahead of time, all we really can conclude is that a number was drawn, and drawing a number was a probability 1 event.

I give more examples of this here.

I think Nick Bostrom is getting at the same thing in his book (page 125):

..your own existence is not in general a ground for thinking that hypotheses are more likely to be true just by virtue of implying that there is a greater total number of observers. The datum of your existence tends to disconfirm hypotheses on which it would be unlikely that any observers (in your reference class) should exist; but that’s as far as it goes. The reason for this is that the sample at hand—you—should not be thought of as randomly selected from the class of all possible observers but only from a class of observers who will actually have existed. It is, so to speak, not a coincidence that the sample you are considering is one that actually exists. Rather, that’s a logical consequence of the fact that only actual observers actually view themselves as samples from anything at all

Related arguments are made in this LessWrong post.  

¹ for simplicity I'm assuming a uniform prior... the prior isn't the issue here