LESSWRONG
LW

Comment Permalink

This certainly doesn't work in all cases:

There is a hidden object which is either green, red or blue. Three people have conflicting opinions about its colour, based on different pieces of reasoning. If you are the one who believes it is green, you have to add up the opponents who say not-green, despite the fact that there is no single not-green position (think of the symmetry -- otherwise everyone could have too great confidence). The same holds true if these are expert opinions.

The above example is basically as general as possible, so in order for your argument to work it will need to add specifics of some sort.

Also, the Koran/Bible case doesn't work. By symmetry, the Koran readers can say that they don't need to add up the Bible readers and the atheists, since they are heterogeneous, so they can keep their belief in the Koran...

JamesAndrix15y40

In practice all arguments will share some premises and some conclusions, in messy asymmetrical ways.

If the not-greens share a a consistent rationale about why the object cannot be green, then I need to take that into account.

If the red supporter contends that all green and blue objects were lost in the color wars, while the blue supporter contends that all objects are fundamentally blue and besides the color wars never happened, then their opinions roughly cancel each other out. (Barring other reasons for me to view one as more rational than the other.)

I ... (read more)

See in context

65 Complexity of Value ≠ Complexity of Outcome

by Wei Dai

30th Jan 2010

3 min read

223

65

Complexity of value is the thesis that our preferences, the things we care about, don't compress down to one simple rule, or a few simple rules. To review why it's important (by quoting from the wiki):

Caricatures of rationalists often have them moved by artificially simplified values - for example, only caring about personal pleasure. This becomes a template for arguing against rationality: X is valuable, but rationality says to only care about Y, in which case we could not value X, therefore do not be rational.
Underestimating the complexity of value leads to underestimating the difficulty of Friendly AI; and there are notable cognitive biases and fallacies which lead people to underestimate this complexity.

I certainly agree with both of these points. But I worry that we (at Less Wrong) might have swung a bit too far in the other direction. No, I don't think that we overestimate the complexity of our values, but rather there's a tendency to assume that complexity of value must lead to complexity of outcome, that is, agents who faithfully inherit the full complexity of human values will necessarily create a future that reflects that complexity. I will argue that it is possible for complex values to lead to simple futures, and explain the relevance of this possibility to the project of Friendly AI.

The easiest way to make my argument is to start by considering a hypothetical alien with all of the values of a typical human being, but also an extra one. His fondest desire is to fill the universe with orgasmium, which he considers to have orders of magnitude more utility than realizing any of his other goals. As long as his dominant goal remains infeasible, he's largely indistinguishable from a normal human being. But if he happens to pass his values on to a superintelligent AI, the future of the universe will turn out to be rather simple, despite those values being no less complex than any human's.

The above possibility is easy to reason about, but perhaps does not appear very relevant to our actual situation. I think that it may be, and here's why. All of us have many different values that do not reduce to each other, but most of those values do not appear to scale very well with available resources. In other words, among our manifold desires, there may only be a few that are not easily satiated when we have access to the resources of an entire galaxy or universe. If so, (and assuming we aren't wiped out by an existential risk or fall into a Malthusian scenario) the future of our universe will be shaped largely by those values that do scale. (I should point out that in this case the universe won't necessarily turn out to be mostly simple. Simple values do not necessarily lead to simple outcomes either.)

Now if we were rational agents who had perfect knowledge of our own preferences, then we would already know whether this is the case or not. And if it is, we ought to be able to visualize what the future of the universe will look like, if we had the power to shape it according to our desires. But I find myself uncertain on both questions. Still, I think this possibility is worth investigating further. If it were the case that only a few of our values scale, then we can potentially obtain almost all that we desire by creating a superintelligence with just those values. And perhaps this can be done manually, bypassing an automated preference extraction or extrapolation process with their associated difficulties and dangers. (To head off a potential objection, this does assume that our values interact in an additive way. If there are values that don't scale but interact nonlinearly (multiplicatively, for example) with values that do scale, then those would need to be included as well.)

Whether or not we actually should take this approach would depend on the outcome of such an investigation. Just how much of our desires can feasibly be obtain this way? And how does the loss of value inherent in this approach compare with the expected loss of value due to the potential of errors in the extraction/extrapolation process? These are questions worth trying to answer before committing to any particular path, I think.

P.S., I hesitated a bit in posting this, because underestimating the complexity of human values is arguably a greater danger than overlooking the possibility that I point out here, and this post could conceivably be used by someone to rationalize sticking with their "One Great Moral Principle". But I guess those tempted to do so will tend not to be Less Wrong readers, and seeing how I already got myself sucked into this debate, I might as well clarify and expand on my position.

Complexity of Value

Frontpage

65

Mentioned in

35Ideal Advisor Theories and Personal CEV

23A Thought Experiment on Pain as a Moral Disvalue

15Superintelligence 23: Coherent extrapolated volition

Complexity of Value ≠ Complexity of Outcome

New Comment

223 comments, sorted by

top scoring

Click to highlight new comments since: Today at 5:32 AM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Toby_Ord15y210

There are a lot of posts here that presuppose some combination of moral anti-realism and value complexity. These views go together well: if value is not fundamental, but dependent on characteristics of humans, then it can derive complexity from this and not suffer due to Occam's Razor.

There are another pair of views that go together well: moral realism and value simplicity. Many posts here strongly dismiss these views, effectively allocating near-zero probability to them. I want to point out that this is a case of non-experts being very much at odds with expert opinion and being clearly overconfident. In the Phil Papers survey for example, 56.3% of philosophers lean towards or believe realism, while only 27.7% lean towards or accept anti-realism.

http://philpapers.org/surveys/results.pl

Given this, and given comments from people like me in the intersection of the philosophical and LW communities who can point out that it isn't a case of stupid philosophers supporting realism and all the really smart ones supporting anti-realism, there is no way that the LW community should have anything like the confidence that it does on this point.

Moreover, I should point out that most of the rea... (read more)

[-]CarlShulman15y540

Among target faculty listing meta-ethics as their area of study moral realism's lead is much smaller: 42.5% for moral realism and 38.2% against.

Looking further through the philpapers data, a big chunk of the belief in moral realism seems to be coupled with theism, where anti-realism is coupled with atheism and knowledge of science. The more a field is taught at Catholic or other religious colleges (medieval philosophy, bread-and-butter courses like epistemology and logic) the more moral realism, while philosophers of science go the other way. Philosophers of religion are 87% moral realist, while philosophers of biology are 55% anti-realist.

In general, only 61% of respondents "accept" rather than lean towards atheism, and a quarter don't even lean towards atheism. Among meta-ethics specialists, 70% accept atheism, indicating that atheism and subject knowledge both predict moral anti-realism. If we restricted ourselves to the 70% of meta-ethics specialists who also accept atheism I would bet at at least 3:1 odds that moral anti-realism comes out on top.

Since the Philpapers team will be publishing correlations between questions, such a bet should be susceptible to objective... (read more)

[-]Wei Dai15y100

In general, those interquestion correlations should help pinpoint any correct contrarian cluster.

This is why I put more weight on Toby's personal position, than on the majority expert position. As far as I know, Toby is in the same contrarian cluster as me, yet he seems to give much more weight to moral realism (and presumably not the Yudkowskian kind either) than I do. Like ciphergoth, I wish he would tell us which arguments in favor of realism, or against anti-realism, that he finds persuasive.

1CarlShulman15y

It seems that would be more likely if some people would put effort into apparently wanting to learn more about moral realism, or would read and present some of the arguments charitably to LW.

1Toby_Ord15y

Thanks for looking that up Carl -- I didn't know they had the break-downs. This is the more relevant result for this discussion, but it doesn't change my point much. Unless it was 80% or so in favour of anti-realism, I think holding something like 95% credence in anti-realism this is far too high for non-experts.

[-]CarlShulman15y270

Atheism doesn't get 80% support among philosophers, and most philosophers of religion reject it because of a selection effect where few wish to study what they believe to be non-subjects (just as normative and applied ethicists are more likely to reject anti-realism).

6Vladimir_Nesov15y

Perhaps we shouldn't look for professional consensus on things we accept with almost-certainty, because things that can be correctly accepted with almost-certainty by amateurs will not be professionally studied, except by people who are systematically confused. Instead, we should ask non-professional opinion of people who are in the position to know most about the subject, but don't study it professionally.

4Toby_Ord15y

You are correct that it is reasonable to assign high confidence to atheism even if it doesn't have 80% support, but we must be very careful here. Atheism is presumably the strongest example of such a claim here on Less Wrong (i.e. one which you can tell a great story why so many intelligent people would disagree etc and hold a high confidence in the face of disagreement). However, this does not mean that we can say that any other given view is just like atheism in this respect and thus hold beliefs in the face of expert disagreement, that would be far too convenient.

9CarlShulman15y

Strong agreement about not overgeneralizing. It does appear, however, that libertarianism about free well, non-physicalism about the mind, and a number of sorts of moral realism form a cluster, sharing the feature of reifying certain concepts in our cognitive algorithms even when they can be 'explained away.' Maybe we can discuss this tomorrow night.

0komponisto15y

Of course not; the substance of one's reasons for disagreeing matters greatly. In this case, I suspect there's probably a significant amount of correlation/non-independence between the reasons for believing atheism and believing something like moral non-realism. One thing we should take away from cases like atheism is that surveys probably shouldn't be interpreted naively, but rather as somewhat noisy information. I think my own heuristic (on binary questions where I already have a strong opinion) is basically to look on which side of 50% my position falls; if the majority agrees with me (or, say, the average confidence in my position is over 50%), I tend to regard that as (more) evidence in my favor, with the strength increasing as the percentage increases. (This, I think, would be part of how I would answer Yvain.)

2Paul Crowley15y

I think the arguments you're developing here go a long way towards answering Toby's point, but what safeguards can we use to ensure we can't use it as a generalized anti-expert defence?

4Vladimir_Nesov15y

The prerequisite for this heuristic is coming to a conclusion with near-certainty on an amateur level. The safeguard has to be general ability to not get that much unjustified overconfidence.

0Paul Crowley15y

Are you proposing a safeguard here or setting out what the safeguard has to achieve?

2Vladimir_Nesov15y

I'm pointing out that there is already a generally applicable enough set of safeguards that covers this case in particular, adequate or not. That is, this heuristic doesn't automatically lead as astray.

0Paul Crowley15y

I don't think I can understand you properly; it reads like you're saying that we can be confident in rejecting expert advice if we've already reached a contrary position with high confidence. That doesn't sound Bayesian. I suspect the error is mine but I'd appreciate your help in finding and fixing it!

3CarlShulman15y

EDIT: I [not Vladimir] would say that if we have one position that we can be confident in (atheism) we can use it as an indicator of expert quality, and pay more attention to those experts on other issues (e.g. moral realism as philosophers define it). And with respect to the selection effect among philosophers of religion, there's overwhelming direct evidence on this in the form of the Catholic Church push on this front.

2Vladimir_Nesov15y

Re: correction: I would say so too, though I wasn't saying that here. It is the mechanism through which we can reject expert opinion, but also as applied to the very claim that is being contested, not just the other slam-dunk claims.

1Paul Crowley15y

Only where there's a relationship of course. We would be unwise to reject medical expertise from a body where atheists were few, unless religion impinged on that advice eg abortion, cryonics. Here a relationship with religion is clear.

3CarlShulman15y

I would say that if on some matter of medical controversy atheist doctors and medical academics tended to come out one way, while the median opinion came out the other way, we should go with the atheist medical opinion, ceteris paribus. Atheism is a proxy for intelligence and scientific thinking, a finding which has a mountain of evidence in its favor.

1Paul Crowley15y

Definitely if the majority opinion among atheist experts differed from the majority opinion among all experts, I'd go for the former, but if say the majority of doctors studying a disease were Catholic for simple geographic reasons, I'd still defer to their expertise.

0Vladimir_Nesov15y

I agree with this interpretation. Zack is making basically the same point here. (This discussion is about meta-level mechanism for agreement, where you accept a conclusion; experts might well have persuasive arguments that inverse one's confidence.)

2RobinZ15y

(cf. Argument Screens Off Authority.)

[-]Zack_M_Davis15y250

Many posts here strongly dismiss [moral realism and simplicity], effectively allocating near-zero probability to them. I want to point out that this is a case of non-experts being very much at odds with expert opinion and being clearly overconfident. [...] For non-experts, I really can't see how one could even get to 50% confidence in anti-realism, much less the kind of 98% confidence that is typically expressed here.

One person's modus ponens is another's modus tollens. You say that professional philosophers' disagreement implies that antirealists shouldn't be so confident, but my confidence in antirealism is such that I am instead forced to downgrade my confidence in professional philosophers. I defer to experts in mathematics and science, where I can at least understand something of what it means for a mathematical or scientific claim to be true. But on my current understanding of the world, moral realism just comes out as nonsense. I know what it means for a computation to yield this-and-such a result, or for a moral claim to be true with respect to such-and-these moral premises that might be held by some agent. But what does it mean for a moral claim to be simply true, full ... (read more)

[-]Eliezer Yudkowsky15y210

But what does it mean for a moral claim to be simply true, full stop?

Well, in my world, it means that the premises are built into saying "moral claim"; that the subject matter of "morality" is the implications of those premises, and that moral claims are true when they make true statements about these implications. If you wanted to talk about the implications of other premises, it wouldn't be the subject matter of what we name "morality". Most possible agents (e.g. under a complexity-based measure of mind design space) will not be interested in this subject matter - they won't care about what is just, fair, freedom-promoting, life-preserving, right, etc.

This doesn't contradict what you say, but it's a reason why someone who believes exactly everything you do might call themselves a moral realist.

In my view, people who look at this state of affairs and say "There is no morality" are advocating that the subject matter of morality is a sort of extradimensional ontologically basic agent-compelling-ness, and that, having discovered this hypothesized transcendental stuff to be nonexistent, we have discovered that there is no morality. In cont... (read more)

[-]Zack_M_Davis15y100

Yes, but I think that my way of talking about things (agents have preferences, some of which are of a type we call moral, but there is no objective morality) is more useful than your way of talking about things (defining moral as a predicate referring to a large set of preferences), because your formulation (deliberately?) makes it difficult to talk about humans with different moral preferences, which possibility you don't seem to take very seriously, whereas I think it very likely.

6Wei Dai15y

So, according to this view, moral uncertainty is just a subset of logical uncertainty, where we restrict our attention to the implication of a fixed set of moral premises. But why is it that I feel uncertain about which premises I should accept? I bet that when most people talk about moral realism and moral uncertainty, that is what they're talking about. Why/how does/should one's moral premises change as one gains knowledge and ability to reflect? (Note that in standard decision theory one's values simply don't change this way.) It seems to me this ought to be the main topic of moral inquiry, instead of being relegated to a parenthetical remark. The subsequent working out of implications seems rather trivial by comparison. Maybe, but we're not there yet.

9Eliezer Yudkowsky15y

You've got meta-moral criteria for judging between possible terms in your utility function, a reconciliation process for conflicting terms, other phenomena which are very interesting and I do wish someone would study in more detail, but so far as metaethics goes it would tend to map onto a computation whose uncertain output is your utility function. Just more logical uncertainty. How can I put it? The differences here are probably very important to FAI designers and object-level moral philosophers, but I'm not sure they're metaethically interesting... or they're metaethically interesting, but they don't make you confused about what sort of stuff morality could possibly be made out of. Moral uncertainty is still made out of a naturalistic mixture of physical uncertainty and logical uncertainty.

5Wei Dai15y

Suppose there's an UFAI loose on the Internet that's not yet very powerful. In order to gain more power, it wants me to change my moral premises (so I'll help it later), and to do that, it places a story on the web for me to find. I read the story, and it "inspires" me to change my values in the direction that the UFAI prefers. In your view, how do we say that this is bad, if this is just what my meta-moral computation did?

[-]Eliezer Yudkowsky15y110

If the UFAI convinced you of anything that wasn't true during the process - outright lies about reality or math - or biased sampling of reality producing a biased mental image, like a story that only depicts one possibility where other possibilities are more probable - then we have a simple and direct critique.

If the UFAI never deceived you in the course of telling the story, but simple measures over the space of possible moral arguments you could hear and moralities you subsequently develop, produce a spread of extrapolated volitions "almost all" of whom think that the UFAI-inspired-you has turned into something alien and unvaluable - if it flew through a persuasive keyhole to produce a very noncentral future version of you who is disvalued by central clusters of you - then it's the sort of thing a Coherent Extrapolated Volition would try to stop.

See also #1 on the list of New Humane Rights: "You have the right not to have the spread in your volition optimized away by an external decision process acting on unshared moral premises."

[-]PeerInfinity15y140

New Humane Rights:

You have the right not to have the spread in your volition optimized away by an external decision process acting on unshared moral premises.

You have the right to a system of moral dynamics complicated enough that you can only work it out by discussing it with other people who share most of it.

You have the right to be created by a creator acting under what that creator regards as a high purpose.

You have the right to exist predominantly in regions where you are having fun.

You have the right to be noticeably unique within a local world.

You have the right to an angel. If you do not know how to build an angel, one will be appointed for you.

You have the right to exist within a linearly unfolding time in which your subjective future coincides with your decision-theoretical future.

You have the right to remain cryptic.

-- Eliezer Yudkowsky

(originally posted sometime around 2005, probably earlier)

[-]Wei Dai15y120

What about the least convenient world where human meta-moral computation doesn't have the coherence that you assume? If you found yourself living in such a world, would you give up and say no meta-ethics is possible, or would you keep looking for one? If it's the latter, and assuming you find it, perhaps it can be used in the "convenient" worlds as well?

To put it another way, it doesn't seem right to me that the validity of one's meta-ethics should depend on a contingent fact like that. Although perhaps instead of just complaining about it, I should try to think of some way to remove the dependency...

(We also disagree about the likelihood that the coherence assumption holds, but I think we went over that before, so I'm skipping it in the interest of avoiding repetition.)

5Eliezer Yudkowsky15y

I think this is about metamorals not metaethics - yes, I'm merely defining terms here, but I consider "What is moral?" and "What is morality made of?" to be problems that invoke noticeably different issues. We already know, at this point, what morality is made of; it's a computation. Which computation? That's a different sort of question and I don't see a difficulty in having my answer depend on contingent facts I haven't learned. In response to your question: yes, if I had given a definition of moral progress where it turned out empirically that there was no coherence in the direction in which I was trying to point and the past had been a random walk, then I should reconsider my attempt to describe those changes as "progress".

0Nick_Tarleton15y

How do you cash "which computation?" out to logical+physical uncertainty? Do you have in mind some well-defined metamoral computation that would output the answer?

0Eliezer Yudkowsky15y

I think you just asked me how to write an FAI. So long as I know that it's made out of logical+physical uncertainty, though, I'm not confused in the same way that I was confused in say 1998.

6Nick_Tarleton15y

"Well-specified" may have been too strong a term, then; I meant to include something like CEV as described in 2004. Is there an infinite regress of not knowing how to compute morality, or how to compute (how to compute morality), or how to compute (how to compute (...)), that you need to resolve; do you currently think you have some idea of how it bottoms out; or is there a third alternative that I should be seeing?

3Roko15y

I think it is a powerful secret of philosophy and AI design that all useful philosophy depends upon the philosopher(s) observing contingent facts from their sensory input stream. Philosophy can be thought of as an ultra high level machine learning technique that records the highest-level regularities of our input/output streams. And the reason I said that this is a powerful AI design principle, is that you realize that your AI can do good philosophy by looking for such regularities.

3Vladimir_Nesov15y

Think of it as a foundational struggle: you've got non-rigorous ideas about what is morally true/right, and you are searching of a way to build a foundation such that any right idea will follow from that foundation deductively. Arguably, this task is impossible within human mind. A better human-level approach would be structural, where you recognize certain (premise) patterns in reliable moral ideas, and learn heuristics that allow to conclude other patterns wherever you find the premise patterns. This constitutes ordinary moral progress, when fixed in culture.

5MichaelVassar15y

I would agree with the above, but I would also substitute 'god', 'fairies', 'chi' and 'UFO abductions', among other things, in place of 'morality'.

[-]Eliezer Yudkowsky15y120

In cases like that, I am perfectly willing to say that we have discovered that the subject matter of "fairies" is a coherent, well-formed concept that turns out to have an empty referent. The closet is there, we opened it up and looked, and there was nothing inside. I know what the world ought to look like if there were fairies, or alternatively no fairies, and the world looks like it has no fairies.

9MichaelVassar15y

I think that a very large fraction of the time, when a possibility appears to be coherent and well formed, it may turn out not to be upon more careful examination. I would see the subject matter of "fairies" as "that which causes us to talk about fairies", the subject matter of "dogs" as "that which causes us to talk about dogs", and the subject matter of "morality" as "that which causes us to talk about morality". All three are interesting.

2wedrifid15y

This is a theme that crops up fairly frequently as a matter of semantic confusion and is a confusion that is difficult to resolve trivially due to inferential differences to the actual abstract concepts. I haven't seen this position explained so coherently in one place before. Particularly the line: ... and the necessary context. I would find it useful to have this as a top level post to link to. Even if, as you have just suggested to JamesAndrix, it is just a copy and paste job. It'll save searching through comments to find a permalink if nothing else.

1matt15y

Copy it to the wiki yourself.

1wedrifid15y

What name?

1Vladimir_Nesov15y

Such things should go through a top-level post first, original content doesn't work well for the wiki.

9MichaelVassar15y

Doctors or medicine, investors or analysis of public information, scientists or science, philosophers or philosophy... maybe it's the process of credentialing that we should be downgrading our credence in. Really, why should the prior for credentials being a very significant form of evidence ever have been very high?

6CarlShulman15y

The philpapers survey is for the top 99 departments. Things do get better as you go up. Among hard scientists, elite schools are more atheist, and the only almost entirely atheist groups are super-elite, like the National Academy of Sciences/Royal Society.

3Technologos15y

Perhaps the fact that they have devoted their lives to a topic suggests that they have a vested interest in making it appear not to be nonsense. Cognitive dissonance can be tricky even for the pros.

2Paul Crowley15y

Maybe they mean something different by it than we're imagining?

0Zack_M_Davis15y

Quite possible. But in that case I would say that we're just talking about things in different ways, and not actually disagreeing on anything substantive.

1timtyler15y

Say we did a survey of 1000 independent advanced civilizations - and found they all broadly agreed on some moral proposition X. That's the kind of evidence that I think would support the idea of morality inherent in the natural world.

[-]Roko15y210

Toby, I spent a while looking into the meta-ethical debates about realism. When I thought moral realism was a likely option on the table, I meant:

Strong Moral Realism: All (or perhaps just almost all) beings, human, alien or AI, when given sufficient computing power and the ability to learn science and get an accurate map-territory distinction, will agree on what physical state the universe ought to be transformed into, and therefore they will assist you in transforming it into this state.

But modern philosophers who call themselves "realists" don't mean anything nearly this strong. They mean that that there are moral "facts". But what use is it if the paperclipper agrees that it is a "moral fact" that human rights ought to be respected, if it then goes on to say it has no desire to act according to the prescription of moral facts, and moral facts can't somehow revoke it.

The force of "scientific facts" is that they constrain the world. If an alien wants to get from Andromeda to here, it has to take at least 2.5 million years, the physical fact of the finite speed of light literally stops the alien from getting here sooner, whether it likes it... (read more)

[-]RobinHanson15y100

I strongly agree with Roko that something like his strong version is the interesting version. What matters is what range of creatures will come to agree on outcomes; it matters much less what range of creatures think their desires are "right" in some absolute sense, if they don't think that will eventually be reflected in agreement.

-4timtyler15y

Roko's question seems engineered to be wrong to me. If this is what people think moral realism means - or should mean - no wonder they disagree with it.

3Vladimir_Nesov15y

In the context of this comment, the goal of FAI can be said to be to constrain the world by "moral facts", just like laws of physics constrain the world by "physical facts". This is the sense in which I mean "FAI=Physical Laws 2.0".

0Roko15y

Only in a useless way: there is a specific FAI that does the "truly right" thing, but the truthhood of rightness doesn't stop you from having to code the rightness in. Goodness is not discoverably true: if you don't already know exactly what goodness is, you can't find out.

0Vladimir_Nesov15y

I'm describing the sense of post-FAI world.

1Roko15y

hmmm. That is interesting. Well, let us define the collection W_i of worlds run by superintelligences with the subscript i ranging over goals. No matter what i is, those worlds are going to look, to any agents in them, like worlds with "moral truths". However, any agent that learned the real physics of such a world would see that the goodness is written in to the initial conditions, not the laws.

3Toby_Ord15y

Roko, you make a good point that it can be quite murky just what realism and anti-realism mean (in ethics or in anything else). However, I don't agree with what you write after that. Your Strong Moral Realism is a claim that is outside the domain of philosophy, as it is an empirical claim in the domain of exo-biology or exo-sociology or something. No matter what the truth of a meta-ethical claim, smart entities might refuse to believe it (the same goes for other philosophical claims or mathematical claims). Pick your favourite philosophical claim. I'm sure there are very smart possible entities that don't believe this and very smart ones that do. There are probably also very smart entities without the concepts needed to consider it. I understand why you introduced Strong Moral Realism: you want to be able to see why the truth of realism would matter and so you came up with truth conditions. However, reducing a philosophical claim to an empirical one never quite captures it. For what its worth, I think that the empirical claim Strong Moral Realism is false, but I wouldn't be surprised if there was considerable agreement among radically different entities on how to transform the world.

7Roko15y

If there's a philosophical claim that intelligent agents across the universe wouldn't display massive agreement on, then I don't really think it is worth its salt. I think that this principle can be used to eliminate a lot of nonsense from philosophy. Which of anti-realism or weak realism is true seems to be a question we can eliminate. Whether strong realism is true or not seems substantive, because it matters to our policy which is true.

0Roko15y

There are clearly some examples where there can be interesting things to say that aren't really empirical, e.g. decision theory, mystery of subjective experience. But I think that this isn't one of them. Suffice it to say I can't think of anything that makes the debate between weak realism and antirealism at all interesting or worthy of attention. Certainly, Friendly AI theorists ought not care about the difference, because the empirical claims about an AI system will do are identical. Once the illusions and fallacies surrounding rationalist moral psychology has been debunked, proponents of other AI motivation methods than FAI also ought not to care about the weak realism vs. anti-realism pseudo-question

0[anonymous]15y

I'm having trouble reconciling this with the beginning of your first comment:

-7timtyler15y

[-]Eliezer Yudkowsky15y180

I am a moral cognitivist. Statements like "ceteris paribus, happiness is a good thing" have truth-values. Such moral statements simply are not compelling or even interesting enough to compute the truth-value of to the vast majority of agents, even those which maximize coherent utility functions using Bayesian belief updating (that is, rational agents) or approximately rational agents.

AFAICT the closest official term for what I am is "analytic descriptivist", though I believe I can offer a better defense of analytic descriptivism than what I've read so far.

EDIT: Looking up moral naturalism shows that Frank Jackson's analytic descriptivism aka moral functionalism is listed as a form of moral naturalism: http://plato.stanford.edu/entries/naturalism-moral/#JacMorFun

Note similarity to "Joy in the Merely Good".

6lukeprog13y

For the interested: A good summary/defense of Jackson's moral functionalism can be found in Jackson (2012), "On ethical naturalism and the philosophy of language." Now, should we call this a form of "moral realism"? I dunno. That's something I'd prefer to taboo. Even famous error theorist Richard Joyce kinda agrees.

[-]JamesAndrix15y100

From your SEP link on Moral Realism: "It is worth noting that, while moral realists are united in their cognitivism and in their rejection of error theories, they disagree among themselves not only about which moral claims are actually true but about what it is about the world that makes those claims true. "

I think this is good cause for breaking up that 56%. We should not take them as a block merely because (one component of) their conclusions match, if their justifications are conflicting or contradictory. It could still be the case that 90% of expert philosophers reject any given argument for moral realism. (This would be consistent with my view that those arguments are silly.)

I may have noticed this because the post on Logical Rudeness is fresh in my mind.

3Toby_Ord15y

You are entirely right that the 56% would split up into many subgroups, but I don't really see how this weakens my point: more philosophers support realist positions than anti-realist ones. For what its worth, the anti-realists are also fragmented in a similar way.

[-]JamesAndrix15y350

Disagreeing positions don't add up just because they share a feature. On the contrary, If people offer lots of different contradictory reasons for a conclusion (even if each individual has consistent beliefs) it is a sign that they are rationalizing their position.

If 2/3's of experts support proposition G , 1/3 because of reason A while rejecting B, and 1/3 because of reason B while rejecting A, and the remaining 1/3 reject A and B; then the majority Reject A, and the majority Reject B. G should not be treated as a reasonable majority view.

This should be clear if A is the koran and B is the bible.

If we're going to add up expert views, we need to add up what experts consider important about a question, not features of their conclusions.

You shouldn't add up two experts if they would consider each other's arguments irrational. That's ignoring their expertise.

7Eliezer Yudkowsky15y

I know it might seem difficult to expand this into a top-level post, but if you just want to post it verbatim, I'd say go for it.

3MichaelVassar15y

Yes James, I'd also appreciate that. Maybe we should encourage more short top-level posts and comment upgrades to posts. I think that would be great if we could develop a good procedure.

5Toby_Ord15y

This certainly doesn't work in all cases: There is a hidden object which is either green, red or blue. Three people have conflicting opinions about its colour, based on different pieces of reasoning. If you are the one who believes it is green, you have to add up the opponents who say not-green, despite the fact that there is no single not-green position (think of the symmetry -- otherwise everyone could have too great confidence). The same holds true if these are expert opinions. The above example is basically as general as possible, so in order for your argument to work it will need to add specifics of some sort. Also, the Koran/Bible case doesn't work. By symmetry, the Koran readers can say that they don't need to add up the Bible readers and the atheists, since they are heterogeneous, so they can keep their belief in the Koran...

4JamesAndrix15y

In practice all arguments will share some premises and some conclusions, in messy asymmetrical ways. If the not-greens share a a consistent rationale about why the object cannot be green, then I need to take that into account. If the red supporter contends that all green and blue objects were lost in the color wars, while the blue supporter contends that all objects are fundamentally blue and besides the color wars never happened, then their opinions roughly cancel each other out. (Barring other reasons for me to view one as more rational than the other.) I suspect that there are things to be said about islam that both atheists and christians would agree on. That's a block that a rational muslim should take into account. Our disagreeing conclusions about god are secondary. If I'm going to update my position because 56% of experts agree on something, then I want to know what I'm going to update to.

3Wei Dai15y

This discussion continues here. BTW, I wish there is a way to upgrade a comment into a post and automatically move all the discussions under the new post as well.

3Douglas_Knight15y

The only reason I can think of to upgrade a comment to a post is to draw attention to it, whether google attention, naturality of external linking, or the attention of the regular readers. In all these cases, it seems to me that it is the duty of the author, who is demanding time from many readers, to spend time summarizing the old discussion and making it easy for new readers to join.

2wedrifid15y

I haven't heard it put that way before. But your explanation makes it seem obvious!

0blogospheroid15y

Ignoring their expertise, but counting only popularity. Moderator, does that mean that Less Wrong's karma system might be modified to take into account why a comment was upvoted? A valid principle James, but a bad example which might be contested by those more knowledgeable of the matter. Islam considers itself the best of the revealed religions and jesus is revered as a prophet in Islam. So, in this case, christians reject the koran, but the muslims do not completely reject the bible. I'm not sure what might serve as a better example, though. The multiple possible explanations of the present recession may serve as a better example, incase you want to make this a top level post.

0Technologos15y

What you say is true while the Koran and the Bible are referents, but when A and B become "Mohammed is the last prophet, who brought the full truth of God's will" and "Jesus was a literal incarnation of God," (the central beliefs of the religions that hold the respective books sacred) then James' logic holds.

0MichaelVassar15y

This applies very generally when the evidential properties of reference classes are brought up.

8TruePath15y

The right response to moral realism isn't to dispute it's truth but to simply observe you don't understand the concept. I mean imagine someone started going around insisting some situations were Heret and others were Grovic but when asked to explain what made a situation Heret or Grovic he simply shrugged and said they were primitive concepts. But you persist and after observing his behavior for a period of time you work out some principle that perfectly predicts which category he will assign a given situation to, even counterfactually but when you present the algorithm to him and ask, "Ohh so is it satisfying this principle that makes one Heret rather than Grovic?" he insists that while your notion will always agrees with his notion that's not what he means. Moreover, he insists that no definition in terms of physical state could capture these concepts. Confused you press him and he says that there are special things which we can't casually interact with that determine Heret or Grovic status. Bracketing your skepticism you ask him to say what properties these new ontological objects must have. After listing a couple he adds that most importantly they can't just be random things with this structure but they also have to be Heret making or Grovic making and that's what distingushes them from all the other casually inaccessible things out there that might otherwise yield some slightly different class of things as Heret and Grovic. Frustrated you curse the guy saying he hasn't really told you anything since you didn't know what it meant to be Heret or Grovic in the first place so you surely don't know what it means to be Heret making or Grovic making. The man's reply is simply to shrug and say, "well it's a fundamental concept, if you don't understand I can't explain it to you anymore than I could explain the perceptual experience of redness to a man who had never experienced color." ---------------------------------------- In such a situation the only thing you c

8Paul Crowley15y

Could you direct us to the best arguments for moral realism, or against anti-realism? Thanks!

[-]Toby_Ord15y130

In metaethics, there are typically very good arguments against all known views, and only relatively weak arguments for each of them. For anything in philosophy, a good first stop is the Stanford Encyclopedia of Philosophy. Here are some articles on the topic at SEP:

I think the best book to read on metaethics is:

An Introduction to Contemporary Metaethics

6Wei Dai15y

Toby, I read through those SEP articles but couldn't find the good arguments against anti-realism that you mentioned. In contrast, the article on deontology laid out the arguments for and against it very clearly. Can you please point us more specifically to the arguments that you find persuasive? Maybe just give us some page numbers in the book that you referenced? Most of us don't really have the time to read something like that cover to cover in search of a few nuggets of information.

1Paul Crowley15y

Thank you for doing that, and may I second this. I started reading those articles, then after a bit started scanning for the anti-realism articles, and worried after not finding them that I'd not read carefully enough, so I'm glad to have your report on this. I really am curious to read these arguments, so I hope someone can point us to them.

1Wei Dai15y

I managed to find a draft of a book chapter titled In Defence of Moral Realism. I'm still wondering what Toby thinks the best arguments are, but alas he doesn't seem to be following this discussion anymore.

1Paul Crowley15y

Thanks! Again, didn't get much from a quick skim, let me know if you find any real meat in there. The thing that really got my attention wasn't the assertion that there are some arguments in favour of realism, but that there are good arguments specifically against anti-realism. I know I've spoken of "skimming" twice here. I promise, if Toby Ord were to say to me "this contains good arguments against anti-realism" I would read it carefully.

2Wei Dai15y

But surely an argument against anti-realism is also an argument for realism? I'm interpreting Toby's comment as saying that there are good arguments for realism in general, but not for any particular realist meta-ethical theory. The author says in the conclusion, "I do not pretend to give any knock-down argument in this chapter for the thesis that objective moral facts or reasons exist, independently of our thoughts and actions." So I think it's mostly a matter of how convincing one finds the argument that he does give. It seems likely, given that the author is a specialist in and proponent of moral realism, that he would give the best arguments that he knew, so this paper seems like good evidence for what kind of arguments for realism is currently available.

0Paul Crowley15y

Will read carefully on that basis. Thanks.

5Paul Crowley15y

Do you have a personal favourite argument against moral anti-realism in there you could point me to?

2Paul Crowley15y

Thanks! There were several points in your PhD thesis where I couldn't work out how to square your position with moral anti-realism - I guess I know why now :-)

3timtyler15y

My case was here: http://lesswrong.com/lw/1m5/savulescu_genetically_enhance_humanity_or_face/1fuv Basically, morality is a product of evolution - which can be expected to favour some moral values over other ones - just as it favours certain physical structures like eyes and legs. Things like: "under most circumstances, don't massacre your relatives or yourself" can be reasonably expected to be widespread values in the universe. The idea gives morality a foundation in the natural world.

0byrnema15y

It is useful that Tim summarizes his position in this context, voted up. My position, developed with no background in philosophy or meta-ethics whatsoever and thus likely to be error-riddled or misguided, is that I consider it an unsolved problem within physical materialism (specifically, within the context of moral anti-realism) how "meaning" (the meaning of life and/or the value of values) can be a coherent or possible concept.

3Paul Crowley15y

Leave humans out of it and try to think about meanings of signals among animals, with an evolutionary perspective.

6Wei Dai15y

I accept this may be a case of the Popularization Bias (speaking for myself). I'd like to see some posts on the arguments against anti-realism...

0CarlShulman15y

Agreed. Perhaps Toby or David Pearce can be persuaded.

0Toby_Ord15y

I don't think I can persuaded. I have many good responses to the comments here, and I suppose I could sketch out some of the main arguments against anti-realism, but there are also many serious demands on my time and sadly this doesn't look like a productive discussion. There seems to be very little real interest in finding out more (with a couple of notable exceptions). Instead the focus is on how to justify what is already believed without finding out any thing else about what the opponents are saying (which is particularly alarming given that many commenters are pointing out that they don't understand what the opponents are saying!). Given all of this, I fear that writing a post would not be a good use of my time.

6CarlShulman15y

Alas. Perhaps some Less Wrongers with more time will write and post a hypothetical apostasy. I invite folk to do so.

4Paul Crowley15y

This is a little unfair; as soon as you take a deflationary stance on anything, you're saying that the other stance doesn't really have comprehensible content, and it's a mistake to turn that into a general-purpose dismissal of deflationary stances. If you think that's more true here than it is in other discussion forums, we're doing something very wrong. I understand that you're not able to spend time writing for this audience, but for those of us who do want to find out more about what moral realists are saying, every link you can provide to existing essays is valuable.

1DonGeddis15y

I, for one, am interested in hearing arguments against anti-realism. If you don't have personal interest in writing up a sketch, that's fine. Might you have some links to other people who have already done so?

6Zack_M_Davis15y

Elsewhere in the thread.

0[anonymous]15y

Toby already linked to the SEP articles on moral realism and anti-realism in another comment.

6Paul Crowley15y

This point has given me a lot of pause, so forgive me my many replies. Part of the problem is that even if I were only 60% confident of moral anti-realism, I would still act on it as if I were 100% confident because I don't understand moral realism at all, and my 60% confidence is in the belief that no-one else does either.

5whpearson15y

Can you give pointers to prominent naturalist realists?

4mattnewport15y

My impression of academic philosophers is that their 'expertise' is primarily in knowledge of what other philosophers have said and in the forms of academic philosophical argument. It is not expertise in true facts about the world. In other words, I would defer to their expertise on the technical details of academically accepted definitions of philosophical terms, or on the writings of Kant, much as I would defer to an expert in literary criticism on the details of what opinions other literary critics have expressed. In neither case however do I consider their opinions to be particularly relevant to the pursuit of true facts about the world. The fact that the survey you link finds 27% of philosophers 'accept or lean towards non-physicalism' increases my confidence in the above thesis.

3Stuart_Armstrong15y

It depends on the expertise; for instance, if we're talking about systems of axioms, then mathematicians may be those with the most relevant opinions as to whether one system has preference over others. And the idea that a unique system of moral axioms would have preference over all others makes no mathematical sense. If philosphers were espousing the n-realism position ("there are systems of moral axioms that are more true than others, but there will probably be many such systems, most mutually incompatible"), then I would have a hard time arguing against this. But, put quite simply, I dismiss the moral realistic position for the moment as the arguments go like this: * 1) There are moral truths that have special status; but these are undefined, and it is even undefined what makes them have this status. * 2) These undefined moral truths make a consistent system. * 3) This system is unique, according to criteria that are also undefined. * 4) Were we to discover this system, we should follow it, for reasons that are also undefined. There are too many 'undefined's in there. There is also very little philosphical literature I've encountred on 2), 3) and 4), which is at least as important as 1). A lot of the literature on 1) seems to be reducible to linguistic confusion, and (most importantly) different moral realists have different reasons for believing 1), reasons that are often contradictory. From a outsider's perspective, these seem powerful reasons to assume that philosphers are mired in confusion on this issue, and that their opinions are not determining. My strong mathematical reasons for claiming that there is no "superiority total ordering" on any general collection of systems of axioms clinches the argument for me, pending further evidence.

2[anonymous]15y

Looking further through the philpapers data, a big chunk of the belief in moral realism seem to be coupled with theism, where anti-realism is coupled with atheism and knowledge of science. The more a field is taught at Catholic or other religious colleges (medieval philosophy, bread-and-butter courses like epistemology and logic) the more moral realism, while philosophers of science go the other way. Philosophers of religion are 87% moral realist, while philosophers of biology are 55% anti-realist. In general, only 61% of respondents "accept" rather than lean towards atheism, and a quarter don't even lean towards atheism. Among meta-ethics specialists, 70% accept atheism, indicating that atheism and subject knowledge both predict moral anti-realism. If we restricted ourselves to the 70% of meta-ethics specialists who also accept atheism I would bet at at least 3:1 odds that moral anti-realism comes out on top. Since the Philpapers team will be publishing correlations between questions, such a bet should be susceptible to objective adjudication within a reasonable period of time. A similar pattern shows up for physicalism. In general, those interquestion correlations should help pinpoint any correct contrarian cluster.

0taw15y

I don't see in what meaningful sense these people are "experts".

-2jhuffman15y

Is there a reason I should care about the % of any group of people that think this or that? Just give us the argument, or write another article about it. It sounds interesting.

6timtyler15y

Re: "Is there a reason I should care about the % of any group of people that think this or that?" Generally speaking, yes, of course. If lots of experts in a relevant field think something is true, then their opinion carries some weight.

1jhuffman15y

In things related to observable facts or repeatable experiments I'd agree. In more abstract things, I'm less interested in what the polls say. Moral realism is a school of thought which has come in and out of style and favor among philosophers. Plato was arguably an moral realist; this isn't a new idea or area of debate amongst philosophers. Telling me where we are on the constantly shifting scale of acceptance is really pretty meaningless. Its like telling me 58% of fashion designers like the color black this year.

3Nick_Tarleton15y

Just to be sure, are you saying that you think there is a fact of the matter about whether moral realism is true, but you don't think philosophers' opinions are significantly correlated with this fact?

0jhuffman15y

Moral realism is a meta-ethical view - I do not know that a such a viewpoint can be as a matter of fact correct or incorrect. Maybe an ethical realist would argue that it is a matter of fact, I'm not sure - an anti-realist might argue that neither viewpoint can be a matter of fact. The whole argument is really about "what are facts" and "what can be objectively true or false" so I suppose that someone may extend this view to the meta-layer where the merits of the viewpoint itself are discussed although I think that would not be very useful.

0Kevin15y

I'm going to deploy what I call the Wittgenstein Chomsky blah blah blah argument. Philosophy is just words in English; there is little ultimate meaning we are going to find here unless we declare our mathematical axioms. Already most of the views here seem reconcilable by redefining what exactly the different words mean. To answer the question: some things can be proven objectively true, some things can be proven objectively false, some things can be proven to be undecidable. A fact is a true statement that follows from your given system of axioms. I personally am unsure if most moral principles or meta ethical systems can be declared objectively true or false with a standard ethical system, but I'm not going to take it seriously until a theorem prover says so. We are never going to convince each other of ultimate philosophical truth by having conversations like this. I suppose this makes me an anti-realist, unless someone feels like redefining realism for me. :D Again, it feels like I am missing something... http://plato.stanford.edu/entries/truth-axiomatic/ helped a little.

0Douglas_Knight15y

While at times Toby Ord refers to 56% as "most" philosophers, a claim that is disputable on grounds of fashion, at other times he draws the line at 20%; the point is that realist philosophers are not a tiny minority, rejecting widely accepted arguments.

0[anonymous]15y

Upvoted for being a legitimate question, from a fairly new poster, that really shouldn't be at -4.

[-]Vladimir_Nesov15y120

To head off a potential objection, this does assume that our values interact in an additive way.

...and this is an assumption of simplicity of value. That we can see individual "values" only reflects the vague way in which we can perceive our preference. Some "values" dictate the ways in which other "values" should play together, so there is no easy way out, no "additive" or "multiplicative" clean decomposition.

[-]Eliezer Yudkowsky15y70

Now censoring replies by DWCrmcm.

4AdeleneDawner15y

Aww, I wanted to play with him. ;)

[-]Paul Crowley15y130

I really don't want us to go there, here; I think it will reduce the quality of the site significantly. At the moment I can follow Recent Comments and find quite a few little nuggets of gold. If we get into arguing with people like this, the good content will be harder to find.

9Vive-ut-Vivas15y

I strongly agree with this. From his own website repeatedly linked here: This is not the kind of "nugget of gold" that we want to see on here, I would think.

2Paul Crowley15y

We've actually done remarkably well - "rationality" is generally a banner to which every green-ink vendor rallies, but I think this is our first full-on green-ink contributor.

0Psy-Kosh15y

"green ink"?

0Cyan15y

I encountered the term here.

0Psy-Kosh15y

Oh, okay. *follows along until he sees the wiki link* aaah. Thanks.

-2[anonymous]15y

Not to worry you won't see anymore. Good luck with your project and God bless.

[-]AngryParsley15y40

Are you a postmodern essay generator?

[-]mattnewport15y20

Your sesquipedalian obscurantism may fool your usual audience but you won't find it very successful here.

[-]avturchin6y10

A possible list of human values which are scalable:

Safety - we prefer that no sources of dangers exist anywhere in the universe

Self-replication - (at least some humans) prefer to gave as many descendants as possible and would be happy to tile the universe with their own grandchildren.

Power - A human often wants to become a king or god. So all the universe must be under his control.

Life extension - some wants immortality

Be the first - one must ensure that he is better than any other being in the universe

Exploration - obviously, scalable

Compassion to other beings.

[-]LucasSloan15y10

You were dropping a lot of unfamiliar terminology, the end result of which was failing utterly to communicate what your point was. If you want us to understand your point, you're going to have to unpack most of your sentences.

(easy example: what does Christian NeoRationalist mean?)

-9[anonymous]15y

[-]timtyler15y10

"rather there's a tendency to assume that complexity of value must lead to complexity of outcome"

The main problem I see here is the other way around:

There's a tendency to assume that complexity of outcome must have been produced by complexity of value.

AFAICS, it is only members of this community that think this way. Noboby else seems to have a problem with the idea of goals that can be concisely expressed - like: "trying to have as many offspring as possible" - leading to immense diversity and complexity.

This is a facet of an even mor... (read more)

2Peter_de_Blanc15y

Are you talking about Kolmogorov complexity or something else? Because the outcome which optimizes a simple goal would have a low Kolmogorov complexity.

-2timtyler15y

Kolmogorov complexity is fine by me. What make you say that? It isn't right. Filling the universe with orgasmium involves interstellar and intergalactic travel, stellar farming, molecular nanotechnology, coordinating stars to leap between galaxies, mastering nuclear fusion, conquering any other civilisations it might meet along the way - and many other complexity-requiring activities.

4Roko15y

Tim, you seem to be failing to distinguish between complex in the technical sense, and complex-looking. Remember that the mandelbrot set is simple, not complex in the technical sense.

0timtyler15y

Indeed - sorry! The r-pentomino's evolution is not a good example of high Kolmogorov complexity - though as you say, it is complex in other senses. I had forgotten that I gave that as one of my examples when I retroactively assented to the use Kolmogorov complexity as a metric.

2Peter_de_Blanc15y

Well, if you had a utility function over a finite set of possible outcomes, then you can run a computer program to check every outcome and pick the one with the highest utility. So the complexity of that outcome is bounded by the complexity of the set of possible outcomes plus the complexity of the utility function plus a constant. EDIT: And none of those things you mentioned require a lot of complexity.

-2timtyler15y

If the things I mentioned are so simple, perhaps you could explain how to do them? I would be especially interested in a "simple" method of conquering any other civilisations which we might meet - so perhaps you might like to concentrate on that?

3Peter_de_Blanc15y

Build AIXItl.

4timtyler15y

Alas, AIXItl is a whole class of things, many of which are likely to be highly complex.

0Paul Crowley15y

This contradicts my understanding of AIXI from Shane Legg's Extrobritannia presentation. What's the variable bit? Not the utility function; that's effectively external and after the fact, and AIXI infers it.

0timtyler15y

I think I answered that in the other sub-thread descended from the parent coment.

-1Peter_de_Blanc15y

If you're referring to the parameters t and l, I'll suggest a googolplex as a sufficiently large number with low Kolmogorov complexity.

0timtyler15y

No. AIXItl will need to have other complexity - if you want it to work in a reasonable quantity of time - e.g. see, for example: "Elimination of the factor 2˜l without giving up universality will probably be a very difficult task. One could try to select programs p and prove VA(p) in a more clever way than by mere enumeration. All kinds of ideas like, heuristic search, genetic algorithms, advanced theorem provers, and many more could be incorporated."" * http://www.hutter1.net/ai/paixi.pdf

3Peter_de_Blanc15y

It seems that you think "complex" means "difficult." It doesn't. Complex means "requires a lot of information to specify." There are no simple problems with complex solutions, because any specification of a problem is also a specification of its solution. This is the point of my original post.

3timtyler15y

So: a galaxy-conquering civilisation has low Kolmogorov complexity - because it has a short description - namely "a galaxy-conquering civilisation"??? If you actually attempted to describe a real galaxy-conquering civilisation, it would take a lot of bits to specify which one you were looking at - because the method of getting there will necessarily have involved time-and-space constraints. Those bits will have come from the galaxy - which is large and contains lots of information. More abstractly, "Find a root of y = sin(x)" is a simple problem with many K-complex solutions. Simple problems really can have K-complex solutions.

4Peter_de_Blanc15y

A particular galaxy-conquering civilization might have high Kolmogorov complexity, but if you can phrase the request "find me a galaxy-conquering civilization" using a small number of bits, and if galaxy-conquering civilizations exist, then there is a solution with low Kolmogorov complexity. Hmm, okay. I should not have said "there are no simple problems with complex solutions." Rather, there are no simple problems whose only solutions are complex. Are we in agreement?

7CronoDAS15y

Joke counterexample: x^2 = -1 is a simple problem that only has complex solutions. ;) (Of course, that's not the meaning of "complex" that you meant.) Serious counterexample: The four-color theorem is relatively simple to describe, but the only known proofs are very complicated.

8wedrifid15y

Gah, don't over-qualify jokes! It's a supplicating behavior and seeking permission to be funny blunts the effect. Just throw the "X^2 = -1" out there (which is a good one by the way) and then go on to say "A more serious counterexample". That's more than enough for people to 'get it' and anyone who doesn't will just look silly. This is the Right (Wedrifid-Laughter-Maximising) thing to do.

2CronoDAS15y

I'm sorry. :(

2Zack_M_Davis15y

Was that a practical joke on wedrifid?

6CronoDAS15y

It is now!

2wedrifid15y

Nice. Die. :P

1Jordan15y

But that complicated proof could be concisely provided via a universal proof algorithm and the statement of the four color theorem.

0Peter_de_Blanc15y

Exactly! The Kolmogorov complexity is not very high.

2timtyler15y

I am not sure. How about: what is the smallest number that can't be described by an English sentence of less than ten thousand words? ;-) Of course, knowing that a K-simple solution existed in the form of the problem specification would not help very much in constructing/implementing it.

1Roko15y

Simple in terms of kolmogorov complexity, that is. Simple to do? No.

2Wei Dai15y

Who are you referring to here? I myself wrote "Simple values do not necessarily lead to simple outcomes either."

-1timtyler15y

AFAICT, the origin of these ideas is here: http://lesswrong.com/lw/l3/thou_art_godshatter/ http://lesswrong.com/lw/lb/not_for_the_sake_of_happiness_alone/ http://lesswrong.com/lw/lq/fake_utility_functions/ http://lesswrong.com/lw/y3/value_is_fragile/ This seems to have led a slew of people to conclude that simple values lead to simple outcomes. You yourself suggest that the simple value of "filling the universe with orgasmium" is one whose outcome would mean that "the future of the universe will turn out to be rather simple". Things like that seem simply misguided to me. IMO, there are good reasons for thinking that that would lead to enormous complexity - in addition to lots of orgasmium.

3Nick_Tarleton15y

...but not in the least convenient possible world with an ontologically simple turn-everything-into-orgasmium button; and the sort of complexity that you mention that (I agree) would be involved in the actual world isn't a sort that most people regard as terminally valuable.

-2timtyler15y

Here we were talking about a superintelligent agent whose "fondest desire is to fill the universe with orgasmium". About the only way such an agent would fail to produce enormous complexity is if it died - or was otherwise crippled or imprisoned. Whether humans would want to live - or would survive in - the same universe as an orgasmium-loving superintelligence seems like a totally different issue to me - and it seems rather irrelevant to the point under discussion.

5Nick_Tarleton15y

Or if the agent has a button that, through simple magic, directly fills the universe with (stable) orgasmium. Did you even read what I wrote? Human morality is the point under discussion, so of course it's relevant. It seems clear that the chief kind of "complexity" that human morality values is that of conscious (whatever that means) minds and societies of conscious minds, not complex technology produced by unconscious optimizers.

-2timtyler15y

Re: Did you even read what I wrote? I think I missed the bit where you went off into a wild and highly-improbable fantasy world. Re: Human morality is the point under discussion What I was discussing was the "tendency to assume that complexity of outcome must have been produced by complexity of value". That is not specifically to do with human values.

[-]Kevin15y10

Does any existing decision theory make an attempt to decide based on existing human values? How would one begin to put human values into rigorous mathematical form?

I've convinced a few friends that the most likely path to Strong AI (i.e. intelligence explosion) is a bunch of people sitting in a room doing math for 10 years. But that's a lot of math before anyone even begins to start plugging in the values.

I suppose it does make sense for us to talk in English about what all of these things mean, so that in 10+ years they can be more easily translated into... (read more)

4Paul Crowley15y

Here's Eliezer's position on that question as of 2004.

3Eliezer Yudkowsky15y

That's not non-English.

5Paul Crowley15y

Sure, but it helps to be familiar with it if you're having this discussion all the same.

[-]xxd13y00

I've struggled with the concept of how an orgasmium optimizing AI could come about or a paperclipper or a bucketmaker or any of the others but this clarifies things. It's the programmer who passes the values on to the AI that is the cause, it's not necessarioy going to be an emergent property.

That makes things easier I believe as it means the code for the seed AI needs to be screened for maximization functions.

[-][anonymous]14y00

-3 lol, Well I can see that you are no closer to AI than you were last year. Do you have a definition of value yet? Life? Complexity?

I thought not.

Respectufly W

[-]Paul Crowley15y00

Please leave.

-3[anonymous]15y

As I see it, you can all treat me affectionately as your own personal crazy. Enjoy me. Criticize my definitions and and my structures. I would love that. That is why I came here. I was looking for intelligent criticism of my model. What I got instead was upsetting and ridicule. I have a neurological disorder and it was acting up. I didn't think. Then after adjusting my meds and my diet I realized that voting down my comments was irrelevant as I could reproduce them on my blog anyway. I overreacted. I'm sorry if I offended anyone. My partner usually has my back, but he didn't know that I was getting upset. My wife was the one who alerted me to it. Anyway I am going through the sequences which is where you are supposed to start. So I can do my thing here or on my blog and on my Face Book page. You decide.

-4[anonymous]15y

I tried to delete my profile and all my comments But to no avail. So until you delete "all" of my content and any references to those comments which are my property, then I will continue to post and link my posts on my blog so that others may see how you treat eccentrics edge dwellers and free thinkers - and how quickly you discount radical ideas as green ink.

[-]LucasSloan15y00

She on the other hand had no clue about what I was trying to express.

The commonality in these situations is you.

8Paul Crowley15y

I urge you to engage with this user only if you want them to stay here. There is no argument that will convince a rock.

2LucasSloan15y

Very well.

-7[anonymous]15y

0[anonymous]15y

Yes it is, that is a good point.

[-]snarles15y00

One more reason why I think Faustian singleton is the most likely final outcome, even if FAI succeeds. Unlike material or social desires, curiosity can scale endlessly--and to the point where humans become willing to suspend their individuality for the sake of computational efficiency.

[-]timtyler15y-20

Re: "the future of the universe will turn out to be rather simple"

You do realise that filling the universe with orgasmium involves interstellar and intergalactic travel, stellar farming, molecular nanotechnology, coordinating stars to leap between galaxies, mastering nuclear fusion, conquering any other civilisations it might meet - and many other high-tech wonders?

How is any of that that "simple"? Do you just mean: "somewhat less complex than it could conceivably be?"

[-]jhuffman15y-20

If it were the case that only a few of our values scale, then we can potentially obtain almost all that we desire by creating a superintelligence with just those values.

Can we really expect a superintelligence to stick with the values we give it ? Our own values change over time; sometimes without even external stimulus just internal reflection. I don't see how we can bound a superintelligence without doing more computation than we expect it to do in its lifetime.

5Zack_M_Davis15y

I tend to file this under "humans are stupid." Messy creatures like ourselves undergo value drift, but decision-theoretically speaking, systems designed to optimize for some particular criterion have a natural incentive to keep that criterion. Cf. "The Basic AI Drives."

1timtyler15y

It is probably best to model those as infections - or sometimes malfunctions. Humans get infected with pathogens that make them do things like sneeze. Their values have not changed to value spreading snot on their neigbours, rather they are infected with germs - and the germs do value that. It's much the same with mind-viruses. A catholic conversion is best modelled as a memetic infection - rather than a genuine change in underlying values. Such people can be cured.

6gregconen15y

The fact that a change is reversible does not make it not real. The fact that the final value system can be modeled as a starting value system modified by "memetic infection" does not make the final value system invalid. They are two different but equivalent ways of modelling the state.

1timtyler15y

Right. The point is that - under the "infection" analogy - people's "ultimate" values change a lot less. How much they change depends on the strength of people's memetic immune system - and there are some people with strong memetic immune systems whose values don't change much at all.

0gregconen15y

I'm not sure I follow you. Are you saying that some agents change their values less often than others (or equivalently, are less likely to acquire "infections")?

0[anonymous]15y

Also, I suspect a lot of people who talk about how human values change are thinking of things, like aesthetics and preferred flavors of ice cream, that aren't plausibly terminal values and that we often want to change over time.

3wedrifid15y

Yes. I once proved that a program will print out only prime numbers endlessly. I really, really wish I kept the working out.

2timtyler15y

Is that program still running? ;-)

0wedrifid15y

Hush you. You weren't supposed to notice that. :D

0timtyler15y

Quite a bit of ink has been spilled on this issue. Eliezer Yudkowsky and Steve Omohundro have argued that it is possible. Have you examined their arguments?

-3Thomas15y

Nothing changes from the inside, unless it is preprogrammed for.

-4jhuffman15y

You cannot pre-program all the routines for handling all future states for anything you can call an AI much less a "superintelligence". AI must be able to learn, and there is no reason all such learning is only based on new external stimuli.

-3Thomas15y

So you say, then a magic happens and something new is born. No, it doesn't. Just the physics acted onto the engraved algorithms and/or data.

0jhuffman15y

No magic; and yes all you have is algorithms and data. Obviously the algorithms contain an aspect of learning, and eventually the data guides decision pathways far more than the original algorithms; and even the algorithms themselves are mutable data. edit: I should note, I'm just talking about some of our crude "AI" systems that we build today. I don't know that this would be the actual software architecture of anything that could become a superintelligence. But it would have these capabilities and more...

3Thomas15y

Crude or non crude AI, a physical configuration at the start and a physical configuration at any time since. You can name it whatever you choose.

[+][anonymous]15y-90

Moderation Log

223Comments