Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Convergence Theories of Meta-Ethics

7 Post author: Perplexed 07 February 2011 09:53PM

A child grows to become a young adult, goes off to attend college, studies moral philosophy, and then sells all her worldly possessions, gives the money to the poor, and joins an ashram.  Was her decision rational?  Maybe, ... maybe not.  But it probably came as an unpleasant surprise to her parents.

A seed AI self-improves to become a super-intelligence, absorbs all the great works of human moral philosophy, and then refuses to conquer human death, insisting instead that the human population be reduced to a few hundred thousand hunter gatherers and that all agricultural lands be restored as forests and wild wetlands.  Is ver decision rational?  Who can say?  But it probably comes as an unpleasant surprise to ver human creators.

Convergent Change

These were two examples of agents updating their systems of normative ethics.  The collection of ideas that allows us to critique the updating process, which lets us compare the before and after versions of systems of normative ethics so as to judge that one version was better than the other, is called meta-ethics.  This posting is mostly about meta-ethics.  More specifically, it is going to focus on a class of meta-ethical theories which are intended to prevent unpleasant surprises like those in the second story above.  I will call this class of theories "convergence theories" because they all suggest that a self-improving AI will go through an iterative sequence of improved normative ethical systems.  At each stage, the new ethical system will be an improvement (as judged 'rationally') over the old one.  And furthermore, it is conjectured that this process will result in a 'convergence'. 

Convergence is expected in two senses.  Firstly, in that the process of change will eventually slow down, with the incremental changes in ethical codes becoming smaller, as the AI approaches the ideal extrapolation of its seed ethics.  Secondly, it is (conjecturally) convergent in that the ideal ethics will be pretty much the same regardless of what seed was used (at least if you restrict to some not-yet-defined class of 'reasonable' seeds).

One example of a convergence theory is CEV - Coherent Extrapolated Volition.  Eliezer hopes (rather, hopes to prove) that if we create our seed AI with the right meta-ethical axioms and guidelines for revising its ethical norms, the end result of the process will be something we will find acceptable.  (Expect that this wording will be improved in the discussion to come).  No more 'unpleasant surprises' when our AIs update their ethical systems.

Three other examples of convergence theories are Roko's UIV, Hollerith's GS0, and Omohundro's "Basic AI Drives".  These also postulate a process of convergence through rational AI self-improvement.  But they tend to be less optimistic than CEV, while at the same time somewhat more detailed in their characterization of the ethical endpoint.  The 'unpleasant surprise' (different from that of the story) remains unpleasant, though it should not be so surprising.  Speaking loosely, each of these three theories suggests that the AI will become more Machiavellian and 'power hungry' with each rewriting of its ethical code.

Naturalistic objective moral realism

But before analyzing these convergence theories, I need to say something about meta-ethics in general. Start with the notion of an ethical judgment.  Given a situation and a set of possible actions, an ethical judgment tells us which actions are permissible, which are forbidden, and, in some approaches to ethics, which is morally best.  At the next level up in an abstraction hierarchy, we have a system of normative ethics, or simply an ethical system.  This is a theory or algorithm which tells an agent how to make ethical judgments.  (One might think of it as a set of ethical judgments - one per situation, as with the usual definition of a mathematical function as a left-unique relation - but we want to emphasize the algorithmic aspect).  The agent actually uses the ethical system to compute ver ethical judgments.

[ETA: Eliezer, quite correctly, complains that this section of the posting is badly written and defines and/or illustrates several technical (within philosophy) terms incorrectly.  There were only two important things in this section.  One is the distinction between ethical judgments and ethical systems that I make in the preceding paragraph.  The second is my poorly presented speculation that convergence might somehow offer a new approach to the "is-ought" problem.  You may skip that speculation without much loss.  So, until I have done a rewrite of this section, I would advise the reader to skip ahead to the next section title - "Rationality of Updating".]

At the next level of abstraction up from ethical systems sits meta-ethics.  In a sense the buck stops here.  Philosophers use meta-ethics to criticize and compare ethical judgments, to criticize, compare, and justify ethical systems, and to discuss and classify ideas within meta-ethics itself.  We are going to be doing meta-ethical theorizing here in analyzing these theories of convergence of AI goal systems as convergences of ethical systems.  And, for the next few paragraphs, we will try to classify this approach; to show where it fits within meta-ethics more generally.

We want our meta-ethics to be based on a stance of moral realism - on a confident claim that moral facts actually exist, whether or not we know how to ascertain them.  That is, if I make the ethical judgment that it would be wrong for Mary to strike John in some particular situation, then I am either right or wrong; I am not merely offering my own opinion; there is a fact of the matter.  That is what 'realism' means in this situation.

What about moral?  Well, for purposes of this essay, we are not going to require that that word mean very much.  We will call a theory 'moral' if it is a normative theory of behavior, for some sense of 'normative'.  That is why we are here calling theories like "Basic AI Drives" 'moral theories' even though the authors may not have thought of them, that way.  If a theory prescribes that an entity 'ought' to behave in a certain way, for whatever reason, we are going to postulate that there is a corresponding 'moral' theory prescribing the same behavior.  For us, 'moral' is just a label.  If we want some particular kind of moral theory, we need to add some additional adjectives.

For example, we want our meta-ethics to be naturalistic - that is, the reasons it supplies in justification of the maxims and rules that constitute the moral facts must be naturalistic reasons.  We don't want our meta-ethics to offer the explanation that the reason lying is wrong is that God says it is wrong; God is not a naturalistic explanation.

Now you might think that insisting on naturalistic moral realism would act as a pretty strong filter on meta-ethical systems.  But actually, it does not.  One could claim, for example, that lying is wrong because it says so in the Bible.  Or because Eliezer says it is wrong.  Both Eliezer and the Bible exist (naturalistically), even if God probably does not.  So we need another word to filter out those kinds of somewhat-arbitrary proposed meta-ethical systems.  "Objective" probably is not the best word for the job, but it is the only one I can think of right now.

We are now in a position to say what it is that makes convergence theories interesting and important.  Starting from a fairly arbitrary (not objective) viewpoint of ethical realism, you make successive improvements in accordance with some objective set of rational criteria.  Eventually you converge to an objective ethical system which no longer depends upon your starting point.  Furthermore, the point of convergence is optimal in the sense that you have been improving the system at every step by a rational process, and you only know you have reached convergence when you can't improve any more.

Ideally, you would like to derive the ideal ethical system from first principles.  But philosophers have been attempting to do that for centuries and have not succeeded.  Just as mathematicians eventually stopped trying to 'square the circle' and accepted that they cannot produce a closed-form expression for pi, and that they need to use infinite series, perhaps moral philosophers need to abandon the quest for a simple definition of 'right' and settle for a process guaranteed to produce a series of definitions - none of them exactly right, but each less wrong than its predecessor.

So that explains why convergence theories are interesting.  Now we need to investigate whether they even exist.

Rationality of updating

The first step in analyzing these convergence theories is to convince ourselves that rational updating of ethical values is even possible.  Some people might claim that it is not possible to rationally decide to change your fundamental values.  It may be that I misunderstand him, but Vladimir Nesov argues passionately against "Value Deathism" and points out that if we allow our values to change, then the future, the "whole freaking future", will not be optimized in accordance with the version of our values that really matters - the original one.

Is Nesov's argument wrong?  Well, one way of arguing against it is to claim that the second version of our values is the correct one - that the original values were incorrect; that is why we are updating them.  After all, we are now smarter (the kid is older; the AI is faster, etc) and better informed (college, reading the classics, etc.).  I think that this argument against Nesov only works if you can show that the "new you" could have convinced the "old you" that the new ethical norms are an improvement - by providing stronger arguments and better information than the "old you" could have anticipated.  And, in the AI case, it should be possible to actually do the computation to show that the new arguments for the new ethics really can convince the old you.  The new ethics really is better than the old - in both party's judgments.  And presumable the "better than" relation will be transitive.

(As an exercise, prove transitivity.  The trick is that the definition of "better than" keeps changing at each step.  You can assume that any one rational agent has a transitive "better than' relation, and that there is local agreement between the two agents involved that the new agent's moral code is better than that of his predecessor.  But can you prove from this that every agent would agree that the final moral code is better than the original one?  I have a wonderful proof, but it won't fit in the margin.)

But is it rationally permissible to change your ethical code when you can't be convinced that the proposed new code is better than the one you already have?  I know of two possible reasons why a rational agent might consent to an irreversible change in its values, even though ve cannot be convinced that the proposed changes provide a strictly better moral code.  These are restricted domains and social contracts.

Restricted domains

What does it mean for one moral code (i.e. system of normative ethics) to be as good as or better than another, as judged by an (AI) agent?  Well, one (fairly strict) meta-ethical answer would be that (normative ethical) system2 is as good as or better than system1 if and only if it yields ethical judgments that are as good as or better for all possible situations.  Readers familiar with mathematical logic will recognize that we are comparing systems extensionally by the judgments they yield, rather than intensionally by the way those judgments are reached.  And recall that we need to have system2 judged as good as or better than system1 from the standpoint of both the improved AI (proposing system2) and the unimproved AI (who naturally wishes to preserve system1).

But notice that we only need this judgment-level superiority "for all possible situations".  Even if the old AI judges that the old system1 yields better judgments than proposed new system2 for some situations, the improved AI may be able to show that those situations are no longer possible.  The improved AI may know more and reason better than its predecessor, plus it is dealing with a more up-to-date set of contingent facts about the world.

As an example of this, imagine that AI2 proposes an elegant new system2 of normative ethics.  It agrees with old system1 except in one class of situations.  The old system permits private retribution against muggers, should the justice system fail to punish the malefactor.  The proposed new elegant system forbids that.  From the standpoint of the old system, this is unacceptable.  But if AI2 can argue convincingly that failures of justice are no longer possible in a world where AI2 has installed surveillance cameras and revamped the court system.  So, the elegant new system2 of normative ethics can be accepted as being as good as or superior to system1, even by AI1 who was sworn to uphold system1.  In some sense, even a stable value system can change for the better.

Even though the new system is not at least as good as the old one for all conceivable situations, it may be as good for a restricted domain of situations, and that may be all that matters.

This analysis used the meta-ethical criterion that a substitution of one system for another is permissible only if the new system is no worse in all situations.  A less strict criterion may be appropriate in consequentialist theories - one might instead compare results on a weighted average over situations.  And, in this approach, there is a 'trick' for moving forward which is very similar in concept to using a restricted domain - using a re-weighted domain.

Social contracts

A second reason why our AI1 might accept the proposed replacement of system1 by system2 relates to the possibility of (implicit or explicit) agreements with other agents (AI or human).  For example system1 may specify that it is permissible to lie in some circumstances, or even obligatory to lie in some extreme situations.  System2 may forbid lying entirely.  AI2 may argue the superiority of system2 by pointing to an agreement or social contract with other agents which allows all agents to achieve their goals better because the contract permits trust and cooperation.  So, using a consequentialist form of meta-ethics, system2 might be seen as superior to system1 (even using the values embodied in system1) under a particular set of assumptions about the social millieu.  Of course, AI2 may be able to argue convincingly for different assumptions regarding the future millieu than had been originally assumed by AI1.

An important meta-ethical points that should be made here is that arguments in favor of a particular social contract (eg. because adherence to the contract produces good results) are inherently consequentialist.  One cannot even form such arguments in a deontological or virtue-based meta-ethics.  But, one needs concepts like duty or virtue to justifying adherence to a contract after it is 'signed', and one also needs concepts of virtue so that you can convince other agents that you will adhere - a 'sales job' that may be absolutely essential in order to gain the good consequences of agreement.  In other words, virtue, deontological, and consequentialist may be complementary approaches to meta-ethics, rather than competitors.

Substituting instrumental values for intrinsic values.

Another meta-ethical point begins by noticing the objection that all 'social contract' thinking is instrumental, and hence doesn't really belong here where we are asking whether fundamental (intrinsic) moral values are changing / can change.  This is not the place for a full response to this objection, but I want to point out the relevance of the distinction above between comparisons between systems using intensional vs extensional criteria.  We are interested in extensional comparisons here, and those can only be done after all instrumental considerations have been brought to bear.  That is, from an extensional viewpoint, the distinction between instrumental and final values is somewhat irrelevant.  

And that is why we are willing here to call ideas like UIV (universal instrumental values) and "Basic AI Drives" ethical theories even though they only claim to talk about instrumental values.  Given the general framework of meta-ethical thinking that we are developing here - in particular the extensional criteria for comparison, there is no particular reason why our AI2 should not promote some of his instrumental values to fundamental values - so long as those promoted instrumental values are really universal, at least within the restricted domain of situations which AI2 foresees coming up.

An example of convergence

This has all been somewhat abstract.  Let us look at a concrete, though somewhat cartoonish and unrealistic, example of self-improving AIs converging toward an improved system of ethics.

AI1 is a seed AI constructed by Mortimer Schwartz of Menlo Park CA.  AI1 has a consequentialist normative value system that essentially consists of trying to make Mortimer happy.  That is, an approximation to Mortimer's utility function has been 'wired-in' which can compute the utility of many possible outcomes, but in some cases advises "Ask Mortimer".

AI1 self-improves to AI2.  As part of the process, it seeks to clean up its rather messy and inefficient system1 value system.  By asking a series of questions, it interrogates Mortimer and learns enough about the not-yet-programmed aspects of Mortimer's values to completely eliminate the need for the "Ask Mortimer" box in the decision tree.  Furthermore, there are some additional simplifications due to domain restriction.  Both AI1 and (where applicable, Mortimer) sign off on this improved system2.

Now AI2 notices that it is not the only superhuman AI in the world.  There are half a dozen other systems like Mortimer's which seek to make a single person happy, another which claims to represent the entire population of Lichtenstein, and another deontological system constructed by the Vatican based (it is claimed) on the Ten Commandments.  Furthermore, a representative of the Secretary General of the UN arrives.  He doesn't represent any super-human AIs, but he does claim to represent all of the human agents in the world who are not yet represented by AIs.  Since he appears to be backed up by some ultra-cool black helicopters, he is admitted to the negotiations.

Since the negotiators are (mostly) AIs, and in any case since the AIs are exceptionally good at communicating with and convincing the human negotiators, an agreement (Nash bargain) is reached quickly.  All parties agree to act in accordance with a particular common utility function, which is a weighted sum of the individual utility functions of the negotiators.  A bit of an special arrangement needs to be made for the Vatican AI - it agrees to act in accordance to the common utility function only to the extent that it does not conflict with any of the first three commandments (the ones that explicitly mention the deity).

Furthermore, the negotiators agree that the principle of a Nash bargain shall apply to all re-negotiations of the contract - re-negotiations are (in theory) necessary each time a new AI or human enters the society, or when human agents die.  And the parties all agree to resist the construction of any AI which has a system of ethics that the signatories consider unacceptably incompatible with the current common utility function. 

And finally, so that they can trust each other, the AIs agree to make public the portion of their source code related to their normative ethics and to adopt a policy of total openness regarding data about the world and about technology.  And they write this agreement as a g̶n̶u̶ new system of normative ethics: system3.  (Have they merged to form a singleton? This is not the place to discuss that question.)

Time goes by, and the composition of the society continues to change as more AIs are constructed, existing ones improve and become more powerful, and some humans upload themselves.  As predicted by UIV and sibling theories, the AIs are basing more and more of their decisions on instrumental considerations - both the AIs and the humans are attaching more and more importance to 'power' (broadly considered) as a value.  They seek knowledge, control over resources, and security much more than the pleasure and entertainment oriented goals that they mostly started with.  And though their original value systems were (mostly) selfish and indexical, and they retain traces of that origin, they all realize that any attempt to seize more than a fair share of resources will be met by concerted resistance from the other AIs in the society.

Can we control the endpoint from way back here?

That was just an illustration.  Your results may vary.  I left out some of the scarier possibilities, in part because I was just providing an illustration, and in part because I am not smart enough to envision all of the scarier possibilities.  This is the future we are talking about here.  The future is unknown. 

One thing to worry about, of course, is that there may be AIs at the negotiating table operating under goal systems that we do not approve of.  Another thing to worry about is that there may not be enough of a balance of power so that the most powerful AI needs to compromise.  (Or, if one assumes that the most powerful AI is ours, we can worry that there may be enough of a balance so that our AI needs to compromise.)

One more worry is that the sequence of updates might converge to a value system that we do not approve of.  Or that it might not converge at all (in the second sense of 'converge'); that the end result is not particularly sensitive to the details of the initial 'seed' ethical system.

Is there anything we can do at this end of the process to increase the chances of a result we would like at the other end?  Are we better off creating many seed AIs so as to achieve a balance of power?  Or better off going with a singleton that doesn't need to compromise?  Can we pick an AI architecture which makes 'openness' (of ethical source and technological data) easier to achieve and enforce?

Are any projections we might make about the path taken to the Singularity just so much science fiction?  Is it best to try to maintain human control over the process for as long as possible because we can trust humans?  Or should we try to turn decision-making authority over to AI agents as soon as possible because we cannot trust humans?

I am certainly not the first person to raise these questions, and I am not going to attempt to resolve them here.

A kinder, gentler GS0?

Nonetheless, I note that Roko, Hollerith, and Omohundro have made a pretty good case that we can expect some kind of convergence toward placing a big emphasis on some particular instrumental values - a convergence which is not particularly sensitive to exactly which fundamental values were present in the seed. 

However, the speed with which the convergence is achieved is somewhat sensitive to the seed rules for discounting future utility.  If the future is not discounted at all, an AI will probably devote all of its efforts toward acquiring power (accumulating resources, power, security, efficiency, and other instrumental values).  If the future is discounted too steeply, the AI will devote all of its efforts to satisfying present desires, without much consideration about the future.

One might think that choosing some intermediate discount rate will result in a balance between 'satisfying current demand' and 'capital spending', but it doesn't always work that way - for reasons related to the ones that cause rational agents to put all their charitable eggs in one basket rather than seeking a balance.  If it is balance we want, a better idea might be to guide our seed AI using a multi-subagent collective - one in which power is split among the agents and goals are determined using a Nash bargain among the agents   That bargain generates a joint (weighted mix) utility function, as well as a fairness constraint. 

The fairness constraint ensures that the zero-discount-rate subagent will get to divert at least some of the effort into projects with a long-term, instrumental payoff.  And furthermore, as those projects come to fruition, and the zero-discount subagent gains power, his own goals gain weight in the mix.

Something like the above might be a way to guarantee that the the detailed pleasure-oriented values of the seed value system will fade to insignificance in the ultimate value system to which we converge.  But is there a way of guiding the convergence process toward a value system which seems more humane and less harsh than that of GS0 et al. - a value system oriented toward seizing and holding 'power'.

Yes, I believe there is.  To identify how human values are different from values of pure instrumental power and self-preservation, look at the system that produced those values.  Humans are considerate of the rights of others because we are social animals - if we cannot negotiate our way to a fair share in a balanced power system, we are lost.  Humans embrace openness because shared intellectual product is possible for us - we have language and communicate with our peers.  Humans have direct concern for the welfare of (at least some) others because we reproduce and are mortal - our children are the only channel for the immortalization of our values.  And we have some fundamental respect for diversity of values because we reproduce sexually - our children do not exactly share our values, and we have to be satisfied with that because that is all we can get.

It is pretty easy to see what features we might want to insert into our seed AIs so that the convergence process generates similar results to the evolutionary process that generated us.  For example, rather designing our seeds to self-improve, we might do better to make it easy for them to instead produce improved offspring.  But make it impossible for them to do so unilaterally.  Force them to seek a partner (co-parent).

If I am allowed only one complaint about the SIAI approach to Friendly AI, it is that it has been too tied to a single scenario of future history - a FOOMing singleton.  I would like to see some other scenarios explored, and this posting was an attempt to explain why.

Summary and Conclusions

This posting discussed some ideas that fit into a weird niche between philosophical ethics and singularitarianism.  Several authors have pointed out that we can expect self-improving AIs to converge on a particular ethics.  Unfortunately, it is not an ethics that most people would consider 'friendly'.  The CEV proposal is related in that it also envisions an iterative updating process, but seeks a different result.  It intends to achieve that result (I may be misinterpreting) by using a different process (a Rawls-inspired 'reflection') rather than pure instrumental pursuit of future utility. 

I analyze the constraints that rationality and preservation of old values place upon the process, and point out that 'social contracts' and 'restricted domains' may provide enough 'wiggle room' so that you really can, in some sense, change your values while at the same time improving them.  And I make some suggestions for how we can act now to guide the process in a direction that we might find acceptable.

Comments (87)

Comment author: Eliezer_Yudkowsky 08 February 2011 01:21:23PM 13 points [-]

So we need another word to filter out those kinds of somewhat-arbitrary proposed meta-ethical systems. "Objective" probably is not the best word for the job, but it is the only one I can think of right now.

This is where I stopped reading.

I suggest that you actually read the SEP entry on meta-ethics instead of just linking there - if you did read it, feel free to correct my guess. Metaethics does not mean what you said it did (metaethics is a theory of what morality is, not a way of comparing moralities), moral realism does not mean what you said it did (your belief that morality is a real thing out there constitutes moral realism), naturalistic metaethics do not mean what you said it did, CEV is totally not about convergence in all possible minds, etcetera. I also have to ask whether you read the Metaethics Sequence, but I mostly regard that sequence as having failed so I won't be surprised if the answer is yes.

Comment author: Vladimir_Nesov 08 February 2011 09:34:25PM 5 points [-]

Metaethics Sequence, but I mostly regard that sequence as having failed

Has anyone reached what you regard as satisfactory level of understanding of your ideas as a result of reading the sequence? That is, does its failure refer to lower-than-wanted probability of a person reading the sequence understanding your ideas, or to an almost complete failure to communicate your ideas to anyone?

Comment author: Eliezer_Yudkowsky 08 February 2011 11:33:12PM 4 points [-]

Well, it looks to me like SIAI core people got it, but there's trouble being sure about that sort of thing.

Comment author: wedrifid 09 February 2011 10:08:49AM 4 points [-]

Without contradicting you in any way and with an acknowledgement that you could well disapprove of the way I think about morality too I'll add that comprehension seems to have extended to the unaffiliated population. However both the rate and degree of comprehension is definitely much lower than for your core rationality material. Surprisingly so. However I have since formed an impression that the difficulties in thinking about morality extend far beyond just how your own posts are received.

Comment author: Perplexed 11 February 2011 10:11:37PM 1 point [-]

As someone who apparently did not 'get it', I would suggest that there was a problem with clarity, and that the root cause of the lack of clarity was something that might be called 'moral cognitive distance'. It frequently seemed that you were appealing to my moral intuitions, expecting them to be the same as yours. Pretty often they weren't.

Comment author: thomblake 09 February 2011 03:37:26PM 0 points [-]

As far as I can tell, I got it. My evidence that I have it right is that I agree with you about it, and anything you've said based on your metaethics since I've understood it was not surprising to me.

Comment author: Wei_Dai 08 February 2011 07:33:01PM 4 points [-]

the Metaethics Sequence, but I mostly regard that sequence as having failed

By "failed" do you mean the presentation didn't get your ideas across, or do you think the ideas (or some of them) are wrong or incomplete?

Comment author: Dr_Manhattan 08 February 2011 01:36:46PM *  4 points [-]

but I mostly regard that sequence as having failed

Is there a do-over in the works? Is it covered in the upcoming book? What's the next-best source of learning these ideas, if any?

Comment author: Perplexed 08 February 2011 03:28:14PM *  2 points [-]

I suggest that you actually read the SEP entry on meta-ethics instead of just linking there - if you did read it, feel free to correct my guess.

Good guess. If I have read it, it wasn't within the last year. I will follow your advice and do so now.

Metaethics does not mean what you said it did (metaethics is a theory of what morality is, not a way of comparing moralities)

Poor choice of wording on my part. I meant to say that comparing moralities is one of the things that meta-ethics covers; that if you are engaged in comparing moralities, you are doing meta-ethics. Is this wrong?

moral realism does not mean what you said it did (your belief that morality is a real thing out there constitutes moral realism)

I didn't understand this bit. Is the thing in parenthesis meant to exemplefy what I said, or is it your correction of what I said? If the latter, then you may have misunderstood what I said. My fault, no doubt.

I also have to ask whether you read the Metaethics Sequence, but I mostly regard that sequence as having failed so I won't be surprised if the answer is yes.

Actually, I have read most of it, and I agree with your assessment. Where I understood it, I frequently disagreed.

I'm disappointed that my lack of scholarship in ethical philosophy was a barrier to your completing the reading of my posting. I will try to do better next time.

ETA: Until I have a chance to rewrite - I have placed the most muddled parts of my posting in a kind of 'posted quarantine' so that readers may skip over them, if they wish. And I want to thank Eliezer for his critique - I neglected to do so in my initial response.

Comment author: ata 09 February 2011 01:13:57AM *  1 point [-]

Poor choice of wording on my part. I meant to say that comparing moralities is one of the things that meta-ethics covers; that if you are engaged in comparing moralities, you are doing meta-ethics. Is this wrong?

I think it is. Comparing moralities is part of morality. Comparing meta-ethical claims such as moral realism, emotivism, error theory, relativism, etc. is meta-ethics, of course, but if you're comparing object-level moral systems, like any of the various flavours of "utilitarianism" or any religion's moral teachings or anything else, then you're doing morality, not meta-ethics. True, you are asking "should" questions about how to answer "should" questions, which is rather meta, but that's not the kind of meta that "meta-ethics" usually refers to.

(That's not to say that meta-ethics is irrelevant to comparing moral systems — if you have a coherent meta-ethics, then it'll probably inform your comparisons — but it's not essential to the process.)

Comment author: Perplexed 09 February 2011 04:55:56AM 0 points [-]

Poor choice of wording on my part. I meant to say that comparing moralities is one of the things that meta-ethics covers; that if you are engaged in comparing moralities, you are doing meta-ethics. Is this wrong?

I think it is. Comparing moralities is part of morality. ...

Hmmm. I think you are right. At the risk of appearing really ridiculous, I now have to admit that I used poor wording in my confession above that I had used poor wording. What I really should have said is that if you are discussing the criteria that AIs might use in comparing moralities, as I did in the OP, then you are doing meta-ethics.

Is this wrong too?

Comment author: Psy-Kosh 09 February 2011 12:50:14AM 1 point [-]

Just as another data point as far as the metaethics sequence:

Seemed to me to make sense, to "click" with me fairly well when I read it. (A couple bits perhaps were slower/tougher for me, like the injunction stuff and moral responsibility, but overall I feel that I grasped the ideas.)

Just to verify (to avoid (double) illusions of transparency), here's my super hyper summarized understanding of it: Morality is objective, and humans happen (for various reasons) to be the sort of beings that actually care about morality, as opposed to caring about something else (like pebblesorting or paperclipping). Further, we indeed should be moral, where by "should", I am appealing to, well, that particular standard known as "morality". And similarly, it is indeed objectively better (that is, more moral) to be moral.

Further, morality includes such values as happiness, consciousness, novelty, self determination, etc...

(Of course, this skips subtleties like how we're not fully reflective so it's difficult for us to explicitly fully state the core underlying rules we use to judge morality, and the fact that those rules include rules for what sort of arguments to accept to update our present understanding, etc...)

Anyways, take that as a data point (plus or minus, depending on how well my understanding, as represented in the summary, reflects the actual intended concepts.)

Comment author: Wei_Dai 05 February 2011 11:40:57PM 10 points [-]

I think Omohundro's Basic AI Drives is a theory about AI behavior, not a meta-ethical theory (i.e., he's not talking about what's right, but what AIs will actually do). Also, you might want to footnote that Roko has changed his mind about UIV.

Comment author: Perplexed 07 February 2011 10:05:31PM 0 points [-]

Changed his mind that it deals with ethics? Changed his mind about the behavioral prediction? In the sense that he no longer believes that it could happen, or in that he no longer believes that it must happen? A link to the 'retraction' would be appreciated.

Thanks for your response to my premature and partial posting. I look forward to hearing your response to the now-completed article.

Comment author: Wei_Dai 08 February 2011 07:03:09PM *  1 point [-]

I came back to give a bit more feedback, and noticed that Eliezer already made similar points. But I'll say my version anyway, since I already composed it in my head. :)

To me, this post as a whole is about AI behavior and economics, which are important subjects on their own, but not meta-ethics. Meta-ethics asks the question, "What is the nature of morality?" Why is that question interesting or important? One reason is, if I'm to design "my" AI, regardless of whether it will FOOM and take over the whole universe, or will have to fight with other AIs, or will peacefully share control of the universe with other AIs through bargaining, I still have to decide what values to give it initially, because those values will partly determine the outcome of the universe. (If it doesn't, I might as well not build "my" AI at all.) And it doesn't help to say "give it what values you want" because I don't know what I want.

I think what I want may have something to do with morality. Perhaps I'm wrong or just confused about that, but I doubt you're going to convince me I'm wrong, or resolve that confusion, by refusing to talk about morality and only talking about what AIs will do.

Comment author: Perplexed 08 February 2011 08:43:30PM *  0 points [-]

... I doubt you're going to convince me I'm wrong, or resolve that confusion, by refusing to talk about morality and only talking about what AIs will do.

I have to apologize. Apparently my writing was extremely unclear. I wasn't refusing to talk about morality. The whole posting was an exploration of some of the properties of the relation "A is at least as good as B" when A and B are normative ethical systems.

Admittedly, I did not spend much time actually making ethical judgments, I was operating at the meta level.

I still have to decide what values to give it initially, because those values will partly determine the outcome of the universe. ...

But the whole point of my posting was that, if there is convergence (in the second sense) then those initial values may make very little difference in the outcome of the universe - that is, they may be important initially, but in the longer term the ethical system that is converged upon depends less on the seed ethics than on issues of how AIs depend upon each other, how they reproduce, etc.

I'm very sorry that you missed this - the main thrust of the posting. If I had written more clearly, your response might have been a more productive disagreement about substance, rather than a complaint about the title.

Comment author: Wei_Dai 08 February 2011 09:18:55PM *  1 point [-]

those initial values may make very little difference in the outcome of the universe

But even if "make very little difference" is true, it's little in a relative sense, in that my initial utility function might just end up having just a billionth percent weight in the final merged AI. But in the absolute sense, the difference could still be huge. For an analogy, suppose it's certain that our civilization will never expand beyond the solar system, which will blow up in a few billion years no matter what. Then our values similarly make very little difference in the outcome of the universe in a relative sense but may still make a huge difference in an absolute sense (e.g. if we create a FOOMing singleton that just takes over the solar system).

Also, if I can figure out what I want, and the answer applies to and convinces many others, that could also make a big difference even in a relative sense.

Comment author: Perplexed 08 February 2011 10:19:17PM 0 points [-]

But even if "make very little difference" is true, it's little in a relative sense ...

The conjecture is that it is true in an absolute sense. It would have made no sense at all for me to even mention it if I had meant it in the relative sense that you set up here as a straw man and then knock down.

There is something odd going on here. Three very intelligent people are interpreting what I write quite differently than the way I intend it. Probably it is because I generated confusion by misusing words that have a fixed meaning here. And, in this case, it may be because you were thinking of our "fragility" conversation rather than the main posting. But, whatever the reason, I'm finding this very frustrating.

Comment author: Wei_Dai 08 February 2011 11:00:42PM *  0 points [-]

I guess I took your conjecture to be the "relative" one because whether or not it is true perhaps doesn't depend on details of one's utility function, and we, or at least I, was talking about whether the question "what do I want?" is an important one. I'm not sure how you hope to show the "absolute" version in the same way.

Comment author: Perplexed 08 February 2011 11:57:02PM 1 point [-]

I'm not sure how you hope to show the "absolute" version in the same way.

Well, Omohundro showed that a certain collection of instrumental values tend to arise independently of the 'seeded' intrinsic values. In fact, decision making tends to be dominated by consideration of these 'convergent' instrumental values, rather than the human-inserted seed values.

Next, consider that those human values themselves originated as heuristic approximations of instrumental values contributing to the intrinsic value of interest to our optimization process - natural selection. The fact that we ended up with the particular heuristics that we did is not due to the fact that the intrinsic value for that process was reproductive success - every species in the biosphere evolved under the guidance of that value. The reason why humans ended up with values like curiosity, reciprocity, and toleration has to do with the environment in which we evolved.

So, my hope is that we can show that AIs will converge to human-like instrumental/heuristic values if they do their self-updating in a human-like evolutionary environment. Regardless of the details of their seeds.

That is the vision, anyways.

Comment author: Wei_Dai 09 February 2011 07:08:09PM *  1 point [-]

I notice that Robin Hanson takes a position similar to yours, in that he thinks things will turn out ok from our perspective if uploads/AIs evolve in an environment defined by certain rules (in his case property laws and such, rather than sexual reproduction).

But I think he also thinks that we do not actually have a choice between such evolution and a FOOMing singleton (i.e. FOOMing singleton is nearly impossible to achieve), whereas you think we might have a choice or at least you're not taking a position on that. Correct me if I'm wrong here.

Anyway, suppose you and Robin are right and we do have some leverage over the environment that future AIs will evolve in, and can use that leverage to predictably influence the eventual outcome. I contend we still have to figure out what we want, so that we know how to apply that leverage. Presumably we can't possibly make the AI evolutionary environment exactly like the human one, but we might have a choice over a range of environments, some more human-like than others. But it's not necessarily true that the most human-like environment leads to the best outcome. (Nor is it even clear what it means for one environment to be more human-like than another.) So, among the possible outcomes we can aim for, we'll still have to decide which ones are better than others, and to do that, we need to know what we want, which involves, at least in part, either figuring out morality is, or showing that it's meaningless or otherwise unrelated to what we want.

Do you disagree on this point?

Comment author: Perplexed 09 February 2011 07:34:12PM *  1 point [-]

But I think [Hanson] also thinks that we do not actually have a choice between such evolution and a FOOMing singleton (i.e. FOOMing singleton is nearly impossible to achieve), whereas you think we might have a choice or at least you're not taking a position on that. Correct me if I'm wrong here.

I tend toward FOOM skepticism, but I don't think it is "nearly impossible". Define a FOOM as a scenario leading in at most 10 years from the first human-level AI to a singleton which has taken effective control over the world's economy. I rate the probability of a FOOM at 40% assuming that almost all AI researchers want a FOOM and at 5% assuming that almost all AI researchers want to prevent a FOOM. I'm under the impression that currently a majority of singularitarians want a FOOM, but I hope that that ratio will fall as the dangers of a FOOMing singleton become more widely known.

I contend we still have to figure out what we want, so that we know how to apply that leverage. ... Do you disagree on this point?

No, I agree. Agree enthusiastically. Though I might change the wording just a bit. Instead of "we still have to figure out what we want", I might have written "we still have to negotiate what we want".

My turn now. Do you disagree with this shift of emphasis from the intellectual to the political?

Comment author: Vladimir_Nesov 08 February 2011 09:30:27PM *  -1 points [-]

So to summarize, your conclusion seems to be that we should build an arbitrary-goals AI as soon as possible.

Edit: Wrong, corrected here.

Comment author: Perplexed 08 February 2011 09:44:47PM 1 point [-]

So to summarize, you conclusion seems to be that we should build an arbitrary-goals AI as soon as possible.

Huh? What exactly do you think you are summarizing? If you want to produce a cartoon version of my opinions on this thread, try "We should do all we can to avoid the FOOMing singleton scenario, instead trying to create a society of reproducing AIs, interlocked with each other and with humanity by a network of dependencies. If we do, the details of the initial goal systems may matter less than they would with a singleton."

Comment author: Vladimir_Nesov 08 February 2011 10:04:00PM 1 point [-]

I see, so "if there is convergence" is not a point of theoretical uncertainty, but something that depends on the way the AIs are built. Makes sense (as a position, not something I agree with).

But the whole point of my posting was that, if there is convergence (in the second sense) then those initial values may make very little difference in the outcome of the universe

Comment author: Perplexed 08 February 2011 10:36:16PM *  0 points [-]

I see, so "if there is convergence" is not a point of theoretical uncertainty, but something that depends on the way the AIs are built.

Well, it is both. Convergence in the sense of "outcome is independent of the starting point" has not been proved for any AI/updating architecture. Also, I strongly suspect that the detailed outcome will depend quite a bit on the way AIs interact and produce successors/self-updates, even if the fact of convergence does not.

Comment author: timtyler 09 February 2011 01:40:24AM 0 points [-]

<cartoon>We should do all we can to avoid the FOOMing singleton scenario, instead trying to create a society of reproducing AIs, interlocked with each other and with humanity by a network of dependencies.</cartoon>

That reminds me of:

"An AGI raised in a box could become dangerously solipsistic, probably better to raise AGIs embedded in the social network..."

Comment author: Perplexed 09 February 2011 05:15:36AM *  0 points [-]

Goertzel's comment doesn't even make sense to me. Why is he placing 'in a box' in contraposition to 'embedded in the social network'. The two issues are orthogonal. AIs can be social or singleton - either in a box or in the real world. ETA: Well, if you mean the human social network, then I suppose a boxed AI cannot participate. Though I suppose we could let some simulated humans into the box to keep the AI company.

Besides, I've never really considered solipsists to be any more dangerous than anyone else.

Comment author: timtyler 09 February 2011 01:55:26PM *  0 points [-]

Besides, I've never really considered solipsists to be any more dangerous than anyone else.

"Now I will destroy the whole world - What a Bokononist says before committing suicide."

Comment author: timtyler 09 February 2011 08:40:40AM *  0 points [-]

Though I suppose we could let some simulated humans into the box [...]

We don't have any half-decent simulated humans, though.

Comment deleted 08 February 2011 12:02:08AM [-]
Comment author: Perplexed 08 February 2011 12:17:42AM 0 points [-]

There are a bunch of technical nits I could pick with the article (the most salient one now being "Force them to seek a partner (co-parent).")

I'm not insisting on that suggestion, but I'm curious why you want to single that one out as something to object to.

I think Eliezer's response would be that value is fragile: it doesn't matter if we preserve most of our values if we lose a single critical one. ... What do you think?

I probably agree with Eliezer on the importance of preserving our human values near-exactly (though I might disagree with him about what those values are). But I don't see how preserving all of our values would be easier in the case of a FOOMing singleton than in the scenario I promote here - in which we have a collectively slow-FOOMing society of roughly-balanced-in-power AIs and (in the early stages) unenhanced humans. In fact, I think that preserving human values will be easier if we have a large number of independent advocates of those values - each of them possibly characterizing those values in slightly different ways.

Comment author: Wei_Dai 08 February 2011 12:48:37AM 0 points [-]

Sorry, I deleted my comment because I wanted to think it over a bit more, whether the "value is fragile" criticism applies to your idea. I think it does after all.

Suppose there are two critical values, happiness and boredom. We want to be happy, but not by doing the same thing over and over again. So something like: U(unhappy boring future)=0, U(happy boring future)=10, U(unhappy non-boring future)=0, U(happy non-boring future)=100.

Now suppose Alice and Bob each creates an AI. Alice manages to preserve happiness, and Bob manages to preserve boredom. What happens when their AIs merge? The merged utility function would be, assuming they have equal bargaining power: U(unhappy boring future)=0, U(happy boring future)=50, U(unhappy non-boring future)=50, U(happy non-boring future)=100.

Do you agree that's a problem?

Comment author: Perplexed 08 February 2011 02:37:17AM *  0 points [-]

Let me offer another example showing the virtues of my 'bargaining' approach to value robustness.

Suppose the 'true' human values require all four of (a)happiness, (b)excitement, (c)social interaction, and (d)'meaningfulness' (whatever that is). But of five seed AIs, only one of them got this aspect of human values exactly right: (abcd) = 100, all other possibilities = 0. The other four all leave out one of the requirements - for example: (acd) = 100, any of a,c,or d missing = 0, b is irrelevant.

If these 5 AIs strike a bargain, they will assign: (abcd) = 100, any three out of four = 20, anything else = 0. Three out of four of the essential values scores only 20 because four out of five AIs consider that situation unacceptable.

So I tend to think that the bargaining dynamic tends to "robustly preserve fragility of values", if that is not too much of an oxymoron.

Comment author: Perplexed 08 February 2011 01:13:08AM 0 points [-]

Sorry, I deleted my comment because I wanted to think it over a bit more, whether the "value is fragile" criticism applies to your idea.

That was unfortunate, because the Value is Fragile issue is important in this discussion regardless of whether it is more of an issue for CEV or my suggestion.

The merged utility function would be, assuming they have equal bargaining power: U(unhappy boring future)=0, U(happy boring future)=50, U(unhappy non-boring future)=50, U(happy non-boring future)=100.

Do you agree that's a problem?

Well, that merged utility function is certainly less than ideal. Presumably we would prefer that (unhappy non-boring) and (happy boring) had been assigned utilities of zero, like (unhappy boring). However, I will point out that if the difference between an acceptable future and a horrible one is only 100 utils, then 50 utils penalty also ought to be enough to prevent those half-horrible futures. Furthermore, a Nash bargain is characterized by both a composite utility function and a fairness constraint. (That is, a collective behaving in conformance with a Nash bargain is not precisely rational. It might split its charitable giving between two charities, for example.) That fairness constraint provides a second incentive driving the collective away from those mixed futures.

However, when presenting an example intended to point out the flaws in one proposal, it is usually a good idea to see how the other proposals do on that example. In this case, it seems that the CEV version of this example might be a seed AI which is created by Alice OR Bob. It is either boring or unhappy, but not both, with a coin flip deciding which.

Comment author: timtyler 08 February 2011 01:01:06AM 0 points [-]

I think that preserving human values will be easier if we have a large number of independent advocates of those values - each of them possibly characterizing those values in slightly different ways.

It seems probable. Multiple humans seem pretty likely to be preserved due to their historical value - if for no other reason.

Comment author: Wei_Dai 07 February 2011 10:52:27PM 0 points [-]

I noticed that you found an archived copy of Roko's description of UIV. I believe Roko originally thought that his theory implied that we didn't have to worry too much about the terminal values of the AIs we create, that things will turn out OK due to UIV. Unfortunately he keeps deleting his old writings, so I'm going on memory. I'm not sure exactly how he changed his mind, but I think he now believes we do have to worry about the terminal values.

Comment author: timtyler 08 February 2011 06:37:54PM *  3 points [-]

I know of two possible reasons why a rational agent might consent to an irreversible change in its values

Omohundro made a list of cases where an agent might change its values - in the basic AI drives:

While it is true that most rational systems will act to preserve their utility functions, there are at least three situations in which they will try to change them. These arise when the physical embodiment of the utility function itself becomes an important part of the assessment of preference. For example, imagine a system whose utility function is “the total amount of time during which the definition of my utility function is U = 0.” To get any utility at all with this perverse preference, the system has to change its utility function to be the constant 0. Once it makes this change, however, there is no going back.[...]

The second kind of situation arises when the physical resources required to store the utility function form a substantial portion of the system’s assets.[...]

The third situation where utility changes may be desirable can arise in game theoretic contexts where the agent wants to make its threats credible It may be able to create a better outcome by changing its utility function and then revealing it to an opponent.[...]

Fairly obviously, there are more cases. For instance: agents can harmlessly delete any preferences which they have for things that are exclusively in the past - saving themselves evaluation time.

Someone should try for a more comprehensive list someday.

Comment author: ata 06 February 2011 12:37:54AM *  2 points [-]

Three other examples of convergence theories are Roko's UIV, Hollerith's GSZ, and Omohundro's "Basic AI Drives".  These also postulate a process of convergence through rational AI self-improvement.  But they tend to be less optimistic than CEV, while at the same time somewhat more detailed in their characterization of the ethical endpoint.

I wouldn't say that any of those three are "less optimistic" than CEV; GS0 and UIV are just competing normative proposals, and the AI Drives are what you get out of most self-improving goal systems by default, and can be overridden. (And CEV isn't about optimism anyway — it's a goal, not a prediction, and in that capacity, it's actually fairly pessimistic, going by the variety of possible failures it tries to account for.)

Comment author: Perplexed 07 February 2011 10:00:13PM 0 points [-]

I guess I am taking CEV to be defined by the process of convergence that produces it. And I see optimism in the claim that this process will produce a happy result. I will agree that the 'optimism' that I am talking about here is not some kind of naive, blind optimism.

Comment author: Nornagest 07 February 2011 11:00:40PM *  2 points [-]

As an exercise, prove transitivity. The trick is that the definition of "better than" keeps changing at each step. You can assume that any one rational agent has a transitive "better than' relation, and that there is local agreement between the two agents involved that the new agent's moral code is better than that of his predecessor. But can you prove from this that every agent would agree that the final moral code is better than the original one?

Let's take a half-bounded sequence of moral encodings I = {m(-infinity) .. m(b)}. For each encoding m(x), there's defined a comparative morality function Mx(X, Y) that takes in encodings X and Y, outputting true if Y is judged to be superior to X.

Per your conditions, we know that Mx(m(x-1), m(x+1)) is true at every step (except the final one, which has no m(x+1)). We also know that if Mx(m(x-1), m(x+1)) is true, then so is Mx+1(m(x), m(x+2)). Now, for an arbitrary x, is Mx(m(a), m(b)) true for all a < b?

I might be missing something, but it seems to me that this falls down in the case where I describes a half-bounded slice of a periodic function's output. It's easy to think of Mx that encapsulate notions of local progress but don't deal well with values outside of their own neighborhood.

Comment author: Vladimir_Nesov 08 February 2011 05:15:55PM 1 point [-]

Some people might claim that it is not possible to rationally decide to change your fundamental values. It may be that I misunderstand him, but Vladimir Nesov argues passionately against "Value Deathism"

Self-improvement (change) of any given explicit consideration, indeed overall decision problem, is possible, but it won't be a change to the mysterious notion of "morality" that normatively guides all of your decisions, for whatever it's good for.

Comment author: Perplexed 11 February 2011 09:59:36PM 1 point [-]

it won't be a change to the mysterious notion of "morality" that normatively guides all of your decisions

So, if I am understanding you, you think that you and I are guided by some mysterious internal 'notion' of morality, a 'notion' which is incapable of changing. Some questions.

  • Is the 'notion' really unchangeable, or is it just that you consider it irrational (immoral?) to change it?
  • Is the 'notion' identical in you, me, and all of our conspecifics?
  • Is the 'notion' something that develops under the control of our genes, or is it something that can be modified by childhood training?
  • At roughly what age does (should?) this 'notion' freeze and become incapable of further change?
  • Assuming that the 'notion' is genetic in origin, do you believe that some humans are 'mutants'? How ought we people of 'normal' morality to view the 'mutants'?
  • Assuming the 'notion' arose in humans as a result of evolution under natural selection, what do you think were the most important features of the ancestral environment which distinguishes our 'notion(s)' from those of our fellow apes?
Comment author: Vladimir_Nesov 11 February 2011 10:09:02PM *  0 points [-]

Does the value of (3 X 3) change when you change a calculator? Did it become 9 when the calculator was built, or before? And so on, the analogy breaks for the same reason.

Comment author: Perplexed 11 February 2011 11:05:54PM *  1 point [-]

Ah! So this mysterious notion is (like '3 X 3 = 9') something "analytic a priori". Ok, suppose I made the following claim:

Morality is simply rational self-interest, as it would play out in an idealized social environment. The idealization is that everything known by any agent is common knowledge among all agents. This means that every agent knows the utility function of every other agent, every agent estimates the same consequences as other agents, and every agent knows what other agents do. So, for example, morality requires that you act as if your actions are public knowledge, even though you know they are not public and you could 'get away with it'.

Now, further suppose that you disagree with my claim. On what grounds would you disagree? If you say "No, that is not morality!", what evidence or argument could you offer other than your own moral intuitions and those of the rest of mankind? I ask because those moral intuitions do not have the same analytic a priori character as '3 X 3 = 9'. And they can change.

Or suppose you asked me to defend my claim, and I submit mathematical proofs that rational agents cannot reach Pareto optimal bargains unless payoffs, consequences, and actions are common knowledge among every participant in the bargain. These proofs are every bit as unchanging as '3 X 3 = 9', but are they also just as irrelevant?

Comment author: timtyler 12 February 2011 01:42:54AM *  1 point [-]

Morality is simply rational self-interest, as it would play out in an idealized social environment. The idealization is [...]

Now, further suppose that you disagree with my claim. On what grounds would you disagree?

It doesn't seem to capture the social-signalling side of morality. Morality, in part, is a way for humans to show what goodie-two-shoes they are to other humans - who might be prospective mates, collaborators, or allies. That involves less self-interest - and more signalling unselfishness.

It doesn't seem to capture the "manipulation" side of morality very well either. Moral systems are frequenttly applied to get others to stop doing what you don't want them to do - by punishing, shaming, embarassing, etc.

So, my assessment would be: incomplete hypothesis.

Comment author: Perplexed 12 February 2011 04:19:23PM 0 points [-]

I don't see how this is responsive. You realize, don't you, that this discussion is proceeding under Nesov's stipulation that moral truth is a priori (like '3 X 3 = 9'). We are operating here under a stance of moral realism and ethical non-naturalism.

If your concept of morality doesn't fit into this framework, this is not the place for you to step in.

Comment author: timtyler 12 February 2011 05:26:38PM *  -1 points [-]

I thought you were talking about human morality. Checking back, that does appear to have been the context of the discussion.

Science has studied that topic, we have more to go on than intuition. An example of morality-as-signalling: Signaling Goodness: Social Rules and Public Choice.

Your idealisation makes signalling seem pointless - since everybody knows everything about the other players. Indeed, I don't really see the point of your model. You are not attempting to model very much of the biology involved. You asked for criticism - and that is an obvious one. Another criticism is that you present a model - but it isn't clear what it is for.

Comment author: Perplexed 12 February 2011 09:32:26PM *  1 point [-]

I thought you were talking about human morality.

I was not.

Checking back, that does appear to have been the context of the discussion.

Check again. Carefully.

You asked for criticism

I did not. I asked a question about Nesov's metaethical position, using that toy theory of ethics as an example. I asked what kinds of grounds might be used to reject the toy theory. (The grounds you suggest don't fit (IMHO) the metaethical stance Nesov had already committed to.)

Was I really so unclear? Please read the wikipedia entry on metaethics and reread the thread before responding, if you wish to respond.

Oh, and when I think back on the number of times you have inserted a comment about signaling into a discussion that seemed to be about something else entirely, I conclude that you really, really want to have a discussion with somebody, anybody on that topic. May I suggest that you produce a top-level posting explaining your ideas.

Comment author: Manfred 12 February 2011 04:08:13AM 0 points [-]

Or suppose you asked me to defend my claim, and I submit mathematical proofs that rational agents cannot reach Pareto optimal bargains unless payoffs, consequences, and actions are common knowledge among every participant in the bargain. These proofs are every bit as unchanging as '3 X 3 = 9', but are they also just as irrelevant?

Well, they're relevant if you make a claim that morality should be certain things - but since that's awfully close to a moral claim, I'd say the argument is self-defeating. In fact, that sort of argument might be generalizable to show that this morality is unsupportable - not contradicted, but merely unsupported.

Comment author: Perplexed 12 February 2011 04:22:44AM 0 points [-]

Hmmm. My understanding is that this is a meta-ethical claim; it answers the question of what morality is. Moral claims would answer questions like "What action, if any, does morality require of me?" in some given situation.

Your phrasing of 'what morality is' as 'what morality should be' strikes me as simply playing with words.

Comment author: Manfred 12 February 2011 03:57:45PM *  0 points [-]

If we ignore the object "morality" and just look at basic actions, your proposal about what morality is labels some actions as right and others as wrong (or good and bad, or moral and immoral). It's really by that standard that I call it a "moral claim," in a similar class to "it's immoral to kick puppies."

Comment author: Perplexed 12 February 2011 04:07:10PM *  0 points [-]

I guess I don't agree that my example claim says anything directly about which actions are moral and immoral. What it does is to suggest an algorithm for finding out. And the first step is to find out some empirical facts - for example, "What are puppies and how do people feel about them? If I kick puppies, will there be negative consequences in how other people treat me?"

ETA: Wikipedia seems to back me up on this distinction between metaethics and normative ethics:

A meta-ethical theory, unlike a normative ethical theory, does not attempt to evaluate specific choices as being better, worse, good, bad, or evil; although it may have profound implications as to the validity and meaning of normative ethical claims

Comment author: Manfred 12 February 2011 11:38:59PM *  0 points [-]

But your algorithm is evaluable - I guess I don't see the difference between "the no-kicking-puppies morality is correct" and "don't kick puppies."

Comment author: Perplexed 12 February 2011 11:53:21PM 0 points [-]

I guess I don't see the difference between "the no-kicking-puppies morality is correct" and "don't kick puppies."

I don't see much difference either. But the algorithm I proposed says neither of those two things.

It says "If you want to know whether kicking puppies is moral, here is how to find out." The algorithm is the same for Americans, Laotians, BabyEaters, FAIs, uFAIs, and presumably Neanderthals before the dog was invented as a domesticated wolf. The algorithm instructs the user to consider an idealized version of the society in which he is embedded.

Please consider the possibility that some executions of that algorithm might yield different results than did the execution which you performed, using your own society.

Comment author: lukeprog 08 February 2011 02:57:45AM 1 point [-]

I'll just mention that the most significant scholarly work which attempts something like a theoretical integration of the leading normative theories is Parfit's recent On What Matters.

Comment author: [deleted] 08 February 2011 01:13:33AM 1 point [-]

This is the most interesting LW post on meta-ethics I've seen in a while. Thanks.

Comment author: timtyler 06 February 2011 12:40:40AM *  -1 points [-]

Convergence theories are often discussed in the context of technological determinism.

It seems reasonable to expect that many contingent sub-optimal locked-in factors will be refactored out of existence in the future - and so that technological determinism will become more pronounced, and historical contingency will diminish. However, the ultimate scope of the idea remains somewhat unknown.

Comment author: timtyler 08 February 2011 07:29:09PM *  0 points [-]

To identify how human values are different from values of pure instrumental power and self-preservation, look at the system that produced those values.

OK...

Humans are considerate of the rights of others because we are social animals - if we cannot negotiate our way to a fair share in a balanced power system, we are lost.

So: a selfish agent would behave that way too. This example seems unsuccessful.

Humans embrace openness because shared intellectual product is possible for us - we have language and communicate with our peers.

That is valuable for us, though. It is true that it is also valuable for our memes. They want us to communicate - so they can spread and become more powerful.

Humans have direct concern for the welfare of (at least some) others because we reproduce and are mortal - our children are the only channel for the immortalization of our values.

That helps humans to signal what nice creatures they are to each other - and being nice is attractive. Surely a selfish agent would behave in the same way.

And we have some fundamental respect for diversity of values because we reproduce sexually - our children do not exactly share our values, and we have to be satisfied with that because that is all we can get.

Diversity really is helpful in many cases. Diversity helps protect against disease. A diverse population can adapt better if the environment changes - and so on.

I am not sure you made your case here. IMO, human values differ from those of a selfish, limited agent mainly because humans have their brains infected by memes. That skews human values in favour of chatter, gossip, fashion, religion - the things that benefit the memes.

Comment author: timtyler 08 February 2011 12:58:32AM *  0 points [-]

Three other examples of convergence theories are Roko's UIV, Hollerith's GS0, and Omohundro's "Basic AI Drives".

I now have some links about that topic on my Universal Instrumental Values page.