Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: PrawnOfFate 23 April 2013 09:38:44PM -2 points [-]

"better according to some shared, motivating standard or procedure of evaluation",

I broadly agree. My thinking ties shoulds and musts to rules and payoffs. Wherever you are operating a set of rules (which might be as localised as playing chess), you have certain localised "musts".

It seems to me that different people (including different humans) can have different motivating standards and procedures of evaluation, and apparent disagreements about "should' sentences can arise from having different standards/procedures, or disagreement about whether something is better according to a shared standard/procedure.

I'm very resitant to the idea, promoted by EY in the thread you refenced, that the meaning of should changes. Does he think chess players have a different concept of "rule" to poker players?

Comment author: CCC 23 April 2013 12:12:14PM 2 points [-]

I don't see the theism/atheism debate as a policy debate. There is a factual question underlying it, and that factual question is "does God exist?" I find it very hard to imagine a universe where the answer to that question is neither 'yes' nor 'no'.

In response to comment by CCC on Qualitatively Confused
Comment author: PrawnOfFate 23 April 2013 12:41:00PM 1 point [-]
Comment author: Kawoomba 23 April 2013 11:02:40AM 1 point [-]

Scenario:

1) You wake up in a bright box of light, no memories. You are told you'll presently be born into an Absolute monarchy, your role randomly chosen. You may choose any moral principles that should govern that society. The Categorical Imperative would on average give you the best result.

2) You are the monarch in that society, you do not need to guess which role you're being born into, you have that information. You don't need to make all the slaves happy to help your goals, you can just maximize your goals directly. You may choose any moral principle you want to govern your actions. The Categorical Imperative would not give you the best result.

A different scenario: Clippy and Anti-Clippy sit in a room. Why can they not agree on epistemic facts about the most accurate laws of physics and other Aumann-mandated agreements, yet then go out and each optimize/reshape the world according to their own goals? Why would that make them not rational?

Lastly, whatever Kant's justification, why can you not optimize for a different principle - peak happiness versus average happiness, what makes any particular justifying principle correct across all - rational - agents. Here come my algae!

Comment author: PrawnOfFate 23 April 2013 12:14:54PM -2 points [-]

You are the monarch in that society, you do not need to guess which role you're being born into, you have that information. You don't need to make all the slaves happy to help your goals, you can just maximize your goals directly. You may choose any moral principle you want to govern your actions. The Categorical Imperative would not give you the best result.

For what value of "best"? If the CI is the correct theory of morality, it will necessarily give your the morally best result. Maybe your complaint is that it wouldn't maximise your personally utility. But I don't see why you would expect that. Things like utilitarianism that seek to maximise group utility, don't promise to make everyone blissfully happy individually. Some will lose out.

A different scenario: Clippy and Anti-Clippy sit in a room. Why can they not agree on epistemic facts about the most accurate laws of physics and other Aumann-mandated agreements, yet then go out and each optimize/reshape the world according to their own goals? Why would that make them not rational?

It would be irrational for Clippy to sing up to an agreement with Beady according to which Beady gets to turn Clippy and all his clips into beads. It is irrational for agents to sign up to anyhting which is not in their interests, and it is not in their interests to have no contract at all. So rational agents, even if they do not converge on all their goals, will negotiate contracts that minimise their disutility Clippy and Beady might take half the universe each.

Lastly, whatever Kant's justification, why can you not optimize for a different principle - peak happiness versus average happiness, what makes any particular justifying principle correct across all - rational - agents.

If you think RAs can converge on an ultimately correct theory of physics (which we don't have), what is to stop them converging on the correct theory of morality, which we also don't have?

Comment author: MindTheLeap 21 April 2013 03:35:00AM 0 points [-]

First of all, thanks for the comment. You have really motivated me to read and think about this more -- starting with getting clearer on the meanings of "objective", "subjective", and "intrinsic". I apologise for any confusion caused by my incorrect use of terminology. I guess that is why Eliezer likes to taboo words. I hope you don't mind me persisting in trying to explain my view and using those "taboo" words.

Since I was talking about meta-ethical moral relativism, I hope that it was sufficiently clear that I was referring to moral values. What I meant by "objective values" was "objectively true moral values" or "objectively true intrinsic values".

The second sentence doesn't follow from the first.

The second sentence was an explanation of the first: not logically derived from the first sentence, but a part of the argument. I'll try to construct my arguments more linearly in future.

If I had to rephrase that passage I'd say:

If there are no agents to value something, intrinsically or extrinsically, then there is also nothing to act on those values. In the absence of agents to act, values are effectively meaningless. Therefore, I'm not convinced that there is objective truth in intrinsic or moral values.

However, the lack of meaningful values in the absence of agents hints at agents themselves being valuable. If value can only have meaning in the presence of an agent, then that agent probably has, at the very least, extrinsic/instrumental value. Even a paperclip maximiser would probably consider itself to have instrumental value, right?

If rational agents converge on their values, that is objective enough.

I think there is a difference between it being objectively true that, in certain circumstances, the values of rational agents converge, and it being objectively true that those values are moral. A rational agent can do really "bad" things if the beliefs and intrinsic values on which it is acting are "bad". Why else would anyone be scared of AI?

Analogy: one can accept that mathematical truth is objective (mathematicians will converge) without being a Platonists (mathematical truths have an existence separate from humans)

I accept the possibility of objective truth values. I'm not convinced that it is objectively true that the convergence of subjectively true moral values indicates objectively true moral values. As far as values go, moral values don't seem to be as amenable to rigorous proofs as formal mathematical theorems. We could say that intrinsic values seem to be analogous to mathematical axioms.

I fin d that hard to follow. If the test i rationally justifiable, and leads to the uniform results, how is that not objective?

I'll have a go at clarifying that passage with the right(?) terminology:

Without the objective truth of intrinsic values, it might just be a matter of testing different sets of assumed intrinsic values until we find an "optimal" or acceptable convergent outcome.

Morality might be somewhat like an NP-hard optimisation problem. It might be objectively true that we get a certain result from a test. It's more difficult to say that it is objectively true that we have solved a complex optimisation problem.

You seem to be using "objective" (having a truth value independent of individual humans) to mean what I would mean by "real" (having existence independent of humans).

Thanks for informing me that my use of the term "objective" was confused/confusing. I'll keep trying to improve the clarity of my communication and understanding of the terminology.

Comment author: PrawnOfFate 23 April 2013 11:06:38AM 0 points [-]

First of all, thanks for the comment. You have really motivated me to read and think about this more

That's what I like to hear!

If there are no agents to value something, intrinsically or extrinsically, then there is also nothing to act on those values. In the absence of agents to act, values are effectively meaningless. Therefore, I'm not convinced that there is objective truth in intrinsic or moral values.

But there is no need for morality in the absence of agents. When agents are there, values will be there, when agents are not there, the absence of values doesn't matter.

I think there is a difference between it being objectively true that, in certain circumstances, the values of rational agents converge, and it being objectively true that those values are moral. A rational agent can do really "bad" things if the beliefs and intrinsic values on which it is acting are "bad". Why else would anyone be scared of AI?

I don't require their values to converge, I require them to accept the truths of certain claims. This happens in real life. People say "I don't like X, but I respect your right to do it". The first part says X is a disvalue, the second is an override coming from rationality.

Comment author: Kawoomba 23 April 2013 10:38:06AM *  0 points [-]

I'm not disputing that there are goals/ethics which may be best suited to take humanity along a certain trajectory, towards a previously defined goal (space exploration!). Given a different predefined goal, the optimal path there would often be different. Say, ruthless exploitation may have certain advantages in empire building, under certain circumstances.

The Categorical Imperative in all its variants may be a decent system for humans (not that anyone really uses it).

But is the justification for its global applicability that "if everyone lived by that rule, average happiness would be maximized"? That (or any other such consideration) itself is not a mandatory goal, but a chosen one. Choosing different criteria to maximize (e.g. noone less happy than x) would yield different rules, e.g. different from the Categorical Imperative. If you find yourself to be the worshipped god-king in some ancient Mesopotanian culture, there may be many more effective ways to make yourself happy, other than the Categorical Imperative. How can it still be said to be "correct"/optimal for the king, then?

So I'm not saying there aren't useful ethical system (as judged in relation to some predefined course), but that because those various ultimate goals of various rational agents (happiness, paperclips, replicating yourself all over the universe) and associated optimal ethics vary, there cannot be one system that optimizes for all conceivable goals.

My argument against moral realism and assorted is that if you had an axiomatic system from which it followed that strawberry is the best flavor of ice cream, but other agents which are just as intelligent with just as much optimizing power could use different axiomatic systems leading to different conclusions, how could one such system possibly be taken to be globally correct and compelling-to-adopt across agents with different goals?

Gandhi wouldn't take a pill which may transform him into a murderer. Clippy would not willingly modify itself such that suddenly it had different goals. Once you've taken a rational agent apart and know its goals and, as a component, its ethical subroutines, there is no further "core spark" which really yearns to adopt the Categorical Imperative. Clippy may choose to use it, for a time, if it serves its ultimate goals. But any given ethical code will never be optimal for arbitrary goals, in perpetuity (proof by example). When then would a particular code following from particular axioms be adopted by all rational agents?

Comment author: PrawnOfFate 23 April 2013 10:50:13AM *  -1 points [-]

But is the justification for its global applicability that "if everyone lived by that rule, average happiness would be maximized"?

Well, not, that's not Kant's justification!

That (or any other such consideration) itself is not a mandatory goal, but a chosen one.

Why would a rational agent choose unhappiness?

If you find yourself to be the worshipped god-king in some ancient Mesopotanian culture, there may be many more effective ways to make yourself happy, other than the Categorical Imperative.

Yes, but that wouldn't count as ethics. You wouldn't want a Universal Law that one guy gets the harem, and everyone else is a slave, because you wouldn't want to be a slave, and you probably would be. This is brought out in Rawls' version of Kantian ethics: you pretend to yourself that you are behind a veil that prevents you knowing what role in society you are going to have, and choose rules that you would want to have if you were to enter society at random.

My argument against moral realism and assorted is that if you had an axiomatic system from which it followed that strawberry is the best flavor of ice cream, but other agents which are just as intelligent with just as much optimizing power could use different axiomatic systems leading to different conclusions,

You don't have object-level stuff like ice cream or paperclips in your axioms (maxims), you have abstract stuff, like the Categorical Imperative. You then arrive at object level ethics by plugging in details of actual circumstances and values. These will vary, but not in an arbitrary way, as is the disadvantage of anything-goes relativism.

how could one such system possibly be taken to be globally correct and compelling-to-adopt across agents with different goals?

The idea is that things like the CI have rational appeal.

Once you've taken a rational agent apart and know its goals and, as a component, its ethical subroutines, there is no further "core spark" which really yearns to adopt the Categorical Imperative.

Rational agents will converge on a number of things because they are rational. None of them will think 2+2-=5.

Comment author: Kawoomba 23 April 2013 09:53:36AM *  0 points [-]

Yea, honestly I've never seen the exact distinction between goals which have an ethics-rating, and goals which do not. I understand that humans share many ethical intuitions, which isn't surprising given our similar hardware. Also, that it may be possible to define some axioms for "medieval Han Chinese ethics" (or some subset thereof), and then say we have an objectively correct model of their specific ethical code. About the shared intuitions amongst most humans, those could be e.g. "murdering your parents is wrong" (not even "murder is wrong", since that varies across cultures and circumstances). I'd still call those systems different, just as different cars can have the same type of engine.

Also, I understand that different alien cultures, using different "ethical axioms", or whatever they base their goals on, do not invalidate the medieval Han Chinese axioms, they merely use different ones.

My problem with "objectively correct ethics for all rational agents" is, you could say, where the compellingness of any particular system comes in. There is reason to believe an agent such as Clippy could not exist (edit: i.e., it probably could exist), and its very existence would contradict some "'rational' corresponds to a fixed set of ethics" rule. If someone would say "well, Clippy isn't really rational then", that would just be torturously warping the definition of "rational actor" to "must also believe in some specific set of ethical rules".

If I remember correctly, you say at least for humans there is a common ethical basis which we should adopt (correct me otherwise). I guess I see more variance and differences where you see common elements, especially going in the future. Should some bionically enhanced human, or an upload on a spacestation which doesn't even have parents, still share all the same rules for "good" and "bad" as an Amazon tribe living in an enclosed reservation? "Human civilization" is more of a loose umbrella term, and while there certainly can be general principles which some still share, I doubt there's that much in common in the ethical codex of an African child soldier and Donald Trump.

Comment author: PrawnOfFate 23 April 2013 10:21:56AM *  -1 points [-]

Yea, honestly I've never seen the exact distinction between goals which have an ethics-rating, and goals which do not

A number of criteria have been put forward. For instance, do as you would be done by. If you don't want to be murdered, murder is not an ethical goal.

My problem with "objectively correct ethics for all rational agents" is, you could say, where the compellingness of any particular system comes in. There is reason to believe an agent such as Clippy could exist, and its very existence would contradict some "'rational' corresponds to a fixed set of ethics" rule. If someone would say "well, Clippy isn't really rational then", that would just be torturously warping the definition of "rational actor" to "must also believe in some specific set of ethical rules".

The argument is not that rational agents (for some vaue of "rational") must believe in some rules, it is rather that they must not adopt arbitrary goals. Also, the argument only requires a statistical majority of rational agents to converge, because of the P<1.0 thing.

Should some bionically enhanced human, or an upload on a spacestation which doesn't even have parents, still share all the same rules for "good" and "bad" as an Amazon tribe living in an enclosed reservation?

Maybe not. The important thing is that variations in ethics should not be arbitrary--they should be systematically related to variations in circumstances.

Comment author: Viliam_Bur 22 April 2013 08:21:25PM 7 points [-]

In real life you sometimes get people who write, using different words, under dozen different articles: "I suspect that this all is just Eliezer's cult designed to extract money from naive people". How much of that is acceptable criticism, and how much is just annoying? Discussing that thing once, thoroughly? Yes, definitely. Dropping the idea around all the time? That's just poluting the space. Problem is that at the moment some people are already deeply annoyed, other people go meta and say we need criticism.

Democracy does not work well online. In real life, one person cannot be at more than one place. Online, one person is enough to be everywhere (within one website). In real life, you can avoid an annoying person by simply going elsewhere and taking your friends with you. Online, you must somehow stop people from doing annoying things, otherwise there is no way to avoid them, except by avoiding the whole website.

I don't have a problem with criticism. I have a problem with boring repetitive criticism. Someone says that having a ceremony is cultish. Okay. Let's discuss that. Someone says again that having a ceremony is cultish. Okay, here is some explanation, here are the differences. Someone says yet again that having a ceremony is cultish. Okay, I heard that already; give me a new information or stop repeating yourself. -- I would have no problem if someone wrote a critical article explaining the dangers of having a ceremony even in its LessWrongian variant, and proposed alternative ways to create personal connections. But dropping hostile comments to other peoples' articles is so much easier. Well, I am not impressed.

People who try hard to appear smart typically have a problem cooperating on anything. For a textbook example, visit Mensa. It is a miracle that Mensa ever gets anything done, because every time anyone proposes an idea, all people loudly disagree, to signal that they are not sheep. Okay, I get it, they are not sheep; but it is still funny how an organization consisting purely of highly intelligent people became such a laughing stock for the rest of the world. Probably they were too busy signalling that they are not sheep, so they missed the forest for the trees.

There is a time to disagree, and there is also a time to agree. If someone has a policy of e.g. never singing a song together with other people (because that might irrationally modify their feelings towards them), I accept if they decide to never sing a song together with fellow rationalists. Yes, they are consistent. I respect that. But if someone is willing to sing a song with random strangers, but would never sing a song with rationalists, that would be reversing stupidity. It means sabotaging yourself and your goals; trying to get some nonexistent points for doing things the hard way.

The proper moment for criticism is when something worth criticising happens. Not when someone merely pattern-matches something to something, and cannot stop obsessing about that. Here is a group of people who all voluntarily decided to share some powerful emotional moments together. Did they commit suicide later? No! Did they donate all their money to Eliezer? No! Did they send disconnection letters to their relatives? No! Did they refuse to talk with their friends who didn't participate in the ritual? No! Did they kill someone or send death threats? No! Are they reduced to mindless zombies? No! Are they keeping the details secret from the rest of the world, threatening to punish whistleblowers? No! -- So why the hell does someone keep repeating that it essentially is the same thing; and why should I pay any attention at all to that kind of criticism?

Comment author: PrawnOfFate 22 April 2013 08:29:07PM *  -1 points [-]

In real life you sometimes get people who write, using different words, under dozen different articles: "I suspect that this all is just Eliezer's cult designed to extract money from naive people". How much of that is acceptable criticism, and how much is just annoying? Discussing that thing once, thoroughly? Yes, definitely. Dropping the idea around all the time? That's just poluting the space. Problem is that at the moment some people are already deeply annoyed, other people go meta and say we need criticism.

Then write the article. (And why are some people annoyed? Can they refute the criticism or not? I find it odd how everyone is running on emotion..)

Democracy does not work well online.

Is the karma system working well?

I have a problem with boring repetitive criticism.

Then write a refutation, post it, and point people to it. (I could add it is easy to get annoyed by claims you can't refute...)

The proper moment for criticism is when something worth criticising happens.

What about criticising theories? Here's a question: has EY ever withdrawn a posting?

No! Did they donate all their money to Eliezer? No! Did they send disconnection letters to their relatives? No! Did they refuse to talk with their friends who didn't participate in the ritual? No! Did they kill someone or send death threats? No! Are they reduced to mindless zombies?

Did they do something known to stimulate the parts of the brain people are irrational with?

So why the hell does someone keep repeating that it essentially is the same thing; and why should I pay any attention at all to that kind of criticism?

They are probably repeating it because it has never been refuted. But I was not exactly saying that. I don't think LW is a full blown cult. I think it is a rather imperfect promoter of rationalism. But nuance gets lost when people get emotional.

Comment author: RichardKennaway 22 April 2013 08:18:14PM 0 points [-]

Tying the moderator reaction to whether or not the criticism is fun to receive, rather than if it is useful to receive, is a recipe for receiving fun but useless criticisms and not receiving unfun but useful criticisms.

Useless criticisms are no fun at all.

Comment author: PrawnOfFate 22 April 2013 08:20:04PM -1 points [-]

What happened to the Search for Truth?

Comment author: Vaniver 22 April 2013 04:24:53PM 2 points [-]

If you have a sufficiently persistent person and inflexible moderation policy, one person really is enough to destroy a website.

I agree that destructive people can do a lot of damage, and that removing them is a good idea. I also agree that destructiveness doesn't even require maliciousness.

The strategy I'd like to see is "cultivate dissent." If someone is being critical in an unproductive way, then show them the productive way to be critical, and if they fail to shape up, then remove them from the community, through a ban or deletion/hiding of comments. Documenting the steps along the way, and linking to previous warnings, makes it clear to observers that dissent is carefully managed, not suppressed.

Tying the moderator reaction to whether or not the criticism is fun to receive, rather than if it is useful to receive, is a recipe for receiving fun but useless criticisms and not receiving unfun but useful criticisms.

Receiving and processing unfun but useful criticisms is a core part of rationality, to the point that there are litanies about it.

Comment author: PrawnOfFate 22 April 2013 04:34:19PM 3 points [-]

The strategy I'd like to see is "cultivate dissent." If someone is being critical in an unproductive way, then show them the productive way to be critical, and if they fail to shape up, then remove them from the community, through a ban or deletion/hiding of comments. Documenting the steps along the way, and linking to previous warnings, makes it clear to observers that dissent is carefully managed, not suppressed.

An excellent suggestion. Not only does it address trolling problems, the encouragement of good quality criticism is something rationalists should be doing anyway.

Comment author: [deleted] 22 April 2013 01:22:22PM 1 point [-]
In response to comment by [deleted] on Ritual Report: Schelling Day
Comment author: PrawnOfFate 22 April 2013 01:29:19PM 0 points [-]

I haven't seen anything to say that is for meta discussion, it mostly isn't de facto, and I haven't seen a "take it elsewhere" notice anywhere as an aternative to downvote and delete.

View more: Next