Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Guardians of the Gene Pool

26 Post author: Eliezer_Yudkowsky 16 December 2007 08:08PM

Followup toGuardians of the Truth

Like any educated denizen of the 21st century, you may have heard of World War II.  You may remember that Hitler and the Nazis planned to carry forward a romanticized process of evolution, to breed a new master race, supermen, stronger and smarter than anything that had existed before.

Actually this is a common misconception.  Hitler believed that the Aryan superman had previously existed—the Nordic stereotype, the blond blue-eyed beast of prey—but had been polluted by mingling with impure races.  There had been a racial Fall from Grace.

It says something about the degree to which the concept of progress permeates Western civilization, that the one is told about Nazi eugenics and hears "They tried to breed a superhuman."  You, dear reader—if you failed hard enough to endorse coercive eugenics, you would try to create a superhuman.  Because you locate your ideals in your future, not in your past.  Because you are creative.  The thought of breeding back to some Nordic archetype from a thousand years earlier would not even occur to you as a possibility—what, just the Vikings?  That's all?  If you failed hard enough to kill, you would damn well try to reach heights never before reached, or what a waste it would all be, eh?  Well, that's one reason you're not a Nazi, dear reader.

It says something about how difficult it is for the relatively healthy to envision themselves in the shoes of the relatively sick, that we are told of the Nazis, and distort the tale to make them defective transhumanists.

It's the Communists who were the defective transhumanists.  "New Soviet Man" and all that.  The Nazis were quite definitely the bioconservatives of the tale.

 

Part of the Death Spirals and the Cult Attractor subsequence of How To Actually Change Your Mind

Next post: "Guardians of Ayn Rand"

Previous post: "Guardians of the Truth"

Comments (73)

Sort By: Old
Comment author: burger_flipper2 16 December 2007 09:10:26PM 4 points [-]

Relatively new to the forum and just watched the 2 1/2 hour Yudkowsky video on Google. Excellent talk that really helped frame some of the posts here for me, though the audience questions were generally a distraction.

My biggest disappointment was the one question that popped up in my mind while watching and was actually posed wasn't answered because it would take about 5 minutes. The man who asked was told to pose it again at the end of the talk, but did not.

This was the question about the friendly AI: "Why are you assuming it knows the outcome of its modifications?"

Any pointer to the answer would be much appreciated.

Comment author: James_D._Miller 16 December 2007 09:53:36PM 0 points [-]

The Soviet new "man" that Stalin wanted to create was a half-ape, half-man super-warrior.

See http://news.scotsman.com/ViewArticle.aspx?articleid=2688011

Comment author: James_Miller 26 November 2011 07:59:49AM 8 points [-]

I no longer trust the validity of this article.

Comment author: [deleted] 26 November 2011 08:20:26AM 11 points [-]

Not a True Scotsman, is it?

Comment author: TGGP4 16 December 2007 10:53:07PM 0 points [-]

This entry reminded me of Blank Slate Asymmetry from Gene Expression. A lot of people would say the difference in our perceptions/opinions result from our general attitude toward progress, but I would suggest that it was contingent on our opposition in war to the Nazis while many of our elites were rather friendly towards the Soviets.

Comment author: JulianMorrison 17 December 2007 02:24:02AM 3 points [-]

The Soviets weren't what I'd call transhumanists, because their New Man wasn't a definable goal or factual trend, he was a utopian catch-all of projected virtue. A transhumanist will be able to break down his goals ("uploading") into subgoals ("AI and brain scans") and roughly sketch a research path ("symbolic AI") that would either approach the goal, or fail in an informative way ("combinatorial explosion"). The Soviets could do no such thing, because NSM was nothing definable. He would certainly pop up as a consequence of the experience of enough Marxism. Timescale couldn't be defined. Success could not be predicted until it was encountered. Keep plugging on and have faith.

I call that religion. It isn't set in the *real* future. It's set in the same never-never land that contains the Second Coming. Importantly, it doesn't lead to a future-oriented culture, which is more precisely a realist goal oriented culture. Nobody works towards NSM (except in propaganda posters) and so he never gets any closer.

Comment author: Richard_Kulisz 17 December 2007 03:23:17AM -1 points [-]

It has nothing to do with poverty of imagination and everything to do with black propaganda. The Soviets were simply never evil enough. And we know that looking forward into the future is evil, therefore the Nazis must have been guilty of that crime. If the Soviets had done it, why it may even have rehabilitated that concept. Can't have that, can we?

The problem isn't that Westerners can't imagine themselves in the shoes of the Romantic Nazis. All to the contrary, the problem is that elite conservative Westerners find it ALL TOO EASY to imagine themselves in the shoes of the Romantic Nazis. So much so that they had to safeguard a part of Nazi ideology, to cherish it and safeguard it, by separating it from the Nazis themselves.

The Romantics turned the Nazis into Transhumanists because it fit their own agenda.

Comment author: TGGP4 17 December 2007 04:07:23AM 1 point [-]

It is often forgotten in the early days of proto-Nazi racial theory the Prussians were said to be the Master Race because they were a combination of German and Slav! Their combination was supposed to be just right from the perspective of Prussians, reminding me of Charles Murray's "Who wants to be an elephant?". Nietzsche also proposed breeding ubermenschen by giving Prussian officers jewish brides (haven't read him myself, just heard he said this in BG&E).

Comment author: FeepingCreature 09 June 2012 09:36:09PM *  0 points [-]

Beyond Good and Evil, Aphorism 251. Nietzsche is not entirely serious there.

For the record, Nietzsche's concept of the Overman is primarily a spiritual genesis of a post-human creature that would have the strength to freely chose their own values in pursuit of a higher goal (If I understand it correctly). It has little to do with eugenics.

Comment author: Joseph_Hertzlinger 17 December 2007 05:29:30AM 2 points [-]

The example of Communism shows that being future-oriented will not always eliminate the "Guardians of Truth" syndrome. Sometimes it will produce people who guard a specific view of the future.

Comment author: HughRistik2 17 December 2007 09:46:49AM 0 points [-]

It says something about the degree to which the concept of progress permeates Western civilization, that the one is told about Nazi eugenics and hears "They tried to breed a superhuman."

What interests me is the frequent opposition to transhumanism because of transhumanism's supposedly mistaken notion of progress. Just because progress might not be smooth, it doesn't mean that we haven't experienced it in various dimensions. Skeptics about progress seem to have a romanticized view of the past, going along with a quasi-religious notion of a fall from grace (due to technology, "patriarchy," techno-patriarchy, or whatever).

I don't mind if other people want to go back to the days before we had fire, or say, the germ theory of medicine—as long as they don't try to take me with them.

Comment author: stcredzero 03 June 2012 06:28:28PM 2 points [-]

I don't mind if other people want to go back to the days before we had fire, or say, the germ theory of medicine as long as they don't try to take me with them.

You should be more specific. If such a group decides to kill off all the technologists first, in a misguided attempt to make their non technological future safe for them, it could be said they didn't try to take you with them.

Comment author: Ben_Jones 17 December 2007 09:49:08AM 2 points [-]

"Sometimes it will produce people who guard a specific view of the future."

Anyone read Joseph's post (just above) and immediately think 'Singularitarians!'?

Certainly the majority of (though not all) people I've met who assign that word to themselves are closer to being Guardians than Seekers. Fairly natural in that all causes want to be cults, but still likely to be harmful to the cause.

Comment author: Richard_Hollerith2 17 December 2007 01:35:44PM 1 point [-]

I have not noticed that, Ben.

Not all of us who believe physics-since-1600 and biology-since-1860 have seen unequivocal progress believe there has been unequivocal progress in popular political or moral opinion. The civil-rights movement of the 1950s and 1960s for example clearly represents an increase in the consistency of the application of the ideal of equality, but it constitutes unequivocal progress only if you believe that the spread of the ideal of equality constitutes unequivocal progress.

Comment author: Caledonian2 17 December 2007 01:47:18PM 5 points [-]

'Progress' is the accumulation of changes towards a pre-defined goal.

Just a few lifespans ago, 'progress' consisted of spreading settlers into sparsely-populated land, killing or driving away the aborigines, draining the wetlands, slaughtering the predators, and converting the ecology into farmland. It was using antibiotics widely and prophylactically, replacing ancient crop strains with monocultures, and designing our communities around the automobile.

'Progress' is the hobgoblin trotted out by everyone who thinks they know what the future should be. Anyone foolish enough to name their political goals 'progressive' ought to be excluded from the political process.

Comment author: Eliezer_Yudkowsky 17 December 2007 02:53:15PM 0 points [-]

"Sometimes it will produce people who guard a specific view of the future."

Anyone read Joseph's post (just above) and immediately think 'Singularitarians!'?

Nope and I challenge you to name two examples.

Comment author: steven 17 December 2007 03:06:37PM 0 points [-]

"Anyone read Joseph's post (just above) and immediately think 'Singularitarians!'?"

Please report to the nearest termination center.

Comment author: Joseph_Hertzlinger 17 December 2007 03:32:12PM 0 points [-]

There are, of course, many different future visions that could be guarded.

Comment author: Ben_Jones 17 December 2007 03:41:53PM 2 points [-]

A Truth-Guardian is someone who 'guards' an Idea by zapping (in its myriad forms) rather than through rational argument.

Are you willing to tell me that you've never met a Singularitarian who has attacked an opponent's authority (zap), or denigrated another's work (zap), or sought to work on their Idea's strong points to the neglect of its weak points (subtle zap), or acted in an elitist manner in order to confer perceived authority on themself (smug zap), or presented new data in such a way as to strengthen their previous predictions (super Bottom Line zap!)? Have you, Eliezer, never ever *guarded* your view of the future rather than argued dispassionately, even against a plainly *wrong* argument?

If you say no, I wholly withdraw my (well-meant) comment. Caledonian can be my second example. :p

The moment anyone makes a biased argument because of their attachment to an Idea, they become a Guardian. Singularitarians are people, and they take criticism, and defend their beliefs, requisitely passionately. Apologies if it seemed as though I was singling anybody out for specific criticism of bias - not my intention. For the record, I'm a firm believer. :)

Comment author: stcredzero 03 June 2012 06:38:24PM 0 points [-]

A Truth-Guardian is someone who 'guards' an Idea by zapping (in its myriad forms) rather than through rational argument.

How does this idea relate to "Well tended gardens die by pacifism?"

Comment author: TheOtherDave 03 June 2012 07:40:34PM *  3 points [-]

Is that an actual question, or an oblique way of suggesting that the thesis of "Well tended gardens die by pacifism" is promoting a form of Truth-Guardianism, and therefore contradicts the thesis of "Guardians of the Truth", and therefore perhaps both theses are flawed?

If it's the latter: yes, yes, very clever.

Assuming charitably that it's the former, my two cents about how they relate:

  • WTGDBF predicts that where local community norms N1 differ from global norms N2 there's a tendency for N2 to displace N1 whenever the local community interacts with the larger world, and suggests that if I consider N1 superior to N2 I have a moral responsibility to counteract this tendency, which sometimes requires violating N2.

  • GOTT suggests that certain norms which involve punishing attempts to challenge or question certain ideas regardless of how novel, well-formed, or carefully reasoned those challenges/questions are, are bad for communities that embrace them, despite being well-protected from outside norms.

  • Combining the two suggests that when I choose to defend my local community norms against corruption by outside norms, I also have a moral responsibility to be right about the superiority of my community's norms.

Comment author: stcredzero 03 June 2012 08:18:49PM 0 points [-]

Is that an actual question

Yes.

or an oblique way of suggesting that the thesis of "Well tended gardens die by pacifism" is promoting a form of Truth-Guardianism, and therefore contradicts the thesis of "Guardians of the Truth", and therefore perhaps both theses are flawed?

Apparent contradictions are often interesting areas of inquiry. Since I had to join my girlfriend for Phở, I only had time to post the one sentence.

It doesn't mean that both theses are flawed. They could be opposing forces. The apparent contradiction might indicate a point where optimization is challenging. This could explain why groups seem doomed to fall to one pathology or another. There's probably positive feedback in either direction, making groups dynamically unstable on this "axis" -- whatever that might be. Maybe this is to be explained by our group cohesion mechanisms being designed to help us survive when the next tribe over decides to attack, and why things too easily devolve into might makes right?

Combining the two suggests that when I choose to defend my local community norms against corruption by outside norms, I also have a moral responsibility to be right about the superiority of my community's norms.

The last phrase makes me cautious. I think one has a moral responsibility to respect the truth by seeking the truth. If we look at ideologies, how well do they deal with the notion of superiority? How many past notions of superiority seem barbaric? Is there a way of transcending or sidestepping this notion of superiority altogether?

Comment author: TheOtherDave 03 June 2012 10:05:44PM 1 point [-]

Agreed that apparent contradictions are often interesting areas of inquiry.

The only ways I know of to sidestep having to decide which norms best align with my values is to adopt values such that either no community's norms are superior to any others', or such that whatever norms happen to emerge victorious from the interaction of social groups are superior to all the norms they displace. Neither of those tempt me at all, though I know people who endorse both.

If I reject both of those options, I'm left with the possibility that two communities C1 and C2 might exist such that C1's norms are superior to C2's, but the interaction of C1 and C2 results in C1's norms being displaced by C2's.

I don't see a fourth option. Do you?

For example... you say I have a moral responsibility to seek truth, which suggests that if I'm in a community whose values oppose truthseeking in certain areas, I have a moral responsibility to violate my community's norms. No?

Comment author: stcredzero 03 June 2012 10:33:50PM *  0 points [-]

This has interesting parallels to the Friendly AI problem. For example, one could posit that material wealth might somehow be a suitable arbiter, but I can imagine plenty of situations where C2 displaces C1 (Corporate lobbying?) followed by global ecological catastrophes. Here, dollars take the place of smiley faces strewn across the solar system. Maybe the problem of a sustainably benevolent truth-seeking group is somehow the same problem as FAI on some level?

Comment author: TheOtherDave 04 June 2012 12:01:21AM 0 points [-]

Solving Friendliness involves capturing desirable ethical guidelines in a robust and sustainable way, so I'd expect the relationship between Friendliness and sustainably benevolent truth-seeking to depend a lot on the relationship between ethics and truth-seeking. I'd agree that they are thematically related, but very much non-identical.

Comment author: pnrjulius 09 June 2012 02:32:36AM 0 points [-]

Yes! The problem of Friendly Corporate Behavior is an urgent and unsolved one. (Indeed, corporations have many of the attributes of artificial intelligences, though of course not all.)

The sustainably benevolent moral group is not Friendly AI; it is Friendly NI (natural intelligence). The two problems are probably closely related, but I can see a few important differences: NIs had to evolve, so they're going to start out optimized for reproduction. AIs are designed, so they're optimized for whatever you optimize them for.

Comment author: stcredzero 09 June 2012 03:07:38PM 0 points [-]

AIs are designed, so they're optimized for whatever you optimize them for.

My prediction: The ones optimized for reproduction are the ones that will be around in the long term.

Comment author: pnrjulius 11 June 2012 01:33:51AM 2 points [-]

Not necessarily, because there's no law saying that AIs have to die. This changes the evolutionary calculus significantly; you don't need to reproduce if you can just keep existing and expand your power over the cosmos.

But you're right, insofar as AIs that rapidly self-destruct and never reproduce are not going to stick around long. (I think this is actually a tautology, but it's a tautology with the character of a mathematical theorem---definitely true, but not obvious or trivial.)

It's also worth considering that there are different constraints between NIs and AIs though. NIs have to change gradually, piece by piece, gene by gene. AIs can be radically overhauled in a single generation. This gives them access to places on the fitness landscape that we could never reach---even places that are in fact evolutionarily stable once you get there.

Comment author: wedrifid 11 June 2012 01:46:12AM 0 points [-]

you don't need to reproduce if you can just keep existing and expand your power over the cosmos.

Apart from the practical lightspeed limitations. You do need to reproduce or in some other way split yourself into space-separated parts if you wish to expand your power over a sufficient distance.

Comment author: stcredzero 13 June 2012 04:47:02AM 0 points [-]

Not necessarily, because there's no law saying that AIs have to die. This changes the evolutionary calculus significantly; you don't need to reproduce if you can just keep existing and expand your power over the cosmos.

As wedrifid pointed out, that depends on what one can do about the lightspeed limit. And thermodynamics. I don't think not dying of old age changes evolution that much. Humans are prone to geriatric diseases because evolution can't do much for us past the reproductive years. Beings without a lifespan won't face that.

I highly doubt that no AI won't ever destroy another, though.

It's also worth considering that there are different constraints between NIs and AIs though. NIs have to change gradually, piece by piece, gene by gene. AIs can be radically overhauled in a single generation. This gives them access to places on the fitness landscape that we could never reach---even places that are in fact evolutionarily stable once you get there.

That just means that they'll evolve without the constraints of genetics, much as designs and memes do.

I think it's a mistake to treat superhuman AI as magic. In some contexts it will seem magic, but not all. Human habitations viewed from 10,000 meters look like growths of lichen. In some contexts, some dogs are "smarter" than some people. Human intelligence gives us a tremendous advantage over all other life on Earth, but it is not magic. Superhuman intelligence is not magic. It's just intelligence.

Comment author: Viliam_Bur 04 June 2012 11:48:59AM *  1 point [-]

There is some difference between group ideas and group norms, although sometimes these two overlap. There is also a difference between challenging group ideas, and breaking group norms.

An example of a group idea: "It is reasonable to give million dollars to an organization that will freeze your head when you die, because someone might scan your brain and make a machine simulation of you, and it will be really you."

An example of a group norm: "We should refrain from political examples, personal attacks, irrational arguments, etc."

An example of challenging a group idea: "I think the machine simulation is not really you. Even if it is 'alive', it is a new life form; and your old self is dead."

An example of breaking group norms: "This is so stupid!!! I guess you have also voted for [political party]!"

Sometimes these two things can be confused. For example it can be a group norm to never challenge group ideas (or to limit challenging them to ways that have no chance to succeed). This should not happen. On the other hand, it is also very frequent to obviously break group norms and then complain about group's intolerance to challenging its ideas -- this is a typical pattern for many internet trolls, and the community should be able to recognize it.

An example: "Cryonics does not work, f*** you!" "Downvoted for swearing." "You just downvote me because I disagree with you, f*** you!"

Comment author: TheOtherDave 04 June 2012 01:33:58PM 0 points [-]

Yes, agreed with all of this. Though as you suggest, the two can overlap. "Give million dollars to an organization that will freeze your head when you die" can become a group norm, and "refrain from political examples, personal attacks, irrational arguments, etc." can be a group idea. And as you say, it is common for one to be confused for the other, sometimes deliberately for rhetorical effect.

Comment author: pnrjulius 09 June 2012 02:34:49AM -1 points [-]

Also sometimes the group's norms are as problematic as its ideas; e.g. KKK, Nazis.

But usually the norms are not too bad, it's just the ideas that are ridiculous (moderate religion in a nutshell). So it definitely makes sense to make a distinction for practical purposes.

Comment author: Ian_C. 17 December 2007 04:12:46PM 2 points [-]

"The moment anyone makes a biased argument because of their attachment to an Idea, they become a Guardian."

I think it's more important what happens when the bias is discovered. Does the group in question reward it or try to eliminate it? For example there is corruption in democracies as well as less free forms of government, what makes the difference is what happens when it is discovered.

Comment author: steven 17 December 2007 04:50:13PM 1 point [-]

Ben, of course no one is 100% un-Guardian-like, but you seemed to be claiming Singularitarians were unusually Guardian-like.

Comment author: Nathan_Myers 17 December 2007 06:17:10PM 0 points [-]

Wouldn't that make them "bio-reactionaries" or "bio-romantics"? Or has the equation of "conservatism" (which once denoted an inclination to preserve the status quo) with "reactionism" (desire to re-instate the status quo ante), "romanticism" (promotion of some vanished, idealized past), or raw fascism (power is its own logic) pervaded even these hallowed halls? Do we have a name for what was once called conservatism, or does the concept no longer have any meaningful referent?

Comment author: pnrjulius 09 June 2012 02:37:08AM 0 points [-]

Part of the problem is that reactionaries call themselves "conservative" even though, you're right, they really aren't.

In the US, equal rights for women is really a conservative idea in the original sense, because it's something that our culture has already mostly accepted. People arguing against it aren't conserving the status quo, they are harkening back to some bygone halycon era.

But think of how weird it sounds to say that feminists are conservative! So I think the term in practice has moved away from its original etymological meaning.

Comment author: TGGP4 17 December 2007 07:55:07PM 0 points [-]

Nathan Myer's, how about "status quo bias"?

Comment author: Eliezer_Yudkowsky 17 December 2007 08:10:33PM 0 points [-]

Caledonian a Singularitarian? I doubt he knows what the word means. I don't recall him on the Singularity Institute donors list, or any of the mailing lists or websites. The term denotes activism, not belief - an "environmentalist" is not someone who believes in the existence of the environment.

Ben Jones, if the standard confirmation/disconfirmation bias is regarded as "Guardianship" then the guardian/discoverer distinction loses all meaning even with respect to scientists versus the Inquisition. The question is whether people exhibit their ordinary human biases to defend the status quo (or status quo ante), or to defend their new ideas and innovations. The latter case, though still ordinarily human-biased, is time-oriented toward the future.

Comment author: Chris 17 December 2007 08:22:10PM 0 points [-]

"an "environmentalist" is not someone who believes in the existence of the environment." Non sequitur. An environmentalist is someone who believes in the value of the environment. sloppy, sloppy, sloppy.......

Comment author: Eliezer_Yudkowsky 17 December 2007 09:07:38PM -2 points [-]

Really, Chris. So if I believe in the value of the environment, but believe that it's much less valuable than the use to be gained by paving it over with strip mines, then I'm an "environmentalist"?

In any case it's a moot point. Mark Plus coined the term "Singularitarian", but didn't do much with it; when I decided to build a Singularitarian movement, I asked Mark Plus for ownership of the word and was granted it; and I define the term to involve activism. If you mean something else by the word, feel free to call yourself a "Singularian" or something.

Comment author: danlowlite 25 October 2010 09:46:35PM 10 points [-]

Were/Are you joking? Seriously. I don't understand how one can own a word. Did I miss something?

I'm not disagreeing that it might involve activism (though I would define activism quite broadly), but how can one "own" a word?

Comment author: Dojan 18 October 2011 07:24:32PM 8 points [-]

Might I suggest open-sourcing the word?

Oh, and also, like, every other word, ever?

Comment author: DimitriK 12 November 2014 10:12:00PM 1 point [-]

I think Chris was talking about value in a relative sense (though ironicly was sloppy and left his statement too vague).

What's more surprising here is that you guys are arguing over a definition of environmentalim. Taboo it and you'd probably agree.

Most surprising of all is seeing you claim you own a word, Eliezer. I may have just started reading these sequences but I'm pretty sure there was a post or two on how you can't just define a word how you want.

Ironically enough you are guarding singularitarianism with your comment. And you're doing it by redefining the word to suit youside. And im pretty sure its a redefinement. The normative use for singularitarianist doesn't involve activism. Nor does environmentalist. You might value one singularitarianist or environmentalist more than another if they are an activist for thecause, but that's another matter.

Comment author: chris4 17 December 2007 09:38:03PM 0 points [-]

Yudkowsky, I was using the colloquial meaning of the word value, that is, positive value. If you insist, positive value of a healthy environment to promote the interests of, and as defined by, the entity that assesses the value. OK ? No prob for 'ownership' of the label, my issue was with the metaphor. BTW, as I respect the issues raised here, and the expertise of those who raise them, I'd love to see a post on the biases around the concept 'ownership'.

Comment author: Caledonian2 18 December 2007 01:24:16AM 2 points [-]

Caledonian a Singularitarian? I doubt he knows what the word means. I don't recall him on the Singularity Institute donors list, or any of the mailing lists or websites.

Ahem.

It's wonderful using words in arguments when you get to redefine them. Who's the source for the alternate definition, the one that replaced "one who believes the concept of a Singularity" and the one more complex than "activist for the Singularity"? Hmmm...

I also love that you equate "working toward bringing about the Singularity" with donating money to your Institute or being on a mailing list.

Seekers of truth do not attempt to hardwire goals and evaluations means into entities they create, whether deistic or merely offspring. Only Guardians value their beliefs so much that they attempt to transmit them as arbitrary, received 'wisdom'.

Comment author: Caledonian2 18 December 2007 01:27:46AM 2 points [-]

I missed this the first time through:

I asked Mark Plus for ownership of the word and was granted it

I... wow. I don't quite know how to respond to a person who makes a statement such as this.

Comment author: Nick_Tarleton 18 December 2007 01:48:53AM 1 point [-]

Seekers of truth do not attempt to hardwire goals and evaluations means into entities they create, whether deistic or merely offspring. Only Guardians value their beliefs so much that they attempt to transmit them as arbitrary, received 'wisdom'.

Values are not "beliefs", "true", or "false". (What about this is so hard to understand?)

Comment author: Peter_de_Blanc 18 December 2007 06:26:00AM 1 point [-]

Values (that is, goals of optimizers) are vastly meaningful; they affect the future shape of the universe.

Comment author: Ben_Jones 18 December 2007 10:12:55AM 1 point [-]

A fair point, Eliezer. I'd agree that if it weren't for dis/confirmation biases, nothing would ever get done. If Einstein, when questioned about what he would have done if his special theory was disproved, had said 'meh, I can take it or leave it,' he probably wouldn't have had the drive to discover it in the first place. Attachment to your Big Idea is often what drives us.

That said, I don't see that a Big Idea About The Future is so different from a Big Idea About The Past in terms of value for humanity. Both can be open or closed, pacifistic or violent, inclusive or exclusive. It's what you do with it that counts! The question of whether the Singularity as currently defined has positive utility for the human race is not a given, neither will it be unanimous.

I've tried and tried, but I can't think of any other Big Ideas that have stemmed from people looking at where science and technology are going, and extrapolating them to a future point. Perhaps someone who's less hungover can think of one. Office Christmas do last night, still coming around.

Caledonian - I'd say that one of the key concepts in my current understanding of the Singularity is that it's the polar opposite of a hard-wired goal. Surely the very idea is that we don't know what happens inside/beyond a singularity, hence the name?

Comment author: Ben_Jones 18 December 2007 10:27:03AM 0 points [-]

Retrospective apologies for the long post - will keep it brief in future!

Comment author: Caledonian2 18 December 2007 01:53:30PM -1 points [-]

Caledonian - I'd say that one of the key concepts in my current understanding of the Singularity is that it's the polar opposite of a hard-wired goal. Surely the very idea is that we don't know what happens inside/beyond a singularity, hence the name?

The whole point of attempting a "Friendly AI" is that its proponents believe that it IS possible to exclude entire branches of possibility from an AI's courses of action - that the superhuman intelligence can be made safe. Not merely friendly in a human sense, but favorable to human interests, not 'evil'.

Of course, they cannot provide an objective and rigorous description of what "being in human interests" actually entails, nor can they explain clearly what 'evil' is. But they know it when they see it, apparently. And since many of them seem to believe that 'values' are arbitrary, they've never bothered subjecting what they value to analysis.

Perhaps the possibility that a consequence of an entity being utterly good might be its being utterly unsafe has never occurred to them. And perhaps the possibility that superhuman general intelligence might analyze their values and find them lacking never occurred to them either. That would explain a lot.

Comment author: pnrjulius 09 June 2012 02:40:08AM -1 points [-]

Why would being good make you unsafe?

Comment author: Ben_Welchner 09 June 2012 03:16:02AM *  0 points [-]

Caledonian hasn't posted anything since 2009, if you said that in hopes of him responding.

Comment author: Matthew2 20 December 2007 09:05:00AM 0 points [-]

caledonian said: "Perhaps the possibility that a consequence of an entity being utterly good might be its being utterly unsafe has never occurred to them."

This describes monotheism rather well. It has occured to me.

Comment author: Ben_Jones 20 December 2007 12:34:05PM 1 point [-]

Caledonian,

Yes, it has occurred to 'them'. I hope you haven't read http://www.singinst.org/AIRisk.pdf, since if you have, you haven't grasped the challenge. The crux isn't excluding branches of possible action by an AI, it's ensuring those avenues aren't attractive options for any reason.

Comment author: Caledonian2 20 December 2007 01:08:45PM -1 points [-]

The crux isn't excluding branches of possible action by an AI, it's ensuring those avenues aren't attractive options for any reason.

(agog)

Would you care to explain what the distinction between those two states is?

Comment author: Ben_Jones 20 December 2007 02:37:26PM 0 points [-]

Sure - it's the difference between not stealing because you think you'll get caught and go to prison, and not stealing because you think theft is irrational/immoral/wrong/you name it. The first is sociopathy, the second is what we'd term normal human reasoning. Can I assume you believe there is no such thing as a Friendly AI?

Comment author: Caledonian2 20 December 2007 03:44:43PM 0 points [-]

When you're determining the value structure of a mind, "ensuring those avenues aren't attractive options for any reason" IS excluding them from the set of possible courses of action. The key phrase there is for any reason.

As for the rest of your argument, reasoning is precisely what the normal human does NOT do, and it's hilarious that you think logical arguments are what keeps most people from theft.

Comment author: Rick_Smith 24 December 2007 01:54:02PM 1 point [-]

Caledonian, shouldn't you check up on who currently owns the word 'reasoning' before stating that?

I guess there must be some sort of register somewhere...

Comment author: thatoliver 06 June 2012 10:24:38PM 2 points [-]

A minor semantic point: wouldn't advocating a return to the ancient Nordic race make them racial reactionaries rather than racial conservatives?

Taking the British National Party as an example of a racial conservative group, we see that they endeavour to PRESERVE the white race. They believe the master race (or, in this case, the race that somehow deserves ownership of the UK) is extant, and must be protected. On the other hand, the Nazi wished to RESTORE a racial standard that they believed had been long buried.

Comment author: Yosarian2 05 January 2013 01:29:34AM *  1 point [-]

This is somewhat true. (It gets even stranger when you find out that they were also trying to similar things with animals, trying to somehow breed dogs back to the first dog ancestor.) However, it's worth noting that Nazis directly tapped into the common "eugenics" mode of thought in our society, and eugenicists in general were trying to "breed better humans" (by doing things like encouraging the forced sterilization of the insane and the physically disabled, ect).

Of course, it's still a fundamental fail of an idea all the way around. Sure, you could do artificial selection on humans even without understanding genetics, the same way we did with dogs, but for that to work you'd have to have absolute and total control over the reproduction of entire large human sub-populations for dozens of generations, nothing short of that would work. And there's no way a tyrannical govenrment that absurd manages to stay in power for that long. On the scale of the kinds of stuff eugenicists were actually doing, it simply couldn't have a significant effect on the human gene pool in any plausible time-frame.

If you really wanted to try to breed better humans, and you weren't all powerful, probably your best best would be to try to ingrain a powerful and ubiquitous "sexy people are smart" meme into the culture and then keep it alive. If you were able to do that and maintain the meme for 1000 years or so, then sexual selection might start to have an effect to increase the average intelligence of the human race.

Comment author: [deleted] 05 January 2013 02:16:54AM 1 point [-]

try to ingrain a powerful and ubiquitous "sexy people are smart" meme into the culture

I've been trying to do that for years. It doesn't seem to be working, as I can't even get laid myself, let alone smart people in general. ;-)

Comment author: Yosarian2 05 January 2013 02:24:25AM 0 points [-]

Actually, looking at the last couple of decades, I would say that he "smart people are sexy" meme really started to take off around the time that a number of computers science geeks suddenly became some of the richest people in the world.

Comment author: Jan_Rzymkowski 09 May 2014 11:07:10PM -1 points [-]

"if you failed hard enough to endorse coercive eugenics"

This might be found a bit too controversial, but I was tempted to come up with not-so-revolting coercive eugenics system. Of course it's not needed, if there is technology for correcting genes, but let's say we only have circa 1900 technology. It has nothing to do with the point of Elizer's note, it's ust my musing.

Coervie eugenics isn't strictly immoral itself. It is a way of protecting people not yet born from genetical flaws - possible diseases, etc. But even giving them less then optimal features - intelligence, strength, looks - is quite equivalent to making them stupidier, weaker, uglier. If you could give your child healthy and pleasent life, yet decide to strip him from that, you are hurting him - it's not like his well-being is your property. But can you have YOUR child, while eugenics prevent you from breeding? Not in genetic sense, but it seems deeply flawed to base parent-child relation simply on genetic code. It's upbringing that matters. Adopted child is in any meaningful way YOUR child. But there are two problems - you can't really use "good genes" people for producing babies for "bad gene" people and "bad gene" mothers may have problem caring newborns without hormonal effect of birth. Way to make eugenics weaker, but overcome these problems, is to limit only mens' breading. When a couple with "good gene man" wants children - let them. If couple with "bad gene man" wants children, then future mother is impregnated by some (possible hired) "good gene man". Normally the couple have protected sex.

It is by no means perfect. But the price for relative well-being of future people is only for a woman to have sex with not her husband, and for husband to be "cheated on". While it seems quite unsettling, it's mainly our cultural norm. While this might be unpleasant for both, it isn't considerably worse then not being able to drink and smoke for woman through pregnancy. Therefore, such coercive eugenics would gradually improve gene pool, while not being considerable more evil then forbidding pregnant woman to smoke cigarettes.

I don't mean to say that such a system would be a good choice. But simply that it would be trading the rights of alive for the rights of not yet born.

I apologize, if above was inappropriate.

Comment author: Jiro 12 May 2014 08:57:34PM 1 point [-]

But even giving them less then optimal features - intelligence, strength, looks - is quite equivalent to making them stupidier, weaker, uglier.

I don't believe that killing someone is equivalent to letting him die. Why should I believe that making someone stupid is equivalent to letting him be stupid?

Also, cheating on someone to improve the health of the offspring results in a non-identity problem since the offspring is not the same one that would have been created without cheating, so whether the offspring is benefited is questionable.

Comment author: Jan_Rzymkowski 13 May 2014 03:09:21PM -1 points [-]

You're right. I got way too far with claiming equivalence.

As for non-identity problem - I have trouble answering it. I don't want to defend my idea, but I can think of an example when one brings up non-identity and comes to wrong conclusion: Drinking alcohol while pregnant can cause a fetus to develop a brain damage. But such grave brain damage means this baby is not the same one, that would be created, if his mother didn't drink. So it is questionable that the baby would benefit from its mother abstinence.

Comment author: PetjaY 24 December 2014 09:23:25PM 0 points [-]

"But can you have YOUR child, while eugenics prevent you from breeding? Not in genetic sense, but it seems deeply flawed to base parent-child relation simply on genetic code. It's upbringing that matters. Adopted child is in any meaningful way YOUR child."

Treating people not genetically your children as if they were is a big minus in our evolutionary game these days. It also helps bad behaviour (making children and letting others raise them), so i´d say that it manages to be bad both for yourself and population, though the second part depends on why the child was given for adoption.

In general improving gene pool would be a good idea, but finding collective solutions for it that don´t cause more bad than good seems hard. Also if our evolution gets rid of the heuristic that sex=children=good which isn´t working anymore and replaces it with something like "acts that lead to you children=good" we then get people spending their money smarter, which increases reproductive success of richer people who tend to be >average intelligent.