"Surely, then, superintelligence would necessarily imply supermorality" Thought the cow, as a bolt plunged into its brain.
Now see, that's exactly the sort of comment that led the young Eliezer to associate criticism of the intelligence-morality link with bad surface analogies. An easy enough monster-argument to slay, but I didn't do quite as well on reconstructing the corpse into something scarier.
Then why don't you go ahead and slay it? I share your dislike for surface analogies, but it seems like this one runs deeper.
Although the cow doesn't have the intelligence to form that thought, the point is that the hypothetical cow thinks "It takes intelligence to increase my utility function, therefore intelligence much greater than mine must lead to greater increases in my utility". It turns out that the cow is wrong, and a counterexample is us. There are supercow intelligences running around, but they kill and eat cows which is presumably not something the cow wants.
If you get the exact same argument out of a human brain, it's just as invalid, though (thankfully) there isn't any real life example to point to.
The deep connection is the same; there is more than one possible utility function.
I like "we are the cards we are dealt", which expresses nicely a problem with common ideas of blame and credit. I disagree that intelligence is the unfairest card of all - I think that a relatively dim person born into affluence in the USA has a much better time of it than a smart person born into poverty in the Congo.
It would be interesting to read a post that describes how a future society would look like if everyone was given the ability of todays top 2% regarding IQ. What would happen, implications, economic output, happiness and so on.
a relatively dim person born into affluence in the USA has a much better time of it than a smart person born into poverty in the Congo.
Taboo 'better'. I wouldn't swap one IQ point for all the silver spoons in the world.
You wouldn't give up one IQ point for say 10 million dollars? It would be a painful decision, but I'm convinced I could have a much better effect on the world with a massive financial head start at only the slightest detriment of my intelligence. A large enough sum of money would afford me the chance to stop working and study and research the rest of my life, probably leading me to be more intelligent in the long run. Right now, I have to waste away my time with a superior level of intelligence just pay for food, shelter and student loans.
Agreed. A lot of what we call intelligence is really speed - both in the short run (how long it takes you to add two numbers in your head, for instance) and in the longer run (how long it takes you to accomplish your ambitious projects). Ten million dollars would free up so much time and let you fake so much long-term speed that it would almost certainly be a gain if you got it for one IQ point. Not that anyone's actually offering this trade.
In general, cows seem to do pretty well out of being eaten - there are now hundreds of millions of them on the planet. If only they could find a way of making themselves more tasty.
Paul, fair point. I'd better say instead that granted a chance to grow up healthy and given a solid (!) education, I'd spend all my other pre-natal person-points on a nice high g. Or at least, I'd say that would make me about the most effective optimiser I could be.
Tim, I think you mean cow genes, rather than cows. Growing large slabs of meat in factories will be great for cow genes, but pretty disastrous for cows. Good for me though, as that's when I plan to jump right off the wagon and into a nice synthetic steak. Eliezer's right though; this has little ...
Warren Buffet uses the 'birth points' idea ("Ovarian Lottery" in his terminology) in a great thought experiment for developing his morality and ethics.
http://rationalangle.blogspot.com/2007/12/warren-buffett-and-hillary-clinton-at.html
At the end he puts a political slant on it, but I've read other instances where he puts it into larger terms.
... if everyone was given the ability of todays top 2% regarding IQ. What would happen, implications, economic output, happiness and so on.
This doesn't seem outlandish. In my former field, advances in gene therapy have been able (in animal models) to improve the function of tissues. Observations such as: the association of autosomal recessive and low-penetrance dominant mutations in Ashkenazim with high intelligence. Without at least heterozygosity for the health disorders associated with the mutations, Ashkenazim are no more intelligent, in the aggregate,...
"Without at least heterozygosity for the health disorders associated with the mutations, Ashkenazim are no more intelligent, in the aggregate, than non-Ashkenazy Jews."
Retired, you're wrong. The largest hypothesized effects of the disease alleles would be only a small fraction of the Ashkenazim advantage: they just aren't frequent enough. If you had a selective pressure for IQ, then it would affect all IQ-influencing alleles, reducing the frequency of rare variants with negative effects (Ashkenazi have lower rates of IQ-reducing PKU alleles), ch...
Ori: It seems to me that what you're describing has already been approximated, due to the filtering effects of certain job markets and employers. Look to Seattle's Eastside, or Silicon Valley. I've never been to the latter, but the former is a lot like heaven, except that the streets aren't slated to be paved with gold until 2014. (Planning takes time.)
I'm uncertain whether Eliezer-1995 was equating intelligence with the ability to self-optimize for utility (ie intelligence = optimization power) or if he was equating intelligence with utility (intelligence is great in and of itself). I would agree with Crowly that intelligence is just one of many factors influencing the utility an individual gets from his/her existence. There are also multiple kinds of intelligence. Someone with very high interpersonal intelligence and many deep relationships but abyssmal math skills may not want to trade places with...
Thank you, Carl Shulman, for correcting my misinformation. It's difficult for one to know which references are reliable, when one is not in the field.
@Carl Shulman: The largest hypothesized effects of the disease alleles would be only a small fraction of the Ashkenazim advantage: they just aren't frequent enough.
Dr. Bostrom cites this paper (so I considered it might be reliable) in his treatise on cognitive enhancement: "Natural History of Ashkenazi Intelligence" by Gregory Cochran, Jason Hardy, Henry Harpending. Speaking of the incidence of of t...
We are the cards we are dealt, and intelligence is the unfairest of all those cards.
I completely agree with that statement, though my interpretation of it might be the opposite of Eliezer's. From The Simpsons:
Lisa: Dad, as intelligence goes up, happiness often goes down. In fact, I made a graph! [She holds up a decreasing, concave upward graph on axes marked "intelligence" and "happiness"] Lisa: [sadly] I make a lot of graphs.
Dear Scott Aaronson:
Pffft.
Let me know if you do not find this argument convincing, and I will expand on it somewhat.
Lara, Eliezer-1995 is pre-Bayesian-enlightenment so he wouldn't have spent a lot of time talking about "utility". But yes, intelligence was a terminal value to him.
As long as you are sharing your development with us, I'd be curious to know why the young Eliezer valued intelligence so highly as to make it a terminal value. He must have enjoyed what he thought was 'intelligence' tremendously, and seen that people who did not share in this intelligence, did not share in his enjoyment and felt sorry for them. Moreover, he must not have been jealous of any enjoyments his less intelligent brethren seemed to partake in that he did not. He probably also did some sort of correlative analysis observing people he considered ...
Eliezer, I don't think there's a necessary tradeoff between intelligence (the academic rather than interpersonal kind) and happiness at the far nerd end of the spectrum---just that the way society is currently organized, it seems to be both true and common knowledge that there is (cf. Lara Foster's comment). Though despite the temptation, I can't justify dwelling on this phenomenon for too long---any more than on physical appearance, parental wealth, or any other aspect of our lives that we might love to "choose wisely" but can't. Unlike many o...
Retired: 59% of the population having alleles that boost IQ by <10 pts only accounts for part of the gap, and most of the alleles probably have effects substantially <10pts (ITD, at about 10pts, is probably the largest effect). Also, what sort of selective pressures would produce only costly alleles of large effect without boosting the frequencies of cheap alleles of small effect? Actually, that's my main problem with the Cochran Harpending hypothesis. The impact from the large alleles seems too large compared to the total advantage. Pressure su...
Retired,
59% frequency isn't enough to explain the size of measured IQ differences between Ashkenazi and non-Ashkenazi populations. This is off-topic, so let's take it to email.
Taboo 'better'. I wouldn't swap one IQ point for all the silver spoons in the world.Possession of that much silver would be a much more potent instrument than a single IQ point for almost any goal I can conceive of, and for any of the 'realistic' goals I can consider.
Even valuing one's self-function highly, I'd say that turning down all of the world's silver spoons for a single IQ point is a poor decision.
If we don't take your statement literally, then you're even wronger than before. By the world's standards, there are countless 'silver spoons' that ha...
Michael, Your question is very ill-defined. I regularly partake in a drug that lowers my IQ in exchange for other utility... It's called alcohol. If you are talking about permanent IQ reductions, I would need to have some sense of what losing one IQ point felt like before I could evaluate a trade. Is it like taking one shot? Would I even notice it missing?
Many psychotropic drugs, especially antipsychotics, 'slow' down the people that take them and thus could be associated with lowering IQ, yet many people choose to take them and lower their IQ for the utility gained by not hearing demonic voices or being allowed to leave a mental institution.
how much money (or other utility-bearing fruit) would you demand (or pay Scott) to take a drug which lowered your IQ by x pts?
Here's the funny thing: given who I am now, I would not pay to have my IQ lowered, and indeed would pay good money to avoid having it lowered, or even to have it raised. But I would also pay to have been, since early childhood, the sort of person who didn't have such an intelligence-centric set of priorities. I'm not transitive in my preferences; I don't want to want what I want.
Interesting, Scott. What priorities do the intelligence-centric type have that make you unhappy? Though I might not necessarily fit into this group, I am confident that I am of above-average intelligence, and I do not believe my litany of worldly woes are attributable to that, so much as to specific personality traits independent of intelligence.
In his comment to Scott Aaronson, Eliezer seems skeptical of extreme intelligence being detrimental to happiness. It is however my understanding that statistics favor Scott's view.
Such statistical data are discussed in this article:
Lara: I regularly partake in a drug that lowers my IQ in exchange for other utility... It's called alcohol.
You've just summarized my complete refusal of all alcoholic beverages better than I ever could. I try not to be too annoying about it, but I really do find the stuff quite horrifying.
Scott: Here's the funny thing: given who I am now, I would not pay to have my IQ lowered, and indeed would pay good money to avoid having it lowered, or even to have it raised. But I would also pay to have been, since early childhood, the sort of person who didn't have such an intelligence-centric set of priorities. I'm not transitive in my preferences; I don't want to want what I want.
How much would you pay to retain your present intelligence, but be born into a world where that intelligence was average? I've never regretted being smart, but I sometimes wish I wasn't smarter. I think that's at least 50% of what people who complain about being smart are really complaining about.
But I try not to complain about that either - it seems like whining, considering all the people who would commit murder to swap places with me. We all have our own troubles and they aren't any less troubling just because other people have troubles too. But I wouldn't want to swap places with those people who want to swap places with me. The grass is greener on this side of the fence.
I don't have a problem, my environment has a problem.
Lara: As far as I can tell, there are four basic problems.
First, if adults constantly praise and reward you for solving math problems, writing stories, and so on, then you aren't forced to develop interpersonal skills to the same extent most kids are. You have a separate source of self-worth, and it may be too late that you realize that source isn't enough. (Incidentally, the sort of interpersonal skills I'm talking about often get conflated with caring for others' welfare, which then leads to moral condemnation of nerds as egotistical and aloof. But th...
I don't have a problem, my environment has a problem.
Eliezer, I'm in complete sympathy with that attitude. I've had only limited success so far at nerdifying the rest of the world, but I'll keep at it!
"Here's the funny thing: given who I am now, I would not pay to have my IQ lowered, and indeed would pay good money to avoid having it lowered, or even to have it raised. But I would also pay to have been, since early childhood, the sort of person who didn't have such an intelligence-centric set of priorities. I'm not transitive in my preferences; I don't want to want what I want."
Scott,
Does this mean that you would take a pill that transferred your neurotransmitter reward responses and so forth to less brainy activities? Assume that it would also transfer your relative skill levels to the new activities, that your friends (old and new) would be compatible with the activities, etc.
There are certain diseases which cause 'brain fog' that could give insight into trade offs of gaining/losing intelligence. I'm cold (literally, often less than 96 degrees) and it effects my cognition at times. The drop in IQ is probably much greater than 1 point. Personally I would do anything short of violence to prevent it.
Interestingly certain quantities of alcohol seem to increase my intelligence, mostly in areas I normally suffer in (like word recall, especially in a foreign language).
Scott: "You have a separate source of self-worth, and it may be too late that you realize that source isn't enough."
Interesting theory of why intelligence might have a negative correlation with interpersonal skills, though it seems like a 'just so story' to me, and I would want more evidence. Here are some alternatives: 'Intelligent children find the games and small-talk of others their own age boring and thus do not engage with them.' 'Stupid children do not understand what intelligent children are trying to tell them or play with them, and th...
Regarding alcohol and lowering of IQ, am I correct in assuming that we are talking here about the temporary negative effects on cognitive ability of occasionally imbibing alcohol in moderation? Or are there studies I'm unaware of that show that occasional alcohol use has an adverse and permanent effect on cognitive ability?
If we're talking about the former, I'd be curious if those who are so vehemently anti-alcohol under all circumstances and with no exceptions are consistent in their application of the rule that anything that temporarily decreases cogniti...
Ori: "It would be interesting to read a post"
It's only a story (intended to illustrate a completely unrelated point), but do see "That Alien Message."
"that describes how a future society would look like if everyone was given the ability of todays top 2% regarding IQ. What would happen, implications, economic output, happiness and so on."
If you mean shifting the entire bell curve two sigmas to the right, I'll say we can't know in detail because the Singularity would happen soon afterwards. If you mean repealing the central limi...
My best guess is . . . the selection [pressure on 'g' in Ashkenazim] being from some trait other than 'g' which is much more enhanced by the alleles in question than 'g' is and which correlates with 'g'. The personality trait Openness would be a good candidate for such a trait.
Interesting conjecture, Michael Vassar. A very salient characteristic of younger Eliezer's writings is great openness -- about things like his history and his internal thought processes. I have been trying to be more open, except when I talk to Muggles, in which case I have been trying to be more circumspect in my speech and not to volunteer negative information about myself.
Rather strangely, all the intelligent people here seem to be talking about "intelligence" as if it could be measured by points on some linear scale. It probably varies with age, time, mood, etc. even for a single person. And there is hardly any good definition of what it is to begin with.
Richard, Openness means with respect to things (ideas/experiences) going in, not things going out.
epwripi: This might sound cockier than I mean it to, but really, I tire of such assertions. I know what intelligence is, and I suspect many here do as well. Plenty of good definitions have been put forth, but somebody is always going to have a nitpick because it doesn't include their favorite metaphor, and there are always going to be people who don't want it to be defined. It can certainly be quantified roughly and relatively, at the least (though "points on a linear scale" may be tending towards a strawman extreme), and when people speak of an ...
Sorry, this I realize is entirely off topic. Where should I move the discussion to? Ppl can take it to email with me if they like (cingulate2000@gmail.com).
Hmm... musing again on the psycho-social development of children and the role of adult approval. Scott suggested that being rewarded by adults for academic development may have impeded his social development.
I wonder if there are any social psychology studies in which a child is chosen at random to be favored by an adult authority figure, an what happens to that child's interactions with peers, and s...
andy, I understand what you say, but I was not referring to the entire discussion in my comment. I posted it impulsively after reading some specific comments that seemed to rather seriously discuss the hypothetical merits of one point of IQ and such. I only meant that it is pointless to discuss it as if everything is so precise and quantifiable, forgetting the fact that the concept itself is hazy (even though, as everyone knows, you can roughly categorize people as being more intelligent/less intelligent when the differences are very clear)
If there is that 'g'/unhappiness correlation, maybe the causality is: unhappiness->'g'. The overly happy, seeing less problems, get less problem solving practice, whereas a tendency to be analytical could boost 'g' over a lifetime, though perhaps not effective intelligence.
I wouldn't expect this to apply to most readers, who get particular pleasure from solving intelligent problems. Think general population.
Difference between "expirience" and "intelligence" is the difference between aproahes of Google and SIAI.
First gives instant results and grow lineary. Second will give no result until some treshold will be passed.
Expirience could gave better prediction in complex situation then pure inteligence without knowleghe
My own intelligence had affected every aspect of my life and mind and personality; that was massively obvious, seen at a backward glance.
Obviously you are a great deal smarter than I was as a child...and maybe more of a contrarian. I have a tendency to smooth conflicts over rather than attack them head-on, which probably makes it harder for me to be a rationalist, and most of what I experience is bumping up against the limits of my native intelligence: concepts I kind of understand, but which are too complex to hold in my working memory all at once so ...
"We are the cards we are dealt, and intelligence is the unfairest of all those cards. More unfair than wealth or health or home country, unfairer than your happiness set-point. "
I hope you were just exaggerating to make a point ^^; Otherwise this comes off as a fairly privileged comment that assumes a lot about how fair the world is when it comes to wealth, health, and country...
I suppose we could taboo "intelligence" and see if we really disagree, but you've already excluded health and wealth so I suspect we really do have approximate...
When I think of poor hands dealt to humans, these days, I think first of death and old age. Everyone's got to have some intelligence level or other, and the important thing from a fun-theoretical perspective is that it should ought to increase over time, not decrease like now.
This is a really important point, and I want to make certain that I get it right - especially to you personally, Mr. Yudkowsky, since you seem like someone with a higher-than-epsilon chance of actually doing something about all of this.
Solve people's lack of motivation and expertise for self-improvement before you handle our old age and death, please. Please.
Because, speaking as someone caught deep in the throes of a flawed optimization loop, the prospect of being caught in such a loop for centuries is terrifying.
Just as initial conditions are hideously unfair, life-paths are also hideously unfair, and the universe does not owe anyone the capacity, let alone the opportunity, to achieve meaning and purpose and happiness in their life.
And I don't know about others, but being condemned to an eternity as myself, damned to struggle futilely to achieve some understanding or purpose that will always be one lev...
I disagree with the fundamental premise here. I would much rather be immortal and stuck in an akratic loop for a few centuries - because a few centuries is very finite and I'll still be alive at the end.
While even if I become absurdly productive and self-controlled, I will still die like a dog of disease & decay in the likely event there is no Singularity and SENS fails.
Remember Steve Jobs: he used all the cutting-edge treatments and even used his billions to buy his way to the head of the transplant line - and died anyway.
My parents always used to downplay the value of intelligence. And play up the value of—effort, as recommended by the latest research? No, not effort. Experience. A nicely unattainable hammer with which to smack down a bright young child, to be sure. That was what my parents told me when I questioned the Jewish religion, for example. I tried laying out an argument, and I was told something along the lines of: "Logic has limits, you'll understand when you're older that experience is the important thing, and then you'll see the truth of Judaism." I didn't try again. I made one attempt to question Judaism in school, got slapped down, didn't try again. I've never been a slow learner.
Whenever my parents were doing something ill-advised, it was always, "We know better because we have more experience. You'll understand when you're older: maturity and wisdom is more important than intelligence."
If this was an attempt to focus the young Eliezer on intelligence uber alles, it was the most wildly successful example of reverse psychology I've ever heard of.
But my parents aren't that cunning, and the results weren't exactly positive.
For a long time, I thought that the moral of this story was that experience was no match for sheer raw native intelligence. It wasn't until a lot later, in my twenties, that I looked back and realized that I couldn't possibly have been more intelligent than my parents before puberty, with my brain not even fully developed. At age eleven, when I was already nearly a full-blown atheist, I could not have defeated my parents in any fair contest of mind. My SAT scores were high for an 11-year-old, but they wouldn't have beaten my parents' SAT scores in full adulthood. In a fair fight, my parents' intelligence and experience could have stomped any prepubescent child flat. It was dysrationalia that did them in; they used their intelligence only to defeat itself.
But that understanding came much later, when my intelligence had processed and distilled many more years of experience.
The moral I derived when I was young, was that anyone who downplayed the value of intelligence didn't understand intelligence at all. My own intelligence had affected every aspect of my life and mind and personality; that was massively obvious, seen at a backward glance. "Intelligence has nothing to do with wisdom or being a good person"—oh, and does self-awareness have nothing to do with wisdom, or being a good person? Modeling yourself takes intelligence. For one thing, it takes enough intelligence to learn evolutionary psychology.
We are the cards we are dealt, and intelligence is the unfairest of all those cards. More unfair than wealth or health or home country, unfairer than your happiness set-point. People have difficulty accepting that life can be that unfair, it's not a happy thought. "Intelligence isn't as important as X" is one way of turning away from the unfairness, refusing to deal with it, thinking a happier thought instead. It's a temptation, both to those dealt poor cards, and to those dealt good ones. Just as downplaying the importance of money is a temptation both to the poor and to the rich.
But the young Eliezer was a transhumanist. Giving away IQ points was going to take more work than if I'd just been born with extra money. But it was a fixable problem, to be faced up to squarely, and fixed. Even if it took my whole life. "The strong exist to serve the weak," wrote the young Eliezer, "and can only discharge that duty by making others equally strong." I was annoyed with the Randian and Nietszchean trends in SF, and as you may have grasped, the young Eliezer had a tendency to take things too far in the other direction. No one exists only to serve. But I tried, and I don't regret that. If you call that teenage folly, it's rare to see adult wisdom doing better.
Everyone needed more intelligence. Including me, I was careful to pronounce. Be it far from me to declare a new world order with myself on top—that was what a stereotyped science fiction villain would do, or worse, a typical teenager, and I would never have allowed myself to be so cliched. No, everyone needed to be smarter. We were all in the same boat: A fine, uplifting thought.
Eliezer1995 had read his science fiction. He had morals, and ethics, and could see the more obvious traps. No screeds on Homo novis for him. No line drawn between himself and others. No elaborate philosophy to put himself at the top of the heap. It was too obvious a failure mode. Yes, he was very careful to call himself stupid too, and never claim moral superiority. Well, and I don't see it so differently now, though I no longer make such a dramatic production out of my ethics. (Or maybe it would be more accurate to say that I'm tougher about when I allow myself a moment of self-congratulation.)
I say all this to emphasize that Eliezer1995 wasn't so undignified as to fail in any obvious way.
And then Eliezer1996 encountered the concept of the Singularity. Was it a thunderbolt of revelation? Did I jump out of my chair and shout "Eurisko!"? Nah. I wasn't that much of a drama queen. It was just massively obvious in retrospect that smarter-than-human intelligence was going to change the future more fundamentally than any mere material science. And I knew at once that this was what I would be doing with the rest of my life, creating the Singularity. Not nanotechnology like I'd thought when I was eleven years old; nanotech would only be a tool brought forth of intelligence. Why, intelligence was even more powerful, an even greater blessing, than I'd realized before.
Was this a happy death spiral? As it turned out later, yes: that is, it led to the adoption even of false happy beliefs about intelligence. Perhaps you could draw the line at the point where I started believing that surely the lightspeed limit would be no barrier to superintelligence. (It's not unthinkable, but I wouldn't bet on it.)
But the real wrong turn came later, at the point where someone said, "Hey, how do you know that superintelligence will be moral? Intelligence has nothing to do with being a good person, you know—that's what we call wisdom, young prodigy."
And lo, it seemed obvious to the young Eliezer, that this was mere denial. Certainly, his own painstakingly constructed code of ethics had been put together using his intelligence and resting on his intelligence as a base. Any fool could see that intelligence had a great deal to do with ethics, morality, and wisdom; just try explaining the Prisoner's Dilemma to a chimpanzee, right?
Surely, then, superintelligence would necessarily imply supermorality.
Thus is it said: "Parents do all the things they tell their children not to do, which is how they know not to do them." To be continued, hopefully tomorrow.
Post Scriptum: How my views on intelligence have changed since then... let's see: When I think of poor hands dealt to humans, these days, I think first of death and old age. Everyone's got to have some intelligence level or other, and the important thing from a fun-theoretical perspective is that it should ought to increase over time, not decrease like now. Isn't that a clever way of feeling better? But I don't work so hard now at downplaying my own intelligence, because that's just another way of calling attention to it. I'm smart for a human, if the topic should arise, and how I feel about that is my own business. The part about intelligence being the lever that lifts worlds is the same. Except that intelligence has become less mysterious unto me, so that I now more clearly see intelligence as something embedded within physics. Superintelligences may go FTL if it happens to be permitted by the true physical laws, and if not, then not.