But how much has your intuitive revulsion at your dependence on others, your inability to do everything by yourself, biased your beliefs about what options you are likely to have. If wishes were horses you know. It is not clear what problems you can really blame on each of us not knowing everything we all know; to answer that you'd have to be clearer on what counterfactuals you are considering.
"But with a sufficient surplus of power, you could start doing things the eudaimonic way. Start rethinking the life experience as a road to internalizing new strengths, instead of just trying to keep people alive efficiently."
It should be noted that this doesn't make the phenomenon of borrowed strength go away, it just outsources it to the FAI. If anything, given the kind of perfect recall and easy access to information that an FAI would have, the ratio of cached historical information to newly created information should be much higher than that of a human. Of course, an FAI wouldn't suffer the problem of losing the information's deep structure like a human would, but it seems to be a fairly consistent principle that the amount of cached data grows faster than the rate of data generation.
The problem here- the thing that actually decreases utility- is humans taking actions without sufficient understanding of the potential consequences, in cases where "Humans seem to do very well at recognizing the need to check for global consequences by perceiving local features of an action." (CFAI 3.2.2) fails. I wonder, out of a sense of morbid curiosity, what the record is for the highest amount of damage caused by a single human without said human ever realizing that they did anything bad.
Robin, I'm not blaming the problem on each of us not knowing everything. To restate my thesis:
(1) The current scenario isn't set up for eudaimonic living; (2) Newton had more fun discovering calculus than you had reading about it; (3) A lot of the reason why people think of technology as a Grey Death Force has to do with their estrangement from their own tools; (4) The future need not be one of opaque gadgets with buttons to press that do complicated things.
Also, I had to learn to distrust knowledge that had only been told me; I could only wish it had been instinctive.
Tom, so long as the AI isn't sentient and would in fact be superintelligent enough to regenerate all the knowledge it has learned, we need be concerned neither with its eudaimonia nor its overreaching.
(2) is true, but forgets about opportunity costs. Newton had more fun discovering calculus than I do reading about it, but Newton took much longer in doing it, huge amounts of that time not necessarily being fun. In density of fun (amount of fun per time), I would say I had much higher density of fun reading the Sequences, or reading about physics (or maths or biology or economics) in books that if I would have to discover just one of the things by myself - at 30 I probably wouldn't have discovered more than one of the great insight, at best.
I love to be able to rebuild my tools by myself if needed - at school they made us re-implement malloc() and similar functions and that was a lot of fun. I would hate to be given a blackbox that does something, and not being able to known anything about how it works. But that doesn't mean I want to make all my tools myself, reinvent/rediscover everything. There is joy in the making/discovery, but there is also joy in using, and one shouldn't depend on the other.
You seem to consider that burrowing someone else's strength is a weakness, I consider it to be our greater strength. So together we can do more than any of us alone could do. And since we have different tastes and skills, everyone can do what he likes the most and/or what he's better at, the other burrowing his strength. A great composer may not be a great player, a great player may not be a great luthier. But when the three combine, they give everyone a lot of fun.
Tom, it's quite possible that the CEV would determine the Right Thing To Do is uplifting humans, in the David Brin sense, until they can once again wipe their own cosmic asses - and then for itself to bow out gracefully, and halt.
It's easy if you're allowed to keep the law of cosines ...
"Borrowing someone else's knowledge really doesn't give you anything remotely like the same power level required to discover that knowledge for yourself." Hmmm. This doesn't seem to me to be the way it works in domains of cumulatively developed competitive expertise such as chess, go, gymnastics and the like. In those domains the depth with which a technique penetrates you when you invent it is far less than that with which it penetrates your students to whom you teach it when they are children, or at least, that's my impression. Of course, if we could alternatively raise and lower our neoteny, gaining adult insights and then returning to childhood to truly learn them our minds might grow beyond what humans have yet experienced.
What you seem to be describing here is leverage. It's clear (more clear these days) that leverage is good for growth, but too much leverage and you are hosed.
Michael, that's an interesting way of looking at it. In retrospect I was reasoning something like this for the Overcoming Bias project - "No matter how much I write about rationality, I can't communicate the generator that I used to write the posts... but if someone reads it as a teenager and then grows up trying to develop it further, they might find it anyway" - but without the explicit generalization that could also apply to Go.
One really does want to try it from age seven, but I'm not sure how much of this stuff even I could have gotten at age seven. It'd be worth trying, though.
"One really does want to try it from age seven, but I'm not sure how much of this stuff even I could have gotten at age seven. It'd be worth trying, though."
I fully intend to teach my children (when I eventually have them, that is) about cognitive biases and rationality from the time they are born. I think that we greatly underestimate what children are capable of understanding. (It is also possible that I am biased, since I was an unusual child and so I can't generalize my experience across all children - but even if that's true, there is at least a good chance it will work with MY children, so much the better for them.) In the future, our children might be taught concepts in their earliest books that we have not even discovered today.
As a soon to be father for the first time I have every intention to similarly teach my future child, I certainly won't count on the culture - much less the educational system - to do it for me, even here in Cambridge UK. As a theatre practitioner I do have some hope and ambition that rationalist principles might begin to find their way into the arts by the time my child is old enough to notice. The arts may not be the most obvious vector for the rationalist meme, but they may well prove surprisingly effective over time. Hopefully Harry Potter and the Methods of Rationality will spawn a new generation of kids books that are fun and useful.
If you figure out some good ways to teach kids, Add them to the site - I think it'd be great to build up a repository of that kind of thing.. after all, I'm sure lots of us will have kids some day and will want to use and build on this knowledge.
I suspect the knowledge you get from reading someones writings is very different than the knowledge you get from working with them or them teaching you. When you work or learn closely with someone they can see your reasoning processes and correct them when they go astray at the right point when they are still newly formed and not too ingrained. Otherwise it relies too much on luck. When in someone intellectual career should you read OB, too early it won't mean too much lacking the necessary background and too late you will be inured against it (assuming it is the right way to go!).
Autodidacts are going to be most intellectually useful when you need to break new ground and the methodologies of the past aren't the way to solve the problems needed to be solved.
I'd say kids are never too young. First, they are already evolved to grow up in an environment of adult ideas, and to pick them up as they are capable. Second, talking to them about complex ideas will teach them not to fear that complexity, even if they don't understand everything. Much of our culture is built around stupidity being cool and irrationality being goodness.
I see a project like seasteading, and it reminds me a lot of a similar failed project I got excited about 10 years ago. If they want to live on the sea, why not buy a boat? Do you HAVE to add the complexity of a new vessel design to all the legal and social challenges? I mean long term habitation of the oceans is a solved problem, use the solution.
The first place I encountered the concept that strength must be earned was eight or nine years ago in a passage from, of all things, Jurassic Park, which stuck in my memory long after the other moments of the book faded from memory. The long version: http://www.stjohns-chs.org/english/Seventeenth/jur.html The short version: """ "I’ll make it simple" Malcolm said. "A karate master does not kill people with his bare hands. He does not lose his temper and kill his wife. The person who kills is the person who has no discipline no restraint, and who has purchased his power in the form of a Saturday night special. And that is the kind of power that science fosters, and permits. And that is why you think that to build a place like this is simple." "It was simple," Hammond insisted. "Then why did it go wrong?" """
"One really does want to try it from age seven, but I'm not sure how much of this stuff even I could have gotten at age seven. It'd be worth trying, though."
Eliezer, it sounds like you need to write a childrens' book.
This post has an enormous noise to content ratio. You gave only one example of a cost from using borrowed strength, and it was unsupported:
"But if no one had been able to use nuclear weapons without, say, possessing the discipline of a scientist and the discipline of a politician - without personally knowing enough to construct an atomic bomb and make friends - the world might have been a slightly safer place."
This is not clear; I would even say it's less than 50% probable. Many scientists, using heuristics against bias that turned out to be wrong in this case, underestimated the aggressiveness of the Soviet Union. Think Bertrand Russell, Albert Einstein, and maybe Oppenheimer. I am cherry-picking; but I don't think a Union of Concerned Scientists could have gotten us through the 1960s without a war with the Soviet Union.
Since kids were brought up, this made me think of the question - any suggestions of how to best teach children rationality from an early age? (This is probably worth a separate post.)
Michael Vassar has a good point that I will take in a different direction: inventors and exploiters are often quite different people. Great explorers are rarely great settlers.
The first person to develop an idea or technology rarely has the best idea of what to do with it. Perhaps s/he is too tied to that development process. Perhaps it takes a different part of the mind to optimize than to discover, and few people have strong modules for both. Sometimes the first mover wins, but the biggest winner is often the later mover who releases a better version.
Citing games again, I look to different sources for ideas and for finished optimizations. Some people can do both, and the more limited the search space the more likely it is that optimizers can find their own ideas. Several people will suggest that X and Y could work well together. They will experiment with things in-game. They will often have the best qualitative grasp of things. Then you bring in the spreadsheet masters to squeeze the last drop of optimization from it. These are the people who calculate the most efficient build for feats, talents, weapons, whatever your game has. Then you can pass that back to the community for people to use.
It takes a certain personality type to explore a new land. It takes a different personality type to start homesteading newly explored territory. It takes yet another to devise regular trade routes back to the mother country.
For anyone wondering where the quote is from: Morisato-san is Keiichi Morisato of the anime/manga franchise Oh My Goddess!, and the speaker is Belldandy.
I missed something obviously. Is there a post I could look at to find Eleizer's understanding of the source of his disagreement with Robin about the singularity?
Cameron, see True Sources of Disagreement.
Also, when in doubt, do what I do and check Andrew Hay's list of my posts.
@Cameron
My understanding of the disagreement appears different than Eli's. My impression is that the core of the disagreement lies in Robin's statement:
"This history of when innovation rates sped up by how much just doesn't seem to support your claim that the strongest speedups are caused by and coincide with new optimization processes, and to a lesser extent protected meta-level innovations"
The fwoom!, god-to-rule-us-all, and issues around models all seem to me to fall out of this contention.
After the discussion around the disagreement, I gave Hal Finney my original and new estimates on these issues. I would be willing to repeat them here if Hal will likewise repeat his, but I'm unsure if anyone is interested anymore.
[Deleted. Tim, you've been requested to stop talking about your views on sexual selection here. --EY]
Eliezer, I think you have somehow gotten very confused about the topic of my now-deleted post.
That post was entirely about cultural inheritance - contained absolutely nothing about sexual selection.
Please don't delete my posts - unless you have a good reason for doing so.
It seems to me that what you object to here as "wielding borrowed strength" are all cases of incompletely wielding borrowed strength... for example, Newton having access to the results of Galileo's celestial discoveries but not to the experience of discovery itself.
And, agreed, that can be dangerous. So can using an acetylene torch without knowing how.
As you suggest, one way around this would be for Newton to have actually made the same discoveries himself. And, sure, with enough surplus of power and time, and the elimination of any goals where reaching them sooner was important, we could set up the system to allow that.
But I submit that another way around it would be for Newton to have access to Galileo's experience. I mean, if the important aspects of Galileo's experience can be encoded in Galileo's brain and subsequently decoded and experienced by Galileo -- which I assume you agree is not only possible but routine -- there seems no in-principle reason why they can't also be encoded by Galileo's brain and subsequently decoded and experienced by Newton.
I think, on reflection, I want a future where experiences are routinely shared among individuals... not just sensory experiences, but cognitive ones. Like what we do today with writing and art, except much much better.
I want a future where if you've discovered Pythagoras' theorem from scratch while sitting on your couch, you can convey that experience to me via a deep encoding and not just via superficial ones, such that I also have the experience of having discovered Pythagoras' theorem from scratch while sitting on your couch.
(Yes, sure, there is some sense in which you actually had the experience and I didn't. And that distinction is important in many contexts... for example, if you discover something valuable, even if I share the experience of having discovered it I'm not entitled to any rewards for having discovered it. But in the context you've raised here, I don't think the distinction is important.)
I recognize that many people don't want this and indeed many are actively repulsed by it. Perhaps you're one of them... judging from the subtext of this post you seem to be, though I'm not entirely certain.
That's OK; I'm not trying to proselytize it. I don't think they/you would have any obligation to share my experiences any more than they/you are obligated today to read my writing.
But neither would I want to be obligated to keep my experiences isolated, or to be restricted to just the experiences my body has had, once experience-sharing technology became available.
So it'd be like playing Minecraft?
On reflection, I'd probably at least ask for a wiki pretty quickly.
Given how little we know right now, even standing on the shoulders of giants, it seems like the problem is not that we try to do things beyond our own strength, but that our own strength is so pathetically weak. The task therefore is to increase our own potential, make it more nearly possible for one person to actually understand everything there is to understand.
"Would Isaac Newton have remained a mystic, even in that earlier era, if he'd lived the lives of Galileo and Archimedes instead of just reading about them?"
Possibly, depending on your definition of "mystic". This is not a simple yes-and-no question because that which commands universal validity in the real world is nonetheless not fundamental to the human psyche. You value the power of intelligence and incessantly work to refine your art of rationality, and yet you complain of low mental energy. I don't think that's necessarily because you are a low-mental-energy person. It could just be that you're an imperfectly "rational" being and are, like the rest of us, ultimately motivated by a complex interplay of sense and emotion that can only be called poetry. (I'm not saying matters shouldn't be corrected by transhumanist methods, just observing that that's how they stand at the moment.) If you were to adopt classical rites known to confer such inspiration, you could be bounding from insight to insight riding a crest of divine frenzy. Such traditions give you access to poetic frameworks refined by generations of thinkers and consequently able to bestow tremendous power. Instead, you choose to immerse yourself in work and socialization. Those are themselves American Protestant rites, and they DO work, but they seem to suit you poorly. Why keep at it regardless, except to fool the eyes of society?
I would ask, would Newton have had the motivation to discover gravity if he hadn't been inspired by astrological mysteries? If so, what would be his incentive? What if modern humanism seemed as insipid to him as it does to me? Elsewhere, I believe you spoke of sacredness not being private, but the fact is, sacredness IS private in the sense that different people find different things sacred and even if you could list all the rational pillars supporting your perspective on the sacredness of a thing that are available to your conscious mind, predictably communicating to others a direct taste of your sensation of holiness would still be an immensely difficult endeavor. I say this because, knowing most of the reasons shuttle launches appear sacred to you, I can readily imagine how someone could find it sacred, and yet I do not share this feeling myself. Exhilarating, tense, joyful, among other things, but sacred? Not really, and I don't think belief and disbelief enter the picture when we're exploring the domain of sensation. Hence, private and incommunicable.
I confess, I worry you might have fallen prey to Post-Christian rationalizationism. See, Christians loved to leach the joy and meaning out of life wherever they didn't understand it, leaving a dry and lifeless husk which they proceeded to arbitrarily label "rational". Not just informally, but as a matter of church doctrine. They then mocked and acted dismissive and when necessary, passive-aggressive to anyone who disagreed with their point of view, which was effective at keeping people in line after centuries of violent evaporative cooling. Is it perhaps possible that everyone acted as though certain modes of behavior are Obviously Rational, and you believed them without systematically questioning their presuppositions? I first suspected the importance of ritual (in a broad sense of the word) in daily life when studying the tenth and final chapter of this book on Neo-Confucian metaphysics: http://faculty.washington.edu/mkalton/ I'm not sure you'd have the desire to read it with as much patience and forbearance as I've had to invest in it.
Who am I to accuse you of unthinking assimilationism anyway? I myself don't practice any traditional rites, though that's because the Neoplatonic rites, (http://www.youtube.com/watch?v=k-PkooJfLRA) which intrigue me the most, are, as far as I'm aware, lost. Thanks again, Christianity! Only the cheap, populist crap, the Christian, Gnostic and Hermetic rites, survive from classical antiquity, out of which the Christian ones are conveniently superior in terms of quality, having received the most attention and polish. Unfortunately, Christianity, at its core, is a constructivist doctrine with a deep distrust of individual self-cultivation (which existed in the West as in the East in Hellenic times) that does not conform to their self-righteous path of ostentatious self-abasement. On the bright side, several important expository texts have come down to us: http://www.scribd.com/doc/31503637/Proclus-on-the-Theology-of-Plato-all-six-books-plus-a-seventh-by-Thomas-Taylor
Followup to: Truly Part of You
"Myself, and Morisato-san... we want to live together by our own strength."
Jared Diamond once called agriculture "the worst mistake in the history of the human race". Farmers could grow more wheat than hunter-gatherers could collect nuts, but the evidence seems pretty conclusive that agriculture traded quality of life for quantity of life. One study showed that the farmers in an area were six inches shorter and seven years shorter-lived than their hunter-gatherer predecessors—even though the farmers were more numerous.
I don't know if I'd call agriculture a mistake. But one should at least be aware of the downsides. Policy debates should not appear one-sided.
In the same spirit—
Once upon a time, our hunter-gatherer ancestors strung their own bows, wove their own baskets, whittled their own flutes.
And part of our alienation from that environment of evolutionary adaptedness, is the number of tools we use that we don't understand and couldn't make for ourselves.
You can look back on Overcoming Bias, and see that I've always been suspicious of borrowed strength. (Even before I understood the source of Robin's and my disagreement about the Singularity, that is.) In Guessing the Teacher's Password I talked about the (well-known) problem in which schools end up teaching verbal behavior rather than real knowledge. In Truly Part of You I suggested one test for false knowledge: Imagine deleting a fact from your mind, and ask if it would grow back.
I know many ways to prove the Pythagorean Theorem, including at least one proof that is purely visual and can be seen at a glance. But if you deleted the Pythagorean Theorem from my mind entirely, would I have enough math skills left to grow it back the next time I needed it? I hope so—certainly I've solved math problems that seem tougher than that, what with benefit of hindsight and all. But, as I'm not an AI, I can't actually switch off the memories and associations, and test myself in that way.
Wielding someone else's strength to do things beyond your own understanding—that really is as dangerous as the Deeply Wise phrasing makes it sound.
I observed in Failing to Learn from History (musing on my childhood foolishness in offering a mysterious answer to a mysterious question): "If only I had personally postulated astrological mysteries and then discovered Newtonian mechanics, postulated alchemical mysteries and then discovered chemistry, postulated vitalistic mysteries and then discovered biology. I would have thought of my Mysterious Answer and said to myself: No way am I falling for that again."
At that point in my childhood, I'd been handed some techniques of rationality but I didn't exactly own them. Borrowing someone else's knowledge really doesn't give you anything remotely like the same power level required to discover that knowledge for yourself.
Would Isaac Newton have remained a mystic, even in that earlier era, if he'd lived the lives of Galileo and Archimedes instead of just reading about them? If he'd personally seen the planets reduced from gods to spheres in a telescope? If he'd personally fought that whole war against ignorance and mystery that had to be fought, before Isaac Newton could be handed math and science as a start to his further work?
We stand on the shoulders of giants, and in doing so, the power that we wield is far out of proportion to the power that we could generate for ourselves. This is true even of our revolutionaries. And yes, we couldn't begin to support this world if people could only use their own strength. Even so, we are losing something.
That thought occurred to me, reading about the Manhattan Project, and the petition that the physicists signed to avoid dropping the atomic bomb on Japan. It was too late, of course; they'd already built the bombs and handed them to the military, and they couldn't take back that gift. And so nuclear weapons passed into the hands of politicians, who could never have created such a thing through their own strength...
Not that I'm saying the world would necessarily have been a better place, if physicists had possessed sole custody of ICBMs. What does a physicist know about international diplomacy, or war? And it's not as if Leo Szilard—who first thought of the fission chain reaction—had personally invented science; he too was using powers beyond his own strength. And it's not as if the physicists on the Manhattan Project raised the money to pay for their salaries and their materials; they were borrowing the strength of the politicians...
But if no one had been able to use nuclear weapons without, say, possessing the discipline of a scientist and the discipline of a politician—without personally knowing enough to construct an atomic bomb and make friends—the world might have been a slightly safer place.
And if nobody had been able to construct an atomic bomb without first discovering for themselves the nature and existence of physics, then we would have been much safer from atomic bombs, because no one would have been able to build them until they were two hundred years old.
With humans leaving the game after just seventy years, we couldn't support this world using only our own strengths. But we have traded quality of insight for quantity of insight.
It does sometimes seem to me that many of this world's problems, stem from our using powers that aren't appropriate to seventy-year-olds.
And there is a higher level of strength-ownership, which no human being has yet achieved. Even when we run, we're just using the muscles that evolution built for us. Even when we think, we're just using the brains that evolution built for us.
I'm not suggesting that people should create themselves from scratch without a starting point. Just pointing out that it would be a different world if we understood our own brains and could redesign our own legs. As yet there's no human "rationalist" or "scientist", whatever they know about "how to think", who could actually build a rational AI—which shows you the limits of our self-understanding.
This is not the sort of thing that I'd suggest as an immediate alteration. I'm not suggesting that people should instantly on a silver platter be given full knowledge of how their own brains work and the ability to redesign their own legs. Because maybe people will be better off if they aren't given that kind of power, but rather have to work out the answer for themselves.
Just in terms of anomie versus fun, there's a big difference between being able to do things for yourself, and having to rely on other people to do them for you. (Even if you're doing them with a brain you never designed yourself.)
I don't know if it's a principle that would stay until the end of time, to the children's children. Maybe better-designed minds could handle opaque tools without the anomie.
But it is part of the commonly retold prophecy of Artificial Intelligence and the Age of the Machine, that this era must be accompanied by greater reliance on things outside yourself, more incomprehensible tools into which you have less insight and less part in their creation.
Such a prophecy is not surprising. That is the way the trend has gone so far, in our culture that is too busy staying alive to optimize for fun. From the fire-starting tools that you built yourself, to the village candleseller, and then from the candleseller to the electric light that runs on strange mathematical principles and is powered by a distant generator... we are surrounded by things outside ourselves and strengths outside our understanding; we need them to stay alive, or we buy them because it's easier that way.
But with a sufficient surplus of power, you could start doing things the eudaimonic way. Start rethinking the life experience as a road to internalizing new strengths, instead of just trying to keep people alive efficiently.
A Friendly AI doesn't have to be a continuation of existing trends. It's not the Machine. It's not the alien force of technology. It's not mechanizing a factory. It's not a new gadget for sale. That's not where the shape comes from. What it is—is not easy to explain; but I'm reminded of doc Smith's description of the Lens as "the physical manifestation of a purely philosophical concept". That philosophical concept doesn't have to manifest as new buttons to press—if, on reflection, that's not what we would want.