I was going to say that this (although very good) wasn't quite Weird enough for your purposes; the principal value of the Baby-Eaters seems to be "individual sacrifice on behalf of the group", which we're all too familiar with. I can grok their situation well enough to empathize quickly with the Baby-Eaters. I'd have hoped for something even more foreign at first sight.
Then I checked out the story title again.
Eagerly awaiting the next installments!
Carl:
Those are instrumental reasons, and could be addressed in other ways.
I wouldn't want to modify/delete hatred for instrumental reasons, but on behalf of the values that seem to clash almost constantly with hatred. Among those are the values I meta-value, including rationality and some wider level of altruism.
I was trying to point out that giving up big chunks of our personality for instrumental benefits can be a real trade-off.
I agree with that heuristic in general. I would be very cautious regarding the means of ending hatred-as-we-know-it in human ...
Carl:
I don't think that automatic fear, suspicion and hatred of outsiders is a necessary prerequisite to a special consideration for close friends, family, etc. Also, yes, outgroup hatred makes cooperation on large-scale Prisoner's Dilemmas even harder than it generally is for humans.
But finally, I want to point out that we are currently wired so that we can't get as motivated to face a huge problem if there's no villain to focus fear and hatred on. The "fighting" circuitry can spur us to superhuman efforts and successes, but it doesn't seem to...
Roko:
Not so fast. We like some of our evolved values at the expense of others. Ingroup-outgroup dynamics, the way we're most motivated only when we have someone to fear and hate: this too is an evolved value, and most of the people here would prefer to do away with it if we can.
The interesting part of moral progress is that the values etched into us by evolution don't really need to be consistent with each other, so as we become more reflective and our environment changes to force new situations upon us, we realize that they conflict with one another. The analysis of which values have been winning and which have been losing (in different times and places) is another fascinating one...
Doug S:
If the broker believes some investment has a positive expectation this year (and is not very likely to crash terribly), he could advise John Smith to invest in it for a year minus a day, take the proceeds and go to Vegas. If he arrives with $550,000 instead of $500,000, there's a betting strategy more likely to wind up with $1,000,000 than the original plan.
The balance of risk and reward between the investment part and the Vegas part should have an optimal solution; but since anything over $1,000,000 doesn't factor nearly as much in John's utility ...
Given your actual reasons for wondering about the world economy in 2040 conditioned on there not having been an extinction/Singularity yet, the survivalist option is actually worth a small hedge bet. If you can go (or convince someone else to go) live in a very remote area, with sufficient skills and resources to continue working quietly on building an FAI if there's a non-existential global catastrophe, that looks like it has a strongly positive expectation (since in those circumstances, the number of competing AI attempts will probably be few if any).
No...
Sexual Weirdtopia: What goes on consensually behind closed doors doesn't (usually) affect the general welfare negatively, so it's not a matter of social concern. However, that particular bundle of biases known as "romantic love" has led to so much chaos in the past that it's become heavily regulated.
People start out life with the love-module suppressed; but many erstwhile romantics feel that in the right circumstances, this particular self-deception can actually better their lives. If a relationship is going well, the couple (or group, perhaps)...
A rogue paperclipper in a mostly Friendly world can probably only be stopped by racial prejudice--to a rational creature, it's always easier to feed him your neighbor than it is to fight him.
A couple of problems with this statement, as I see it:
You could be using &quo...
Richard, You're making the exact point Eliezer just did, about how modeling the effects of intelligence doesn't generally proceed by running a simulation forward. The "ordinarily" he speaks of, I assume, refers to the vast majority of physical systems in the Universe, in which there are no complicated optimization processes (especially intelligences) affecting outcomes on the relevant scales.
Vladimir,
Just to clarify (perhaps unnecessarily): by an attractor I mean a moral framework from which you wouldn't want to self-modify radically in any direction. There do exist many distinct attractors in the space of 'abstracted idealized dynamics', as Eliezer notes for the unfortunate Pebblesorters: they might modify their subgoals, but never approach a morality indifferent to the cardinality of pebble heaps.
Eliezer's claim of moral convergence and the CEV, as I understand it, is that most humans are psychologically constituted so that our moral framew...
I agree— and I balk at the concept of "the" Coherent Extrapolated Volition precisely because I suspect there are many distinct attractors for a moral framework like ours. Since our most basic moral impulses come from the blind idiot god, there's no reason for them to converge under extrapolation; we have areas of agreement today on certain extrapolations, but the convergence seems to be more a matter of cultural communication. It's not at all inconceivable that other Everett branches of Earth have made very different forms of moral progress fro...
Well, I find that my metamorality meets those criteria, with one exception.
To reiterate once, I think that the foundations of morality as we understand it are certain evolved impulses like the ones we can find in other primates (maternal love, desire to punish a cheater, etc); these are like other emotions, with one key difference: the social component that we expect and rely on others having the same reaction, and accordingly we experience other emotions as more subjective and our moral impulses as more objective.
Note that when I'm afraid of something, an...
Hmm. These doubts might seem sophomoric to us, since the "idiot god" of evolution couldn't conspire against our reasoning with the thoroughness of the Dark Lords of the Matrix. But it makes sense to consider these questions in the course of programming an AI, who will have cause to wonder whether its creators might have intentionally circumscribed its reasoning faculties...
Also, the problem with "cogito, ergo sum" is that it tempts us to posit a self distinct from the act of thinking, thus an immaterial soul, when the best interpretat...
Silas: Some of the more progressive Christian denominations, perhaps? Most of the elite members have become entirely embarrassed of claiming things like the unique divinity of Jesus, but manage to keep this relatively silent (with the partial exception of defectors like ex-Bishop Spong) so as not to offend the more traditional believers in their communion (who of course know about the elites' unbelief).
The Episcopal Communion, in particular, is going more into schism the more people start to reveal their real theologies.
I fall closer to the morality-as-preference camp, although I'd add two major caveats.
One is that some of these preferences are deeply programmed into the human brain (i.e. "Punish the cheater" can be found in other primates too), as instincts which give us a qualitatively different emotional response than the instincts for direct satisfaction of our desires. The fact that these instincts feel different from (say) hunger or sexual desire goes a long way towards answering your first question for me. A moral impulse feels more like a perception of...
What would I do?
When faced with any choice, I'd try and figure out my most promising options, then trace them out into their different probable futures, being sure to include such factors as an action's psychological effect on the agent. Then I'd evaluate how much I prefer these futures, acknowledging that I privilege my own future (and the futures of people I'm close to) above others (but not unconditionally), and taking care not to be shortsighted. Then I'd try to choose what seems best under those criteria, applied as rationally as I'm capable of.
You know, the sort of thing that we all do anyway, but often without letting our conscious minds realize it, and thus often with some characteristic errors mixed in.
Eliezer,
Every time I think you're about to say something terribly naive, you surprise me. It looks like trying to design an AI morality is a good way to rid oneself of anthropomorphic notions of objective morality, and to try and see where to go from there.
Although I have to say the potshot at Nietzsche misses the mark; his philosophy is not a resignation to meaninglessness, but an investigation of how to go on and live a human or better-than-human life once the moral void has been recognized. I can't really explicate or defend him in such a short remark...
HA:
Those are interesting empirical questions. Why jump to the conclusion?
I didn't claim it was a proof that some sort of algorithm was running; but given the overall increased effectiveness at maximizing utility that seems to come with the experience of deliberation, I'd say it's a very strongly supported hypothesis. (And to abuse a mathematical principle, the Church-Turing Thesis lends credence to the hypothesis: you can't consistently compete with a good algorithm unless you're somehow running a good algorithm.)
Do you have a specific hypothesis you thin...
Usually I don't talk about "free will" at all, of course! That would be asking for trouble - no, begging for trouble - since the other person doesn't know about my redefinition.
Boy, have we ever seen that illustrated in the comments on your last two posts; just replace "know" with "care". I think people have been reading their own interpretations into yours, which is a shame: your explanation as the experience of a decision algorithm is more coherent and illuminating than my previous articulation of the feeling of free will ...
...I actually can't see how the world would be different if I do have free will or if I don't. (Stephen Weeks)
In order for you to have free will, there has to be a "you" entity in the first place. . . (Matthew C.)
I have an idea where Eliezer is going with this, and I think the above comments are helpful in it.
Seems to me that the reason people intuitively feel there must be some such thing as free will is that there's a basic notion of free vs. constrained in social life, and that we project physical causality of our thoughts to be of the same fo...
David,
You're right not to feel a 'blow to your immortality' should that happen; but consider an alternate story:
You step into the teleport chamber on Earth and, after a weird glow surrounds you, you step out on Mars feeling just fine and dandy. Then somebody tells you that there was a copy of you left in the Earth booth, and that the copy was just assassinated by anti-cloning extremists.
The point of the identity post is that there's really no difference at all between this story and the one you just told, except that in this story you subjectively feel yo...
Dave,
Well, if you resolve not to sign up for cryonics and if the thinking on Quantum Immortality is correct, you might expect a series of weird (and probably painful) events to prevent you indefinitely from dying; while if you're signed up for it, the vast majority of the worlds containing a later "you" will be the ones revived after a peaceful death. So there's a big difference in the sort of experience you might anticipate, depending on whether you've signed up.
Caledonian,
Oh, sure, ant colonies are optimization processes too. But there are a few criteria by which we can distinguish the danger of an ant colony from the danger of a human from the danger of an AGI. For example:
(1) How powerful is the optimization processâ how tiny is the target it can achieve? A sophisticated spambot might reliably achieve proper English sentences, but I work towards a much smaller target (namely, a coherent conversation) which the spambot couldn't reliably hit.
Not counting the production of individual ants (which is the result...
Eliezer,
I also think that considering the particular topics is helpful here. In the math book, you were pretty confident the statement was wrong once you discovered a clear formal proof, because essentially there's nothing more to be said.
On the interpretation of quantum mechanics, since you believe we have almost all the relevant data we'll ever have (save for observed superpositions of larger and larger objects) and the full criteria to decide between these hypotheses given that information, you again think that disagreement is unfounded.
(I suggest you ...
Caledonian, I think Eliezer's going off of his distinction (in Knowability of AI and elsewhere) between "optimal" and "optimized", which more colloquial senses of the words don't include. There may be more optimal ways of achieving our goals, but that doesn't take away from the fact that we regularly achieve results that
(1) we explicitly set out to do (2) we can distinguish clearly from other results (3) would be incredibly unlikely to achieve by random effort.
I.e. this comment isn't close to optimal, but it's optimized enough as a coh...
Bambi,
The 'you gotta believe me technology' remark was probably a reference to the AI-Box Experiment.
Phillip,
None of the defenses you mentioned are safe against something that can out-think their designers, any more than current Internet firewalls are really secure against smart and determined hackers.
And blocking protein nanotech is as limited a defense against AGI as prohibiting boxcutters on airplanes is against general terrorist attack. Eliezer promoted it as the first idea he imagined for getting into physical space, not the only avenue.
Frank, I think you have an idea that many-worlds means a bunch of parallel universes, each with a single past and future, like parallel train tracks. That is most emphatically not what the interpretation means. Rather*, all of the universes with my current state in their history are actual futures that the current me will experience (weighted by the Born probabilities).
If there's an event which I might or might not witness (but which won't interfere with my existence), then that's really saying that there are versions of me that witness it and versions o...
Bad analogy, actually. If I have an incurable terminal illness today and fall asleep, I'll still have an incurable terminal illness in most of the worlds in which I wake upâ so I should assign a very low subjective probability to finding myself cured tomorrow. (Or, more precisely, the vast majority of the configurations that contain someone with all my memories up to that point will be ones in which I'm waking up the next day with the illness.)
I'm not quite sure how it might play out subjectively at the very end of life sans cryonics; this is where the...
Sorry to be late to the partyâ but has nobody yet mentioned the effect that MWI has on assessing cryonics from a personal standpoint; i.e. that your subjective probability of being revived should very nearly be your probability estimate that revival will happen in some universe? If 9/10 of future worlds destroy all cryogenic chambers, and 9/10 of the ones left don't bother to revive you, then it doesn't matter to you: you'll still wake up and find yourself in the hundredth world. Such factors only matter if you think your revival would be a significant...
Does anyone else suspect that the last full paragraph is meant to give us the assignment for tomorrow morning?
As for my answers, I think that the particulars of this paradigm shift have to enter into it on some level— because as Eliezer pointed out earlier, the Schrödinger's Cat thought experiment really should have suggested the possibility of superimposed observers to someone, and from there the MWI doesn't seem too remote.
So I'd have to ascribe the delay in the MWI proposal in great part to the fact that it doesn't immediately cohere with our subjective...
Well, now I think I understand why you chose to do the QM series on OB. As it stands, the series is a long explication of one of the most subtle anthropocentric biases out there— the bias in favor of a single world with a single past and future, based on our subjective perception of a single continuous conscious experience. It takes a great deal of effort before most of us are even willing to recognize that assumption as potentially problematic.
Oh, and one doesn't even have to assume the MWI is true to note this; the single-world bias is irrationally strong in us even if it turns out to correspond to reality.
I just wanted to say I've benefited greatly from this series, and especially from the last few posts. I'd studied some graduate quantum mechanics, but bailed out before Feynman paths, decoherence, etc; and from what I'd experienced with it, I was beginning to think an intuitive explanation of (one interpretation of) quantum mechanics was nigh-impossible. Thanks for proving me wrong, Eliezer.
The argument (from elegance/Occam's Razor) for the many-worlds interpretation seems impressively strong, too. I'll be interested to read the exchanges when you let the one-world advocates have their say.
More to the point: (P or ~P) isn't a theorem, it's an axiom. It is (so far as we can tell) consistent with our other axioms and absolutely necessary for many important theorems (any proof by contradiction— and there are some theorems like Brouwer's Fixed Point Theorem which, IIRC, don't seem to be provable any other way), so we accept a few counterintuitive but consistent consequences like (G or ~G) as the price of doing business. (The Axiom of Choice with the Banach-Tarski Paradox is the same way.)
OK, I've said enough on that tangent.
Actually, you can't quite escape the problem of the excluded middle by asserting that "This sentence is false" is not well-formed, or meaningful; because Gödel's sentence G is a perfectly well-formed (albeit horrifically complicated) statement about the properties of natural numbers which is undecidable in exactly the same way as Epimenides' paradox.
Mathematicians who prefer to use the law of excluded middle (i.e. most of us, including me) have to affirm that (G or ~G) is indeed a theorem, although neither G nor ~G are theorems! (This doesn't lead to a contradiction within the system, fortunately, because it's also impossible to formally prove that neither G nor ~G are theorems.)
This matters emotionally, even though it shouldn't (or seems like it shouldn't).
Hypothetical money is not treated as equivalent to possessed money.
My point exactly. It's perfectly understandable that we've evolved a "bird in the hand/two in the bush" heuristic, because it makes for good decisions in many common contexts; but that doesn't prevent it from leading to bad decisions in other contexts. And we should try to overcome it in situations where the actual outcome is of great value to us.
A utility function can take things other than money in...
How do the commenters who justify the usual decisions in the face of certainty and uncertainty with respect to gain and loss account for this part of the post?
There are various other games you can also play with certainty effects. For example, if you offer someone a certainty of $400, or an 80% probability of $500 and a 20% probability of $300, they'll usually take the $400. But if you ask people to imagine themselves $500 richer, and ask if they would prefer a certain loss of $100 or a 20% chance of losing $200, they'll usually take the chance of losing...
Interesting. Since people are commenting on fiction vs. non-fiction, it's interesting to note that my formative books were all non-fiction (paleontology, physics, mathematics, philosophy), and that I now find myself much more easily motivated to try understanding the problems of the world than motivated to try fixing them.
Plural of anecdote, etc, etc.