I'm not sure, but was this line:
But, from the first species, we learned a fact which this ship can use to shut down the Earth starline
supposed to read "the Huygens starline"?
I was going to say that this (although very good) wasn't quite Weird enough for your purposes; the principal value of the Baby-Eaters seems to be "individual sacrifice on behalf of the group", which we're all too familiar with. I can grok their situation well enough to empathize quickly with the Baby-Eaters. I'd have hoped for something even more foreign at first sight.
Then I checked out the story title again.
Eagerly awaiting the next installments!
(E.g. repeating the mantra "Politics is the Mind-Killer" when tempted to characterize the other side as evil)
Uh, I don't mean that literally, though doing up a whole Litany of Politics might be fun.
Carl:
Those are instrumental reasons, and could be addressed in other ways.
I wouldn't want to modify/delete hatred for instrumental reasons, but on behalf of the values that seem to clash almost constantly with hatred. Among those are the values I meta-value, including rationality and some wider level of altruism.
I was trying to point out that giving up big chunks of our personality for instrumental benefits can be a real trade-off.
I agree with that heuristic in general. I would be very cautious regarding the means of ending hatred-as-we-know-it in human ...
Carl:
I don't think that automatic fear, suspicion and hatred of outsiders is a necessary prerequisite to a special consideration for close friends, family, etc. Also, yes, outgroup hatred makes cooperation on large-scale Prisoner's Dilemmas even harder than it generally is for humans.
But finally, I want to point out that we are currently wired so that we can't get as motivated to face a huge problem if there's no villain to focus fear and hatred on. The "fighting" circuitry can spur us to superhuman efforts and successes, but it doesn't seem to...
Roko:
Not so fast. We like some of our evolved values at the expense of others. Ingroup-outgroup dynamics, the way we're most motivated only when we have someone to fear and hate: this too is an evolved value, and most of the people here would prefer to do away with it if we can.
The interesting part of moral progress is that the values etched into us by evolution don't really need to be consistent with each other, so as we become more reflective and our environment changes to force new situations upon us, we realize that they conflict with one another. The analysis of which values have been winning and which have been losing (in different times and places) is another fascinating one...
Doug S:
If the broker believes some investment has a positive expectation this year (and is not very likely to crash terribly), he could advise John Smith to invest in it for a year minus a day, take the proceeds and go to Vegas. If he arrives with $550,000 instead of $500,000, there's a betting strategy more likely to wind up with $1,000,000 than the original plan.
The balance of risk and reward between the investment part and the Vegas part should have an optimal solution; but since anything over $1,000,000 doesn't factor nearly as much in John's utility ...
Given your actual reasons for wondering about the world economy in 2040 conditioned on there not having been an extinction/Singularity yet, the survivalist option is actually worth a small hedge bet. If you can go (or convince someone else to go) live in a very remote area, with sufficient skills and resources to continue working quietly on building an FAI if there's a non-existential global catastrophe, that looks like it has a strongly positive expectation (since in those circumstances, the number of competing AI attempts will probably be few if any).
No...
Sexual Weirdtopia: What goes on consensually behind closed doors doesn't (usually) affect the general welfare negatively, so it's not a matter of social concern. However, that particular bundle of biases known as "romantic love" has led to so much chaos in the past that it's become heavily regulated.
People start out life with the love-module suppressed; but many erstwhile romantics feel that in the right circumstances, this particular self-deception can actually better their lives. If a relationship is going well, the couple (or group, perhaps)...
A rogue paperclipper in a mostly Friendly world can probably only be stopped by racial prejudice--to a rational creature, it's always easier to feed him your neighbor than it is to fight him.
A couple of problems with this statement, as I see it:
You could be using &quo...
You might want to footnote, before anyone starts making noise about the ant example, that colony selection is not a case of group selection but a case of individual selection on the queen (and drones), since the rest of the ants don't reproduce.
Richard, You're making the exact point Eliezer just did, about how modeling the effects of intelligence doesn't generally proceed by running a simulation forward. The "ordinarily" he speaks of, I assume, refers to the vast majority of physical systems in the Universe, in which there are no complicated optimization processes (especially intelligences) affecting outcomes on the relevant scales.
Vladimir,
Just to clarify (perhaps unnecessarily): by an attractor I mean a moral framework from which you wouldn't want to self-modify radically in any direction. There do exist many distinct attractors in the space of 'abstracted idealized dynamics', as Eliezer notes for the unfortunate Pebblesorters: they might modify their subgoals, but never approach a morality indifferent to the cardinality of pebble heaps.
Eliezer's claim of moral convergence and the CEV, as I understand it, is that most humans are psychologically constituted so that our moral framew...
I agree— and I balk at the concept of "the" Coherent Extrapolated Volition precisely because I suspect there are many distinct attractors for a moral framework like ours. Since our most basic moral impulses come from the blind idiot god, there's no reason for them to converge under extrapolation; we have areas of agreement today on certain extrapolations, but the convergence seems to be more a matter of cultural communication. It's not at all inconceivable that other Everett branches of Earth have made very different forms of moral progress fro...
Well, I find that my metamorality meets those criteria, with one exception.
To reiterate once, I think that the foundations of morality as we understand it are certain evolved impulses like the ones we can find in other primates (maternal love, desire to punish a cheater, etc); these are like other emotions, with one key difference: the social component that we expect and rely on others having the same reaction, and accordingly we experience other emotions as more subjective and our moral impulses as more objective.
Note that when I'm afraid of something, an...
Hmm. These doubts might seem sophomoric to us, since the "idiot god" of evolution couldn't conspire against our reasoning with the thoroughness of the Dark Lords of the Matrix. But it makes sense to consider these questions in the course of programming an AI, who will have cause to wonder whether its creators might have intentionally circumscribed its reasoning faculties...
Also, the problem with "cogito, ergo sum" is that it tempts us to posit a self distinct from the act of thinking, thus an immaterial soul, when the best interpretat...
Silas: Some of the more progressive Christian denominations, perhaps? Most of the elite members have become entirely embarrassed of claiming things like the unique divinity of Jesus, but manage to keep this relatively silent (with the partial exception of defectors like ex-Bishop Spong) so as not to offend the more traditional believers in their communion (who of course know about the elites' unbelief).
The Episcopal Communion, in particular, is going more into schism the more people start to reveal their real theologies.
I fall closer to the morality-as-preference camp, although I'd add two major caveats.
One is that some of these preferences are deeply programmed into the human brain (i.e. "Punish the cheater" can be found in other primates too), as instincts which give us a qualitatively different emotional response than the instincts for direct satisfaction of our desires. The fact that these instincts feel different from (say) hunger or sexual desire goes a long way towards answering your first question for me. A moral impulse feels more like a perception of...
What would I do?
When faced with any choice, I'd try and figure out my most promising options, then trace them out into their different probable futures, being sure to include such factors as an action's psychological effect on the agent. Then I'd evaluate how much I prefer these futures, acknowledging that I privilege my own future (and the futures of people I'm close to) above others (but not unconditionally), and taking care not to be shortsighted. Then I'd try to choose what seems best under those criteria, applied as rationally as I'm capable of.
You know, the sort of thing that we all do anyway, but often without letting our conscious minds realize it, and thus often with some characteristic errors mixed in.
Interesting. Since people are commenting on fiction vs. non-fiction, it's interesting to note that my formative books were all non-fiction (paleontology, physics, mathematics, philosophy), and that I now find myself much more easily motivated to try understanding the problems of the world than motivated to try fixing them.
Plural of anecdote, etc, etc.