Given that the full title of the book is "The Moral Landscape: How Science Can Determine Human Values," I think that conclusion is the major one, and certainly the controversial one. "Science can help us judge things that involve facts" and similar ideas aren't really news to anyone who understands science. Values aren't a certain kind of fact.

I don't see where Sean's conclusions are functionally different from those in the metaethics sequence. They're presented in a much less philosophically rigorous form, because Sean is a physicist, not a philosopher (and so am I). For example, this statement of Sean's:

But there’s no reason why we can’t be judgmental and firm in our personal convictions, even if we are honest that those convictions don’t have the same status as objective laws of nature.

and this one of Eliezer's:

Yes, I really truly do believe that humanity is better than the Pebblesorters! I am not being sarcastic, I really do believe that.

seem to express the same sentiment, to me.

If you really object to Sean's writing, take a look at Russell Blackford's review of the book. (He is a philosopher, and a transhumanist one at that.)

except about the true nature of terminal values, which is a major conclusion

In quick approximation, what was this conclusion?

I try not to rationalize.

I don't think we should reward correct guesses that were made for the wrong reasons (and are only correct by certain stretches of vocabulary). Talking about superintelligences is more precise and avoids vast planes of ambiguity and negative connotations, so why not just do that?

tl;dr - If you're going to equate morality with taste, understand that when we measure either of the two, taking agents into the process is a huge fact we can't leave out

I'll be upfront about having not read Sam Harris' book yet, though I did read the blog review to get a general idea. Nonetheless, I take issue with the following point:

Carroll's point at the end about attempting to find the 'objective truth' about what is the best flavor of ice cream echoes my thoughts so far on the "Moral Landscape".

I've found that an objective truth about the best flavor of ice cream can be found if one figures out which disguised query they're after. (Am I looking for "If I had to guess, what would random person z's favorite flavor of ice cream be, with no other information?" or am I looking for something else).

This attempt at making morality too subjective to measure by relating it to taste has always bothered me because people always ignore a main factor here: agents should be part of our computation. When I want to know what flavor of ice cream is best, I take into account people's preferences. If I want to know what would be the most moral action, I need to take into account it's effects on people (or myself, should I be a virtue ethicist, or how it aligns with my rules, should I be a deontologist). Admittedly the latter is tougher than the former, but that doesn't mean we have no hoped of dealing with it objectively. It just means we have to do the best we can with what we're given, which may mean a lot of individual subjectivity.

In his book Stumbling on Happiness, Daniel Gilbert writes about studying the subjective as objectively as possible when he decides on the three premises for understanding happiness: 1] Using imperfect tools sucks, but it's better than no tools. 2] An honest, real-time insider view is going to be more accurate than our current best outside views. 3] Abuse the law of real numbers to get around the imperfections of 1] and 2] (a.k.a measure often)

I also would like to see a link to that post, if anyone recognizes it.

I'll agree that to (atheist) me, it certainly seems that one big support for religious belief is the natural human tendency toward wishful thinking. However, it doesn't do much good to provide convincing arguments against religion as atheists picture it. You need convincing arguments against religion as its practitioners see it.

Once you have your maybe-they're-believing-this-because-that-would-be-a-cool-thing-to-believe lenses on, it all seems very transparent.

Yeah, I know what you mean. Pity I can't turn that around and use it against simulationism. :)

Dennett tells a similar "agentification" story:

"I think we can discern religion’s origins in superstition, which grew out of an overactive adoption of the intentional stance,” he says. “This is a mammalian feature that we share with, say, dogs. If your dog hears the thud of snow falling off the roof and jumps up and barks, the dog is in effect asking, ‘Who’s there?’ not, ‘What’s that?’ The dog is assuming there’s an agent causing the thud. It might be a dangerous agent. The assumption is that when something surprising, unexpected, puzzling happens, treat it as an agent until you learn otherwise. That’s the intentional stance. It’s instinctive.” The intentional stance is appropriate for self-protection, Dennett explains, and “it’s on a hair trigger. You can’t afford to wait around. You want to have a lot of false positive, a lot of false alarms [...]” He continues: “Now, the dog just goes back to sleep after a minute. But we, because we have language, we mull it over in our heads and pretty soon we’ve conjured up a hallucinated agent, say, a little forest god or a talking tree or an elf or something ghostly that made that noise. Generally, those are just harmless little quirks that we soon forget. But every now and then, one comes along that has a little bit more staying power. It’s sort of unforgettable. And so it grows. And we share it with a neighbor. And the neighbor says, ‘What do you mean, a talking tree? There’s no talking trees.’ And you say, ‘I could have sworn that tree was talking.’ Pretty soon, the whole village is talking about the talking tree.

I think that is usually called Patternicity these days. See:

Patternicity: Finding Meaningful Patterns in Meaningless Noise

Why the brain believes something is real when it is not - By Michael Shermer

Extended digression about how to adjust Solomonoff induction for making anthropic predictions plz

This just means you have a very narrow (Abrahamic) conception of God that not even most Christians have. (At least, most Christians I talk to have super-fuzzy-abstract ideas about Him, and most Jews think of God as ineffable and not personal these days AFAIK.) Otherwise your distinction makes little sense. (This may very well be an argument against ever using the word 'God' without additional modifiers (liberal Christian, fundamentalist Christian, Orthodox Jewish, deistic, alien, et cetera), but it's not an argument that what people sometimes mean by 'God' is a wrong idea. Saying 'simulator' is just appealing to an audience interested in a different literary genre. Turing equivalence, man!)

Of note is that the less memetically viral religions tend to be saner (because missionary religions mostly appealed to the lowest common denominator of epistemic satisfiability). Buddhism as Buddha taught it is just flat out correct about nearly everything (even if you disagree with his perhaps-not-Good but also not-Superhappy goal of eliminating imperfection/suffering/off-kilteredness). Many Hindu and Jain philosophers were good rationalists (in the sense that Epicurus was a good rationalist), for instance. To a first and third and fifth approximation, every smart person was right about everything they were trying to be right about. Alas, humans are not automatically predisposed to want to be right about the super far mode considerations modern rationalists think to be important.

To me it seems nakedly obvious that "God faked the details of evolution" is a far more outre and improbable theory than "our universe is a simulation and the simulation is approximate". I should've been able to leave filling in the details as an exercise to the reader.

Trusting ones 'gut' impressions of the "nakedly obvious" like that and 'leaving the details as an exercise' is a perfectly reasonable thing to do when you have a well-tuned engine of rationality in your possession and you just need to get some intellectual work done.

But my impression of the thrust of the OP was that he was suggesting a bit of time-consuming calibration work so as to improve the tuning of our engines. Looking at our heuristics and biases with a bit of skepticism. Isn't that what this community is all about?

But enough of this navel gazing! I also would like to see that digression on Solomonoff induction in an anthropic situation.

Seconding Kevin's request. Seeing a sentence like that with no followup is very frustrating.

Thx. But I don't read that as arguing against religion. Instead it seems to be an argument against one feature of modern religion - its claim to unfalsifiability (since it deals with a Non-Overlapping MAgisterium, 'NOMA' using the common acronym). Eliezer thinks this is pretty wimpy. He seems to have more respect for old-time religion, like those priests of Baal who stuck their necks out, so to speak, and submitted their claims to empirical testing.

Can this attitude of critical rationalism be redeployed against simulationist claims? Or at least against the claims of those modern simulationists who keep their simulations unfalsifiable and don't permit interaction between levels of reality? Against people like Bostrom who stipulate that the simulations that they multiply (without necessity) should all be indistinguishable from the real thing - at least to any simulated observer? I will leave that question to the reader. But I don't think that it qualifies as a posting in which Eliezer argues against religion in toto. He is only arguing against one feature of modern apologetics.

I'm afraid I don't see the application to simulation arguments. You will have to spell it out.

I fully agree with EY that Occam is not a valid argument against MWI. For that matter, I don't even see it as a valid argument against the Tegmark Ultimate Ensemble. But I do see it as a valid argument against either a Creator (unneeded entity) or a Simulator (also an unneeded entity). The argument against our being part of a simulation is weakened only if we already know that simulations of universes as rich as ours are actually taking place. But we don't know that. We don't even know that it is physically and logically possible.

Nevertheless, your mention of MWI and simulation in the same posting brings to mind a question that has always bugged me. Are simulations understood to cover all Everett branches of the simulated world? And if they are understood to cover all branches, is that broad coverage achieved within a single (narrow) Everett branch of the universe doing the simulating?

Have you read the paper?

Yes, I had read Bostrom's paper.

I'm not convinced of it for a few reasons, but I'd consider it located at least.

I would express my opinion of that argument using less litotes. But as to locating the hypotheses, I suppose I agree.

Which leads me to ask, have you read the catechism? Like most Catholic schoolchildren, I was encouraged to memorize much of it in elementary school, though I have since forgotten almost all of it. It also locates one hypothesis, a hypothesis considerably more popular than Bostrom's.

Thanks!

That's a good point, but where do you recommend getting contrarian glee separate from communication?

I agree, though I was describing the case where I can do both simultaneously (when I'm talking to people who either don't mind or join in on the fun). This post was more an example of just not realizing that the use of the word 'theism' would have such negative and distracting connotations.

This does not follow. It is not necessary for my argument that different religions all be related to each other; it is only necessary that BTanism not be related to any of them, and (this part I asserted implicitly by linking to Religion's Claim to be Non-Disprovable) that it not have been generated by a similar process.

OK, I think I now understand the implicit part; I think you mean that religions of old made total, and not merely ontological, claims, which BTanism doesn't (I wasn't sure before what you were picking up from Religion's Claim to be Non-Disprovable, which I do know and read before; I thought it had something to do with disprovability).

I think you're right to point to that distinction.

Varieties of "theism" which have similar epistemological status to BTanism are not subject on LW to the same kind of dismissal as religion, to the best of my knowledge. Nor should they be. But for the sake of avoiding confusion and undesirable connotations, they certainly shouldn't be called "theism".

Well, why not, if they're varieties of theism? Perhaps it'd be better if LW found another word to condemn, other than theism?

Such a word could be... theism! It does have two definitions, a broad and a narrow one. I checked a few dictionaries to be sure, and one of them helpfully elucidated the broad one as "the opposite of atheism", and the narrow one as "the opposite of deism".

If what you mean here is "merely community-cohesiveness driven phenomenon", then I disagree entirely.

"Largely", rather than "merely", is how I would put it. I'm not certain I understand the rest of your paragraph. To my mind, atheism (or, more precisely, strong dismissal of theism) being incidental to LW's charter doesn't mean it can't become a way to cohere the group, to nurture a sense of belonging. Note, by the way, that rejection of theism made it to the Welcome post, and is a unique example of a specific shared LW value there. Although that may be for pragmatic rather than signalling reasons.

For my part, I see "open-mindedness" toward theism mostly as manifesting an inability to come to gut-level terms with the fact that large segments of the human population can be completely, totally wrong.

That's an interesting theory I'd have to think about. Do you consider agnosticism as a subset of "open-mindedness", and thus the above as the primary explanation of agnosticism?

Which?

I don't know; there are several possibilities and it'd be impolite, not to mention fruitless, on my part to speculate.

Again, this is Less Wrong, not a random internet forum. It is not possible to recapitulate the Sequences in every comment; that doesn't mean that strong opinions whose justifications lie therein are inadequately supported.

Agreed in general.

Not sure how well this applies in the particular case. This thread has focused on two assertions in your original comment: "[not] memetically related" and "superficial resemblance ... is so slight that you would never notice it unless you were motivated to do so, or heard it from someone who was". You cited a Sequence post in your follow-up comment about the former (but I don't see any reference to that post or the idea of total claims of religions in your original comment - correct me if you disagree), and after some thickness on my part I acknowledge its relevance here. You don't seem to rely on anything from the Sequences for the latter.

You're making good points, with which I largely agree, with some reservations (see below). I'd just point out that this wasn't the argument Komponisto was making - he was talking only about relatedness in the ancestry sense.

Your list of attributes is probably good enough to distinguish e.g. a simulation argument from "religions" and justify not calling it one. There are two difficulties, however. One is that adherence to these attributes isn't nearly as uniform among religions as it's often rhetorically assumed on LW to be. There's a tendency to: start talking about theism; assume in your argument that you're dealing with something like an omnipresent, omniscient monotheistic God of Judaism/Christianity whose believers are all Bible literalists; draw the desired conclusion and henceforth consider it applying to "theism" or "religion" in general. I find this fallacious tendency to be frequent in discussions of theism on LW. This comment from the earlier discussion is relevant, as are some other comments there. In this post, Eliezer comments that believing in simulation/the Matrix means you're believing in powerful aliens, not deities. Well, consider ancient Greek gods; they are not omniscient, not omnipresent, they can die... they're not more powerful than the simulation runners, and arguably not very ontologically different; are they not deities, but aliens? Was that not religion? [1]

It's kind of understandable that one thinks of the concept of God and Jehovah pops into view. But if you stick with Jehovah - and not even any Jehovah, but a particular, highly literally interpreted kind - it's no good pretending afterwards that you've dealt a blow to religion or to theism.

So proper account of what religions are actually out there makes your list of attribute much less universal, and the dividing line between religions and something like BTanism much less sharp. But, to be clear, I still think this line can be usefully drawn.

The second difficulty is something I've already written to Komponisto above: OK, it's not a religion, so what? The really important thing is whether it's like a religion in those things that ought to make a rationalist not glibly and gleefully dismiss one if they're psyched about another. And among those things worship and sacred texts are arguably less important than e.g. falsifiability. Have you seen a good way to falsify a simulation claim recently?

[1] I just remembered that Dan Simmons develops this theme in Ilium/Olympos. The second book is much worse than the first one.

I suspect that a full universe simulation of all of space-time, fifteen billion years of an entire universe, may have a cost complexity such that it could never be realized in any currently conceivable computer due to speed of light limitations. Even a galaxy sized black hole may not be sufficient. You are talking about a Tipler-like scenario that would probably require some massive re-engineering of the entire universe. I can't rule this out, but from what I've read of astrophysicist's reactions, it is questionable whether it is possible even in principle to collapse the universe in the fashion required. (Tipler figures it requires tachyons in his later response writings)

So no, that would not be much easier to simulate - it would be vastly more difficult, and may not even be possible in principle.

The more likely simulation is one run by our posthuman ancestors after a local Singularity on earth, where they have a massive amount of computation, enough to simulate perhaps a galaxy or galaxies full of virtual humans, but not the entire history of our universe. We must remember that they will want to simulate many possible samples as well. They will also probably simulate hypothetical aliens and hypothetical contact scenarios. Basically they will simulate future important sample time-slices.

Today humanity as a whole spends a large amount of time thinking about the present, slightly alternate versions of the present, historical time heavily weighted based on importance, and projected futures. We already are engaging in the limited creation of simulated realities. The phenomenon has already begun, it started with dreams, language, thought and is more recently amplified with computer simulation and graphics, and just chart that trajectory out into the future and amplify it by an exponential vastening . . . .

You can't not be allowed to take the observer-ladenness of a universe as evidence.

Limiting case: Property X is true of a universe if and only if it has observers. May we take the fact that observers exist in our universe as evidence that observers exist there?

Probabilities were for example purposes only. I made them up because they were nice to calculate with and sounded halfway reasonable. I will not defend them. If you request that I come up with my real probability estimates, I will have to think harder.

The post is here. The reasoning as written is:

Cryonics is reasonable - Due to reading and understanding the quantum physics sequence, I ended up contacting Rudi Hoffman for a life insurance quote to fund cryonics. It’s only a few hundred dollars a year for me. It’s well within my budget for caring about myself and others... such as my future selves in forward branching multi-verses.

My comments on the subject (having cut out the tree debating MWI) can be found here.

Hmm from my reading of the thread it doesn't look like much of a consensus.

I may want to revive this - the arguments against practicality don't seem convincing from an engineering perspective.

From a high quality upload's scanned mind one should get a great deal of information about the upload's closest friends, relatives, etc. The data from any one such upload many not be overwhelming, but you'd start with a large population of such uploads. People who were well known and loved would be easier cases, but you could also supplement the data in many cases with low-quality scans from poorly preserved bodies.

This should give one prior generation. Going back another previous generation would get murkier, but is still quite possible, especially with all the accessory historical records.

The farther back you go, the less 'accurate' the uploads become, but the less and less important this 'accuracy' becomes.

For example, assuming I become a posthuman, I will be interested in bring back my grandfather. There a huge space of possible minds that could match my limited knowledge and beliefs about this person I never met. Each of them would fully be my grandfather from my subjective perspective and would fully be my grandfather from their subjective perspective.

There is no objective standard frame of reference from which to evaluate absolute claims of personal identity. It is relative.