Matthew C quoting Einstein: "A human being is a part of the whole, called by us, "Universe," a part limited in time and space. He experiences himself, his thoughts and feelings as something separated from the rest -- a kind of optical delusion of his consciousness."

Further to this point, and Eliezer's description of the Rubicon: It seems that recognizing (or experiencing) that perceived separation is a step necessary to its eventual resolution. Those many who've never even noticed to ask the question will not notice the answer, no matter how close to them it may be.

Eliezer, A few years ago I sat across from you at dinner and mentioned how much you reminded me of my younger self. I expected, incorrectly, that you would receive this with the appreciation of a person being understood, but saw instead on your face an only partially muted expression of snide mirth. For the next hour you sat quietly as the conversation continued around us, and on my drive home from the Bay Area back to Santa Barbara I spent a bit more time reflecting on the various interactions during the dinner and updating my model of others and you.

For as long as I can remember, well before the age of 4, I've always experienced myself from both within and without as you describe. On rare occasions I've found someone else who knows what I'm talking about, but I can't say I've ever known anyone closely for whom it's such a strong and constant part of their subjective experience as it has been for me. The emotions come and go, in all their intensity, but they are both felt and observed. The observations of the observations are also observed, and all this up to typically and noticeably, about 4 levels of abstraction. (Reflected in my natural writing style as well.)

This leads easily and naturally to a model representing a part of oneself dealing with another part of oneself. Which worked well for me up until about the age of 18, when a combination of long-standing unsatisfied questions of an epistemological nature on the nature of induction and entropy, readings (Pirsig, Buckminster Fuller, Hofstadter, Dennett, and some of the more coherent and higher-integrity books on Buddhism) lead me to question and then reorganize my model of my relationship to my world. At some point about 7 years later (about 1985) it hit me one day that I had completely given up belief in an essential "me", while fully embracing a pragmatic "me". It was interesting to observe myself then for the next few years; every 6 months or so I would exclaim to myself (if no one else cared to listen) that I could feel more and more pieces settling into a coherent and expanding whole. It was joyful and liberating in that everything worked just as before, but I had to accommodate one less hypothesis, and certain areas of thinking, meta-ethics in particular, became significantly more coherent and extensible. [For example, a piece of the puzzle I have yet to encounter in your writing is the functional self-similarity of agency extended from the "individual" to groups.]

Meanwhile I continued in my career as a technical manager and father, and had yet to read Cosmides and Tooby, Kahneman and Tversky, E.T. Jaynes or Judea Pearl -- but when I found them they felt like long lost family.

I know of many reasons why its difficult to nigh impossible to convey this conceptual leap, and hardly any reason why one would want to make it, other than one who's values already drive him to continue to refine his model of reality.

I offer this reflection on my own development, not as a "me too" or any sort of game of competition of perceived superiority, but only as a gentle reminder that, as you've already seen in your previous development, what appears to be a coherent model now, can and likely will be upgraded (not replaced) to accommodate a future, expanded, context of observations.

@G: " if ethics were all about avoiding "getting caught", then the very idea that there could be an ethical "right thing to do" as opposed to what society wants one to do would be incoherent."

Well, I don't think anyone here actually asserted that the basis of ethics was avoiding getting caught, or even fear of getting caught. It seems to me that Eliezer posited an innate moral sense inhibiting risk-taking in the moral domain, and in my opinion this is more a reflection of his early childhood environment of development than any innate moral sense such as pride or disgust. Even though I think Eliezer was working from the wrong basis, I think he's offered a valid observation on the apparent benefit of "deep wisdom" with regard to tending to avoid "black swans."

But there seems to be an even more direct problem with your query, in that it's strictly impractical in terms of the information model it would entail, that individual agents would somehow be equipped with the same model of "right" as the necessarily larger model supported by society.

Apologies in advance, but I'm going to bow out of this discussion now due to diminishing returns and sensitivity to our host.

@George Weinberg: "...from an evolutionary perspective: why do we have a sense that we ought to do what is right as opposed to what society wants us to do?"

In other words, why don't humans function as mindless drones serving the "greater good" of their society? Like ants or bees? Well, if you were an ant or a bee, even one capable of speculating on evolutionary theory, you wouldn't ask that question, but rather its obverse. ;-)

Peter Watts wrote an entertaining bit of fiction, _Blindsight_ on a similar question, but to ask why would evolution do X rather than Y, imputes an inappropriate teleology.

Otherwise, if you were asking as to the relative merits of X versus Y, I think the most powerful answer would hinge on the importance of diversity at multiple levels for robust adaptability, rather than highest degree of adaptation.

And, it might help to keep in mind that biological organisms are adaptation executers, not fitness maximizers, and also that evolutionary economics favors satisficing over "optimizing."

@Caledonian: "...we must therefore conclude that a fatal flaw exists in our model..."

It's not necessarily that a "fatal flaw" exists in a model, but that all models are necessarily incomplete.

Eliezer's reasoning is valid and correct -- over a limited context of observations supporting meaning-making. It may help to consider that groups promote individual members, biological organisms promote genes, genes promote something like "material structures of increasing synergies"...

In cybernetic terms, in the bigger picture, there's nothing particularly privileged about the role of the gene, nor about biological evolutionary processes as a special case of a more fundamental organizing principle.

Eliezer: "The problem is that it's nigh mathematically impossible for group selection to overcome a countervailing individual selection pressure..."

While Eliezer's point here is quite correct within its limited context of individual selection versus group selection, it seems obvious, supported by numerous examples in nature around us, that his case is overly simplistic, failing to address multi-level or hierarchical selection effects, and in particular, the dynamics of **selection between groups**.

This would appear to bear also on difficulty comprehending selection between (and also within) multi-level agencies in the moral domain.

odf23ds: "Ack. Could you please invent some terminology so you don't have to keep repeating this unwieldy phrase?"

I'm eager for an apt idiom for the concept, and one also for "increasing coherence over increasing context."

It seems significant, and indicative of our cultural unfamiliarity -- even discomfort -- with concepts of systems, information, and evolutionary theory, that we don't have such shorthand.

But then I look at the gross misunderestimation of almost every issue of any complexity at every level of supposed sophistication of social decision-making, and then geek speak seems not so bad.

Suggestions?

Russell: "ethics consists of hard-won wisdom from many lifetimes, which is how it is able to provide me with a safety rail against the pitfalls I have yet to encounter in my single lifetime."

Yes, generations of selection for "what works" encoded in terms of *principles* tends to outweigh assessment within the context of an individual agent in terms of *expected utility* -- to the extent that the present environment is representative of the environment of adaptation. To the extent it isn't, then the best one can do is rely on the increasing weight of principles perceived hierarchically as increasingly effective over increasing scope of consequences, e.g. action on the basis of the principle known as the "law of gravity" is a pretty certain bet.

I'm in strong agreement with Peter's examples above. I would generalize by saying that the epistemic "dark side" tends to arise whenever there's an implicit discounting of the importance of increasing context. In other words, whenever, for the sake of expediency, "the truth", "the right", "the good". etc., is treated categorically rather than contextually (or equivalently, as if the context were fixed or fully specified.)

Phil: "Is that on this specific question, or a blanket "I never respond to Phil or Jef" policy?"

I was going to ask the same question, but assumed there'd be no answer from our gracious host. Disappointing.