@Eliezer: I can't imagine why I might have been amused at your belief that you are what a grown-up Eliezer Yudkowsky looks like.
No, but of course I wasn't referring to similarity of physical appearance, nor do I characteristically comment at such a superficial level. Puhleease.
I don't know if I've mentioned this publicly before, but as you've posted in this vein several times now, I'll go ahead and say it:
functional self-similarity of agency extended from the 'individual' to groups
I believe that the difficult-to-understand, high-sounding ultra-abstract concepts you use with high frequency and in great volume, are fake. I don't think you're a poor explainer; I think you have nothing to say.
If I don't give you as much respect as you think you deserve, no more explanation is needed than that, a conclusion I came to years ago.
Well that explains the ongoing appearance of disdain and dismissal. But my kids used to do something similar and then I was sometimes gratified to see shortly after an echo of my concepts in their own words.
Let me expand on my "fake" hint of a potential area of growth for your moral epistemology:
If you can accept that the concept of agency is inherent to any coherent meta-ethics, then we might proceed. But, you seem to preserve and protect a notion of agency that can't be coherently modeled.
You continue to posit agency that exploits information at a level unavailable to the system, and wave it away with hopes of math that "you don't yet have." Examples are your post today that has "real self" somehow dominating lesser aspects of self as if quite independent systems, or with your "profound" but unmodelable interpretation of ishoukenmei which bears only a passing resemblance to the very realistic usage I learned while living in Japan.
You continue to speak (and apparently think) in terms of "goals", even when such "goals" can't be effectively specified in the uncertain context of a complex evolving future, and you don't seem to consider the cybernetic or systems-theoretic reality that ultimately no system of interesting complexity, including humans, actually attains long-term goals so much as it simply tries to null out the difference between its (evolving) internal model and its perceptions of its present reality. All the intelligence is in the transform function effecting its step-wise actions. And that's good enough, but never absolutely perfect. But the good enough that you can have is always preferable to the absolutely perfect that you can never have (unless you intend to maintain a fixed context.)
You posit certainty (e.g. friendliness) as an achievable goal, and use rigorous-sounding terms like "invariant goal" in regard to decision-making in an increasingly uncertain future, but blatantly and blithely ignore concerns addressed to you over the years by myself and others as to how you think that this can possibly work, given the ineluctable combinatorial explosion, and the fundamentally critically underspecified priors.
I realize it's like a Pascal's Wager for you, and I admire your contributions in a sense somewhat tangential to your own, but like an isolated machine intelligence of high processing power but lacking an environment of interaction of complexity similar to its own - eventually you run off at high speed exploring quite irrelevant reaches of possibility space.
As to my hint to you today, if you have a workable concept of agency, then you might profit from consideration of the functional self-similarity of agencies composed of agencies, and so on, self-similar with increasing scale, and how the emergent (yeah, I know you dismiss "emergence" too) dynamics will tend to be perceived as increasingly moral (from within the system, as each of us necessarily is) due to the multi-level selection and therefore alignment for "what works" (nulling out the proximal difference between their model and their perceived reality, wash, rinse, repeat) by agents each acting in their own interest within an ecology of competing interests.
Sheesh, I may be abstract, I may be a bit too out there to relate to easily, but I have a hard time with "fake."
I meant to shake your tree a bit, in a friendly way, but not to knock you out of it. I've said repeatedly that I appreciate the work you do and even wish I could afford to do something similar. I'm a bit dismayed, however, by the obvious emotional response and meanness from someone who prides himself on sharpening the blade of his rationality by testing it against criticism.
Everything I am, is surely my brain; but I don't accept everything my brain does, as "me".
Though I don't like the fact, it does seem in my case to be true, that reflecting upon a passion can diminish it. It does seem that in taking a step back from myself to look at what my brain is doing, that this introduces a greater emotional distance from the world, than when I am not reflecting. Though I am extremely cautious when it comes to the moral issue of how to modify human nature, this is one thing that I would change, I think.
The point at which you start thinking that not everything that goes on your brain is to be identified with "you", is something of a Rubicon, I fear. And those who prize the intensity of a simple passionate existence, above success in life or a complex layered personality, should not cross that Rubicon.
(Though in this case you should accept advice from rationalists about practical policies for guarding yourself, because this is a dangerous world in which to be unreflective. For example: if you don't follow the link to the rest of this post, then sign the #$!@ up for cryonics regardless of what your simple passions say on the subject.)
Still here? Then here is the Rubicon of which I speak:
The first scenario is what I would call the intensity of simple passion; nothing between the world and the emotions. The sentence contains no "I" to get in the way. There is nothing to think about except the world itself, the sadness and the missing person.
In the second scenario it is not the world that is sad, but some particular person, an "I"; and the sadness of this "I" is something that can be called into question.
And in the third scenario, the borders of "I" have been drawn in a way that doesn't include everything in the brain, so that "I" is the victim of the brain, struggling against it. And this is not paradoxical. Everything that I am, has to be in my brain somewhere, because there is nowhere else for it to be. But that doesn't mean I have to identify with everything that my brain does. Just as I draw the border of "me" to include my brain but exclude my computer's CPU - which is still a sensible decision at least for now - I can define the border of myself to exclude certain events that happen in my brain, which I do not control, do not want to happen, and do not agree with.
That time I faced down the power-corrupts circuitry, I thought, "my brain is dumping this huge dose of unwanted positive reinforcement", and I sat there waiting for the surge to go away and trying not to let it affect anything.
Thinking "I am being tempted" wouldn't have quite described it, since the deliberate process that I usually think of as "me" - the little voice inside my own head - was not even close to being swayed by the attempted dose of reward that neural circuit was dumping. I wasn't tempted by power; I'd already made my decision, and the question was enforcing it.
But a dangerous state of mind indeed it would have been, to think "How tempting!" without an "I" to be tempted. From there it would only be a short step to thinking "How wonderful it is to exercise power!" This, so far as I can guess, is what the brain circuit is supposed to do to a human.
So it was a fine thing that I was reflective, on this particular occasion.
The problem is when I find myself getting in the way of even the parts I call "me". The joy of helping someone, or for that matter, the sadness of death - these emotions that I judge right and proper, which must be me if anything is me - I don't want those feelings diminished.
And I do better at this, now that my metaethics are straightened out, and I know that I have no specific grounds left for doubting my feelings.
But I still suspect that there's a little distance there, that wouldn't be there otherwise, and I wish my brain would stop doing that.
I have always been inside and outside myself, for as long as I can remember. To my memory, I have always been reflective. But I have witnessed the growth of others, and in at least one case I've taken someone across that Rubicon. The one now possesses a more complex and layered personality - seems more to me now like a real person, even - but also a greater emotional distance. Life's lows have been smoothed out, but also the highs. That's a sad tradeoff and I wish it didn't exist.
I don't want to have to choose between sanity and passion. I don't want to smooth out life's highs or even life's lows, if those highs and lows make sense. I wish to feel the emotion appropriate to the event. If death is horrible then I should fight death, not fight my own grief.
But if I am forced to choose, I will choose stability and deliberation, for the sake of what I protect. And my personality does reflect that. What you are willing to trade off, will sometimes get traded away - a dire warning in full generality.
This post is filed under "morality" because the question "Which parts of my brain are 'me'?" is a moral question - it's not predicted so much as chosen. You can't perform a test on neural tissue to find whether it's in or out. You have to accept or reject any particular part, based on what you think humans in general, and yourself particularly, ought to be.
The technique does have its advantages: It brings greater stability, being less subject to sudden changes of mind in the winds of momentary passion. I was unsettled the first time I met an unreflective person because they changed so fast, seemingly without anchors. Reflection conveys a visibly greater depth and complexity of personality, and opens a realm of thought that otherwise would not exist. It makes you more moral (at least in my experience and observation) because it gives you the opportunity to make moral choices about things that would otherwise be taken for granted, or decided for you by your brain. Waking up to reflection is like the difference between being an entirely helpless prisoner and victim of yourself, versus becoming aware of the prison and getting a chance to escape it sometimes. Not that you are departing your brain entirely, but the you that is the little voice in your own head may get a chance to fight back against some circuits that it doesn't want to be influenced by.
And the technique's use, to awaken the unreflective, is as I have described: First you must cross the gap between events-in-the-world just being terribly sad or terribly important or whatever, of themselves; and say, "I feel X". Then you must begin to judge the feeling, saying, "I do not want to feel this - I feel this way, but I wish I didn't." Justifying yourself with "This is not what a human should be", or "the emotion does not seem appropriate to the event".
And finally there is the Rubicon of "I wish my brain wouldn't do this", at which point you are thinking as if the feeling comes from outside the inner you, imposed upon you by your brain. (Which does not say that you are something other than your brain, but which does say that not every brain event will be accepted by you as you.)
After crossing this Rubicon you have set your feet fully upon the reflective Way; and I've yet to hear of anyone turning back successfully, though I think some have tried, or wished they could.
And once your feet are set on walking down that path, there is nothing left but to follow it forward, and try not to be emotionally distanced from the parts of yourself that you accept as you - an effort that a mind of simple passion would not need to make in the first place. And an effort which can easily backfire by drawing your attention to the layered depths of your selfhood, away from the event and the emotion.
Somewhere at the end of this, I think, is a mastery of techniques that are Zenlike but not Zen, so that you have full passion in the parts of yourself that you identify with, and distance from the pieces of your brain that you reject; and a complex layered personality with a stable inner core, without smoothing out those highs or lows of life that you accept as appropriate to the event.
And if not, then screw it, let's hack the brain so that it works that way. I have no confidence in my ability to judge how human nature should change, and would sooner leave it up to a more powerful mind in the same metamoral reference frame. But if I had to guess, I think that's the right thing to do.