Maybe don't restrict "understand" to "be able to model and predict".

If you want "rational" to include moral, then you're not actually disagreeing with LessWrong about rationality (the thing), but rather about "rationality" (the word).

Likewise if you want "understanding" to also include "empathic understanding" (suffering when other people suffer, taking joy when other people take joy), you're not actually disagreeing about understanding (the thing) with people who want to use the word to mean "modelling and predicting" you're disagreeing with them about "understanding" (the word).

Are all your disagreements purely linguistic ones? From the comments I've read of you so far, they seem to be so.

I (whowhowho) was not defending that claim.

I quoted DP making that claim, said that claim confused me, and asked questions about what that claim meant. You replied by saying that you think DP is saying something which you then defended. I assumed, I think reasonably, that you meant to equate the thing I asked about with the thing you defended.

But, OK. If I throw out all of the pre-existing context and just look at your comment in isolation, I would certainly agree that Clippy is incapable of having the sort of understanding of suffering that requires one to experience the suffering of others (what you're calling a "full" understanding of suffering here) without preferring not to cause suffering, all else being equal.

Which is of course not to say that all else is necessarily equal, and in particular is not to say that Clippy would choose to spare itself suffering if it could purchase paperclips at the cost of its suffering, any more than a human would necessarily refrain from doing something valuable solely because doing so would cause them to suffer.

But I haven't yet seen any convincing evidence there could be an alien qualia-space whose inherently (dis)valuable textures map on to the (dis)valuable textures of the pain-pleasure axis.

Meaning mapping the wrong way round, presumably.

Without hedonic tone, how can anything _matter _ at all?

Good question.

if we encountered an agent who wanted to maximise paperclips today, we wouldn't think, ""wow, how incomprehensibly alien"

Agreed, as far as it goes. Hell, humans are demonstrably capable of encountering Eliza programs without thinking "wow, how incomprehensibly alien".

Mind you, we're mistaken: Eliza programs are incomprehensibly alien, we haven't the first clue what it feels like to be one, supposing it even feels like anything at all. But that doesn't stop us from thinking otherwise.

but, "aha, autism spectrum disorder".

Sure, that's one thing we might think instead. Agreed.

we're assuming a hypothetical axis of (un)clippiness whose (dis)valuable nature is supposedly orthogonal to the pleasure-pain axis. But what grounds have we for believing such a qualia-space could exist?

(shrug) I'm content to start off by saying that any "axis of (dis)value," whatever that is, which is capable of motivating behavior is "non-orthogonal," whatever that means in this context, to "the pleasure-pain axis," whatever that is.

Before going much further, though, I'd want some confidence that we were able to identify an observed system as being (or at least being reliably related to) an axis of (dis)value and able to determine, upon encountering such a thing, whether it (or the axis to which it was related) was orthogonal to the pleasure-pain axis or not.

I don't currently have any grounds for such confidence, and I doubt anyone else does either. If you think you do, I'd like to understand how you would go about making such determinations about an observed system.

Shminux, I wonder if we may understand "understand" differently. Thus when I say I want to understand what it's like to be a bat, I'm not talking merely about modelling and predicting their behaviour. Rather I want first-person knowledge of echolocatory qualia-space. Apaarently, we can know all the third-person facts and be none the wiser.

The nature of psychopathic cognition raises difficult issues. There is no technical reason why we couldn't be designed like mirror-touch synaesthetes (cf. http://www.daysyn.com/Banissy_Wardpublished.pdf) impartially feeling carbon-copies of each other's encephalised pains and pleasures - and ultimately much else besides - as though they were our own. Likewise, there is no technical reason why our world-simulations must be egocentric. Why can't the world-simulations we instantiate capture the impartial "view from nowhere" disclosed by the scientific world-picture? Alas on both counts accurate and impartial knowledge would put an organism at a disadvantage. Hyper-empathetic mirror-touch synaesthetes are rare. Each of us finds himself or herself apparently at the centre of the universe. Our "mind-reading" is fitful, biased and erratic. Naively, the world being centred on me seems to be a feature of reality itself. Egocentricity is a hugely fitness-enhancing adaptation. Indeed, the challenge for evolutionary psychology is to explain why aren't we all psychopaths, cheats and confidence trickers all the time...

So in answer to your point, yes. a psychopath can often model and predict the behaviour other sentient beings better than the subjects themselves. This is one reason why humans can build slaughterhouses and death camps. [Ccompare death-camp commandant Franz Stangl's response in Gitta Sereny's Into That Darkness to seeing cattle on the way to be slaughtered: http://www.jewishvirtuallibrary.org/jsource/biography/Stangl.html] As you rightly note too, a psychopath can also know his victims suffer. He's not ignorant of their sentience like Descartes, who supposed vivisected dogs were mere insentient automata emitting distress vocalisations. So I agree with you on this score as well. But the psychopath is still in the grip of a hard-wired egocentric illusion - as indeed are virtually all of us, to a greater or less degree. By contrast, if the psychopath were to acquire the rich empathetic understanding of a generalised mirror-touch syarnesthete, i.e. if he had the cognitive capacity to represent the first-person perspective of another subject of experience as though it were literally his own, then he couldn't wantonly harm another subject of experience: it would be like harming himself. Mirror-touch synaesthetes can't run slaughterhouses or death camps. This is why I take seriously the prospect that posthuman superintelligence will practise some sort of high-tech Jainism. Credible or otherwise, we may presume posthuman superintelligence won't entertain the false notions of personal identity adaptive for Darwinian life.

[sorry shminux, I know our conceptual schemes are rather different, so please don't feel obliged to respond if you think I still don't "get it". Life is short...]

I can't see why DP's definition of understanding needs more defence than yours. You are largely disagreeing about the meaning of this word, and I personally find the inclusion of empathy in understanding quite intuitive.

No other definition [of "understanding"] I could find (not even Kant's pure understanding) implies empathy

"She is a very understanding person, she really empathises when you explain a problem to her".

"one is able to think about it and use concepts to deal adequately with that object."

In other words, you can model their behavior accurately.

I don't think that is an uncontentious translation. Most of the forms of modelling we are familiar with don't seem to involve concepts.

I think others have addressed EY;s arguments. Sometimes centuries before he made them.