Disagreeing with what you think of the system is the same as not intuiting it.

Missing a 'not' I think.

Yes, that's the one I was referring to.

I can't tell which way your sarcasm was supposed to cut.

The obvious interpretation is that you think rationality is somehow hindered by paying attention to form rather than substance, and the "exemplary rationality" was intended to be mocking.

But your comment being referenced was an argument that form has something very relevant to say about substance, so it could also be that you were actually praising gwern for practicing what you preach.

His early actions on LW were positive and some were voted up.

His 'early' actions on LW were recent and largely negative, and one was voted up significantly (though I don't see why - I voted that comment down).

At his best he's been abrasive, confrontational, and rambling. Not someone worth engaging.

Yes. I wonder if there's a good explanation why narrow AI folks are so much more sensible than AGI folks on those subjects.

Yes, I'd say so. It isn't helpful here to say that a system lacks a theory of mind if it has a mechanism that allows it to make predictions about reported beliefs, intentions, etc.

Either people have fundamentally different, irreconcilable, values or they don't. If they do, then the argument I made is valid. If they don't, then CEV(any random person) will give exactly the same result as CEV(humanity).

This is a false dilemma. If people have some values that are the same or reconcilable, then you will get different output from CEV(any random person) and CEV(humanity).

And note that an actual move by virtue ethicists is to exclude sociopaths from "humanity".

Yes, that's possible. It's still possible that you could get a lot done with strategy #2 without being able to make that prediction.

I agree that if 2 systems have the same inputs and outputs, their internals don't matter much here.

There is no reason to think that this is because they are less intelligent or less well-informed than me, as opposed to actually having different preferences.

There are lots of reasons to think so. For example, they might want the death of your friends because they mistakenly believe that a deity exists.

What is it that we would actually be disagreeing about, other than what English phrase to use to describe the system's underlying model(s)?

We would be disagreeing about the form of the system's underlying models.

2 different strategies to consider:

1. I know that Steve believes that red blinking lights before 9 AM are a message from God that he has not been doing enough charity, so I can predict that he will give more money to charity if I show him a blinking light before 9 AM.

2. Steve seeing a red blinking light before 9 AM has historically resulted in a 20% increase of charitable donation for that day, so I can predict that he will give more money to charity if I show him a blinking light before 9 AM.

You can model humans with or without referring to their mental states. Both kinds of models are useful, depending on circumstance.