That's true.

I wonder whether professional philosophers have made any progress with this kind of an approach? At least in retrospect it feels rather obvious, but I don't recall hearing anyone mention something like this before.

Yes, that's the idea.

In that case, I struggle to see why the "defeater critique" wouldn't seriously undermine practice (1) in most cases. Philosophers can't simply assume intuited contents p and then move from p to q. We want to know how likely p is to be true, and if our primary reason for thinking p is true is some unreliable cognitive algorithm (rather than, say, hard scientific data or a mathematical proof), then we are left without much reason to be confident that p is true.

Suppose a theist says he knows by Holy Spirit Communication (HSC) that Jesus is magic. An atheist replies, "HSC is not a reliable method. See all this experimental data on people making judgments based on the deliverances of (what they claim is) HSC." The theist then says, "No, I'm not arguing from the HSC mental state to the conclusion that Jesus is magic. I'm arguing from the HSC contents (that is, from proposition p) to the conclusion that Jesus is magic."

The atheist would be unimpressed, and correctly so.

A quick review (for the benefit of others): Bugmaster asked: "Is there really any kind of a serious debate in modern philosophy circles regarding whether or not our personal intuitions can be generally trusted?" I replied: "Yes... the debate over intuitions is one of the hottest in philosophy today." But Richard is right to say that most of the philosophical debate about intuitions "isn't about whether intuitions are reliable, but rather over whether the critics have accurately specified the role they play in traditional philosophical methodology." So, I apologize for my sloppy wording.

Now, a few words on intuitionist methodology. When I read the defenders of intuitionist methodology, I'm reminded of something John Doris said in my interview with him (slight paraphrase for clarity and succinctness; see the exact quote at the bottom of the transcript):

If experimentalists say that something is a mistake, then no one will admit they said it. And I'm no different: "Oh, is that false? Then I disagree. That's not what I really meant."

When experimentalists pointed out that our brains don't store concepts as necessary and sufficient conditions, many philosophers rushed to say that philosophers had never been assuming this in the first place. But clearly, many philosophers were making such false assumptions about how concepts worked, since the "classical" view of concepts — concepts as mental representations captured by necessary and sufficient conditions — held sway for quite some time, even after Wittgenstein (1953). (For a review, see Murphy 2004.)

Or, given that experimentalists have raised worries about using intuitions as evidence in general, Ichikawa (forthcoming) now rushes to say that philosophers generally don't rely on intuitions in a "central" way. (To narrow our discussion, I'll focus on this, the first article you sent me.) What does Ichikawa mean by this? He distinguishes three metaphilosophical claims:

1. Intuited contents are (often) taken as important evidence/reasons/data/input in armchair philosophy.
2. Intuited contents are (often) taken as important evidence/reasons/data/input in armchair philosophy because they are intuited.
3. Intuition states, or facts about intuition states, are (often) taken as important evidence/reasons/data/input in armchair philosophy.

As far as I can tell, Ichikawa wants to argue that (1) represents philosophical practice better than (2) or (3), and that (1) is not particularly undermined by experimentalist critiques. Have I got that right? (I'll hold my reply until I hear whether you agree with my interpretation. I found Ichikawa to be unclear on this, though not as unclear as Yudkowsky or Muehlhauser often are in their philosophical writings.)

So does that mean this:

I think Richard is saying that what is "right" is rigidly determined by my current (idealised) desires - so in a possible world where I desired to murder, murder would still be wrong

...is your real claim here, independent of any points about language use?

If so, I think I would just straightforwardly modify my paragraph above to say that my statements are not trying to talk about language use or human brains / desires, albeit that desire is both an optimization target of, and a quotation of, morality.

Multiple philosophers have suggested that this stance seems similar to "rigid designation", i.e., when I say 'fair' it intrinsically, rigidly refers to something-to-do-with-equal-division. I confess I don't see it that way myself - if somebody thinks of Euclidean geometry when you utter the sound "num-berz" they're not doing anything false, they're associating the sound to a different logical thingy. It's not about words with intrinsically rigid referential power, it's that the words are window dressing on the underlying entities.

I just wanted to agree with Tristanhaze here that this usage strikes me as non-standard. I want to put this in my own words so that Tristanhaze/Eliezer/others can correct me if I've got the wrong end of the stick.

If something is a rigid designator it means that it refers to the same thing in all possible worlds. To say it's non-rigid is to say it refers to different things in some possible worlds to others. This has nothing to do with whether different language users that use the phrase must always be referring to the same thing. So George Washington may be a rigid designator in that it refers to the same person in all possible worlds (bracketing issues of transworld identity) but that doesn't mean that in all possible worlds that person is called George Washington or that in all possible worlds people who use the name George Washington must be referring to this person or even that in the actual world all people who use the name George Washington must be referring to this person.

To say "water" is a rigid designator is to say that whatever possible world I am talking about, I am picking out the same thing when I use the word water (in a way that I wouldn't be when I say, "the tallest person in the world" - this would pick out different things in different worlds). But it doesn't say anything about whether I mean the same thing as other language users in this or other possible worlds.

ETA: So the relevance to the quoted section is that rigid designators aren't about whether someone that thinks of Euclidean geometry when you say "numbers" is right or wrong - it's about whether whatever they associate with that word is the same thing in all possible worlds (or whether it's a different thing in some worlds).

ETA 2: I take it that Eliezer's paragraph here is in response to comments like these. I'm in a bit of a rush and need to think about it some more but I think Richard may be making a different point here to the one Eliezer's making (on my reading). I think Richard is saying that what is "right" is rigidly determined by my current (idealised) desires - so in a possible world where I desired to murder, murder would still be wrong because "right" is a rigid designator (that is, right from the perspective of my language, a different language user - like the me that desires murder - might still use "right" to refer to something else according to which murder is right. See the point about George Washington being able to be rigid even if people in other possible worlds use that name to name someone else). On the other hand, my reading of Eliezer was that he was taking the claim that "right" (or "fair") is a rigid designator to mean something about the way different language users use the word "fair". Eliezer seemed to be suggesting that rigid designation implied that words intrinsically mean certain things and hence that rigid designation implies that if someone uses a word in a different way they are wrong (using numbers to refer to geometry). I could have misunderstood either of these two comments but if I haven't then it seems to me that Eliezer is using rigid designator in a non-standard way.

Can you cite a specific paper on book chapter which makes the kind of argument you're suggesting here?

Is there really any kind of a serious debate in modern philosophy circles regarding whether or not our personal intuitions can be generally trusted?

Yes! The Experimental Philosophy: An Introduction book I linked to is a very brief, up-to-date summary of that debate. The debate over intuitions is one of the hottest in philosophy today, and has been since about 1998.

“I toss in a pebble whenever a sheep passes,” I point out.

“When a sheep passes, you toss in a pebble?” Mark says. “What does that have to do with anything?”

“It’s an interaction between the sheep and the pebbles,” I reply.

“No, it’s an interaction between the pebbles and you,” Mark says. “The magic doesn’t come from the sheep, it comes from you. Mere sheep are obviously nonmagical. The magic has to come from somewhere, on the way to the bucket.”

I point at a wooden mechanism perched on the gate. “Do you see that flap of cloth hanging down from that wooden contraption? We’re still fiddling with that – it doesn’t work reliably – but when sheep pass through, they disturb the cloth. When the cloth moves aside, a pebble drops out of a reservoir and falls into the bucket. That way, Autrey and I won’t have to toss in the pebbles ourselves.”

Mark furrows his brow. “I don’t quite follow you… is the cloth magical?”

I shrug. “I ordered it online from a company called Natural Selections. The fabric is called Sensory Modality.” I pause, seeing the incredulous expressions of Mark and Autrey. “I admit the names are a bit New Agey. The point is that a passing sheep triggers a chain of cause and effect that ends with a pebble in the bucket."

(From "The Simple Truth", a parable about using pebbles in a bucket to keep count of the sheep in a pasture.)

“My pebbles represent the sheep!” Autrey says triumphantly. “Your pebbles don’t have the representativeness property, so they won’t work. They are empty of meaning. Just look at them. There’s no aura of semantic content; they are merely pebbles. You need a bucket with special causal powers.”

“Ah!” Mark says. “Special causal powers, instead of magic.”

“Exactly,” says Autrey. “I’m not superstitious. Postulating magic, in this day and age, would be unacceptable to the international shepherding community. We have found that postulating magic simply doesn’t work as an explanation for shepherding phenomena. So when I see something I don’t understand, and I want to explain it using a model with no internal detail that makes no predictions even in retrospect, I postulate special causal powers. If that doesn’t work, I’ll move on to calling it an emergent phenomenon.”

“What kind of special powers does the bucket have?” asks Mark.

“Hm,” says Autrey. “Maybe this bucket is imbued with an about-ness relation to the pastures. That would explain why it worked – when the bucket is empty, it means the pastures are empty.”

“Where did you find this bucket?” says Mark. “And how did you realize it had an about-ness relation to the pastures?”

“It’s an ordinary bucket,” I say. “I used to climb trees with it… I don’t think this question needs to be difficult.”

“I’m talking to Autrey,” says Mark.

“You have to bind the bucket to the pastures, and the pebbles to the sheep, using a magical ritual – pardon me, an emergent process with special causal powers – that my master discovered,” Autrey explains.

Autrey then attempts to describe the ritual, with Mark nodding along in sage comprehension.

“And this ritual,” says Mark, “it binds the pebbles to the sheep by the magical laws of Sympathy and Contagion, like a voodoo doll.”

Autrey winces and looks around. “Please! Don’t call it Sympathy and Contagion. We shepherds are an anti-superstitious folk. Use the word ‘intentionality’, or something like that.”

“Can I look at a pebble?” says Mark.

“Sure,” I say. I take one of the pebbles out of the bucket, and toss it to Mark. Then I reach to the ground, pick up another pebble, and drop it into the bucket.

Autrey looks at me, puzzled. “Didn’t you just mess it up?”

I shrug. “I don’t think so. We’ll know I messed it up if there’s a dead sheep next morning, or if we search for a few hours and don’t find any sheep.”

“But -” Autrey says.

“I taught you everything you know, but I haven’t taught you everything I know,” I say.

Mark is examining the pebble, staring at it intently. He holds his hand over the pebble and mutters a few words, then shakes his head. “I don’t sense any magical power,” he says. “Pardon me. I don’t sense any intentionality.”

“A pebble only has intentionality if it’s inside a ma- an emergent bucket,” says Autrey. “Otherwise it’s just a mere pebble.”

“Not a problem,” I say. I take a pebble out of the bucket, and toss it away. Then I walk over to where Mark stands, tap his hand holding a pebble, and say: “I declare this hand to be part of the magic bucket!” Then I resume my post at the gates.

Autrey laughs. “Now you’re just being gratuitously evil.”

I nod, for this is indeed the case.

“Is that really going to work, though?” says Autrey.

I nod again, hoping that I’m right. I’ve done this before with two buckets, and in principle, there should be no difference between Mark’s hand and a bucket. Even if Mark’s hand is imbued with the elan vital that distinguishes live matter from dead matter, the trick should work as well as if Mark were a marble statue.

(The moral: In this sequence, I explained how words come to 'mean' things in a lawful, causal, mathematical universe with no mystical subterritory. If you think meaning has a special power and special nature beyond that, then (a) it seems to me that there is nothing left to explain and hence no motivation for the theory, and (b) I should like you to say what this extra nature is, exactly, and how you know about it - your lips moving in this, our causal and lawful universe, the while.)

No, I'm saying that you could hold all of the physical facts fixed and my cat might still not be fuzzy. This is somewhat absurd, but I have a tremendously good imagination; if I can imagine zombies, I can imagine fuzz-zombies.