You're right about the ignorance part of privilege-- and contrary to SJW, it's quite possible for people in the less privileged categories to be ignorant about at least some of the problems of people in the more privileged categories.

I'd love to find a way to disentangle the ignorance part of the idea of privilege from the power grab, but I haven't figured out how to do it.

I think that "privilege" (in its more reasonable forms) basically refers to a special case of the Typical Mind Fallacy, one where people are prone to dismissing or understating the problems of one group because they don't personally experience them in the same way. For a relatively neutral example, there's this bit in Yvain's post:

I can't deal with noise. If someone's being loud, I can't sleep, I can't study, I can't concentrate, I can't do anything except bang my head against the wall and hope they stop. I once had a noisy housemate. Whenever I asked her to keep it down, she told me I was being oversensitive and should just mellow out. I can't claim total victory here, because she was very neat and kept yelling at me for leaving things out of place, and I told her she needed to just mellow out and you couldn't even tell that there was dust on that dresser anyway. It didn't occur to me then that neatness to her might be as necessary and uncompromisable as quiet was to me, and that this was an actual feature of how our minds processed information rather than just some weird quirk on her part.

I would say that these are pretty much perfect examples of privilege: situations in which the perfectly reasonable problems of one party are completely invisible to the other, to the point that the other cannot even see what the problem is and thinks that the other person is just complaining about nothing.

Similarly, Eliezer has explicitly used the term "metabolic privilege" in pretty much this sense:

The metabolically privileged don't believe in metabolic privilege, since they are able to lose weight by trying! harder! to diet and exercise, and the diet and exercise actually work the way they're supposed to…

So "privilege" is a useful concept, one which has actually already seen use in the LW community. In this context, "check your privilege" is a call to re-evaluate one's assumptions and to take into account the factors which make the situation genuinely problematic for others but a non-problem for you.

Even the "privilege means you're not allowed to have any opinion other than the social justice consensus" sense can be a somewhat reasonable one - there are plausibly positions where people frequently and commonly become guilty of the Typical Mind Fallacy, and where a consensus of the people who've given the issue some thought agrees on this, and people who disagree are likely to just be flat-out wrong. (You could say that it's the SJW version of "read the Sequences".)

A classic SJW example of privilege that I think is justified is the case of sexual harassment of women, where men frequently react to cases of harassment with variations of "I don't see the problem here, if someone did that to me I'd just be flattered". In that case, the fallacy involves an inability to take into account the fact that a behavior that one might consider flattering if it only happened rarely will become unbearable if repeated sufficiently often (obligatory link), and also that men being stronger women creates a sense of accompanying danger that wouldn't be present in the case of women harassing men.

I thought Of Dogs and Lizards was also a nice illustration of these concepts:

This is where things get a bit tricky to understand. Because most examples of social privilege aren’t that straightforward. Let’s take, for example, a basic bit of male privilege:

A man has the privilege of walking past a group of strange women without worrying about being catcalled, or leered at, or having sexual suggestions tossed at him.

A pretty common male response to this point is “that’s a privilege? I would love if a group of women did that to me.”

And that response, right there, is a perfect shining example of male privilege.

To explain how and why, I am going to throw a lengthy metaphor at you. In fact, it may even qualify as parable. Bear with me, because if it makes everything crystal clear, it will be worth the time.

Imagine, if you will, a small house, built someplace cool-ish but not cold, perhaps somewhere in Ohio, and inhabited by a dog and a lizard. The dog is a big dog, something shaggy and nordic, like a Husky or Lapphund – a sled dog, built for the snow. The lizard is small, a little gecko best adapted to living in a muggy rainforest somewhere. Neither have ever lived anywhere else, nor met any other creature; for the purposes of this exercise, this small house is the entirety of their universe.

The dog, much as you might expect, turns on the air conditioning. Really cranks it up, all the time – this dog was bred for hunting moose on the tundra, even the winter here in Ohio is a little warm for his taste. If he can get the house to fifty (that’s ten C, for all you weirdo metric users out there), he’s almost happy.

The gecko can’t do much to control the temperature – she’s got tiny little fingers, she can’t really work the thermostat or turn the dials on the A/C. Sometimes, when there’s an incandescent light nearby, she can curl up near it and pick up some heat that way, but for the most part, most of the time, she just has to live with what the dog chooses. This is, of course, much too cold for her – she’s a gecko. Not only does she have no fur, she’s cold-blooded! The temperature makes her sluggish and sick, and it permeates her entire universe. Maybe here and there she can find small spaces of warmth, but if she ever wants to actually do anything, to eat or watch TV or talk to the dog, she has to move through the cold house.

Now, remember, she’s never known anything else. This is just how the world is – cold and painful and unhealthy for her, even dangerous, and she copes as she knows how. But maybe some small part of her thinks, “hey, it shouldn’t be like this,” some tiny growing seed of rebellion that says who she is right next to a lamp is who she should be all the time. And she and the dog are partners, in a sense, right? They live in this house together, they affect each other, all they’ve got is each other. So one day, she sees the dog messing with the A/C again, and she says, “hey. Dog. Listen, it makes me really cold when you do that.”

The dog kind of looks at her, and shrugs, and keeps turning the dial.

This is not because the dog is a jerk.

This is because the dog has no fucking clue what the lizard even just said.

Consider: he’s a nordic dog in a temperate climate. The word “cold” is completely meaningless to him. He’s never been cold in his entire life. He lives in an environment that is perfectly suited to him, completely aligned with his comfort level, a world he grew up with the tools to survive and control, built right in to the way he was born.

So the lizard tries to explain it to him. She says, “well, hey, how would you like it if I turned the temperature down on you?”

The dog goes, “uh… sounds good to me.”

What she really means, of course, is “how would you like it if I made you cold.” But she can’t make him cold. She doesn’t have the tools, or the power, their shared world is not built in a way that allows it – she simply is not physically capable of doing the same harm to him that he’s doing to her. She could make him feel pain, probably, I’m sure she could stab him with a toothpick or put something nasty in his food or something, but this specific form of pain, he will never, ever understand – it’s not something that can be inflicted on him, given the nature of the world they live in and the way it’s slanted in his favor in this instance. So he doesn’t get what she’s saying to him, and keeps hurting her.

Most privilege is like this.

A straight cisgendered male American, because of who he is and the culture he lives in, does not and cannot feel the stress, creepiness, and outright threat behind a catcall the way a woman can. His upbringing has given him fur and paws big enough to turn the dials and plopped him down in temperate Ohio. When she says “you don’t have to put up with being leered at,” what she means is, “you don’t ever have to be wary of sexual interest.” That’s male privilege. Not so much that something doesn’t happen to men, but that it will never carry the same weight, even if it does.

So what does this mean? And what are we asking you to do, when we say “check your privilege” or “your privilege is showing”?

Well, quite simply, we want you to understand when you have fur. And, by extension, when that means you should listen. See, the dog’s not an asshole just for turning down the temperature. As far as he knows, that’s fine, right? He genuinely cannot feel the pain it causes, he doesn’t even know about it. No one thinks he’s a bad person for totally accidentally doing harm.

Here’s where he becomes an asshole: the minute the gecko says, “look, you’re hurting me,” and he says, “what? No, I’m not. This ‘cold’ stuff doesn’t even exist, I should know, I’ve never felt it. You’re imagining it. It’s not there. It’s fine because of fur, because of paws, because look, you can curl up around this lamp, because sometimes my water dish is too tepid and I just shut up and cope, obviously temperature isn’t this big deal you make it, and you’ve never had to deal with mange anyway, my life is just as hard.”

And then the dog just ignores it. Because he can. That’s the privilege that comes with having fur, with being a dog in Ohio. He doesn’t have to think about it. He doesn’t have to live daily with the cold. He has no idea what he’s talking about, and he will never, ever be forced to learn. He can keep making the lizard miserable until the day they both die, and he will never suffer for it beyond the mild annoyance of her complaining. And she, meanwhile, gets to try not to freeze to death.

You overlap through at least one of the following categories:heterosexual, male, white, high socioeconomic bracket

An interesting set. So let's see who doesn't overlap at least one category -- it got to be a lesbian (or at least bi) poor non-white woman.

So everybody who is not a lesbian poor non-white woman (which I would estimate to be 98-99% of the population) is vulnerable to the cry of Check Your Privilege! Interesting...

I haven't reached any conclusions about him, I've just got these hypotheses, these plausible explanations to what's in front of me, that are based on previous experiences with other people, and that give me enough pause not to attempt to tackle this discussion with someone I don't know well.

"While in principle I'm glad I have the freedom to have sex with a man (and would prefer if e.g. it became illegal to fire me for doing so), I don't think I'd ever take advantage of that freedom."

Thing is, I'm pretty sure I would... as soon as I've finished cutting all ties with my ancestral land so that the consequences wouldn't come to bite me in some roundabout way. But I might not be that prudent or patient. Give me liberty and give me cock(adooloo)! :P

Now, for the misunderstanding:

"and you immediately pattern-matched me as a racist and decided to ignore me. "

Not quite. I just decided the chance was high enough that I wasn't willing to risk my time and effort. Not quite the same thing as pegging you as guilty until proven otherwise and then dismissing you outright.

"What would you think about someone who responded the same way to the suggestion that god didn't exist, or that astrology didn't work, or that the earth was six billion years old?"

That they've been faced with people promoting these ideas who were both smug and unconvincing, engaged them in good faith, and got burned for it, learning nothing and feeling hurt and insulted. I would blame those people who ruined the audience for me, like a salesman finding a hostile populace in the wake of Howard Hill.

And I've met atheists who were jerks (Atheism Warriors?), back when I was religious, in societies where they were powerful enough that they could take their stance for granted. If it weren't for my confidence in my own capacity to counter their arguments (my actual belief in the dragon in my garage), and my endless reserves of youthful energy, Same for Trotskyst and Anarchist jerks ("Engineers are the most alienated of all!"), libertarian jerks ("Poverty? What's poverty? Having less cable channels than the neighbors?"), and so on and so forth.

At my age, please forgive me for being very cautious about engaging anyone who even looks like a Warrior of any sort in debate. If we get to know each other later and I find that I can trust you not to be a Warrior, I'll discuss this at length with you... by PM (because, frankly, our reputation is bad enough as is).