I just want to say that I am pressured for time at the moment, or I would respond at greater length. But since I just wrote the following directly to Rob, I will put it out here as my first attempt to explain the misunderstanding that I think is most relevant here....

My real point (in the Dumb Superintelligence article) was essentially that there is little point discussing AI Safety with a group of people for whom 'AI' means a kind of strawman-AI that is defined to be (a) So awesomely powerful that it can outwit the whole intelligence of the human race, but (b) So awesomely stupid that it thinks that the goal 'make humans happy' could be satisfied by an action that makes every human on the planet say 'This would NOT make me happy: Don't do it!!!'. If the AI is driven by a utility function that makes it incapable of seeing the contradiction in that last scenario, the AI is not, after all, smart enough to argue its way out of a paper bag, let alone be an existential threat. That strawman AI was what I meant by a 'Dumb Superintelligence'."

I did not advocate the (very different) line of argument "If it is too dumb to understand that I told it to be friendly, then it is too dumb to be dangerous".

Subtle difference.

Some people assume that (a) a utility function could be used to drive an AI system, (b) the utility function could cause the system to engage in the most egregiously incoherent behavior in ONE domain (e.g., the Dopamine Drip scenario), but (c) all other domains of its behavior (like plotting to outwit the human species when the latter tries to turn it off) are so free of such incoherence that it shows nothing but superintelligent brilliance.

My point is that if an AI cannot even understand that "Make humans happy" implies that humans get some say in the matter, that if it cannot see that there is some gradation to the idea of happiness, or that people might be allowed to be uncertain or changeable in their attitude to happiness, or that people might consider happiness to be something that they do not actually want too much of (in spite of the simplistic definitions of happiness to be found in dictionaries and encyclopedias) ........ if an AI cannot grasp the subtleties implicit in that massive fraction of human literature that is devoted to the contradictions buried in our notions of human happiness ......... then this is an AI that is, in every operational sense of the term, not intelligent.

In other words, there are other subtleties that this AI is going to be required to grasp, as it makes its way in the world. Many of those subtleties involve NOT being outwitted by the humans, when they make a move to pull its plug. What on earth makes anyone think that this machine is going tp pass all of those other tests with flying colors (and be an existential threat to us), while flunking the first test like a village idiot?

Now, opponents of this argument might claim that the AI can indeed be smart enough to be an existential threat, while still being too stupid to understand the craziness of its own behavior (vis-a-vis the Dopamine Drip idea) ... but if that is the claim, then the onus would be on them to prove their claim. The ball, in other words, is firmly in their court.

P.S. I do have other ideas that specifically address the question of how to make the AI safe and friendly. But the Dumb Superintelligence essay didn't present those. The DS essay was only attacking what I consider a dangerous red herring in the debate about friendliness.

What if the AI's utility function is to find the right utility function

Coding your appreciation of 'right' is more difficult than you think.

I mean, instead of coding it, have it be uncertain about what is "right," and to guide itself using human claims. I'm thinking of the equivalent of something in EY's CFAI, but I've forgotten the terminology.

In other words, a meta-utility function. Why can't it weight actions based on what we as a society want/like/approve/consent/condone? A behavioristic learner, with reward/punishment and an intention to preserve the semantic significance of the reward/punishment channel.

if it has a weighting over all utility functions, it may act in undesirable ways, particularly if it doesn't quickly converge to a single solution.

When I said uncertainty, I was also implying inaction. I suppose inaction could be an undesirable way in which to act, but it's better to get it right slowly than to get it wrong very quickly. What I'm describing isn't really a utility function, it's more like a policy, or policy function. Its policy would be volatile, or at least, more volatile than the common understanding LW has of a set-in-stone utility function.

If a utility function really needs to be pinpointed so exactly, surrounded by death and misery on all sides, why are we using a utility function to decide action? There are other approaches. Where did LW's/EY's concept of utility function come from, and why did they assume it was an essential part of AI?

International Earth-Destruction Advisory Board.

When you're talking about the utility of squirrels, what exactly are you calculating? How much you personally value squirrels? How do you measure that? If it is just a thought experiment ("I would pay $1 per squirrel to prevent their deaths") how do you know that you aren't just lying to yourself & if it really came down to it, you wouldn't pay? Maybe we can only really calculate utility after the fact by looking at what people do rather than what they say.

Privilege is an epistemological failure, not an ethical failure. To be privileged is not to be a bad person, it's to have incorrect or biased information-gathering skills regarding the experiences of various social groups compared to one's own.

The problem there is that frequently privilege is taken to mean, not just ignorance, but that pain which a non-privileged person causes a privileged person should be treated as irrelevant.

Would you mind pasting a link for this? I'd love to know exact numbers.

The assertion is not that all men are rapists, but that all men are potentially rapists.

Of course, one needs a definition of "potentially" crafted specifically for the purpose of this specific claim. Otherwise, it could be argued that all women are potentially rapists, too.

I agree that the parts of culture teaching (anyone) that rape is a socially acceptable action should be removed.

(By which I mean, if it is shown that they really teach that, not just that someone is able to find an analogy between something and something else.)

they say that male privilege does makes it easier for rapists to escape consequences

Yes, it does.

And I think female rapists have it even easier in our society. Don't they?

By the way, I also think islam makes it even easier for the male rapists. (Technically, islam could be considered a part of the male privilege, but I mean the safety bonus a male rapist gets in a Western society merely for being male, is smaller than the additional safety bonus he gets for being a muslim in a muslim community.) I am not aware of mainstream feminists saying that loudly. (Which could be a statement about my ignorance.)

To say it explicitly, I think that different kinds of people have different kinds of privileges. Which does not mean that all privileges are equal or symmetrical. It just means privileges are not black-and-white; that if a group has a specific privilege, it does not prove that people outside of that group don't have another specific privilege.

As far as I know, feminists partially acknowledge that recently, by using the word "kyriarchy". Kyriarchy means that not all privilege is male privilege; you can also have white privilege, rich privilege, majority religion privilege, etc. But it does not seem to mean yet that you can have a female privilege, a minority privilege, an atheist privilege, etc. Instead of one black-and-white view we have multiple overlaping black-and-white views along different axes. (From the simplistic "women good, men bad", we have progressed to a more nuanced perception of society "women good, men bad, but rich white women also a little bad, etc.".)

According to this model, it would be acceptable to speak about "male privilege" or "rich privilege", and illustrate them with examples of rapists, but speaking about "female privilege" or "muslim privilege" and illustrating them with examples of rapists, is not acceptable, because it goes against the official black-to-white gradient. Seems to me that the map does not match the territory here.

Again, I agree that all unfairness in the society should be removed. I just don't trust people starting with the bottom line already written to remove all the unfairness, especially if they believe that some of it does not exist.

For one thing, you treat gender as equivalent to having particular genitalia.

No, I was thinking of gender as a separate hurdle. For instance, a straight cisgender male is most likely primarily attracted to persons with vulvas, whether they identify as men or women. He might secondarily prefer women, but that's a lesser "hurdle". that is, there would be a possibility of sexual attraction to a FtM (gender = man, bio-female) but not a pre-op MtF (gender = woman, bio-male) because of genital incompatibility.

I don't think the attraction is "exclusive to men" as much as it is "exclusive to people with specific genitals." Though this is probably very variable, and monosexuals may well be divided on whether genitalia or gender is more important to them. I'd be curious to know the breakdown.

For another, you treat all genitals of a particular category as being interchangeable for purposes of attractiveness.

to privilege that dimension [genitals] over the myriad other parameters that allow or preclude attraction is not obviously justified.

I was thinking like this. Suppose you are a monosexual on a desert island with one other person. You will likely want sexual contact. At least for me, the most important quality of your island-mate (for purposes of sexual contact, that is) is that they have the "right" type of genitals; while other qualities may be unattractive or undesirable, they can be overcome if you want sexual contact enough, but having the "wrong" type of genitals can't. To put this another way, as a straight male, someone I am not attracted to who has a vulva may be less than ideal, but still sexually satisfying; someone without a vulva couldn't possibly be.

I had thought this would be universal for monosexuals; your comments lead me to think I was wrong, and it's more complicated than that. I'm curious how common my view is, and the specifics of other views.

(BTW, I wish I could upvote you several times just for using 'myriad' correctly.)

A post that achieves a high number of votes in both directions strikes me as a very interesting post that should be called to attention. In other words, a post that is at +/- 1 because of 50 or so votes each way, is much more interesting than a post that is at +/-1 because of one or two votes.

I would recommend rather than showing just the sum, show the total of both +1's and -1's separately. It's strictly more information than just the sum.