Perhaps not the specific intention, but the concept of what it means to have an intention

Funny, I would've phrased that the other way around.

That no superintelligence could be dumb enough to misinterpret what we mean is the particular belief in question

I didn't come up with that myself, I got it from MugaSofer: 'Well, since the whole genie thing is a metaphor for superintelligence, "this genie is trying to be Friendly but it's too dumb to model you well" doesn't really come up.'

Under reasonable definitions of "superintelligence" it does follow that a superintelligence must know what you mean, but if you pick some other definition and state so outright, I won't argue with it. (It is, however, still subject to "talk to the intelligence to figure out what it's going to do".)

Of course if you are a computer scientist they will say, "But Jiro is not an elite computer scientist!", and if you were an elite computer scientist they would say, "Elite computer scientists don't currently take the issue seriously enough to think about it properly...

I think you're making my case for me.

PS: If you want to reply please post a new reply to the root message since I can't afford the karma hits to respond to you.

That no superintelligence could be dumb enough to misinterpret what we mean is the particular belief in question

Some off-the-cuff thoughts on why "a superintelligence dumb enough to misinterpret what we mean" may be a contradiction in terms, given the usual meaning of superintelligence:

Intelligence is near-synonymous with "able to build accurate models and to update those models accurately", with 'higher intelligence' denoting a combination of "faster model-building / updating" and/or "less prone to systematic / random errors".

'Super' as a qualifier is usually applied on both dimensions, i.e. "faster and more accurately". While this seems more like a change in degree (one intelligence hypothesis, a devoted immortal fool with an endless supply of paper and pencils could simulate the world), it also often is a change in kind, since in practice there always are resource-constraints (unless Multivax reverses entropy), often relevant enough to bar a slower-modeling agent from achieving its goals within the given constraints.

"Able to build accurate models and to update those models accurately", then, proportionally increases "powerful, probably able to pursue its goals effectively, conditional on those goals being related to the accurate models".

Given a high degree of the former, by definition it is not exactly very hard to acquire and emulate the shared background on which inter-human understanding is built. For an AI, understanding humans would be relevant near-regardless of its actual goals; accurate models of humans as the sine-qua-non for e.g. breaking out of the AI box. Being able to build such models quickly and accurately is what classifies the agent as "superintelligent" in the first place! If there was no incentive for the agent to model humans at all, why would there be interactions with humans, such as the human asking the agent to "rescue grandma from the burning building"? The agent, when encountering rocks and precious minerals, will probably seek models reflecting a deep understanding of those. It will do the same when encountering humans.

See, I'm d'accord with statements such as "less intelligent agents would be expected to misinterpret what we mean", but a superintelligent agent -- i.e. an agent good at building accurate models --, should by its definition by able to understand human-level intentions. If it does not, then in that respect, I wouldn't call it a superintelligent agent.

In addition, I'd question who'd call a domain-limited expert system which is great with models only on some small subject-spectrum, but evidently abysmal with building models relevant to its goals in other respects, a "superintelligent agent", with its connotations of general intelligence. Does the expression "a superintelligent chessbot" make sense? Or saying "x is a superintelligent human, except for doing fractions, which he absolutely cannot do"?

Before you label me an idiot who'd expect the AI to fall in love with a human princess on top of the Empire State building, allow me to stress I'm not talking about the goal-specification phase, for which no shared basis for interpretation can be expected. "The humans constructed me to stop cancer. Now, I have come to understand that humans want that in order to live longer, and I use that and all my other refined models of the human psyche to fulfill my goal. Which I do: I stop cancer, by wiping out humanity." (Refined models cannot be used to change terminal goals, only to choose actions and subgoals to attain those goals.) More qualifications apply:

At first, such human-related models would of course be quite lacking, but probably converge fast (by definition). The problem remains of why the superintelligent agent would do what the monkeys intend it to (nevermind what they explicitly told it to), and how the monkeys could make sure of that in a way which survives self-modification. The intend-it-to / programmed-it-to dichotomy remains a problem then, since terminal goals are presumably not subject to updating/reflection, at least not as part of the 'superintelligence' attribute.

tl;dr: A superintelligent agent's specified goals must be airtightly constructed, but if those include "do what the human intends, not what he says", then the step from "words" to "intent" should be trivial. (Argument that superintelligent agents will not misinterpret humans does not apply to the goal-setting phase!)

ETA: News at 11 - News at 11 - Kawoomba solved FAI: use / leverage the foomed AIs superior model building ability (which entails that it knows what we want better than we do) by letting it solve the problem: let its initial (invariant) goal be to develop superior models of anything it encounters without affecting it (which should be easier to formalize than "friendliness"), then time that such that it will ask for "ENTER NEW GOALS" once it already established its superior models, at which point you simply tell it "ok glass, use as your new goal system that which I'd most want you to use".

NEXT!