Ah, that's why I think reductionism would be very useful for you. Everything can be broken down and understood in such a way that nothing remains that doesn't represent testable consequences. definitely read How an Algorithm Feels As the following quote represents what you may be thinking when you wonder if something is really intelligent.
Now suppose that you have an object that is blue and egg-shaped and contains palladium; and you have already observed that it is furred, flexible, opaque, and glows in the dark. [all the characteristics implied by the label "blegg"]
This answers every query, observes every observable introduced. There's nothing left for a disguised query to stand for.
So why might someone feel an impulse to go on arguing whether the object is really a blegg [is truly intelligent]?
[brackets] are my additions.
Oh, sure, but the real question is what are all the characteristics implied by the label "intelligent".
The correctness of a definition is decided by the purpose of that definition. Before we can argue what's the proper meaning of the word "intelligent" we need to decide what do we need that meaning for.
For example, "We need to decide whether that AI is intelligent enough to just let it loose exploring this planet" implies a different definition of "intelligent" compared to, say, "We need to decide whether that AI is intelligent enough to be trusted with a laser cutter".
A stub on a point that's come up recently.
If I owned a paperclip factory, and casually told my foreman to improve efficiency while I'm away, and he planned a takeover of the country, aiming to devote its entire economy to paperclip manufacturing (apart from the armament factories he needed to invade neighbouring countries and steal their iron mines)... then I'd conclude that my foreman was an idiot (or being wilfully idiotic). He obviously had no idea what I meant. And if he misunderstood me so egregiously, he's certainly not a threat: he's unlikely to reason his way out of a paper bag, let alone to any position of power.
If I owned a paperclip factory, and casually programmed my superintelligent AI to improve efficiency while I'm away, and it planned a takeover of the country... then I can't conclude that the AI is an idiot. It is following its programming. Unlike a human that behaved the same way, it probably knows exactly what I meant to program in. It just doesn't care: it follows its programming, not its knowledge about what its programming is "meant" to be (unless we've successfully programmed in "do what I mean", which is basically the whole of the challenge). We can't therefore conclude that it's incompetent, unable to understand human reasoning, or likely to fail.
We can't reason by analogy with humans. When AIs behave like idiot savants with respect to their motivations, we can't deduce that they're idiots.