For example, "We need to decide whether that AI is intelligent enough to just let it loose exploring this planet" implies a different definition of "intelligent" compared to, say, "We need to decide whether that AI is intelligent enough to be trusted with a laser cutter".
Those sound more like safety concerns than inquiries involving intelligence. Being clever and able to get things done doesn't automatically make something share enough of your values to be friendly and useful.
Better questions would be "We need to decide whether that AI is intelligent enough to effectively research and come to conclusions about the world if we let it explore without restrictions" or "We need to decide if the AI is intelligent enough to correctly use a laser cutter".
Although, given large power (i.e. a laser cutter) and low intelligence, it might not achieve even its explicate goal correctly, and may accidentally do something bad. (i.e. laser cut a person)
one attribute of intelligence is the likelihood of said AI producing bad results non-purposefully. The more it does, the less intelligent it is.
one attribute of intelligence is the likelihood of said AI producing bad results non-purposefully.
Nah, that's an attribute of complexity and/or competence.
My calculator has a very very low likelihood of producing bad results non-purposefully. That is not an argument that my calculator is intelligent.
A stub on a point that's come up recently.
If I owned a paperclip factory, and casually told my foreman to improve efficiency while I'm away, and he planned a takeover of the country, aiming to devote its entire economy to paperclip manufacturing (apart from the armament factories he needed to invade neighbouring countries and steal their iron mines)... then I'd conclude that my foreman was an idiot (or being wilfully idiotic). He obviously had no idea what I meant. And if he misunderstood me so egregiously, he's certainly not a threat: he's unlikely to reason his way out of a paper bag, let alone to any position of power.
If I owned a paperclip factory, and casually programmed my superintelligent AI to improve efficiency while I'm away, and it planned a takeover of the country... then I can't conclude that the AI is an idiot. It is following its programming. Unlike a human that behaved the same way, it probably knows exactly what I meant to program in. It just doesn't care: it follows its programming, not its knowledge about what its programming is "meant" to be (unless we've successfully programmed in "do what I mean", which is basically the whole of the challenge). We can't therefore conclude that it's incompetent, unable to understand human reasoning, or likely to fail.
We can't reason by analogy with humans. When AIs behave like idiot savants with respect to their motivations, we can't deduce that they're idiots.