The AI-box for hunter-gatherers

Alex Flint

The following is a minor curiosity that occurred to me regarding real-world analogies to the AI-box concept.

Fundamentally, the reason that we fear a randomly-chosen super-intelligent AI is twofold:

It would be smarter than us, so it could outwit us no matter what its goals.
We have no reason to expect its goals to exactly coincide with ours, so we expect its actions to be detrimental to us.

Now, let's say you're playing a game with a wide range of outcomes, over which you have a well-defined utility function. Let's say you can choose a partner from some pool of potential partners, each with a random utility function. Let's say the intelligence of each player is known to you. It would be unwise to choose a player more intelligent than yourself because

They could outwit you.
You have no reason to expect their goals to coincide with yours.

On the other hand, if you pick a less intelligent player then perhaps you could trick them into furthering your own goals, or at least ascertain whether their motives coincide with yours. At the very least you would be able to keep them from subverting your own goal-directed actions.

I conjecture that this explains why people sometimes fear those more intelligent than themselves, and also why people sometimes act dumb. Imagine a hunter-gatherer group considering inviting an outsider to join them. If the outsider's motives are uncertain then the more intelligent the outsider, the less well-advised the group would be to let em in.

In fact, a situation analogous to an AI-box could arise:

Group member 1: We should let this intelligent outsider in, but we should keep tabs on them should they act against us.

Group member 2: But the fact that they're more intelligent than us means that you *can't* expect to keep tabs on them.

Group member 1: But couldn't we set a test of their motives and only allow them in if they proves to have motives coinciding with ours?

Group member 2: No matter what test you set, we can expect them to outwit us -- that's what intelligence means.

The following is a minor curiosity that occurred to me regarding real-world analogies to the AI-box concept.

Fundamentally, the reason that we fear a randomly-chosen super-intelligent AI is twofold:

It would be smarter than us, so it could outwit us no matter what its goals.
We have no reason to expect its goals to exactly coincide with ours, so we expect its actions to be detrimental to us.

They could outwit you.
You have no reason to expect their goals to coincide with yours.

In fact, a situation analogous to an AI-box could arise:

Group member 1: We should let this intelligent outsider in, but we should keep tabs on them should they act against us.

Group member 2: But the fact that they're more intelligent than us means that you *can't* expect to keep tabs on them.

Group member 1: But couldn't we set a test of their motives and only allow them in if they proves to have motives coinciding with ours?

Group member 2: No matter what test you set, we can expect them to outwit us -- that's what intelligence means.

In the case of people however group member 2 is often incorrect because the intelligence difference is small enough that some level of judgement is often possible. Unlike with a super intelligent AI where it is currently not even known how to judge its level of intelligence because it can self improve in short order.

Fair point, the analogy I made is a bit of a stretch, really.

Still, I think that uncertainty about "boxing" increases rapidly enough with intelligence that these considerations are significant even for intelligence differences that we observe between humans.

11

The AI-box for hunter-gatherers

11

11

11

The AI-box for hunter-gatherers

11

11