I'm thinking of a dilemma that I thought was called the farmer's dilemma, but that redirects to the prisoner's dilemma on wikipedia, and google doesn't help me out either. Is there a standard name for this dilemma?
Two farmers have adjacent fields. If either of them irrigates (U-1), both get the benefits from it (U+5). If I cooperate, my opponent can get a payoff of 4 by cooperating or 5 by defecting, so he has an incentive to defect; my own payoff is guaranteed 4. If I defect, my opponent can get a payoff of 4 by cooperating (and I get 5), or 0 by defecting (and I also get 0), so he has an incentive to cooperate. We both want at least of us to cooperate, but as long as the other cooperates, we have a mild preference for defecting.
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d)
And you have an amazingly good tool-user, that still doesn't innovate or surprise you.
That's not entirely true. It might surprise us by, say, showing us the precise way to use an endoscopic cauterizer to cut off blood flow to a tumor without any collateral damage. But it can't, by definition, invent a new tool entirely.
I'm not sure the solution to the AI friendliness problem is "Creating AI that is too narrow-minded to be dangerous". You throw out most of what is intended to be achieved by AI in the first place, and achieve little more than evolutionary algorithms are already capable of. (If you're capable of modeling the problem to that extent, you can just toss it, along with the toolset, into an evolutionary algorithm and get something pretty close to just as good.)
Can humans "invent a new tool entirely", when all we have to work with are a handful of pre-defined quarks, leptons and bosons? AIXI is hard-coded to just use one tool, a Turing Machine; yet the open-endedness of that tool make it infinitely inventive.
We can easily put a machine shop, or any other manufacturing capabilities, into the abstract room. We could ignore the tedious business of manufacturing and just include a Star-Trek-style replicator, which allows the AI to use anything for which is can provide blueprints.
Also, we can easily be surprised by actions taken in the room. For example, we might simulate the room according to known scientific laws, and have it automatically suspend if anything strays too far into uncertain territory. We can then either abort the simulation, if something dangerous or undesirable is happening within, or else perform an experiment to see what would happen in that situation, then feed the result back in and resume. That would be a good way to implement an artificial scientist. Similar ideas are explored in http://lambda-the-ultimate.org/node/4392