That's the friendly AI problem. If you have a piece of planning software that seems to work fine, and you give it more and more options and resources, how do you know that it will keep generating non-extreme plans?
If it terminates as soon as it hits a plan that achieves the goal, and the possible actions are ordered in terms of how extreme they are, then increasing the available resources can't cause trouble, but increasing the available options can (because your ordering might go from correct to incorrect).
In general optimization terms, this is the difference between local optimum solutions and global optimum solutions. If you have a reasonable starting point and use gradient descent, to end up at a reasonable ending point you only need the local solution space to be reasonable because the total distance you'll travel is likely to be short (relative to the solution space and dependent on its topology, of course). If you have a global optimum solution, you need the entire solution space to be reasonable.
I've since edited the previous comment to agree with you in principle, but I think this particular objection doesn't really work.
Let's say Lawrence asks the AI to get him a cheeseburger with probability at least 90%. The AI can't use its usual plan because the local burger place is closed. It picks the next simplest plan, which involves using a couple more computers for additional planning and doesn't specify any further details. These computers receive the subgoal "maximize the probability no matter what", because it's slightly simpler mathemati...
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should be posted in Discussion, and not Main.
4. Open Threads should start on Monday, and end on Sunday.