There's a danger, though, in building something that's superhumanly intelligent, and has goals that it acts on, that doesn't include some of our goals. You would have to make sure it's not an expectation-maximizing agent.

I think an assumption of the FAI project is that you shouldn't do what Nancy is proposing, because you can't reliably build a superhumanly-intelligent self-improving agent and cripple it in a way that prevents it from trying to maximize its goals.