There's also the problem of actually building such a thing.

edit: I should add, the problem of building this particular thing is above and beyond the already difficult problem of building any AGI, let alone a friendly one: how do you make a thing's utility function correspond to the world and not to its perceptions? All it has immediately available to it is perception.