But it also seems too easy, as if you could take ten paper-clip maximizers competing to convert things into differently colored paperclips, and ended out with utopia.

That's essentially what happened with bacteria - they really don't have goals beyond converting the universe into bacteria. But through an endless process of exhausting resources, competition, and cooperation, they worked out different ways of dealing with each other, until they eventually became multicellular - and so the process began again. We've reached the point where superorganisms composed of many multicellular organisms are arising.

Give utility-maximizers long enough to develop, and they'll develop increasingly-sophisticated strategies for dealing with each other. 'Nice' has nothing to do with it.

As for the main thrust of this post - normal rationality is self-correcting, but it's good not to be able to make the mistake in the first place. But minimize one type of error, increase the possibility of others. Always take the sure course and you'll miss opportunities; never take risks, and you'll accomplish far less than you possibly could. Whether that's a good thing depends on how serious screwing up is.