If we're going to be building a Seed AI, I really don't think a good design would involve the AI reasoning using multiple, partially overlapping, possibly inconsistent models, especially since I'm not sure how the AI would go about updating those models if it made contradictory observations. For example, upon receiving contradictory evidence, which of its models would it update? One? Two? All of them? If you decide to work with ad hoc hypotheses that contradict not only reality, but each other, just because it's useful to do so, the price you pay is throwing the entire idea of updating out the window.
If it's uncertainty you're concerned about, you don't need to go to the trouble of having multiple models; good old Bayesian reasoning is designed to deal with uncertainties in reasoning--no overlapping models required. Moreover, I have a difficult time believing that a sufficiently intelligent AI would face much of an issue with regard to processing speed or memory capacity; if anything, working with multiple models might actually take longer in some situations, e.g. when dealing with a scenario in which several different models could apply. In short, the "super all encompassing model" would seem to work just fine.
Bayesianism works well with known unknowns. But it doesn't work any better than any other system else with unknown unknowns. I would say that while Bayesian reasoning can deal well with risk, it's not great with uncertainty - that's not to say uncertainty invalidates Bayesianism, only to say that Bayesianism is not so spectacularly strong it is able to overwhelm such fundamental difficulties of epistemology.
To my mind, using multiple models of reality is more or less essential. My reasons for thinking this are difficult to articulate because they're mired ...
Another month, another rationality quotes thread. The rules are: