You do realize that people are working on logical uncertainty under limited time, and this could tell an AI how to re-examine its assumptions? I admit that Gaifman at Columbia deals only with a case where we know the possibilities beforehand (at least in the part I read). But if the right answer has a description in the language we're using, then it seems like E.T. Jaynes theoretically addresses this when he recommends having an explicit probability for 'other hypotheses.'
Then again, if this approach didn't come up when the authors of "Tiling Agents" discuss utility maximization, perhaps I'm overestimating the promise of formalized logical uncertainty.
Folks here should be familiar with most of these arguments. Putting some interesting quotes below:
http://aeon.co/magazine/being-human/david-deutsch-artificial-intelligence/
"Creative blocks: The very laws of physics imply that artificial intelligence must be possible. What's holding us up?"
He also says confusing things about induction being inadequate for creativity which I'm guessing he couldn't support well in this short essay (perhaps he explains better in his books). Not quoting here. His attack on Bayesianism as an explanation for intelligence is valid and interesting, but could be wrong. Given what we know about neural networks, something like this does happen in the brain, and possibly even at a concept level.
His final conclusions are disagreeable. He somehow concludes that the principal bottleneck in AGI research is a philosophical one.
In his last paragraph, he makes the following controversial statement:
This would be false if, for example, the mother controls gene expression while a foetus develops and helps shape the brain. We should be able to answer this question definitively once we can grow human babies completely in vitro. Another problem would be the impact of the cultural environment. A way to answer this question would be to see if our Stone Age ancestors would be classified as AGIs under a reasonable definition