= 27d567bc2d48d9f40e9ccc20ea370b15)
PhilGoetz comments on Metaphilosophical Mysteries - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (255)
A prior can be wrong if it assigns zero weight to the true state of the world. For example, if our universe does in fact contain halting problem oracles, the Bayesian superintelligence with a TM-based universal prior will never be able to believe that, no matter how many hard math problems get successfully solved by this weird black box. But a human would converge on the true belief pretty quickly. All this stuff, and more, is in Wei Dai's examples.
I think this problem would vanish if you spelled out what "believe" means. The Bayesian superintelligence would quickly learn to trust the opinion of the halting problem oracle; therefore, it would "believe" it.
I am having a few problems in thinking of a sensible definition of "believe" in which the superintelligence would fail to believe what its evidence tells it is true. It would be especially obvious if the machine was very small. The superintelligence would just use Occcam's razor - and figure it out.
Of course, one could imagine a particularly stupid agent, that was too daft to do this - but then it would hardly be very much of a superintelligence.