= 27d567bc2d48d9f40e9ccc20ea370b15)
Liron comments on Metaphilosophical Mysteries - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (255)
What does it mean to "realize that a prior is wrong"? The mechanics of belief change in a Bayesian agent are fixed by the prior itself.
Bayesian updating is always the right thing to do. The only question is how to approximate a proper Bayesian update using efficient data structures and algorithms.
I would stay in the fruit tree metaphor and say they might be "hanging right over our heads".
A prior can be wrong if it assigns zero weight to the true state of the world. For example, if our universe does in fact contain halting problem oracles, the Bayesian superintelligence with a TM-based universal prior will never be able to believe that, no matter how many hard math problems get successfully solved by this weird black box. But a human would converge on the true belief pretty quickly. All this stuff, and more, is in Wei Dai's examples.
AIXI with a TM-based universal prior will always produce predictions about the black box, and predictions about the rest of the universe based on what the black box says, that are just as good as any prediction the human can come up with. After all, the human is in there somewhere. If you think of AIXI as embodying all computable ways of predicting the universe, rather than all computable models of the universe, you may begin to see that's not quite as narrow as you thought.
Eliezer, that was your position in this thread, and I thought I had convinced you that it was wrong. If that's not the case, can you please re-read my argument (especially the last few posts in the thread) and let me know why you're not convinced?
So... the part I found potentially convincing was that if you ran off a logical view of the world instead of a Solomonoff view (i.e., beliefs represented in e.g. higher-order logic instead of Turing machines) and lived in a hypercomputable world then it might be possible to make better decisions, although not better predictions of sensory experience, in some cases where you can infer by reasoning symbolically that EU(A) > EU(B), presuming that your utility function is itself reasoning over models of the world represented symbolically. On the other hand, cousin_it's original example still looks wrong.
You can make better predictions if you're allowed to write down your predictions symbolically, instead of using decimal numbers. (And why shouldn't that be allowed?)
ETA: I made this argument previously in the one-logic thread, in this post.
ETA 2: I think you can also make better (numerical) predictions of the form "this black box is a halting-problem oracle" although technically that isn't a prediction of sensory experience.
Why would you want to make any predictions at all? Predictions are not directly about value. It doesn't seem that there is a place for the human concept of prediction in a foundational decision theory.
I think that's right. I was making the point about prediction because Eliezer still seems to believe that predictions of sensory experience is somehow fundamental, and I wanted to convince him that the universal prior is wrong even given that belief.
Still, universal prior does seem to be a universal way of eliciting what the human concept of prediction (expectation, probability) is, to the limit of our ability to train such a device, for exactly the reasons Eliezer gives: whatever is the concept we use, it's in there, among the programs universal prior weights.
ETA: On the other hand, the concept thus reconstructed would be limited to talk about observations, and so won't be a general concept, while human expectation is probably more general than that, and you'd need a general logical language to capture it (and a language of unknown expressive power to capture it faithfully).
ETA2: Predictions might still be a necessary concept to express the decisions that agent makes, to connect formal statements with what the agent actually does, and so express what the agent actually does as formal statements. We might have to deal with reality because the initial implementation of FAI has to be constructed specifically in reality.
Umm... what about my argument that a human can represent their predictions symbolically like "P(next bit is 1)=i-th bit of BB(100)" instead of using numerals, and thereby do better than a Solomonoff predictor because the Solomonoff predictor can't incorporate this? Or in other words, the only reason the standard proofs of Solomonoff prediction's optimality go through is that they assume predictions are represented using numerals?
Surely predictions of sensory experience are pretty fundamental. To understand the consequences of your actions, you have to be able to make "what-if" predictions.
Re: "It doesn't seem that there is a place for the human concept of prediction in a foundational decision theory."
You can hardly steer yourself effectively into the future if you don't have an understanding of the consequences of your actions.
Yes, it might be necessary exactly for that purpose (though consequences don't reside just in the "future"), but I don't understand this well enough to decide either way.
Yes, the human is in there somewhere, but so are many other, incorrect predictors. To adopt their predictions as its own, AIXI neds to verify them somehow, but how? (I'm very confused here and may be missing something completely obvious.)
ETA: yeah, this is wrong, disregard this.
That took two days to parse, but now I understand how it works. You're right. I apologize to everyone for having defended an incorrect position.
My misconception seems to be popular, though. Maybe someone should write a toplevel post on the right way to think about the universal prior. Though seeing that some other people are even more hopelessly confused than me, and seem to struggle with the idea of "prior" per se, I'm not sure that introducing even more advanced topics would help.
I don't know much about Solomonoff induction, so I may be wrong about this, but is it not the case that the universal prior only takes into account computable functions which exactly output the sensory data? If that is the case, consider the following scenario:
We have a function F which takes an unbounded natural number N as input and is provably uncomputable for all valid inputs. We have a computable algorithm A which provably outputs lower and upper bounds for F for any valid input. Furthermore, it is provable that no computable algorithm can provably produce tighter bounds on F's output than A (regardless of N). We can see that A outputs the bounds for a closed interval in the set of real numbers. We know that all such intervals (for which the lower and upper bounds are not equal) are uncountable. Now imagine a physical hypercomputer which outputs F(0), then F(1), then F(2), etc. to infinity. No computable algorithm will be able to predict the next symbol output by this hypercomputer, but there will be computable minds capable of recognizing the pattern and so of using A to place stronger bounds on its predictions of future sensory experience than AIXI can.
EDIT: Actually, this scenario might be broken. Specifically, I'm not sure what it physically means to 'output' an uncomputable number, and I think that AIXI's problem dissolves if we limit ourselves to the computable (and thus countable) subsets of the output intervals.
Is there a good exposition of this semantics (more generally, for algorithmic probability)?
I think this problem would vanish if you spelled out what "believe" means. The Bayesian superintelligence would quickly learn to trust the opinion of the halting problem oracle; therefore, it would "believe" it.
I am having a few problems in thinking of a sensible definition of "believe" in which the superintelligence would fail to believe what its evidence tells it is true. It would be especially obvious if the machine was very small. The superintelligence would just use Occcam's razor - and figure it out.
Of course, one could imagine a particularly stupid agent, that was too daft to do this - but then it would hardly be very much of a superintelligence.
P(true) = 0 - or p(false) = 1 - seem like trivial mistakes to avoid.
A "expected utility maximizer programmed with a TM-based universal prior" would surely not care very much if it was programmed with wrong priors after a while - since it would not be depending on the details of its priors much any more - due to having a big mountain of experience concerning what the actual expected frequency of events was. Its priors would be swamped by data - unless its priors were completely crazy.
The OP must be thinking of some different type of construct from me - and he doesn't seem to explain what it is.
Unfortunately they aren't. A universal prior must enumerate all the ways a universe could possibly be. If your prior is based on Turing machines that compute universes, but our actual universe is uncomputable, you're screwed forever no matter what data comes in. Maybe the problem can be solved by a better universal prior, as Nesov suggests elsewhere in the thread, but as far as I understand it's an open problem right now.
ETA: pretty much this whole comment is wrong. The prior is over algorithms that generate sequences of sensory input, not over algorithms that define universes. This is an important distinction, sorry for missing it when I wrote this comment.
Natural selection solves this problem.
Being forced to use the nearest computable approximation to an uncomputable function does not make you screwed forever.
That depends on the uncomputable function. Some can make you very well screwed indeed. It's all there in Wei Dai's examples on everything-list and one-logic, I really wish people would read them, maybe we'd have an actual discussion then. Sorry for sounding harsh.
Right, but it's not necessarily true, or even likely, hence my point.
I did read the links, (including the link to the empty stub article!), and the google group discussions all seemed to end, from my brief perusing of them, with them coming to the consensus that Wei Dai hadn't established his provacative, counterintuitive point. (And some of the exchanges here show the same.)
At the very least, he should summarize the reasoning or examples, as per standard practice, so we know there's something to be gained from going to the links. This is especially true given that most readers had assumed that the opposite of Wei Dai's premises are true and uncontroversial.
To avoid such a trivial mistake, just follow the advice on:
http://lesswrong.com/lw/mp/0_and_1_are_not_probabilities/
Yeah, he really saw the light, but dropped the ball, when writing that stormy bag of mixed metaphors.
Gee, that was obviously supposed to be a non-mixed metaphor about urban foraging. Yeah that's it. :)
Seriously, I thought about sticking with the fruit tree metaphor, but "hanging right over our heads" makes the problem sound too easy, so I decided to favor accuracy over literary elegance.