= 27d567bc2d48d9f40e9ccc20ea370b15)
Eliezer_Yudkowsky comments on Metaphilosophical Mysteries - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (255)
AIXI with a TM-based universal prior will always produce predictions about the black box, and predictions about the rest of the universe based on what the black box says, that are just as good as any prediction the human can come up with. After all, the human is in there somewhere. If you think of AIXI as embodying all computable ways of predicting the universe, rather than all computable models of the universe, you may begin to see that's not quite as narrow as you thought.
Eliezer, that was your position in this thread, and I thought I had convinced you that it was wrong. If that's not the case, can you please re-read my argument (especially the last few posts in the thread) and let me know why you're not convinced?
So... the part I found potentially convincing was that if you ran off a logical view of the world instead of a Solomonoff view (i.e., beliefs represented in e.g. higher-order logic instead of Turing machines) and lived in a hypercomputable world then it might be possible to make better decisions, although not better predictions of sensory experience, in some cases where you can infer by reasoning symbolically that EU(A) > EU(B), presuming that your utility function is itself reasoning over models of the world represented symbolically. On the other hand, cousin_it's original example still looks wrong.
You can make better predictions if you're allowed to write down your predictions symbolically, instead of using decimal numbers. (And why shouldn't that be allowed?)
ETA: I made this argument previously in the one-logic thread, in this post.
ETA 2: I think you can also make better (numerical) predictions of the form "this black box is a halting-problem oracle" although technically that isn't a prediction of sensory experience.
Why would you want to make any predictions at all? Predictions are not directly about value. It doesn't seem that there is a place for the human concept of prediction in a foundational decision theory.
I think that's right. I was making the point about prediction because Eliezer still seems to believe that predictions of sensory experience is somehow fundamental, and I wanted to convince him that the universal prior is wrong even given that belief.
Still, universal prior does seem to be a universal way of eliciting what the human concept of prediction (expectation, probability) is, to the limit of our ability to train such a device, for exactly the reasons Eliezer gives: whatever is the concept we use, it's in there, among the programs universal prior weights.
ETA: On the other hand, the concept thus reconstructed would be limited to talk about observations, and so won't be a general concept, while human expectation is probably more general than that, and you'd need a general logical language to capture it (and a language of unknown expressive power to capture it faithfully).
ETA2: Predictions might still be a necessary concept to express the decisions that agent makes, to connect formal statements with what the agent actually does, and so express what the agent actually does as formal statements. We might have to deal with reality because the initial implementation of FAI has to be constructed specifically in reality.
Umm... what about my argument that a human can represent their predictions symbolically like "P(next bit is 1)=i-th bit of BB(100)" instead of using numerals, and thereby do better than a Solomonoff predictor because the Solomonoff predictor can't incorporate this? Or in other words, the only reason the standard proofs of Solomonoff prediction's optimality go through is that they assume predictions are represented using numerals?
Re: "what about my argument that a human can [adapt its razor a little] and thereby do better than a Solomonoff predictor because the Solomonoff predictor can't incorporate this?"
There are at least two things "Solomonoff predictor" could refer to:
An intelligent agent with Solomonoff-based priors;
An agent who is wired to use a Solomonoff-based razor on their sense inputs;
A human is more like the first agent. The second agent is not really properly intelligent - and adapts poorly to new environments.
BB(100) is computable. Am I missing something?
Maybe... by BB I mean the Busy Beaver function Σ as defined in this Wikipedia entry.
Humans are (can be represented by) turing machines. All halting turing machines are incorporated in AIXI. Therefore, anything that humans can do to more effectively predict something than a "mere machine" is already incorporated into AIXI.
More generally, anything you represent symbolically can be represented using binary strings. That's how that string you wrote got to me in the first place. You converted the turing operations in your head into a string of symbols, a computer turned that into a string of digits, my computer turned it back into symbols and my brain used computable algorithms to make sense of them. What makes you think that any of this is impossible for AIXI?
Am I going crazy, or did you just basically repeat what Eliezer, Cyan, and Nesov said without addressing my point?
Do you guys think that you understand my argument and that it's wrong, or that it's too confusing and I need to formulate it better, or what? Everyone just seems to be ignoring it and repeating the standard party line....
ETA: Now reading the second part of your comment, which was added after my response.
ETA2: Clearly I underestimated the inferential distance here, but I thought at least Eliezer and Nesov would get it, since they appear to understand the other part of my argument about the universal prior being wrong for decision making, and this seems to be a short step. I'll try to figure out how to explain it better.
The fact that AIXI can predict that a human would predict certain things, does not mean that AIXI can agree with those predictions.
Surely predictions of sensory experience are pretty fundamental. To understand the consequences of your actions, you have to be able to make "what-if" predictions.
Re: "It doesn't seem that there is a place for the human concept of prediction in a foundational decision theory."
You can hardly steer yourself effectively into the future if you don't have an understanding of the consequences of your actions.
Yes, it might be necessary exactly for that purpose (though consequences don't reside just in the "future"), but I don't understand this well enough to decide either way.
Yes, the human is in there somewhere, but so are many other, incorrect predictors. To adopt their predictions as its own, AIXI neds to verify them somehow, but how? (I'm very confused here and may be missing something completely obvious.)
ETA: yeah, this is wrong, disregard this.
That took two days to parse, but now I understand how it works. You're right. I apologize to everyone for having defended an incorrect position.
My misconception seems to be popular, though. Maybe someone should write a toplevel post on the right way to think about the universal prior. Though seeing that some other people are even more hopelessly confused than me, and seem to struggle with the idea of "prior" per se, I'm not sure that introducing even more advanced topics would help.
I don't know much about Solomonoff induction, so I may be wrong about this, but is it not the case that the universal prior only takes into account computable functions which exactly output the sensory data? If that is the case, consider the following scenario:
We have a function F which takes an unbounded natural number N as input and is provably uncomputable for all valid inputs. We have a computable algorithm A which provably outputs lower and upper bounds for F for any valid input. Furthermore, it is provable that no computable algorithm can provably produce tighter bounds on F's output than A (regardless of N). We can see that A outputs the bounds for a closed interval in the set of real numbers. We know that all such intervals (for which the lower and upper bounds are not equal) are uncountable. Now imagine a physical hypercomputer which outputs F(0), then F(1), then F(2), etc. to infinity. No computable algorithm will be able to predict the next symbol output by this hypercomputer, but there will be computable minds capable of recognizing the pattern and so of using A to place stronger bounds on its predictions of future sensory experience than AIXI can.
EDIT: Actually, this scenario might be broken. Specifically, I'm not sure what it physically means to 'output' an uncomputable number, and I think that AIXI's problem dissolves if we limit ourselves to the computable (and thus countable) subsets of the output intervals.
Is there a good exposition of this semantics (more generally, for algorithmic probability)?