= 27d567bc2d48d9f40e9ccc20ea370b15)
gwern comments on Open Thread: September 2011 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (441)
I keep running into problems with various versions of what I internally refer to as the "placebo paradox", and can't find a solution that doesn't lead to Regret Of Rationality. Simple example follows:
You have an illness from wich you'll either get better, or die. The probability of recovering is exactly half of what you estimate it to be due to the placebo effect/positive thinking. Before learning this you have 80% confidence in your recovery. Since you estimate 80%, your actual chance is 40% so you update to this. Since the estimate is now 40%, the actual chance is 20%, so you update to this. Then it's 10%, so you update to that. etc. Until both your estimated and actual chance of recovery are 0. then you die.
An irrational agent, on the other hand, upon learning this could self delude to 100% certainty of recovery, and have a 50% chance of actually recovering.
This is actually causing me real world problems, such as inability to use techniques based on positive thinking, and a lot of cognitive dissonance.
Another version of this problem features in HP:MoR, in the scene where harry is trying to influence the behaviour of dementors.
And to show this isn't JUST a quirk of human mind design, one can envision Omega setting up an isomorphic problem for any kind of AI.
An AI can presumably self-modify. For a sufficient reward from Omega, it is worth degrading the accuracy of one's beliefs, especially if the reward will immediately allow one to make up for the degradation by acquiring new information/engaging in additional processing.
(A hypothetical: Omega offers me 1000 doses of modafinil, if I will lie on one PredictionBook.com entry and say -10% what I truly believe. I take the deal and chuckle every few minutes the first night, when I register a few hundred predictions to make up for the falsified one.)
This entirely misses the point. Yes, you could self modify, but it's a self modification away from rationality and that gives rise to all sorts of trouble as has been elaborated many times in the sequences. For example: http://lesswrong.com/lw/je/doublethink_choosing_to_be_biased/
Also, LYING about what you believe has nothing to do with this. Omega can read your mind.
I was trying to apply the principle of charity and interpret your post as anything but begging the question: 'assume rational agents are penalized. How do they do better than irrational agents explicitly favored by the rules/Omega?'
Question begging is boring, and if that's really what you were asking - 'assume rational agents lose. How do they not lose?' - then this thread is deserving only of downvotes.
And Eliezer was talking about humans, not the finer points of AI design in a hugely arbitrary setup. It may be a bad idea for LWers to choose to be biased, but a perfectly good idea for AIXI stuck in a particularly annoying computable universe.
Since I'm not an AI with direct access to my beliefs in storage on a substrate, I was using an analogy to as close as I can get.
Sorry, I were hoping that there were some kind of difference between "penalize this specific belief in this specific way" and "penalize rationality as such in general", some kind of trick to work around the problem, that I hadn't noticed and which resolved the dilemma.
And your analogy didn't work for me, is all I'm saying.