You understand Bayes' Theorem. You enter into a situation with an intuitive "common sense" prior. You observe the situation, and then you shut up and multiply.
And then you go to update, you compute the desired behavior to maximize utility... and some cognitive module buried in your brain says "no".
Example:
I realize I need a physical examination. I have no rational reason to fear going to the doctor. I am, in fact, acutely aware that my fear of going to the doctor is based on a fear that they will find something wrong that I can't afford to fix, but the truth is already so. So I bite the bullet, make an appointment, and then at the scheduled time I get in the car and drive to the doctor's office.
And then I just keep driving past the doctor's office, turn around and go home.
I tell myself that if I'm not going, I should call the doctor to avoid a $100 no-show fee, but I don't.
And then I get home, and I tell myself that that was dumb, and that I need to update my behavior - and that physically punishing myself for not going to the doctor is not an efficient use of my energy.
So I punch my hand through a mirror.
NOW, finally, I have an excuse to go to the doctor - so I wrap my hand in bandages and go back to bed, instead.
What do you do when your computed probabilities and utility function have NO EFFECT WHATSOEVER on your actual behavior?
hrm. Historically, when I've done that, I've got called on it, and then socially sanctioned. (I.e., "you say you think , but then I see you doing . I'm going to stop believing anything you say until you start being more honest.")
I think I may be bogged down with too many cached constraints, but I have no idea which ones to purge, let alone how to stop following them.
Also, knowledge is a tricky thing. While I've always followed something like a Bayesian heuristic for knowledge when left to my own devices, it's reasonably easy to convince me to abandon it in favor of a kind of radical skepticism against my own thoughts and qualia.
In person that can certainly be a problem with some emotional/irrational people but luckily we can't read facial expressions and body language here on the internet :)
It's a possibil... (read more)