To begin, here are some Fun Psychology Facts:
People who were asked to describe a face after seeing it are worse at recognizing the same face later.
People who are asked to describe a wine after drinking it are worse at recognizing the same wine later.
People who are asked to give reasons for their preferences among a collection of jellies are worse at identifying their own preferences among those jellies.
This effect, known as Verbal Overshadowing, occurs primarily when a principally non-verbal process is disrupted by a task which involves verbalization. The above generalizations (and Verbal Overshadowing effects more generally), do not occur among what we can term "Verbal Experts": individuals who are as good at verbalizing the relevant process as they are at doing it implicitly or automatically. This seems like it will be very important to keep in mind when cultivating our own Rationality.
Sorry, I misread. I thought that was just a restating of the original concern. Mind rephrasing it? Thanks.
(note, however, that I'm talking about what "ideal rational agents that don't want to be stupid" do. As I indicated in the warning, trying to actually fully and completely translate a human's entire preferences to this is a highly nontrivial task)
I am thinking more like this: I am a scaredy-cat about roller coasters. So I prefer the tea cups to big thunder mountain rail road. And I maintain that preference after choosing the Tea Cups (I don't regret my decision). However, had I ridden Big Thunder Mountain Rail Road, I would have been able to appreciate that it is awesome, and would have preferred Big Thunder Mountain Rail Road to the Tea Cups.
Since this case seems pretty possible, if the sorts of lessons you are going to draw only apply to hyper-idealized agents who know all their preferences perfectly and whose preferences are stable over time, that is a good thing to note, since the lessons may not apply to those of us with dynamic preference sets.