I have some similar problems and I'll try to explain them.
So I think that some of my failings may be attributable to having many contradictory values rather than to being failures of rationality. An example of values that oppose one another are the desire for social interaction and the strong sense of guilt that I experience at any social failing. If I try to optimize for one then I take a hit from the other and I still don't net very many hedons. The rational action here would be to modify and remove the errant value, but that's difficult.
In Neurotic spells in the past I've also acquired some desire to see myself fail as well and this flares up from time to time. I can be fairly rational while alone, but in social situations I seem to get overwhelmed with negative emotion and my rationality gets 'knocked out' . Well, not literally knocked out, it's just that I update on all sorts of misperceived social cues and wind up with weird aliefs and beliefs, temporarily. I can leave a social situation anticipating on different levels that a good friend doesn't actually like me, or some new idea that I had and was confidant about, is actually just terrible (in situations were neither of these was actually the case). I can point to some large past irrationalities that were the source of some of this, but the values are persistent.
Raw exposure to social situations has reduced some of my wild update problems, but I works very well if I actually interact which is something that is hard to ensure. I also don't that much else out of raw social interaction, so it's not an attractive option.
If anyone wants to link me articles on here about thinking about and dealing with perverse desires I'd love that. It would also be awesome if anyone could point out mistakes that I seem to be making here.
This sounds pretty similar to a lot of my problems. Using this community's terminology, I can have all the beliefs I want, but if I have sufficiently powerful overriding aliefs, I'm screwed - since the alief-guided motivational system is actually closer to the motor control subprocessors than the belief-guided motivational system (aka "Amygdala hijack").
Worse, the alief-driven submodule is operating on its own utility table, which often is a nearly antiparallel eigenvector to my belief-driven submodule's utility table. So I have two submodules each with strong impetus vectors towards/away from various attractors within the solution domain, and... well, thrashing happens.
My name is Brent, and I'm probably insane.
I can perform various experimental tests to verify that I do not perform primate pack-bonding rituals correctly, which is about half of what we mean by "insane". This concerns me simply from a utilitarian perspective (separation from pack makes ego-depletion problems harder; it makes resources harder to come by; and it simply sucks to experience "from the inside"), but these are not the things that concern me most.
The thing that concerns me most is this:
What if the very tools that I use to make decisions are flawed?
I stumbled upon Bayesian techniques as a young child; I was lucky enough to have the opportunity to perform a lot of self-guided artificial intelligence "research" in Junior High and High School, due to growing up in a time and place when computers were utterly mysterious, so no one could really tell me what I was "supposed" to be doing with them - so I started making simple video games, had no opponents to play them against due to the aforementioned failures to correctly perform pack-bonding rituals, decided to create my own, became dissatisfied with the quality of my opponents, and suddenly found myself chewing on Hopfstaedter and Wiener and Minsky.
I'm filling in that bit of detail to explain that I have been attempting to operate as a rational intelligence for quite some time, so I believe that I've become very familiar with the kinds of "bugs" that I will tend to exhibit.
I've spent a very long time attempting to correct for my cognitive biases, edit out tendencies to seek comfortable-but-misleading inputs, and otherwise "force" myself to be rational, and often, the result is that my "will" will crack under the strain. My entire utility-table will suddenly flip on its head, and attempt to maximize my own self-destruction rather than allow me to continue to torture it with endlessly recursive, unsolvable problems that all tend to boil down to "you do not have sufficient social power, and humans are savage and cruel no matter how much you care about them."
Most of my energy is spent attempting to maintain positive, rational, long-term goals in the face of some kind of regedit-hack of my utility table itself, coming from somewhere in my subconscious that I can't seem to gain write-access to.
Clearly, the transhumanist solution would be to identify the underlying physical storage where the bug is occurring, and replace it with a less-malfunctioning piece of hardware.
Hopefully someday someone with more self-control, financial resources, and social resources than I will invent a method to do that, and I can get enough of a partial personectomy to create something viable with the remaining subroutines.
In the meantime, what is someone who wishes to be rational supposed to do, when the underlying hardware simply won't cooperate?