I think I understand Moore's Paradox a bit better now, after reading some of the comments on Less Wrong. Jimrandomh suggests:
Many people cannot distinguish between levels of indirection. To them, "I believe X" and "X" are the same thing, and therefore, reasons why it is beneficial to believe X are also reasons why X is true.
I don't think this is correct—relatively young children can understand the concept of having a false belief, which requires separate mental buckets for the map and the territory. But it points in the direction of a similar idea:
Many people may not consciously distinguish between believing something and endorsing it.
After all—"I believe in democracy" means, colloquially, that you endorse the concept of democracy, not that you believe democracy exists. The word "belief", then, has more than one meaning. We could be looking at a confused word that causes confused thinking (or maybe it just reflects pre-existing confusion).
So: in the original example, "I believe people are nicer than they are", she came up with some reasons why it would be good to believe people are nice—health benefits and such—and since she now had some warm affect on "believing people are nice", she introspected on this warm affect and concluded, "I believe people are nice". That is, she mistook the positive affect attached to the quoted belief, as signaling her belief in the proposition. At the same time, the world itself seemed like people weren't so nice. So she said, "I believe people are nicer than they are."
And that verges on being an honest mistake—sort of—since people are not taught explicitly how to know when they believe something. As in the parable of the dragon in the garage; the one who says "There is a dragon in my garage—but it's invisible", does not recognize his anticipation of seeing no dragon, as indicating that he possesses an (accurate) model with no dragon in it.
It's not as if people are trained to recognize when they believe something. It's not like they're ever taught in high school: "What it feels like to actually believe something—to have that statement in your belief pool—is that it just seems like the way the world is. You should recognize this feeling, which is actual (unquoted) belief, and distinguish it from having good feelings about a belief that you recognize as a belief (which means that it's in quote marks)."
This goes a long way toward making this real-life case of Moore's Paradox seem less alien, and providing another mechanism whereby people can be simultaneously right and wrong.
Likewise Kurige who wrote:
I believe that there is a God—and that He has instilled a sense of right and wrong in us by which we are able to evaluate the world around us. I also believe a sense of morality has been evolutionarily programmed into us—a sense of morality that is most likely a result of the formation of meta-political coalitions in Bonobo communities a very, very long time ago. These two beliefs are not contradictory, but the complexity lies in reconciling the two.
I suspect, Kurige, you have decided that you have reasons to endorse the quoted belief that God has instilled a sense of right and wrong in us. And also that you have reasons to endorse the verdict of science. They both seem like good communities to join, right? There are benefits to both sets of beliefs? You introspect and find that you feel good about both beliefs?
But you did not say:
"God instilled a sense of right and wrong in us, and also a sense of morality has been evolutionarily programmed into us. The two states of reality are not inconsistent, but the complexity lies in reconciling the two."
If you're reading this, Kurige, you should very quickly say the above out loud, so you can notice that it seems at least slightly harder to swallow—notice the subjective difference—before you go to the trouble of rerationalizing.
This is the subjective difference between having reasons to endorse two different beliefs, and your mental model of a single world, a single way-things-are.
The reason why saying "There is a God and He instilled..." is harder than saying "I believe that there is a God and He instilled..." is because the words "I believe that" are weasel words. The literal meaning of "I believe that" is irrelevant; any other weasel words would have the same effect. Consider the same sentence, but replace "I believe that" with "It is likely that", or "Evidence indicates that", or any similar phrase, and it's just as easy.
Just because people are aware of a concept, and have words which ought to refer to that concept, does not mean that they consistently connect the two. The best example of this comes from the way people refer to things as [good] and [bad]. When people dislike something, but don't know why, they generate exemplars of the concept "bad", and call it evil, ugly, or stupid. This same mechanism lead to the widespread use of "gay" as a synonym for "bad", and to racial slurs directed at anonymous online rivals who are probably the wrong race for the slur. I think that confidence markers are subject to the same linguistic phenomenon.
People think with sentences like "That's a [good] car" or "[Weasel] God exists". The linguistic parts of their mind expand them to "That's a sweet car" and "I believe God exists" when speaking, and performs the inverse operation when listening. They don't think about how the car tastes, and they don't think about beliefs, even though literal interpretation of what they say would indicate that they do.
Ah, but the point is that "believe" is the weasliest of words. I know a few, and would guess there are quite a lot more, intelligent people who readily states "I believe that there is a God" but who would be very hesitant if you asked them to use "Evidence indicates that".
I would say that what you call weasel words occupy a scale and that its not just as easy to use them all in any given context, at least not for reasonably intelligent people.