I agree (see, e.g., The Second Law of Thermodynamics, and Engines of Cognition for why this is the case). Unfortunately, I see this as a key inferential gap between people who are and aren't trained in rationality.

The problem is that many people-- dare I say most-- feel no obligation to gather evidence for their intuitive feelings, or to let empirical evidence inform their feelings. They don't think of intuitive feelings as predictions to be updated by Bayesian evidence; they treat their intuitive feelings as evidence.

It's a common affair (at least in the United States) to see debaters use unsubstantiated intuitive feelings as linchpins of their arguments. It's even common on internet debates to see whole chains of reasoning in which every link is supported by gut feeling alone. This style of argument is not only unpersuasive to anyone who doesn't share those intuitions already-- it prevents the debater from updating, as long as his intuitions don't change.

You argument reminds me of a thought experiment I did concerning the "GOD Operator...

1+ 1 = 2

1 -1 = 0

1 * 1 = 1

1 / 1 = 1

Etc...

The operator is the +, -, * /, etc

The god operator is inclusive of all known operators and allows such things as:

1 GOD 1 = whatever answer Fits

AND

1 GOD 1 = sqrt(-1), PI, Etc..

How do we define the operator when GOD can be "whatever works"?

My main thoughts then went to the idea of a "universal machine" much like Turing...

What specifically is the mechanism of the human mind that would allow both of the above examples?