I think you can differentiate between people who say that about a skill, and people who say it about a concept.
Consider driving: I recall driving being a System 2 activity for at least a year while I was learning. It was certainly stressful enough to induce tears on a regular basis (give me a pass, I was a teenager). Slowly, driving under normal conditions became integrated into System 1, and now I don't feel like crying when I have to change lanes. Sufficient practice of any skill can turn it into a System 1 activity.
Currently, programming is a System 2 activity for me. My husband, however, has more than a decade of experience programming. When he helps debug my code, he doesn't painstakingly go over every line, first thing. He glances, skims, says "This doesn't look right..." and then uses a combination of instinct and experience to find my error. I can't imagine being a professional programmer until it's a System 1 activity at least half of the time.
So: the difference between saying "System 1 is integral to my profession and execution of a skill," and "System 1 is all the evidence I need for the existence of a deity" is very large. In the first case, we can take the statement as evidence, appropriately weighted against the speakers track record with that skill, that System 1 has been beneficial. In the second case, people are saying the equivalent of "My instinct = God, and that's the only test I need!" The weight that bears in your Bayesian calculation should be nothing, or almost nothing, because there is no way to develop a God-detecting skill and integrate it into System 1.
I can't imagine being a professional programmer until it's a System 1 activity at least half of the time.
This. In just about every field. It helps that in programming, and some of the sciences, that there is quick feedback. I have a physics degree but my job is fuzzier than that. After 4 years a lot of the time judgments start as "This doesn't sound right " and a lot of the work is reconstructing a communicable line of reasoning.
In this video, Douglas Crockford (JavaScript MASTER) says:
1
I don't think he has "absolutely no evidence". In worlds where DOUGLAS CROCKFORD has a gut feeling about something related to programming, how often does that gut feeling end up being correct? Probably a lot more than 50% of the time. So according to Bayes, his gut feeling is definitely evidence.
The problem isn't that he lacks evidence. It's that he lacks communicable evidence. He can't say "I believe A because X, Y and Z." The best he could do is say, "just trust me, I have a feeling about this".
Well, "just trust me, I have a feeling about this" does qualify as evidence if you have a good track record, but my point is that he can't communicate the rest of the evidence his brain used to produce the resulting belief.
2
How do you handle a situation where you're having a conversation with someone and they say, "I can't explain why I believe X; I just do."
Well, as far as updating beliefs, I think the best you could do is update on the track record of the person. I don't see any way around it. For example, you should update your beliefs when you hear Douglas Crockford say that he has a gut feeling about something related to programming. But I don't see how you could do any further updating of your beliefs. You can't actually see the evidence he used, so you can't use it to update your beliefs. If you do, the Bayes Police will come find you.
Perhaps it's also worth trying to dig the evidence out of the other persons subconscious.
3
Ok, now let's talk about what you shouldn't do. You shouldn't say, "Well if you can't provide any evidence, you shouldn't believe what you do." The problem with that statement is that it assumes that the person has "no evidence". This was addressed in Section 1. It's akin to saying, "Well Douglas Crockford, you're telling me that you believe X and you have a fantastic track record, but I don't know anything about why you believe it, so I'm not going to update my beliefs at all, and you shouldn't either."
Brains are weird and fantastic thingys. They process information and produce outputs in the form of beliefs (amongst other things). Sometimes they're nice and they say, "Ok Adam - here is what you believe, and here is why you believe it". Other times they're not so nice and the conversation goes like this:
Just because brains could be mean doesn't mean they should be discounted.