I think this is what many people find confusing about Bayesian reasoning: the nods to subjectivity. Some people I've discussed it with often say things like "But it's still subjective!" or "You're pulling those numbers out of your ass!" Well, yes, maybe. But calibration games, immediate-feedback systems like the Good Judgement Project, and general awareness of priors are being shown as better than chance, and better than intuition, when the RESULTS are objectively measured.
If I had to speculate on why people react so negatively, I'd say it's because of the false dichotomy between "objective" and "subjective." Objective: numbers, math, computer programs. Subjective: fuzzy, things that aren't science. So, saying that something can be evidence in a rational approach AND be subjective...is confusing. They aren't thinking about weighting the evidence accordingly. They're annoyed that it's counted at all.
In this video, Douglas Crockford (JavaScript MASTER) says:
1
I don't think he has "absolutely no evidence". In worlds where DOUGLAS CROCKFORD has a gut feeling about something related to programming, how often does that gut feeling end up being correct? Probably a lot more than 50% of the time. So according to Bayes, his gut feeling is definitely evidence.
The problem isn't that he lacks evidence. It's that he lacks communicable evidence. He can't say "I believe A because X, Y and Z." The best he could do is say, "just trust me, I have a feeling about this".
Well, "just trust me, I have a feeling about this" does qualify as evidence if you have a good track record, but my point is that he can't communicate the rest of the evidence his brain used to produce the resulting belief.
2
How do you handle a situation where you're having a conversation with someone and they say, "I can't explain why I believe X; I just do."
Well, as far as updating beliefs, I think the best you could do is update on the track record of the person. I don't see any way around it. For example, you should update your beliefs when you hear Douglas Crockford say that he has a gut feeling about something related to programming. But I don't see how you could do any further updating of your beliefs. You can't actually see the evidence he used, so you can't use it to update your beliefs. If you do, the Bayes Police will come find you.
Perhaps it's also worth trying to dig the evidence out of the other persons subconscious.
3
Ok, now let's talk about what you shouldn't do. You shouldn't say, "Well if you can't provide any evidence, you shouldn't believe what you do." The problem with that statement is that it assumes that the person has "no evidence". This was addressed in Section 1. It's akin to saying, "Well Douglas Crockford, you're telling me that you believe X and you have a fantastic track record, but I don't know anything about why you believe it, so I'm not going to update my beliefs at all, and you shouldn't either."
Brains are weird and fantastic thingys. They process information and produce outputs in the form of beliefs (amongst other things). Sometimes they're nice and they say, "Ok Adam - here is what you believe, and here is why you believe it". Other times they're not so nice and the conversation goes like this:
Just because brains could be mean doesn't mean they should be discounted.