This is probably very arrogant of me to say, but my advice would be: "Listen to the domain expert when he tells you what you should do... and then find a Bayesian and let them explain to you why that works."

In my defense, this was my personal experience with statistics at school. I was very good at math in general, but statistics somehow didn't "click". I always had this feeling as if what was explained was built on some implicit assumptions that no one ever mentioned explicitly, so unlike with the rest of the math, I had no other choice here but to memorize that in a situation x you should do y, because, uhm, that's what my teachers told me to do. -- More than ten years later, I read LW, and here I am told that yes, the statistics that I was taught does have implicit assumptions, and suddenly it all makes sense. And it makes me very angry that no one told me this stuff at school. -- I am a "deep learner" (this, not this), and I have problem learning something when I am told how, but I can't find out why. Most people probably don't have a problem with this, they are told how, and they do, and can be quite successful with it; and probably later they will also get an idea of why. But I need to understand the stuff from the very beginning, otherwise I can't do it well. Telling me to trust a domain expert does not help; I may put a big confidence in how, but I still don't know why.