Based on our rational approach we are at a disadvantage for discovering these truths.
As I argued, assigning accurate (perhaps low, perhaps high) probabilities to the truth of such claims (of the general category which lucid dreaming falls into) does not make it harder -- not even a little harder -- to discover the truth about lucid dreaming. What makes it hard is the large number of similar but bogus claims to sift through, as well as the difficulty of lucid dreaming itself. Assigning an appropriate probability based on past experience with these sorts of claims only helps us because it allows us to make good decisions about how much of our time to spend investigating such claims.
What you seem to be missing (maybe?) is that we need to have a general policy which we can be satisfied with in "situations of this kind". You're saying that what we should really do is trust our friend who is telling us about lucid dreaming (and, in fact, I agree with that policy). But if it's rational for us to ascribe a really low probability (I don't think it is), that's because we see a lot of similar claims to this which turn out to be false. We can still try a lot of these things, with an experimental attitude, if the payoff of finding a true claim balances well against the number of false claims we expect to sift through in the process. However, we probably don't have the attention of looking at all such cases, which means we may miss lucid dreaming by accident. But this is not a flaw in the strategy; this is just a difficulty of the situation.
I'm frustrated because it seems like you are misunderstanding a part of the response Kindly and I are making, but you're doing a pretty good job of engaging with our replies and trying to sift out what you think and where you start disagreeing with our arguments. I'm just not quite sure yet where the gap between our views is.
I don't think there is a gap. I am pointing towards a difficulty. If you are acknowledging the difficulty (which you are) then we are in agreement. I am not sure why it feels like a disagreement, Don't forget that at the start you had a reason for disagreeing which was my erroneous use of the word rationality. I have now corrected that so maybe we are arguing from the momentum of our first disagreement :P
I would like to explore certain kinds of experiential knowledge that appear to me to be difficult to investigate rationally as the rational attitude itself might be the cause of a reluctance to explore. If this is already covered in one of the articles on the site please refer me to it.
In this thought experiment we will use the example of lucid dreaming. Lucid dreaming is a state in which a person realises they are dreaming while they are dreaming. The subtleties of the state are not relevant to this discussion.
Circumstances
[1] We will assume the experiment takes place at a time where the existence of the experience of lucid dreaming hasn't been scientifically proven yet. We will also assume that a proof is not possible in the current state of technological or methodological development.
[2] Person A has a (true) belief on the existence of lucid dreaming that is based on his personal experience of the state.
[3] He is trying to communicate the existence of lucid dreaming to someone else. Let us call him person B.
[4] Actually becoming lucid in a dream is quite a complex process that requires among other things1:
[4.1] Expending large amounts of effort.
[4.2] Following guidelines and exercises that appear strange.
[4.3] A time investment of significant length.
In the described circumstances we have an internal experience that has not be scientifically proven but is nevertheless true. We know this in our time through scientific studies but B does not know it in his world. Person B would have to actually believe in the existence of lucid dreaming and trust A to guide him through the process. But since there is no sufficient evidence to support the claim of A, the required effort is significantly large and the methods appear strange to those not understanding the state how can B rationally decide to expend the effort?
Proposed Conclusion
[5] People focusing on rational assessment can be mislead when dealing with experiential knowledge that is not yet scientifically proven, is not easily testable and has no obvious external function but is, nevertheless, experientially accessible.
1 Even if you disagree with the level of difficulty or the steps required please accept [4] and its sub-headings as being accurate for the duration of the argument.