The value of a test A for learning about B is measured by the mutual information I(A;B). The tradeoff between this and how easy the test is to perform is left up to you.
Here is a brief overview of the subject. As far as notation goes: I want to distinguish the test A from its outcomes, which I will denote a and -a.
The information content I(a) from an outcome given by the formula I(a) = - log Pr[a]. (The log is often taken to be base 2, in which case the units of information are bits.) The formula is motivated by our desire that if tests A1, A2 are independent, I(a1 and a2) = I(a1) + I(a2); the information we gain from learning both outcomes at once is the sum of the information learned from each outcome separately.
The entropy of a test A is the expected information content from learning its outcome: H(A) = I(a) Pr[a] + I(-a) Pr[-a]. Intuitively, it measures our uncertainty about the outcome of A; it is maximized (at 1 bit) when a and -a are equally likely, and approaches 0 when either a or -a approaches certainty. Ultimately, H(B) is the parameter you're trying to reduce in this problem.
We can easily condition on an outcome: H(B|a) is given by replacing all probabilities with conditional ones. It is our (remaining) uncertainty about B if we learn that a was the outcome of test A.
The conditional entropy H(B|A) is the expected value H(B|a) Pr[A] + H(B|-a) Pr[-a]. In other words, this is the expected uncertainty remaining about B after performing test A.
Finally, the mutual information I(A;B) = H(B) - H(B|A) measures the reduction in uncertainty about B from performing test A. As a result, it is a measure of the value of test A for learning about B. Irrelevantly but cutely, it is symmetric: I(A;B) = I(B;A).
...so if I perform tests A and B simultaneously 25 times and out of 25 of them obtain Pr[a], Pr[-a], Pr[b] and Pr[-b] and calculate I(A;B), and THEN I look at the results for A26, I should be able to predict B26, right? And if I(A;B)>I(C;B)>I(D;B), then I take test A as the most useful predictor? But if the set from which the sample was taken is large, and probably heterogenous, and there might be other factors I haven't included in my analysis, then the test A might mislead me about the outcome of B. (Which will be Bayesian evidence, if it happens.) How many iterations should I run? Is there a rule of thumb? Thank you for such helpful answers.
I think it's past time for another Stupid Questions thread, so here we go.
This thread is for asking any questions that might seem obvious, tangential, silly or what-have-you. Please respect people trying to fix any ignorance they might have, rather than mocking that ignorance.