how do we know if the 60 considered studies were testing the hypothesis that there was a link or the hypothesis that there was not a link?

I think the answer to this is "because they're using NHST." They say "we couldn't detect an effect at the level that random chance would give us 5% of the time, thus we are rather confident there is no effect." But that we don't see our 5% false positives suggests that something about the system is odd.

Which 5%?

No, that 5% is the probability of a false positive, [...]

No, "that" 5% is the probability from my cooked-up example, which was the probability of a false-negative.

You're saying (and Phil says also in several places) that in his example the 5% is the probability of a false positive. I don't disagree, a priori, but I would like to know, how do we know this? This is a necessary component of the full argument that seems to be missing so far.

(A) if the true conclusion is 'positive', any test can yield a negative with 5% probability.

No, that 5% is the probability of false positive, not the probability of false negative. Phil has the number he needs and uses it correctly.

I don't think that it's necessarily suspicious in that, a priori, I wouldn't have a problem with 60 tests all being negative even though they're all only 95% confident.

The reason being, depending on the nature of the test, the probability of a false negative might indeed be 5% while the probability of a false positive could be tiny. Suppose this is indeed the case and let's consider the two cases that the true answer is either 'positive' or 'negative'.

(A) if the true conclusion is 'positive', any test can yield a negative with 5% probability. (this test will be reported as a negative with 95% confidence, though one would expect most tests to yield the positive conclusion.)

(B) if the true conclusion is 'negative', any test that yields a negative will still be reported with the 95% confidence because of the possibility of case (A). Though if it is case (B), we should not expect any positive conclusion, even over 60 tests, because the false-positive rate is so low.

I have no idea if this lack of symmetry is the case for the set of MMR and autism studies. (It probably isn't -- so I apologize that I am probably accomplishing nothing but making it more difficult to argue what is likely a true intuition.)

But it is easy to think of an example where this asymmetry would apply: consider that you are searching for someone that you know well in a crowd, but you are not sure they are there. Consider a test to be looking for them over a 15 minute period, and you estimate that if they are there, you are likely to find them during that 15 minute period with 95% probability. Suppose they are there but you don't find them in 15 minutes -- that is a false negative with 5% probability. Supopse they are not there and you do not find them -- you again say they are not there with 95% probability. But in this case where they are not there, even if you have 60 people looking for them over 15 minutes, no one will find them because the probability of a false positive is pretty much zero.

(I do see where you addressed false positives versus false negatives in several places, so this explanation was not for you specifically since I know you are familiar with this. But it is not so clear which is which in these studies from the top, and it is fleshing this out that will ultimately make the argument more difficult, but more water-tight.)

How do you decide whether to count down or to count up?

The signature lines you're writing under your posts are confusing and caused me to follow your post history to understand what was going on, arriving here, so that I can now tell you adding a block of extra text to all your posts in an effort to be terse seems like a lost purpose.

This is one of the strangest posts I have ever read on Less Wrong..

50 comments 50 words or less. #48

Depending on opportunities in your field, academia may provide favorable amounts of freedom, job security and impact. However, for the quintessential academic, academia is not a calculated optimization but a personality type:

It’s awesome to be supported while you learn and think, if that’s what you wanted to do anyway.

50 comments 50 words or less. #49

I had already noticed I needed to adjust in this direction. I'm going to try being more concise and see how it goes.

Specifically, 50 words or less for 50 comments. (#50)

I interpreted it as indicating that there are multiple ways of thinking about the problem, some of which produce the right answer and some of which produce the wrong answer. There's an element of chance involved in which one the child happens to employ, and children who are farther along in their development are more likely but not certain to pick the correct one on any single trial.

"Acquiring a concept" is a little ambiguous of an expression - suppose there's some subsystem or module in the child's brain which has learned to apply the right logic and hits upon on the right answer each time, but that subsystem is only activated and applied to the task part of the time, and on other occasions other subsystems are applied instead. Maybe the brain has learned that this system/mode of thought is the right way to think about the issue in some situations, but it hasn't yet reliably learned to distinguish what those situations are.

Not sure how analogous this really is, but I'm reminded of the fact that IBM's Watson used a wide variety of algorithms for scoring possible answer candidates, and then used a metalearning algorithm for figuring out the algorithms whose outputs were the most predictive of the correct answer in different situations (i.e. doing model combination and adjustment). So it, too, had some algorithms which produced the right answer, but it didn't originally know which ones they were and when they should be applied.

That kind of an explanation would still be compatible with a sudden boost in math talent, if things suddenly clicked and the learner came to more reliably apply the correct ways of thinking. But I'm not entirely sure if it's necessarily a developmental thing, as opposed to just being a math-related skill that was acquired by practice. Jonah wrote:

Because there was substantial overlap in the algebraic techniques utilized in the different subjects I was studying, my exposure to them per day was higher, so that when I learned them, they stuck in my long-term memory.

And if there is a specific "recognize the situations that can be thought of in algebraic terms and where algebraic reasoning is appropriate" skill, for example, then simultaneously studying multiple different subjects employing the same algebraic techniques in different contexts sounds just like the kind of thing that would be good practice for it.