Comment on Problems in Education

Fri, 12 Apr 2013 16:16:27 GMT

Only if you make the very strong assumptions that there is no systematic bias or selection effect or regression to the mean or anything which might cause the unstability to favor an increase.

What effects could cause an increase of 8 points on a properly normed test across the board? Why would there a significant benefit to being in the control group of this study?

Plus we already know from the pairs of before-afters that these researchers are either incredibly incompetent or actively dishonest.

You can rule out that they were using a test which produced the scores that they recorded, perhaps by using raw score rather than normed output. You can rule out every other explanation for why the recorded results aren't valid scores. You can even rule out that they were competently dishonest, since competent dishonesty would be nontrivial to detect; your only possible conclusion is incompetence, which isn't evidence which should change your priors.

Incompetence is the social equivalent of the null hypothesis, and there is very rarely any significant evidence against it.

Therefore, the IQ test used simply wasn't properly normed; if we assume that it was equally improperly normed for all students in the study,

Which claim and assumption we will make because we are terminally optimistic, and to borrow from the '90s, "I want to believe!"

Assuming only incompetence as you have, the expected result would be equally erratic for all students. You can assign any likelihood to the assumption that the incompetence was the primary factor and that dishonesty doesn't modify it significantly, but you have already concluded systemic incompetent dishonesty across a large number of studies.

Wow, you still aren't giving up on the Pygmalion study? Just let it go already. You don't even have to give up on your wish for self-fulfilling expectations - there are plenty of followup studies which turned in your desired significant effects.

As you say, it's been confirmed by other studies. I'm not insisting that a particular study was done correctly, I'm explaining why their conclusions being true is consistent with the errors in their study. (Which means that a study with those flaws would be expected to reach the same conclusions, if those conclusions were true)

Discuss