= 27d567bc2d48d9f40e9ccc20ea370b15)
AlexSchell comments on Open Thread, May 26 - June 1, 2014 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (245)
Hello. I've been a lurker here for quite some time now, but this is the first time I am making an appearance. I would like to consult everyone here regarding what I perceive to be irrationality on my part. I hope that you will be patient towards me and refrain from downvoting out of irritation, as I would prefer not to have my comment hidden, since that would greatly reduce my chances of getting feedback.
The issue is this: while I am fully aware that anecdotes do not constitute data, I have a very difficult time believing that test preparation only has a modest positive effect (if any) on SAT scores, even though this has been noted by several studies. Such a finding is completely incongruent with my personal experiences growing up in an East Asian country where most students (regardless of their socioeconomic status) attend cram schools or hire after-school private tutors -- many of these students have managed to perform much better than they otherwise would have, due to the extra lessons and revision. (They usually go through several iterations of testing themselves using old SAT papers before sitting for the actual test, and there are often very significant gains -- sometimes as high as several hundred points -- in their performances.)
So, my request to my fellow LW readers is this: Please help me reconcile this jarring chasm between research findings and my own personal experiences. There might be some information I am missing that is contributing to my inability to believe the findings. Is there anyone here who has done extensive reading on this topic?
SAT is very g-loaded, so it would be susceptible to the same practice effect that IQ tests in general are susceptible to. If you look at SAT/IQ tables, the 20-40 point increase Tyler cites corresponds to about 1.5-3 IQ points. This is consistent with the typical magnitude (< 5 points) of the practice effects on IQ test scores. Your "hundreds of points" are wildly inconsistent with this. The only way I could see that happen is if quite a bit of the SAT would test for skills that can be practiced but don't correlate with g. Not very likely.
The way to reconcile your experience with the evidence is to note that the score on a low-stakes practice test is just not comparable to the score on the real thing (with or without test prep). It's not that implausible to believe that, say, 10% of people (more than enough to account for your anecdotes) will score at least 100 points less on an early practice test than they could score on the real thing at the same level of preparation. It's hard to trick your brain into believing that something is high-stakes when it isn't.
ETA: On reflection, the low-stakes hypothesis probably doesn't account for too much of the puzzle. In particular, it doesn't explain any gain between consecutive low-stakes practice tests. I think James Miller's explanation takes the cake. The SAT-g correlation is likely a lot lower for a population not proficient in English.
Not likely?? It's certain!
If you know the scoring rules and their implications like when to guess and when to leave it blank, that can get you points you might miss from leaving it blank and reduce your penalty on things you'd have gotten wrong.
If you know better how to manage your time, then you won't end up rushed.
Simply having done it before reduces the stress of the situation and can enable better focus.
Being familiar with the style of questions asked will help a lot - you'll know to expect certain odd phrasings that can trip up a naive test-taker, and in some cases you will barely need to parse, simply pattern-match. 'Yup, this is that kind of question.'
And that's setting aside just studying the words they're likely to ask you about.
None of these have all that much to do with g, and I can see them producing a swing of 40 points easily, perhaps more at the lower end (you know, in the case where there are hundreds of points to gain).
This isn't to say that intense SAT prep is a huge difference on average - it could end up inducing more freakout instead of less, it could induce someone to stay up late and not be rested, it may primarily be used by those who would already do well, it may be used as a crutch by those wouldn't... all sorts of confounding things. But the idea that there is no significant component of the SAT that's practicable non-g is hard to believe.
This comment is very insightful -- you managed to articulate a lot of non-g factors that would explain my own observations. Thank you.
You are right that a test being g-loaded is not inconsistent with test takers receiving significant gains from repeatedly taking this test. This is why formal IQ tests used by experts are not available to the general public because practicing them can significantly raise scores and so give an inaccurate estimate of IQ.
Edited because I misread the above comment.
This is one possibility. Thanks for bringing it up.