LESSWRONG
LW

Comment Permalink

[anonymous]12y20

Too true. Last year, I crammed seven AP tests (Calc BC, Env. Science, Psychology, Comp Sci A, Statistics, Biology, English Language) in two weeks. Got 5 in all but stat (A 4. I crammed this in eight hours the day before the test and started without even knowing what a binomial distribution was.). I knew the Calc and CS material cold, and I speak and write in decent English so maybe three were deserved. I had some knowledge of biology, and knew nothing about the other subjects apart from what trivia I had picked up on the net. And it's not just the cutoffs. I was scoring near perfect scores in practice tests for many of these subjects (Multiple choice sections, the essays were harder to game). The problems are simply too darn easy for people with lots of background knowledge. Eliminating obviously wrong answers usually got me the right answer even if I didn't know the material.

I suppose I'm an outlier and most people won't be able to manage this, but I feel that something's seriously broken with the tests when I score as qualified on semester or year-long courses after studying for maybe twenty hours at most on one subject.

I'm not sure how much of a problem this is though. Colleges are free to not accept AP tests, are they not? At worst, cheaters like me will have a temporary advantage in admissions until everyone gets their act together. I suppose it could get worse if colleges didn't act while the tests were being exploited to oblivion, but that points to deeper problems than just easy AP tests.

The problems are simply too darn easy for people with lots of background knowledge. Eliminating obviously wrong answers usually got me the right answer even if I didn't know the material.

So, multiple choice questions are easier to hack than other kinds of questions, but if you "guess" the right answer 80% of the time, that's not really guessing. I think you underestimate the degree to which lots of background knowledge implies lots of subject knowledge / how much autodidactic knowledge clever people pick up just by paying attention and being c... (read more)

12

Advanced Placement exam cutoffs and superficial knowledge over deep knowledge

1st Sep 2013

2 min read

12

When I was in high school, I noticed is that it was possible to score the top mark on an Advanced Placement (AP) Exam by answering a relatively small portion of the questions correctly.

During my junior year, I self-studied calculus, and took the AP Calculus AB exam. I was very surprised that I scored a 5 (the top mark), because at the time when I took the exam, I didn't know some very basic things that were on the syllabus.

The College Board gives the raw score to AP score conversions for the exams that have been most recently released. The percentages needed to get a 5 are as follows:

Art History: 71%
Biology: 63%
Calculus AB: 63%
Calculus BC: 63%
Chemistry: 67%
Computer Science A: 77%
English Language and Composition: 75%
English Literature and Composition: 76%
Environmental Science: 71%
European History: 66%
French Language: 80%
German Language: 86%
Comparative Government & Politics: 70%
US Government and Politics: 77%
Human Geography: 61%
Latin: Vergil: 69%
Music Theory: 70%
Macroeconomics: 81%
Microeconomics: 83%
Physics B: 62%
Physics C: Mechanics: 55%
Physics C: Electricity and Magnetism: 59%
Psychology: 75%
Spanish Language: 78%
Spanish Literature: 76%
Statistics: 63%
US History: 61%
World History 64%

Conspicuously, these percentages are all well below the standard 90% needed for the top mark (A) in a high school or college course. Of course, the threshold of 90% that's typically used is arbitrary, and some courses have harder exams than others, but the fact that the above percentages are lower than 90% (and in some cases *much* lower) is a weak argument that the standards for getting a 5 on an AP exam are lenient (and in some cases *very* lenient).

On an object level, based on my experience taking the AP calculus exams as a high schooler, my experience teaching calculus for three years at University of Illinois, and my revisiting the exams, I think that students who score 90% on an AP calculus exam know the material very well, and that students who score 63% (the lowest percentage needed to get a 5) have only marginal knowledge of the material.

It's natural to ask what effect this has on incentives. Note that AP courses are taught to prepare students to get high marks on the AP exams, so that the grading scheme for AP exams propagates backward to the grading schemes for courses, which feed into college admissions. Taking many AP courses and performing just above the "5" level looks better than taking few AP courses and performing at the ~90% level.

So one might reasonably suppose that high schoolers who want to get into selective colleges focus on developing superficial knowledge of lots of academic subjects rather than deep knowledge of a smaller number of academic subjects. And indeed, this is precisely what I remember most students doing in high school.

To what degree does your own experience reflect this as well?
What are some other contexts in which this sort of thing occurs?
How much of a problem is this (if at all)?

EducationPracticalWorld Modeling

12

Mentioned in

32LessWrong analytics (February 2009 to January 2017)

18High School, Human Capital, Signaling and College Admissions

9High school activities and medical school admissions

New Comment

22 comments, sorted by

Click to highlight new comments since: Today at 4:43 AM

[-]JoshuaZ12y80

I don't think percentages in this sort of context are a good metric. People under time pressure or stress make a lot of mistakes often.

Now, as to the issue of superficial knowledge rather than deep knowledge, if this is the case (and I suspect it is although for reasons largely unrelated to the AP exam pressures), this shouldn't be a big deal. At the highschool level most kids have little idea what they enjoy or are genuinely talented at. Having them get a few tastes of more advanced material in a variety of subjects is therefore good. Also, at a practical level, actually getting kids at that point to have an in depth understanding of subjects is often difficult. For example in the calculus case, sequences and series are one of the last things taught, and they are substantially more abstract and are simply easier to teach when students have had more hands on experience with what calculus can do.

I don't think percentages in this sort of context are a good metric. People under time pressure or stress make a lot of mistakes often.

Still, doesn't 63% for calculus seem low? I think that if you know all of the material like the back of your hand, you can get 85+% right even with mistakes.

At the highschool level most kids have little idea what they enjoy or are genuinely talented at. Having them get a few tastes of more advanced material in a variety of subjects is therefore good.

A possible concern is that what people learn might be too superficial for them to get even a "taste."

Also, at a practical level, actually getting kids at that point to have an in depth understanding of subjects is often difficult. For example in the calculus case, sequences and series are one of the last things taught, and they are substantially more abstract and are simply easier to teach when students have had more hands on experience with what calculus can do.

The case of sequences and series is interesting. My impression is that most students who get 5's on AP Calculus BC really don't understand the topic. If I were designing the curriculum, I think that I would pare down that portion of the course so that students got a more gentle introduction.

85% of what? I guarantee you that I can write a calculus exam, using only AP-level material, that is long enough or algebraically messy enough that you can't get 85% of it right in three hours (or whatever the actual time limit is). There is no fundamental reason the AP exams should have exactly the same difficulty curve (5% of students can get 100%, another 10% can get over 90%, another 30% can get over 80% or whatever) as a high school class.

85% of what?

Of the points on the AP exam (based on examining the questions and scoring criteria myself).

[-]Price41247y40

I write this as a current highschool studnet enrolled in 6 ap classes. For some subjects such as World History or Human Geography the test is very comprhensive and teachers have no contention with teaching to the test. In World History we are required to write a DBQ (Document Based Question), LAQ (Long answer Question), as well as a slew of SAQ (Short Answer Questions). These questions require a significant amount of background knowledge to answer comprehensivly. In addition having low test cut-offs don't necessarily correlate to a high percentage of studnets actually receving a 5. In WHAP for example you cite the cut-off rate as 64% however, only 6.5% of students acheive this cutoff. I believe the reason for this low five rate is how comprhensive the test is in measusring the learning a students incurs throughout the year. I realize that I am very bias in wanting these tests to be comphrensive for my own benefit but I stronly believe in history and perhaps other subjects a 5 is acceptable even with these low cut-off rates. Because of how low the 5 rate is even with a low cutoff rate means those students contain a comprehenvise understanding of the material even if they are scoring barely above a 64%.

[-]VipulNaik12y30

Here's the combined score distribution data for all the Advanced Placement examinations.

There's a countervailing consideration: a low cutoff for a 5 means that one could in principle achieve that cutoff by understanding some subtopics of the syllabus very well and ignoring others entirely (for instance, many AP BC 5 students may be very thorough with derivatives and integrals and have marginal knowledge of series). This is most so if the test question are individually hard and the students know in advance what fraction of the test covers what topic(s). To the extent this is true, it points in the opposite direction to your concern once we consider specialization within subjects.

[-]Mushroom Boi5y20

I imagine the cutoffs are based on the percentage of students able to achieve a certain score. I wouldn't make a whole lot of sense to have the score for "5" set at 90% if only 1 in 200 students got above a 90%. For example, the national average on most biology written responses is well below 50% of the points, and considering that statistic, I think the lower cutoffs are quite reasonable.

[-][anonymous]12y20

My own personal experience from three years of being an undergraduate math instructor:

Typically during the first week of class, around 10% (six or seven students) will approach me asking if they really need to take the class if they got N on the AP Calculus exam, where N is usually greater than 3 and they rarely distinguish between AB and BC.

I encourage them to stay, because they're usually misjudging the difficulty of the course from the first week or so of classes. I can't recall any of them doing well. There's a lot of confounding factors in this account, however....

[-]Vaniver12y20

I don't remember exactly how many AP tests I took, or what I got on them- the number 13 seems to be available, and I remember getting mostly 4s and 5s. I do remember that I got a year and a half's worth of credits out of it (which is the other benefit of taking lots of different tests), and so had almost all of my 'core credits' out of the way when I stepped onto campus.

Conspicuously, these percentages are all well below the standard 90% needed for the top mark (A) in a high school or college course.

The 90% is generally after adjustment, and with taking into account lots of "effort" points which are generally viewed by professors and students as mostly free. Many of my hard science courses in the course of my undergraduate degree had average marks on exams in the 60% range, with the grades curved such that about 30% of the class would get an A, 40% a B, and so on. (I'm sure the percentages vary widely across classes and schools.)

In that view, a better question to ask is sometimes "what percentage of students get a 5 on the AP X test?". If you look at Calculus (in 2008), the scores were split roughly uniformly on the AB test, such that about 20% of students got a 5. On BC, about 40% of students got a 5 (which I would explain by students being much more selective in deciding to take BC).

Nature doesn't grade on a curve :-) I agree that the AP grading isn't out of line with course grading at most colleges. But that doesn't mean that students are learning the material well.

[-][anonymous]12y20

Too true. Last year, I crammed seven AP tests (Calc BC, Env. Science, Psychology, Comp Sci A, Statistics, Biology, English Language) in two weeks. Got 5 in all but stat (A 4. I crammed this in eight hours the day before the test and started without even knowing what a binomial distribution was.). I knew the Calc and CS material cold, and I speak and write in decent English so maybe three were deserved. I had some knowledge of biology, and knew nothing about the other subjects apart from what trivia I had picked up on the net. And it's not just the cutoffs. I was scoring near perfect scores in practice tests for many of these subjects (Multiple choice sections, the essays were harder to game). The problems are simply too darn easy for people with lots of background knowledge. Eliminating obviously wrong answers usually got me the right answer even if I didn't know the material.

I suppose I'm an outlier and most people won't be able to manage this, but I feel that something's seriously broken with the tests when I score as qualified on semester or year-long courses after studying for maybe twenty hours at most on one subject.

I'm not sure how much of a problem this is though. Colleges are free to not accept AP tests, are they not? At worst, cheaters like me will have a temporary advantage in admissions until everyone gets their act together. I suppose it could get worse if colleges didn't act while the tests were being exploited to oblivion, but that points to deeper problems than just easy AP tests.

[-]Vaniver12y10

The problems are simply too darn easy for people with lots of background knowledge. Eliminating obviously wrong answers usually got me the right answer even if I didn't know the material.

So, multiple choice questions are easier to hack than other kinds of questions, but if you "guess" the right answer 80% of the time, that's not really guessing. I think you underestimate the degree to which lots of background knowledge implies lots of subject knowledge / how much autodidactic knowledge clever people pick up just by paying attention and being curious.

(My school didn't offer Econ classes, but I took both AP Econ tests with about a day's worth of prep with an exam book, and believe I got 5 on both of them. Not surprising for someone who read econ books for fun and argued about economics on the internet in his spare time!)

So, multiple choice questions are easier to hack than other kinds of questions, but if you "guess" the right answer 80% of the time, that's not really guessing.

It depends how obvious it is that some of the answers are wrong.

[-]Vaniver12y30

It depends how obvious it is that some of the answers are wrong.

Obvious, of course, is a two-place word.

One experience I had that highlighted the importance of background knowledge for me was playing Alexei's Calibration Game. I had literally no knowledge about the sports questions- after confirming that, I just clicked A every time, and had 50% accuracy. But one of the categories of questions was "which of these postmasters general of the US served first?", which I was able to get 60-70% accuracy on, just by replacing that with the question "which of these two American names is more old-timey?"

I doubt a recent Chinese immigrant to the US would be able to hit 60% accuracy with that approach, because they don't have that good a sense of what's "old-timey" in the US. (And that's with only two options!) I've seen a handful of prep tests for subjects I know very little about, and my guessing accuracy there is close to chance.

Another way to put this: with multiple choice tests, reversed stupidity is intelligence, and that's a failing of the test. But identifying stupidity is a skill that requires some expertise.

But identifying stupidity is a skill that requires some expertise

But the expertise might not be in the area that's ostensibly being tested.

[-]Priya Keswani4y10

do you know if this still applies in 2021 or not? I suspect not, but I can't find raw score conversions like this for all the APs for this year. I'm taking AP Bio, AP Stats, and APUSH and wanted to know what the 5 cutoffs were.

Personally my largest problem with AP tests is the reliance on multiple choice. Even though there is a free response section, half of the score still comes from multiple choice questions. For most subjects I'd say that the free responses are by far a more accurate gauge of subject knowledge. It is much harder to fake your way out of a free response question with various test taking and guessing strategies. Additionally, not only is it easy to guess your way out of multiple choice questions you may not know, but it is also much easier to make simple mistakes as someone who knows the material and get something wrong, the AP has no idea why you got a multiple choice question incorrect. In contrast, free response favors a student who might be having a rough day / test but knows the material. They may not get all the points, but it is likely that their response will still demonstrate their knowledge. I am taking 3 AP exams this year, CS, Phys C, and BC and even though I have been coding for years and have never actually taken BC (I took AB last year and have studied the rest over the course of this year) CS is my most difficult practice test. This is because I tend to misread multiple choice questions and am not great at standardized testing so even though I get 90-100% on the free response every time, it is easy to get a run of multiple choice questions wrong. Running though code by hand and answering multiple choice questions is a very different skill set from real coding, and it is frustrating that this skill is weighted as 50% of your AP points. Of course knowing the material helps, but after a certain point, reading errors and knowledge gaps become hard to distinguish within a multiple choice test. The low cut offs merely indicate the college boards acknowledgment of it. They've chosen to give everyone the benefit of the doubt that the wrong answers were careless rather than knowledge based and as such most schools simply end up having to retest everyone anyways.

"Performing at the 90% level" doesn't mean anything unless you assume that the questions are precisely calibrated so that someone with a good understanding of the material will get about 90% of the questions right in the allowed amount of time. While this is very common in high school, in my five years of college I never took a class that used this scale. Why should you assign problems so easy that someone with almost no grasp of the material gets 60% of them right (D) and a decently smart person who studies a lot can be assured of 100% (A+)?

The question you should be asking is "Is it possible to get the percentage needed for a 5 without a good understanding of the subject?" An exam with five truly difficult problems, where you get a 5 if even one is fully solved, would be a good exam by this standard despite looking very bad by the standard you are using.

I agree that AP exams are often too easy and do not require a good understanding of the material, but

I was very surprised that I scored a 5 (the top mark), because at the time when I took the exam, I didn't know some very basic things that were on the syllabus.

is the only thing in your whole post that's really relevant to this.

I agree that AP exams are often too easy and do not require a good understanding of the material, but [...] is the only thing in your whole post that's really relevant to this.

I also wrote:

"On an object level, based on my experience taking the AP calculus exams as a high schooler, my experience teaching calculus for three years at University of Illinois, and my revisiting the exams, I think that students who score 90% on an AP calculus exam know the material very well, and that students who score 63% (the lowest percentage needed to get a 5) have only marginal knowledge of the material."

It's natural to ask what effect this has on incentives. Note that AP courses are taught to prepare students to get high marks on the AP exams, so that the grading scheme for AP exams propagates backward to the grading schemes for courses, which feed into college admissions. Taking many AP courses and performing just above the "5" level looks better than taking few AP courses and performing at the ~90% level.

Oh, so is that why schools use grades rather than percentages - an attempt to disincentive overspecialization and/or reduce pressure to overachieve? I've always wondered about that.

A more sane way to do this would be to give a percentage, but obfuscate it above a certain level. As in, John is at 78%, Wang is at 79%, but Ajeet, Alisha, Antwan are at 80%+ and the exact number shall not be revealed.

This keeps the disadvantage of undue emphasis on cut-off numbers (the 1% point that distinguishes C+ from B-) to a minimum while retaining the desired effect on incentives. There would still be a disadvantage at the single cut-off point (since 79% is significantly worse than 80+%) but at least the harm is contained around just one cut off point instead of several.

Using percentages requires difficulty to be very precisely calibrated and requires you to throw out most of your range. If you make all your problems harder so I can now only do half rather than 100% in the allotted time, my grade should remain the same rather than going from an A to an F. Percentile is much more relevant than percentage, though it's not perfect either.

Percentile is good when you want to identify where an individual is with respect to peers. You use it when you don't care about cohort effects and stuff like that. If I need 10 doctors, and I want the best doctors, then I can take the top 10 percentile in a class of 100. Even if they are all shitty MCAT scorers, I can't do without doctors so I've got to take the best of what I have. Likewise, even if they are all amazing scorers, I can't possibly take them all, so I'll just take the best.

Percentage is good when you want to identify an absolute level of skill. For example, if you wanted to see which students in a class of 100 were ready for algebra, you'd want to measure their absolute addition/subtraction/multiplication skills, not relative skills. If everyone is ready, then everyone can go on to Algebra. Likewise if no one is ready, they can all stay behind. I don't care where people are with respect to peers - I only care where they are.

I suppose the types of situations where you would attempt obfuscation would usually be of the "percentile" variety.

More from JonahS

Curated and popular this week