Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Advanced Placement exam cutoffs and superficial knowledge over deep knowledge

4 Post author: JonahSinick 01 September 2013 09:01PM

When I was in high school, I noticed is that it was possible to score the top mark on an Advanced Placement (AP) Exam by answering a relatively small portion of the questions correctly.

During my junior year, I self-studied calculus, and took the AP Calculus AB exam. I was very surprised that I scored a 5 (the top mark), because at the time when I took the exam, I didn't know some very basic things that were on the syllabus.

The College Board gives the raw score to AP score conversions for the exams that have been most recently released. The percentages needed to get a 5 are as follows:

  • Art History: 71%
  • Biology: 63%
  • Calculus AB: 63%
  • Calculus BC: 63%
  • Chemistry: 67%
  • Computer Science A: 77%
  • English Language and Composition: 75%
  • English Literature and Composition: 76%
  • Environmental Science: 71%
  • European History: 66%
  • French Language: 80%
  • German Language: 86%
  • Comparative Government & Politics: 70%
  • US Government and Politics: 77%
  • Human Geography: 61%
  • Latin: Vergil: 69%
  • Music Theory: 70%
  • Macroeconomics: 81%
  • Microeconomics: 83%
  • Physics B: 62%
  • Physics C: Mechanics: 55%
  • Physics C: Electricity and Magnetism: 59%
  • Psychology: 75%
  • Spanish Language: 78%
  • Spanish Literature: 76%
  • Statistics: 63%
  • US History: 61%
  • World History 64%
Conspicuously, these percentages are all well below the standard 90% needed for the top mark (A) in a high school or college course. Of course, the threshold of 90% that's typically used is arbitrary, and some courses have harder exams than others, but the fact that the above percentages are lower than 90% (and in some cases *much* lower) is a weak argument that the standards for getting a 5 on an AP exam are lenient (and in some cases *very* lenient).

On an object level, based on my experience taking the AP calculus exams as a high schooler, my experience teaching calculus for three years at University of Illinois, and my revisiting the exams, I think that students who score 90% on an AP calculus exam know the material very well, and that students who score 63% (the lowest percentage needed to get a 5) have only marginal knowledge of the material.

It's natural to ask what effect this has on incentives. Note that AP courses are taught to prepare students to get high marks on the AP exams, so that the grading scheme for AP exams propagates backward to the grading schemes for courses, which feed into college admissions. Taking many AP courses and performing just above the "5" level looks better than taking few AP courses and performing at the ~90% level.

So one might reasonably suppose that high schoolers who want to get into selective colleges focus on developing superficial knowledge of lots of academic subjects rather than deep knowledge of a smaller number of academic subjects. And indeed, this is precisely what I remember most students doing in high school.

  • To what degree does your own experience reflect this as well?
  • What are some other contexts in which this sort of thing occurs?
  • How much of a problem is this (if at all)?

Comments (18)

Comment author: JoshuaZ 02 September 2013 05:36:11AM 4 points [-]

I don't think percentages in this sort of context are a good metric. People under time pressure or stress make a lot of mistakes often.

Now, as to the issue of superficial knowledge rather than deep knowledge, if this is the case (and I suspect it is although for reasons largely unrelated to the AP exam pressures), this shouldn't be a big deal. At the highschool level most kids have little idea what they enjoy or are genuinely talented at. Having them get a few tastes of more advanced material in a variety of subjects is therefore good. Also, at a practical level, actually getting kids at that point to have an in depth understanding of subjects is often difficult. For example in the calculus case, sequences and series are one of the last things taught, and they are substantially more abstract and are simply easier to teach when students have had more hands on experience with what calculus can do.

Comment author: JonahSinick 02 September 2013 07:48:15PM 1 point [-]

I don't think percentages in this sort of context are a good metric. People under time pressure or stress make a lot of mistakes often.

Still, doesn't 63% for calculus seem low? I think that if you know all of the material like the back of your hand, you can get 85+% right even with mistakes.

At the highschool level most kids have little idea what they enjoy or are genuinely talented at. Having them get a few tastes of more advanced material in a variety of subjects is therefore good.

A possible concern is that what people learn might be too superficial for them to get even a "taste."

Also, at a practical level, actually getting kids at that point to have an in depth understanding of subjects is often difficult. For example in the calculus case, sequences and series are one of the last things taught, and they are substantially more abstract and are simply easier to teach when students have had more hands on experience with what calculus can do.

The case of sequences and series is interesting. My impression is that most students who get 5's on AP Calculus BC really don't understand the topic. If I were designing the curriculum, I think that I would pare down that portion of the course so that students got a more gentle introduction.

Comment author: kalium 05 September 2013 06:35:49PM 0 points [-]

85% of what? I guarantee you that I can write a calculus exam, using only AP-level material, that is long enough or algebraically messy enough that you can't get 85% of it right in three hours (or whatever the actual time limit is). There is no fundamental reason the AP exams should have exactly the same difficulty curve (5% of students can get 100%, another 10% can get over 90%, another 30% can get over 80% or whatever) as a high school class.

Comment author: JonahSinick 05 September 2013 07:02:13PM *  1 point [-]

85% of what?

Of the points on the AP exam (based on examining the questions and scoring criteria myself).

Comment author: [deleted] 04 September 2013 02:43:54PM 2 points [-]

My own personal experience from three years of being an undergraduate math instructor:

Typically during the first week of class, around 10% (six or seven students) will approach me asking if they really need to take the class if they got N on the AP Calculus exam, where N is usually greater than 3 and they rarely distinguish between AB and BC.

I encourage them to stay, because they're usually misjudging the difficulty of the course from the first week or so of classes. I can't recall any of them doing well. There's a lot of confounding factors in this account, however....

Comment author: SolveIt 02 September 2013 02:04:40PM *  2 points [-]

Too true. Last year, I crammed seven AP tests (Calc BC, Env. Science, Psychology, Comp Sci A, Statistics, Biology, English Language) in two weeks. Got 5 in all but stat (A 4. I crammed this in eight hours the day before the test and started without even knowing what a binomial distribution was.). I knew the Calc and CS material cold, and I speak and write in decent English so maybe three were deserved. I had some knowledge of biology, and knew nothing about the other subjects apart from what trivia I had picked up on the net. And it's not just the cutoffs. I was scoring near perfect scores in practice tests for many of these subjects (Multiple choice sections, the essays were harder to game). The problems are simply too darn easy for people with lots of background knowledge. Eliminating obviously wrong answers usually got me the right answer even if I didn't know the material.

I suppose I'm an outlier and most people won't be able to manage this, but I feel that something's seriously broken with the tests when I score as qualified on semester or year-long courses after studying for maybe twenty hours at most on one subject.

I'm not sure how much of a problem this is though. Colleges are free to not accept AP tests, are they not? At worst, cheaters like me will have a temporary advantage in admissions until everyone gets their act together. I suppose it could get worse if colleges didn't act while the tests were being exploited to oblivion, but that points to deeper problems than just easy AP tests.

Comment author: Vaniver 02 September 2013 06:43:23PM 2 points [-]

The problems are simply too darn easy for people with lots of background knowledge. Eliminating obviously wrong answers usually got me the right answer even if I didn't know the material.

So, multiple choice questions are easier to hack than other kinds of questions, but if you "guess" the right answer 80% of the time, that's not really guessing. I think you underestimate the degree to which lots of background knowledge implies lots of subject knowledge / how much autodidactic knowledge clever people pick up just by paying attention and being curious.

(My school didn't offer Econ classes, but I took both AP Econ tests with about a day's worth of prep with an exam book, and believe I got 5 on both of them. Not surprising for someone who read econ books for fun and argued about economics on the internet in his spare time!)

Comment author: JonahSinick 02 September 2013 07:51:19PM 1 point [-]

So, multiple choice questions are easier to hack than other kinds of questions, but if you "guess" the right answer 80% of the time, that's not really guessing.

It depends how obvious it is that some of the answers are wrong.

Comment author: Vaniver 02 September 2013 09:39:17PM 2 points [-]

It depends how obvious it is that some of the answers are wrong.

Obvious, of course, is a two-place word.

One experience I had that highlighted the importance of background knowledge for me was playing Alexei's Calibration Game. I had literally no knowledge about the sports questions- after confirming that, I just clicked A every time, and had 50% accuracy. But one of the categories of questions was "which of these postmasters general of the US served first?", which I was able to get 60-70% accuracy on, just by replacing that with the question "which of these two American names is more old-timey?"

I doubt a recent Chinese immigrant to the US would be able to hit 60% accuracy with that approach, because they don't have that good a sense of what's "old-timey" in the US. (And that's with only two options!) I've seen a handful of prep tests for subjects I know very little about, and my guessing accuracy there is close to chance.

Another way to put this: with multiple choice tests, reversed stupidity is intelligence, and that's a failing of the test. But identifying stupidity is a skill that requires some expertise.

Comment author: JonahSinick 02 September 2013 09:47:10PM 4 points [-]

But identifying stupidity is a skill that requires some expertise

But the expertise might not be in the area that's ostensibly being tested.

Comment author: VipulNaik 08 September 2013 05:35:53PM *  1 point [-]

Here's the combined score distribution data for all the Advanced Placement examinations.

There's a countervailing consideration: a low cutoff for a 5 means that one could in principle achieve that cutoff by understanding some subtopics of the syllabus very well and ignoring others entirely (for instance, many AP BC 5 students may be very thorough with derivatives and integrals and have marginal knowledge of series). This is most so if the test question are individually hard and the students know in advance what fraction of the test covers what topic(s). To the extent this is true, it points in the opposite direction to your concern once we consider specialization within subjects.

Comment author: Vaniver 02 September 2013 06:38:33PM 1 point [-]

I don't remember exactly how many AP tests I took, or what I got on them- the number 13 seems to be available, and I remember getting mostly 4s and 5s. I do remember that I got a year and a half's worth of credits out of it (which is the other benefit of taking lots of different tests), and so had almost all of my 'core credits' out of the way when I stepped onto campus.

Conspicuously, these percentages are all well below the standard 90% needed for the top mark (A) in a high school or college course.

The 90% is generally after adjustment, and with taking into account lots of "effort" points which are generally viewed by professors and students as mostly free. Many of my hard science courses in the course of my undergraduate degree had average marks on exams in the 60% range, with the grades curved such that about 30% of the class would get an A, 40% a B, and so on. (I'm sure the percentages vary widely across classes and schools.)

In that view, a better question to ask is sometimes "what percentage of students get a 5 on the AP X test?". If you look at Calculus (in 2008), the scores were split roughly uniformly on the AB test, such that about 20% of students got a 5. On BC, about 40% of students got a 5 (which I would explain by students being much more selective in deciding to take BC).

Comment author: JonahSinick 02 September 2013 07:49:58PM 1 point [-]

Nature doesn't grade on a curve :-) I agree that the AP grading isn't out of line with course grading at most colleges. But that doesn't mean that students are learning the material well.

Comment author: kalium 05 September 2013 06:19:07PM *  0 points [-]

"Performing at the 90% level" doesn't mean anything unless you assume that the questions are precisely calibrated so that someone with a good understanding of the material will get about 90% of the questions right in the allowed amount of time. While this is very common in high school, in my five years of college I never took a class that used this scale. Why should you assign problems so easy that someone with almost no grasp of the material gets 60% of them right (D) and a decently smart person who studies a lot can be assured of 100% (A+)?

The question you should be asking is "Is it possible to get the percentage needed for a 5 without a good understanding of the subject?" An exam with five truly difficult problems, where you get a 5 if even one is fully solved, would be a good exam by this standard despite looking very bad by the standard you are using.

I agree that AP exams are often too easy and do not require a good understanding of the material, but

I was very surprised that I scored a 5 (the top mark), because at the time when I took the exam, I didn't know some very basic things that were on the syllabus.

is the only thing in your whole post that's really relevant to this.

Comment author: JonahSinick 05 September 2013 07:01:30PM 1 point [-]

I agree that AP exams are often too easy and do not require a good understanding of the material, but [...] is the only thing in your whole post that's really relevant to this.

I also wrote:

"On an object level, based on my experience taking the AP calculus exams as a high schooler, my experience teaching calculus for three years at University of Illinois, and my revisiting the exams, I think that students who score 90% on an AP calculus exam know the material very well, and that students who score 63% (the lowest percentage needed to get a 5) have only marginal knowledge of the material."

Comment author: Ishaan 03 September 2013 06:26:42AM *  -1 points [-]

It's natural to ask what effect this has on incentives. Note that AP courses are taught to prepare students to get high marks on the AP exams, so that the grading scheme for AP exams propagates backward to the grading schemes for courses, which feed into college admissions. Taking many AP courses and performing just above the "5" level looks better than taking few AP courses and performing at the ~90% level.

Oh, so is that why schools use grades rather than percentages - an attempt to disincentive overspecialization and/or reduce pressure to overachieve? I've always wondered about that.

A more sane way to do this would be to give a percentage, but obfuscate it above a certain level. As in, John is at 78%, Wang is at 79%, but Ajeet, Alisha, Antwan are at 80%+ and the exact number shall not be revealed.

This keeps the disadvantage of undue emphasis on cut-off numbers (the 1% point that distinguishes C+ from B-) to a minimum while retaining the desired effect on incentives. There would still be a disadvantage at the single cut-off point (since 79% is significantly worse than 80+%) but at least the harm is contained around just one cut off point instead of several.

Comment author: kalium 05 September 2013 06:27:38PM 1 point [-]

Using percentages requires difficulty to be very precisely calibrated and requires you to throw out most of your range. If you make all your problems harder so I can now only do half rather than 100% in the allotted time, my grade should remain the same rather than going from an A to an F. Percentile is much more relevant than percentage, though it's not perfect either.

Comment author: Ishaan 05 September 2013 11:16:51PM *  -1 points [-]

Percentile is good when you want to identify where an individual is with respect to peers. You use it when you don't care about cohort effects and stuff like that. If I need 10 doctors, and I want the best doctors, then I can take the top 10 percentile in a class of 100. Even if they are all shitty MCAT scorers, I can't do without doctors so I've got to take the best of what I have. Likewise, even if they are all amazing scorers, I can't possibly take them all, so I'll just take the best.

Percentage is good when you want to identify an absolute level of skill. For example, if you wanted to see which students in a class of 100 were ready for algebra, you'd want to measure their absolute addition/subtraction/multiplication skills, not relative skills. If everyone is ready, then everyone can go on to Algebra. Likewise if no one is ready, they can all stay behind. I don't care where people are with respect to peers - I only care where they are.

I suppose the types of situations where you would attempt obfuscation would usually be of the "percentile" variety.