Comment Permalink

JonahS12y00

So, multiple choice questions are easier to hack than other kinds of questions, but if you "guess" the right answer 80% of the time, that's not really guessing.

It depends how obvious it is that some of the answers are wrong.

Vaniver12y30

It depends how obvious it is that some of the answers are wrong.

Obvious, of course, is a two-place word.

One experience I had that highlighted the importance of background knowledge for me was playing Alexei's Calibration Game. I had literally no knowledge about the sports questions- after confirming that, I just clicked A every time, and had 50% accuracy. But one of the categories of questions was "which of these postmasters general of the US served first?", which I was able to get 60-70% accuracy on, just by replacing that with the question &... (read more)

See in context

12

[Link] The Collapse of Complex Societies

2012-12-31T12:11:04.764Z

TGGP a frequent commenter at Overcoming Bias (and hence old LessWrong), writes about his thoughts on a book by Joseph Tainter . I’ve seen Joseph Tainter’s “The Collapse of Complex Societies” recommended in a few different places. Jared Diamond’s book might be one of them, the guest-posts of Captain David Ryan aka “Tony Comstock” for James Fallows at the Atlantic might be another. The sidebar of John Robb’s “Global Guerrillas” blog is the only one I remember with certainty. It’s a not a very long book, and you can get the gist of it from Tainter’s wikipedia page . Lots of people have found civilizational collapses to be interesting, and Tainter reviews many of their theories while finding them wanting. The “eleven major themes in the explanation of collapse” he lists are depletion/cessation of a vital resource, establishment of a new resource base (which I found too stupid to take seriously even momentarily), insurmountable catastrophe, insufficient response to circumstances (which is almost tautological), other complex societies, intruders, class conflict or elite mismanagement, social dysfunction, mystical factors, chance concatenation of events (almost tautological if you don’t think collapse is predetermined) and economic factors. Like Tainter, I find the “mystical” theories to not really constitute theories at all, although some of the most popular writers on the subject (Spengler, Toynbee, various ancients) are included there. Tainter often contrasts “integrative” (or “functional”) theories on the origin of the state/complexity vs “conflict” theories, and acknowledges that he is more partial toward the former. Unfortunately, most of the latter theorists he lists are Marxists and carry a lot of baggage. The observation that throughout much of history some set of people ruled over others as a result of military victory regardless of any benefit to the subjects (though a Leviathan may happen to have upsides) predates Marx, with Ibn Khaldun being one of the few non-Marxist examples Tainter mentions. That’s not to say Tainter is anti-Marxism, he actually compares Marxist “social science” to Einsteinian physics and Darwinian biology! I suppose there is (or was) just such a heavy representation of Marxists among academic anthropologists and historians that Tainter regards Marxism as somewhat normative, whereas to me it’s something weird and laughable like Holocaust revisionism. Resource depletion is the reverse of the theory I found so absurd, and (showing there is hope for humanity) it is a much more popular theory. J. Donald Hughes blamed Rome’s collapse in part on deforestation, but W. Groenman van Waateringe some years later provided evidence at the time that cereal pollens declined while forest pollens increased. That of course is not a causal proof, since it is documented that when the empire was declining many agricultural regions became depopulated. Waateringe blames agricultural intensification for increasing the population and thus the demands on agriculture, but to me that just raises the question of why marginal agricultural lands weren’t reclaimed. There actually is an explanation for that depopulation, but like Tainter I’m not going to get to that in a hurry. Tainter finds this theory (like most other theories he rejects) unsatisfactory because complex societies should have leaders who notice the depletion and think of a response. I am reminded of David St. Hubbin’s girlfriend in Spinal Tap who says “It’s just a problem! It get’s solved!” Sometimes a solution is not within a society’s feasible choice-set. Tainter briefly acknowledges that possibility (noting that it would have to be proved, which is difficult given how little information we have about many ancient societies) but spends more time castigating imaginary opponents depicted as claiming societal elites just stood around slack-jawed rather than attempting to deal with the situation. I would call that a strawman, except that Jared Diamond’s “Collapse” bears too much resemblance. He also mentions Richard Wilkinson’s documenting that deforestation spurred development in late/post medieval England, which is really just extra evidence that Hughes was wrong (as had already been mentioned) rather than a broader point against a class of theories. I should acknowledge that Tainter also cites evidence on the differential abandonment of cities and the failure to correlate with expected environmental characteristics, which is just the sort of thing that would later puncture Jared Diamond’s take on the Maya . His point that greed is constant enough (or its variance poorly enough understood) to make it a poor explanation of a variable situation is fair enough, but he can’t dismiss theories of collapse based on mismanagement because by their nature they should keep their society going, even if only out of self-interest. There are basic agency problems that mean one shouldn’t identify the interests of elite (or non-elite, for that matter) actors with that of a larger organization. In an uncertain world it also makes sense to discount the future (you or your dynasty might be replaced, might as well get what you can while you can). Steve Sailer once critiqued Diamond’s thesis by noting that societies tend to die from homicide rather than suicide , and lumping together two of Tainter’s rejected explanations would make for a very popular theory. Tainter, however, would exclude most cases clearly caused by another complex society because those involve absorbtion rather than collapse (indigenous populations thoroughly devastated by disease before Europeans even arrived would be exceptions). So his question is then why a complex society would succumb to less complex intruders. Sometimes it may not be so easy to disentangle the two scenarios, such as when the persian & eastern roman empires exhausted themselves fighting each other, leaving themselves open to the Muslim invaders exploding out of Arabia (although it was only the persians that succumbed in fairly short order, and Tainter wouldn’t consider that a collapse). Tainter says it is “unsatisfactory [...] that a recurrent process – collapse – is explained by a random variable, by historical accident”. If random numbers for that variable (I’m imagining a stochastic process with a threshold for collapse rather than a binary control variable) are constantly being generated over time, it shouldn’t be that unsatisfactory that they recur throughout history. Tainter does make the legitimate point that elites, with a number of Roman emperors being good examples, have often proved capable of dealing with barbarian intrusions. But there’s no guarantee that they will always be successful. He also wonders why invaders would “destroy those things which repay conquest”. The obvious answer is that, by the second law of thermodynamics, it’s very easy to break things, and that includes during the process of conquering & looting. Some relatively sophisticated barbarians may conquer a territory and leave much of the administrative apparatus intact to rule as before, others may have no particular interest (or competence) in being bound to a territory and collecting scraps of taxes from farmers. The collapse of Rome is probably the most famous example (at least to westerners) and forms one of his three case studies, paradigmatic of the most complex sort of society to collapse. I find it more enlightening than the others because, and call me a drunk looking under a lamppost if you will, it’s the most well documented. Among the things documented is that the proximate cause of collapse was invasion by various (mostly Germanic) barbarians. That is discussed extensively in Peter Heather’s “Empires and Barbarians” which I have discussed earlier . It’s because I read that one so recently that a few of Tainter’s remarks stuck out. Focusing on the internal soundness of a society and its affordable scale/complexity he writes “The Germanic kingdoms that succeeded Roman rule in the West were more successful at resisting invasions”. If he limited that claim to the particular kingdoms which survived past the dark ages, it would be rather tautological. But if he means germanic kingdoms generally, then it just doesn’t seem to be the case. They got invaded and replaced all the time, we just don’t remember the ones that died out. Part of the reason the Romans had such problems with barbarians is that one group would get invaded by another, and then start moving around and displacing other barbarians. The western Roman empire, in contrast, was able to survive many invasions before the last Roman emperor was toppled. Tainter portrays the formation of the empire as a process in which a territory is able to summon the resources to mobilize a force to conquer more territory to extract its resources, then rinse and repeat in a self-sustaining cycle until it expanded too far to get many marginal returns. There is some truth to that, but it overlooks the non-extractive aspect of Roman rule which increased productivity in conquered territories, thereby making those territories more attractive as a target for raiding. In Heather’s story, barbarian confederations on the border engaged in a process of competitive selection for the strength to hold an attractive position (for reasons of trade, raiding and diplomatic subsidy) and eventually the size and cohesion necessary to survive and settle within Roman territory. Focusing on the internals of the collapsed societies, he overlooks any dynamics occurring within outside societies that could give them the capability of defeating the imperial power. Heather’s account is similar to Peter Turchin’s in “ War and Peace and War ” except that, like Khaldun, Turchin focuses more on the softening effects of metropolitan decadence that renders old dynasties vulnerable to the hardened asabiya-endowed border marchers. Tainter’s two other case studies are the Mayan lowland citystates and Chaco canyon cliff-dwellers. The Mayans are less complex (or at least less well-documented, since the conquistadors destroyed many of the remaining documents) and the Chacoans even less so. He also used the Ik as an example of an extremely simple society that collapsed even further below the level of familial organization, but he didn’t discuss it all that much and I’m not sure how reliable primary source Colin Turnbull was ( supposedly they hadn’t been hunter-gatherers for centuries when their supposed “livelihood” of hunting was banned). The interesting thing about the Maya is that there were multiple relatively equivalent city-states rather than one dominant hegemon. Tainter includes them as a case study of collapse, even as he states elsewhere that collapse is not an option for “peer polity competition” because the weakening of one peer just invites conquest by another. Also, rather than devoting most of their resources under duress to a standing army (something documented in the Roman case) Tainter discusses the building of monuments as conspicuous consumption to demonstrate how powerful and brutal (per the depictions of torture) the city was, rather similar to the story Diamond tells. I don’t know what kind of evidence we have for the scale of their military expenditures, although we know they did war from time to time. There was no writing whatsoever in Chaco canyon, so we are left with the old archeological standby of potsherds and whatnot. Tainter does make the interesting point that the culture benefitted from uniting different ecological niches, with higher elevation territories having more agricultural productivity in cold wet years while lower elevation ones were more productive in warm dry ones. An economist would say that this diversified portfolio allowed for more consumption smoothing. However, I was confused by Tainter’s argument that as more outlier territories were incorporated diversity and gains from exchange went down. As long as the ratio between high and low places was stable, incorporating more territories should not cause any problems in that respect. Admittedly, this does mean that there are more viable subsets of communities that would be individually stable if they withdrew, which is indeed what he claims happened eventually. But he also seemed to be suggesting that the system overally was degrading its performance, without clearly stating whether an excess of a particular type of environment was upsetting the balance. Tainter’s theory to explain collapse is declining marginal returns. This is a common concept in economics, but it is normally used to understood how equilibrium can develop. Applied to a society, we would expect the declining returns to territorial expansion or administrative complexity (the former often requiring some degree of the latter) to result in eventual stasis rather than collapse. David Friedman has an interesting paper on how the advantages of taxing trade, land or labor gave rise to different equilibria for the sizes & boundaries of polities during the Roman, medieval and nationalist eras in Europe. In Friedman’s theory, each shift between eras resulted from some exogenous change rather being part of the internal logic of societies. Tainter relates some various interesting bits from C. Northcote Parkinson’s “Parkinson’s Law, and Other Studies in Administration”. For example, while “between 1914 and 1967, the number of capital ships in the British Navy declined by 78.9 percent, the number of officers and enlisted men by 32.9 percent, and the number of dockyard workers by 33.7 percent [...] the number of dockyard officials and clerks increased by 247 percent, and the number of Admiralty officals by 767 percent” (emphasis added). Mencius Moldbug would not be surprised to learn that “between 1935 and 1954 the number of officals in the British Colonial office increased by 447 percent” even though “the empire administered by these officials shrank considerably”. These examples are important because they do not demonstrate an increasingly large requirement of administrators for a marginal increase in size/complexity of an entity to be administered, but paying more for less. Parkinson’s explanation was bureaucratic self-serving, which Tainter rejects because he finds trends of increasing hierarchical specialization in the private sector. But because Tainter fails to distinguish between declining marginal returns (eventually reaching zero at a steady-state) and NEGATIVE returns he doesn’t specify whether the latter occurs in the private sector (though Karl Smith would not be surprised if it does for many publicly owned corporations whose shareholders would be better served by liquidation of assets). The growth of administration in higher education would also count, but as a heavily subsidized non-profit sector I can’t say it would qualify. At one point Tainter acknowledges “In many cases this increased, more costly complexity will yield no increased benefits, at other times the benefits will not be proportionate to costs” (emphasis in the original). This is precisely the question at issue of elite mismanagement or the out-of-control inertia of expanding administrative bureaucracies, but as noted he rejected Parkinson’s theory and mocks the idea of societies as runaway trains as self-evidently absurd. Instead he portrays collapse as a choice which is preferable once marginal returns have declined to a certain point. This didn’t entirely make sense to me, because if a society has accidentally shot part the point of zero marginal returns to one of negative returns, the sensible thing is just to reduce that marginal increase in complexity to return to the steady state with zero marginal returns. The Roman empire sometimes seemed to behave in such a manner, losing some territory and sticking with a more defensible and adminstrable domain (although in Heather’s account some of the lost territory was among the most agriculturally productive), although Tainter thinks the conquests of Britain & Dacia never paid for themselves. So why the path dependency so that changes are not simply reversible? There could be consumption of a not easily renewable resource, a sort of borrowing from the future that leaves future generations deeper in the red. This could happen with soil deterioration, though Tainter doesn’t discuss that much (odd, despite his focus on societies as means of managing sources of energy). His example of Roman emperors increasingly resorting to the debasing of the currency could count (by Diocletian’s time it collected taxes in kind rather than the currency it had rendered nearly worthless), as well as the selling of imperial land. The larger problem in Rome seemed to be an increasingly large portion of subjects who were citizens (both urban proletariat and squabbling elites) subject to fewer or no taxes, while marginal lands were abandoned by overtaxed farmers. An odd feature of the empire was that election officials had to cover the costs of their own office, and as expenses rose there were fewer wealthy people willing to come forward as candidates, until the position was made hereditary. It became obligatory to farm certain deserted lands, with peasants drafted by local city Senates, and Constantine made soldiery a hereditary profession (which required a number of new laws over time to deal with sons who’d rather not follow that career). Taxation of land was simplistic and did not vary based on its quality or yield, so a farmer of marginal land would often be better off working for the owner of a more productive territory and paying rent than failing to cover the taxes on his own plot. With agricultural labor becoming legally tied to the land, we can see the clear beginnings of serfdom and the manorial system. As mentioned, Tainter views the Roman collapse as a choice (as he does others), although of course accounts from the time were more apt to regard it as unfortunate failure or divine punishment. Interestingly, the “peer polity competition” that replaced Roman civilization is a situation he regards as invulnerable to collapse as opposed to absorbtion, and by removing that “option” he thinks this made peasants demand democratic representation. He acknowledges that this did not happen in the “Warring States” period of China, and instead the Confucian ideology of governance developed. He suggests “Perhaps participatory governance was simply not possible in ancient societies that were so much larger, demographically and territorially, than the Greek city-states”. Someone should have told James Madison (and I’m not being sarcastic). Interestingly enough, there was a civilization of Greek city states which did collapse, just as we’ve mentioned the lowland Maya doing. These are the Mycenaean Greeks who preceded the Dark Ages of Homer’s time. Their collapse is usually attributed to invasion by Dorian Greeks, but Tainter isn’t convinced there’s enough evidence for the Dorians’ presence. Because “ Collapse occurs, and can only occur, in a power vacuum ” (emphasis in the original) both the Mycenaean Greek and lowland Maya polities must have experienced simultaneous collapse. The choice of peasants may be limited to passively withdrawing support and just not working very hard (I’d have more to say on that if I’d read James Scott’s “Weapons of the Weak”), but even then I don’t think it’s a desirable outcome for peasants. I’ve mentioned Heather on the greater productivity of Roman territory, and what do you think happened to the masses when that productivity crashed? My understanding of the current archaelogical consensus is that the population crashed as well. Tainter talks about the malnourished skeletons of the peasantry as evidence for the undesirability of certain degrees of complexity, but we also know that English peasants ate better after so many of their peers died of the bubonic plague (it’s also known that peasants have poorer diets than hunter-gatherers, though in Darwinian terms you definitely want to be a farmer). As in James Scott’s account of highland southeast asia (which I don’t entirely buy) was there much cultural defection of the peasantry to the greener pastures outside civilization? Tainter writes that “In 378 [...] Balkan miners went over en masse to the Visigoths”, and that others wished to be conquered/liberated. It is precisely due to the risk of being conquered that he argues is the reason many societies don’t simply revert to a lower level of complexity “even if marginal returns are unfavorable”, but it’s unclear whether conquest is one of outcomes being factored into those marginal returns. Few people are going to read this book without speculating on their own complex society’s liability to collapse. John Robb and James Kunstler (along with some others in the “Peak Oil” camp) are going to place a high probability on it, while the Singularitarians have the opposite view. Globalization could mean the entire world is now in a state of “peer polity competition” but modern norms (and economic incentives ) against conquest and giving war a chance means “failed states” can keep failing for a long time without someone replacing the bad management. Tainter’s studied societies are also Malthusian agricultural ones, it’s hard to know if the same logic will generalize past the industrial revolution. In modern technological economies the costs and benefits of advances may not be simple increasing or declining curves. Robin Hanson doesn’t even consider nearly free energy (which would very important to Tainter) to be nearly as important as the replacement of most human labor by computers (since the latter takes up so much more of GDP). When Tainter was writing there was still just the slightest possibility of nuclear armageddon, now the most likely candidates for death by complexity are grey goo or an unstoppable manmade pandemic. My two cents are that collapse is unlikely in my lifetime, and that’s for the better considering how much worse things could be.

Advanced Placement exam cutoffs and superficial knowledge over deep knowledge

by JonahS

1st Sep 2013

2 min read

12

When I was in high school, I noticed is that it was possible to score the top mark on an Advanced Placement (AP) Exam by answering a relatively small portion of the questions correctly.

During my junior year, I self-studied calculus, and took the AP Calculus AB exam. I was very surprised that I scored a 5 (the top mark), because at the time when I took the exam, I didn't know some very basic things that were on the syllabus.

The College Board gives the raw score to AP score conversions for the exams that have been most recently released. The percentages needed to get a 5 are as follows:

Art History: 71%
Biology: 63%
Calculus AB: 63%
Calculus BC: 63%
Chemistry: 67%
Computer Science A: 77%
English Language and Composition: 75%
English Literature and Composition: 76%
Environmental Science: 71%
European History: 66%
French Language: 80%
German Language: 86%
Comparative Government & Politics: 70%
US Government and Politics: 77%
Human Geography: 61%
Latin: Vergil: 69%
Music Theory: 70%
Macroeconomics: 81%
Microeconomics: 83%
Physics B: 62%
Physics C: Mechanics: 55%
Physics C: Electricity and Magnetism: 59%
Psychology: 75%
Spanish Language: 78%
Spanish Literature: 76%
Statistics: 63%
US History: 61%
World History 64%

Conspicuously, these percentages are all well below the standard 90% needed for the top mark (A) in a high school or college course. Of course, the threshold of 90% that's typically used is arbitrary, and some courses have harder exams than others, but the fact that the above percentages are lower than 90% (and in some cases *much* lower) is a weak argument that the standards for getting a 5 on an AP exam are lenient (and in some cases *very* lenient).

On an object level, based on my experience taking the AP calculus exams as a high schooler, my experience teaching calculus for three years at University of Illinois, and my revisiting the exams, I think that students who score 90% on an AP calculus exam know the material very well, and that students who score 63% (the lowest percentage needed to get a 5) have only marginal knowledge of the material.

It's natural to ask what effect this has on incentives. Note that AP courses are taught to prepare students to get high marks on the AP exams, so that the grading scheme for AP exams propagates backward to the grading schemes for courses, which feed into college admissions. Taking many AP courses and performing just above the "5" level looks better than taking few AP courses and performing at the ~90% level.

So one might reasonably suppose that high schoolers who want to get into selective colleges focus on developing superficial knowledge of lots of academic subjects rather than deep knowledge of a smaller number of academic subjects. And indeed, this is precisely what I remember most students doing in high school.

To what degree does your own experience reflect this as well?
What are some other contexts in which this sort of thing occurs?
How much of a problem is this (if at all)?

EducationPracticalWorld Modeling

Personal Blog

12

Mentioned in

32LessWrong analytics (February 2009 to January 2017)

18High School, Human Capital, Signaling and College Admissions

9High school activities and medical school admissions

New Comment

22 comments, sorted by

top scoring

Click to highlight new comments since: Today at 12:45 AM

[-]JoshuaZ12y80

I don't think percentages in this sort of context are a good metric. People under time pressure or stress make a lot of mistakes often.

Now, as to the issue of superficial knowledge rather than deep knowledge, if this is the case (and I suspect it is although for reasons largely unrelated to the AP exam pressures), this shouldn't be a big deal. At the highschool level most kids have little idea what they enjoy or are genuinely talented at. Having them get a few tastes of more advanced material in a variety of subjects is therefore good. Also, at a practical level, actually getting kids at that point to have an in depth understanding of subjects is often difficult. For example in the calculus case, sequences and series are one of the last things taught, and they are substantially more abstract and are simply easier to teach when students have had more hands on experience with what calculus can do.

[-]JonahS12y00

I don't think percentages in this sort of context are a good metric. People under time pressure or stress make a lot of mistakes often.

Still, doesn't 63% for calculus seem low? I think that if you know all of the material like the back of your hand, you can get 85+% right even with mistakes.

At the highschool level most kids have little idea what they enjoy or are genuinely talented at. Having them get a few tastes of more advanced material in a variety of subjects is therefore good.

A possible concern is that what people learn might be too superficial for them to get even a "taste."

Also, at a practical level, actually getting kids at that point to have an in depth understanding of subjects is often difficult. For example in the calculus case, sequences and series are one of the last things taught, and they are substantially more abstract and are simply easier to teach when students have had more hands on experience with what calculus can do.

The case of sequences and series is interesting. My impression is that most students who get 5's on AP Calculus BC really don't understand the topic. If I were designing the curriculum, I think that I would pare down that portion of the course so that students got a more gentle introduction.

[-]kalium12y40

85% of what? I guarantee you that I can write a calculus exam, using only AP-level material, that is long enough or algebraically messy enough that you can't get 85% of it right in three hours (or whatever the actual time limit is). There is no fundamental reason the AP exams should have exactly the same difficulty curve (5% of students can get 100%, another 10% can get over 90%, another 30% can get over 80% or whatever) as a high school class.

[-]JonahS12y00

85% of what?

Of the points on the AP exam (based on examining the questions and scoring criteria myself).

[-]Price41247y40

I write this as a current highschool studnet enrolled in 6 ap classes. For some subjects such as World History or Human Geography the test is very comprhensive and teachers have no contention with teaching to the test. In World History we are required to write a DBQ (Document Based Question), LAQ (Long answer Question), as well as a slew of SAQ (Short Answer Questions). These questions require a significant amount of background knowledge to answer comprehensivly. In addition having low test cut-offs don't necessarily correlate to a high percentage of studnets actually receving a 5. In WHAP for example you cite the cut-off rate as 64% however, only 6.5% of students acheive this cutoff. I believe the reason for this low five rate is how comprhensive the test is in measusring the learning a students incurs throughout the year. I realize that I am very bias in wanting these tests to be comphrensive for my own benefit but I stronly believe in history and perhaps other subjects a 5 is acceptable even with these low cut-off rates. Because of how low the 5 rate is even with a low cutoff rate means those students contain a comprehenvise understanding of the material even if they are scoring barely above a 64%.

[-]VipulNaik12y30

Here's the combined score distribution data for all the Advanced Placement examinations.

There's a countervailing consideration: a low cutoff for a 5 means that one could in principle achieve that cutoff by understanding some subtopics of the syllabus very well and ignoring others entirely (for instance, many AP BC 5 students may be very thorough with derivatives and integrals and have marginal knowledge of series). This is most so if the test question are individually hard and the students know in advance what fraction of the test covers what topic(s). To the extent this is true, it points in the opposite direction to your concern once we consider specialization within subjects.

[-]Mushroom Boi5y20

I imagine the cutoffs are based on the percentage of students able to achieve a certain score. I wouldn't make a whole lot of sense to have the score for "5" set at 90% if only 1 in 200 students got above a 90%. For example, the national average on most biology written responses is well below 50% of the points, and considering that statistic, I think the lower cutoffs are quite reasonable.

[-][anonymous]12y20

My own personal experience from three years of being an undergraduate math instructor:

Typically during the first week of class, around 10% (six or seven students) will approach me asking if they really need to take the class if they got N on the AP Calculus exam, where N is usually greater than 3 and they rarely distinguish between AB and BC.

I encourage them to stay, because they're usually misjudging the difficulty of the course from the first week or so of classes. I can't recall any of them doing well. There's a lot of confounding factors in this account, however....

[-]Vaniver12y20

I don't remember exactly how many AP tests I took, or what I got on them- the number 13 seems to be available, and I remember getting mostly 4s and 5s. I do remember that I got a year and a half's worth of credits out of it (which is the other benefit of taking lots of different tests), and so had almost all of my 'core credits' out of the way when I stepped onto campus.

Conspicuously, these percentages are all well below the standard 90% needed for the top mark (A) in a high school or college course.

The 90% is generally after adjustment, and with taking into account lots of "effort" points which are generally viewed by professors and students as mostly free. Many of my hard science courses in the course of my undergraduate degree had average marks on exams in the 60% range, with the grades curved such that about 30% of the class would get an A, 40% a B, and so on. (I'm sure the percentages vary widely across classes and schools.)

In that view, a better question to ask is sometimes "what percentage of students get a 5 on the AP X test?". If you look at Calculus (in 2008), the scores were split roughly uniformly on the AB test, such that about 20% of students got a 5. On BC, about 40% of students got a 5 (which I would explain by students being much more selective in deciding to take BC).

[-]JonahS12y00

Nature doesn't grade on a curve :-) I agree that the AP grading isn't out of line with course grading at most colleges. But that doesn't mean that students are learning the material well.

[-][anonymous]12y20

Too true. Last year, I crammed seven AP tests (Calc BC, Env. Science, Psychology, Comp Sci A, Statistics, Biology, English Language) in two weeks. Got 5 in all but stat (A 4. I crammed this in eight hours the day before the test and started without even knowing what a binomial distribution was.). I knew the Calc and CS material cold, and I speak and write in decent English so maybe three were deserved. I had some knowledge of biology, and knew nothing about the other subjects apart from what trivia I had picked up on the net. And it's not just the cutoffs. I was scoring near perfect scores in practice tests for many of these subjects (Multiple choice sections, the essays were harder to game). The problems are simply too darn easy for people with lots of background knowledge. Eliminating obviously wrong answers usually got me the right answer even if I didn't know the material.

I suppose I'm an outlier and most people won't be able to manage this, but I feel that something's seriously broken with the tests when I score as qualified on semester or year-long courses after studying for maybe twenty hours at most on one subject.

I'm not sure how much of a problem this is though. Colleges are free to not accept AP tests, are they not? At worst, cheaters like me will have a temporary advantage in admissions until everyone gets their act together. I suppose it could get worse if colleges didn't act while the tests were being exploited to oblivion, but that points to deeper problems than just easy AP tests.

[-]Vaniver12y10

The problems are simply too darn easy for people with lots of background knowledge. Eliminating obviously wrong answers usually got me the right answer even if I didn't know the material.

So, multiple choice questions are easier to hack than other kinds of questions, but if you "guess" the right answer 80% of the time, that's not really guessing. I think you underestimate the degree to which lots of background knowledge implies lots of subject knowledge / how much autodidactic knowledge clever people pick up just by paying attention and being curious.

(My school didn't offer Econ classes, but I took both AP Econ tests with about a day's worth of prep with an exam book, and believe I got 5 on both of them. Not surprising for someone who read econ books for fun and argued about economics on the internet in his spare time!)

[-]JonahS12y00

So, multiple choice questions are easier to hack than other kinds of questions, but if you "guess" the right answer 80% of the time, that's not really guessing.

It depends how obvious it is that some of the answers are wrong.

[-]Vaniver12y30

It depends how obvious it is that some of the answers are wrong.

Obvious, of course, is a two-place word.

I doubt a recent Chinese immigrant to the US would be able to hit 60% accuracy with that approach, because they don't have that good a sense of what's "old-timey" in the US. (And that's with only two options!) I've seen a handful of prep tests for subjects I know very little about, and my guessing accuracy there is close to chance.

Another way to put this: with multiple choice tests, reversed stupidity is intelligence, and that's a failing of the test. But identifying stupidity is a skill that requires some expertise.

[-]JonahS12y40

But identifying stupidity is a skill that requires some expertise

But the expertise might not be in the area that's ostensibly being tested.

[-]Priya Keswani4y10

do you know if this still applies in 2021 or not? I suspect not, but I can't find raw score conversions like this for all the APs for this year. I'm taking AP Bio, AP Stats, and APUSH and wanted to know what the 5 cutoffs were.

[-]petraA8y10

Personally my largest problem with AP tests is the reliance on multiple choice. Even though there is a free response section, half of the score still comes from multiple choice questions. For most subjects I'd say that the free responses are by far a more accurate gauge of subject knowledge. It is much harder to fake your way out of a free response question with various test taking and guessing strategies. Additionally, not only is it easy to guess your way out of multiple choice questions you may not know, but it is also much easier to make simple mistakes as someone who knows the material and get something wrong, the AP has no idea why you got a multiple choice question incorrect. In contrast, free response favors a student who might be having a rough day / test but knows the material. They may not get all the points, but it is likely that their response will still demonstrate their knowledge. I am taking 3 AP exams this year, CS, Phys C, and BC and even though I have been coding for years and have never actually taken BC (I took AB last year and have studied the rest over the course of this year) CS is my most difficult practice test. This is because I tend to misread multiple choice questions and am not great at standardized testing so even though I get 90-100% on the free response every time, it is easy to get a run of multiple choice questions wrong. Running though code by hand and answering multiple choice questions is a very different skill set from real coding, and it is frustrating that this skill is weighted as 50% of your AP points. Of course knowing the material helps, but after a certain point, reading errors and knowledge gaps become hard to distinguish within a multiple choice test. The low cut offs merely indicate the college boards acknowledgment of it. They've chosen to give everyone the benefit of the doubt that the wrong answers were careless rather than knowledge based and as such most schools simply end up having to retest everyone anyways.

[-]kalium12y00

"Performing at the 90% level" doesn't mean anything unless you assume that the questions are precisely calibrated so that someone with a good understanding of the material will get about 90% of the questions right in the allowed amount of time. While this is very common in high school, in my five years of college I never took a class that used this scale. Why should you assign problems so easy that someone with almost no grasp of the material gets 60% of them right (D) and a decently smart person who studies a lot can be assured of 100% (A+)?

The question you should be asking is "Is it possible to get the percentage needed for a 5 without a good understanding of the subject?" An exam with five truly difficult problems, where you get a 5 if even one is fully solved, would be a good exam by this standard despite looking very bad by the standard you are using.

I agree that AP exams are often too easy and do not require a good understanding of the material, but

I was very surprised that I scored a 5 (the top mark), because at the time when I took the exam, I didn't know some very basic things that were on the syllabus.

is the only thing in your whole post that's really relevant to this.

[-]JonahS12y00

I agree that AP exams are often too easy and do not require a good understanding of the material, but [...] is the only thing in your whole post that's really relevant to this.

I also wrote:

"On an object level, based on my experience taking the AP calculus exams as a high schooler, my experience teaching calculus for three years at University of Illinois, and my revisiting the exams, I think that students who score 90% on an AP calculus exam know the material very well, and that students who score 63% (the lowest percentage needed to get a 5) have only marginal knowledge of the material."

[-]Ishaan12y00

It's natural to ask what effect this has on incentives. Note that AP courses are taught to prepare students to get high marks on the AP exams, so that the grading scheme for AP exams propagates backward to the grading schemes for courses, which feed into college admissions. Taking many AP courses and performing just above the "5" level looks better than taking few AP courses and performing at the ~90% level.

Oh, so is that why schools use grades rather than percentages - an attempt to disincentive overspecialization and/or reduce pressure to overachieve? I've always wondered about that.

A more sane way to do this would be to give a percentage, but obfuscate it above a certain level. As in, John is at 78%, Wang is at 79%, but Ajeet, Alisha, Antwan are at 80%+ and the exact number shall not be revealed.

This keeps the disadvantage of undue emphasis on cut-off numbers (the 1% point that distinguishes C+ from B-) to a minimum while retaining the desired effect on incentives. There would still be a disadvantage at the single cut-off point (since 79% is significantly worse than 80+%) but at least the harm is contained around just one cut off point instead of several.

[-]kalium12y10

Using percentages requires difficulty to be very precisely calibrated and requires you to throw out most of your range. If you make all your problems harder so I can now only do half rather than 100% in the allotted time, my grade should remain the same rather than going from an A to an F. Percentile is much more relevant than percentage, though it's not perfect either.

[-]Ishaan12y00

Percentile is good when you want to identify where an individual is with respect to peers. You use it when you don't care about cohort effects and stuff like that. If I need 10 doctors, and I want the best doctors, then I can take the top 10 percentile in a class of 100. Even if they are all shitty MCAT scorers, I can't do without doctors so I've got to take the best of what I have. Likewise, even if they are all amazing scorers, I can't possibly take them all, so I'll just take the best.

Percentage is good when you want to identify an absolute level of skill. For example, if you wanted to see which students in a class of 100 were ready for algebra, you'd want to measure their absolute addition/subtraction/multiplication skills, not relative skills. If everyone is ready, then everyone can go on to Algebra. Likewise if no one is ready, they can all stay behind. I don't care where people are with respect to peers - I only care where they are.

I suppose the types of situations where you would attempt obfuscation would usually be of the "percentile" variety.

Moderation Log