LESSWRONG
LW

Comment Permalink

But how do those skew the result? It's not enough to say that there's bias; you would have to show that the bias works against the claimed conclusion. So with respect to your points:

1) If you don't have a reason to believe the bias skews one way or the other, knowledge that it could be higher or lower doesn't matter.

2) The wealthier you are, the easier you are to track down, so the survey would tend to over-state the average income.

3) True, but again, do you have a reason to believe it goes one way rather than the other? (I admit I will never understand what kind of person actually responds to surveys. For people in my demographic, responding to a pollster is a mark of shame, and my parents, less reliant on cell phones, refuse to answer pollsters as well.)

RobinZ16y00

If you have reason to believe that there are significant sources of sample bias in a poll, you will have to discount the results severely.

See in context

21 Efficient prestige hypothesis

by taw

16th Nov 2009

1 min read

21

There's a contrarian theory presented by Robin that people go to highly reputable schools, visit highly reputable hospitals, buy highly reputable brands etc. to affiliate with high status individuals and institutions.

But what would a person who completely didn't care about such affiliations do? Pretty much the same thing. Unless you know a lot about schools, hospitals, and everything else, you're better off simply following prestige as proxy for quality (in addition to price and all the other usual criteria). There's no denying that prestige is better indicator of quality than random chance - the question is - is it the best we can do?

It's possible to come up with alternative measures, which might correlate with quality too, like operation success rates for hospitals, graduation rates for schools etc. But if they really indicated quality that well, wouldn't they be simply included in institution's prestige, and lose their predictive status? The argument is highly analogous to one for efficient market hypothesis (or to some extent with Bayesian beauty contest with schools, as prestige might indicate quality of other students). Very often there are severe faults with alternative measures, like with operation success rates without correcting for patient demographics.

If you postulate that you have better indicator of quality than prestige, you need to do some explaining. Why is it not included in prestige already? I don't propose any magical thinking about prestige, but we shouldn't be as eager to throw it away completely as some seem to be.

Personal Blog

21

New Comment

37 comments, sorted by

top scoring

Click to highlight new comments since: Today at 3:58 AM

[-]RobinHanson16y130

Consider the analogous hypotheses about attractive real estate brokers, which are in higher demand. You might claim that broker attractiveness is just an indicator of broker quality. One hypothesis says attractive brokers are in more demand mostly because they sell houses faster, while another hypothesis says the are in more demand mostly because people like to hang out with attractive people. We could compare these theories empirically by seeing if in fact attractive brokers sell houses faster or for more money. If they don't, or if their gains aren't enough to cover their added costs, the second hypothesis would be favored.

[-]taw16y10

The second hypothesis would not be favored, even if the data turned out to be so.

As far as I know, nobody has done such tests, or at least no such test is in public knowledge. So obviously it couldn't have been incorporated in prestige yet.

When averaged over all human activities, it's not too hard to see that physically attractive people perform somewhat better. Incorporating it as part of default prestige mix makes sense when we don't know anything about real estate brokers.

If we suddenly came out with super-precise quality metrics for some field, it would turn out that prestige overestimates performance of some and underestimates performance of others. We just don't have any idea what would the adjustment be.

[-]RobinHanson16y10

Well by your standards once one had collected such data one could see if it made any difference to new customers, when they were informed of it.

[-]bogdanb16y10

One could look if it made any difference, but the reason for what they see might not be obvious to understand. The obvious expected change would be an increase of the quality metrics and a decrease of the (previous) prestige metrics in correlation of which service customers buy. (In principle, this is calculable.)

In practice, I'd expect the effects would be lower than “rational”, at least at first. Such a precise quality metric is likely to be complicated to obtain (since we don't have one yet). People would naturally be mistrustful of it; in effect, unless one can do the entire analysis again and re-derive the new metric themselves, one must balance the previous prestige measure with the prestige of the researcher(s) that introduced the new metric. (In fact, even if I could re-derive the metric myself, I wouldn't apply 100% confidence to my calculations. In all psychological studies that I read researchers seemed to assume that the rational action would be to apply 100% confidence to new information, with the exception of studies about confidence...)

The effect might be even reversed, in the case of more-or-less paranoid customers suspecting a conspiracy.

[-]taw16y00

Assuming it wasn't incorporated in the prestige metrics already or too insignificant to bothered with, yes. Prediction is that new customers would change their behaviour in this direction.

[-]wedrifid16y00

Good test. Has someone done it?

[-]CannibalSmith16y00

Good test. Does anyone here has a house to sell?

[-]wedrifid16y20

Good test. Does anyone have a bunch of grad students to go and rate real estate brokers and compare with existing sales data?

[-]Technologos16y90

I think the trick is that prestige is never composed solely of quality-assessing measures. If nothing else, prestige is path-dependent: it takes a lot of work for a new university to prove itself.

Even if all relevant quality metrics are included in prestige, additional lower-correlation components can make prestige correlate worse than any of the individual quality metrics.

[-]Alicorn16y60

Prestige is a complicated indicator, and might underperform simpler indicators that are already incorporated in prestige evaluations. Cf. Michael Bishop, "In Praise of Epistemic Irresponsibility".

[-]Psychohistorian16y50

But what would a person who completely didn't care about such affiliations do? Pretty much the same thing.

I actually doubt this is true. Specifically, for the case of private university tuition. Suppose people actually sat down and calculated their lifetime expected income, given their admissions metrics, and their parents/the state said, "Here's tuition at a private university. Go where you want and keep the change." Then, they forget all about prestige. Do you think people would be spending those dollars at more-prestigious private schools, or at comparably effective but lower-cost, lower-amenity public schools? Similarly, do you think they'd be going to famous research institutions, or famous teaching institutions?

Precise Hanson's theory is not, but it does seem to explain why students pay top dollar to listen to professors who don't really want to teach them.

There may admittedly be a significant collective action problem: if everyone ignored prestige, maybe you should pick a top public school over a top private one. But if it's just you, well, signal away.

[-]Nanani16y10

What if their lifetime expected income is highly dependent on the connections they make in those years of education?

For example, making partner, or meeting the right people to begin a startup.

If school prestige attratcs the kind of people you need to meet, professor interest in students isn't as important.

[-]Psychohistorian16y00

This is certainly true for some students, particularly at the high end, but I would be very surprised if it's true for your typical student making this decision. That is admittedly something that would need to be factored in to lifetime expected income, I admit.

[-]gwern16y30

As I recall, haven't we linked and discussed before studies showing that lifetime earnings of those who turned down Ivies for top-but-cheaper schools were comparable to those who went? By that standard, prestige is inefficient. (And I've seen advice to this effect, too, in college guide materials.)

I have also heard that an increasingly common strategy among undergraduates in California is to go to one of the feeder community colleges and only go to the prestigious schools like UCLA or Berkeley for the last 2 years. (Although I read this in Steve Sailer who was busy mixing it up with his theories about canny Asian parents maximizing the bang for their buck, so reader beware.)

These 3 points suggest to me that it is known that prestige is not identical to quality. There are a few possible explanations that occur to me.

Perhaps the target demographics of the 3 points are people who cannot take advantage of the financial aid offered by prestigious universities but perhaps not by the higher-quality but less prestigious institutions. (Prestige attracts donations, which enables things like Harvard's attend-free-if-you're-poor. Obviously qualified poor kids will much prefer Harvard to, say, Cornell, even if Cornell is more quality; it's basic math. But for a middle-class kid who doesn't qualify?)

Perhaps the lifetime earnings are similar or higher elsewhere but one is compensated for the poorer education/higher-expense with intangibles like being part of a tradition, everyone knowing of where you went, or...

Having more power thanks to the prestigious institution & networking. I think of John Kerry; leaving aside the marrying-into-wealth, he has exercised a great deal of power of the years, so you certainly couldn't call him a failure, but I suspect his lifetime earnings are vastly less than if he had gone to a higher-quality-but-lesser-prestige school which funneled him into private law. (Kerry first really got into politics at Yale, but if you don't think his example works, how about Barack Obama? Without the prestige of Harvard, would he have gone into local politics/community-organizing & thereby forfeiting the lucrative lawyering career one could expect? But notice how much power he got in exchange.)

So that's 3 or 4 ways in which prestige could be very closely aligned to quality and yet remain permanently different.

[-]Douglas_Knight16y40

As I recall, haven't we linked and discussed before studies showing that lifetime earnings of those who turned down Ivies for top-but-cheaper schools were comparable to those who went? By that standard, prestige is inefficient.

NO! they said the opposite! They said that school SAT scores don't predict income, but they also say that the cost of the school, even among private schools, predicts income. In recent cohorts, it was only a normal return on investment, but it was positive.

The final version also makes, if only by omission of the previous version, the claim that school prestige doesn't predict income. This is facially absurd and very hard to reconcile with the tuition effect.

[-]gwern16y00

http://www.nytimes.com/2008/04/19/business/19money.html doesn't seem to mention anything about SAT or cost of school correlating with income:

Actual research on the subject is scant, and what exists offers conflicting evidence. One often-cited study from 1998, however, concludes that attending a more selective or elite institution does not translate to an economic advantage for students later on, as measured by their reported income. Attending a more elite college does seem to affect the later incomes of poorer students. The study, written by Alan B. Krueger, a professor of economics at Princeton, and Stacy Berg Dale, then a researcher at the Andrew W. Mellon Foundation, concluded that the qualities that students themselves bring to their education may be what matters most.

Are you looking at the actual studies or something?

[-]RobinZ16y00

Oddly, I was just reading How to Lie with Statistics the other day, and, well ...

One often-cited study from 1998, however, concludes that attending a more selective or elite institution does not translate to an economic advantage for students later on, as measured by their reported income.

... these data might be very poor, for several reasons.

Self-reporting is unlikely to be accurate - people both overstate and understate income for various reasons.
The surveys can only go out to people that can be tracked down - and those who cannot will be disproportionately the poorly-off.
If the response rate is not close to 100%, there may be a strong difference between those who reply and those who do not.

These are just the three effects I can remember that apply to a study of just this kind.

[-]SilasBarta16y00

But how do those skew the result? It's not enough to say that there's bias; you would have to show that the bias works against the claimed conclusion. So with respect to your points:

1) If you don't have a reason to believe the bias skews one way or the other, knowledge that it could be higher or lower doesn't matter.

2) The wealthier you are, the easier you are to track down, so the survey would tend to over-state the average income.

[-]RobinZ16y00

If you have reason to believe that there are significant sources of sample bias in a poll, you will have to discount the results severely.

[-]Douglas_Knight16y00

Are you looking at the actual studies or something?

Yes.

[-]taw16y00

That's essentially what I said - networking is part of quality; lifetime earnings are part of quality; all that added together is approximated by prestige, and there's little evidence that any of those metrics alone is better quality estimator than prestige alone.

[-]gwern16y20

I don't think I said what you said. I think quality is different from lifetime earnings, which is different from prestige/networking.

Quality in science can easily depress lifetime earnings (do the most brilliant scientists work in academia or for commercial interests? Where do they earn more), for example, and I already pointed out how prestige/networking can push down lifetime earnings because it offers a chance at power and power doesn't always come with as much money as one would have elsewise.

These are all in general correlated, much like IQ is correlated with success, health, non-criminality etc. but no one would say that IQ is a better metric to use than measures just of health or non-criminality.

[-]bgrah44916y30

I hate to be "that guy" but it depends on what you mean by quality! I think what you have in mind is something like, does the school educate you well? But the only way you'd know if it educated you well is if you succeed in life, i.e. achieved status. ("No, standardized tests!" -- but how the questions on the test are deemed important is irrevocably intertwined with status to begin with!) So we see that education is a tool for achieving status. It's much more direct if the school just conferred status -- which is what we find.

So yes! Prestige is a great indicator of quality - as long as you define "quality" as "prestigious"! (Which we mostly do anyway.)

If that's not clear (it's hard for me to get ideas out of my head sometimes), I guess another way to say it is: Which causes the other? Because I think "prestige" is an input to quality, whereas you're saying "quality" is an input to "prestige."

[-]Larks16y20

Students using prestige only as a quality-indicator wouldn't be willing to pay as much for a high-prestige college, as it is an imperfect indicator for them, as opposed to a perfect indicator for prestige hunters.

[-]vinayak15y10

There seem to exist certain measures of quality that are second level, in the sense that they measure quality in a kind of indirect way, mostly because the indirect way seems to be easier. One example is sex appeal. The "quality" of a potential mate should be measured just by the number of healthy offsprings it can give birth to. However, that's difficult to find out and hence evolution has programmed our genes to refer to the sex appeal instead, that is, the number of people who will find the person in question attractive. However, the only problem with such second level measures is that they can be feigned. An impotent person can still look attractive.

Similarly, imagine a car manufacturing company that produces two different models. However, the only differences between them are in the looks and the cost. Assuming that the looks are not clearly better than the looks of the other, people who use prestige as the measure of quality will always buy the more expensive car.

[-]wedrifid15y00

The "quality" of a potential mate should be measured just by the number of healthy offsprings it can give birth to.

Modified, naturally, by how sexy the healthy children are themselves.

[-]Blueberry16y10

One example for an alternative metric is provided by the various sets of ratings for law schools. The most famous ratings, the US News ratings, include many different factors, only one of which is "prestige" (as measured by asking lawyers and professors what they think of schools). They also include more objective factors such as library size, money spent on students, and average test scores and GPAs. There are alternative rankings available as well, which look at factors such as debt repayment, minority enrollment, likelihood of getting X job, quality of student life, quality of teaching, and so forth.

When students choose a law school, it's very common to look at these sets of rankings, taking prestige into account as one of many factors.

[-]Mike Bishop16y10

I agree there is something to this, but given that people have different goals, it usually makes sense to look at factors other than just prestige.

On an only somewhat related note. I highly recommend this paper by Henrich and Gil-White on the Evolution of Prestige (defined as "freely conferred deference.")

[-]Neil16y00

I wonder if you could in theory separate out part of the prestige value. Part of the prestige value of a product would be related to its exclusivity - things that are easily got don't confer prestige for obvious reasons.

So suppose you were looking at two schools that were equally prestigous but one was smaller, more expensive, required better social connections and higher academic achievement to access, and was more preferred by people in higher circles than the other. Then you might conclude that this smaller school derived more of its prestige from its exclusivity than the other school did, and hence on other indicators which might matter more, the larger, less exclusive school was actually better.

[-]knb16y00

You need to operationalize "quality". Quality in this context simply means "goodness", which is subjective. If by quality you mean "leads to life success", then prestige seem to be a good proxy for "quality" (though the selection effect makes it less clear).

For example, going to Harvard Business school might lead to life success due to possible employers wanting to affiliate with your high status associations, even if the education leads to lower productivity gains than community college business courses and a few years of employment in the field.

[-]djcb16y00

BTW, regardless of the discussion here, the OB-article linked is well worth reading, as are the commments. One of the better OB-articles of recent times.

[-]fburnaby16y00

There's no denying that prestige is better indicator of quality than random chance - the question is - is it the best we can do?

Where does the prestige come from? Likely, it's got a lot to do with public perception of quality in the first place. If we can improve objectivity in the judgment of this quality, then that's great; but the prestige would follow it along. We won't 'do better' than following the prestige, the prestige will 'do better' at following the quality.

[-]Vladimir_Nesov16y00

The usual assumption is that public perception of quality is systematically biased and that individuals willing to do better shouldn't automatically agree with it. It's not a given that good indicators of quality known to experts are widely accepted. This post presents a hypothesis that public perception may be a pretty good indicator, incorporating other indicators as they become known.

[-][anonymous]16y00

I thought the post's idea was that perception (by the "right" public) is status, and that's all there is to it - it's not a proxy but the real thing.

[-]saliency16y00

“But what would a person who completely didn't care about such affiliations do? Pretty much the same thing.”

I disagree. This is a case of different populations having different utility formulas for the same item. Those who enjoy being affiliated with high status institutions are receiving utility other populations do not.

I think that people often pre-screen their selection options removing those options they think are overpriced aka do not fit their utility formula. They filter out signals belonging to other populations who use a different utility function.

An item may have many utility functions. Populations who’s utility formulas match the market price see correct signals. Populations who’s formulas do not match see the item as either a deal or rip-off. I imagine that populations that view the item as a deal must be small. Those that see the item as a rip-off probably purchase alternatives based off of signaling data they feel is valid.

In some cases the utility function map to the same value but in others they do not.

[-][anonymous]16y00

I recently decided to reverse my old policy and start buying brand name clothes. Previously I would ignore brand information and just buy whatever looked like reasonably high quality clothes at a good price.

The problem is that quality matters, for reasons that are difficult to articulate, and I am unable to identify high quality clothes. So I have a closet full of junky no-brand clothing that I'll never wear again.

Sometimes I buy brand clothes and am disappointed, but this happens much less often than with no-brand stuff.

[-]NancyLebovitz16y-10

Prestige is always retrospective. It's building on what brought prestige in the past. If you're willing to bet on a new field or a new angle on an old field, and you think learning that (or associating with the up-and-comers) is best served at a less prestigious university, then you have a more complicated choice.

Or you might think an institution is going to lose prestige. Will the Harvard business school continue to be respected?

Personal taste: quality of life is the bottom line, and it isn't that bad even if you aren't at the top. You might be looking for something that isn't offered at the most prestigious institutions.

About those studies of the return for getting into a top university: Do they include the costs of competing to get in?

Moderation Log