Prestige is always retrospective. It's building on what brought prestige in the past. If you're willing to bet on a new field or a new angle on an old field, and you think learning that (or associating with the up-and-comers) is best served at a less prestigious university, then you have a more complicated choice.
Or you might think an institution is going to lose prestige. Will the Harvard business school continue to be respected?
Personal taste: quality of life is the bottom line, and it isn't that bad even if you aren't at the top. You might be looking for something that isn't offered at the most prestigious institutions.
About those studies of the return for getting into a top university: Do they include the costs of competing to get in?
There's a contrarian theory presented by Robin that people go to highly reputable schools, visit highly reputable hospitals, buy highly reputable brands etc. to affiliate with high status individuals and institutions.
But what would a person who completely didn't care about such affiliations do? Pretty much the same thing. Unless you know a lot about schools, hospitals, and everything else, you're better off simply following prestige as proxy for quality (in addition to price and all the other usual criteria). There's no denying that prestige is better indicator of quality than random chance - the question is - is it the best we can do?
It's possible to come up with alternative measures, which might correlate with quality too, like operation success rates for hospitals, graduation rates for schools etc. But if they really indicated quality that well, wouldn't they be simply included in institution's prestige, and lose their predictive status? The argument is highly analogous to one for efficient market hypothesis (or to some extent with Bayesian beauty contest with schools, as prestige might indicate quality of other students). Very often there are severe faults with alternative measures, like with operation success rates without correcting for patient demographics.
If you postulate that you have better indicator of quality than prestige, you need to do some explaining. Why is it not included in prestige already? I don't propose any magical thinking about prestige, but we shouldn't be as eager to throw it away completely as some seem to be.