What are the differences between the 'big names' of higher education, in comparison to other places?

For example, I often hear about MIT, Oxford, and to a lesser extent, Cambridge. Either there's some sort of self-selection, or do graduates from there have better prospects than graduates of 'University of X, YZ'?

In a little bit of unintended self-reflection I noticed that I have a strange binary way of thinking of higher education. It feels that if I don't go to one of the top n, my effort is wasted. Not sure why.

I'm just becoming somewhat paranoid regarding the real world after reading HPMOR because I always get a 'how much do I really know?' feeling. I'm not sure how my impressions were formed and I better double-check how well does the ideas in my mind reflect the real-world truth but at the same time I'm not even sure what's a reliable indicator.

Post-high education LWers, do you think the place you studied at had a significant effect on your future prospects?