I think the trick is that prestige is never composed solely of quality-assessing measures. If nothing else, prestige is path-dependent: it takes a lot of work for a new university to prove itself.

Even if all relevant quality metrics are included in prestige, additional lower-correlation components can make prestige correlate worse than any of the individual quality metrics.