And then, why privilege penrose over the ~1e3 experts who disagree with him?

It is valuable to have a list of "people who believe X" and "people who believe ~X," and one might suspect that, if X, the majority position, is incorrect, the people on the ~X list to disproportionately be higher quality. I'm not a good enough science historian to know if that's the case historically, especially because you would want to use contemporary measurements of quality, as many people who believed ~X and it turned out to be right are more highly estimated by hindsight.

(More broadly, there may be systematic patterns to public support on scientific controversies, such that 1) one shouldn't compare length of lists or treat positions of individuals as giving completely independent evidence and 2) there may be patterns that suggest known kinds of events.)