Some poor bastard had to come to terms with knowing that he was obviously the dumbest physicist on the Manhattan Project, a net drag on the team

This doesn't follow. It might be that the dumbest person is still contributing productively. They'll just be contributing the least. Moreover, there might be enough variation in specific skill sets that no one is actually the dumbest (although I find this second argument to be weak. The truth is that some people really are better than others). Now it is more plausible that some of the dumber people who also worked less ended up distracting people in the project enough that they were net negatives. But that sort of argument requires that they be not only stupid but lazy and disruptive. In practice, few people with those traits last long in serious research.