Yes, some subjects are just hard. But there are limits to this. How much one needs is a function of how much one wants to focus on a particular subject. So for example most physicists probably need three semesters of calc, linear algebra, and stats, at minimum. But only some of the physicists will need group theory, while others will need additional stats, and others will need differential geometry. But almost no physicist will need all of these things. Similarly, some degree of specialization may make sense if one wants to do philosophy.

That's in fact already the case: the moral philosopher has a read a lot more about the history of moral philosophy, and same for the person studying epistemology, or other basic aspects of things. So to some extent the issue isn't the amount of learning that is required, but a disagreement with what is required, and how cross-disciplinary it should be.