Also, another consideration you haven't mentioned is the belief that focusing on improving the "product of EA" (e.g., quality of research, etc.) is better for movement building purposes than outright outreach. For example, Paul Christiano:

Sometimes “movement-building” is offered as an example of an activity with very high rates of returns. At the moment I am somewhat skeptical of these claims, and my suspicion is that it is more important for the “effective altruism” movement to have a fundamentally good product and to generally have our act together than for it to grow more rapidly, and I think one could also give a strong justification for prioritization research even if you were primarily interested in movement-building. But that is a much longer discussion.

This seems plausible to me, though I'm not completely convinced.