Basically, he claims that working in an amoral environment will eventually turn you into a worse person than you would otherwise be, and weaken your resolve and desire to fulfill your original goal.

This is a valid consideration, but that is partly why having a lively community of effective altruists is so important, as a way to prevent value erosion.