I had the same problem, but it went away immediately after one simple change: stop using shaving cream. Instead, just apply warm water before you shave (it helps to do it after a shower). Before I made the change, my face was always irritable the day of a shave, and exercising would make it flare up; now, nothing. (Having a good multi-blade razor still matters though.)

Maybe you're using the wrong cream? Using just water was much worse for me than using cream. (I've also found than massaging for a while to spread the cream rather than just sticking it on my face helps.)