Note that, conventionally, fitnesses can decline

Yes, I did notice that. That is why I wrote spelling out the assumptions:

R.A. Fisher proved that (under the admittedly unrealistic assumption of an unchanging environment) the average 'fitness' of the organisms in a population subject to natural selection can only increase, so long as the mutation rate is moderate.