Since elementary particles can come and go, what's really conserved is some arbitrary energy. Infinities won't save you from arbitrariness here, because energy is locally conserved too, and our energy density is (thank goodness) definitely not infinite.