Assuming zero space inflation, the “exit” mouth of the wormhole can’t travel faster than c with respect to the entry. So for expansion purposes (where you don’t need (can’t, actually, due to lack of space) to go back), you’re limited to c (radial) expansion. Which is the same as without wormholes.

In other words, the volume covered by wormholes expands as (c×t)³ relative to when you start sending wormholes. The number of people is exponential relative to when you start reproducing. Even if you start sending wormholes a long time before you start reproducing exponentially, you’re still going to fill the wormhole-covered volume.

(The fault in your statement is that you can go in “less” than half the time only for travel within the volume already covered by wormholes. For arbitrarily far distances you still need to wait for the wormhole exit to reach there, with is still below c.)

Space inflation doesn’t help that much. Given long enough time, the “distance” between the wormhole entry and exit point can grow at more than c (because the space between the two expands; the exit points still travel below c). In other words, far parts of the Universe can fall outside your event horizon, but the wormhole can keep them still accessible (for various values of can...). This can allow you unbounded-growth in the volume of space for expansion (exponentially, if the inflation is exponential), but note that the quantity of matter accessible is still the same volume that was in your (c×t)³ (without inflation) volume of space.