With classical computers Moore's law improves serial and parallel performance simulataneously - by making components smaller.
With quantum computers serial and parallel performance are decoupled - more qubits improves parallel performance and minaturisation has no effect on the number of qubits, but improves serial processing performance. So, there are two largely independent means of speeding up quantum computing. Which one supposedly doubles twice as fast as classical computers? Neither - AFAICS.
Sorry, my original response should have been "yes, you aren't getting into the spirit of the counterfactual."
Eliezer Yudkowsky and Scott Aaronson - Percontations: Artificial Intelligence and Quantum Mechanics
Sections of the diavlog: