Shouldn't I be equally worried about stirring my coffee?
I'm willing to consider that this quantum computer could be a novel situation that demands real consideration.
The vector space of your coffee is basically flat whereas the vector space of the algorithm Deutsch is describing is unimaginably vast. The fact that both are embedded in quantum physics is somewhat besides the point.
From David Deutsch's The Beginning of Infinity:
I'm not so sure we have the computing power to "simulate a person," but suppose we did. (Perhaps we will soon.) How would you respond to this worry?