I'm so confused, I can't even tell if we disagree. What I am thinking of is essentially the argument in Eliezer Yudkowsky's "Inductive Bias":

The more inductive bias you have, the faster you learn to predict the future, but only if your inductive bias does in fact concentrate more probability into sequences of observations that actually occur. If your inductive bias concentrates probability into sequences that don't occur, this diverts probability mass from sequences that do occur, and you will learn more slowly, or not learn at all, or even - if you are unlucky enough - learn in the wrong direction.

Inductive biases can be probabilistically correct or probabilistically incorrect, and if they are correct, it is good to have as much of them as possible, and if they are incorrect, you are left worse off than if you had no inductive bias at all. Which is to say that inductive biases are like any other kind of belief; the true ones are good for you, the bad ones are worse than nothing. In contrast, statistical bias is always bad, period - you can trade it off against other ills, but it's never a good thing for itself. Statistical bias is a systematic direction in errors; inductive bias is a systematic direction in belief revisions.