When a theory starts being inconsistent with the data you just throw it out. But what if you were wrong about the theory being inconsistent?

That's just not what you do. "Throwing out theories" is more strongly suggested by falsificationism. A Bayesian approach recognises that observations are uncertain and fallible. Observations inconsistent with a theory are strong negative evidence, but they don't really "falsify" a theory.

Here is Yudkowsky on the topic:

On the other hand, Popper's idea that there is only falsification and no such thing as confirmation turns out to be incorrect. Bayes' Theorem shows that falsification is very strong evidence compared to confirmation, but falsification is still probabilistic in nature; it is not governed by fundamentally different rules from confirmation, as Popper argued.

Popper "missed" confirmation by rejecting induction. He didn't get it, and now we know better.

A related point is that science does not start with a set of observations or data.

Observations come second. Priors come first.

Solomonoff induction doesn't care about criticism, let alone that it is pivotal in knowledge creation.

Critics might have a role to play for a resource-limited agent - for instance if they pointed out explanations that were short and were not yet receiving the proper consideration - or if they supplied more data.

Also some theories can't be refuted by empirical means, so what does Solomonoff Induction do about those?

It says to prefer the shorter one.