I don't find myself forced to believe in uncomputable things in the same way that I find myself forced to accept the existence of countable infinite sets .

I'm not saying that uncomputable things exist. I'm just saying that they might.

I think what's bugging me is that a TM with a finite tape seems more complicated than a TM with an infinite tape.

So? Just because the computer can run forever doesn't mean that the program never halts or repeats.

Also, Occam's razor doesn't work that way. If you add a constant of a specific value, that makes it less likely because the probability has to be split over all possible values, and it's unlikely to be that specific one. If you're just suggesting that there is a constant, this does not apply.