Hidden variables aren't random; they are fixed, but unknown. Maybe we are using different definitions of randomness here. Yet I can't see why you are comfortable with a hidden deterministic algorithm setting hidden variables; wouldn't such an algorithm itself be random by your definition?
There is no point in arguing, which of the hypotheses producing the same results is "really true". We should just pick the simplest one according to the Occam razor. But the simplest hypothesis isn't just the one which involves less objects (like hidden variables), but rather, the one for which our theories fit with minimal stretch. If you agree with the interpretation of probabilities as a measure of uncertainty, then it's simpler to use the fundamentally random processes interpretation which fits into this framework -- the one with hidden variables.
I just don't see any distinction between a hidden variable and a random variable. That it's fixed has nothing to do with anything. It's the difference between having a random number generator inside your program, or having a deterministic program which is called with a bunch of randomly generated arguments.
Either way you still have to ask the question of where the numbers are coming from, and if they are truly random. If they are the result of some simple deterministic algorithm. If we could, at least in principle, predict it with total accuracy, or if it'...
I was trying to reconcile the fact that in a deterministic universe there could be life with free will, but I am going full circle now and am starting to think that everything is really random, if not I don't see how there could be free will in a deterministic universe.
If mathematicians measure randomness with probability, then there must be some things that have a 100% occurrence probability (in the current universe above atomic levels I presume), which now I see as special cases of randomness rather than opposites to randomness, and these lead us to think that there is determinism.
I think we may have this cognitive bias (deterministic views of reality) because it is extremely helpful to use these 100% probability occurrence things to model the universe rationally, learn, and to predict the future, but it is not the whole story or at least a complete description of reality.
What do you think?
EDIT 1: Thank you all for the comments below. I recognize I am naive in this topic.
Although I am not convinced yet, I think my possible argumentative error is:
P1: I observe free will in the behavior of living things.
P2: Deterministic physical mechanical processes don't permit free will.
C: Therefore physics must include random processes.
I think I only see a solution of free will in randomness, but maybe there are other solutions ( I will read the Free Will Sequence here on LW!)
EDIT 2: After reading some articles of the Free Will Sequence I realize the problem of investing energy around free will questions if free will is just a mistake in our thinking process.
It is something like why ask about time travel if time doesn't exist? or, why explore the mechanics of randomness vs determinism if randomness doesn't exist and thus the dichotomy "randomness vs determinism" doesn't exist in the first place?