The main mathematical issue here is no uniform probability distribution on a countable set:
Minor nitpick: You mean infinite set. Any finite set is, of course, both countable and has a uniform probability distribution, and your point is correct for all (measurable) infinite sets.
It depends on your conventions. Some people use "countable" and "countably infinite" to refer to what I refer to as "at most countable" and "countable."
I saw this conundrum at Alexander Pruss's blog and I thought LWers might enjoy discussing it: