Temporarily adopting this sort of model of "AI capabilities are useful compared to human IQs":

With IQ 100 AGI (i.e. could do about the same fraction of tasks as well as a sample of IQ 100 humans), progress may well be hyper exponentially fast: but the lead-in to a hyper-exponentially fast function could be very, very slow. The majority of even relatively incompetent humans in technical fields like AI development have greater than IQ 100. Eventually quantity may have a quality of its own, e.g. after there were very large numbers of these sub-par researcher equivalents running at faster than human and coordinated better than I would expect average humans to be.

Absent enormous numerical or speed advantages, I wouldn't expect substantial changes in research speed until something vaguely equivalent to IQ 160 or so.

Though in practice, I'm not sure that human measures of IQ are usefully applicable to estimating rates of AI-assisted research. They are not human, and only hindsight could tell what capabilities turn out to be the most useful to advancing research. A narrow tool along the lines of AlphaFold could turn out to be radically important to research rate without having anything that you could characterize as IQ. On the other hand, it may turn out that exceeding human research capabilities isn't practically possible from any system pretrained on material steeped in existing human paradigms and ontology.

Oh, I see that I misread.

One problem is that "every possible RNG call" may be an infinite set. For a really simple example, a binary {0,1} RNG with program "add 1 to your count if you roll 1 and repeat until you roll 0" has infinitely many possible rolls and no maximum output. It halts with probability 1, though.

If you allow the RNG to be configured for arbitrary distributions then you can have it always return a number from such a distribution in a single call, still with no maximum.

My guess is "no" because both of you would die first. In the context of "largest numbers" 10^10^100 is baby's first step, but is still a number with more digits than you will ever succeed in printing.

In principle the "you" in this scenario could be immortal with unbounded resources and perfect reliability, but then we may as well just suppose you are a superintelligence smarter than the AI in the problem (which isn't looking so 'S' anymore).

1. It's an arbitrary convention. We could have equally well chosen a convention in which a left hand rule was valid. (Really a whole bunch of such conventions)
2. In the Newtonian 2-point model gravity is a purely radial force and so conserves angular momentum, which means that velocity remains in one plane. If the bodies are extended objects, then you can get things like spin-orbit coupling which can lead to orbits not being perfectly planar if the rotation axes aren't aligned with the initial angular momentum axis.
   If there are multiple bodies then trajectories can be and usually will be at least somewhat non-planar, though energy losses without corresponding angular momentum losses can drive a system toward a more planar state.
   Zero dimensions would only be possible if both the net force and initial velocity were zero, which can't happen if gravity is the only applicable force and there are two distinct points.
   In general relativity gravity isn't really a force and isn't always radial, and orbits need not always be planar and usually aren't closed curves anyway. Though again, many systems will tend to approach a more planar state.

I believe that there is already far too much "hate sharing".

Perhaps the default in a social media UI should be that shared content includes a public endorsement of whatever content it links to, and if you want to "hate share" anything without such an endorsement, you have to fight a hostile UI to do so.

In particular, "things that are worth sharing" absolutely should not overlap with "want to see less of". If you want to see less of some type of thing, it's self-defeating to distribute more copies of it. Worse, if you even suspect that any of your own readers are anything like you, why are you inflicting it on them?

The conclusion does not follow from the argument.

The argument suggests that it is unlikely that a perfect replica of the functioning of a specific human brain can be emulated on a practical computer. The conclusion generalizes that out to no conscious emulation of a human brain, at all.

These are enormously different claims, and neither follows from the other.

For all practical purposes, such credences don't matter. Such scenarios certainly can and do happen, but in almost all cases there's nothing you can do about them without exceeding your own bounded rationality and agency.

If the stakes are very high then it may make sense to consider the probability of some sort of trick, and attempt to get further evidence of the physical existence of the coin and that its current state matches what you are seeing.

There is essentially no point in assigning probabilities to hypotheses of failures of your mind itself. You can't reason your way out of serious mind malfunction using arithmetic. At best you could hope to recognize that it is malfunctioning, and try not to do anything that will make things worse. In the case of mental impairment severe enough to have false memories or sensations this blatant, a rational person should expect that a person so affected wouldn't be capable of correctly carrying out quantified Bayesian reasoning.

My own background credences are generally not insignificant for something like this or even stranger, but they play essentially zero role in my life and definitely not in any probability calculations. Such hypotheses are essentially untestable and unactionable.