I think you're getting downvoted for your TL;DR, which is extremely difficult to parse. May I suggest:

TL;DR: Treating "computers running minds" as discrete objects might cause a paradox in probability calculations that involve self-location.

"Without suffering" seems like a really high bar. Additionally, do we really want a system that can, presumably, put an utter genius with leet rationality skillz in the top position, and not gain from it? If the correlation between doing well and having a smart leader is literally zero, that's what you get.

1) I expect to see AI with human-level thought but 100x as slow as you or I first. Moore's law will probably run out sooner than we get AI, and these days Moore's law is giving us more cores, not faster ones.

I was measuring the Kolmogorov complexity of the evidence, but now that you mention it that does make for a bit of circular reasoning.

Additional justification: if your data can be perfectly described by a 1000 bit program, you will find that program after at most 1000 + c bits of evidence.

1) Yes, brains have lots of computational power, but you've already accounted for that when you said "human-level AI" in your claim. A human level AI will, with high probability, run at 2x human speed in 18 months, due to Moore's law, even if we can't find any optimizations. This speedup by itself is probably sufficient to get a (slow-moving) intelligence explosion.

2) It's not read access that makes a major difference, it's write access. Biological humans probably will never have write access to biological brains. Simulated brains or AGIs probably will have or be able to get write access to their own brain. Also, DNA is not the source code to your brain, it's the source code to the robot that builds your brain. It's probably not the best tool for understanding the algorithms that make the brain function.

3) As said elsewhere, the question is whether the speed at which you can pick the low hanging fruit dominates the speed at which increased intelligence makes additional fruit low-hanging. I don't think this has an obviously correct answer either way.

[EDIT note: This is completely different from what I originally wrote in response to lavalamp's question, because originally I completely misunderstood it. Sorry.]

You can't "count every possible program equally". (What probability will you give each possible program? If it's positive then your total probability will be infinite. If it's zero then your total probability will be zero. You can do a lot of probability-like things on a space with infinite total measure, in which case you could give every program equal weight, but that's not generally what one does.)

Leaving that aside: Your argument suggests that whatever probabilities we give to programs, the resulting probabilities on outcomes will end up favouring outcomes that can be produced by simpler programs. That's true. But a universal prior is doing something stronger than that: it gives (in a particular way) higher probabilities to simpler programs as well. So outcomes generated by simpler programs will (so to speak) be doubly favoured: once because those simple programs have higher probabilities, and once because there are "more" programs that generate those outcomes.

In fact, any probability assignment with finite total probability (in particular, any with total probability 1, which is of course what we usually require) must "in the limit" give small probabilities to long programs. But a universal prior is much more specific, and says how program length corresponds to probability.

But a universal prior is doing something stronger than that: it gives (in a particular way) higher probabilities to simpler programs as well.

The only programs allowed in the Solomonoff distribution are ones that don't have any extended versions that produce the same output observed so far. So it's not that the longer programs are given lower probability - it's that they are given undefined probability, and are entirely "represented" by the most truncated version.

I learned this by reading Anatoly Vorobey's forthcoming article!

I have been thinking that the universal prior is tautological. Given a program, there are an infinite number of longer programs which perform the same computation (or an indistinguishable variation) but only a finite number of shorter programs having this characteristic. If you count every possible program equally, you'll find that each short program represents a host of longer programs. However, now that I write this down, I'm no longer sure about it. Can someone say why/if it's wrong?

Do concern trolls actually exist? I've never seen one (or maybe they were subtle enough that I didn't notice).