The following is a counterargument to some of the claims of the OP. I'm not convinced that it's a correct counterargument, but it's sufficient to make me doubt some of the central claims of the OP.

I think I understand the reasoning that leads to claims like

Unlike a human toddler, AIXI can never entertain the possibility of its own death. However much it learns, it will never recognize its mortality.

and

they won't take seriously the idea that their cognition can be modified

But from my reductionism+materialism+computationalism perspective, it seems obvious that a computer program that gets this sort of thing right must exist (because humans get these things right and humans exist, or even just because "naturalized induction" would implement such a program). So the argument must be that the Solomonoff prior gives much higher weights to equally good predictors that get these things wrong.

But sequence prediction isn't magic -- we can expect that the shortest programs that survive a reasonable training period are the ones with (possibly highly compressed) internal representations that carve reality at its joints. If the training period can pack in enough information, then AIXI can learn the same sorts of things we think we know about being embedded in the world and how the material that forms our nervous systems and brains carries out the computations that we feel from the inside. In other words, if I can learn to predict that my perceptual stream will end if I drop an anvil on my head even though I've never tried it, why can't an AIXI agent?

What am I missing?

(I regard the preference solipsism point as obviously correct -- you don't want to give an autonomous AI motivations until well after it's able to recognize map/territory distinctions.)