Have the AI predict the reward rather than the observation. The AI can improve itself because it can empirically learn that some modifications to itself lead to more rewards. Assuming that you use probabilistic or approximate models rather than perfect fitting hypotheses.

I don't know about the anvil problem, but if "HALT" instructions would work (I'm not sure if they would), then just make hypotheses with that instruction have higher prior probability.

Preference solipsism is a general problem with reinforcement learning and I think it might actually be unsolvable. To do something other than reinforcement learning requires strong assumptions about the universe you exist in, and desiring to get it to a desired state (as opposed to getting merely your input to a desired state.) How is that even possible? We can't perfectly define the universe it exists in beforehand, because we don't know ourselves. We can't decide the state we want the universe to exist in either.

Also it seems really interesting that almost all the problems you describe apply (to some degree) to humans. Perhaps that really is the natural state of intelligence and these problems aren't completely solvable.