That's okay, there's no formalized theory behind it. But for the sake of conversation:
It seems you once agreed that multiple agents in the same epistemic state in different possible worlds can define strategies over their future observations in a way that looks like trading utilities: http://lesswrong.com/lw/102/indexical_uncertainty_and_the_axiom_of/sht
When I treat priors as a kind of utility, that's interpretation #4 from this Wei Dai post: http://lesswrong.com/lw/1iy/what_are_probabilities_anyway/
Really the only things that seems in any way novel here are the idea that the space of possible worlds might include worlds that work by different mathematical rules and that possibility is contingent on the agent's priors. I don't know how to characterize how math works in a different world, other than by saying explicitly what the outcome of a given computation will be. You can think of that as forcing the structural equation that would normally compute "1+1" to output "5", where the graph setup would somehow keep that logical fact from colliding with proofs that "3-1=2" (for worlds that don't explode) (which is what I thought Eliezer meant by creating a factored DAG of mathematics here). That's for a very limited case of illogical-calculation where our reasoning process produced results close enough to their analogues in the target world that we're even able to make some valid deductions. Maybe other worlds don't have a big book of platonic truths (ambiguity or instability) and cross-world utility calculations just don't work. In that case, I can't think of any sensible course of action.
I don't think this is totally worthless speculation, even if you don't agree that "a world with different math" makes sense, because an AI with faulty hardware / reasoning will still need to reason about mathematics that work differently from its mistaken inferences, and that probably requires a partial correspondence between how the agent reasons and how the world works, just like how the partial correspondence between worlds with different mathematical rules allows some limited deductions with cross-world or other-world validity.
Suppose you wake up as a paperclip maximizer. Omega says "I calculated the millionth digit of pi, and it's odd. If it had been even, I would have made the universe capable of producing either 1020 paperclips or 1010 staples, and given control of it to a staples maximizer. But since it was odd, I made the universe capable of producing 1010 paperclips or 1020 staples, and gave you control." You double check Omega's pi computation and your internal calculator gives the same answer.
Then a staples maximizer comes to you and says, "You should give me control of the universe, because before you knew the millionth digit of pi, you would have wanted to pre-commit to a deal where each of us would give the other control of the universe, since that gives you 1/2 probability of 1020 paperclips instead of 1/2 probability of 1010 paperclips."
Is the staples maximizer right? If so, the general principle seems to be that we should act as if we had precommited to a deal we would have made in ignorance of logical facts we actually possess. But how far are we supposed to push this? What deal would you have made if you didn't know that the first digit of pi was odd, or if you didn't know that 1+1=2?
On the other hand, suppose the staples maximizer is wrong. Does that mean you also shouldn't agree to exchange control of the universe before you knew the millionth digit of pi?
To make this more relevant to real life, consider two humans negotiating over the goal system of an AI they're jointly building. They have a lot of ignorance about the relevant logical facts, like how smart/powerful the AI will turn out to be and how efficient it will be in implementing each of their goals. They could negotiate a solution now in the form of a weighted average of their utility functions, but the weights they choose now will likely turn out to be "wrong" in full view of the relevant logical facts (e.g., the actual shape of the utility-possibility frontier). Or they could program their utility functions into the AI separately, and let the AI determine the weights later using some formal bargaining solution when it has more knowledge about the relevant logical facts. Which is the right thing to do? Or should they follow the staples maximizer's reasoning and bargain under the pretense that they know even less than they actually do?
Other Related Posts: Counterfactual Mugging and Logical Uncertainty, If you don't know the name of the game, just tell me what I mean to you