I don't understand the title. You're talking about a reform to the democratic process, and you're comparing it with 'finance reform'. Those only seem tangentially related.

Why not "P1: C, P2: Y", which maximizes the sum of the two utilities, and is the optimal precommitment under the Rawlian veil-of-ignorance prior?

If we multiply player 2's utility function by 100, that shouldn't change anything because it is an affine transformation to a utility function. But then "P1: B, P2: Y" would maximize the sum. Adding values from different utility functions is a meaningless operation.

P1: .5C .5B

P2: Y

It's not a Nash equilibrium, but it could be a timeless one. Possibly more trustworthy than usual for oneshots, since P2 knows that P1 was not a Nash agent assuming the other player was a Nash agent (classical game theorist) if P2 gets to move at all.

But this is different than requiring that the inputs be drawn from a certain probability distribution (in other words, requiring 5% of the inputs to be 0000, 7% to be 0001, 6% to be 0010, etc.)

Well, I don't know of a single piece of software which requires that its inputs come from specific probability distributions in addition to satisfying some set of properties.

In fact, wouldn't that software object have to maintain some running counts to even be able to estimate from which distribution do its inputs come?

I think you've missed the point. When I said

Unfortunately, if you do that, the resulting formalism is no longer AIXItl.

I meant it literally and formally. The resulting machine may or may not be smart, but regardless, it does not necessarily obey the AIXI equations after the first timestep, and it lacks some of the formal properties of the AIXItl model. The AIXItl model assumes that the machine will continue to be AIXItl. Proofs about how AIXItl behaves (e.g., that its environment model improves over time) do not apply to AIXItls that can modify their code. I don't know what properties this variant has, but I'm not yet convinced that they are nice ones.

Perhaps the variant can still act intelligently. Perhaps it cannot. Perhaps there's a clever way to train it so that it happens to work in this particular game. I don't know. My point is only that AIXItl was designed with an optimality property in mind ("superior to any other time t and length l bounded algorithm"), and that an embodied AIXItl lacks this property (regardless of the "router").

This post attempts to explain why AIXItl is not a superior agent, and attempts to impart an intuition for why the field of AGI is not reducible to constructing better and better approximations of AIXI. It sounds like you already believe this point, so I won't try to convince you of it further :-)

Rather, I think any problem AIXI has in reasoning about itself is either one humans also have in reasoning about themselves or analogous to a problem it has reasoning about other things. In this case it is a problem humans also have.

Be that as it may, the argument "humans can't do X" is not a compelling reason to stop caring about X. It seems to me that an ideal artificial agent should be able to win at the HeatingUp game.

However, my concern is that in addition to its usual stupidity, you think AIXI-tl has an additional obstacle in terms of some sort of 'Cartesian boundary problem', and that there exists some sort of 'naturalized induction' which humans have and which AIXI and AIXI-tl don't have. I am unconvinced by this, and I think it is an unproductive line of research.

I have made no claims in this post about research strategy, I'm only trying to point out why AIXI is not an ideal (as this concept still seems foreign to many people). Again, it seems you are already on board with this idea.

I hear your complaints about what you've assumed are my research goals, but I think we should save the research strategy discussion for another place and time :-)

Intuitively, this limitation could be addressed by hooking up the AIXItl's output channel to its source code. Unfortunately, if you do that, the resulting formalism is no longer AIXItl.

I dispute this. Any robot which instantiates AIXI-tl must consist of two parts: First, there must be a component which performs the actual computations for AIXI-tl. Second, there is a router, which observes the robot's environment and feeds it to the first compoment as input, and also reads the first component's output and translates it into an action the robot performs. The design of the router must be neccessity make additional arbitrary choices not present in the pure description of AIXI-tl. For example, the original description of AIXI described the output as a bit-string, which in this scenario must somehow be converted into a constree for the output register. If the router is badly designed then it can make problems that no program of any intelligence can overcome. For example, imagine the router can't perform the action 'move right'.

The problem described here is not at all in AIXI-tl, but entirely in the design of the router. This can be seen from how at no point you look into the internal components of AIXI-tl or what output it would generate. If you allowed the router to change the internal registers of the robot, it would still be AIXI-tl, just that it would have a different output router.

I think that if the robot use such a router then it would kill itself in experimentation before it would have the chance to solve the problem, but you haven't established that. I would like to see an argument against AIXI-tl that does not really rely what it is or is not physically capable of doing, but rather on what it is intelligent enough to choose to do. After all, humans, despite supposedly being capable of "naturalized induction", would not do well in this problem either. A human cannot by force of will reprogram her brain into a static set of commands, nor can she make her brain stop emitting heat.

Finally, I want to say why I am making these arguments. It is not because I want to advocate for AIXI-tl and argue for its intelligence. The way I think of it AIXI is the dumbest program that is still capable of learning the right behavior eventually. Actually it's worse than that; my argument here has convince me that even with exponential resources AIXI-tl can't argue itself out of a paper bag (Note argument does look into the internals of AIXI-tl rather than treating it as a black-box). So if anything I think you might be overestimating the intelligence of AIXI-tl. However, my concern is that in addition to its usual stupidity, you think AIXI-tl has an additional obstacle in terms of some sort of 'Cartesian boundary problem', and that there exists some sort of 'naturalized induction' which humans have and which AIXI and AIXI-tl don't have. I am unconvinced by this, and I think it is an unproductive line of research. Rather, I think any problem AIXI has in reasoning about itself is either one humans also have in reasoning about themselves or analogous to a problem it has reasoning about other things. In this case it is a problem humans also have.

According to this analogy, we previously thought e to be 2. Now it's 2.6 recurring. We're making progress, of a sort.

But suppose I'm missing something, and there is a genuine non-arbitrary distinction between being convinced and being coerced.

There need not be a distinction between them. If you prefer, you could contrast an AI willing to "convince" its humans to behave in any way required, with one that is unwilling to sacrifice their happiness/meaningfulness/utility to do so. The second is still at a disadvantage.

Is there a convenient place to see just what changed from the old to the new?

Online diff tools aren't usefully handling the paragraphs when I copy-paste, and my solution of download -> insert line breaks -> run through my favorite diff program is probably inconvenient for most.

People are (rightly or wrongly) less concerned about putting their best foot forward with text.

As an ergonomic matter, typing all day long, although fatiguing, consumes less energy than talking all day long.

Text can be created in fits and bursts. An audio or video needs to be recorded more or less in a continuous sitting.

Note that many people seem to prefer e.g. Skype calls over text chats because (to these people) voice chat requires less energy than writing, and feels like just having a normal conversation and thus effortless, whereas writing is something that requires actually thinking about what you say and thus feels much more laborious.

A lot of people also seem to find audio easier to consume than text: podcasts would be a lot less popular otherwise. (I never understood podcasts at first. Why not just write? Finally I realized that non-nerds actually find listening easier than reading.)

You can't play background music while having a video conversation

Headphones and a good call quality together fix this, I think? Haven't tried, though.