Why do you think that any logical language (of the sort we're currently familiar with) is sufficiently expressive for this purpose?

The general idea seems right. If the existing languages are inadequate, they at least seem adequate for a full-featured prototype: figure out decision theory (and hence notion of preference) in terms of standard logic, then move on as necessary for extending expressive power. This should stop at some point, since this exercise at formality is aimed at construction of a program.

I'm not sure. One way to think about it is whether the question "what is the right prior?" is more like "what is the right decision theory?" or more like "what is the right utility function?" In What Are Probabilities, Anyway? I essentially said that I lean towards the latter, but I'm highly uncertain.

I don't see clearly the distinction you're making, so let me describe how I see it. Some design choices in constructing FAI would certainly be specific to our minds (values), but the main assumption to my approach to FAI is exactly that a large portion of design choices in FAI can be specified as a natural category in human brains, something we can point a simple mirror at and say "there!", with the mirror doing most of the work in determining what goes into the FAI. I call the automated design choices "preference", and the mirror (theory of mirror) "decision theory", with the slot "notion of preference" that is to be filled in automatically. So, there is no question of which one of "decision theory" and "preference" is "essential", both play a role. The worry is about the necessary size of the manually designed "decision theory" part, and whether it's humanly possible to construct it.