could you provide a hard philosophical problem (of the kind for which feedback is impossible) together with an argument that this problem must be resolved before human-level AGI arrives?

I can't provide a single example because it depends on the FAI design. I think multiple design approaches are possible but each involves its own hard philosophical problems.

To try to make my point clearer (though I think I'm repeating myself): we can aim to build machine intelligences which pursue the outcomes we would have pursued if we had thought longer (including machine intelligences that allow human owners to remain in control of the situation and make further choices going forward, or bootstrap to more robust solutions). There are questions about what formalization of "thought longer" we endorse, but of course we must face these with or without machine intelligence.

At least one hard problem here is, at you point out, how to formalize "thought longer", or perhaps "remain in control". Obviously an AGI will inevitably influence the options we have and the choices we end up making, so what does "remain in control" mean? I don't understand your last point here, that "we must face these with or without machine intelligence". If people weren't trying to build AGI and thereby forcing us to solve these kinds of problems before they succeed, we'd have much more time to work on them and hence a much better chance of getting the answers right.

For the most part, the questions involved in building such an AI are empirical though hard-to-test ones---would we agree that the AI basically followed our wishes, if we in fact thought longer?---and these don't seem to be the kinds of questions that have proved challenging, and probably don't even count as "philosophical" problems in the sense you are using the term.

If we look at other empirical though hard-to-test questions (e.g., what security holes exist in this program) I don't see much reason to be optimistic either. What examples are you thinking of, that makes you say "these don't seem to be the kinds of questions that have proved challenging"?

I suspect I also object to your degree of pessimism regarding philosophical claims, but I'm not sure and that is probably secondary at any rate.

I'm suspecting that even the disagreement we're current discussing isn't the most important one between us, and I'm still trying to figure out how to express what I think may be the most important disagreement. Since we'll be meeting soon for the decision theory workshop, maybe we'll get a chance to talk about it in person.