What epistemic algorithms would you run to discover more about your arbitrary preferences and to make sure you were interpreting them correctly? (Assuming you don't have access to an FAI.) For example, what kinds of reflection/introspection or empiricism would you do, given your current level of wisdom/intelligence and a lot of time?
It's a good question, and ruling out the FAI takes away my favourite strategy!
One thing I consider is how my verbal expressions of preference will tend to be biased. For example if I went around saying "I'd willingly give up immortality to prevent 28 strangers from starving" then I would triple check my belief to see if it was an actual preference and not a pure PR soundbite. More generally I try to bring the question down to the crude level of "what do I want?", eliminating distracting thoughts about how things 'should' be. I visualiz...
From Mike Darwn's Chronopause, an essay titled "Would You Like Another Plate of This?", discussing people's attitudes to life:
Conclusion, graphs, and references in article. As usual, I recommend reading Chronopause.com as Darwin has many good articles; to quickly link a few: