I haven't said we haven't some way of determining the numbers. I have said that I can't concisely formulate a rule whose domain of definition is the set of all possible information. What you are asking for is basically outlining a large part of the code of a general artificial intelligence. This is out of reach, but it doesn't mean we can't update at all. Some probabilities plugged in will almost certainly be generated by intuition, but I don't think method of reasoning has to remove all arbitrariness to be called such.
What you are asking for is basically outlining a large part of the code of a general artificial intelligence.
Kind of! I'm asking for the best algorithm for human intelligence we can come up with. I guess that indeed, the phrase "apply Bayes' rule" is significantly better than nothing at all.
I've been on Less Wrong since its inception, around March 2009. I've read a lot and contributed a lot, and so now I'm more familiar with our jargon, I know of a few more scientific studies, and I might know a couple of useful tricks. Despite all my reading, however, I feel like I'm a far cry from learning rationality. I'm still a wannabe, not an amateur. Less Wrong has tons of information, but I feel like I haven't yet learned the answers to the basic questions of rationality.
I, personally, am a fan of the top-down approach to learning things. Whereas Less Wrong contains tons of useful facts that could, potentially, be put together to answer life's important questions, I really would find it easier if we started with the important questions, and then broke those down into smaller pieces that can be answered more easily.
And so, that's precisely what I'm going to do. Here are, as far as I can tell, the basic questions of rationality—the questions we're actually trying to answer here—along with what answers I've found:
Q: Given a question, how should we go about answering it? A: By gathering evidence effectively, and correctly applying reason and intuition.