Of course, Bayes' theorem has the obvious problem that carrying out all of the necessary calculations is practically impossible. I mentioned a bunch of properties that a good system (to take a hint from roryokane, an algorithm) ought to have; surely we can come up with something that has those properties, without being impossible for a human to execute.
When creating such a general algorithm, we must keep a human limitation in mind: subconscious, unsystemized thought. A practical algorithm must account for and exploit it.
There are two types of subconscious thought that an algorithm has to deal with. One is the top-level type that is part of being a human. It is only our subconscious that can fire off the process of choosing to apply a certain conscious algorithm. We won’t even start running our algorithm if we don’t notice that it applies in this situation, or if we don’t remember it, or if we feel bored ...
So, I've been thinking. We ought to have a system for rationality. What do I mean?
Well, consider a real-time strategy game like Starcraft II. One of the most important things to do in SC2 is macromanagement: making sure that your resources are all being used sensibly. Now, macromanagement could be learned as a big, long list of tips. Like this: