In the chapter 5 of the Probability Theory: Logic of Science you can read about so-called device of imaginary results which seems to go back to the book of I J Good named Probability and the Weighing of Evidence.
The idea is simple and fascinating:
1) You want to estimate your probability of something, and you know that this probability is very, very far from 0.5. For the sake of simplicity, let's assume that it's some hypothesis A and P(A|X) << 0.5
2) You imagine the situation where the A and some well-posed alternative ~A are the only possibilities.
(For example, A = "Mr Smith has extrasensory perception and can guess the number you've written down" and ~A = "Mr Smith can guess your number purely by luck". Maybe Omega told you that the room where the experiment is located makes it's impossible for Smith to secretly look at your paper, and you are totally safe from every other form of deception.)
3) You imagine the evidence which would convince you otherwise: P(E|A,X) ~ 1 and P(E|~A,X) is small (you should select E and ~A that way that it's possible to evaluate P(E|~A,X) )
4) After a while, you feel that you are truly in doubt about A: P(A|E1,E2,..., X) ~ 0.5
5) And now you can backtrack everything back to your prior P(A|X) since you know every P(E|A) and P(E|~A).
After this explanation with the example about Mr Smith's telepathic powers, Jaynes gives reader the following exercise:
Exercise 5.1. By applying the device of imaginary results, find your own strength of
belief in any three of the following propositions: (1) Julius Caesar is a real historical
person (i.e. not a myth invented by later writers); (2) Achilles is a real historical person;
(3) the Earth is more than a million years old; (4) dinosaurs did not die out; they are
still living in remote places; (5) owls can see in total darkness; (6) the configuration of
the planets influences our destiny; (7) automobile seat belts do more harm than good;
(8) high interest rates combat inflation; (9) high interest rates cause inflation.
I have trouble tackling the first two propositions and would be glad to hear your thoughts about another seven. Anybody care to help me?
(I decided not to share details of my attempt to solve this exercise unless asked. I don't think that my perspective is so valuable and anchoring would be bad.)
UPD: here is my attempt to solve the Julius Caesar problem.
(As promised, I post description of my attempts here)
Case 1: Julius Caesar. The existence of Caesar seems very likely for me. Therefore, I will think about evidence that would convince me that Caesar is a myth.
I decided that the following three pieces of evidence will be enough for me to start doubting existence of Julius Caesar:
1) I don't know anything about the process of burial of the Roman emperors. Hence it wouldn't be inconsistent to assume that there is an official "emperor's tomb", a luxurious necropolis where the crypts of every emperor are located. Having assumed that this true, I imagine a discovery saying that the crypt of Julius Caesar is missing, or his tomb is empty, or the body inside couldn't belong to Julius, or something along those lines.
2) Similarly, the existence of a reliable independent Arabian historian similar to Herodotus wouldn't contradict my worldview. So I can assume that there indeed was such a historian. If his book about Roman Empire had failed to mention Julius Caesar at all, that would have been an evidence pointing to Julius being a myth.
3) Why can't I imagine the second Arabian history book failing to mention Caesar?
Having imagined all that, I decided that it would be enough for me to start doubting Julius Caesar's existence.
Let's turn to the probabilities.
1) P(no crypt|no Julius Caesar) ~ 1. P(no crypt|Julius Caesar) = ?
It doesn't look like there's some way to easily estimate that quality. I notice that I'm confused, but let's try anyway. It's possible to convert this probability into the relative frequency: there were cases of some monarch's bodies having been removed from their graves. I know two such cases: False Dmitriy I of Russian Tsars and Akhenaten of the Eighteenth dynasty of Egypt. Counting the other people of their lineage gives the approximate figure around 2/350 which I consider kind of satisfying, though it appears to be somewhat higher than desired. However, I had to consult Wikipedia to get this estimate, and that kind of violates the "no gathering other relevant real-world data" rule.
2) and 3) P(no mention of Caesar in the Arabic book|no Julius Caesar) ~ 1 P(no mention of Caesar in the Arabic book|Julius Caesar) = ?
I have trouble estimating that at all. I have no slightest idea how to estimate this without a lot of imaginary betting, and imaginary betting kind of defeats the whole point. Why shouldn't I estimate P(Julius Caesar existed) via betting?
Well, betting makes sense only when my utility function is linear in money, and that holds only when the probabilities are sufficiently close to 0.5. Maybe I should break my prior into several parts via Good's device and then estimate the parts via betting.
It seems plausible. However, I think that if Jaynes had meant his exercise to be done in that way, he would have explicitly told so.
I notice that I'm confused. Let's try to find my mistake:
1) The alternative hypothesis "Julius Caesar existed" is too vague, and hence it is difficult to come up with the conditional probabilities. It seems likely, but I can't come up with something better.
2) The experiment is bad. I should think about something else: for example, I draw a random portrait of Roman emperor from the urn containing portraits of every Roman emperor, how many non-Caesar's portraits are enough? Coming up with something like that seems very difficult for me. I'm very bad at selling non-apples.
This is a reason I wanted to find a new perspective and don't wanted to spoil anyone by anchoring him/her to my vision.
3) The evidence is bad. I should come up with some new evidence such that I can calculate the probability P(E|Julius Caesar)
4) There is a good way to estimate conditional probabilities, I just missed it.
I haven't worked much with the other cases, but it seems that it would be difficult to calculate their relevant probabilities as well (how can I calculate anything about geology in (3), for example?). So I think that I've misunderstood something, and maybe someone there can describe the correct way of doing this exercise to me.
It seems to me that it would be more effective to work from evidence that you have encountered personally or in the case of hypothetical evidence, could have hypothetically encountered. In the case of historical figures, unless you happen to be an archaeologist yourself, the majority of the evidence you have is through secondary and tertiary sources. For example, if a publication alleged that Julius was a title, not a name, and was used by many Caesars, and thus many acts attributed to the person Julius Caesar were in fact performed by separate individuals... (read more)