That is exactly the problem I am trying to address. On one hand, I can't figure out how to estimate the likelihood of a situation. On the other hand, it's quite evident that some people would fake a picture as mentioned above since many people find it of some importance. I just can't figure out how to try and evaluate the likelihood of one versus the other. When should I be confused?
But how can I apply this sort of logic to the problems I've described above? It still seems to me like I need in theory to sum over all of the probabilities in some set A that contains all these improbable events but I just don't understand how to even properly define A, as its boundaries seem fuzzy and various thing "kinda fit" or "doesn't quite fit, but maybe?" instead of plain true and false.
My main point I think, is that this is a more general problem. Some configurations of observations can seem extremely unlikely, yet the sum over all of these configurations might be fairly probable. Like if an airplane has an engine failure above your home town, and is about to crash unto it. The probability of it crashing right near your house is small (if you live in a big town), but it has to crash near someone's house. And that person that had the airplane crash right on his front lawn would go and say "What do you know, what are the odds? So...
I think it's worth dividing blackmail into two distinct types:
1. Blackmailing on information that is harmful to society.
2. Blackmailing on information that is not harmful to society, but which the victim feels private about.
Your arguments stand somewhat well for the first type. For example, if one is stealing money from the cash register where he works on a weekly basis, then we would not want such behavior to persist. But for the latter type, for example, if someone is secretly a homosexual and is afraid of what his family would say or do if they kne...
I think the title is a little bit misleading, and perhaps he didn't put much emphasis on this, but it seems he isn't claiming correct models are generally bad, just that there are also possible downsides to holding correct models and it's probably a good idea to be aware to these flaws when applying these models to reality.
Also, it seems to me as he is defining 'correct model' as a model in which the reasoning is sound and can be used for some applications, however does not necessarily fully describe every aspect of the problem.
Finally the true face of the LessWrong admins is revealed. I would do the sensible thing and go back to using myspace.