It would be a powerful tool to be able to dismiss fringe phenomena, prior to empirical investigation, on firm epistemological ground.
Thus I have elaborated on the possibility of doing so using Bayes, and this is my result:
Using Bayes to dismiss fringe phenomena
What do you think of it?
Going outside the argument, if you come up with 50% chance that any given observation of a UAP was actually observing a UAP, you've done something wrong.
More specifically,
P(UAP): What is the prior probability that a broadly unknown aerial phenomena UAP exists? In this case of UAP's, there exists large bodies of observation data. One governmental study of 3200 observations concluded that 22% of the observations could not be identified. These were separate from another 9% of cases which could not be identified due to insufficient observation data. Thus we can tentatively assess P(UAP) to be 0.22
Didn't click through to the study, but what is an 'observation' in this case? I make 3200 observations a day, but I don't observe 70 UAPs a day. It seems far more likely that this was 3200 instances of someone reporting a UAP - in other words, the 22% is pointing towards P(UAP|UO), not P(UAP). But "we couldn't identify this" doesn't necessarily mean it was a UAP, so this is still an overestimate.
We don't need to (and for the sake of internal consistency, shouldn't) estimate this probability on its own.
You probably should, for sanity checking. You calculate P(UO1) as 0.332, which is clearly way too high - I think that most people don't see a single thing in their lifetime that they think is a UAP. If you estimate something, and then calculate a number which is different my that many orders of magnitude, you can go back and check your workings.
Previously we decided P(UO1|UAP) to be 0.8 [...] Thus P(UO1|¬UAP) must be 1−0.8=0.2
No.
You're correct about the study. What they actually found was that a certain fraction of UFO reports (I.e., what loldrup calls UO) had reported descriptions that didn't match any known class of object. So yes, it's more like P(UAP|UO) in loldrup's notation; and yes, it's not "thing known not to be a known class of object" but "thing whose reported description we didn't find a good match for" which is of course consistent not only with what loldrup calls the UAP hypothesis but also with inaccurate reporting and with known classes of object having currently-unknown behaviour.
[EDITED to fix formatting.]
Going outside the argument, if you come up with 50% chance that any given observation of a UAP was actually observing a UAP, you've done something wrong.
This percentage is conditional on P( observation | UAP ) being high (0.8). I guess only a minority of such observations (if any) are that high.
Didn't click through to the study, but what is an 'observation' in this case? I make 3200 observations a day, but I don't observe 70 UAPs a day. It seems far more likely that this was 3200 instances of someone reporting a UAP - in other words, the 22% is pointing towards P(UAP|UO), not P(UAP). But "we couldn't identify this" doesn't necessarily mean it was a UAP, so this is still an overestimate.
If this is so, then I can't use any empirical study to estimate P( UAP ). What value should I then guesstimate P( UAP ) to be? If I choose a generic 0.5, then the result will be an even higher P( UAP | observation ) than 0.53. Thus it would be even harder to dismiss the observation.
If I instead make my own estimation, say, 0.0000001, then one could ask if this was still an epistemologically sound approach. My choice would be rather arbitrary.
Previously we decided P(UO1|UAP) to be 0.8 [...] Thus P(UO1|¬UAP) must be 1−0.8=0.2
No.
Could you elaborate?
Thanks for your feedback :)
If you don't have an epistemically sound approach, then you should probably say "I don't know" instead of using an epistemically unsound one, or at least say "this is really bad, and you shouldn't put high confidence in my conclusions, but it's the best I can do, so..."
That said, one option you have is not to calculate P(UAP) at all, and instead calculate a likelihood ratio:
P(UAP|UO1) / P(¬UAP|UO1) = P(UO1|UAP) / P(UO1|¬UAP) × P(UAP)/P(¬UAP)
So if you just calculate P(UO1|UAP) / P(UO1|¬UAP), then anyone can update their P(UAP) appropriately, regardless of where it started.
Could you elaborate?
P(rain | clouds) might be something like 0.7, and that means that P(¬rain | clouds) is 0.3. But P(rain | ¬clouds) is 0.
You simply can't calculate P(UO1|¬UAP) from P(UO1|UAP). You need to work it out some other way.
I also don't think that asking for P(UO1|UAP) and P(UO1|¬UAP) is reasonable without knowing anything about UO1. Right now I'm observing my watch tick; that's no more or less likely to happen in UAP-world than ¬UAP-world, so the likelihood ratio is one. If tomorrow night I go outside and see lots of bright lights in the sky, and a crop circle the next morning (which is especially weird because there didn't used to be any crops there at all), and the news reports that lots of other people have seen the same thing and the government is passing it off as a sighting of Venus, then that's somewhat more likely in UAP-world than ¬UAP-world.
If tomorrow night I go outside and see lots of bright lights in the sky, and a crop circle the next morning (which is especially weird because there didn't used to be any crops there at all), and the news reports that lots of other people have seen the same thing and the government is passing it off as a sighting of Venus, then that's somewhat more likely in UAP-world than ¬UAP-world.
This example suggests that you're confusing P(OU1|UAP) with P(UAP|OU1). To determine P(OU1|UAP), image you live in a world where UAP is true.
Unfortunately, the analysis so far hasn't been clear on what we mean by P(UAP): does it mean the probability that there are alien visitors within Earth's atmosphere or that their ship is flying over head right now?
Assuming the former, to estimate P(OU1|UAP) assume there are aliens on Earth, if that is the case what's the probability of you observing the light in the sky. Obviously this is hard to estimate but one would start by speculating about the potential motives and behavior of the aliens.
I'm afraid I can't tell which direction you think I'm confused in. That example was intended to be an instance of UO1 for which P(UO1|UAP) > P(UO1|¬UAP), and that still seems true to me, even if P(UO1|UAP) is still low.
(I'm taking UAP to be something like "Earth is sometimes visited by aliens".)
If you don't have an epistemically sound approach, then you should probably say "I don't know" instead of using an epistemically unsound one, or at least say "this is really bad, and you shouldn't put high confidence in my conclusions, but it's the best I can do, so..."
Instead of relying on dubious priors couldn't one simply avoid having to reliably estimate a prior probability P(UAP) by choosing a canonical dataset of observations, choosing a generic prior P(UAP) = 0.5 and then repeatedly update P(UAP | observation x) for each observation x in the dataset?
In this way, the unreliable prior should gradually be deluted, through the iterations. In the end, it will be overshadowed by the influence of the canonical observation data.
If so, how could one do this programmatically? And how could one do this analytically? (links are welcome!)
I also hinted at these options in the section 'Future work' in the article. But I don't know how to approach this approach..
I also don't think that asking for P(UO1|UAP) and P(UO1|¬UAP) is reasonable without knowing anything about UO1. Right now I'm observing my watch tick; that's no more or less likely to happen in UAP-world than ¬UAP-world, so the likelihood ratio is one. If tomorrow night I go outside and see lots of bright lights in the sky, and a crop circle the next morning (which is especially weird because there didn't used to be any crops there at all), and the news reports that lots of other people have seen the same thing and the government is passing it off as a sighting of Venus, then that's somewhat more likely in UAP-world than ¬UAP-world.
As the goal is to say something prior to investigating the observation, I must assume as little as possible about the nature of the given observation. In the article I assumed P ( observation | UAP ) to be 0.8.
If I could reuse this bit of information to say something about P(UO1|UAP) and P(UO1|¬UAP), then I haven't broken the "let's assume as little as possibly"-premise any further.
Is that bit of information sufficient to say something useful about P(UO1|UAP) and P(UO1|¬UAP)?
canonical dataset of observations [...] unreliable prior should gradually be diluted
Indeed, if you have enough observations then the prior eventually doesn't matter. The difficulty is in the selection of the observations. Ideally you should include every potentially relevant observation -- including, e.g., every time someone looks up at the sky and doesn't see an alien spaceship, and every time anyone operates a radar or a radio telescope or whatever and sees nothing out of the ordinary.
In practice it's simply impractical to incorporate every potentially relevant observation into our thinking. But that makes it awfully easy to have some bias in selection, and that can make a huge difference to the conclusions.
In practice it's simply impractical to incorporate every potentially relevant observation into our thinking. But that makes it awfully easy to have some bias in selection, and that can make a huge difference to the conclusions.
Yes these circumstances induce bias and this is unfortunate if one wants to say anything about frequency and such things.
Another somewhat simpler question is this: given n observations of something the observer thinks is a UAP, what is the probability that at least one of these observations originated from a UAP?
If for each of these observations P( observation | UAP ) is strictly greater than 0, then I suspect P(UAP) will go towards 1, monotonously, as the number of observations increases.
Is this hunch somewhat correct? How do I express this hunch mathematically..?
I also touch on this question in the section 'Future work' in my article, but I don't have the answer.
If for each of these observations P( observation | UAP ) is strictly greater than 0, then I suspect P(UAP) will go towards 1, monotonously, as the number of observations increases.
No. This violates the law of conservation of expected evidence. The relevant question is whether P( observation | UAP ) is bigger or smaller than P( observation | ~UAP ).
The problem, as I mentioned above, is that it's hard to estimate P( observation | UAP ).
What if we have n observations where P( observation | ~UAP ) through investigation has been found to be 0 and, while hard to determine, P( observation | UAP ) is reasonably said to be strictly greater than 0.
Then P(UAP) will go towards 1, monotonously, as the number of observations increases, right?
What if we have n observations where P( observation | ~UAP ) through investigation has been found to be 0
Um, I don't think you understand what it means for P( observation | ~UAP ) to equal 0. If P( observation | ~UAP ) were really 0, then a single such observation would be enough to comclude the P(UAP) is 1.
So how should one interpret findings like this: "We investigated n observations and out of these there were k observations which had sufficient observation data to rule out all known aerial phenomena as being the cause".
So that would imply that P(UAP) is pretty much 1?
So what remains is "merely" to determine what lies in this set 'UAP' as it could pretty much be anything.
So how should one interpret findings like this: "We investigated n observations and out of these there were k observations which had sufficient observation data to rule out all known aerial phenomena as being the cause".
If I take that statement at face value it means the observations were caused by some unknown phenomenon. Therefore, unknown phenomena of this type exist.
What value should I then guesstimate P( UAP ) to be? If I choose a generic 0.5, then the result will be an even higher P( UAP | observation ) than 0.53. Thus it would be even harder to dismiss the observation.
When I think of it, I recall I'm not the only one trying to estimate P(UAP). The Fermi paradox concludes something like this:
"Aliens surely do exist, so how come we don't see them?"
In this sentence lies an estimate of P(UAP). The only problem is that it is rather high ( 'surely' == 0.9 ?), thus making it even harder again to dismiss P( UAP | observation )
That isn't an estimate of P(UAP). The following two propositions are very different:
The first of these can be true without the second -- maybe there are aliens but they never come here. (That was rather the point of the Fermi paradox.)
The second can be true without the first -- maybe people have actually seen angels, or time travellers from earth's future, or spontaneous rips in the fabric of spacetime.
Fringe phenomena is an interesting reference class. What do you consider it to mean?
A lot of scientists invest a lot of effort into replicating hard to detect effects. Are they chasing fringe phenomena? Is the reason that they aren't that they are respected members of the establishment?
Generally, the hard to detect effects that those scientists research are not ones which
-- have been enthusiastically promoted by nonscientists
-- conflict with basic scientific principles that are well-studied and well-understood (often the proponents don't even understand that what they are suggesting conflicts with such principles)
-- have been well-studied themselves and already found to be false
-- conflict with basic scientific principles that are well-studied and well-understood (often the proponents don't even understand that what they are suggesting conflicts with such principles)
Given the Fermi paradox the existence of aliens doesn't violate scientific principles. Yet aliens are outside of what you can study scientifically.
-- have been well-studied themselves and already found to be false
When it comes to that class we don't really talk about judging them "prior to empirical investigation".
-- have been enthusiastically promoted by nonscientists
It looks like it. It's about status.
It looks like it. It's about status.
No, it's a Bayseian update based on "the probability that something is true, conditional on being enthusiastically promoted by nonscientists and rejected or ignored by scientists, is really really low". That's what you use Bayseian updates for. Science works; it may not do so with complete certainty, but the odds heavily favor it.
It's no more about status than wanting to go to a medical doctor instead of a faith healer is about status.
A lot of things get ignored by scientists because you don't get funding for studying the topic. Xrisk would be a good example. FHI finds it hard to raise money via the traditional way for the subject.
Science works
It works through empirical investigation. It doesn't do much prior to empirical investigation.
As a heuristic, I suspect ignoring things ignored by most scientists will actually work pretty well for you. Its not an unreasonable assumption to say that "given no other information, the majority of scientists dismissing a subject lowers my probability that that subject has any grounding". Thats a sensible thing to do, and does indeed use a simple Bayesian logic.
Note that we essentially do this for all science, in that we tend to accept the scientific consensus. We can't be subject specialists in everything, so while we can do a bit of reading, its probably fine to just think: what most scientists think is probably the closest to correct I am capable of being without further study.
As a heuristic, I suspect ignoring things ignored by most scientists will actually work pretty well for you. Its not an unreasonable assumption to say that "given no other information, the majority of scientists dismissing a subject lowers my probability that that subject has any grounding".
If you don't have any information then that might be true. Usually you however do have some information.
Note that we essentially do this for all science, in that we tend to accept the scientific consensus.
That's only true for fields that are studied enough for there to be an evidence based scientific consensus.
As a heuristic, I suspect ignoring things ignored by most scientists will actually work pretty well for you.
There is an interesting exception -- if you are scientist yourself.
A lot of things get ignored by scientists because you don't get funding for studying the topic.
Bayseian calculation doesn't work on "a lot", it works on the odds, and the odds are much lower for such things.
As a prequel to wiring my article, I looked into studies of UAPs. None of these studies concluded that all investigated UAP's turns out to be either known phenomena, or solely unidentifiable due to insufficient observation data. All studies show that a minor percentage of UAPs resist identification (between 5 and 20 percent).
Thus we cannot say that we currently have a scientific understanding of all aerial phenomena.
When this is combined with Fermi's paradox, how come we don't conclude that we should study them some more?
If there's any inaccuracy in the reporting, any mundane event can "resist identification". Eyewitnesses are not as accurate as most people think they are.
And while you claim that ones that have insufficient observation data were excluded, I'll believe it when I see a study, because that can mean a lot of things. (If there is enough information to rule out causes X and Y, but not Z or A, is that 'insufficient observation data'? What if they ruled out all sorts of causes but didn't rule out the possibility of, say, a hoax?)
I applaud looking at the studies. I included references to 7 studies and 4 case collections (including one collection solely of radar backed observations) in the References section of my article:
P(A | B) is not equal to 1 - p(A | not B). You are thinking p(A | B) = 1 - p(not A | B). Example:
p(A=0,B=0) = 0.1, p(A=0,B=1) = 0.2, p(A=1,B=0) = 0.3, p(A=1,B=1) = 0.4.
p(A=0 | B=0) = p(A=0,B=0) / ( p(A=1,B=0) + p(A=0,B=0) ) = 0.1 / (0.3+0.1) = 0.1/0.4 = 1/4
p(A=0 | B=1) = p(A=0,B=1) / ( p(A=0,B=1) + p(A=1,B=1) ) = 0.2 / (0.2+0.4) = 0.2/0.6 = 2/6 = 1/3
1/4 is not 1 - 1/3.
Someone else pointed this out already, are you updating on basic math errors?
I am aware of the error and will correct it - it's on my todo list :)
[EDIT] fixed! (hopefully!)
Hint: what happened to UFO sightings once everyone started to carry a high-resolution camera (in a smartphone) with them at all times?
What happened is that UFO sightings basically disappeared. Turns out it's much harder to talk about seeing UFOs when you can't answer the question "So, why didn't you take a picture with your phone?"
references?
[edit]: This graph shows the frequency of reported UFO sightings inn Canada over the last 25 years. There is a steady increase in sightings over the years:
http://www.canadianuforeport.com/survey/images/ttlreports2013.gif
The graph originates from this survey: http://survey.canadianuforeport.com/
conducted by the Canadian astronomer Chris A. Rutkowski and non-astronomer Geoff Dittman
I'm partial to the reference class, "theories that make lots of excuses for why it's hard to confirm or reject when it should be very easy, but nonetheless an ape-like creature ran into it one day."
I believe that what you have proven is that it will probably not help your career to investigate fringe phenomena. Unfortunately, science needs the occasional martyr who is willing to be completely irrational in their life path (unless you assign a really large value to having "he was right after all" written on your tombstone) while maintaining very strict rationality in their subject of interest. For example, the theory that "falling stars were" were caused rocks falling out of the sky was considered laughable since this had already been lumped together with ghosts, etc.
If you don't have any information then that might be true. Usually you however do have some information.
That's only true for fields that are studied enough for there to be an evidence based scientific consensus.