Outside the Laboratory
"Outside the laboratory, scientists are no wiser than anyone else." Sometimes this proverb is spoken by scientists, humbly, sadly, to remind themselves of their own fallibility. Sometimes this proverb is said for rather less praiseworthy reasons, to devalue unwanted expert advice. Is the proverb true? Probably not in an absolute sense. It seems much too pessimistic to say that scientists are literally no wiser than average, that there is literally zero correlation.
But the proverb does appear true to some degree, and I propose that we should be very disturbed by this fact. We should not sigh, and shake our heads sadly. Rather we should sit bolt upright in alarm. Why? Well, suppose that an apprentice shepherd is laboriously trained to count sheep, as they pass in and out of a fold. Thus the shepherd knows when all the sheep have left, and when all the sheep have returned. Then you give the shepherd a few apples, and say: "How many apples?" But the shepherd stares at you blankly, because they weren't trained to count apples - just sheep. You would probably suspect that the shepherd didn't understand counting very well.
Now suppose we discover that a Ph.D. economist buys a lottery ticket every week. We have to ask ourselves: Does this person really understand expected utility, on a gut level? Or have they just been trained to perform certain algebra tricks?
One thinks of Richard Feynman's account of a failing physics education program:
"The students had memorized everything, but they didn't know what anything meant. When they heard 'light that is reflected from a medium with an index', they didn't know that it meant a material such as water. They didn't know that the 'direction of the light' is the direction in which you see something when you're looking at it, and so on. Everything was entirely memorized, yet nothing had been translated into meaningful words. So if I asked, 'What is Brewster's Angle?' I'm going into the computer with the right keywords. But if I say, 'Look at the water,' nothing happens - they don't have anything under 'Look at the water'!"
Suppose we have an apparently competent scientist, who knows how to design an experiment on N subjects; the N subjects will receive a randomized treatment; blinded judges will classify the subject outcomes; and then we'll run the results through a computer and see if the results are significant at the 0.05 confidence level. Now this is not just a ritualized tradition. This is not a point of arbitrary etiquette like using the correct fork for salad. It is a ritualized tradition for testing hypotheses experimentally. Why should you test your hypothesis experimentally? Because you know the journal will demand so before it publishes your paper? Because you were trained to do it in college? Because everyone else says in unison that it's important to do the experiment, and they'll look at you funny if you say otherwise?
No: because, in order to map a territory, you have to go out and look at the territory. It isn't possible to produce an accurate map of a city while sitting in your living room with your eyes closed, thinking pleasant thoughts about what you wish the city was like. You have to go out, walk through the city, and write lines on paper that correspond to what you see. It happens, in miniature, every time you look down at your shoes to see if your shoelaces are untied. Photons arrive from the Sun, bounce off your shoelaces, strike your retina, are transduced into neural firing frequences, and are reconstructed by your visual cortex into an activation pattern that is strongly correlated with the current shape of your shoelaces. To gain new information about the territory, you have to interact with the territory. There has to be some real, physical process whereby your brain state ends up correlated to the state of the environment. Reasoning processes aren't magic; you can give causal descriptions of how they work. Which all goes to say that, to find things out, you've got to go look.
Now what are we to think of a scientist who seems competent inside the laboratory, but who, outside the laboratory, believes in a spirit world? We ask why, and the scientist says something along the lines of: "Well, no one really knows, and I admit that I don't have any evidence - it's a religious belief, it can't be disproven one way or another by observation." I cannot but conclude that this person literally doesn't know why you have to look at things. They may have been taught a certain ritual of experimentation, but they don't understand the reason for it - that to map a territory, you have to look at it - that to gain information about the environment, you have to undergo a causal process whereby you interact with the environment and end up correlated to it. This applies just as much to a double-blind experimental design that gathers information about the efficacy of a new medical device, as it does to your eyes gathering information about your shoelaces.
Maybe our spiritual scientist says: "But it's not a matter for experiment. The spirits spoke to me in my heart." Well, if we really suppose that spirits are speaking in any fashion whatsoever, that is a causal interaction and it counts as an observation. Probability theory still applies. If you propose that some personal experience of "spirit voices" is evidence for actual spirits, you must propose that there is a favorable likelihood ratio for spirits causing "spirit voices", as compared to other explanations for "spirit voices", which is sufficient to overcome the prior improbability of a complex belief with many parts. Failing to realize that "the spirits spoke to me in my heart" is an instance of "causal interaction", is analogous to a physics student not realizing that a "medium with an index" means a material such as water.
It is easy to be fooled, perhaps, by the fact that people wearing lab coats use the phrase "causal interaction" and that people wearing gaudy jewelry use the phrase "spirits speaking". Discussants wearing different clothing, as we all know, demarcate independent spheres of existence - "separate magisteria", in Stephen J. Gould's immortal blunder of a phrase. Actually, "causal interaction" is just a fancy way of saying, "Something that makes something else happen", and probability theory doesn't care what clothes you wear.
In modern society there is a prevalent notion that spiritual matters can't be settled by logic or observation, and therefore you can have whatever religious beliefs you like. If a scientist falls for this, and decides to live their extralaboratorial life accordingly, then this, to me, says that they only understand the experimental principle as a social convention. They know when they are expected to do experiments and test the results for statistical significance. But put them in a context where it is socially conventional to make up wacky beliefs without looking, and they just as happily do that instead.
The apprentice shepherd is told that if "seven" sheep go out, and "eight" sheep go out, then "fifteen" sheep had better come back in. Why "fifteen" instead of "fourteen" or "three"? Because otherwise you'll get no dinner tonight, that's why! So that's professional training of a kind, and it works after a fashion - but if social convention is the only reason why seven sheep plus eight sheep equals fifteen sheep, then maybe seven apples plus eight apples equals three apples. Who's to say that the rules shouldn't be different for apples?
But if you know why the rules work, you can see that addition is the same for sheep and for apples. Isaac Newton is justly revered, not for his outdated theory of gravity, but for discovering that - amazingly, surprisingly - the celestial planets, in the glorious heavens, obeyed just the same rules as falling apples. In the macroscopic world - the everyday ancestral environment - different trees bear different fruits, different customs hold for different people at different times. A genuinely unified universe, with stationary universal laws, is a highly counterintuitive notion to humans! It is only scientists who really believe it, though some religions may talk a good game about the "unity of all things".
As Richard Feynman put it:
"If we look at a glass closely enough we see the entire universe. There are the things of physics: the twisting liquid which evaporates depending on the wind and weather, the reflections in the glass, and our imaginations adds the atoms. The glass is a distillation of the Earth's rocks, and in its composition we see the secret of the universe's age, and the evolution of the stars. What strange array of chemicals are there in the wine? How did they come to be? There are the ferments, the enzymes, the substrates, and the products. There in wine is found the great generalization: all life is fermentation. Nobody can discover the chemistry of wine without discovering, as did Louis Pasteur, the cause of much disease. How vivid is the claret, pressing its existence into the consciousness that watches it! If our small minds, for some convenience, divide this glass of wine, this universe, into parts — physics, biology, geology, astronomy, psychology, and so on — remember that Nature does not know it! So let us put it all back together, not forgetting ultimately what it is for. Let it give us one more final pleasure: drink it and forget it all!"
A few religions, especially the ones invented or refurbished after Isaac Newton, may profess that "everything is connected to everything else". (Since there is a trivial isomorphism between graphs and their complements, this profound wisdom conveys exactly the same useful information as a graph with no edges.) But when it comes to the actual meat of the religion, prophets and priests follow the ancient human practice of making everything up as they go along. And they make up one rule for women under twelve, another rule for men over thirteen; one rule for the Sabbath and another rule for weekdays; one rule for science and another rule for sorcery...
Reality, we have learned to our shock, is not a collection of separate magisteria, but a single unified process governed by mathematically simple low-level rules. Different buildings on a university campus do not belong to different universes, though it may sometimes seem that way. The universe is not divided into mind and matter, or life and nonlife; the atoms in our heads interact seamlessly with the atoms of the surrounding air. Nor is Bayes's Theorem different from one place to another.
If, outside of their specialist field, some particular scientist is just as susceptible as anyone else to wacky ideas, then they probably never did understand why the scientific rules work. Maybe they can parrot back a bit of Popperian falsificationism; but they don't understand on a deep level, the algebraic level of probability theory, the causal level of cognition-as-machinery. They've been trained to behave a certain way in the laboratory, but they don't like to be constrained by evidence; when they go home, they take off the lab coat and relax with some comfortable nonsense. And yes, that does make me wonder if I can trust that scientist's opinions even in their own field - especially when it comes to any controversial issue, any open question, anything that isn't already nailed down by massive evidence and social convention.
Maybe we can beat the proverb - be rational in our personal lives, not just our professional lives. We shouldn't let a mere proverb stop us: "A witty saying proves nothing," as Voltaire said. Maybe we can do better, if we study enough probability theory to know why the rules work, and enough experimental psychology to see how they apply in real-world cases - if we can learn to look at the water. An ambition like that lacks the comfortable modesty of being able to confess that, outside your specialty, you're no better than anyone else. But if our theories of rationality don't generalize to everyday life, we're doing something wrong. It's not a different universe inside and outside the laboratory.
Addendum: If you think that (a) science is purely logical and therefore opposed to emotion, or (b) that we shouldn't bother to seek truth in everyday life, see "Why Truth?" For new readers, I also recommend "Twelve Virtues of Rationality."
Loading…
Subscribe to RSS Feed
Comments (142)
But when it comes to the actual meat of the religion, prophets and priests follow the ancient human practice of making everything up as they go along. And they make up one rule for women under twelve, another rule for men over thirteen; one rule for the Sabbath and another rule for weekdays; one rule for science and another rule for sorcery...
???
I thought those rules were the outcome of competition between different factions. The factions with the better rules were more likely to win. For example, a century or two ago, part of the Jewish community decided to try ignoring the requirement to not eat shrimp etc. It looks like that isn't working very well. As far as Jews are concerned, God really does hate shrimp.
Not only competition, but what seemed logical. I'm only 5 years late to this, but I figure I'll add this regardless: Shrimp made people sick, so it only made sense to make rules against eating shrimp, regardless of the reason behind it making people sick. A lot of the old testament is pretty much a survival guide.
That link is however a church, and as far as I can tell does not represent the Jewish faith. From what I know, it's not that shrimp were bad, and hated by God, but that since people got sick, it was not a great idea to eat it. Same logic that founded rules about washing your hands before dinner - they didn't think God hated your hands, they just figured out some correlation between sickness, and filth.
That said, it's not all good, but it seems to me that at least SOME rules were based on logic. And that whoever had the worse rules DID die more frequently.
Eh, I see this as a purely selective / survivorship-bias process:
All the little minority groups that didn't have weird rules got assimilated into the mainstream culture and lost their identity as little minority groups. They became Persians or Greeks or Romans or Christians or Muslims, when those empires were in ascendancy. Therefore, all the little minority groups that have remained distinct for thousands of years have weird rules.
It's not that the weird rules were good for individuals' survival. Pretty often, you're better off individually if you join the mainstream. But weird rules are good for maintaining group identity.
Interesting. I've not thought of it like that, but it would make sense - groups would drop their weird rules if they didn't fit the larger group which they were integrated into.
However, in this case at least, it IS so that the weird rules increased survival. Rules about keeping clean were seen as weird, but were generally beneficial for the individual. Example linked to the discussion: During the Black Plague fewer Jews got infected, mainly due to the weird rules. Only negative was that this was suspicious, and these Jews were believed to be the cause... A bit of a lose-lose situation, with good intentions.
What I hear there is that in one particular circumstance, the weird rules may have increased survival from disease, but decreased survival from persecution. Net result probably nil for the individual. But persecution also maintains the minority group's distinct status.
True. In this case, it most likely did harm in the long run, but the intentions behind were good, and logical. It's not always rational to generalize, but you make a good argument. Though I'm not sure - for the most part, weird rules in religion seem to be based on public opinion as much as group identity or logic. In short: Can be good or bad depending on circumstances, no matter what it is based on.
But it's late and I'm beginning to fear for my mind. I'll stop before I embarrass myself too much.
Joseph, how did they get these "competing rules" in the first place? By making them up as they went along. So, in accordance with human psychology, they make up lots of different rules for different occasions that "feel different". Both sides (or all sides) of any religious battle do this, and it doesn't matter who wins, they still won't come up with a unified answer.
Shouldn't that lead to at least some (if very poor) "testing" of rules over time? Some (such as taboos which strengthen social cohesion or which inadvertently help avoid dangerous behavior) would help the ground adapt, whilst others (which do neither) would be unlikely to continue.
Hmmm...
Q) Why do I believe that special relativity is true? A) Because scientists have told me their standards of evidence, and that the evidence for special relativity meets those standards.
I haven't seen anything contract when moving close to the speed of light. I haven't measured the speed of light in a vacuum and found that it is independent of the non-accelerating motion of the observer. I haven't measured a change in mass during nuclear reactions. I simply hear what people tell me, and decide to believe it.
George Orwell put it far more elegantly, and you can read what he wrote at http://www.newenglishreview.org/blog_direct_link.cfm?blog_id=4274
I can try to apply filters to determine who I can regard as a legitimate authority on various topics. Anyone whose arguments are logically inconsistent is obviously right out. I can check credentials. I can ask people why they accept a claim, and if I disapprove of their standard of evidence, I can give their claims less credence. I can see if the topic is controversial among those whose standards of evidence I respect, and if it is, I can refrain from judgment on the grounds that if there were strong evidence either way, there would be no controversy.
Many things tend to be such that we have to act without anywhere near the amount of evidence that even the social sciences demand. How should I invest my money? What will make me more attractive to potential mates? Who should I vote for? Is (insert enemy here) really a dire threat that my country needs to fight and defeat? What career should I pursue? Which person should I hire? It's really hard to design and perform experiments to answer questions like this. Heck, we still don't even know what kind of food is best to eat!
Seems to me that a lack of patience is part of the problem. Some people would like to be able to really understand why special relativity is true and go through the argument and experiments but they'd have to invest quite some time doing so, before they'd find out for themselves. So too various other things people would like to know, but believe they haven't got the time to deeply examine. Couple that with a compressed curriculum in education where students now need to know more than ever before and know it in less time. Couple that with our society that puts information into increasingly small packets, that spends vast amounts of advertising dollars on convincing people in the shortest optimum time to buy some item, and it's revealed that people are time poor when it comes to deeply understanding and investigating what it is they want to know.
Now with regards to "we still don't even know what kind of food is best to eat!" That is a question that we do know! But you probably won't find it in advertising material, you probably won't find it one particular book, and you most certainly won't find it in one particular eatery/restaurant. You will find the answer from a professional dietician/nutritionist (whatever your country calls them) that's spent about 3 years studying to find out the answer in all its complexity. Shall we trust that professional, shall we have faith in that professional? Or do we want to find out the answer for ourselves... whilst we struggle with paying the mortgage, getting the kids to school and meeting our work commitments?
When we dismiss "faith" and "trust", and I don't mean in a deity, I mean when we dismiss faith and trust in other humans, we are left in a very precarious position of having to work it all out for ourselves.
Are you asserting that there is no controversy among credentialed nutritionists about what kind of food is best to eat?
I assert that suitably diligent nutritionists do make a series of measurements of a particular individual and then offer accurate advice on what are the best kinds of food to eat in that circumstance, that they will retest those measurements and refine their advice as appropriate.
It's my opinion that much of the "controversy" with regards to what are the best kinds of food to eat is based in the fact that many people, including some of those who hold a certification in nutrition/diet make no measurements before they make a judgment. My opinion is that it's the generalist answer that is controversial, not the specific.
Yes, academics largely train people to follow various standard procedures as social conventions, instead of getting people to really understand the reasons for those conventions. Apparently it is very hard to teach and test regarding the underlying reasons. That is the fact that really gives me pause.
If you'd like to move past your pause, then study an education degree at a reputable school. "Deep Learning" is something that is very much a part of the subject matter in the units I'm studying at Charles Darwin Universities Bachelor of Teaching and Learning Preservice.
What's the -1 for please?
Probably because you suggested that someone who has their own stuff to do go get a college degree to resolve one problem they think they see in education. Also, your second sentence sounds a bit like an advertisement for Charles Darwin Universities Bachelor of Teaching and Learning Preservice.
To do better, just say "I'm studying education, and they're spending a lot of time on this stuff called 'deep learning,'" and then maybe you could spend a few sentences actually talking about what you're studying and explaining it to Robin, rather than expecting him to sign up for classes to find out.
Thanks for your response Manfred.
So it's a "9 point" positive to say something that reiterates a commonly perceived problem and offers no solution and also makes a factual error, but to direct someone to a place that actually addresses their problem is a -1. Cool, I'm getting a feel for the website now, cheers.
Let's try this. It's actually pretty easy to test for deeper learning. For example multiple choice questions have previously been considered as examples of shallow learning, or if you prefer shallow testing, and in the past that was accurate and indeed in some still existing multiple choice questions there isn't a path towards deep learning. However consider this question.
The 11 letters in the word PROBABILITY are written on 11 pieces of paper, and a piece of paper chosen at random from a bag. Which of the following statements are true? a)The probability of selecting a "B" is less than the probability of selecting an "I". b)There is a greater chance of obtaining a consonant than of obtaining a vowel. c)A vowel is less likely than a consonant. d)If you repeated the experiment a very large number of times, approximately 63% of the results would be consonants. Make your selection, note that you may select more than zero answers.
Now since "advertising" for CDU might be deemed as somewhat negative, am I permitted to share the details of a book one could read to understand how deep learning for mathematics can be taught to middle school students? Or would that be advertising also?
Elementary & Middle School Mathematics : Teaching Developmentally by John A. Van De Walle, Karen S. Karp and Jennifer M. Bay-Williams, published by Pearson International.
If you're Robin Hanson, sure. Also note that the comment is from '07: the voting system wasn't put in place until after the move from Overcoming Bias (Hanson's blog) to LessWrong. If a similar comment was made today it would probably be voted up much higher. Or maybe downvoted, who knows.
Here, I gave you an upvote. Now it's a 0. Karma means too little to stress out over like this.
Good day to you too.
Telling someone to read a book is still probably expecting too much of them. An online article is pretty much the limit, and it's even better if you can tell the other person what you want them to learn in your own words. Referencing a book is good, though.
Cheers Manfred, Hopefully I've given a glimpse of how one can apply and test for deep learning... and if I a "lowly" first year undergrad in education can do it, seems it's probably not that hard. [smirk]
I'm getting a few mixed signals from the site, and that's to be expected. Some want high detail, others want little. Some will downgrade me for restating something that's been said, others don't seem to be downgraded for saying something that's common knowledge. I'll get the hang of it. I'm well aware that joining an internet site/forum/community involves some amount of fitting in, some amount of taking hits cause you're the new guy or just aren't getting it, and some amount of adjusting to the norms, indeed some good old fashion relationship building too.
I was hoping this might be a place where solutions are discussed more than problems are complained about. Perhaps that's true and I just haven't discovered it yet, don't know.
I remain open minded and am working my way through the sequences, they're good reading.
Disconfirmation bias - love that concept and discovered it on lesswrong, hadn't found that one in the Social Psych or Education Psych textbook yet. Even if I get too peeved off and leave to never return, LW will be remembered fondly for that one concept alone.
Could you give an example of "shallow learning" alongside one of "deep learning," and explain the difference? "Deep learning" definitely sounds like something that's better than "shallow learning," but you haven't made if very clear what it actually is.
Desrtopa, sure thing mate. Deep learning example : The 11 letters in the word PROBABILITY are written on 11 pieces of paper, and a piece of paper chosen at random from a bag. Which of the following statements are true? a)The probability of selecting a "B" is less than the probability of selecting an "I". b)There is a greater chance of obtaining a consonant than of obtaining a vowel. c)A vowel is less likely than a consonant. d)If you repeated the experiment a very large number of times, approximately 63% of the results would be consonants. Make your selection, note that you may select more than zero answers.
Shallow learning example. The 11 letters of FOUNDATIONS are written on 11 pieces of paper, and a piece of paper chosen at random from a bag. What is the probability that an "O" is selected? a) 3/12. b) 1/11. c) 2/11. d) 2/12. Select only 1 answer.
The Deep learning example uses the word “probability” that’s a way to prime the student to thinking in terms of probability, it is an interconnection, it enhances learning, the word "foundations" doesn't do this. The question is “which of the following statements are true?” – that’s a question that is more open than “What is the probability that an “O” is selected?” - open questions evoke deep learning better than closed questions. The answer selections in the deep learning are worded, they require interpretation, they need an understanding of consonants and vowels – which again provides an interconnection with English, and interconnections are deep learning. Whilst the shallow learning has 4 numbers for options, they require no interpretation and they don’t interconnect with English as much as does the deep learning example. The Deep learning questions “make you selection, note that you may select more than zero answers” gives the reader a pause… how many can I select, what does more than zero mean, it requires some interpretation, could be 1, 2, 3, or 4! The “Select only 1 answer” doesn’t need interpretation, it’s closed – just 1. Now about the answers themselves. Shallow learning multiple choice questions typically have 2 options that are readily visible as incorrect and can be quickly discarded, 3/12 can be quickly discarded because there aren’t 3 O’s nor are there 12 letters. 1/11 can be quickly discarded because there are 2 “O”s. 2/11 is the correct answer and so the person doesn’t even need to assess answer d. Where as in the deep learning each answer needs to be assessed for their truth value, and no answer can be quickly discarded.
"Now suppose we discover that a Ph.D. economist buys a lottery ticket every week. We have to ask ourselves: Does this person really understand expected utility, on a gut level?"
Tricky question. It we look purely at the financial return, the odds, then no. If we look at the return in utility, possibly yes.
Is $1 too much to pay for a couple of days of pleasurable dreams about what one would do if one won? Don't we think that such fleeing from reality has some value to the one entering such a fantasy, a suspension of the rules of the real world?
If we don't agree that that has some value then it's going to be terribly difficult to explain why people spend $8 to do to the movies for 90 minutes.
I don't buy lottery tickets.. but I still dream about what I'd do if I won. I realised a while back that i don't actually have to pay to have those dreams.
In sum, I agree, but one small issue I take is when you argue that someone acts contrary to their learning it demonstrates that they don't really understand it. I'm sure this is often the case, but sometimes it's a matter of akrasia: the person knows what they should do and why, even deep down inside, yet finds themselves unable to do it.
Humans suffer heavily from their biases. I recall at in middle school I came to the conclusion that no deities existed, yet it took me a long while to act on it because of social pressures, so I continued to behave contrary to my beliefs out of fear. It was only later in life that I gained the self-confidence and bravery to act upon my beliefs, no matter how contrary to the social norm.
You might say that I didn't really understand and that if I did I would have acted differently, but I find this contrary to my own experience, and this is only one such example. The human brain is a mine field, and even when we understand, we may still fail to act correctly.
Depending on the circumstances and your priorities, pretending to have religious beliefs might have been the most rational thing to do (not knowing either, I don't know if that's true of course).
Tao was described (I wouldn't say invented...) long before Isaac Newton, and yet expresses Feynman's sentiments almost exactly. But then it isn't a religion, either.
Apparently it is very hard to teach and test regarding the underlying reasons.
Does "apparently" (in general) mean you aren't using additional sources of information? In this case, are you concluding that it's difficult simply from the fact that it isn't done? That only seems to me like evidence that it's not worth it. Unfortunately, the value driving the system is getting published, not advancing science.
Douglas, I have found it hard to teach when I have tried, but I'm sure another reason it is rarely done is that academic rewards for it tend to be small relative to the costs.
Tim Worstall, if a PhD economist has pleasurable dreams about winning the lottery, that is exactly what I would call "failing to understand probability on a gut level". Look at the water! A calculated probability of 0.0000001 should diminish the emotional strength of any anticipation, positive or negative, by a factor of ten million. Otherwise you've understood the probability as little symbols on paper but not what it *means* in real life.
Also, a good economist should be aware that winning the lottery often does not make people happy - though one must take into account that they were the sort of people who bought lottery tickets to begin with.
A calculated probability of 0.0000001 should diminish the emotional strength of any anticipation, positive or negative, by a factor of ten million.
I don't play the lottery, but I sometimes have pleasurable daydreams about what I'd do if I were some great success - found the cure for cancer, proved P=NP, won a Nobel prize... objectively speaking, the probability is extremely low, but it doesn't scale my pleasure down by a million times.
A calculated probability of 0.0000001 should diminish the emotional strength of any anticipation, positive or negative, by a factor of ten million.
And there goes Walter Mitty and Calvin, then. If it is justifiable to enjoy art or sport, why is it not justifiable to enjoy gambling for its own sake?
if the results are significant at the 0.05 confidence level. Now this is not just a ritualized tradition. This is not a point of arbitrary etiquette like using the correct fork for salad.
The use of the 0.05 confidence level is itself a point of arbitrary etiquette. The idea that results close to identical, yet one barely meeting the arbitrary 0.05 confidence level and the other not, can be separated into two categories of "significant" and "not significant" is a ritualized tradition indeed perhaps not understood by many scientists. There are important reasons for having an arbitrary point to mark significance, and of having that custom be the same throughout science (and not chosen by the experimenter). But the actual point is arbitrary etiquette.
The commonality of utensils or traffic signals in a culture is important, even though the specific forms that they take are arbitrary. The exact confidence level used is arbitrary; it's important that there is a standard.
Nor is Bayes's Theorem different from one place to another.
No, but the statistical concept of "confidence" depends on how an experimenter thinks that a study was designed. See for example this discussion of the likelihood principle.
If Alice conducts 12 trials with 3 successes and 9 failures, do we reject the null hypothesis p = .5 versus p < .5 at the 0.05 confidence level? It turns out that the answer depends in the classical frequentist sense on whether Alice decided ahead of time to conduct 12 trials or decided to conduct trials until 3 successes were achieved. What if Alice drops dead after recording the results of the trials but not the setup? Then Bob and Chuck, finding the notebook, may disagree about significance. The "significance" depends on the design of the experiment rather than the results alone, according to classical methods.
How many scientists understand that?
The vast majority of scientist, by your standard, don't really understand science. Humans have certain built-in biases and consistently make certain kinds of bad judgements. Even statisticians and mathematicians make common errors of judgement. The fact is that people are often not rational and are driven by emotion, biases, and other non-rational factors.
While it's useful to study and understand these biases and it's healthy to try to avoid commoon errors of judgement, it's not accurate to declare that anyone who acts irrationally is not truly a scientist or doesn't actually understand science. You are merely observing that they are human.
I apologize for responding to this where you are highly unlikely to see ... but you seem to be missing an essential point. It is not necessary to understand science to do science any more than it is necessary to understand control theory to balance on one leg. What is disappointing is that even the population of scientists - who would appear the most likely to understand science - make errors that demonstrate that they do not.
Even so, we rationalists ought not to be deterred from improving our minds by their failure to. That would be an improper use of humility.
Comments like the parent are the reason I'm glad we don't have a norm against responding to ancient comments.
Probability theory still applies.
Ah, but which probability theory? Bayesian or frequentist? Or the ideas of Fisher?
How do you feel about the likelihood principle? The Behrens-Fisher problem, particularly when the variances are unknown and not assumed to be equal? The test of a sharp (or point) null hypothesis?
It does no good to assume that one's statistics and probability theory are not built on axioms themselves. I have rarely met a probabilist or statistician whose answer about whether he or she believes in the likelihood principle or in the logically contradicted significance tests (or in various solutions of the Behrens-Fisher problem) does not depend on some sort of axiom or idea of what simply "seems right." Of course, there are plenty of scientists who use mutually contradictory statistical tests, depending on what they're doing.
John, I consider myself a 'Bayesian wannabe' and my favorite author thereon is E. T. Jaynes. As such, I follow Jaynes in vehemently denying that the posterior probability following an experiment should depend on "whether Alice decided ahead of time to conduct 12 trials or decided to conduct trials until 3 successes were achieved". See Jaynes's _Probability Theory: The Logic of Science_.
The 0.05 significance level is not just "arbitrary", it is demonstrably too high - in some fields the actual majority of "statistically significant" results fail to replicate, but the failures to replicate don't get into the prestigious journals, and are not talked about and remembered.
I'm sorry, that seems just wrong. The statistics work if there's an unbiased process that determines which events you observe. If Alice conducts trails until 3 successes were achieved, that's a biased process that's sure to ensure that the data ends with a least one success.
Surely you accept that if Alice conducts 100 trials and only gives you the successes, you'll get the wrong result no matter the statistical procedure used, so you can't say that biased data collection is irrelevant. You have to either claim that continuing until 3 successes were achieved is an unbiased process, or retreat from the claim that that procedure for collecting the data does not influence the correct interpretation of the results.
The universe doesn't care about Alice's intentions. The trials give information and that information would have been the same even if the trials were run because a rock fell on Alice's keyboard when she wasn't watching.
Yes, he does.
Here is where the mistake starts creeping in. You are setting up "biased data collection" to mean selective reporting. Cherry picking the trials that succeed while discarding trials that do not. But in the case of Alice the evidence is all being considered.
The necessary claim is "continuing until 3 successes are achieved does not produce biased data", which is true.
This is a question that is empirically testable. Run a simulation of agents that try to guess, say, which of a set of weighted dice are in use. Pit your 'care what Alice thinks' agents against the bayesian agent. Let them bet among themselves. See which one ends up with all the money.
If Alice decides to conduct 12 trials, then the sampling distribution of the data is the binomial distribution. If Alice decides to sample until 3 successes are achieved, then the sampling distribution of the data is the negative binomial distribution. These two distributions are proportional when considered as functions of the parameter p (i.e., as likelihood functions). So in this specific case, from a Bayesian point of view the sampling mechanism does not influence the conclusions. (This is in contradistinction to inference based on p-values.)
In general, you are correct to say that biased data collection is not irrelevant; this idea is given a complete treatment in Chapter 6 (or 7, I forget which) of Gelman et al.'s Bayesian Data Analyses, 2nd ed.
I thought the exact same thing, and wrote a program to test it. Program is below:
Turns out they actually are equivalent. I tested with all manner of probabilities of success. Obviously, if what you're actually doing is running a set number of trials in one case and running trials until you reach significance or give up in the second case, you will come up with different results. However, if you have a set number of trials and a set success threshold set beforehand, it doesn't matter whether or not you run all the trials, or just run until the success threshold (which actually seems fairly obvious in retrospect). Edit: formatting sucks
Upvoted for actually testing the theory :)
I don't believe this is true. Every individual trial is individual Bayesian evidence, unrelated to the rest of the trials except in the fact that your priors are different. If you run until significance you will have updated to a certain probability, and if you run until you're bored you'll also have updated to a certain probability.
Sure, if you run a different amount of trials, you may end up with a different probability. At worst, if you keep going until you're bored, you may end up with results insignificant for the strict rules of "proof" in Science. But as long as you use Bayesian updating, neither method produces some form of invalid results.
Ding ding ding! That's my hindsight-bias-reminder-heuristic going off. It tells me when I need to check myself for hindsight bias, and goes off on thoughts like "That seems obvious in retrospect" and "I knew that all along." At the risk of doing your thinking for you, I'd say this is a case of hindsight bias: It wasn't obvious beforehand, since otherwise you wouldn't have felt the need to do the test. This means it's not an obvious concept in the first place, and only becomes clear when you consider it more closely, which you did. Then saying that "it's obvious in retrospect" has no value, and actually devalues the time you put in.
Try this:
(From the Comment Formatting Help)
You have to be very careful you're actually asking the same question in both cases. In the case I tested above, I was asking exactly the same question (my intuition said very strongly that I wasn't, but that's because I was thinking of the very similar but subtly different question below). The "fairly obvious in retrospect" refers to that particular phrasing of the problem (I would have immediately understood that the probabilities had to be equal if I had phrased it that way, but since I didn't, that insight was a little harder-earned).
The question I was actually thinking of is as follows.
Scenario A: You run 12 trials, then check whether your odds ratio reaches significance and report your results.
Scenario B: You run trials until either your odds ratio reaches significance or you hit 12 trials, then report your results.
I think scenario A is different from scenario B, and that's the one I was thinking of (it's the "run subjects until you hit significance or run out of funding" model).
A new program confirms my intuition about the question I had been thinking of when I decided to test it. I agree with Eliezer that it shouldn't matter whether the researcher goes to a certain number of trials or a certain number of positive results, but I disagree with the implication that the same dataset always gives you the same information.
The program is here, you can fiddle with the parameters if you want to look at the result yourself.
I did. It didn't indent properly. I tried again, and it still doesn't.
Actually, it's quite interesting what happens if you run trials until you reach significance. Turns out that if you want a fraction p of all trials you do to end up positive, but each trial only ends up positive with probability q<p, then with some positive probability (a function of p and q) you will have to keep going forever.
(This is a well-known result if p=1/2. Then you can think of the trials as a biased random walk on the number line, in which you go left with probability q<1/2 and right otherwise, and you want to return to the place you started. The probability that you'll ever return to the origin is 2q, which is less than 1.)
Ah, but that's not what it means to run until significance -- in my interpretation in any case. A significant result would mean that you run until you have either p < 0.005 that your hypothesis is correct, or p < 0.005 that it's incorrect. Doing the experiment in this way would actually validate it for "proof" in conventional Science.
Since he mentions "running until you're bored", his interpretation may be closer to yours though.
I don't follow this. If we meet a scientist who is a Marxist, or a Democrat, or a libertarian, or a Republican, or whatever, we don't point out that there is no empirical proof that any of those political programs would achieve its desired aims, and that no true scientist would hold political beliefs. We accept that political decisions are made on different (some would say weaker) evidence that scientific decisions. More generally, there are many questions that the methods of science can't be used to answer.
Assuming consistent priorities it should be possible to empirically determine (assuming sufficent ability to experiment and proper procedures) what policies will best achieve those objectives. It should also be possible to predict empirically (to a degree of accuracy superior to intuition) if a policy will achieve its goals.
I consider myself a 'Bayesian wannabe' and my favorite author thereon is E. T. Jaynes.
Ah, well then I agree with you. However, I'm interested in how you reconcile your philosophical belief as a subjectivist when it comes to probability with the remainder of this post. Of course, as a mathematician, arguments based on the idea of rejecting arbitrary axioms are inherently less impressive than to some other scientists. After all, most of us believe in the Axiom of Choice for some reason like that the proofs needing it are too beautiful and must be true; this is despite the Banach-Tarski paradox and knowing that it is logically independent of the other axioms of Zermelo-Fraenkel set theory.
it is demonstrably too high
Hmm. I lean towards agreeing that it may be too high, but at the same time there would be problems introduced from a lower standard as well. In particular, one such silly problem is that from testing many relationships at the same time, and one then inevitably finding that (from random chance) one is "significant," another thing that many scientists are not aware of, particularly when doing demographic studies. I shudder at the idea of ridiculous demographic data dredging and multiple comparisons being even more widespread.
That said, I, being largely a Bayesian, question the entire concept of null hypotheses. If you are truly "vehemently denying that the posterior probability following an experiment should depend on whether Alice decided ahead of time to conduct 12 trials or decided to conduct trials until 3 successes were achieved," then you must logically reject the entire concept of point hypothesis testing, not merely believe that it's arbitrary or too high, and favor something like Bayes factor.
Of course, it's hard for any of us to be completely consistent in our statistical tests or even understand them all or understand all the completely arbitrary axioms that go into our reasoning.
Sorry, ambiguous wording. 0.05 is too weak, and should be replaced with, say, 0.005. It would be a better scientific investment to do fewer studies with twice as many subjects and have nearly all the reported results be replicable. Unfortunately, this change has to be standardized within a field, because otherwise you're deliberately handicapping yourself in an arms race. This probably deserves its own post.
In my head, I always translate so-called "statistically significant" results into (an often poorly-computed approximation to) a likelihood ratio of 0.05 over the null hypothesis. I believe that experiments should report likelihood ratios.
I am an infinite set atheist - have you ever actually *seen* an infinite set?
I am a "subjective/objective" Bayesian. If we are ignorant about a phenomenon, this is a fact about our state of mind, not a fact about the phenomenon. Probabilities are in the mind, not in the environment. Nonetheless I follow a correspondence, rather than a coherentist, theory of truth: we are trying to concentrate as much subjective probability mass as possible into (the mental representation that corresponds to) the real state of affairs. See my "The Simple Truth" and "A Technical Explanation of Technical Explanation".
I'd rather prefer two studies with 0.05% on the same claim by different scientifists to one study with 0.005%. Proving replicable of scientific studies with actually replicating them is better than going for a even lower p value.
I wouldn't. Two studies opens the door to publication bias concerns and muddles the 'replication': rarely do people do a straight replication.
From Nickerson in http://lesswrong.com/lw/g13/against_nhst/
Agreed. It's much easier for a false effect to garner two 'statistically significant' studies with p < .05 than to gain one statistically significant study with p < .005 (though you really want p < .0001).
If you put the general significance standard at P<0.005 you will even further decrease the amount of straight replications. We need more straight replication instead of less.
A single study can wrong due to systematic bias. One researcher could engage in fraud and therefore get a P<0.005 result. He could also simply be bad at blinding his subjects properly. There are many possible ways to get a P<0.005 result by messing up the underlying science in a way that you can't see by reading a paper.
Having a second researcher reproduce the effects is vital to know that the first result is not due to some error in the experiment setup of the first study.
"And there goes Walter Mitty and Calvin, then. If it is justifiable to enjoy art or sport, why is it not justifiable to enjoy gambling for its own sake?"
You don't have to believe (at any level) that there's a higher chance of you winning than there actually is to enjoy gambling. You just have to consider that the "thrill" payoff inherent in the uncertainty itself is high enough to justify the money that will be statistically spent. I think exactly the same argument could be made about sport.
OT: Eliezer, what do you think about null hypotheses? E.g. what's the correct null hypothesis regarding the probability distribution of the size of (cubic) boxes produced by the box factory, where a flat distribution over the length would produce a polynomial distribution over surface area and volume, for instance?
pdf, that gets into the issue of ignorance priors which is a whole bagful o' worms in its own right. I tend to believe that we should choose more fundamental and earlier elements of a causal model. The factory was probably built by someone who had in mind a box of a particular volume, and so that, in the real world, is probably the earliest element of the causal model we should be ignorant about. If the factory poofed into existence as a random collection of machinery that happened to manufacture cubic boxes, it would be appropriate to be ignorant about the side length.
Sorry, ambiguous wording. 0.05 is too weak, and should be replaced with, say, 0.005. It would be a better scientific investment to do fewer studies with twice as many subjects and have nearly all the reported results be replicable. Unfortunately, this change has to be standardized within a field, because otherwise you're deliberately handicapping yourself in an arms race.
Ah, yes, I see. I understand and lean instinctively towards agreeing. Certainly I agree about the standardization problem. I think it's rather difficult to determine what is the best number, though. 0.005 is as equally pulled out of a hat as Fisher's 0.05.
From your "A Technical Explanation of Technical Explanation":
Similarly, I wonder how many betters on horse races realize that you don't win by betting on the horse you think will win the race, but by betting on horses whose payoffs exceed what you think are the odds. But then, statistical thinkers that sophisticated would probably not bet on horse races.
Now I know that you aren't familiar with gambling. The latter is precisely what the professional gamblers do, and some of them do bet on horse races, or sports. Professional gamblers, unlike the amateurs, are sophisticated statistical thinkers. (And horse races are acceptable for sophisticated gamblers because there's only the small vigorish involved, *and* there's plenty of area for specialized knowledge.)
I think you've made a common statistical fallacy. Perhaps "someone who bets on horse races is probably not a sophisticated statistical thinker." But it does not necessarily follow that "someone who is a sophisticated statistical thinker probably does not bet on horse races." Bayes's Theorem, my man. :)
I know plenty of math Ph.D.s and grad students who do gamble online and look for arbitrage in a variety on ways. Whether they're representative I don't know.
To add to the comment about gambling-- professional gamblers are well aware of the term Dutch book, if not necessarily with arbitrage (though arbitrage is becoming more commonly used).
Heh. Fair enough, John, I suppose that someone has to arbitrage the books. I'll add it to Jane Galt's observation regarding the genuine usefulness of salad forks.
I agree that 0.005 is equally pulled out of a hat. But I also agree on your earlier observation regarding there being some necessity for standardization here.
Personally, I would prefer to standardize "small", "medium", and "large" effect sizes, then report likelihood ratios over the point null hypothesis. A very strong advantage of this approach is that it lets someone do a large study and report a startling likelihood advantage of 1000 for "no effect" over "small effect", rather than just the boring old phrase "not statistically significant". This is probably worth its own post, but I may not get around to writing it.
The point is not that scientists should be perfect in all spheres of human endeavor. But neither should anyone who really understands science, deliberately start believing things without evidence. It's not a moral question, merely a gross and indefensible error of cognition. It's the equivalent of being trained to say that 2 + 2 = 4 on math tests, but when it comes time to add up a pile of candy bars you decide that 2 + 2 ought to equal 5 because you want 5 candy bars. You may do well on math tests, when you apply the rules that have been trained into you, but you don't understand numbers. Similarly, if you deliberately believe without evidence, you don't understand cognition or probability theory. You may understand quarks, or cells, but not science.
Newton may have been a hotshot physicist by the standards of the 17th century, but he wasn't a hotshot rationalist by the standards of this one. (Laplace, on the other hand, was explicitly a probability theorist as well as a physicist, and he was an outstanding rationalist by the standards of that era.)
What makes you think it's a deliberate act to start believing things without evidence?
What if it's somewhere along a spectrum of time required to make a rational decision. On x-axis on the far left we've got no time, on the far right we've got all the time of our lives. On the y-axis we've got the effectiveness of decision making, the higher it is the better the performance. Looks like the Yerkes-Dodson inverted "U" relationship.
If we spend very little time on making the decision, then it's likely an ineffective decision. If we spend heaps of time making the decision, then it's possible the decision is over analysed and could well be a less effective decision than one where we've spent some optimum amount of time making the decision.
How much time could we spend on deciding to eat an apple? We could just grab it off the shelf and eat it - that might be ok, or it could result in us taking a bite of an rotten apple.
We could examine the apple for rottenness, we could examine the shop for their overall health standards, we could trace the journey of the apple back through the transport system, all the way back onto the tree, we could do a soil and pest analysis of the environment the apple grew in - this is probably over analysis.
Instead we could have an optimum decision with only 30 seconds of observing the apple, squeezed it and it didn't squish, looked over it's surface and there are no obvious holes or yucky markings.
The scientist does increase their time spent on making a decision within their field, they believe that their optimum amount of decision making process is moved to the right in the aforementioned graph, because that's their field, their job and reputation. When they turn off their "work" processes they will move back to the left on the graph. Are they now being irrational, or have they simply acknowledged that their optimum decision making no longer needs to be so strict.
How much evidence is required to decide that the apple is safe?
What standard is reasonable for deciding to believe in something, and is context relevant to that standard?
Yes, there have been many great scientists who believed in utter crap - though fewer of them and weaker belief, as you move toward modern times.
And there have also been many great jugglers who didn't understand gravity, differential equations, or how their cerebellar cortex learned realtime motor skills. The vast majority of historical geniuses had no idea how their own brains worked, however brainy they may have been.
You can make an amazing discovery, and go down in the historical list of great scientists, without ever understanding what makes Science work. Though you couldn't build a scientist, just like you couldn't build a juggler without knowing all that stuff about gravity and differential equations and error correction in realtime motor skills.
I still wouldn't trust the one's opinion about a controversial issue in which they had an emotional stake. I couldn't rely on them to know the difference between evidence versus a wish to believe. If they can compartmentalize their brains for a spirit world, maybe they compartmentalize their brains for scientific controversies too - who knows? If they gave into temptation once, why not again? I'll find someone else to ask for their summary of the issues.
If 'spirituality' is a catch-all dumpster for 'a common explanation toward things I cannot show empirically at this very moment,' then the criticism is correct: this is a useless discarding of knowledge.
On the other hand, if you don't start with an assumption of first principles, you have no science at your disposal. Certainly, new knowledge can build upon established knowledge; but what did that knowledge build upon? Sooner or later, we must come up with a blank. No question can be asked, and tested, unless it starts with something to build upon. The scientist who says "I believe God did it; so how is it put together, and what can cause its condition to change?", is hardly incapable of making honest and useful observations about what s/he finds, nor -- at that broad of a level -- is s/he particularly different from the scientist who says "I have no God; so I'll investigate it, and ascribe its origin and operation to a chain of mechanisms that I was not around to observe, and which must ultimately go backwards to infinity."
It is also worth reiterating, as others have noted, that no man on earth has empirically proven everything he believes to be accurate (unless, perhaps, he is of an extraordinarily shallow and uncurious intellect). Time is on no-one's side; sooner or later, you must take someone else's word for it. The moment you are clutching your chest in agony is not the time to go enroll in a medical school's cardiology program, for example. You have a good-faith reason to trust in the surgeon's knowledge. Nor is the surgeon obligated to explain all of his knowledge and proofs to you at that moment, and you won't get past the mortuary by demanding otherwise. You can accept or decline the offered help on the basis of what you do know about your predicament; the wise man simply accepts the help and sets aside all other questions until a better time.
Which could be extended into a very interesting discussion of 'religious' faith, or more correctly who or whom you place your faith in (Jesus Christ? Buhdda? Mohammad? Someone else?), and what that person's credentials are to require it of you. Even better, it would be a discussion that does not plead to the rationality of being willfully ignorant of things that one cannot personally, empirically prove right now. No human really lives that way, nor can anyone do so. Death will come first.
have you ever actually *seen* an infinite set?
Wait, are you an finitist or an intuitionist when it comes to the philosophy of mathematics? I don't think I've ever met one before in person?
Clearly you have to deal with infinite sets in order to apply Bayesian probability theory. So do you deal with mathematics as some sort of dualism where infinite sets are allowed so long as you aren't referring to the real world, or do you use them as a sort of accounting fiction but always assume that you're really dealing with limits of finite things but it makes the math and concepts easier?
Do you believe in the Axiom of Choice? Would the Banach-Tarski paradox make you less likely to?
Does the two envelopes problem make you less likely to believe the Bayesian theory of probability?
Can you justify your acceptance of the Bayesian theory of probability or the other mathematical axioms to which you hold through pure evidence?
Does it bother you that (as shown by Godel) no theory which contains elementary arithmetic (addition and multiplication of the natural numbers) can be both consistent and complete, and that no theory that contains elementary arithmetic and the concepts of formal provability can include a statement about its own consistency without being inconsistent? Does this evidence cause you to reject elementary arithmetic, based on the importance of consistency, rational logic, and the need for all true statements to be proved?
Are you also an empiricist in mathematics, akin to Quine and Putnam?
Sorry, posted too soon. I'm a little confused because you said that you rejected coherentist views of truth, but most mathematical empiricists these days use the idea of coherence to justify mathematics. (Mathematics is necessary for these scientific theories; these theories must be taken as a whole; therefore there is reason to accept mathematics, to grossly simplify.)
Q) Why do I believe that special relativity is true? A) Because scientists have told me their standards of evidence, and that the evidence for special relativity meets those standards.
I also know that GPS satellites work with high precision, and that they wouldn't if they didn't correct for relativity.
GPS? You can do better than that! I believe special relativity because it's implied by Maxwell's equations, which I have experienced. Normal human speeds are enough to detect contraction, if you do it by comparing E&M.
Does anyone know how to do this?
Looks like Douglas_Knight2 hasn't been here for a while, so he probably isn't going to say. I don't think the path ahead of me is going to have its colors shift as I run faster, so the simplest approach isn't going to work. This would be a really cool science experiment if it were really possible.
That seems rather bizarre. Was he making some kind of joke? Humans aren't fast, heavy, small or sensitive enough to notice anything that that advanced happening to ourselves.
It takes very low speeds to see macroscopic magnetic effects from electric charges. I'm not sure that that 'implies special relativity', because it's also consistent with the previous theory. But from a relativistic point of view, that's a relativistic effect of much the same kind as time dilation/length contraction.
I have a vague memory from an electrodynamics course more than twenty years ago that the electromagnetic field is a four-vector that transforms the same way that spacetime vectors transform under boosts.
So what in Victorian physics were two separate things became components of one thing, in the same way that space and time merged into spacetime. And Maxwell's four equations in three dimensions + time became two equations in spacetime.
With the old physics, if you had two stationary charged things, they'd attract each other by means of the electric field and there would be no magnetism involved.
But two things moving side by side (i.e. the same situation but you've changed your idea of what it means to stop), attracting each other in exactly the same way, had to be explained by saying things like 'a moving charge generates a magnetic field, and the other charge, moving in a magnetic field, feels a force.
Another way of saying that is that by moving, you can turn electric fields into magnetic fields.
In relativistic physics, there's just the one thing, 'the electromagnetic field', and your motion affects your measurements of the two different components, in much that same way that there's only 'spacetime', and your motion affects your measurements of space and time.
Because the electric and magnetic fields are so strong, this interchange is perceptible with simple instruments at low speeds.
It was all a long time ago. Perhaps a passing physicist can explain better or correct my flailings?
John Thacker:
I consider myself a finitist, but not an ultrafinitist; I believe in the existence of numbers expressed using Conway chained arrow notation. I am also willing to reject finitism iff a physical theory is constructed which requires me to believe in infinite quantities. I tentatively believe in real numbers and differential equations because physics requires (though I also hold out hope that e.g. holographic physics or some other discrete view may enable me to go digital again). However, I don't believe that the real numbers in physics are really made of Dedekind cuts, or any other sort of infinite set. I am willing to relinquish my skepticism if a high-energy supercollider breaks open a real number and we find an infinite number of rational numbers bopping around inside it.
I consider the Axiom of Choice to be a work of literary fiction, like "Lord of the Rings".
Bayesian probability theory works quite well on finite sets. Real-world problems are finite. Why should I need to accept infinity to use Bayes on real-world problems?
The two-envelopes problem shows the necessity of having a finite prior.
Godel's Completeness theorem shows that any first-order statement true in all models of a set of first-order axioms is provable from those axioms. Thus, the failure of Peano Arithmetic to prove itself consistent is because there are many "supernatural" models of PA in which PA itself is not consistent; that is, there exist supernatural numbers corresponding to proofs of P&~P. PA shouldn't prove itself consistent because that assertion does not in fact follow from the axioms of PA. (This view was suggested to me by Steve Omohundro.) Now, I don't believe in these supernatural numbers, but PA hasn't been given enough information to rule them out, and so it is behaving properly in refusing to assert its own consistency.
I have no desperate psychological need for absolute certainty or proof, which, even if PA proved itself sound, I couldn't have in any case, because I would have to believe in PA's soundness before I trusted its proof of soundness. Or maybe I'm in the grips of a Cartesian demon playing with my mathematical abilities.
Correspondence, not coherence, very easily justifies mathematics. Math can make successful predictions, ergo, it's probably true. No one has ever seen an infinite set, ergo, they probably don't exist, and at any rate I have no reason to believe in them.
Shouldn't you add probability theory to the list [physics, differential equations]? Only because probabilities are usually taken to be real numbers. I'm curious what you think of real numbers... how would you construct them? I guess it must be some way that looks a limit of finite processes operating on finite sets, right?
Many schools of math philosophy do without infinity:
http://en.wikipedia.org/wiki/Finitism
http://en.wikipedia.org/wiki/Ultrafinitism
http://en.wikipedia.org/wiki/Mathematical_constructivism
http://en.wikipedia.org/wiki/Intuitionism
So if someone (A) pubishes a proof of theorem T in a maths journal, it isnt actually true until someone else shows that it corresponds to reality in a lab, and publishes that in a science journal?
Or maybe (B) all we need is for some theorems of it to work, in which case we can batrack and suppose the axioms are correct, and then foreward-track to all the theorems derivable from those axioms, which is a much larger set than those known to corresopond to reality?
I havent seen e, i, pi or 23 either.
Eliezer wrote: "Godel's Completeness theorem shows that any first-order statement true in all models of a set of first-order axioms is provable from those axioms. Thus, the failure of Peano Arithmetic to prove itself consistent is because there are many "supernatural" models of PA in which PA itself is not consistent; that is, there exist supernatural numbers corresponding to proofs of P&~P."
This is getting far from the topic but... I really don't see how Completeness entails anything about PA's failure to prove itself consistent (much less how it suggests an explanation in terms of "supernatural models", whatever that is supposed to mean). PA is not expressible as a first-order statement, so Completeness has nothing to say about PA or its limitations.
"PA is not expressible as a first-order statement." Countable sets of first-order statements still count. But, yes, this is getting rather far off-topic.
http://en.wikipedia.org/wiki/G%C3%B6del%27s_completeness_theorem
"Those not willing to examine this evidence are following in the footsteps of Cardinal Bellarmine with his refusal to look through Galileo's telescope. And the refusal is for the same essential reasons: sociology and arrogance."
From the Crackpot Index: "40 points for comparing yourself to Galileo, suggesting that a modern-day Inquisition is hard at work on your case, and so on."
Anyway, based on what I've read, Sheldrake's experiments do return statistically significant results, but there tend to be problems with the experiments themselves that suggest the results are not caused by anything currently unknown to physics. For more details, just check out www.randi.org and search for Sheldrake's name.
Alot of these comments are getting pretty far off topic. Why not create new posts on these interesting topics?
From the Crackpot Index: "40 points for comparing yourself to Galileo, suggesting that a modern-day Inquisition is hard at work on your case, and so on."
Are you seeking truth, or seeking to confirm your current beliefs? Do you deny that the mainstream of the scientific establishment has sociological parameters and taboos, and that these are extremely hostile to the possibility of telepathy and related topics? In that case, you might find this essay by the editor of the Journal of Consciousness Studies, of interest (JCS is a mainstream journal a mainstream journal that publishes material from scholars like Daniel Dennett, in addition to a recent issue discussing some of Sheldrake's research):
http://www.imprint.co.uk/Editorial12_6.pdf
Rather than reading apologia from self-proclaimed guardians like Randi (who is not a scientist, but rather a successful entertainer and propagandist for an official version of truth, the very idea of which is anathema to science), why not read Sheldrake's papers for yourself and come up with your own criticisms? Sheldrake discusses some of Randi's attacks, which turn out to be totally off base or even fabricated. For example:
http://www.sheldrake.org/controversies/randi.html
I can only suggest that you read Sheldrake's published papers with an open mind, not prejudging that their results are "impossible" without reading his research methods and results. The papers are quite accessible, generally clearly written and straightforward, with a general lack of the masses of domain-specific jargon that mars so much journal writing.
The fact that Randi and Shermer, leave alone investigate Sheldrake's claim, did not even give it a primary reading, reminded me too much of Prof's Verres's pseudo rationality in HPMOR . In particular, they don't seem to follow this little dictum
I know for a fact that some scientists, even some world-renowned scientists, are morons outside of their own field. I used to manage construction at a Big 10 University, and had many conversations like this one:
BRILLIANT SCIENTIST, looking over my estimate for a remodelling project on his floor: "What the heck is this, $4000 for a door? A door? I just replaced the front door of my house for $500!"
ME: "Sir, your house is made of wood, and the doors don't have to meet any particular fire code. This building is concrete and steel, and the doors have to be 90-minute fire-rated. This means, among other things, that the door slab has to be hollow metal, which means it is heavy, which means that the frame, hinges, latch, and handle all have to be much sturdier than the hardware on wood doors. Also, the carpenter who will install this door is probably getting paid more than carpenters who work residential, and he's going to have to spend more time on it because it is more complicated. Finally, the lock core has to match all the rest of the cores in this building, so as not to mess up the keying system."
BS: "Don't give me that! This is ridiculous!"
I wish I had a dime for every time this happened . . . .
Do you have any idea of whether the first flash of stubborn anger (probably status driven) ever gets undercut by later reflection?
I haven't seen anyone complain about the doors, but the chairs.... oh my, the chairs. They have to be certified as both totally fireproof/acid proof/base proof and highly ergonomic. Unlike the above case, you can see why, and if it didn't cost 100 times as much, I'd agree. But it does. One certified blessed fireproof / corrosionproof / rustproof / knidproof +5 ergonomics chair costs multiple thousands of dollars. A nice chair satisfying all of the first set of criteria costs multiple tens of dollars. It just won't be +5 ergonomic.
Okay… but…
We don't sit in one place for 20 minutes, let alone 8 hours! Give us reasonably comfortable metal chairs and stools! Sigh.
A very, very belated note: there's an interesting scene in the first season of The Wire where one protagonists is running his kid brother through some math homework - some problem with people boarding and getting off buses. The kid brother is completely lost. Then the protagonist restates the problem in terms of the drugs being sold outside, and the kid brother gets the answer perfectly.
Hmm. Personally, as a Christian and a student of science (doing a Bachelor of Aviation Technology), I have to say that my thought processes were entirely different from what you described in your article.
I went with Pascal's Wager, or at least a modified version of it. Any sort of existence is infinitely better than not existing at all; this eliminates atheism, Buddhism, and Hinduism from consideration, along with other reincarnation-oriented religions. Judaism is almost impossible to convert into, so it's out of the running. Of the religions that remain, most of the pagan ones have relatively mediocre afterlives compared to the heavens of Christianity and Islam, and similarly mediocre punishments if I'm wrong as long as I live virtuously. If I do follow a pagan religion, and Christianity or Islam is correct, I'll suffer eternal hellfires. Therefore, I will be either a Christian or a Muslim. Since Christianity doesn't require me to attempt to overthrow Western civilization, has generally easier requirements to attain Heaven, and will probably allow me to avoid Hell if Islam is correct, I chose to be a Christian.
Of course, simply believing something to be true does not neccessarilly make it true, so I plan to put off testing that belief as long as humanly possible. Or, more accurately, as posthumanly possible, considering I plan to become a posthuman robot god and live forever.
Hmm? If Atheism is correct, I cease to exist after I die no matter what I believe in. If it isn't, I'll either wind up burning in Hell, going to a relatively mediocre afterlife, or ceasing to exist, depending on which religion is correct.
What incentive could I possibly have to decide to be an atheist? It seems to be more likely to be true judging by most present science, but that doesn't automatically make it the most rational decision to make. The best-case scenario is that I'm wrong and I wind up as a minor functionary in the Celestial Bureaucracy or something.
Care to tell? If it's "Pascal's Wager is insufficiently broad" I believe that I have stated I examined a more generalized version of Pascal's Wager before deciding.
I don't believe that I'm an atheist; basically, what I was saying was that it's impossible to know anything to be true with 100 percent certainty; science can only disprove things. The physical evidence indicates atheism is probably true, but the optimal decision for what belief to choose might well to be to ignore that and believe something else.
Can you describe what you mean by "choosing to believe" in something? Right now it's raining where I am, and I don't seem to be able to choose to believe otherwise. I have the same difficulty in choosing to believe things I don't know the truth of, like whether it will stop raining by the time I go home.
On the other hand, I know someone who became interested in paganism, tested it by believing in it, and found it worked, so continued to believe. I would have been fascinated to probe him further on the matter, but I didn't think I could manage to not sound like an anthropologist inquiring somewhat condescendingly into the strange superstitions of tribal savages.
how paganism worked for him? pagan rituals were cool and pagan chicks were hot, or something more? :P
Worshipping the Goddess that infuses Nature made a difference in his life, or something like that. As I say, I didn't feel comfortable about pressing him on the subject. It would have been like asking what sex is like.
The last sentence shouldn't be "nobody" but "no Bayesian rationalist."
They need to have inconsistent attitudes about how they calculate probability, or estimate probabilities by inherently irrational means such as assigning likelyhood based on what hypothesis they want to be true the most and acting like that belief is certain. Empirically, I've met individuals who claim that no amount of evidence would alter some of their beliefs so something like this may be going on. It is however possible that trying to model these beliefs as probabilities implies a degree of rationality that they simply lack. The human mind is not generally a good Bayesian.
It may be a matter of language use--- if I assign something a probability of 1, it means that everything I know now points in that direction, but I leave the possibility open that I might come to know more.
I think my underlying premise is "no evidence could ever convince me otherwise" is so ridiculous that it doesn't need to be included in the schema.
Any interpretation where you can fix a broken model. I can imagine a conversation like this...
Prankster: I'm holding a die behind my back. If I roll it, what probability would you assign to a 1 coming up?
cupholder: Is it loaded?
Prankster: No.
cupholder: Are you throwing it in a funny way, like in one of those machines that throws it so it's really likely to come up a 6 or something?
Prankster: No, no funny tricks here. Just rolling it normally.
cupholder: Then you've got a 1/6 probability of rolling a 1.
Prankster: And what about rolling a 2?
cupholder: Well, the same.
Prankster: And so on for all the other numbers, right?
cupholder: Sure.
Prankster: So you assign a probability of 1 to a number between 1 and 6 coming up?
cupholder: Yeah.
Prankster: Surprise! It's 20-sided!
cupholder: Huh. I'd better change my estimate from 1 to 6/20.
The bigger problem is that accepting Pascal wager is just first step on the road to faith. And walking the road means to live and pray as if you had faith.
Somewhat, I do not see this guy doing it...
To avoid being punished by the God of Rationality. Since there's no evidence for gods, It sends all theists to Hell.
And I'd thank him for it, since it's better to spend eternity burning in Religious Hell than ceasing to exist. At least in Religious Hell, I'm still me. ;)
Also, I should probably be going to bed since I live in Australia and it's half-past midnight and I have university tomorrow.
That belief in an afterlife tends to go with belief in a deity doesn't make disbelief in an afterlife a logical consequence of atheism.
Yes, but it seems fair to say that P(Afterlife|A deity exists) > P(Afterlife|~ A deity exists).
why? how do you measure that P of caring personal god who saves human souls from extinction is higher that P of unthinking mechanism ("akashic chronicle", "reincarnation wheel") doing the same?
I don't, but something like a reincarnation wheel or an akashic chronicle is not inconsistent with the existence of a deity so I don't need to.
For real life example: one Russian kook preaches exactly this doctrine - strong atheism combined with strong belief of immortality of souls. Add holocaust denial, moon landing denial and admiration of Stalin as greatest hero that ever lived and you have something that sells dozens of books and gains many dedicated followers. Any more about him would belong to "irrationality quotes" thread if one existed...
Interesting. Never head of this guy. Link?
|Interesting.
as interesting as picking up rocks and observing insects crawling under them, IMHO
|Never head of this guy. Link?
http://en.wikipedia.org/wiki/Yury_Ignatyevich_Mukhin
most of his works are online, in Russian of course, links from Russian wiki page
What, insects are fascinating!
This seems far from exhaustive.
Edit: To clarify, my objection is not that you've ignored certain current religions; my objection is that you've restrained the field to current religions in the first place, as if they were somehow inherently more plausible than the vast unexplored majority of religionspace.
Good for you. Now you only have to renounce all pride, glory and luxury and spend your life praying for the gift of faith. It will eventually come, as Pascal reassures us.
http://www.indepthinfo.com/extended-quotes/necessity-of-the-wager.shtml
(scroll down to note 233 for Pascal's famous wager argument in its full context)
I'm curious if you actually put as much thought into this as you claim to. I'm also curious if you grew up in a largely Christian environment. This entire piece sounds a bit like motivated cognition. In particular, I have to wonder whether your justification for throwing out Judaism as being "almost impossible to convert into" reflects an actual attempt to investigate this matter. Depending on the denomination/movement, the time it takes can vary from a few weeks or months (in some Reform versions) to as long as 2-3 years (in Orthodox forms). It also seems like you didn't do much research because under your framework there are much stronger reasons to reject Judaism. In particular, the vast majority of forms of Judaism don't believe in eternal damnation, and those that do generally severely limit the set of people whom it applies to. You seem to have an associated problem in generalizing about Christianity and Islam in that there are universalist or close to universalist forms of both those religions. Not only that, but even among non-universalists there is a chance for members of other religions to go to heaven. (If one were just looking at the Abrahamic religions for example and trying to minimize one's chance of hell, Judaism might make the most sense since many forms of Christianity and Islam are ok with that). But you seem to have also simply avoided thinking about many religious traditions, such as Mormonism and the Ba'hai.
I'd like to think I put as much thought into it as I think so! :P I don't think I wrote down the answer first and then filled in the proof, but I suppose I can't be totally sure I didn't. I did get raised in a Christian environment, but we were hardly the type who'd go to Church every week.
One of my friends as a teenager had his mother converting into Judaism; apparently people who convert into the religion have to go through the diet strictures and whatnot extra-strictly. That's what I meant by "almost impossible to convert into". My understanding of the Jewish view of the afterlife is that they either go to Heaven or cease existing (Sheol) which is infinitely worse than eternal hellfire, and a decent Christian will still get into Jewish heaven since we'd follow the Noahide laws, so that way I'm covered.
Mormonism was rejected because the guy who founded it was a known con man, and the nature of the Book of Mormon is such that if it is true, you can't not believe in it and go to Heaven, and if it isn't, then you can't believe in it and go to Heaven. Since he was a con man and therefore it probably isn't true, it's probably not a good idea to believe in it.
I'll admit that I don't think I did much looking into Bahai, other than seeing that they were basically a religion that splintered off of Islam. Looking it up on Wikipedia, though, it looks like they believe in reincarnation? Bleh. My mind is who I am; if it gets deleted when I go onto the next world, there's no point.
This remark makes it sound even more like you didn't do much research. The belief that one ceases to exist was historically floating around in some sects but wasn't a prominent viewpoint from about 100 CE to 1800 CE where it again got picked up by the most weak theistic and deistic strains of Judaism (such as some Reform and Conservative types). Most Orthodox for example believe in a heaven and (temporary) hell pretty similar to that of Christianity (although even this is complicated by the lack of any strong doctrinal statements. There's a lot more fracturing without anything like the statements of faith or catechisms found in many forms of Christianity). Also, while it is clear that Muslims follow the Noachide laws by most approaches it is actually far from clear that Christians count as such. In particular, the belief in the divinity of a human, Jesus, according to many opinions runs afoul of the prohibition on idolatry. Islam doesn't have this problem when running into the Noachide laws because no claim is made that Muhammad is divine, indeed quite the opposite.
ETA: Also the thing about converts keeping laws extra strictly is only true in some strains also. Note also that this simply amounts in some strains to actually requiring converts to keep the rules (for example in the United States only about half of all Conservative Jews keep kashrut but it is expected that converts keep some form. The Conservative Movement leaders believes that everyone should keep Kashrut but in practice they can't get most of their members to actually do so).
That's not true. Many Mormons believe that non-Mormons can go to heaven. The only caveat is that non-Mormons don't progress as much as Mormons.
Also, keeping kashrut only seems almost impossible if it's something you don't want to do. Obviously, there are a great many people who do it, though the feasibility depends greatly on where you live.
The sort of conversion which seem to be extremely difficult is one which will get you Israeli citizenship.
One other thought: If you are as concerned about continuing to exist as you say you are then you should be much more worried about religions in which believers don't stop existing and non-believers do stop at death. In that case, your options become a bit more limited. I take it you aren't either a Jehovah Witness or a classical Karaite?
This seems like it is somehow connected to the fact that pantheism and solipsism are identical beliefs with different terminology for that which provides sensation.
Not all scientists go around tallying up the expectations payed by their beliefs. If they have a freeloading belief they hadn't examined, one that doesn't affect their science, so what of it?
There's something fundamentally different between a gambling economist and a theist scientist: the thinking required for economics constantly forces you to acknowledge that gambling is dumb (although...). The thinking required for most sciences barely ever runs into the problem of God / spirit world / other wacky nonsense.
Most scientific reasoning treats God as a non-agent while never actually claiming atheism. A scientist who's never evicted a freeloading belief isn't necessarily a bad scientist. Theism is a warning flag only when it causes real-life expectations.
To use slightly different language I would suggest it is always a warning flag but only an actual problem when it causes real-life expectations or field related claims. I say always a warning flag because the kind of brain that can maintain religious belief despite scientific education and experience tends to have traits that I distrust.
Placing a warning flag onto a theist scientist's work would only be justified if you had evidence in support of the claim: P ( good science | scientist is theist ) < P ( good science ) .
Less Wrong provides many excellent philosophical examples in support of that claim. But what about real world examples? Do theist scientists actually tend to do lower-quality science?
Don't confuse a prior with a priori. ;)
Fixed. Thanks. I didn't realize that my statement read, "A priori reasoning can only be justified if it's a posteriori."
Edit: so what about my actual statement? Or, are we done having this discussion?
I've see statistics showing that scientists tend to be less theistic than the general population and that the best scientists (National Academy members, for example) tend to be less theistic than scientists in general. So that provides the correlation you are asking for. But, I strongly suspect that in this case, correlation does not imply causation.
I have seen numerous examples, though, in which scientific enquiry with the choice of subject matter motivated by theism is of lower quality than science done without that motivation. However, the same kinds of bad results can arise from motivation by social activism or personal animosity or simply prideful intransigence.
Absolutely. Hence, the warning flag. A scientist expecting to find the evidence of God doesn't just have freeloading beliefs, but beliefs that pay rent in wrong expectations. That's akin to a gambling economist.
I'd say it's good evidence in favor of P ( good science | scientist is theist ) < P ( good science ) . Of course, your point about correlation not causation is very valid, too.
Someone in the discussion once said that atheism on average adds ~40 to IQ (I might be remembering incorrectly). I suppose high IQ is correlated with both excellence as a scientist and an ability to reconsider and abandon theism if the question ever arose.
My specific interest is whether or not atheism alone makes scientists better.
Buster, that's the kind of brain you have. We're not built well, and not built too differently either. Even if you don't believe in a big dude in the sky who will preserve your identity after your physical form is destroyed, you have a brain that is completely suited to believing that, and your non-belief is a sign of the particular experiences you have had.
The question is whether you believe that the set of experiences required to become a good scientist necessarily include those experiences that force one to adopt atheism. I think the number of important discoveries made by theists throughout history, and even in the modern day, indicates otherwise.
Do not refer to me as buster.
You may note that in the very sentence you quote I refer to experiences, a rather critical part of my claim.
While I am not inclined to go into detail on personality research right now there is, in fact, a relationship between the strength of a person's compartmentalisation ability and other important traits. Genetics plays a critical part in the formation of beliefs from stimulus and there is some information that can be inferred from the expression of said beliefs.
I apologize, but I am also confused. Is this an issue with gender, formality, or something else? I feel like I should be able to generalize you taking issue with that to other things, and also avoid all of those, but it would be helpful for you to explain.
I still feel that, in MoreOn's terms, P ( good science | scientist is theist ) is close enough to P ( good science ) that starting from the position of distrust is probably over-filtering. I don't think that resorting to explaining the personality traits that might explain that relation are important, unless we know an individual's traits well enough to use those to estimate the kind of science she will produce.
There is a positive correlation between an individual thinking well in one area and thinking well in another area, a relationship which I do not consider terribly controversial. A (loosely) related post is the Correct Contrarian Cluster.
Like being able to judge if some knowledge is dangerous and public relations?
Correlations. Not deductive certainties. A correlation that has perhaps been fully accounted for and then some in that case.
And do we really need to bring that up? Really, it's all been said already...
AAaand the graph gives me a coughing fit. Good job.
Oh, trust me, I wouldn't defend this one.
Some profs showed it as an example of a utility function for which gambling would make sense, rationally. I'd say if your utility function looks like this, you have problems far worse than gambling.
That depends on whether you think a propensity to compartmentalize is a good thing or not.
In learning we have two drives, to find truth and to avoid error. These rarely do come in conflict, and even more rarely do they come into any major kind of conflict, but there still are times that we have to decide whether we consider it to be more important to find truths or to avoid errors. Religion, for instance.
Let us imagine that a time traveler arrives from some distant point in the future and teaches me five facts which are not currently known to the world. Four of these facts can be explained with current scientific knowledge, and when they are tested they are proven correct. The fifth is as far beyond our current scientific knowledge as quantum mechanics is beyond the state of science as it was in the first dynastic period of Egypt. We not only are unable to understand the explanation or how to test it, but we are unable to understand the information that would let us understand the explanation, and so on and so forth for a great many regressions.
Nevertheless, because I am more concerned with finding truth than I am with avoiding error, and because the time-traveler's first four facts were true, I would conclude that it is reasonable for me to believe the fifth fact until I am proven to be in error or at least have sufficient reason to doubt it.
This same process applies to religion. The concept that there is some kind of higher being or beings is a concept which causes the universe to make more sense to me in several ways. Some of these ways are ones which I cannot even express well if at all, making it even more difficult to find possible alternatives, but I am not very distressed by this situation. There are concepts which you yourself no doubt believe and which similarly fulfill and answer things which I cannot express to my satisfaction.
Where there would be issue is in knowingly believing an error. While certain interpretations of Deity have been proven wrong either by history or mere logic, I do not believe that theism as a whole has been proven utterly incorrect. Perhaps this is straying into the territory of "Religion cannot be proven incorrect and is an entirely different magisterium," but in the end, if your concept of God is flexible enough, God actually can't be proven to not exist. Some hypotheses can't be tested. It makes them utterly useless for scientific purposes, true, but I've never thought that we can prove that God exists or not, only that we can prove that a particular religious concept is incorrect.
Knowing this, I revise my religious beliefs in accordance with scientific knowledge. I consider evolution to be true, and I am well aware of such things as the craziness of our retinas, and so I do not believe that the Divine created everything in six days, or even that the Divine did anything at all to guide evolutionary processes. I am aware that there are studies which raise interesting questions about consciousness, free will, and the relationship between the two, and I have had to redefine my conception of the soul as a result.
I do not believe that there is any issue with a scientist who has religious beliefs. The problem lies with scientists who have religious beliefs which are incompatible with science and basic reason.
Good day, sir. I look forward to continuing my reading of your site and can hope only that, whether you agree or disagree, I have at least managed to state my case in a way that does not, through its ludicrousness, lend support to the idea that some people shouldn't be allowed to reproduce or even talk to other people.