Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Perpetual Motion Beliefs

31 Post author: Eliezer_Yudkowsky 27 February 2008 08:22PM

Followup toThe Second Law of Thermodynamics, and Engines of Cognition

Yesterday's post concluded:

To form accurate beliefs about something, you really do have to observe it. It's a very physical, very real process: any rational mind does "work" in the thermodynamic sense, not just the sense of mental effort...  So unless you can tell me which specific step in your argument violates the laws of physics by giving you true knowledge of the unseen, don't expect me to believe that a big, elaborate clever argument can do it either.

One of the chief morals of the mathematical analogy between thermodynamics and cognition is that the constraints of probability are inescapable; probability may be a "subjective state of belief", but the laws of probability are harder than steel.

People learn under the traditional school regimen that the teacher tells you certain things, and you must believe them and recite them back; but if a mere student suggests a belief, you do not have to obey it.  They map the domain of belief onto the domain of authority, and think that a certain belief is like an order that must be obeyed, but a probabilistic belief is like a mere suggestion.

They look at a lottery ticket, and say, "But you can't prove I won't win, right?"  Meaning:  "You may have calculated a low probability of winning, but since it is a probability, it's just a suggestion, and I am allowed to believe what I want."

Here's a little experiment:  Smash an egg on the floor.  The rule that says that the egg won't spontaneously reform and leap back into your hand is merely probabilistic.  A suggestion, if you will.  The laws of thermodynamics are probabilistic, so they can't really be laws, the way that "Thou shalt not murder" is a law... right?

So why not just ignore the suggestion?  Then the egg will unscramble itself... right?

It may help to think of it this way - if you still have some lingering intuition that uncertain beliefs are not authoritative:

In reality, there may be a very small chance that the egg spontaneously reforms.  But you cannot expect it to reform.  You must expect it to smash.  Your mandatory belief is that the egg's probability of spontaneous reformation is ~0.  Probabilities are not certainties, but the laws of probability are theorems.

If you doubt this, try dropping an egg on the floor a few decillion times, ignoring the thermodynamic suggestion and expecting it to spontaneously reassemble, and see what happens.  Probabilities may be subjective states of belief, but the laws governing them are stronger by far than steel.

I once knew a fellow who was convinced that his system of wheels and gears would produce reactionless thrust, and he had an Excel spreadsheet that would prove this - which of course he couldn't show us because he was still developing the system.  In classical mechanics, violating Conservation of Momentum is provably impossible.  So any Excel spreadsheet calculated according to the rules of classical mechanics must necessarily show that no reactionless thrust exists - unless your machine is complicated enough that you have made a mistake in the calculations.

And similarly, when half-trained or tenth-trained rationalists abandon their art and try to believe without evidence just this once, they often build vast edifices of justification, confusing themselves just enough to conceal the magical steps.

It can be quite a pain to nail down where the magic occurs - their structure of argument tends to morph and squirm away as you interrogate them.  But there's always some step where a tiny probability turns into a large one - where they try to believe without evidence - where they step into the unknown, thinking, "No one can prove me wrong".

Their foot naturally lands on thin air, for there is far more thin air than ground in the realms of Possibility.  Ah, but there is an (exponentially tiny) amount of ground in Possibility, and you do have an (exponentially tiny) probability of hitting it by luck, so maybe this time, your foot will land in the right place!  It is merely a probability, so it must be merely a suggestion.

The exact state of a glass of boiling-hot water may be unknown to you - indeed, your ignorance of its exact state is what makes the molecules' kinetic energy "heat", rather than work waiting to be extracted like the momentum of a spinning flywheel.  So the water might cool down your hand instead of heating it up, with probability ~0.

Decide to ignore the laws of thermodynamics and stick your hand in anyway, and you'll get burned.

"But you don't know that!"

I don't know it with certainty, but it is mandatory that I expect it to happen.  Probabilities are not logical truths, but the laws of probability are.

"But what if I guess the state of the boiling water, and I happen to guess correctly?"

Your chance of guessing correctly by luck, is even less than the chance of the boiling water cooling your hand by luck.

"But you can't prove I won't guess correctly."

I can (indeed, must) assign extremely low probability to it.

"That's not the same as certainty, though."

Hey, maybe if you add enough wheels and gears to your argument, it'll turn warm water into electricity and ice cubes!  Or, rather, you will no longer see why this couldn't be the case.

"Right! I can't see why couldn't be the case!  So maybe it is!"

Another gear?  That just makes your machine even less efficient.  It wasn't a perpetual motion machine before, and each extra gear you add makes it even less efficient than that.

Each extra detail in your argument necessarily decreases the joint probability.  The probability that you've violated the Second Law of Thermodynamics without knowing exactly how, by guessing the exact state of boiling water without evidence, so that you can stick your finger in without getting burned, is, necessarily, even less than the probability of sticking in your finger into boiling water without getting burned.

I say all this, because people really do construct these huge edifices of argument in the course of believing without evidence.  One must learn to see this as analogous to all the wheels and gears that fellow added onto his reactionless drive, until he finally collected enough complications to make a mistake in his Excel spreadsheet.

Comments (38)

Sort By: Old
Comment author: jonvon 27 February 2008 09:26:38PM 1 point [-]

there are so many beautiful things about this post. i have only a smallish idea what you are talking about a good deal of the time, but i can easily understand this: "To form accurate beliefs about something, you really do have to observe it."

your posts here have helped solidify this for me. this is true, i feel strongly, for all great art, for reliable science, and for joy in living. observation is everything.

this sentence: "One of the chief morals of the mathematical analogy between thermodynamics and cognition is that the constraints of probability are inescapable; probability may be a "subjective state of belief", but the laws of probability are harder than steel."

man. this is one of those sentences i'll be meditating on for a while. god that's a good one. just simply as a piece of writing, that is nice work. i'm going to put it on a t-shirt or something. maybe a bumper sticker, see how many accidents i can cause on the way home...

Comment author: denis_bider 27 February 2008 11:31:07PM 0 points [-]

Reading today's fare is a bit like eating unflavored oatmeal. :-)

It seems to me that the person who can read this and understand it, already knows it.

But the person who does not know it, cannot understand it and will be frustrated by reading it.

I'm not sure what your intention is with the whole series of posts, but if you'd like to enligthen the muggles, the trick is to explain it in a concise, striking, unusual, easily understood, entertaining manner.

Of course, that takes genius. :-)

But otherwise you are writing primarily for people who already know it.

In yet other words: some of your posts, I will forward to my wife. Others, I won't. This one is one of the latter.

Comment author: denis_bider 27 February 2008 11:32:55PM 0 points [-]

Interesting stuff about the preservation of phase space volume, though. I appreciate it, I previously knew nothing about that.

Comment author: The_Epicurean_Dealmaker 28 February 2008 12:04:15AM 0 points [-]

I would be interested in seeing you talk about belief and probability in cases where the deck is not quite so stacked as it is in your thermodynamic examples. The principles should be the same, but I imagine your argument will be less tractable. No?

Comment author: brent 28 February 2008 02:04:03AM 1 point [-]

I thought this post was actually very easy to follow. It actually gave me the vocab I needed to settle scores of arguments with dumbarse religious types who say daft things like "Evolution is only a religion" and "You can't PROVE that blah blah blah magic fairy dust blah blah blah blah".

> "But you don't know that!" > > I don't know it with certainty, but it is mandatory that I expect it to happen. Probabilities are not logical truths, but the laws of probability are.

I love that. "I don't know it with certainty - that's because of your ridiculous understanding of what 'certainty' means - but it is mandatory that I expect it to happen."

Cheers.

Comment author: brent 28 February 2008 02:06:51AM 0 points [-]

hang on - they don't say "evolution is only a religion" they say "Evolution is only a theory".

....dumbarse atheistic types who don't proofread their own rantings....

Comment author: AndyCossyleon 26 August 2010 07:47:36PM 1 point [-]

Well the incarcerated* Kent Hovind did used to say that evolution is a religion. But I never heard him saying it was "just" a religion.

*HAD to include that

Comment author: bigjeff5 01 March 2011 09:10:23PM 8 points [-]

I have to point out that Kent Hovind is in prison for tax evasion, not for being wrong about evolution (though I'm sure he'd like you to think that!).

I've actually met the man, I went to a Christian school as a child and he was a guest speaker at more than one assembly. It's only years later that I realize how ignorant and intentionally blind to the facts he is, and that his arguments rested entirely on straw men and a misunderstanding of the evidence. It slowed my understanding of science by at least five years, maybe more, and I'm more than just a little pissed about that.

It wasn't until I saw japanese mud-skippers in a nature show a few years ago that it really clicked - one of his slams against evolution was "why don't you see fish crawling out of the water?" Well, there you go. 100% fish, living on land - they crawl into mud periodically to breath, pretty cool really. Their front fins are even elongated and act more like fingerless arms than paddles. I wouldn't be surprised if there were a hundred other similar species of fish out there.

When you realize that the theory of evolution has been accurately guiding biologists' expectations for life in bizarre places, and archaeologists' and paleontologists' expectations for artifacts and fossils, for many decades, it becomes clear who was right and who was wrong, just as a matter of practical application. It can be hard to see this when you've been taught to dismiss evolution out of hand, though, and Hovind's intellectually bankrupt arguments give you a fake foundation on which to base such dismissal, so long as you don't question it too much.

Comment author: NancyLebovitz 01 March 2011 09:54:37PM 1 point [-]

Walking catfish-- they actually just sort of slither, but they can locomote.

Comment author: Eliezer_Yudkowsky 28 February 2008 03:18:33AM 6 points [-]

I would be interested in seeing you talk about belief and probability in cases where the deck is not quite so stacked as it is in your thermodynamic examples.

Okay: It's less stacked for lottery tickets than thermodynamics, and it's less stacked for roulette wheels than lottery tickets.

If you stick your fingers in the game anyway, it is mandatory to expect your fingers to get toasted, with not quite as extreme a probability.

The principles should be the same, but I imagine your argument will be less tractable. No?

The principles are exactly the same, and the real force of the argument is exactly unchanged. But to the unenlightened ones the argument seems to have less emotional force, and they do indeed try to ignore the laws of probability, and the expected proportion of them burn their hands.

Comment author: Yelsgib 28 February 2008 05:40:50AM 1 point [-]

What do you think the relation between the mental category of "certainty" and probability is?

For the primitive it is not true that "the sun will rise with 100% certainty" - it is simply "certain that the sun will rise." What's more, I think these statements are -not equivalent-.

For the "educated westerner" it is true that "the sun will rise with certainty very close to 100%, given some assumptions about the nature of the universe in earth's neighborhood." Certainty is not a necessity any longer.

My claim would be that, for most, heuristic descriptions of possibility/probability and an understanding of the mathematical laws or probability are absolutely disjoint. The reason that you can even think about low probability events is not mere knowledge - you must actually switch the context in which you are framing the problem - you must "step back" and examine the lottery in the context of theory in which you (rightly) believe.

What I'm saying here is that arguing against -heuristic descriptions- with -actual probabilities- (even if just approximations) is like arguing against a shaman's perception of the weather with modern supercomputer-driven approximations. You have to consider that people have an investment in their heuristic descriptions - to leave them would be like to leave a nice warm place which makes you happy (most of the time) but might have some nagging problems (e.g. playing the lottery).

Ya dig?

Comment author: Matthew2 28 February 2008 07:02:55AM 0 points [-]

the ep. dealmaker said: "I would be interested in seeing you talk about belief and probability in cases where the deck is not quite so stacked as it is in your thermodynamic examples."

It seems reasonable for a financial analyst to understand that the lottery and coin flipping aren't "stacked".

Hmm, what else could he mean...

Perhaps he means something like the weather.

Comment author: Ben_Jones 28 February 2008 09:47:30AM 4 points [-]

New rule for the lottery: every time you play but don't win, stick your hand in a pan of hot water!

Comment author: Ron_Hardin 28 February 2008 09:55:59AM 0 points [-]

I favor taking energy from earth rotation. Put a horizontal gyroscope across a circular track around the North Pole, and let the earth rotate under it. Take energy from the relative motion.

Comment author: Caledonian2 28 February 2008 01:00:57PM 0 points [-]

I suspect the real problem with the lottery is that people are familiar with the amounts of money necessary to purchase a ticket and recognize that it won't bring them much happiness or satisfaction... but they're not familiar with the amounts of money typically given away in a lottery jackpot, and they imagine that it will make them much, much happier than it is actually likely too.

Even people who know that the lottery has a negative expected financial value buy tickets. They'll accept a resource loss if it leads to greater utility, and people tend to perceive the slim chance of winning lots of money to be sufficiently valuable that they'll take an expected loss to have the chance.

The irrationality comes not from dismissing the statistical loss of money, but from believing that winning offers much greater utility than it will.

Comment author: danlowlite 02 December 2010 10:39:01PM *  0 points [-]

I suspect the people who suspect a real problem with the lottery have never played it.

I don't play regularly, or at all anymore. I can actually count on one hand the number of times I have, but in all those occasions the primary joy from that was not the possibility that I might become more wealthy. It was because it was fun to engage with my peers in a group discussion of "What If."

From what I have witnessed, this seemed to be a popular activity: the discussion of fantasy. This didn't mean that anyone had any illusions about the possibility of winning. I can do that math.

Simply viewing it as a probability game ignores a motivation: it's fun to dream. And it's fun to do so together. "What would you get?" "Who would you give money to?" "Would you quit right away or give two-weeks' notice?" and so on.

Of course, because I only bought lottery tickets with people who bought lottery tickets with me means that my sample is biased towards those who bought them with me. And that I bought lottery tickets.

Edit: Just a note that the "What If" game need not be a social activity. Obviously.

Comment author: taryneast 28 December 2010 02:38:54PM 2 points [-]

The "what if" game can be played even if you don't buy the ticket.

What's more, there's another "what if" game that you're neglecting... that's the "what if I invest this money in something actually achievable here and now?"

This is the game that investors and entrepreneurs play, and if you actually put money into the end-result of that game you have a higher expected payoff than that with the "lottery ticket what if" game

Comment author: Vaniver 28 December 2010 02:47:05PM 5 points [-]

You may find a waste of hope interesting. Like taryneast suggests, everyone plays the "what if?" game- what matters is what you play it about. "What if Brad Pitt leaves Angelina Jolie for me?" is a less profitable question to think about than "What if I talk to the cute guy at the coffee shop?". And since you only think about one or two of those questions at a time, there is a real trade-off involved with planning for the first instead of the second.

Comment author: Tiiba2 28 February 2008 07:18:53PM 1 point [-]

"I favor taking energy from earth rotation. Put a horizontal gyroscope across a circular track around the North Pole, and let the earth rotate under it. Take energy from the relative motion."

The energy to put it there will come out of your allowance.

Comment author: Doug_S. 29 February 2008 04:19:37PM 1 point [-]

A more practical way to take energy from the rotations of the earth-moon system is to exploit the tides...

Comment author: Ron_Hardin 29 February 2008 09:27:39PM 1 point [-]

gyroscope

That was actually a joke. Though people would be hard-pressed to guess what happens if you try it.

Gyroscopes are very unintuitive, because people intuitively but incorrectly think that pushing on something changes its position, a mistake that gyroscopes bring out.

Comment author: NancyLebovitz 25 March 2010 06:06:34PM *  3 points [-]

How do you distinguish between principles as solid as the conservation laws vs. the commonly held belief (recently established as wrong) that adults don't have significant neuroplasticity?

Comment author: Psy-Kosh 25 March 2010 06:17:44PM 5 points [-]

Which ones seem to have stupidly huge amounts of evidence, lower complexity, deeper ties to the rest of our theories/models of reality, etc?

ie, the usual way: downgrade based on complexity (more complex assumptions = lower probability), upgrade based on huge amounts of evidence, etc.

Or do I misunderstand your question?

Comment author: NancyLebovitz 25 March 2010 09:15:43PM *  3 points [-]

No, you understood me correctly.

The problem is a result of confusing consensus with knowledge.

And that's a really easy mistake to make-- it isn't as though there's a handy index to how much evidence there is for commonly held beliefs.

Comment author: Kenny 10 February 2013 04:48:53AM 0 points [-]

I hope there will be handy indexes once we've accumulated enough accurate beliefs, widely.

It doesn't help that our most accurate beliefs (e.g. the standard model of physics) are some of the most difficult to understand, or that beliefs with lots of evidence (e.g. evolution, the age of Earth) are not widely held.

Comment author: David_Gerard 04 January 2011 04:04:08PM -1 points [-]

But there's always some step where a tiny probability turns into a large one

This is gold.

Comment author: jdinkum 13 February 2012 05:58:15PM -1 points [-]

"The rule that says that the egg won't spontaneously reform and leap back into your hand is merely probabilistic."

This example requires a level of education that doesn't match my belief of the expected audience of this post.

The low importance in the distinction between mathematical certainty and realistic likelihood is valid, but involving quantum probability kills the post for me.

Comment author: Nisan 13 February 2012 09:35:19PM 0 points [-]

The example doesn't require quantum physics. Just ordinary classical mechanics.

Comment author: jdinkum 14 February 2012 04:22:40PM -2 points [-]

My point still holds. Most people, myself included, don't have a belief that an egg will spontaneously reform according any laws of physics. To use it as an example of the difference between certainty and likelihood is ineffective.

Comment author: JDM 17 June 2013 01:32:39AM 0 points [-]

If it were something too open to debate, it would take away from the point.

The point is as stated. There is a non-zero probability it will happen, so you shouldn't use "certain", but any reasonable person will act on the belief it isn't going to happen.

If he used religion, which is also extremely unlikely to be correct, it would distract from the point.

Comment author: NancyLebovitz 10 February 2013 05:19:46AM 1 point [-]

This looks like a valuable special case of scope insensitivity.