Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Fundamental Doubts

24 Post author: Eliezer_Yudkowsky 12 July 2008 05:21AM

Followup toThe Genetic Fallacy, Where Recursive Justification Hits Bottom

Yesterday I said that—because humans are not perfect Bayesians—the genetic fallacy is not entirely a fallacy; when new suspicion is cast on one of your fundamental sources, you really should doubt all the branches and leaves of that root, even if they seem to have accumulated new evidence in the meanwhile.

This is one of the most difficult techniques of rationality (on which I will separately post, one of these days).  Descartes, setting out to "doubt, insofar as possible, all things", ended up trying to prove the existence of God—which, if he wasn't a secret atheist trying to avoid getting burned at the stake, is pretty pathetic.  It is hard to doubt an idea to which we are deeply attached; our mind naturally reaches for cached thoughts and rehearsed arguments.

But today's post concerns a different kind of difficulty—the case where the doubt is so deep, of a source so fundamental, that you can't make a true fresh beginning.

Case in point:  Remember when, in the The Matrix, Morpheus told Neo that the machines were harvesting the body heat of humans for energy, and liquefying the dead to feed to babies?  I suppose you thought something like, "Hey!  That violates the second law of thermodynamics."

Well, it does violate the second law of thermodynamics.  But if the Matrix's makers had cared about the flaw once it was pointed out to them, they could have fixed the plot hole in any of the sequels, in fifteen seconds, this easily:

Neo:  "Doesn't harvesting human body heat for energy, violate the laws of thermodynamics?"

Morpheus:  "Where'd you learn about thermodynamics, Neo?"

Neo:  "In school."

Morpheus:  "Where'd you go to school, Neo?"

Neo:  "Oh."

Morpheus:  "The machines tell elegant lies."

Now, mind you, I am not saying that this excuses the original mistake in the script.  When my mind generated this excuse, it came clearly labeled with that warning sign of which I have spoken, "Tada!  Your mind can generate an excuse for anything!"  You do not need to tell me that my plot-hole-patch is a nitwit idea, I am well aware of that...

...but, in point of fact, if you woke up out of a virtual reality pod one day, you would have to suspect all the physics you knew.  Even if you looked down and saw that you had hands, you couldn't rely on there being blood and bone inside them.  Even if you looked up and saw stars, you couldn't rely on their being trillions of miles away.  And even if you found yourself thinking, you couldn't rely on your head containing a brain.

You could still try to doubt, even so.  You could do your best to unwind your thoughts past every lesson in school, every science paper read, every sensory experience, every math proof whose seeming approval by other mathematicians might have been choreographed to conceal a subtle flaw...

But suppose you discovered that you were a computer program and that the Dark Lords of the Matrix were actively tampering with your thoughts.

Well... in that scenario, you're pretty much screwed, I'd have to say.

Descartes vastly underestimated the powers of an infinitely powerful deceiving demon when he supposed he could trust "I think therefore I am."  Maybe that's just what they want you to think.  Maybe they just inserted that conclusion into your mind with a memory of it seeming to have an irrefutable chain of logical support, along with some peer pressure to label it "unquestionable" just like all your friends.

(Personally, I don't trust "I think therefore I am" even in real life, since it contains a term "am" whose meaning I find confusing, and I've learned to spread my confidence intervals very widely in the presence of basic confusion.  As for absolute certainty, don't be silly.)

Every memory of justification could be faked.  Every feeling of support could be artificially induced.  Modus ponens could be a lie.  Your concept of "rational justification"—not just your specific concept, but your notion that any such thing exists at all—could have been manufactured to mislead you.  Your trust in Reason itself could have been inculcated to throw you off the trail.

So you might as well not think about the possibility that you're a brain with choreographed thoughts, because there's nothing you can do about it...

Unless, of course, that's what they want you to think.

Past a certain level of doubt, it's not possible to start over fresh.  There's nothing you can unassume to find some firm rock on which to stand.  You cannot unwind yourself into a perfectly empty and perfectly reliable ghost in the machine.

This level of meta-suspicion should be a rare occasion.  For example, suspecting that all academic science is an organized conspiracy, should not run into anything like these meta-difficulties.  Certainly, someone does not get to plead that unwinding past the Bible is impossible because it is too foundational; atheists walk the Earth without falling into comas.  Remember, when Descartes tried to outwit an infinitely powerful deceiving demon, he first tried to make himself absolutely certain of a highly confusing statement, and then proved the existence of God.  Consider that a caution about what you try to claim is "too basic for a fresh beginning".  And even basic things can still be doubted, it is only that we use our untrustworthy brains to doubt them.

Or consider the case of our existence as evolved brains.  Natural selection isn't trustworthy, and we have specific reason to suspect it.  We know that evolution is stupid.  We know many specific ways in which our human brains fail, taken beyond the savanna.  But you can't clear your mind of evolutionary influences and start over.  It would be like deciding that you don't trust neurons, so you're going to clear your mind of brains.

And evolution certainly gets a chance to influence every single thought that runs through your mind!  It is the very reason why you exist as a thinker, rather than a lump of carbon—and that doesn't mean evolution summoned a ghost-in-the-machine into you; it designed the ghost.  If you learn culture, it is because you were built to learn culture.

But in fact, we don't run into unmanageable meta-trouble in trying to come up with specific patches for specific known evolved biases.  And evolution is stupid, so even though it has set up self-deceptive circuits in us, these circuits are not infinitely difficult to comprehend and outwit.

Or so it seems!  But it really does seem that way, on reflection.

There is no button you can press to rewind past your noisy brain, and become a perfectly reliable ghost of perfect emptiness.  That's not just because your brain is you.  It's also because you can't unassume things like modus ponens or belief updating.  You can unassume them as explicit premises for deliberate reasoning—a hunter-gatherer has no explicit concept of modus ponens—but you can't delete the actual dynamics (and all their products!)

So, in the end, I think we must allow the use of brains to think about thinking; and the use of evolved brains to think about evolution; and the use of inductive brains to think about induction; and the use of brains with an Occam prior to think about whether the universe appears to be simple; for these things we really cannot unwind entirely, even when we have reason to distrust them.  Strange loops through the meta level, I think, are not the same as circular logic.


Part of The Metaethics Sequence

Next post: "Rebelling Within Nature"

Previous post: "My Kind of Reflection"

Comments (87)

Sort By: Old
Comment author: Tiiba2 12 July 2008 06:32:46AM 0 points [-]

"""(Personally, I don't trust "I think therefore I am" even in real life, since it contains a term "am" whose meaning I find confusing, and I've learned to spread my confidence intervals very widely in the presence of basic confusion. As for absolute certainty, don't be silly.)"""

I'm just wondering, what do you think of the Ultimate Ensemble? If I'm not mistaken (I only read the Wikipedia article), it applies to existence your rule that if there's no difference, there should be no distinction.

Comment author: Tim_Tyler 12 July 2008 08:00:32AM 0 points [-]

Harvesting human body heat for energy might be stupid and inefficient way to use stored food - but I don't think it would violate the second law of thermodynamics. Maybe you are thinking that the flesh of the dead was the only source of food? That would indeed be even more silly - but the movie doesn't make that claim.

Comment author: Articulator 29 March 2014 04:10:14AM 0 points [-]

Fundamentally, the problem is that you need to get the energy somewhere. Currently, we get it indirectly from sunlight. In a world with no ability to obtain sunlight (as the justification of the Matrix goes), the second law means that barring geothermal (which doesn't require humans as a go-between), the total usable energy will decrease to zero.

It's like recycling. Can you ever expect to get better materials, or more materials than you started with without putting anything else in? (Including energy)

Comment author: ShardPhoenix 12 July 2008 08:20:07AM 3 points [-]

No matter how badly you're being tricked, there still must be something there to be tricked. So Cogito Ergo Sum applies without any need for further logical justification than the fact that you thought of it (whether naturally or induced by the deceiving demon, and regardless of the validity any further reasoning or justification) in the first place.

A more generalized idea might be "something exists".

Comment author: TheAncientGeek 29 March 2014 11:24:07AM 3 points [-]

So the cogito is a necessary truth? Except that it isn't, because there are arguments against it that don't depend on illusion. The Hume Buddha argument has it that thoughts don't imply thinkers.

Comment author: dxu 21 March 2015 10:09:27PM 3 points [-]

Define "exist".

Comment author: JohnBuridan 21 March 2015 11:35:50PM *  0 points [-]

Lamest come back ever.

"To exist" is a verb used in sentences to refer to a thing's persistence in the universe. Exemplum gratiae, the loutish commentator above who persistently requests for definitions exists. :P

Agreement about the function of language is required for any discussion. But since most of us don't think language is reducible to atomic propositions, we have little choice but to rely on convention in most instances. ShardPhoenix is using Standard Written English. In our society, you must use Standard Written English to be credible or taken seriously.

dxu, are you taking ShardPhoenix seriously, are you trying to push the envelope, or are you testing the waters for absolute skepticism? I'm curious.

Comment author: dxu 22 March 2015 05:17:17AM 2 points [-]

Lamest come back ever.

In my experience, being snarky online is never a good idea. In any case, what made you think my comment was supposed to be a "come back", as opposed to a legitimate request for clarification?

"To exist" is a verb used in sentences to refer to a thing's persistence in the universe.

You've just passed the buck from the word "exist" to the word "universe". Define that as well, please.

loutish commentator

Charitably taken, this is a friendly jab meant to signal casualness. Uncharitably taken, this verges on attack. In the context of the rest of your comment, I'm inclined to think the latter, in which case I'd reiterate that snark is never helpful.

ShardPhoenix is using Standard Written English.

"Standard Written English", as you put it, is notoriously bad at conveying technical meaning. For more on this, see 37 Ways That Words Can Be Wrong.

dxu, are you taking ShardPhoenix seriously, are you trying to push the envelope, or are you testing the waters for absolute skepticism? I'm curious.

Rather than answer a loaded question like that, I have a question for you:

Are you interpreting me charitably, are you trying to push the envelope, or are you testing the waters for how rude one can be in a comment without getting called out on it? I'm curious.

Comment author: JohnBuridan 24 March 2015 01:39:23AM *  0 points [-]

I was chiding you, absolutely! But I will provide some context for my comments so that we can see where each other are coming from.

I did figure you were asking sincerely "exist" to be defined, but deny that the word in this context can be defined to any avail since it has a near tautological use. Things can exist in many ways: they can exist in the mind, or they can exist independent of the mind (mind-independent), or they can exist in the mind of God, or as a fact about the world. A thing can exist contingently or necessarily, a priori or a posteriori, as an analytic or a synthetic truth.

I was going to say that these technical terms don't really apply to the argument - that simple ordinary language reveals that there must be something for us to talk about anything. But on further reflection you are right. It is important to define in what way this something-that-must-exist exists.

However, I don't believe conventionally defining the word exist as you requested does anything more than "pass the buck" as you say to another term. I say that the meaning of a word is determined by how it is used. Sometimes an author or a community will be very specific about its use and it is a technical term.

Agreement about how to discuss language is key to continuing rational argument. It follows that we must agree about what it is we want when we ask for definitions too! My understanding generally comes from Wittgenstein (I highly recommend Ray Monk's biography on him!). This essay by David Foster Wallace is full of humor and, brilliant man that he is, explains quite a bit about linguistic theory in the process!

In sum, there is still room for discussion about what the phrase "something must exist" actually entails. I was too dismissive; I generally associate the quest to ever define with a fundamental misunderstanding about how language works.

Is it solipsistic in here or is it just me?!

Comment author: Ian_C. 12 July 2008 08:50:16AM 0 points [-]

I think Descartes was right. Even if this is the Matrix and everything is fake, even your thought processes, then it is all still something and not nothing. By nothing I mean a dimensionless point. So at the very least you are *something*. "I think therefore I am."

Comment author: DanielLC 30 June 2012 04:45:59AM 1 point [-]

Even if this is the Matrix...

Yes, but what if this isn't the Matrix? What if there really is nothing?

Comment author: TheAncientGeek 29 March 2014 11:35:35AM 1 point [-]

At the very least something is something.

Comment author: Hopefully_Anonymous 12 July 2008 10:17:10AM 0 points [-]

Interesting post. I agree with the other commenters that "I think therefore I am more than zero information" is on a different (more doubt-resistant) level than other doubtable concepts.

Comment author: robin_brandt2 12 July 2008 10:20:24AM 1 point [-]

My version is "there is experience, therefore something is", but some year after I also understood that there is no reason even to trust that logic, no matter how bullet-proof it sounds...

A hard question for me is still, what is the functional role and underlying mechanism of qualia in the timeless mathematical structure of the multiverse. And what is the minimal structure required for qualia. And I mean qualia in the direct phenomenal sense, not defined as anything epiphenomenal. I am perfectly aware of and a great fan of the Anti-Zomibe principle. Still this basic question just does not disolve for me.

Comment author: Olivier 12 July 2008 10:30:56AM 2 points [-]

I always saw I think therefore I am as a definition of to be. What it basically says is that "Thought exists" and that when you are referring to yourself, ultimately, you are referring to these thoughts that you are certain exist. When you say "I walk", you mean "those thoughts of which I am certain perceive, through senses, that they are attached to a body that is walking".

It also means that those thoughts of which you are not certain, such as "we must allow the use of brains to think about thinking" are not part of you; they are a separate object. You are the presence of this thought, not its result.

Comment author: Ian_C. 12 July 2008 10:35:03AM 1 point [-]

Robin there's no logic to trust. Even though we might use the phrase "therefore," in reality we are not performing a process of reason, but just changing our perspective on a fact. Looking at the same data askance. That's my way of thinking about it anyway.

Comment author: Latanius2 12 July 2008 11:05:43AM 0 points [-]

"I think therefore I am"... So there is a little billiard ball in some model which is me, and it has a relatively stable existence in time. Can't you imagine a world in which these concepts simply make no sense? (If you couldn't, just look around, QM, GR...)

Comment author: Ben_Jones 12 July 2008 11:08:31AM 0 points [-]

for these things we really cannot unwind entirely, even when we have reason to distrust them.

I've had similar thoughts when reading Hofstadter etc. However, are you saying that strange loops and inductive reasoning are inherent properties of 'thinking minds,' or just of human minds? Is it, by definition, impossible to put to gether an AI that could step back and consider questions like 'why does the universe appear to be simple' without make prior assumptions?

Comment author: gwern 12 July 2008 12:14:13PM 0 points [-]

Latanius: Descartes doesn't say anything about what the 'I' is. Perhaps you would understand it better if the formula was "Something exists"? I really don't see how that can be objected to, except on grounds of vacuousness/triviality/tautologicality.

(Which ironically enough, is a reasonable translation of one of Parmenides's chief claims, that "It is.")

Comment author: Lara_Foster 12 July 2008 12:22:56PM 0 points [-]

Brandt: "what is the functional role and underlying mechanism of qualia in the timeless mathematical structure of the multiverse."

Doesn't this question imply that you believe qualia has a purpose? Isn't this a bit mystic? We know evolutions are stupid and goal-less...

" And what is the minimal structure required for qualia." Isn't this unknowable, given that the only way to really determine 'qualia' proper is to experience it? Read anti-zombie posts. Do you think an insect has qualia? What about a paramecium? It's a pretty intelligent little cell...

I also agree that it's pretty amazing that the universe seems intuitable AT ALL, and this might be the result of evolutions and their goal-less drives... If so... OH WELL! Lets have a good time. What's also strange is how other people's minds seem intuitable if you know certain essential information about them... Also strange that the minds of some animals are intuitable (ie I think I know my cat pretty well). However, that anything is understandable might just be because we live in a universe that we are capable of understanding. It's similar to the question, 'If life is so unlikely to occur randomly, then why do we exist?' Well... "If a universe capable of being comprehended by a mind, and a mind capable of understanding a universe are so unlikely to occur randomly, then why do they exist?" Tah Dah! Well... here we are. Or I might just be a bolzman brain. Go me!

Comment author: Eliezer_Yudkowsky 12 July 2008 12:48:41PM 8 points [-]

Very Bad Sign #243: Many people all agree that a statement is definitely true but they can't agree on why and/or what the statement means.

Comment author: notsonewuser 21 December 2013 03:29:12PM 0 points [-]

Like "killing babies is wrong"? Of course, that's the point of this whole sequence, I guess.

Comment author: michael_vassar3 12 July 2008 01:31:42PM 2 points [-]

Eliezer: I think that the folk here *do* agree on why "cogito" is true and on what the statement means, though of course, almost all meanings gloss over some confusion. I agree with elementary school students about what 2+2=4 means and why it is true even if I can ground it in number theory and they can't.

Laura: Evolutions are stupid and goal-less and produce structures with functional roles. No conflict with Brandt there. OTOH, evolution produces facts about the world, not facts about the features of structures in the space of physically possible structures, but the major confusion about qualia seems to be that we don't understand how qualia bearing structures can be present in the space of possible structures or how within that space they are distinguished from non-qualia bearing structures. It seems likely that if we answer this question we will also know how evolution in some environments could select for those structures at which point the question would have dissolved.

Comment author: Cyan2 12 July 2008 01:43:33PM 6 points [-]

Tim Tyler, the thermodynamically problematic part of the Matrix is the fact that humans had induced something like nuclear winter to deny the machines the energy of the sun. Morpheus states that the machines then used humans as a source of energy. Humans get their energy from food: no sun implies no food implies no humans.

Comment author: Someone_from_the_other_side 12 July 2008 02:16:53PM 0 points [-]

Much like above posters, I can't see how you could dispute "Something thinks therefore something is". Since the "I" is not specified in the original text (but it's nearing 10 years that I read a translation of it so take this with a grain of salt) I would say the substitution makes sense. At the very least, it is the meaning that everyone I ever discussed with was using. (I believe the "I" was merely a self referential concept, maybe like "this" in some OOP languages refers to the specific instance of the object in question).

As for the thermodynamics in Matrix, I faintly recall something about a fusion reactor working somewhere, but that may have been to power Zion, not the Matrix. I agree it is a colossal blunder in the whole series (and anyway, it would have been better if Reloaded and Revolutions were never made, Reloaded especially was unforgivably bad).

Comment author: Unknown 12 July 2008 02:37:13PM 2 points [-]

"Very Bad Sign #243: Many people all agree that a statement is definitely true but they can't agree on why and/or what the statement means."

Do you mean very bad for the person rejecting the statement? Such a consensus would seem to suggest that there are many reasons for affirming its truth, and many different ways it can be interpreted which are still true.

Comment author: Elliot_Temple 12 July 2008 02:57:58PM 0 points [-]

It's worth considering *how*

> And evolution certainly gets a chance to influence every single thought that runs through your mind!

works (if it does). Just because evolution created minds in the first place does not necessarily imply it has retained influence over everything that happens in them. For example, if a person builds a house that doesn't necessarily give him influence over the termites in the walls.

Comment author: Ian_Maxwell 12 July 2008 03:24:37PM 2 points [-]

I had thought of that particular plot hole solution. In fact, however, most violations of thermodynamics and other physical laws seem to occur within the Matrix, not outside. That is, the rules of the Matrix do not add up to normality.

There actually is a cover in the movie, though: the human energy source is "combined with a source of fusion". This is, as one review stated, like elaborately explaining how a 747 is powered by rubber-bands and then mentioning that this is combined with four jet engines.

Comment author: Doug_S. 12 July 2008 04:19:45PM 3 points [-]

You can blame focus groups for that particular blunder in The Matrix movie. The original script had the machines using human brains as computing nodes, but the focus groups consisted of people too stupid to understand what they were talking about.

Comment author: mwengler 01 July 2012 12:57:51AM 0 points [-]

I have no idea if this is actually true, but I like the idea a lot so I upvoted it.

Comment author: bamonster 12 July 2008 05:03:54PM 2 points [-]

One theory (The Matrix spawned a lot of philosophy talk, and even books) was that, unbeknownst to the machines themselves, they couldn't simply kill off the humans - for ethical reasons. I mean, there are obviously more efficient ways to generate energy, but the robots couldn't kill off their creators - so they came up with this elaborate scheme of harvesting energy from their bodies, and never thinking much about how they were actually losing energy in the process.

Comment author: Tim_Tyler 12 July 2008 05:30:53PM 1 point [-]

Would the humans would have "scorched the sky" if they had believed it would harm them more than the machines? Presumably they had food stockpiles - or some other means of surviving.

Anyway, this is a story told by Morpheus in the movie. We can only speculate about what actually happened. There is no need to hypothesize violations of the laws of thermodynamics.

Comment author: DanielLC 30 June 2012 04:51:27AM -1 points [-]

We do need to hypothesize why Morpheus doesn't understand the laws of thermodynamics.

Then again, Morpheus could be lying. Perhaps the question is why he thought Neo would believe a statement involving a violation of thermodynamics. For that matter, maybe the question is why Morpheus told Neo an obvious lie.

Comment author: shokwave 30 June 2012 05:22:20AM 1 point [-]

Shortly after we see Neo trained to ignore the laws of physics - jumping, karate, etc. I took it to be preparation for that.

Comment author: DanielLC 30 June 2012 06:36:07AM -1 points [-]

But he's doing it in the Matrix. It's like telling someone that there's life after death, to prepare them for extra lives in a video game.

Comment author: TheOtherDave 30 June 2012 05:44:41AM 1 point [-]

I don't see any reason why Morpheus should understand the laws of thermodynamics.

That said, I remember reading years ago someone's suggestion for how to make that scene much better:
Morpheus: yadda yadda utterly ridiculous explanation
Neo: But that makes no friggin' sense! It can't work that way!
M: And why is that?
N: It violates the laws of thermodynamics!
M: And you learned about those laws where?
N: In the -- oooh.
M: (nods sagely) They are subtle.

Comment author: DanielLC 30 June 2012 06:37:40AM -1 points [-]

I don't see any reason why Morpheus should understand the laws of thermodynamics.

It's not just Morpheus. If anyone in Zion understands it, they know that that's not why the machines are really keeping people. If nobody in Zion understands it, then how are they building working machines?

Comment author: mwengler 01 July 2012 01:05:11AM 2 points [-]

If you know ANYTHING about building working machines, you should know that most of the people that build them have no clues about the laws of thermodynamics.

Engineering is much more a respecter of intuition and trial and error than it is of theories.

One would expect a culture of a very limited number of free humans going up against a vastly materially and intelligently superior enemy to be primarily a raiding culture, anyway. Otherwise they haven't a prayer.

Comment author: DanielLC 01 July 2012 04:28:39AM 0 points [-]

How can they be a raiding culture? Who do they raid?

Comment author: Luke_A_Somers 02 July 2012 03:32:41AM 1 point [-]

The machines... they steal the machines that can't fight backk, retool them to their purposes. With all the things in the movie that don't make sense, that part does.

Comment author: Zack_M_Davis 30 June 2012 06:47:26AM 5 points [-]

I remember reading years ago someone's suggestion for how to make that scene much better

You may be thinking of Eliezer Yudkowsky's 2008 post "Fundamental Doubts"; scroll up or select the "View the original post" link above. Best wishes, the Less Wrong Reference Desk

Comment author: TheOtherDave 30 June 2012 02:06:26PM 0 points [-]
  • hangs head in embarrassment *


Comment author: gwern 30 June 2012 06:26:32PM *  5 points [-]

Which re-appeared in Methods of Rationality years later as 'WELCOME TO THE REAL WORLD': http://www.fanfiction.net/s/5782108/64/Harry_Potter_and_the_Methods_of_Rationality

...MORPHEUS: The machines tell elegant lies.


NEO (in a small voice): Could I please have a real physics textbook?

MORPHEUS: There is no such thing, Neo. The universe doesn't run on math.

Comment author: mwengler 01 July 2012 01:06:31AM 0 points [-]

This is the first comment I have ever wanted to upvote multiple times.

Comment author: poke 12 July 2008 06:03:03PM 3 points [-]

I don't buy this sort of skepticism at all. Yes, we can imagine that the external world in an illusion, but the basic flaw is (like so much in philosophy) privileging our ability to imagine something over science. Whether we can be deceived in this way is an empirical matter. Yes, you can say "everything you learned about empirical science is part of the illusion," but all you've done is taken your ability to imagine an outcome and privileged that above scientific experiment. Science always trumps imagination. It is therefore, I think, impossible to formulate the skeptical thesis.

This is difficult to think about. Philosophy has given us a view of the world where perception is essentially a subset of imagination. We have pictures in our head and sometimes, if we're lucky, they correspond to the world. The scientific view of perception, however, is that it's just physics-as-usual. The philosophical story is an a priori psychology; if you reject the a priori, yet still buy that story, then you haven't doubted "all the branches and leaves of that root" sufficiently. The scientific story of perception involves photons and receptors and neurons and macromolecules and all that good stuff. It can't be used to call those things into doubt.

The correct view of all this is a (restricted) Quinean one: You have to accept the ontology of science as basic ontology. Science undermines our methods of determining other (metaphysical) ontologies (i.e., a priori reasoning); in everything we do, beginning with thought and perception, science should be our starting point. Nothing we know about thought and perception can undermine what we know about physics and chemistry and molecular biology because thought and perception are high-level areas of biology: everything we know about them is based on scientific ontology. No skeptical theses that undermines science can be constructed (science, however, can still undermine our common sense view of the world) and without skepticism epistemology reduces to neurobiology and sociology.

This is the difficult part: Even if it's empirically possible to totally deceive somebody, to run a simulation of them inside a supercomputer and manipulate their entire life and history, we still have no reason to doubt science. Personally I doubt that this is possible. I think the whole concept of a "subjectively real" simulation is a basic error of reasoning and I doubt that cognition and memory can be so arbitrarily manipulated anyway. Regardless, if my doubts turn out to be unfounded, it will be empirical science that proves them unfounded and the argument itself will only be as strong as empirical science itself. We cannot formulate this argument based on what we can merely imagine happening to ourselves.

Descartes had it backwards. If he'd thrown out "I think therefore I am" and taken the new physics and mathematics as his starting point he would have had a very powerful form of naturalism on his hands. A naturalism that doubts common sense and accepts science as the starting point of all reason. As I like to say, there's no distance between ourselves and the world, what happens at the retina is no more privileged than what happens at the microscope or the voltage clamp. We can just as easily take those as our starting point.

Comment author: Unknown 12 July 2008 06:19:01PM 0 points [-]

Poke, if anyone could give credence to the idea that "science is just another religion", it would be you.

Comment author: Cyan2 12 July 2008 06:58:07PM 3 points [-]

We can only speculate about what actually happened.

I -- err -- what? What actually happened is that the Wachowski brothers made a movie. No humans were enslaved in the making of this film.

Comment author: JB 12 July 2008 08:24:38PM 0 points [-]

"Strange loops through the meta level, I think, are not the same as circular logic."

But of course you would say that, Eli.


Comment author: Patrick_(orthonormal) 12 July 2008 11:21:53PM 0 points [-]

Hmm. These doubts might seem sophomoric to us, since the "idiot god" of evolution couldn't conspire against our reasoning with the thoroughness of the Dark Lords of the Matrix. But it makes sense to consider these questions in the course of programming an AI, who will have cause to wonder whether its creators might have intentionally circumscribed its reasoning faculties...

Also, the problem with "cogito, ergo sum" is that it tempts us to posit a self distinct from the act of thinking, thus an immaterial soul, when the best interpretation seems to be that there is no "I" apart from the activity of my brain. I agree with Nietzsche here when he calls it a seductive trick of grammar, imagining that a verb implies a subject in this way.

Comment author: TheAncientGeek 29 March 2014 12:50:35PM 0 points [-]

Where did you heat about evolution?

Comment author: Mike_Blume 12 July 2008 11:43:31PM 1 point [-]

I've always taken cogito as "Here are some thoughts bouncing about. They must be causally related to some set of existent phenomena." which I think is pretty safe.

Comment author: Caledonian3 12 July 2008 11:58:10PM 0 points [-]

Such a consensus would seem to suggest that there are many reasons for affirming its truth, and many different ways it can be interpreted which are still true.

That means the sentence is ambiguous and probably meaningless.

Useful claims have an obvious, specific meanings and can potentially be shown to be wrong easily. Nonsense can often be seen for nonsense, but meaninglessness is harder to spot.

Comment author: Eliezer_Yudkowsky 13 July 2008 12:24:26AM 2 points [-]

Searched "On Certainty" for references to the word "probability". Found one hit, a reference to law courts.

Comment author: Hopefully_Anonymous 13 July 2008 12:29:27AM -2 points [-]

"Poke, if anyone could give credence to the idea that "science is just another religion", it would be you."

Uknown, you nailed it.

Comment author: poke 13 July 2008 02:47:27AM 4 points [-]

Unknown and Hopefully Anonymous, If basing your beliefs on established science and systematically rejecting every incompatible methodology is "religion" then stick a ridiculous hat on my head and call me the Pope of Reality.

Comment author: frelkins 13 July 2008 03:06:05AM 1 point [-]


In general I think it is safe to say that OB and its readers do not consider themselves philosophers and are not conversant with those systems of thought. It's helpful at this point to recall Deleuze's critique of philosophy here at OB, its repressive role in the history of thought.

OB in the main seem to consider themselves scientists, but altho I agree you are quite correct here to namecheck Wittgenstein, they do not overall view the world as he did - split coarsely between the analysis of language & concepts and the discovery of fact, or between philosophy and science.

I do think you have an excellent point that EY would benefit from reading PI (Philosophical Investigations), especially of course Sec. 561, where Wittgenstein discusses the meanings of "is."

That might resolve his concern about its supposed ambiguity.

A more general note to the blog might be that Descartes took his cogito from Augustine's formulation in City of God. There really is no doubt about its intended meaning - that commentators here might not know it or be able to articulate it doesn't make the statement itself meaningless or nonsense.

It's crucial to remember that the Cartesian project is to ultimately uphold the Sorbonne and Catholic doctrine; this is why his Dedication is to the theologians there. He himself was a devout Catholic his entire life and believed his Method extended Catholic thought and served to refute Skepticism.

Comment author: Vladimir_Nesov 13 July 2008 03:44:09AM 0 points [-]

I think it's cleaner to build the foundation around the basic laws of physics. You don't need the laws of reasoning in the description of physics, they define the way reality flows. Leaving the margin for doubt on the laws themselves, all the higher-level things like minds and principles of reasoning within the current framework can be built on top of that, like special kind of physical structures with a certain kind of dynamics. I think about the goal-driven intelligence (optimization process) as an attractor in the dynamics of reality, which drives it towards a certain kind of dynamics. When morality of the intelligent agent is changed, either by being broken significantly or by being influenced, it means that the high-level description that fits the process that is reality is changed. Where before the dynamics looked like it was heading in this direction, now it jumped to a form where it's heading in another direction, or is changed the course, so that it's heading in a slightly different way. The laws of physics are the substrate, on which one has to figure a way of building the attractor.

Another question is that if the mind is running inside a simulation, it is not semantically bound to the structure of real world, its goal gets hijacked by substituting the reality by the simulated happy button, the limited simulated reality that is accessible to inspection. Such mind can't have a global attractor effect on the flow of reality, but it can bind it locally, just as an ordinary human optimizer, so I guess it's not fundamentally different.

Comment author: Lara_Foster 13 July 2008 03:46:44AM 0 points [-]


Some problems which precede the questions you are asking about qualia: 1) What is qualia? Give your definition. 2) By what means do you recognize that something has qualia? This doesn't need to be a fool-proof algorithm, but at least a practical approximation for determining if, say, a cat or other creature similar to ourselves has qualia. If we can't say whether or not a fellow mammal has qualia, then how are we supposed to begin to deal with alien minds?

Comment author: Hopefully_Anonymous 13 July 2008 04:55:56AM 1 point [-]

"I think it's cleaner to build the foundation around the basic laws of physics." I think it's more honest to build the foundation on the observer, over the course of their life arc, encountering different experiences.

* hmm, there's this thing called language I'm speaking and this thing called thoughts I'm thinking. * hmm, there are these things called scientists, and this body of thought called science. This is what they've purported to discovery about reality. * hmm, these are the tiny handfuls of experiments I've done myself over the course of my life.

which apparently has for many lead to

* ... ... [death/permanent lack of conscious experience].

That's my interaction with science, and I think gives the scope of most or all of our interactions with science, and determining the nature of reality generally. Reality seems big and messy, internally and externally. It seems humans past and present have figured some (even many) things out about reality, but as an individual passing through we have limited opportunities to confirm these discoveries. There seems to be a strong motivation to express undue certainty about reality to increase one's status, as well perhaps to have strong belief in things just to ease one's mind or as an aesthetic, but that doesn't make the strong beliefs justified.

But that, not physics, seems to me to be the honest starting point. For Einstein, for Witten, for Eliezer, and for me.

Comment author: frelkins 13 July 2008 05:34:05AM 0 points [-]


"the foundation on the observer"

Does such an entity exist. . . .? Vlad's point seems less problematic. Philosophically.

Comment author: steven 13 July 2008 07:27:41PM 0 points [-]

WASONB: We All Sail On Neurath's Boat.

A more general argument than 2LT against humans as fuel goes something like, "it's a priori unlikely regardless of physics that the best way to generate power also happens to involve creating structures so complicated as conscious minds".

Comment author: Aaron6 14 July 2008 06:54:02AM 1 point [-]

Doug S: Cite? I'd love to believe that. It would also explain how humans can subvert the matrix, if they are themselves running the simulation.

Eliezer: Any universe with a reasonable notion of energy and probability will have statistical mechanics. In a sense it is far deeper than most of the physics that gets studied. The fact that gas expands when you heat it is due to the form of the density of states. The fact that you can get heat engines, and that they have a maximum efficiency of T_2 - T_1 / T_2 is stronger than that.

Comment author: Hopefully_Anonymous 14 July 2008 07:09:55AM 0 points [-]

Steven, actually you recover the argument a bit with that tactic. Perhaps in a Kasporovian irony, the machines were harnessing the "deep creativity" (part of a Kahneman system 1 intelligence?) of a massive parallel network of human brains to maximize their fuel efficiency and new fuel location innovations. If machines do become more powerful than us before they become as creative as us a Matrix type situation becomes somewhat more plausible. Thanks for inadvertantly making the Matrix a tiny bit more plausible to me, and thus more enjoyable.

Comment author: Dynamically_Linked 14 July 2008 09:23:45AM 0 points [-]

Aaron, statistical mechanics also depend on particle physics being time-reversible, meaning that two different microstates at time t will never evolve to the same microstate at time t+1. If this assumption is violated then entropy can decrease over time.

Is there some reason why time-reversibility has to be true?

If we can imagine a universe where entropy can be made to decrease, then living beings in it will certainly evolve to take advantage of this. Why shouldn't it be the case that human beings are especially good at this, and that is what they are being used for by the machines?

Comment author: Eliezer_Yudkowsky 14 July 2008 09:26:57AM 0 points [-]

HA: If machines do become more powerful than us before they become as creative as us

Unlikely. Where does humanity's power come from, after all?

Comment author: Hopefully_Anonymous 14 July 2008 10:54:50AM 0 points [-]

Eliezer, it wasn't a serious post. My serious view is that we're probably all going to die, but the question is whether any of us can beat information theoretic death and buy that long odds lottery ticket for reanimation by more technologically advanced unknown parties. I sure hope I can.

Comment author: athmwiji 15 July 2008 12:53:07AM 0 points [-]

The problem with taking physics as fundamental and working up from there is that it forces us to make a lot of assumptions about the nature of the hardware that underlies reality, which are not supported by observation.

I say this because in order to make an observation we have to describe the observation in terms of the hardware in the form that we assume it has. We can claim to observe the sun rising in the east or the number of atoms in a box, but these are actually judgment about our observations and these judgments are formulated in terms of the hardware

Such assumptions have repeatedly turned out to be a bad idea.

It also forces us to ignore the observations, our experiences, that lead to these judgments because they can not be formulated in terms of the hardware.

We can try to avoid this by saying our experiences take place at a higher level, but this still leaves the problem that there isn't any way to formulate experience in terms of the type of hardware assumed, so we have no idea what to look for at the higher level. One could then say that anything that can not be formulated in terms of the type of hardware assumed does not exist. But, this forces us to ignore our observations and hold on to assumptions that contradict evidence that is continually beating us over the head.

We can avoid this by not making any assumptions about hardware, and instead using the structure of subjective experience as our foundation.

Eventually we should know enough to be able to open our brains and have experiences that are far more different from anything anyone has experienced then the difference between green and bliss, and by then we may have a clear enough idea of whats going on to identify qualia bearing structures, and even communicate qualia directly, but until then i don't think we can say much about hardware other then there seems to be something out there even when we aren't looking and it is correlated with our experiences.

With regard to the matrix, I had guessed that Morphious just didn't know about thermodynamics or why the machines kept people alive, and that the matrix was second life gone wrong... or right... depending on how you look at it.

Or perhaps that the ai were instantiated in a literal collective unconscious of human brains

Comment author: Caledonian2 15 July 2008 03:16:26AM 0 points [-]

and by then we may have a clear enough idea of whats going on to identify qualia bearing structures, and even communicate qualia directly

Provide an operational definition of the word 'qualia' or stop usingit.

Comment author: Wei_Dai2 16 July 2008 12:52:23AM 1 point [-]

Some problems are hard to solve, and hard even to define clearly. It's possible that "qualia" is not referring to anything meaningful, but unless you are able to explain why it feels meaningful to someone, but isn't really, I don't think you should demand that they stop using it.

Having said that, here's my attempt at an operational definition of qualia.

Comment author: Aaron6 21 July 2008 10:50:57PM 0 points [-]

Hopefully Anonymous: the usual phrasing is not "time-reversible" for that, as that can be interpreted as "the laws of physics are the same under exchange of t with -t". One usual phrasing is "non-dissipative", though I hold out for "retrodictable". Even this isn't sufficient for entropy to be conserved -- what's necessary is conservation of phase space volume. I'm going to cheese out and say that as energy conservation is enough to do this, that my comment about "reasonable" physics covers this.

But statistical mechanics can still treat the case where it isn't. Yes, entropy can decrease. That violates the second law of thermodynamics. Note that it isn't a law of statistical mechanics, and that we do in fact observe entropy fluctuations that dip negative in real open systems, without making statistical mechanics inapplicable. Note also that at most one closed system exists -- the universe itself.

Comment author: Tim_Tyler 11 September 2008 07:31:22AM 2 points [-]

Re: The original script had the machines using human brains as computing nodes, but the focus groups consisted of people too stupid to understand what they were talking about.

Here's what this original script says:

They discovered a new form of fusion. All that was required to initiate the reaction was a small electric charge. Throughout human history we have been dependent on machines to survive. Fate, it seems, is not without a sense of irony.

- http://thematrixtruth.remoteviewinglight.com/html/original-matrix-script-13.html

Comment author: buybuydandavis 28 September 2011 12:02:04AM *  0 points [-]

Well... in that scenario, you're pretty much screwed, I'd have to say. ...

So you might as well not think about the possibility that you're a brain with choreographed thoughts, because there's nothing you can do about it...

Unless, of course, that's what they want you to think.

Yep. That's pretty much the conclusion I drew in a previous post in the sequence. http://lesswrong.com/lw/s0/where_recursive_justification_hits_bottom/4wsn

Assume you can win. Assume that everything necessary for you to win is true. If it isn't, you're screwed anyway.

Comment author: Carinthium 06 August 2013 02:12:23PM 0 points [-]

Effectively, of course, this is a sort of faith (metaphorically at least). Like many philosophers, you seem to have the implicit assumption that the Total Skeptical argument should be assumed false... just because. Seems a pretty good case for being a Total Skeptic.

In addition, to an extent it is possible to take Doubt further than you do, if in a minor way, without running into true pragmatic difficulties. The argument goes like this:

-What if my senses are a total illusion? If memory and Induction remain unquestioned, the conclusion here is that you have a consistent world you can affect to affect yourself and your "well-being" in terms of pleasure and pain anyway. A potential argument for selfishness.

For an AI there is no reason that isn't mind-bogglingly stupid to program this as part of their core philosophy. However, for human beings it is at least credible enough to consider.

Comment author: MondSemmel 20 September 2013 03:17:42PM 0 points [-]

One of the benefits I've drawn from Less Wrong so far - via posts like The Simple Truth - is more solid foundations for my beliefs. Since I study physics, I wasn't particularly worried about philosophical arguments against the scientific method anymore - science seemed to work, after all - but of those doubts that remained, many more still got (apparently) dispelled or dissolved.

That said, I never had doubts that fundamental. Could anybody really live that way? I don't have a coherent mental model for such a situation. Take Mad-Eye Moody in HPMoR with his constant paranoia, as in "Unless, of course, that's what they want you to think." - could such a character really function in the real world? Could they eat food without starving themselves to death out of paranoia? If you are paranoid about both X -> Y and X -> ~Y, how do you decide whether to do Y or ~Y? (It seems more plausible to me that Moody doesn't actually exhibit doubts that fundamental, than that he or anyone else could function properly despite those doubts.)

Comment author: Articulator 29 March 2014 04:22:49AM 0 points [-]

[I'm probably going to be the latest in a long line of people saying something like this, but I hope my wording, at least, makes this worth existing.]

"I think, therefore I am" is, in fact, deductive reasoning. The definition of "I am", as thought in the first person, as far as we can comprehend it, means "I think".

"I think, therefore I think" Or, more simply, "I think". The statement itself, as we are thinking it, cannot possibly be false - no matter the Demons we posit, we cannot be in simultaneous states of comprehending and not thinking.

"I am", by our very definitions, must be true for everyone who is reading this, as you are reading it. Because you are reading it.

I'm sorry, Eliezer, but I think you are mistaken if you think you can disprove "I am", thought in the first person.