Think Like Reality

49 Eliezer_Yudkowsky 02 May 2007 06:36AM

Whenever I hear someone describe quantum physics as "weird" - whenever I hear someone bewailing the mysterious effects of observation on the observed, or the bizarre existence of nonlocal correlations, or the incredible impossibility of knowing position and momentum at the same time - then I think to myself:  This person will never understand physics no matter how many books they read.

Reality has been around since long before you showed up.  Don't go calling it nasty names like "bizarre" or "incredible".  The universe was propagating complex amplitudes through configuration space for ten billion years before life ever emerged on Earth.  Quantum physics is not "weird".  You are weird.  You have the absolutely bizarre idea that reality ought to consist of little billiard balls bopping around, when in fact reality is a perfectly normal cloud of complex amplitude in configuration space.  This is your problem, not reality's, and you are the one who needs to change.

continue reading »

Better Disagreement

70 lukeprog 24 October 2011 09:13PM

Honest disagreement is often a good sign of progress.

- Gandhi

 

Now that most communication is remote rather than face-to-face, people are comfortable disagreeing more often. How, then, can we disagree well? If the goal is intellectual progress, those who disagree should aim not for name-calling but for honest counterargument.

To be more specific, we might use a disagreement hierarchy. Below is the hierarchy proposed by Paul Graham (with DH7 added by Black Belt Bayesian).1

 

DH0: Name-Calling. The lowest form of disagreement, this ranges from "u r fag!!!" to "He’s just a troll" to "The author is a self-important dilettante."

DH1: Ad Hominem. An ad hominem ('against the man') argument won’t refute the original claim, but it might at least be relevant. If a senator says we should raise the salary of senators, you might reply: "Of course he’d say that; he’s a senator." That might be relevant, but it doesn’t refute the original claim: "If there’s something wrong with the senator’s argument, you should say what it is; and if there isn’t, what difference does it make that he’s a senator?"

DH2: Responding to Tone. At this level we actually respond to the writing rather than the writer, but we're responding to tone rather than substance. For example: "It’s terrible how flippantly the author dimisses theology."

DH3: Contradiction. Graham writes: "In this stage we finally get responses to what was said, rather than how or by whom. The lowest form of response to an argument is simply to state the opposing case, with little or no supporting evidence." For example: "It’s terrible how flippantly the author dismisses theology. Theology is a legitimate inquiry into truth."

DH4: Counterargument. Finally, a form of disagreement that might persuade! Counterargument is "contradiction plus reasoning and/or evidence." Still, counterargument is often directed at a minor point, or turns out to be an example of two people talking past each other, as in the parable about a tree falling in the forest.

DH5: Refutation. In refutation, you quote (or paraphrase) a precise claim or argument by the author and explain why the claim is false, or why the argument doesn’t work. With refutation, you're sure to engage exactly what the author said, and offer a direct counterargument with evidence and reason.

DH6: Refuting the Central Point. Graham writes: "The force of a refutation depends on what you refute. The most powerful form of disagreement is to refute someone’s central point." A refutation of the central point may look like this: "The author’s central point appears to be X. For example, he writes 'blah blah blah.' He also writes 'blah blah.' But this is wrong, because (1) argument one, (2) argument two, and (3) argument three."

DH7: Improve the Argument, then Refute Its Central Point. Black Belt Bayesian writes: "If you’re interested in being on the right side of disputes, you will refute your opponents' arguments. But if you're interested in producing truth, you will fix your opponents' arguments for them. To win, you must fight not only the creature you encounter; you [also] must fight the most horrible thing that can be constructed from its corpse."2 Also see: The Least Convenient Possible World.

 

Having names for biases and fallacies can help us notice and correct them, and having labels for different kinds of disagreement can help us zoom in on the parts of a disagreement that matter.

continue reading »

Created Already In Motion

27 Eliezer_Yudkowsky 01 July 2008 06:03AM

Followup toNo Universally Compelling Arguments, Passing the Recursive Buck

Lewis Carroll, who was also a mathematician, once wrote a short dialogue called What the Tortoise said to Achilles.  If you have not yet read this ancient classic, consider doing so now.

The Tortoise offers Achilles a step of reasoning drawn from Euclid's First Proposition:

(A)  Things that are equal to the same are equal to each other.
(B)  The two sides of this Triangle are things that are equal to the same.
(Z)  The two sides of this Triangle are equal to each other.

Tortoise:  "And if some reader had not yet accepted A and B as true, he might still accept the sequence as a valid one, I suppose?"

Achilles:   "No doubt such a reader might exist.  He might say, 'I accept as true the Hypothetical Proposition that, if A and B be true, Z must be true; but, I don't accept A and B as true.'  Such a reader would do wisely in abandoning Euclid, and taking to football."

Tortoise:  "And might there not also be some reader who would say, 'I accept A and B as true, but I don't accept the Hypothetical'?"

continue reading »

Passing the Recursive Buck

15 Eliezer_Yudkowsky 16 June 2008 04:50AM

Followup toArtificial Addition, The Ultimate Source, Gödel, Escher, Bach: An Eternal Golden Braid

Yesterday, I talked about what happens when you look at your own mind, reflecting upon yourself, and search for the source of your own decisions.

Let's say you decided to run into a burning orphanage and save a young child.  You look back on the decision and wonder: was your empathy with children, your ability to imagine what it would be like to be on fire, the decisive factor?  Did it compel you to run into the orphanage?

No, you reason, because if you'd needed to prevent a nuclear weapon from going off in the building next door, you would have run to disarm the nuke, and let the orphanage burn.  So a burning orphanage is not something that controls you directly.  Your fear certainly didn't control you.  And as for your duties, it seems like you could have ignored them (if you wanted to).

So if none of these parts of yourself that you focus upon, are of themselves decisive... then there must be some extra and additional thing that is decisive!  And that, of course, would be this "you" thing that is looking over your thoughts from outside.

continue reading »

Existential Angst Factory

45 Eliezer_Yudkowsky 19 July 2008 06:55AM

Followup toThe Moral Void

A widespread excuse for avoiding rationality is the widespread belief that it is "rational" to believe life is meaningless, and thus suffer existential angst.  This is one of the secondary reasons why it is worth discussing the nature of morality.  But it's also worth attacking existential angst directly.

I suspect that most existential angst is not really existential.  I think that most of what is labeled "existential angst" comes from trying to solve the wrong problem.

Let's say you're trapped in an unsatisfying relationship, so you're unhappy.  You consider going on a skiing trip, or you actually go on a skiing trip, and you're still unhappy.  You eat some chocolate, but you're still unhappy.  You do some volunteer work at a charity (or better yet, work the same hours professionally and donate the money, thus applying the Law of Comparative Advantage) and you're still unhappy because you're in an unsatisfying relationship.

So you say something like:  "Skiing is meaningless, chocolate is meaningless, charity is meaningless, life is doomed to be an endless stream of woe."  And you blame this on the universe being a mere dance of atoms, empty of meaning.  Not necessarily because of some kind of subconsciously deliberate Freudian substitution to avoid acknowledging your real problem, but because you've stopped hoping that your real problem is solvable.  And so, as a sheer unexplained background fact, you observe that you're always unhappy.

continue reading »

Sorting Pebbles Into Correct Heaps

75 Eliezer_Yudkowsky 10 August 2008 01:00AM

Followup toAnthropomorphic Optimism

Once upon a time there was a strange little species—that might have been biological, or might have been synthetic, and perhaps were only a dream—whose passion was sorting pebbles into correct heaps.

They couldn't tell you why some heaps were correct, and some incorrect.  But all of them agreed that the most important thing in the world was to create correct heaps, and scatter incorrect ones.

Why the Pebblesorting People cared so much, is lost to this history—maybe a Fisherian runaway sexual selection, started by sheer accident a million years ago?  Or maybe a strange work of sentient art, created by more powerful minds and abandoned?

But it mattered so drastically to them, this sorting of pebbles, that all the Pebblesorting philosophers said in unison that pebble-heap-sorting was the very meaning of their lives: and held that the only justified reason to eat was to sort pebbles, the only justified reason to mate was to sort pebbles, the only justified reason to participate in their world economy was to efficiently sort pebbles.

The Pebblesorting People all agreed on that, but they didn't always agree on which heaps were correct or incorrect.

continue reading »

The Design Space of Minds-In-General

19 Eliezer_Yudkowsky 25 June 2008 06:37AM

Followup toThe Psychological Unity of Humankind

People ask me, "What will Artificial Intelligences be like?  What will they do?  Tell us your amazing story about the future."

And lo, I say unto them, "You have asked me a trick question."

ATP synthase is a molecular machine - one of three known occasions when evolution has invented the freely rotating wheel - which is essentially the same in animal mitochondria, plant chloroplasts, and bacteria.  ATP synthase has not changed significantly since the rise of eukaryotic life two billion years ago.  It's is something we all have in common -  thanks to the way that evolution strongly conserves certain genes; once many other genes depend on a gene, a mutation will tend to break all the dependencies.

Any two AI designs might be less similar to each other than you are to a petunia.

continue reading »

JFK was not assassinated: prior probability zero events

20 Stuart_Armstrong 27 April 2016 11:47AM

A lot of my work involves tweaking the utility or probability of an agent to make it believe - or act as if it believed - impossible or almost impossible events. But we have to be careful about this; an agent that believes the impossible may not be so different from one that doesn't.

Consider for instance an agent that assigns a prior probability of zero to JFK ever having been assassinated. No matter what evidence you present to it, it will go on disbelieving the "non-zero gunmen theory".

Initially, the agent will behave very unusually. If it was in charge of JFK's security in Dallas before the shooting, it would have sent all secret service agents home, because no assassination could happen. Immediately after the assassination, it would have disbelieved everything. The films would have been faked or misinterpreted; the witnesses, deluded; the dead body of the president, that of twin or an actor. It would have had huge problems with the aftermath, trying to reject all the evidence of death, seeing a vast conspiracy to hide the truth of JFK's non-death, including the many other conspiracy theories that must be false flags, because they all agree with the wrong statement that the president was actually assassinated.

But as time went on, the agent's behaviour would start to become more and more normal. It would realise the conspiracy was incredibly thorough in its faking of the evidence. All avenues it pursued to expose them would come to naught. It would stop expecting people to come forward and confess the joke, it would stop expecting to find radical new evidence overturning the accepted narrative. After a while, it would start to expect the next new piece of evidence to be in favour of the assassination idea - because if a conspiracy has been faking things this well so far, then they should continue to do so in the future. Though it cannot change its view of the assassination, its expectation for observations converge towards the norm.

If it does a really thorough investigation, it might stop believing in a conspiracy at all. At some point, the probability of a miracle will start to become more likely than a perfect but undetectable conspiracy. It is very unlikely that Lee Harvey Oswald shot at JFK, missed, and the president's head exploded simultaneously for unrelated natural causes. But after a while, such a miraculous explanation will start to become more likely than anything else the agent can consider. This explanation opens the possibility of miracles; but again, if the agent is very thorough, it will fail to find evidence of other miracles, and will probably settle on "an unrepeatable miracle caused JFK's death in a way that is physically undetectable".

But then note that such an agent will have a probability distribution over future events that is almost indistinguishable from a normal agent that just believes the standard story of JFK being assassinated. The zero-prior has been negated, not in theory but in practice.

 

How to do proper probability manipulation

This section is still somewhat a work in progress.

So the agent believes one false fact about the world, but its expectation is otherwise normal. This can be both desirable and undesirable. The negative is if we try and control the agent forever by giving it a false fact.

To see the positive, ask why would we want an agent to believe impossible things in the first place? Well, one example was an Oracle design where the Oracle didn't believe its output message would ever be read. Here we wanted the Oracle to believe the message wouldn't be read, but not believe anything else too weird about the world.

In terms of causality, if X designates the message being read at time t, and B and A are event before and after t, respectively, we want P(B|X)≈P(B) (probabilities about current facts in the world shouldn't change much) while P(A|X)≠P(A) is fine and often expected (the future should be different if the message is read or not).

In the JFK example, the agent eventually concluded "a miracle happened". I'll call this miracle a scrambling point. It's kind of a breakdown in causality: two futures are merged into one, given two different pasts. The two pasts are "JFK was assassinated" and "JFK wasn't assassinated", and their common scrambled future is "everything appears as if JFK was assassinated". The non-assassination belief has shifted the past but not the future.

For the Oracle, we want to do the reverse: we want the non-reading belief to shift the future but not the past. However, unlike the JFK assassination, we can try and build the scrambling point. That's why I always talk about messages going down noisy wires, or specific quantum events, or chaotic processes. If the past goes through a truly stochastic event (it doesn't matter whether there is true randomness or just that the agent can't figure out the consequences), we can get what we want.

The Oracle idea will go wrong if the Oracle conclude that non-reading must imply something is different about the past (maybe it can see through chaos in ways we thought it couldn't), just as the JFK assassination denier will continue to be crazy if can't find a route to reach "everything appears as if JFK was assassinated".

But there is a break in the symmetry: the JFK assassination denier will eventually reach that point as long as the world is complex and stochastic enough. While the Oracle requires that the future probabilities be the same in all (realistic) past universes.

Now, once the Oracle's message has been read, the Oracle will find itself in the same situation as the other agent: believing an impossible thing. For Oracles, we can simply reset them. Other agents might have to behave more like the JFK assassination disbeliever. Though if we're careful, we can quantify things more precisely, as I attempted to do here.

The Level Above Mine

42 Eliezer_Yudkowsky 26 September 2008 09:18AM

Followup toThe Proper Use of Humility, Tsuyoku Naritai

(At this point, I fear that I must recurse into a subsequence; but if all goes as planned, it really will be short.)

I once lent Xiaoguang "Mike" Li my copy of "Probability Theory: The Logic of Science".  Mike Li read some of it, and then came back and said:

"Wow... it's like Jaynes is a thousand-year-old vampire."

Then Mike said, "No, wait, let me explain that—" and I said, "No, I know exactly what you mean."  It's a convention in fantasy literature that the older a vampire gets, the more powerful they become.

I'd enjoyed math proofs before I encountered Jaynes.  But E.T. Jaynes was the first time I picked up a sense of formidability from mathematical arguments.  Maybe because Jaynes was lining up "paradoxes" that had been used to object to Bayesianism, and then blasting them to pieces with overwhelming firepower—power being used to overcome others.  Or maybe the sense of formidability came from Jaynes not treating his math as a game of aesthetics; Jaynes cared about probability theory, it was bound up with other considerations that mattered, to him and to me too.

For whatever reason, the sense I get of Jaynes is one of terrifying swift perfection—something that would arrive at the correct answer by the shortest possible route, tearing all surrounding mistakes to shreds in the same motion.  Of course, when you write a book, you get a chance to show only your best side.  But still.

continue reading »

Illusion of Transparency: Why No One Understands You

57 Eliezer_Yudkowsky 20 October 2007 11:49PM

In hindsight bias, people who know the outcome of a situation believe the outcome should have been easy to predict in advance.  Knowing the outcome, we reinterpret the situation in light of that outcome.  Even when warned, we can't de-interpret to empathize with someone who doesn't know what we know.

Closely related is the illusion of transparency:  We always know what we mean by our words, and so we expect others to know it too.  Reading our own writing, the intended interpretation falls easily into place, guided by our knowledge of what we really meant.  It's hard to empathize with someone who must interpret blindly, guided only by the words.

June recommends a restaurant to Mark; Mark dines there and discovers (a) unimpressive food and mediocre service (b) delicious food and impeccable service.  Then Mark leaves the following message on June's answering machine:  "June, I just finished dinner at the restaurant you recommended, and I must say, it was marvelous, just marvelous."  Keysar (1994) presented a group of subjects with scenario (a), and 59% thought that Mark's message was sarcastic and that Jane would perceive the sarcasm.  Among other subjects, told scenario (b), only 3% thought that Jane would perceive Mark's message as sarcastic.  Keysar and Barr (2002) seem to indicate that an actual voice message was played back to the subjects.

continue reading »

View more: Next