LESSWRONG
LW

Comment Permalink

Wow, I'm glad this kind of analysis is showing up in mainstream publications.

Norvig is describing an important insight from information theory: the amount of information you get from learning something is equal to the log of the inverse of the probability you assigned to it (log 1/p). (This value is called the "surprisal" or "self-information".)

So, always getting results you expect (i.e. put a high p on), means you're getting little information out of the experiments, and you should be doing ones where you expect the result to be less probable.

Therefore, to have a good experiment, you want to maximize the "expected surprisal" (i.e. sum over p * log(1/p)), which is equivalent to the entropy, and probably the basis for the method you mention.

Oscar_Cunningham14y10

Is LW broken for everyone?

ETA: When I wrote this, the "Comments" page was one of the few I could access, hence it being posted in such a strange place.

See in context

10 Rationality quotes: August 2010

by Cyan

3rd Aug 2010

1 min read

216

10

This is our monthly thread for collecting these little gems and pearls of wisdom, rationality-related quotes you've seen recently, or had stored in your quotesfile for ages, and which might be handy to link to in one of our discussions.

Please post all quotes separately, so that they can be voted up/down separately. (If they are strongly related, reply to your own comments. If strongly ordered, then go ahead and post them together.)
Do not quote yourself.
Do not quote comments/posts on LW/OB.
No more than 5 quotes per person per monthly thread, please.

Rationality Quotes

Personal Blog

10

Rationality quotes: August 2010

New Comment

216 comments, sorted by

top scoring

Click to highlight new comments since: Today at 1:04 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Rain14y1780

Personally, I've been hearing all my life about the Serious Philosophical Issues posed by life extension, and my attitude has always been that I'm willing to grapple with those issues for as many centuries as it takes.

-- Patrick Nielsen Hayden

[-]gwern14y140

This is a good example of how some areas are most concisely dealt with by ridicule.

1knb14y

What are the Serious Philosophical Issues posed by life extension? I can see many serious practical issues, but in what way should philosophical opinions change due to the mere extension of human life span?

8NihilCredo14y

I conjecture that we're supposed to read it under the classical definition of "philosophy", which used to include pretty much every type of intellectual discussion, including such practical issues as how to properly raise children, how to organise a political society, etc.

1jsnider32y

"Who wants to live forever when love must die?"

1goldfish14y

There is a great ted lecture on this subject. I thought he did a good job at addressing the concerns. At least to the point of defending that research should continue to at least allow future generations to decide if they think it is acceptable. http://www.ted.com/talks/aubrey_de_grey_says_we_can_avoid_aging.html

0[anonymous]14y

I don't know.

0[anonymous]14y

What is a day for?

0Document13y

Is the source online? The first page of Google results seems to be just people quoting the line.

4Rain13y

I tracked down the source as rec.arts.sf.fandom, 09 Aug 2000 (previous to any use as a quote), in a thread titled, "Do cats go to heaven?" It's (sort of) available online in the Google Groups archive. However, the provided link does not seem to work due to the huge length of the thread (it's trying to create a tree threaded discussion out of 4189 posts), but I was able to see several references to it inside, including the beginning as a snippet.

7Rain11y

Due to the new release of Google Groups, I was able to go through this thread and found the original reference on page 41 of the thread, posted 08 Aug 2000.

0scotherns14y

Is Rain's quote the most upvoted entry of all time? Its currently at +62.

0[anonymous]14y

I win!

0katydee14y

It's the most upvoted comment I've ever seen. I'm not sure about top-level posts, though.

0Rain14y

The current first page of 'Recent Posts' shows 7 with a higher point total than 62; two posts are above 100 points, both by Yvain.

0Apprentice14y

It would be interesting to see a compilation of the most upvoted entries from all the quote threads.

4Rain14y

DanielVarga created a top quote list, last updated in March.

[-]Randaly14y400

"Any sufficiently analyzed magic is indistinguishable from SCIENCE!"

~Girl Genius

[-]DSimon14y210

Young Agatha Clay: But how can they protect me if they aren't here? That's illogical.

Uncle Barry: Um...It's science.

Young Agatha Clay: Ah. You mean you'll explain when I have a sufficiently advanced educational background.

[-]JoshuaZ14y100

Yes, but in context this doesn't quite mean what it sounds like. In Girl Genius, "Science" is almost a password for any weird things built by sparks, rather than what a rationalist would call science.

6DSimon14y

Well, presumably the sparks have a rational understanding of the things they build, even if the reader doesn't. Excepting things like the clanks Agatha built when she was asleep, but she's something of an odd one even among sparks...

3Zubon14y

Girl Genius has visited that border between magic and science. Note also the first two frames, in which the spark rational mind is put to use in pursuit of spark emotional needs. "Look. I'm a girl with needs. Okay?"

5orthonormal14y

Indeed. It's Science as Attire in more ways than one.

5Zack_M_Davis14y

Two more: ----------------------------------------

5pjeby14y

But it should have... and Tarvek correctly notices that he is confused. (The reason for his confusion doesn't become clear until later, though.)

2Yitz2y

Does this make Brandon Sanderson scifi?

[-]DSimon14y350

My hotel doesn't have a 13th floor because of superstition, but people on the 14th floor, you should know what floor you're really on. If you jump out the window, you will die sooner than you expect.

-- Mitch Hedberg (Quoted from memory)

[-]djcb14y150

Superstition produces bad luck

-- Anonymous

[-]Richard_Kennaway14y310

When I was a young man about to go out into the world, my father says to me a very valuable thing. He says to me like this... "Son," the old guy says, "I am sorry that I am not able to bank roll you to a very large start, but not having any potatoes which to give you, I am now going to stake you to some very valuable advice. One of these days in your travels, a guy is going to come to you and show you a nice, brand new deck of cards on which the seal has not yet been broken. This man is going to offer to bet you that he can make the jack of spades jump out of the deck and squirt cider in your ear. Now son, do not bet this man, for as sure as you stand there, you are going to wind up with an earful of cider."

-- Sky Masterson, a character in "Guys and Dolls"

6Psychohistorian14y

My recollection of this involved "bet you five thousand dollars he can..." and "going to wind up five grand in the hole with an earful of cider," which makes the quote more entertaining. Nonetheless, an excellent quote.

[-]Jayson_Virissimo14y290

The fact that you are giving money to charity does not mean that you need not try to find out whether that charity is a fraud or not.

-C. S. Lewis, Mere Christianity

5SilasBarta14y

B... But, but he wants ... he ... says you, it's a good idea to ... argh!

8MartinB14y

His failing in one area, does not make his quotes untrue. Just a bit iffy. Thats why I try not to quote Gandhi or Churchill anymore.

2Matt_Simpson14y

2SilasBarta14y

Yes, my thought exactly. This isn't the first time someone's posted a quote from Lewis that he didn't follow. My reply was me expressing shock that someone like him would have grounds to lecture others about not giving money to questionable charities.

6NihilCredo14y

Lewis was a convert to Christianity. The usual self-analytical deficiencies of religious believers are often gigantified in converts, possibly because they adopted their beliefs out of need rather than simple habit and thus will hug them much tighter.

0magfrump14y

Similar ideas discussed here.

3NihilCredo14y

I think there are two slightly different situations at work here. Lewis did grow up in a very Christian environment, and as such possessed the "antibodies" described in the linked post. His being a convert from atheism - by means of some kind of emotional breakdown - didn't, thus, make him into an advocate of following every passage of the Bible and every idea ever presented by the Magisterium; rather, it made him a zealous defender of every excuse for why you don't need to do so in order to be a good Christian.

1magfrump14y

That's fair--I didn't mean to say that it was the exact same case, simply that similar topics had been discussed before. That wasn't clear from the original comment so I'll change it to make it a little more so.

4sark14y

Just an observation of how those most motivated to figure out the effectiveness of a charity are those who don't want to donate. But of course, this only applies to seeking out evidence of non-effectiveness. Charities don't usually think of other charities as competition. Also, most of the time, most just rationalize a reason for non-effectiveness.

[-]Apprentice14y270

Upon his death man must leave everything behind ... and depart forever from the world he has known. He must of necessity go to that foul land of death, a fact which makes death the most sorrowful of all events. ... Some foreign doctrines, however, teach that death should not be regarded as profoundly sorrowful. ... These are all gross deceptions contrary to human sentiment and fundamental truths. Not to be happy over happy events, not to be saddened by sorrowful events, not to show surprise at astonishing events, in a word, to consider it proper not to be moved by whatever happens, are all foreign types of deception and falsehood. They are contrary to human nature and extremely repugnant to me.

-- Motoori Norinaga (1730-1801) - quoted from Blocker, Japanese Philosophy, p. 109

Motoori was as far as you can get from being a rationalist but this quote was so Yudkowskian that I felt it belonged here.

[-]Kazuo_Thow14y270

... the history of mathematics is a history of horrendously difficult problems being solved by young people too ignorant to know that they were impossible.

-- Freeman Dyson, "Birds and Frogs"

[-]sketerpot14y250

If you do experiments and you're always right, then you aren't getting enough information out of those experiments. You want your experiment to be like the flip of a coin: You have no idea if it is going to come up heads or tails. You want to not know what the results are going to be.

-- Peter Norvig, in an interview about being wrong. When I saw this, I thought it sounded a lot like entropy pruning in decision trees, where you don't even bother asking questions that won't make you update your probability estimates significantly. Then I remembered that Norvig was the co-author of the AI textbook that I had learned about decision trees from. Interesting interview.

6SilasBarta14y

Wow, I'm glad this kind of analysis is showing up in mainstream publications. Norvig is describing an important insight from information theory: the amount of information you get from learning something is equal to the log of the inverse of the probability you assigned to it (log 1/p). (This value is called the "surprisal" or "self-information".) So, always getting results you expect (i.e. put a high p on), means you're getting little information out of the experiments, and you should be doing ones where you expect the result to be less probable. Therefore, to have a good experiment, you want to maximize the "expected surprisal" (i.e. sum over p * log(1/p)), which is equivalent to the entropy, and probably the basis for the method you mention.

1Oscar_Cunningham14y

Is LW broken for everyone? ETA: When I wrote this, the "Comments" page was one of the few I could access, hence it being posted in such a strange place.

5Richard_Kennaway14y

No, only for the ones who can't reply to your comment.

0NancyLebovitz14y

It's broken for me, at least. Instead of the old-fashioned way of breaking, with new comments around a certain time not being visible, I can't get to older comments or posts. I don't know what the extent of the loss of access is. The report issues page (link at bottom of page) is read only. There was going to be a :"brief outage" for that page at 7AM PDT. That would be almost 7 hours ago. I think the stock of humorous messages at the "you tried a link which isn't working" has been improved.

0NancyLebovitz14y

The last line should have been I think the stock of humorous messages at the "you tried a link which isn't working" page has been improved. Or to put it another way, the permalink which would normally make it possible for me to edit isn't working either.

0NancyLebovitz14y

And, though this is minor, if I hit the comment button, it seems as though nothing has happened. However, if I refresh the comments page, my comment shows up. As a minor mercy, hitting the comment button more than once doesn't seem to produce duplicate comments.

0[anonymous]14y

It is for me.

0cupholder14y

The tree metaphor reminds me of this...

[-]Rain14y240

A great many people think they are thinking when they are merely rearranging their prejudices.

-- William James

[-]ata14y200

Farnsworth: My god, is it possible?
Fry: It must be possible. It's happening.

— Futurama: "The Late Philip J. Fry"

[-]Richard_Kennaway14y180

Man cannot understand the perfection and imperfections of his chosen art if he cannot see the value in other arts. Following rules only permits development up to a point in technique and then the student and artist has to learn more and seek further. It makes sense to study other arts as well as those of strategy. Who has not learned something more about themselves by watching the activities of others? To learn the sword study the guitar. To learn the fist study commerce. To just study the sword will make you narrow-minded and will not permit you to grow outward.

-- Musashi, "A Book of Five Rings"

3MichaelVassar14y

Pretty much the opposite of the foundation of modern education and social organization.

[-]simplicio14y150

On rationalization, aka the giant sucking cognitive black hole.

Though [Ben Franklin] had been a vegetarian on principle, on one long sea crossing the men were grilling fish, and his mouth started watering:

I balanc'd some time between principle and inclination, till I recollectd that, when the fish were opened, I saw smaller fish taken out of their stomachs; then thought I, "if you eat one another, I don't see why we mayn't eat you." So I din'd upon cod very heartily, and continued to eat with other people, returning only now and then occasionally to a vegetable diet.

Franklin concluded: "So convenient a thing it is to be a reasonable creature, since it enables one to find or make a reason for every thing one has a mind to do."

-Jonathan Haidt, "The Happiness Hypothesis"

5wedrifid14y

I really like the quote about cod but I'm not particularly inspired by the moral given for the story. I'd prefer "I eliminated a non-terminal ethical principal when I realised my thinking was pretentious bullshit, moving towards a more coherent ethical framework. Yay me!"

[-]simplicio14y130

I noticed that too; of course not eating fish is an ethical non-issue given how much other low-hanging consequentialist fruit there is.

However, note that his justification for his change of heart is pure rationalization. Whatever good reasons there might be for eating fish, or for abandoning vegetarianism, "they eat each other" is a bad one, a confabulation.

Fish and other animals are not capable of reflecting ethically on their actions, so they are ethically blameless for whatever they do. That does not mean their suffering doesn't count. Franklin knew that.

[-]SilasBarta14y130

I know I'm bringing Drescher up a lot recently, but this exchange reminds me of some of his points, and how, after reading Good and Real, I see Haidt's work (among other people's) in a different light.

Drescher's theory of ethics and decision making is, "You should do what you [self-interestedly] wish all similarly situated beings would do" on the basis that "if you would regard it as the optimal thing to do, then-counterfactually they would too".

He claims it implies you should cast a wide net in terms of which beings you grant moral status, but not too wide: you draw the line at beings that don't make choices (in the sense of evaluating alternatives and picking one for the sake of a goal), as that breaks a critical symmetry between you and them.

Taking your premise that fish don't reflect on their actions, this account would claim that they likewise do not have the moral status of humans. But it would also agree with you that it's insufficient to point to how they eat each other, because "I would not want some superbeing to eat me simply on the basis that I eat less intelligent beings."

Also, Drescher accounts for our moral intuitions by saying that the... (read more)

2simplicio14y

I've got Good and Real on hold at the library. :) Currently working through Cialdini's Influence, muahaha... This sounds to me like a modernized version of Kantian deontology... interesting. Where I really trip up with this argument is in the 'granting moral status' step. What does it mean if I decide to say 'a fish has no moral status?' Let's do a reductio. Say fish have no moral status. Does that mean it's permissible to torture them, say by superstimulating pain centres in their brains? I don't think so, even if the torture achieved some small useful end. I don't think suffering should be taken out of the equation in favour of symmetries. The latter have no obvious moral weight.

1SilasBarta14y

I don't have a good answer for the rest of your comment, but I can answer this: Drescher does a good job of making sure that nothing depends on choice of terminology. In this case, "a fish has no moral status" cashes out to "I should not count a fish's disutility/pain/etc. against the optimality of actions I am considering." You can take "should" to mean anything under Drescher's account, and, as long as you're consistent with its usage, it has non-absurd implications. Under common parlance, you can take "should" to mean "the action that I will choose" or "the action I regard as optimal". Then, you can see how this sense of the term applies: "If I would regard it as optimal to kill weaker beings, then-counterfactually beings who are stronger than me would regard it as optimal to kill me, to the extent that their relation to me mirrors my relation to the weaker beings under consideration." I didn't give a full exposition of how exactly you apply such reasoning to fish, but under this account, you would need to look at what is counterfactually entailed by your reasoning to cause pain to fish.

2wedrifid14y

No, that isn't implied. There are all sorts of coherent value systems which make ethical distinctions between killing things that kill other things and killing things that don't kill other things. Maybe Franklin was confabulating, but again, that moral does not inspire me. In most cases the reasoning is sound and does move the values a step towards coherency. There is a difference between dastardly rationalisation and updating your ethical position by eliminating obviously poor thinking. A lot of people are good at not reflecting ethically too, and it does help them get away with stuff (via more effective signalling). This is not a feature of the universe over which I rejoice and nor is it one that I encourage via my ethical signalling.

2Richard_Kennaway14y

His comment on the matter suggests he thought he was. The context does not record whether he returned to vegetarianism once away from the temptation.

2wedrifid14y

Yes. Hence the lack of inspiration. It's the same old moral: "Thoughts and ethical intuitions are enemies. Ethical intuitions are good and you should follow them. Thinking your ethics through is bad. Submit to the will of the tribe!" I say if subjecting your ethical intuitions to rational analysis doesn't lead you to change them in some way then you are probably doing it wrong.

3Oligopsony14y

How subject ethical intuitions should be to rational analysis (in the sense of being changed by them) depends on how much you endorse the fact-value distinction and how fundamental the intuition is. Reason leads me (though perhaps my reasoning is flawed) to conclude that "others' abject suffering is bad" isn't any more justified a desire than "others' abject suffering is good;" they're as equivalent as a preference for chocolate or vanilla ice cream. But so what? I don't abandon my preference for vanilla just because it doesn't follow from reason. Morality works the same way, except that ideally, I care about it enough to force my preferences on others.

3wedrifid14y

Yes. It is non-terminal ethical intuitions that I expect to be updated. "Should not do X because Y" should be discarded when it becomes obvious that Y is bullshit.

[-]Rain14y150

Read, every day, something no one else is reading. Think, every day, something no one else is thinking. Do, every day, something no one else would be silly enough to do. It is bad for the mind to be always part of unanimity.

-- Christopher Morley

[-]Randaly14y140

"A joke told by Warren Buffett comes to mind: a patient, after hearing from a doctor that he has cancer, tells the doctor, “Doc, I don’t have enough money for the surgery, but maybe could I pay you to touch up the x-ray?” Hope and self-deception are not a strategy."

~ Vitaliy Katsenelson

[-]RobinZ14y130

The most difficult subjects can be explained to the most slow-witted man if he has not formed any idea of them already; but the simplest thing cannot be made clear to the most intelligent man if he is firmly persuaded that he knows already, without a shadow of doubt, what is laid before him.

- Leo Tolstoy, The Kingdom of God is Within You (1894), ch. III

2wedrifid14y

You know, I'm going to remember that one and try to remind myself of it whenever I notice my status is interfering with my thoughts.

-5[anonymous]14y

[-]Baughn14y120

On the nature of ethics:

There is no justice in the laws of Nature, Headmaster, no term for fairness in the equations of motion. The universe is neither evil, nor good, it simply does not care. The stars don't care, or the Sun, or the sky. But they don't have to! We care! There is light in the world, and it is us.

~ Eliezer Yudkowsky c/o Harry Potter, Methods of Rationality chapter 39.

1jsnider32y

This is too much of a bare assertion to be a good rationality quote.

[-]Airedale14y110

I entered the Lager (Auschwitz) as a non-believer, and as a non-believer I was liberated and have lived to this day. Actually, the experience of the Lager with its frightful iniquity confirmed me in my nonbelief. It has prevented me, and still prevents me, from conceiving of any form of providence or transcendent justice. . . . I must nevertheless admit that I experienced (and again only once) the temptation to yield, to seek refuge in prayer. This happened in October 1944, in the one moment in which I lucidly perceived the imminence of death . . . naked and compressed among my naked companions with my personal index card in hand, I was waiting to file past the “commission” that with one glance would decide whether I should go immediately into the gas chamber or was instead strong enough to go on working. For one instance I felt the need to ask for help and asylum; then, despite my anguish, equanimity prevailed; one does not change the rules of the game at the end of the match, nor when you are losing. A prayer under these conditions would have been not only absurd (what rights could I claim? and from whom?) but blasphemous, obscene, laden with the greatest impiety of which a nonbeliever is capable. I rejected the temptation; I knew that otherwise were I to survive, I would have to be ashamed of it

-- Primo Levi, The Drowned and the Saved (quoted by Damon Linker at The New Republic ; h/t Andrew Sullivan)

[-]orthonormal14y110

In short, whatever emotional impulse we may have toward altruism and empathy, and to whatever extent it may be genetically hardwired, it does not obviate the need for explicit judgments about right and wrong. If it did not seem correct to act with kindness and fairness, even at a net personal cost—if there were no sensible reason for so acting, beyond a raw impulse to do so—then we would have reason to regard the raw impulse as pointlessly self-destructive—like a disposition to alcoholism or a purely visceral (so to speak) aversion to surgery—and we would have a reason to attempt to overcome it.

Gary Drescher, Good and Real

[-]Rain14y110

Your imaginings can have as much power over you as your reality, or even more.

-- Charles T. Tart

[-]wedrifid14y100

"There are worse things than seeming irresponsible. Losing, for example."

~ Paul Graham

[-]NihilCredo14y90

Therefore, after intellectually evaluating your problems through common sense and drawing on what psychiatry has taught us, if you still cannot emotionally release yourself from unwarranted guilt, and put your theories into action, then you should learn to make your guilt work for you. You should act upon your natural instincts, and then, if you cannot perform without feeling guilty, revel in your guilt. This may sound like a contradiction in terms, but if you will think about it, guilt can often add a fillip to the senses. Adults would do well to take a

... (read more)

6mkehrt14y

Completely out of curiosity, why do you cite him by his birth name rather than his pen name of Anton Szander LaVey?

2NihilCredo14y

As a token of obligatory disrespect towards the juvenile pomposity of adopting a spoooky name for oneself.

[-]DSimon14y150

And "NihilCredo" isn't a spooky adopted name? :-)

2NihilCredo14y

Heh, good quip! :) (the difference is obvious, anyway.)

1CronoDAS14y

I know this very well. Video games get so much more appealing when I have some work to avoid.

5steven046114y

It could also be a selection effect -- if you're doing something even though you're not supposed to, it must be something you really like. Similarly, unhealthy food tastes good because substances that are neither healthy nor tasty aren't classified as food.

2pjeby14y

Or an evolutionary nudge in the direction of hypocrisy. More precisely, if the tribe has bothered to prohibit it, it must be something that advantages you at someone else's (possibly the group's) expense, and therefore you should be: a) Encouraged to do it more, while... b) Exhibiting plausible self-punishment when caught, to maintain group ties. The former explains the increased reward, the latter the social behavior aspect.

6CronoDAS14y

It's specifically the "I have something unpleasant to do, and I'm not doing it!" that's exciting. I'll play even a bad game when I have work to avoid.

[-]XiXiDu14y80

You use a metaphor to describe some concept. The metaphor isn’t the thing you describe - it’s just a tool that you use. But someone takes the metaphor, and runs with it, making arguments that are built entirely on metaphor, but which bear no relation to the real underlying concept. And they believe that whatever conclusions they draw from the metaphor must, therefore, apply to the original concept.

— Mark Chu-Carroll, Metaphorical Crankery: a bad metaphor is like a steaming pile of …

[-]XiXiDu14y70

A little bit of knowledge is a dangerous thing. It can convince you that an argument this idiotic and this sloppy is actually profound. It can convince you to publicly make a raging jackass out of yourself, by rambling on and on, based on a stupid misunderstanding of a simplified, informal, intuitive description of something complex.

— Mark Chu-Carroll, The Danger When You Don’t Know What You Don’t Know : Good Math, Bad Math

[-]WrongBot14y60

I'll believe anything, no matter how wild and ridiculous, if there is evidence for it.

-- Isaac Asimov

[-]Nic_Smith14y60

"Someday I want to be so powerful that I can defeat myself in a single blow." -- Silence, commenting on a character in Prism Ark that says they want to be stronger.

5Will_Newsome14y

I don't understand. Anybody who lives near a cliff or owns a gun has that much power. Explain? What is the connection to rationality?

2[anonymous]14y

That's the best summary of rationality I've ever seen.

3wedrifid14y

Wouldn't a good summary of rationality be a little more comprehensible? I'm sure there is a deep insight that Nic sees behind the quote and from what you say it must be related to rationality but at face value, without context that most people, myself included, don't have it makes no sense.

3Nic_Smith14y

Wow, I had no idea that this quote would be so controversial. I will attempt to explain. To begin, I think there is something that can be learned from the quote, but I wouldn't call it a "deep insight." Or, at least, I wouldn't have before uninverted's reply. Silence, a friend of mine, was making fun of the focus of many anime on literal, physical strength, often detached from anything to protect or, indeed, any real overall goal at all (I don't remember if this is the case with Prism Ark or not). It's thus a light warning against focusing too much on method over end results. Also, if there were two of you, it'd be impossible for you[a] to be decisively stronger and more skilled than you[b]. Uninverted, I'm guessing, was thinking of akrasia and overcoming cognitive biases, which can be considered "part of self" and, at the same time, something to "defeat" as easily as possible. Hence, the quote can also be read as a summary of rationality.

1[anonymous]14y

Yes, that's what I thought exactly.

-1sketerpot14y

With a big enough hammer, that goal is within the reach of anybody strong enough to swing it at their own head.

[-]Cyan14y60

Sometimes, I feel the fear of uncertainty stinging clear
And I can't help but ask myself how much I let the fear
Take the wheel and steer
It's driven me before
And it seems to have a vague, haunting mass appeal
But lately I'm beginning to find that I
Should be the one behind the wheel

- Incubus, Drive

2Matt_Simpson14y

Wow... I hadn't noticed that before

1Cyan14y

I'd quote the whole song, but I think that might violate a copyright or something.

0Matt_Simpson14y

I doubt it, but there are several sites with the full lyrics if anyone wants to see them. Edit 24 July 2013: Removed link to lyricsbay at their request. – admin

0gwern14y

[-]knb14y50

On the subject of noble lies:

No: at all costs, it is to be prayed by all men that shams may cease. Good heavens, to what depths have we got, when this to many a man seems strange! Yet strange to many a man it does seem; and to many a solid Englishman, wholesomely digesting his pudding among what are called the cultivated classes, it seems strange exceedingly; a mad ignorant notion, quite heterodox, and big with mere ruin. He has been used to decent forms long since fallen empty of meaning, to plausible modes, solemnities grown ceremonial,--what you in yo

... (read more)

[-]SilasBarta14y50

I had just recently seen the two Christopher Nolan Batman movies (Batman Begins and The Dark Knight). Here are my favorite quotes; please add any that you like. (Character attribution left off to prevent pre-judgment.) Wikiquote lists.

"It's not who you are on the inside; it’s what you do that defines you." (Compare: Functionalism, substrate independence, timeless identity – I know, that’s probably not what was intended, but note how it’s used in other contexts.)

"Criminals thrive on the indulgence of society's understanding." (Compare ... (read more)

[-]Morendil14y140

Nolan's Memento is also interesting from a rationalist perspective - it gives "running on untrusted hardware" a quite concrete meaning.

6NancyLebovitz14y

It's a complicated issue-- as nearly as I can tell, the people who argue for no understanding assume that they can just use their intuitions about punishment, and not update about whether they're getting the effects they want.

2SilasBarta14y

I agree. I wasn't trying to argue in favor of some kind of unlimited punishment, or against all understanding whatsoever -- just that this kind of understanding can be misused, especially when you discount the offense for being in the past. (I had recently read Drescher's account in Good and Real of why the pastward, inalterable aspect of a transgression, and the fact that the punishment only causes things in the future, are no reason not to punish.) Edit3: And, of course [rot13] vg jnf n onq thl jub fnvq gung va gur zbivr, naq gur guvatf lbh jnea nobhg ner unaqyrq va gur zbivr, nf gung punenpgre'f ernfbavat yrnqf uvz gb qb ubeevoyr guvatf ba gung onfvf, yvxr gel gb qrfgebl na ragver pvgl. Ng gur fnzr gvzr, V guvax ur qbrf unir n cbvag.

[-]mikerpiker14y50

The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents. We live on a placid island of ignorance in the midst of black seas of infinity, and it was not meant that we should voyage far. The sciences, each straining in its own direction, have hitherto harmed us little; but some day the piecing together of dissociated knowledge will open up such terrifying vistas of reality, and of our frightful position therein, that we shall either go mad from the revelation or flee from the deadly light into the peace and safety of a new dark age.

-H.P. Lovecraft

[-]DSimon14y110

Sounds like Caveman Science Fiction to me. "Why should we risk learning about new things, when there's a possibility they'll be scary?"

9AlanCrowe14y

This belongs on the parody site http://morewrong.com. Please build it :-)

[-]Scott Alexander14y190

I don't know; the more Less Wrong I read, the more I start to think Lovecraft was on to something.

Delving too far in our search for knowledge is likely to awaken vast godlike forces which are neither benevolent nor malevolent but horrifyingly indifferent to humanity. Some of these forces may be slightly better or worse than others, but all of them could and would swat our civilization away like a mosquito. Such forces may already control other star systems.

The only defense against such abominations is to study the arcane knowledge involved in summoning or banishing these entities; however, such knowledge is likely to cause its students permanent psychological damage or doom them to eternities of torture.

5Larks14y

We've got Harry Potter and the Methods of Rationality; maybe you should write Cthulhu Mythos and Rationality ? Then again, it might be unwise to disseminate it openly.

[-]steven046114y160

At the Mountains of Sanity

3gwern14y

I've always enjoyed Vernor Vinge's name for AI: "Applied Theology". (In, I think, A Fire upon the Deep.)

0Jayson_Virissimo11y

"Theological engineering" has a nice ring to it.

8Kyre14y

I never read Lovecraft as being any kind of metaphor for the real world, so I wouldn't vote this up as a rationalist quote for that reason. But I like it as a device used Lovecraft to try to convey a sheer magnitude of horror. Can you imagine discovering something so horrific you wished you could delete the whole thing from your memory ? The more you pride yourself as a rationalist, the more horrific it would have to be.

0toto14y

This seems to be the premise of Isaac Asimov's "Nightfall".

[-]WrongBot14y50

No man is an island entire of itself; every man is a piece of the continent, a part of the main; if a clod be washed away by the sea, Europe is the less, as well as if a promontory were, as well as any manner of thy friends or of thine own were; any man's death diminishes me, because I am involved in mankind. And therefore never send to know for whom the bell tolls; it tolls for thee.

-- John Donne

4Will_Newsome14y

Any man's death diminishes me, because their agenthood and qualia are probabilistically similar to mine, and it would not have taken many counterfactual changes for me to be not at all unlike that man, compared to a babyeater, whose pain bothers me less. Thus altruism is found in the egoist, to an extent.

7ata14y

Getting from "other people's minds are probably similar to mine" to "I care about other people's minds" still requires some implicit premises or some psychological features beyond egoism (e.g. empathy).

3Will_Newsome14y

Specifically it requires "I care about myself" with saner boundaries around what counts as 'self' as the implicit premise. Added: Any specific reason for the downvote?

0ShardPhoenix14y

Evolution (of genes and/or memes) is probably sufficient to generate this.

-1[anonymous]14y

That's a better response to Will's post than to the Donne quote. Donne only states that all humans influence each other's existence to some minimal degree, butterfly-effect-style (I would object, incidentally, that such influence may not always be desirable); it's Will that brings up the similarities between humans as his moral foundation.

1ata14y

Indeed. But my comment was a reply to Will's.

0NihilCredo14y

Mine eyes, they deceive me! Deleted.

[-]WrongBot14y220

Please don't delete comments. It makes it hard to understand orphaned replies. Adding an [Edit: Withdrawn] at the end of the comment serves the same purpose, but maintains conversational continuity.

1WrongBot14y

What are qualia?

3ata14y

Are you unfamiliar with the term, or are you asking him to demonstrate that he understands it well enough to permit his using it as a quiet assumption in making a point?

6gwern14y

WrongBot has been around enough that one can safely assume that his ignorance is Socratic.

2WrongBot14y

I am familiar with the term, but I don't seem to have any. And no one's been able to tell me what they do, so I like to ask when they come up so that maybe someday I'll find out.

9orthonormal14y

I don't believe in qualia as a real entity, but when people talk about them they're referring to a genuine phenomenon which you also experience: that your conscious understanding of the experience of perception is only the merest shadow of the perception itself. Seeing red doesn't mean seeing something with a little XML "red" tag attached, but something much more complicated that happens beyond your conscious introspection. You can imagine the state of having switched that "red" experience with the "green" experience, in all your memories as well as in current perception, and still instantly knowing that the switch had occurred. This phenomenon is not an illusion, just a blind spot of conscious knowledge which happens to confuse the hell out of naive philosophers.

2SilasBarta14y

Thank you, well said! I've seen people go so far in dissolving qualia that they think they have to deny their own conscious experience, or think the confusion is extinguished as soon as you have the terminology nailed down.

1WrongBot14y

Of course. If I had perfect knowledge of my brain's functioning, now that would be a very strange thing indeed. No, I can't. If all my memories had been altered to agree with my newly-altered perception system, what difference would I detect? How would I detect it? Different from what?

7orthonormal14y

The hypothetical situation I mean is one where your current retina is reprogrammed to switch red and green stimuli, and your memories are edited so that you don't figure it out from inconsistencies, but everything else is left the same. The fact that there's subconscious cognitive content to red vs. green can be deduced from things like instinctive reactions† to the sight of blood: the brain doesn't check the color against the memory of other blood, it reacts faster than that to to perception. The emotional valence of colors would seem off somehow after a switch, because those don't appear to operate fully through memory, either. Snap judgments of peoples' attractiveness would backfire as your subconscious applied the rule "green tint means sickly" to someone with a healthy complexion. I don't think you'd be able to consciously articulate what exactly seemed "red" about that green grass, but parts of your mind would be telling you that something's gone wrong, because they're hooked up not just to labels "red" and "green" but to full systems of processing that would be running on suddenly different stimuli. †Similarly, chimps raised by humans in captivity will still freak out when exposed to a fake snake, because certain patterns have been encoded deep within. There's no reason for such patterns to be raised to the level of conscious knowledge.

7WrongBot14y

Ahhh, so you'd only be reprogramming part of my brain. Well, of course I'd run into problems then. All that means is that there are more parts of my brain than those I have conscious access to, which seems pretty obvious to me even before I start to think about what I know of neurology. I think we agree with each other.

0ChristianKl14y

I wouldn't be sure, the vision system has an amazing ability to adept to rewiring. Monkeys were able to see another color through gene therapy that their species hadn't seen before.

2orthonormal14y

1. Indeed there's rewiring over time, but it wouldn't be instant and it wouldn't be total, so the point stands. 2. That's a really interesting experiment-- can you find me a link?

4ChristianKl14y

http://www.wired.com/wiredscience/2009/09/colortherapy/

0orthonormal14y

Thanks!

0billswift14y

Most philosophical definitions are pretty weird. On the rare occasions I use "qualia", it means the inside view of your sense perceptions. What things look/sound/feel like to the person doing the sensing.

2Randaly14y

The subjective way we experience things.

0WrongBot14y

What do you mean by "experience"? And "subjective"? I'm not sure what you're talking about.

2Randaly14y

By experience I mean anything which we detect with one of our senses. The subjective part is, IMHO, the key to qualia. Suppose that you've never seen red light, and that you are then told all of its properties in perfect detail. You would still gain new information by actually seeing red light, because you still don't know "what it feels like" to see it. The qualia is not the objective facts, but rather what seeing the light "feels like", your perception of the effect produced on your brain by the light. (Qualia are usually taken to be an argument against materialism- because after you know every objective fact about something, you still gain new information (qualia) by experiencing it.)

[-]orthonormal14y130

This "Mary's Room" argument, like the "Chinese Room" argument†, contains a subtle sleight of hand.

On the one hand, for the learning to be about just the qualia rather than about externally observable features of vision processing, the subject would need to learn immensely more than the physical properties of red light. (The standard version of Mary's Room does so, postulating Mary to also deeply understand her own visual cortex and the changes it would undergo upon being exposed to that color.) In fact, the depth of conscious theoretical understanding that this would require is far beyond any human being, and it's wrong and silly to naively map our mind-states onto those of such a mind.

On the other hand, it plays on the everyday intuition that if I've never seen the color red, but have been given a short list of facts about it and am consciously representing my limited intuition for that set of facts, that doesn't add up to the experience of seeing red.

The equivocation consists of thinking that a superhuman level of detailed understanding of (and capability to predict) the human brain can be analogized to that everyday intuition, rather than being unimaginably o... (read more)

3red7514y

Intuitions don't matter. If Mary can't activate her neural pathways participating in creation of experience of seeing red, then she has no means of knowing how she will experience redness. All models she can create in her mind will be external to her as the mind created by actions of human being in Chinese room is external to that human being. It is not only conscious understanding that is required, we will need a conscious control of individual neurons and synapses to be able to experience qualia given just a description of it. For example, to be able to name color and imagine color given its name, Mary (roughly speaking) should manually connect neurons in her visual cortex to the neurons in her Broca's area and to the neurons in her auditory cortex. So I think that, contrary to Dennet, Mary will get new information when she will see colors, as human's brain construction doesn't allow to acquire that information by other means. Thus in a sense human's qualia cannot be reduced.

5Blueberry14y

You may be interested in this paper which makes a similar argument.

1red7514y

Thanks. It is identical argument modulo my inability to make all reasoning and premises sufficiently transparent.

5thomblake14y

That contradicts one of the assumptions in the thought experiment. You're establishing qualia as a physical property; in that case, "what it feels like to see red" is amongst the things Mary knows about, by hypothesis. Also, if it just comes down to activating those neurons, then Mary knows that too and can perform an experiment to activate those neurons without having a 'red thing' in front of her, using her incredible superhuman intelligence and resources.

1red7514y

I am not establishing qualia as physical properties of brain's activity, I think of them as descriptions of specific neural activity in the terms of human's self-model. And limitations of that self-model (it's not sufficiently detailed to refer to individual neurons) don't allow to establish unambiguous correspondence between physical description of brain and self-model description of brain within that self-model. And what is a difference between seeing red thing and activation of those neurons? The point of "Mary's room" is to know what seeing red means without actually seeing it.

0thomblake14y

Depends who's using it. For Dennett, for instance, the point of Mary's room is to point out how ridiculous this notion of qualia is, or at least how silly the thought experiment is. As stated, she knows everything physical about red. So she knows, for instance, how to build a machine that will activate her red-seeing neurons in the absence of the color. Also as stated, she can perform whatever experiments she needs to in order to become an expert color scientist. So she can have whatever experience would come from having those neurons activated. If you think there's nothing else to the experience, then I think we're in agreement so far.

0SilasBarta14y

So we have no intuition for that understanding level's qualia? ;-)

0orthonormal14y

Yeah, I realized the unintended recursion there, and have edited accordingly...

7WrongBot14y

Well, of course a verbal description of red light is different from seeing red light. One is an auditory stimulus, and one is a visual stimulus. They do different things to my neurons. Are qualia about something other than neurons?

4JanetK14y

I voted up WrongBot's redefinition of his question, 'are qualia about something other than neurons?'. Is qualia something other than a word that has been awkwardly defined? Why is colour always the example used to illustrate qualia? Is there something different between colour and things like position, texture, pitch? Has anyone thought of a better or even different way to experience our experiences?

2SilasBarta14y

Readers may be interested in my approach to the problem, or rather, the problem that remains even after any terminology issues are settled. Summary: What we identify as "qualia" is the encoding of memories that we cannot yet compare directly between people, to the extent we can't compare them. This incommensurability can easily arise among agents who are similar, but who self-modify in a way that does not place any priority on the ability to directly transfer memories to other agents. In that case, their methods of storing memories are ad-hoc, and look like garbage to each other -- but with the right assumptions and interaction, they can achieve a limited ability to compare, and thereby have terminology like "red" that means something to all agents, even as it doesn't call up exactly the same idea for each one.

1mkehrt14y

I was intrigued when I first read this when you last posted it, and I thought about it for a while. The problem with it, it seems to me, is that this is a good explanation for why qualia are ineffable, but it doesn't seem to be come any close to explaining what they are or how they arise. So, I could imagine a world (it may even be this one!) where people's brains happen to be organized similarly enough that two people really could transfer qualia between them, but this still doesn't explain anything about them.

1SilasBarta14y

You're right. But I believe that that the ineffable aspect is closely related to the other two questions, although I don't have an answer in the same detail as the ineffability question (which would still be progress!). To give a sketch of what I have in mind, my best explanation is this: conscious minds form when a subsystem is able to screen itself off from the entropizing forces of the environment (similar in kind to a refrigerator or other control system). This necessarily decouples it from the patterns that exist in the environment, as well as other minds that have done the same. So the formation of a conscious mind will coincide with the formation of incompatible encoding methods, unless special care is taken to ensure that the encoding protcols are the same. Therefore, we shouldn't be surprised to notice that, "hey, everything that's conscious, also has ineffable experiences with the other conscious things." But again, I don't claim this part is as well-developed or thought-out.

0LucasSloan14y

What is it that you feel/see/touch/taste/think/etc. instead of simply acting? Why is there a "you" you experience, instead of mere rote action? We label these sorts of things that we use to distinguish between empty existence and our own subjective (personally observed/felt) experience. The thing about humans that distinguish them from P-zombies.

[-]WrongBot14y120

What is it that you feel/see/touch/taste/think/etc. instead of simply acting?

Why do you group together sense perceptions (which I have) with thoughts (which I have), and call them qualia (which I don't have)?

Why is there a "you" you experience, instead of mere rote action?

How are these different?

We label these sorts of things that we use to distinguish between empty existence and our own subjective (personally observed/felt) experience.

How can existence be "empty"? Is subjective experience just sense perception? Because sense perception doesn't seem like it warrants all this mysteriousness.

The thing about humans that distinguish them from P-zombies.

That's odd. I thought the sequence on P-zombies made it pretty clear that they don't exist. Why do we need to be distinguished from confused, impossible thought experiments?

5Richard_Kennaway14y

Perhaps you simply do not have qualia or subjective experience. Some people do not have visual mental imagery, strange though that may seem to those of us who do. Similarly, maybe some people do not have anything they are moved to describe as subjective experience. Such people, if they exist, are the opposite of the logically absurd p-zombies. P-zombies falsely claim that they do have these things; people without them truthfully claim that they do not. You might just be Socratically role-playing, but even so, there may be other people who actually do not have these things. That is, they would express puzzlement at talk about "the redness of red", "awareness of one's own self", and so forth (and without having been tutored into such puzzlement by philosophers arguing that they cannot be experiencing what in fact they do experience). Is there anyone here who does experience that puzzlement, even before knowing anything of the philosophical controversy around the subject?

2Emile14y

There is this example: I see that as a cheap way out, I think "do I have free will ?" is just a confused question whose answer depends of the way you unconfuse it. I'm just in the minority of humans who refuses to answer that confused question - I'd like to say I refuse to answer all confused questions, but that's probably not true. Still, it is possible that confusion and disagreement about "qualia" and "free will" are just due to differences in personal experience, not to different interpretation of those labels.

1WrongBot14y

People who lack visual mental imagery have atypical performance on certain kinds of cognitive tests, as Yvain's article describes, and if I believed that such people existed, I would expect that testable difference. What type of test should I expect to distinguish between those who have qualia and those who do not?

4Richard_Kennaway14y

The most direct test would be this: "Do you have qualia?" * Yes * No But you'd have to use naive subjects who haven't philosophised themselves into ignoring their own experience. A little more indirectly, people without qualia would profess puzzlement at the very idea, and argue that there is no such thing. If they are philosophers, they will write articles on the incoherence of the concept. If they are psychologists, they will practice psychology on the basis that mental phenomena do not exist. If they are teachers, they will see the brain as a pot to be filled, not the mind as a fire to be ignited. Those who do have qualia will be as tenacious on the other side. Nothing that those who do have qualia say about qualia will make sense to those who don't, and those who don't will have no difficulty in demonstrating that it is nonsense. Those who do have qualia will be unable to explain them even to each other, since they know no more about what they are than they know about how thought happens. All of their supposed explanations will only be disguised descriptions of what it feels like to have them. Looks pretty much like our world, doesn't it?

5Morendil14y

How is that direct? First you'd have to explain what you mean by that, and "understanding" such an explanation would pretty much require convincing oneself that there are such things to be had in the first place. There are some things you can test by asking; I can imagine asking someone, "do you ever get a twisty kind of feeling in your stomach or nearby, when you've just had something very bad happen to you and it slipped your mind for a while but then it intrudes again on your awareness - and the twisty feeling comes precisely at that moment". That's a feeling. It's describable. I have it sometimes. It's an empirical matter whether other people recognize an experience of theirs in that description, or not. It's much like pointing to a red thing and asking people "is this red", and then they confirm that it's red to them. How are "qualia" different?

1Richard_Kennaway14y

Failing to understand would amount to a "No".

1byrnema14y

It does. If psychologists came out with a study that 1 out of 10 people don't experience qualia, I would feel rather certain that I was one of those out of 10 that don't experience it. Just like WrongBot, I think. However, my actual expectation is that we are all the same at that level of brain organization, and wonder what aspect of my experience people are labeling 'qualia'. Above, Orthonormal wrote, Actually, this is exactly what I hypothesized qualia were: little reference tags of meaning that we attach to things we recognize. When using an entirely new medium, I feel like I experience the creation of new qualia. For example, here on Less Wrong, each comment has a username. After some experience on Less Wrong, it feels like the username is different from (and more than) a set of green underlined letters in bold font at the upper right hand corner that tells you the person who wrote the comment-- it's like a separate object that means the source of the comment, and as soon as it has that extra meaning, it gains this elusive quale-like aspect.

2Richard_Kennaway14y

I have an opposite hunch: that the further removed any part of our internal constitution is from the world outside our skins, the more we vary. My reason is that there are many ways of doing the right thing to survive and reproduce. The genome isn't big enough to contain a blueprint for a whole brain, so evolution has come up with a general mechanism (which no-one actually knows anything about yet) for the whole thing to organise itself when the newborn is dropped into an unknown environment. The organisation an individual brain ends up with is constrained by nothing more than the requirement to make the organism function in that environment. Look around you at the variation in people's personalities. They're even more different inside their heads than that.

[-]cousin_it14y40

“The greatest obstacle to discovering the shape of the earth, the continents and the ocean was not ignorance but the illusion of knowledge.” -- Daniel Boorstin

[-]Thomas14y40

If I understand it, I can build it.

Richard Feynman's inverted (not A implies not B == B implies A) quote.

1Vladimir_Nesov14y

Dualized.

[-]Richard_Kennaway14y30

Dear Miss Manners: How is a hat correctly worn?

Gentle Reader: Same as always; on the head.

-- Judith Martin ("Miss Manners")