Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Rationality quotes: August 2010

5 Post author: Cyan 03 August 2010 12:16AM

This is our monthly thread for collecting these little gems and pearls of wisdom, rationality-related quotes you've seen recently, or had stored in your quotesfile for ages, and which might be handy to link to in one of our discussions.

  • Please post all quotes separately, so that they can be voted up/down separately.  (If they are strongly related, reply to your own comments.  If strongly ordered, then go ahead and post them together.)
  • Do not quote yourself.
  • Do not quote comments/posts on LW/OB.
  • No more than 5 quotes per person per monthly thread, please.

Comments (201)

Comment author: DSimon 03 August 2010 03:27:01AM 27 points [-]

My hotel doesn't have a 13th floor because of superstition, but people on the 14th floor, you should know what floor you're really on. If you jump out the window, you will die sooner than you expect.

-- Mitch Hedberg (Quoted from memory)

Comment author: djcb 03 August 2010 09:33:48PM 11 points [-]

Superstition produces bad luck

-- Anonymous

Comment author: Apprentice 03 August 2010 01:15:30PM *  23 points [-]

Upon his death man must leave everything behind ... and depart forever from the world he has known. He must of necessity go to that foul land of death, a fact which makes death the most sorrowful of all events. ... Some foreign doctrines, however, teach that death should not be regarded as profoundly sorrowful. ... These are all gross deceptions contrary to human sentiment and fundamental truths. Not to be happy over happy events, not to be saddened by sorrowful events, not to show surprise at astonishing events, in a word, to consider it proper not to be moved by whatever happens, are all foreign types of deception and falsehood. They are contrary to human nature and extremely repugnant to me.

-- Motoori Norinaga (1730-1801) - quoted from Blocker, Japanese Philosophy, p. 109

Motoori was as far as you can get from being a rationalist but this quote was so Yudkowskian that I felt it belonged here.

Comment author: Rain 03 August 2010 12:56:46AM 127 points [-]

Personally, I've been hearing all my life about the Serious Philosophical Issues posed by life extension, and my attitude has always been that I'm willing to grapple with those issues for as many centuries as it takes.

-- Patrick Nielsen Hayden

Comment author: gwern 03 August 2010 04:23:00AM 13 points [-]

This is a good example of how some areas are most concisely dealt with by ridicule.

Comment author: knb 03 August 2010 05:51:03PM 1 point [-]

What are the Serious Philosophical Issues posed by life extension?

I can see many serious practical issues, but in what way should philosophical opinions change due to the mere extension of human life span?

Comment author: NihilCredo 03 August 2010 09:39:23PM *  6 points [-]

I conjecture that we're supposed to read it under the classical definition of "philosophy", which used to include pretty much every type of intellectual discussion, including such practical issues as how to properly raise children, how to organise a political society, etc.

Comment author: goldfish 04 August 2010 06:11:05PM 1 point [-]

There is a great ted lecture on this subject. I thought he did a good job at addressing the concerns. At least to the point of defending that research should continue to at least allow future generations to decide if they think it is acceptable.

http://www.ted.com/talks/aubrey_de_grey_says_we_can_avoid_aging.html

Comment author: Document 27 August 2011 10:05:58PM 0 points [-]

Is the source online? The first page of Google results seems to be just people quoting the line.

Comment author: Rain 27 August 2011 10:08:15PM *  2 points [-]

I tracked down the source as rec.arts.sf.fandom, 09 Aug 2000 (previous to any use as a quote), in a thread titled, "Do cats go to heaven?"

It's (sort of) available online in the Google Groups archive.

However, the provided link does not seem to work due to the huge length of the thread (it's trying to create a tree threaded discussion out of 4189 posts), but I was able to see several references to it inside, including the beginning as a snippet.

Comment author: Rain 24 June 2013 07:12:19PM 4 points [-]

Due to the new release of Google Groups, I was able to go through this thread and found the original reference on page 41 of the thread, posted 08 Aug 2000.

Comment author: scotherns 16 August 2010 10:36:09AM 0 points [-]

Is Rain's quote the most upvoted entry of all time? Its currently at +62.

Comment author: katydee 16 August 2010 10:37:28AM 0 points [-]

It's the most upvoted comment I've ever seen. I'm not sure about top-level posts, though.

Comment author: Rain 16 August 2010 02:01:35PM *  0 points [-]

The current first page of 'Recent Posts' shows 7 with a higher point total than 62; two posts are above 100 points, both by Yvain.

Comment author: Apprentice 16 August 2010 11:07:30AM 0 points [-]

It would be interesting to see a compilation of the most upvoted entries from all the quote threads.

Comment author: Rain 16 August 2010 01:58:08PM *  3 points [-]

DanielVarga created a top quote list, last updated in March.

Comment author: Kazuo_Thow 03 August 2010 03:56:18AM *  18 points [-]

... the history of mathematics is a history of horrendously difficult problems being solved by young people too ignorant to know that they were impossible.

-- Freeman Dyson, "Birds and Frogs"

Comment author: sketerpot 04 August 2010 07:10:17PM 17 points [-]

If you do experiments and you're always right, then you aren't getting enough information out of those experiments. You want your experiment to be like the flip of a coin: You have no idea if it is going to come up heads or tails. You want to not know what the results are going to be.

-- Peter Norvig, in an interview about being wrong. When I saw this, I thought it sounded a lot like entropy pruning in decision trees, where you don't even bother asking questions that won't make you update your probability estimates significantly. Then I remembered that Norvig was the co-author of the AI textbook that I had learned about decision trees from. Interesting interview.

Comment author: SilasBarta 04 August 2010 07:46:53PM 4 points [-]

Wow, I'm glad this kind of analysis is showing up in mainstream publications.

Norvig is describing an important insight from information theory: the amount of information you get from learning something is equal to the log of the inverse of the probability you assigned to it (log 1/p). (This value is called the "surprisal" or "self-information".)

So, always getting results you expect (i.e. put a high p on), means you're getting little information out of the experiments, and you should be doing ones where you expect the result to be less probable.

Therefore, to have a good experiment, you want to maximize the "expected surprisal" (i.e. sum over p * log(1/p)), which is equivalent to the entropy, and probably the basis for the method you mention.

Comment author: Oscar_Cunningham 04 August 2010 08:32:19PM *  1 point [-]

Is LW broken for everyone?

ETA: When I wrote this, the "Comments" page was one of the few I could access, hence it being posted in such a strange place.

Comment author: RichardKennaway 04 August 2010 09:26:07PM 5 points [-]

No, only for the ones who can't reply to your comment.

Comment author: NancyLebovitz 04 August 2010 08:49:13PM 0 points [-]

It's broken for me, at least.

Instead of the old-fashioned way of breaking, with new comments around a certain time not being visible, I can't get to older comments or posts. I don't know what the extent of the loss of access is.

The report issues page (link at bottom of page) is read only. There was going to be a :"brief outage" for that page at 7AM PDT. That would be almost 7 hours ago.

I think the stock of humorous messages at the "you tried a link which isn't working" has been improved.

Comment author: NancyLebovitz 04 August 2010 09:00:56PM 0 points [-]

The last line should have been

I think the stock of humorous messages at the "you tried a link which isn't working" page has been improved.

Or to put it another way, the permalink which would normally make it possible for me to edit isn't working either.

Comment author: NancyLebovitz 04 August 2010 09:03:25PM 0 points [-]

And, though this is minor, if I hit the comment button, it seems as though nothing has happened. However, if I refresh the comments page, my comment shows up.

As a minor mercy, hitting the comment button more than once doesn't seem to produce duplicate comments.

Comment author: cupholder 04 August 2010 07:20:53PM 0 points [-]
Comment author: RichardKennaway 03 August 2010 07:54:49AM 16 points [-]

When I was a young man about to go out into the world, my father says to me a very valuable thing. He says to me like this... "Son," the old guy says, "I am sorry that I am not able to bank roll you to a very large start, but not having any potatoes which to give you, I am now going to stake you to some very valuable advice. One of these days in your travels, a guy is going to come to you and show you a nice, brand new deck of cards on which the seal has not yet been broken. This man is going to offer to bet you that he can make the jack of spades jump out of the deck and squirt cider in your ear. Now son, do not bet this man, for as sure as you stand there, you are going to wind up with an earful of cider."

-- Sky Masterson, a character in "Guys and Dolls"

Comment author: Psychohistorian 03 August 2010 11:46:47PM *  5 points [-]

My recollection of this involved "bet you five thousand dollars he can..." and "going to wind up five grand in the hole with an earful of cider," which makes the quote more entertaining. Nonetheless, an excellent quote.

Comment author: Rain 03 August 2010 12:48:23PM 14 points [-]

Read, every day, something no one else is reading. Think, every day, something no one else is thinking. Do, every day, something no one else would be silly enough to do. It is bad for the mind to be always part of unanimity.

-- Christopher Morley

Comment author: RichardKennaway 03 August 2010 07:44:31AM *  13 points [-]

Man cannot understand the perfection and imperfections of his chosen art if he cannot see the value in other arts. Following rules only permits development up to a point in technique and then the student and artist has to learn more and seek further. It makes sense to study other arts as well as those of strategy. Who has not learned something more about themselves by watching the activities of others? To learn the sword study the guitar. To learn the fist study commerce. To just study the sword will make you narrow-minded and will not permit you to grow outward.

-- Musashi, "A Book of Five Rings"

Comment author: MichaelVassar 04 August 2010 03:49:31AM 4 points [-]

Pretty much the opposite of the foundation of modern education and social organization.

Comment author: Randaly 03 August 2010 04:45:58AM 29 points [-]

"Any sufficiently analyzed magic is indistinguishable from SCIENCE!"

~Girl Genius

Comment author: DSimon 03 August 2010 03:47:34PM *  16 points [-]

Young Agatha Clay: But how can they protect me if they aren't here? That's illogical.

Uncle Barry: Um...It's science.

Young Agatha Clay: Ah. You mean you'll explain when I have a sufficiently advanced educational background.

Comment author: JoshuaZ 03 August 2010 04:18:41PM 7 points [-]

Yes, but in context this doesn't quite mean what it sounds like. In Girl Genius, "Science" is almost a password for any weird things built by sparks, rather than what a rationalist would call science.

Comment author: orthonormal 20 August 2010 05:36:00PM 4 points [-]

Indeed. It's Science as Attire in more ways than one.

Comment author: DSimon 03 August 2010 09:32:56PM 4 points [-]

Well, presumably the sparks have a rational understanding of the things they build, even if the reader doesn't. Excepting things like the clanks Agatha built when she was asleep, but she's something of an odd one even among sparks...

Comment author: Zubon 08 August 2010 11:39:09PM 2 points [-]

Girl Genius has visited that border between magic and science. Note also the first two frames, in which the spark rational mind is put to use in pursuit of spark emotional needs. "Look. I'm a girl with needs. Okay?"

Comment author: Zack_M_Davis 09 January 2011 12:22:39AM *  4 points [-]

Two more:

ZEETHA: hmf. She sounds like an idiot.
GIL: Well, yes. But she was never a malicious one.
ZEETHA: Is that important?
GIL: Heavens, yes! If I let everyone I thought was an idiot die---there wouldn't be many people left.
---Girl Genius


TARVEK: (to clank:) Stop! I am prince Tarvek of the House of Sturmvoraus. I am the direct descendant of Andronicus Avlois and heir to the Lightning Throne. I am the Storm King---and you were created to serve me!
(Clank throws Tarvek backwards; sound effect: "POW!")
MOLOCH: Um---that never works, you know.
--- Girl Genius (relevance: No Universally Compelling Arguments)

Comment author: pjeby 09 January 2011 04:04:21AM 3 points [-]

Um---that never works, you know.

But it should have... and Tarvek correctly notices that he is confused. (The reason for his confusion doesn't become clear until later, though.)

Comment author: ata 25 August 2010 04:59:33AM *  10 points [-]

Farnsworth: My god, is it possible?
Fry: It must be possible. It's happening.

— Futurama: "The Late Philip J. Fry"

Comment author: Baughn 22 August 2010 02:37:43PM *  10 points [-]

On the nature of ethics:

There is no justice in the laws of Nature, Headmaster, no term for fairness in the equations of motion. The universe is neither evil, nor good, it simply does not care. The stars don't care, or the Sun, or the sky. But they don't have to! We care! There is light in the world, and it is us.

~ Eliezer Yudkowsky c/o Harry Potter, Methods of Rationality chapter 39.

Comment author: Randaly 14 August 2010 03:15:15AM 10 points [-]

"A joke told by Warren Buffett comes to mind: a patient, after hearing from a doctor that he has cancer, tells the doctor, “Doc, I don’t have enough money for the surgery, but maybe could I pay you to touch up the x-ray?” Hope and self-deception are not a strategy."

~ Vitaliy Katsenelson

Comment author: orthonormal 06 August 2010 07:00:12PM 10 points [-]

In short, whatever emotional impulse we may have toward altruism and empathy, and to whatever extent it may be genetically hardwired, it does not obviate the need for explicit judgments about right and wrong. If it did not seem correct to act with kindness and fairness, even at a net personal cost—if there were no sensible reason for so acting, beyond a raw impulse to do so—then we would have reason to regard the raw impulse as pointlessly self-destructive—like a disposition to alcoholism or a purely visceral (so to speak) aversion to surgery—and we would have a reason to attempt to overcome it.

  • Gary Drescher, Good and Real
Comment author: RobinZ 11 August 2010 12:32:56PM *  9 points [-]

The most difficult subjects can be explained to the most slow-witted man if he has not formed any idea of them already; but the simplest thing cannot be made clear to the most intelligent man if he is firmly persuaded that he knows already, without a shadow of doubt, what is laid before him.

- Leo Tolstoy, The Kingdom of God is Within You (1894), ch. III

Comment author: wedrifid 19 August 2010 02:01:46PM 2 points [-]

You know, I'm going to remember that one and try to remind myself of it whenever I notice my status is interfering with my thoughts.

Comment deleted 19 August 2010 02:13:08PM *  [-]
Comment author: SilasBarta 19 August 2010 03:13:45PM 2 points [-]

I don't see how. Whatever truth it might hold for a 140/100 IQ gap, it doesn't hold for arbitrarily smart beings, who could tell a lesser being all the self-modifying it would need to do to reach cognitive parity.

In any case, as those who have seen my posts here know, my warning lights go off whenever someone claims that something "can't be explained, even given infinite time and space".

Comment author: Kingreaper 19 August 2010 04:23:14PM 2 points [-]

The point being made seems to be the contradiction of a common AI-theorist, futurist position.

That position being that once a computer algorithm of effective AI IQ of 100 is produced, it can increase it's intelligence to arbitrary levels by the addition of additional hard-drive space, ram, and processing power.

IMO the analogy slightly fails. It fails to include anything analogous to the increase in ram, which is a very important factor, as it allows complex concepts to be dealt with as a whole.

Comment author: XiXiDu 19 August 2010 03:34:45PM *  3 points [-]

The original quote said, "the most difficult subjects can be explained to the most slow-witted man". This contradicts in my opinion what Michael Anissimov (Media Director, SIAI) thinks to be the case, namely that "a person with an IQ of 100 cannot understand certain concepts that people with an IQ of 140 can".

Michael Anissimov is responsible for compiling, distributing, and promoting SIAI media materials.

I was just being pedantic here but thought that highlighting this point would be good as other people, like Greg Egan, seem to disagree. This is an important question regarding the dangers posed by AI.

Comment author: wedrifid 19 August 2010 04:13:34PM 0 points [-]

In any case, as those who have seen my posts here know, my warning lights go off whenever someone claims that something "can't be explained, even given infinite time and space".

Infinite time and space. That's a lot of time and space. I suspect my warning lights would go off too. Do people make claims like that often?

Comment author: SilasBarta 19 August 2010 05:33:18PM 2 points [-]

Anissimov sure did: "no matter how many time and notebooks they have."

Comment author: wedrifid 19 August 2010 04:14:33PM 1 point [-]

I'm not sure what your point is exactly, but if it is anything like "That quote by Anissimov is largely mistaken" then you are correct.

Comment author: XiXiDu 19 August 2010 04:46:48PM 0 points [-]

It's an open question. I just don't know if that is the case and I'm very curious to know more about this. The chimp-human example is very convincing. Further someone with down-syndrome probably cannot understand what you can comprehend. So where is the gap, is there one? It looks like. This says, yes, I should believe what the SIAI claims. However, the original quote claims that "the most difficult subjects can be explained to the most slow-witted man". If this was the case, it would hint at the possibility that superhuman AI would merely be superhuman fast and had superhuman memory and ram, yet wouldn't reside on a different conceptual level.

Comment author: RobinZ 19 August 2010 04:57:17PM 0 points [-]

Both Anissimov and Tolstoy appear to me to be engaging in hyperbole. Further, Tolstoy was writing in the end of the nineteenth century - computers didn't exist, much less the idea of general artificial intelligence.

Comment author: Kingreaper 19 August 2010 04:17:25PM *  0 points [-]

While a perfectly valid, and somewhat relevant, point if true, that position does need support.

Comment author: timtyler 19 August 2010 05:18:47PM 0 points [-]

It seems pretty intuitively obvious - unless the notebooks have special things written on them.

Comment author: Kingreaper 19 August 2010 11:15:31PM *  2 points [-]

Really?

Give an example of something that you believe the average person, given 200 years to study completely focused on that task, couldn't possibly understand.

Comment author: wedrifid 20 August 2010 12:07:08AM *  0 points [-]

It seems pretty intuitively obvious - unless the notebooks have special things written on them.

Intuitively obvious but also wrong. IQ primarily makes a difference in how long it takes to master something and which knowledge can be created independently, without the need to load it from a notebook. Both of these factors have been assumed in infinite measure. Obviously IQ will make a difference in how effectively those concepts can be applied.

I make the tentative prediction "There are no concepts that can be discovered by typical people with an IQ of 140 that can not be understood by an average person with IQ of 100, no matter how much time and how many notebooks the latter is given." I would also be comfortable extending the IQ limit to 200, so long as a clear condition of (neuro)typical is maintained.

Comment author: timtyler 19 August 2010 05:17:13PM 0 points [-]

This quote seems undermine the SIAI.

How do you mean?

Comment author: Jayson_Virissimo 03 August 2010 02:38:38AM *  20 points [-]

The fact that you are giving money to charity does not mean that you need not try to find out whether that charity is a fraud or not.

-C. S. Lewis, Mere Christianity

Comment author: sark 20 December 2010 03:57:57PM 2 points [-]

The fact that you don't want to give money to a charity does not mean you need to find out whether the charity is a fraud or not.

Just an observation of how those most motivated to figure out the effectiveness of a charity are those who don't want to donate. But of course, this only applies to seeking out evidence of non-effectiveness. Charities don't usually think of other charities as competition. Also, most of the time, most just rationalize a reason for non-effectiveness.

Comment author: SilasBarta 03 August 2010 03:07:35AM 3 points [-]

B... But, but he wants ... he ... says you, it's a good idea to ... argh!

Comment author: MartinB 03 August 2010 04:41:50AM 6 points [-]

His failing in one area, does not make his quotes untrue. Just a bit iffy. Thats why I try not to quote Gandhi or Churchill anymore.

Comment author: Matt_Simpson 03 August 2010 03:59:41AM 2 points [-]

?

Comment author: SilasBarta 03 August 2010 04:20:07AM *  2 points [-]

Yes, my thought exactly.

This isn't the first time someone's posted a quote from Lewis that he didn't follow.

My reply was me expressing shock that someone like him would have grounds to lecture others about not giving money to questionable charities.

Comment author: NihilCredo 03 August 2010 04:26:57AM *  5 points [-]

Lewis was a convert to Christianity. The usual self-analytical deficiencies of religious believers are often gigantified in converts, possibly because they adopted their beliefs out of need rather than simple habit and thus will hug them much tighter.

Comment author: magfrump 03 August 2010 04:44:40AM *  0 points [-]

Similar ideas discussed here.

Comment author: NihilCredo 03 August 2010 05:06:49AM 2 points [-]

I think there are two slightly different situations at work here. Lewis did grow up in a very Christian environment, and as such possessed the "antibodies" described in the linked post.

His being a convert from atheism - by means of some kind of emotional breakdown - didn't, thus, make him into an advocate of following every passage of the Bible and every idea ever presented by the Magisterium; rather, it made him a zealous defender of every excuse for why you don't need to do so in order to be a good Christian.

Comment author: magfrump 03 August 2010 05:19:53AM 1 point [-]

That's fair--I didn't mean to say that it was the exact same case, simply that similar topics had been discussed before. That wasn't clear from the original comment so I'll change it to make it a little more so.

Comment author: Rain 03 August 2010 12:56:20AM 20 points [-]

A great many people think they are thinking when they are merely rearranging their prejudices.

-- William James

Comment author: wedrifid 19 August 2010 01:55:32PM *  7 points [-]

"There are worse things than seeming irresponsible. Losing, for example."

~ Paul Graham

Comment author: XiXiDu 08 August 2010 07:07:44PM 6 points [-]

You use a metaphor to describe some concept. The metaphor isn’t the thing you describe - it’s just a tool that you use. But someone takes the metaphor, and runs with it, making arguments that are built entirely on metaphor, but which bear no relation to the real underlying concept. And they believe that whatever conclusions they draw from the metaphor must, therefore, apply to the original concept.

— Mark Chu-Carroll, Metaphorical Crankery: a bad metaphor is like a steaming pile of …

Comment author: simplicio 04 August 2010 05:32:50AM *  14 points [-]

On rationalization, aka the giant sucking cognitive black hole.

Though [Ben Franklin] had been a vegetarian on principle, on one long sea crossing the men were grilling fish, and his mouth started watering:

I balanc'd some time between principle and inclination, till I recollectd that, when the fish were opened, I saw smaller fish taken out of their stomachs; then thought I, "if you eat one another, I don't see why we mayn't eat you." So I din'd upon cod very heartily, and continued to eat with other people, returning only now and then occasionally to a vegetable diet.

Franklin concluded: "So convenient a thing it is to be a reasonable creature, since it enables one to find or make a reason for every thing one has a mind to do."

-Jonathan Haidt, "The Happiness Hypothesis"

Comment author: wedrifid 04 August 2010 06:28:25AM *  5 points [-]

I really like the quote about cod but I'm not particularly inspired by the moral given for the story. I'd prefer "I eliminated a non-terminal ethical principal when I realised my thinking was pretentious bullshit, moving towards a more coherent ethical framework. Yay me!"

Comment author: simplicio 04 August 2010 06:40:38AM 8 points [-]

I noticed that too; of course not eating fish is an ethical non-issue given how much other low-hanging consequentialist fruit there is.

However, note that his justification for his change of heart is pure rationalization. Whatever good reasons there might be for eating fish, or for abandoning vegetarianism, "they eat each other" is a bad one, a confabulation.

Fish and other animals are not capable of reflecting ethically on their actions, so they are ethically blameless for whatever they do. That does not mean their suffering doesn't count. Franklin knew that.

Comment author: SilasBarta 04 August 2010 12:39:59PM *  6 points [-]

I know I'm bringing Drescher up a lot recently, but this exchange reminds me of some of his points, and how, after reading Good and Real, I see Haidt's work (among other people's) in a different light.

Drescher's theory of ethics and decision making is, "You should do what you [self-interestedly] wish all similarly situated beings would do" on the basis that "if you would regard it as the optimal thing to do, then-counterfactually they would too".

He claims it implies you should cast a wide net in terms of which beings you grant moral status, but not too wide: you draw the line at beings that don't make choices (in the sense of evaluating alternatives and picking one for the sake of a goal), as that breaks a critical symmetry between you and them.

Taking your premise that fish don't reflect on their actions, this account would claim that they likewise do not have the moral status of humans. But it would also agree with you that it's insufficient to point to how they eat each other, because "I would not want some superbeing to eat me simply on the basis that I eat less intelligent beings."

Also, Drescher accounts for our moral intuitions by saying that they are a case of us being choice machines which recognize acausal means-end links (i.e. relationships between our choices and the achievement of goals that do not require the choice to [futurewardly] cause the goal). This doesn't necessarily contradict Haidt's argument that we judge things as right because of e.g. ingroup/outgroup distinctions (he says that functional equivalence to acausal means-ends links is all that matters, even if the agent simply feels that they "care" about others), but it does tend to obviate that kind of supposition. [/show-off]

Comment author: simplicio 06 August 2010 06:03:10AM 2 points [-]

I've got Good and Real on hold at the library. :) Currently working through Cialdini's Influence, muahaha...

Drescher's theory of ethics and decision making is, "You should do what you [self-interestedly] wish all similarly situated beings would do" on the basis that "if you would regard it as the optimal thing to do, then-counterfactually they would too".

He claims it implies you should cast a wide net in terms of which beings you grant moral status, but not too wide: you draw the line at beings that don't make choices (in the sense of evaluating alternatives and picking one for the sake of a goal), as that breaks a critical symmetry between you and them.

This sounds to me like a modernized version of Kantian deontology... interesting.

Where I really trip up with this argument is in the 'granting moral status' step. What does it mean if I decide to say 'a fish has no moral status?'

Let's do a reductio. Say fish have no moral status. Does that mean it's permissible to torture them, say by superstimulating pain centres in their brains? I don't think so, even if the torture achieved some small useful end.

I don't think suffering should be taken out of the equation in favour of symmetries. The latter have no obvious moral weight.

Comment author: SilasBarta 06 August 2010 04:03:17PM *  1 point [-]

I don't have a good answer for the rest of your comment, but I can answer this:

Where I really trip up with this argument is in the 'granting moral status' step. What does it mean if I decide to say 'a fish has no moral status?'

Drescher does a good job of making sure that nothing depends on choice of terminology. In this case, "a fish has no moral status" cashes out to "I should not count a fish's disutility/pain/etc. against the optimality of actions I am considering."

You can take "should" to mean anything under Drescher's account, and, as long as you're consistent with its usage, it has non-absurd implications. Under common parlance, you can take "should" to mean "the action that I will choose" or "the action I regard as optimal". Then, you can see how this sense of the term applies:

"If I would regard it as optimal to kill weaker beings, then-counterfactually beings who are stronger than me would regard it as optimal to kill me, to the extent that their relation to me mirrors my relation to the weaker beings under consideration."

I didn't give a full exposition of how exactly you apply such reasoning to fish, but under this account, you would need to look at what is counterfactually entailed by your reasoning to cause pain to fish.

Comment author: wedrifid 04 August 2010 12:00:41PM 1 point [-]

Whatever good reasons there might be for eating fish, or for abandoning vegetarianism, "they eat each other" is a bad one, a confabulation.

No, that isn't implied. There are all sorts of coherent value systems which make ethical distinctions between killing things that kill other things and killing things that don't kill other things. Maybe Franklin was confabulating, but again, that moral does not inspire me. In most cases the reasoning is sound and does move the values a step towards coherency.

There is a difference between dastardly rationalisation and updating your ethical position by eliminating obviously poor thinking.

Fish and other animals are not capable of reflecting ethically on their actions, so they are ethically blameless for whatever they do.

A lot of people are good at not reflecting ethically too, and it does help them get away with stuff (via more effective signalling). This is not a feature of the universe over which I rejoice and nor is it one that I encourage via my ethical signalling.

Comment author: RichardKennaway 04 August 2010 12:22:54PM *  2 points [-]

Maybe Franklin was confabulating

His comment on the matter suggests he thought he was.

The context does not record whether he returned to vegetarianism once away from the temptation.

Comment author: wedrifid 05 August 2010 01:59:10AM 1 point [-]

His comment on the matter suggests he thought he was.

Yes. Hence the lack of inspiration. It's the same old moral: "Thoughts and ethical intuitions are enemies. Ethical intuitions are good and you should follow them. Thinking your ethics through is bad. Submit to the will of the tribe!"

I say if subjecting your ethical intuitions to rational analysis doesn't lead you to change them in some way then you are probably doing it wrong.

Comment author: Oligopsony 05 August 2010 02:08:46AM 2 points [-]

How subject ethical intuitions should be to rational analysis (in the sense of being changed by them) depends on how much you endorse the fact-value distinction and how fundamental the intuition is.

Reason leads me (though perhaps my reasoning is flawed) to conclude that "others' abject suffering is bad" isn't any more justified a desire than "others' abject suffering is good;" they're as equivalent as a preference for chocolate or vanilla ice cream. But so what? I don't abandon my preference for vanilla just because it doesn't follow from reason. Morality works the same way, except that ideally, I care about it enough to force my preferences on others.

Comment author: wedrifid 05 August 2010 02:16:18AM 2 points [-]

How subject ethical intuitions should be to rational analysis (in the sense of being changed by them) depends on how much you endorse the fact-value distinction and how fundamental the intuition is.

Yes. It is non-terminal ethical intuitions that I expect to be updated. "Should not do X because Y" should be discarded when it becomes obvious that Y is bullshit.

Comment author: WrongBot 27 August 2010 02:24:43AM *  5 points [-]

I'll believe anything, no matter how wild and ridiculous, if there is evidence for it.

-- Isaac Asimov

Comment author: XiXiDu 08 August 2010 07:04:38PM *  5 points [-]

A little bit of knowledge is a dangerous thing. It can convince you that an argument this idiotic and this sloppy is actually profound. It can convince you to publicly make a raging jackass out of yourself, by rambling on and on, based on a stupid misunderstanding of a simplified, informal, intuitive description of something complex.

— Mark Chu-Carroll, The Danger When You Don’t Know What You Don’t Know : Good Math, Bad Math

Comment author: SilasBarta 03 August 2010 04:49:35PM *  5 points [-]

I had just recently seen the two Christopher Nolan Batman movies (Batman Begins and The Dark Knight). Here are my favorite quotes; please add any that you like. (Character attribution left off to prevent pre-judgment.) Wikiquote lists.

"It's not who you are on the inside; it’s what you do that defines you." (Compare: Functionalism, substrate independence, timeless identity – I know, that’s probably not what was intended, but note how it’s used in other contexts.)

"Criminals thrive on the indulgence of society's understanding." (Compare to counterfactual reasoning: if we would sympathize with every defection, and felt that “the past is the past”, those wishing to defect would have no reason not to.)

“And what about escalation? […] We start carrying semi-automatics, they buy automatics. We start wearing Kevlar, they buy armor-piercing rounds. […] And you’re wearing a mask, and jumping off rooftops!”

(Also, I watched the earlier Batman movies after seeing this, and frankly, by comparison, they look like campy garbage.)

Comment author: NancyLebovitz 03 August 2010 05:37:25PM 6 points [-]

"Criminals thrive on the indulgence of society's understanding." (Compare to counterfactual reasoning: if we would sympathize with every defection, and felt that “the past is the past”, those wishing to defect would have no reason not to.)

It's a complicated issue-- as nearly as I can tell, the people who argue for no understanding assume that they can just use their intuitions about punishment, and not update about whether they're getting the effects they want.

Comment author: SilasBarta 03 August 2010 05:45:37PM *  2 points [-]

I agree. I wasn't trying to argue in favor of some kind of unlimited punishment, or against all understanding whatsoever -- just that this kind of understanding can be misused, especially when you discount the offense for being in the past. (I had recently read Drescher's account in Good and Real of why the pastward, inalterable aspect of a transgression, and the fact that the punishment only causes things in the future, are no reason not to punish.)

Edit3: And, of course [rot13] vg jnf n onq thl jub fnvq gung va gur zbivr, naq gur guvatf lbh jnea nobhg ner unaqyrq va gur zbivr, nf gung punenpgre'f ernfbavat yrnqf uvz gb qb ubeevoyr guvatf ba gung onfvf, yvxr gel gb qrfgebl na ragver pvgl. Ng gur fnzr gvzr, V guvax ur qbrf unir n cbvag.

Comment author: Morendil 03 August 2010 04:53:26PM 12 points [-]

Nolan's Memento is also interesting from a rationalist perspective - it gives "running on untrusted hardware" a quite concrete meaning.

Comment author: Nic_Smith 03 August 2010 05:08:21PM 4 points [-]

"Someday I want to be so powerful that I can defeat myself in a single blow." -- Silence, commenting on a character in Prism Ark that says they want to be stronger.

Comment author: Will_Newsome 07 August 2010 05:16:12AM 4 points [-]

I don't understand. Anybody who lives near a cliff or owns a gun has that much power. Explain? What is the connection to rationality?

Comment author: [deleted] 03 August 2010 09:00:37PM 1 point [-]

That's the best summary of rationality I've ever seen.

Comment author: wedrifid 07 August 2010 08:31:55AM 3 points [-]

Wouldn't a good summary of rationality be a little more comprehensible? I'm sure there is a deep insight that Nic sees behind the quote and from what you say it must be related to rationality but at face value, without context that most people, myself included, don't have it makes no sense.

Comment author: Nic_Smith 07 August 2010 10:19:47PM 2 points [-]

Wow, I had no idea that this quote would be so controversial. I will attempt to explain.

To begin, I think there is something that can be learned from the quote, but I wouldn't call it a "deep insight." Or, at least, I wouldn't have before uninverted's reply. Silence, a friend of mine, was making fun of the focus of many anime on literal, physical strength, often detached from anything to protect or, indeed, any real overall goal at all (I don't remember if this is the case with Prism Ark or not). It's thus a light warning against focusing too much on method over end results.

Also, if there were two of you, it'd be impossible for you[a] to be decisively stronger and more skilled than you[b].

Uninverted, I'm guessing, was thinking of akrasia and overcoming cognitive biases, which can be considered "part of self" and, at the same time, something to "defeat" as easily as possible. Hence, the quote can also be read as a summary of rationality.

Comment author: [deleted] 09 August 2010 07:35:43AM 1 point [-]

Yes, that's what I thought exactly.

Comment author: sketerpot 03 August 2010 10:33:34PM 0 points [-]

With a big enough hammer, that goal is within the reach of anybody strong enough to swing it at their own head.

Comment author: Airedale 11 August 2010 04:48:08PM 8 points [-]

I entered the Lager (Auschwitz) as a non-believer, and as a non-believer I was liberated and have lived to this day. Actually, the experience of the Lager with its frightful iniquity confirmed me in my nonbelief. It has prevented me, and still prevents me, from conceiving of any form of providence or transcendent justice. . . . I must nevertheless admit that I experienced (and again only once) the temptation to yield, to seek refuge in prayer. This happened in October 1944, in the one moment in which I lucidly perceived the imminence of death . . . naked and compressed among my naked companions with my personal index card in hand, I was waiting to file past the “commission” that with one glance would decide whether I should go immediately into the gas chamber or was instead strong enough to go on working. For one instance I felt the need to ask for help and asylum; then, despite my anguish, equanimity prevailed; one does not change the rules of the game at the end of the match, nor when you are losing. A prayer under these conditions would have been not only absurd (what rights could I claim? and from whom?) but blasphemous, obscene, laden with the greatest impiety of which a nonbeliever is capable. I rejected the temptation; I knew that otherwise were I to survive, I would have to be ashamed of it

-- Primo Levi, The Drowned and the Saved (quoted by Damon Linker at The New Republic ; h/t Andrew Sullivan)

Comment author: NihilCredo 03 August 2010 04:42:45AM *  8 points [-]

Therefore, after intellectually evaluating your problems through common sense and drawing on what psychiatry has taught us, if you still cannot emotionally release yourself from unwarranted guilt, and put your theories into action, then you should learn to make your guilt work for you. You should act upon your natural instincts, and then, if you cannot perform without feeling guilty, revel in your guilt. This may sound like a contradiction in terms, but if you will think about it, guilt can often add a fillip to the senses. Adults would do well to take a lesson from children: children often take great delight in doing something they know they are not supposed to.

-- Howard Stanton Levey, The Satanic Bible

(hopefully unnecessary disclaimer: I am in no way a Satanist)

Comment author: mkehrt 03 August 2010 07:18:36AM 4 points [-]

Completely out of curiosity, why do you cite him by his birth name rather than his pen name of Anton Szander LaVey?

Comment author: NihilCredo 03 August 2010 09:07:49AM *  1 point [-]

As a token of obligatory disrespect towards the juvenile pomposity of adopting a spoooky name for oneself.

Comment author: DSimon 03 August 2010 03:42:14PM 11 points [-]

And "NihilCredo" isn't a spooky adopted name? :-)

Comment author: NihilCredo 03 August 2010 05:12:49PM 2 points [-]

Heh, good quip! :) (the difference is obvious, anyway.)

Comment author: CronoDAS 05 August 2010 12:04:03AM 1 point [-]

This may sound like a contradiction in terms, but if you will think about it, guilt can often add a fillip to the senses. Adults would do well to take a lesson from children: children often take great delight in doing something they know they are not supposed to.

I know this very well. Video games get so much more appealing when I have some work to avoid.

Comment author: steven0461 05 August 2010 12:21:03AM *  4 points [-]

It could also be a selection effect -- if you're doing something even though you're not supposed to, it must be something you really like.

Similarly, unhealthy food tastes good because substances that are neither healthy nor tasty aren't classified as food.

Comment author: pjeby 05 August 2010 01:10:30AM 2 points [-]

It could also be a selection effect -- if you're doing something even though you're not supposed to, it must be something you really like.

Or an evolutionary nudge in the direction of hypocrisy.

More precisely, if the tribe has bothered to prohibit it, it must be something that advantages you at someone else's (possibly the group's) expense, and therefore you should be:

a) Encouraged to do it more, while...

b) Exhibiting plausible self-punishment when caught, to maintain group ties.

The former explains the increased reward, the latter the social behavior aspect.

Comment author: CronoDAS 05 August 2010 01:40:36AM 6 points [-]

It's specifically the "I have something unpleasant to do, and I'm not doing it!" that's exciting. I'll play even a bad game when I have work to avoid.

Comment author: Rain 03 August 2010 12:55:31AM 8 points [-]

Your imaginings can have as much power over you as your reality, or even more.

-- Charles T. Tart

Comment author: cousin_it 26 August 2010 03:53:55PM 3 points [-]

“The greatest obstacle to discovering the shape of the earth, the continents and the ocean was not ignorance but the illusion of knowledge.” -- Daniel Boorstin

Comment author: Thomas 04 August 2010 07:47:18AM 3 points [-]
  • If I understand it, I can build it.

Richard Feynman's inverted (not A implies not B == B implies A) quote.

Comment author: Vladimir_Nesov 04 August 2010 07:59:53AM 1 point [-]

Dualized.

Comment author: knb 04 August 2010 04:46:57AM 3 points [-]

On the subject of noble lies:

No: at all costs, it is to be prayed by all men that shams may cease. Good heavens, to what depths have we got, when this to many a man seems strange! Yet strange to many a man it does seem; and to many a solid Englishman, wholesomely digesting his pudding among what are called the cultivated classes, it seems strange exceedingly; a mad ignorant notion, quite heterodox, and big with mere ruin. He has been used to decent forms long since fallen empty of meaning, to plausible modes, solemnities grown ceremonial,--what you in your iconoclast humour call shams,—all his life long; never heard that there was any harm in them, that there was any getting-on without them.

Thomas Carlyle--The Latter-Day Pamphlets

Comment author: Jayson_Virissimo 03 August 2010 06:57:40AM 3 points [-]

I cannot prove, but am prepared to affirm, that if you take care of clarity in reasoning, most good causes will take care of themselves, while some bad ones are taken care of as a matter of course.

-Anthony de Jasay

Comment author: NihilCredo 03 August 2010 04:22:18AM *  4 points [-]

A drawing instead of a quote.

(This one is also interesting. I didn't spot much else worth sharing in the rest of the "comic", however.)

Comment author: RichardKennaway 03 August 2010 07:50:43AM 3 points [-]

Dear Miss Manners: How is a hat correctly worn?

Gentle Reader: Same as always; on the head.

-- Judith Martin ("Miss Manners")

See also: The Simple Truth.

Comment author: mikerpiker 03 August 2010 02:55:55AM *  6 points [-]

The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents. We live on a placid island of ignorance in the midst of black seas of infinity, and it was not meant that we should voyage far. The sciences, each straining in its own direction, have hitherto harmed us little; but some day the piecing together of dissociated knowledge will open up such terrifying vistas of reality, and of our frightful position therein, that we shall either go mad from the revelation or flee from the deadly light into the peace and safety of a new dark age.

-H.P. Lovecraft

Comment author: Kyre 03 August 2010 05:35:17AM *  8 points [-]

I never read Lovecraft as being any kind of metaphor for the real world, so I wouldn't vote this up as a rationalist quote for that reason.

But I like it as a device used Lovecraft to try to convey a sheer magnitude of horror. Can you imagine discovering something so horrific you wished you could delete the whole thing from your memory ? The more you pride yourself as a rationalist, the more horrific it would have to be.

Comment author: AlanCrowe 03 August 2010 11:12:26AM 7 points [-]

This belongs on the parody site http://morewrong.com. Please build it :-)

Comment author: Yvain 03 August 2010 10:13:43PM 15 points [-]

I don't know; the more Less Wrong I read, the more I start to think Lovecraft was on to something.

Delving too far in our search for knowledge is likely to awaken vast godlike forces which are neither benevolent nor malevolent but horrifyingly indifferent to humanity. Some of these forces may be slightly better or worse than others, but all of them could and would swat our civilization away like a mosquito. Such forces may already control other star systems.

The only defense against such abominations is to study the arcane knowledge involved in summoning or banishing these entities; however, such knowledge is likely to cause its students permanent psychological damage or doom them to eternities of torture.

Comment author: Larks 06 August 2010 12:14:57AM 4 points [-]

We've got Harry Potter and the Methods of Rationality; maybe you should write Cthulhu Mythos and Rationality ?

Then again, it might be unwise to disseminate it openly.

Comment author: steven0461 06 August 2010 12:18:44AM 11 points [-]

At the Mountains of Sanity

Comment author: gwern 15 November 2010 11:21:16PM *  3 points [-]

I've always enjoyed Vernor Vinge's name for AI: "Applied Theology".

(In, I think, A Fire upon the Deep.)

Comment author: Jayson_Virissimo 04 April 2014 07:51:56PM 1 point [-]

"Theological engineering" has a nice ring to it.

Comment author: DSimon 03 August 2010 03:21:28AM *  9 points [-]

Sounds like Caveman Science Fiction to me. "Why should we risk learning about new things, when there's a possibility they'll be scary?"

Comment author: toto 06 August 2010 08:06:27AM 0 points [-]

This seems to be the premise of Isaac Asimov's "Nightfall".

Comment author: Cyan 03 August 2010 12:20:11AM *  5 points [-]

Sometimes, I feel the fear of uncertainty stinging clear
And I can't help but ask myself how much I let the fear
Take the wheel and steer
It's driven me before
And it seems to have a vague, haunting mass appeal
But lately I'm beginning to find that I
Should be the one behind the wheel

- Incubus, Drive

Comment author: Matt_Simpson 03 August 2010 12:57:30AM 2 points [-]

Wow... I hadn't noticed that before

Comment author: Cyan 03 August 2010 03:16:15AM 1 point [-]

I'd quote the whole song, but I think that might violate a copyright or something.

Comment author: Matt_Simpson 03 August 2010 03:57:57AM *  0 points [-]

I doubt it, but there are several sites with the full lyrics if anyone wants to see them.

Edit 24 July 2013: Removed link to lyricsbay at their request. – admin

Comment author: gwern 03 August 2010 04:21:58AM 0 points [-]

Fear is the little-death that brings total obliteration

Comment author: WrongBot 03 August 2010 01:38:38AM *  3 points [-]

No man is an island entire of itself; every man is a piece of the continent, a part of the main; if a clod be washed away by the sea, Europe is the less, as well as if a promontory were, as well as any manner of thy friends or of thine own were; any man's death diminishes me, because I am involved in mankind. And therefore never send to know for whom the bell tolls; it tolls for thee.

-- John Donne

Comment author: Will_Newsome 03 August 2010 02:01:51AM 3 points [-]

Any man's death diminishes me, because their agenthood and qualia are probabilistically similar to mine, and it would not have taken many counterfactual changes for me to be not at all unlike that man, compared to a babyeater, whose pain bothers me less. Thus altruism is found in the egoist, to an extent.

Comment author: ata 03 August 2010 02:10:27AM 5 points [-]

Getting from "other people's minds are probably similar to mine" to "I care about other people's minds" still requires some implicit premises or some psychological features beyond egoism (e.g. empathy).

Comment author: Will_Newsome 03 August 2010 07:08:40PM *  2 points [-]

Specifically it requires "I care about myself" with saner boundaries around what counts as 'self' as the implicit premise.

Added: Any specific reason for the downvote?

Comment author: ShardPhoenix 03 August 2010 04:50:07AM *  0 points [-]

Evolution (of genes and/or memes) is probably sufficient to generate this.

Comment deleted 03 August 2010 02:52:31AM *  [-]
Comment author: ata 03 August 2010 02:55:49AM *  1 point [-]

Indeed. But my comment was a reply to Will's.

Comment author: NihilCredo 03 August 2010 03:23:18AM 0 points [-]

Mine eyes, they deceive me! Deleted.

Comment author: WrongBot 03 August 2010 04:26:10AM 15 points [-]

Please don't delete comments. It makes it hard to understand orphaned replies. Adding an [Edit: Withdrawn] at the end of the comment serves the same purpose, but maintains conversational continuity.

Comment author: WrongBot 03 August 2010 02:21:07AM 1 point [-]

What are qualia?

Comment author: Randaly 03 August 2010 02:33:04AM 2 points [-]

The subjective way we experience things.

Comment author: WrongBot 03 August 2010 04:17:09AM 0 points [-]

What do you mean by "experience"? And "subjective"? I'm not sure what you're talking about.

Comment author: Randaly 03 August 2010 04:55:41AM 2 points [-]

By experience I mean anything which we detect with one of our senses.

The subjective part is, IMHO, the key to qualia.

Suppose that you've never seen red light, and that you are then told all of its properties in perfect detail. You would still gain new information by actually seeing red light, because you still don't know "what it feels like" to see it. The qualia is not the objective facts, but rather what seeing the light "feels like", your perception of the effect produced on your brain by the light.

(Qualia are usually taken to be an argument against materialism- because after you know every objective fact about something, you still gain new information (qualia) by experiencing it.)

Comment author: orthonormal 03 August 2010 04:23:10PM *  10 points [-]

This "Mary's Room" argument, like the "Chinese Room" argument†, contains a subtle sleight of hand.

On the one hand, for the learning to be about just the qualia rather than about externally observable features of vision processing, the subject would need to learn immensely more than the physical properties of red light. (The standard version of Mary's Room does so, postulating Mary to also deeply understand her own visual cortex and the changes it would undergo upon being exposed to that color.) In fact, the depth of conscious theoretical understanding that this would require is far beyond any human being, and it's wrong and silly to naively map our mind-states onto those of such a mind.

On the other hand, it plays on the everyday intuition that if I've never seen the color red, but have been given a short list of facts about it and am consciously representing my limited intuition for that set of facts, that doesn't add up to the experience of seeing red.

The equivocation consists of thinking that a superhuman level of detailed understanding of (and capability to predict) the human brain can be analogized to that everyday intuition, rather than being unimaginably other to it. So I don't see that an agent who was really possessed of that level of self-understanding would necessary feel that the actual experience added an ineffable otherness to what they already knew.

That sense of ineffable otherness, IMO, comes from the levels of detail in the mental processing of color which we don't have conscious access to. Our conscious mind isn't built to understand what we're doing when we visually perceive, at the level that we actually do it-- there's no evolutionary need to communicate all the richness of color perception, so the conscious mind didn't evolve to encompass it all. And this limitation of our conscious understanding feels to us like a thing we have which cannot in principle be reduced.

† The application of this same principle to the Chinese Room argument is a trivial exercise, left to the reader.

Comment author: red75 03 August 2010 05:39:45PM *  2 points [-]

Intuitions don't matter. If Mary can't activate her neural pathways participating in creation of experience of seeing red, then she has no means of knowing how she will experience redness. All models she can create in her mind will be external to her as the mind created by actions of human being in Chinese room is external to that human being.

And this limitation of our conscious understanding feels to us like a thing we have which cannot in principle be reduced.

It is not only conscious understanding that is required, we will need a conscious control of individual neurons and synapses to be able to experience qualia given just a description of it. For example, to be able to name color and imagine color given its name, Mary (roughly speaking) should manually connect neurons in her visual cortex to the neurons in her Broca's area and to the neurons in her auditory cortex.

So I think that, contrary to Dennet, Mary will get new information when she will see colors, as human's brain construction doesn't allow to acquire that information by other means. Thus in a sense human's qualia cannot be reduced.

Comment author: Blueberry 03 August 2010 07:36:18PM 4 points [-]

It is not only conscious understanding that is required, we will need a conscious control of individual neurons and synapses

You may be interested in this paper which makes a similar argument.

Comment author: red75 03 August 2010 08:17:11PM 1 point [-]

Thanks. It is identical argument modulo my inability to make all reasoning and premises sufficiently transparent.

Comment author: thomblake 03 August 2010 05:47:37PM 3 points [-]

That contradicts one of the assumptions in the thought experiment. You're establishing qualia as a physical property; in that case, "what it feels like to see red" is amongst the things Mary knows about, by hypothesis.

Also, if it just comes down to activating those neurons, then Mary knows that too and can perform an experiment to activate those neurons without having a 'red thing' in front of her, using her incredible superhuman intelligence and resources.

Comment author: red75 03 August 2010 06:11:21PM *  1 point [-]

I am not establishing qualia as physical properties of brain's activity, I think of them as descriptions of specific neural activity in the terms of human's self-model. And limitations of that self-model (it's not sufficiently detailed to refer to individual neurons) don't allow to establish unambiguous correspondence between physical description of brain and self-model description of brain within that self-model.

Mary knows that too and can perform an experiment to activate those neurons without having a 'red thing' in front of her, using her incredible superhuman intelligence and resources.

And what is a difference between seeing red thing and activation of those neurons? The point of "Mary's room" is to know what seeing red means without actually seeing it.

Comment author: SilasBarta 03 August 2010 04:25:36PM 0 points [-]

In fact, the depth of understanding that this would require is far beyond any human being, and we really have no intuition for what it would be like to have it.

So we have no intuition for that understanding level's qualia? ;-)

Comment author: orthonormal 03 August 2010 04:28:56PM 0 points [-]

Yeah, I realized the unintended recursion there, and have edited accordingly...

Comment author: WrongBot 03 August 2010 05:02:31AM 6 points [-]

Well, of course a verbal description of red light is different from seeing red light. One is an auditory stimulus, and one is a visual stimulus. They do different things to my neurons. Are qualia about something other than neurons?

Comment author: JanetK 03 August 2010 10:50:16AM 4 points [-]

I voted up WrongBot's redefinition of his question, 'are qualia about something other than neurons?'. Is qualia something other than a word that has been awkwardly defined? Why is colour always the example used to illustrate qualia? Is there something different between colour and things like position, texture, pitch? Has anyone thought of a better or even different way to experience our experiences?

Comment author: SilasBarta 03 August 2010 03:59:06PM *  1 point [-]

Readers may be interested in my approach to the problem, or rather, the problem that remains even after any terminology issues are settled.

Summary: What we identify as "qualia" is the encoding of memories that we cannot yet compare directly between people, to the extent we can't compare them. This incommensurability can easily arise among agents who are similar, but who self-modify in a way that does not place any priority on the ability to directly transfer memories to other agents.

In that case, their methods of storing memories are ad-hoc, and look like garbage to each other -- but with the right assumptions and interaction, they can achieve a limited ability to compare, and thereby have terminology like "red" that means something to all agents, even as it doesn't call up exactly the same idea for each one.

Comment author: mkehrt 03 August 2010 05:49:44PM 1 point [-]

I was intrigued when I first read this when you last posted it, and I thought about it for a while. The problem with it, it seems to me, is that this is a good explanation for why qualia are ineffable, but it doesn't seem to be come any close to explaining what they are or how they arise.

So, I could imagine a world (it may even be this one!) where people's brains happen to be organized similarly enough that two people really could transfer qualia between them, but this still doesn't explain anything about them.

Comment author: SilasBarta 03 August 2010 07:51:18PM *  1 point [-]

The problem with it, it seems to me, is that this is a good explanation for why qualia are ineffable, but it doesn't seem to be come any close to explaining what they are or how they arise.

You're right. But I believe that that the ineffable aspect is closely related to the other two questions, although I don't have an answer in the same detail as the ineffability question (which would still be progress!).

To give a sketch of what I have in mind, my best explanation is this: conscious minds form when a subsystem is able to screen itself off from the entropizing forces of the environment (similar in kind to a refrigerator or other control system). This necessarily decouples it from the patterns that exist in the environment, as well as other minds that have done the same.

So the formation of a conscious mind will coincide with the formation of incompatible encoding methods, unless special care is taken to ensure that the encoding protcols are the same. Therefore, we shouldn't be surprised to notice that, "hey, everything that's conscious, also has ineffable experiences with the other conscious things."

But again, I don't claim this part is as well-developed or thought-out.

Comment author: LucasSloan 03 August 2010 04:26:04AM 0 points [-]

What is it that you feel/see/touch/taste/think/etc. instead of simply acting? Why is there a "you" you experience, instead of mere rote action? We label these sorts of things that we use to distinguish between empty existence and our own subjective (personally observed/felt) experience. The thing about humans that distinguish them from P-zombies.

Comment author: WrongBot 03 August 2010 04:46:41AM *  10 points [-]

What is it that you feel/see/touch/taste/think/etc. instead of simply acting?

Why do you group together sense perceptions (which I have) with thoughts (which I have), and call them qualia (which I don't have)?

Why is there a "you" you experience, instead of mere rote action?

How are these different?

We label these sorts of things that we use to distinguish between empty existence and our own subjective (personally observed/felt) experience.

How can existence be "empty"? Is subjective experience just sense perception? Because sense perception doesn't seem like it warrants all this mysteriousness.

The thing about humans that distinguish them from P-zombies.

That's odd. I thought the sequence on P-zombies made it pretty clear that they don't exist. Why do we need to be distinguished from confused, impossible thought experiments?

Comment author: RichardKennaway 03 August 2010 03:58:49PM 4 points [-]

Perhaps you simply do not have qualia or subjective experience. Some people do not have visual mental imagery, strange though that may seem to those of us who do. Similarly, maybe some people do not have anything they are moved to describe as subjective experience. Such people, if they exist, are the opposite of the logically absurd p-zombies. P-zombies falsely claim that they do have these things; people without them truthfully claim that they do not.

You might just be Socratically role-playing, but even so, there may be other people who actually do not have these things. That is, they would express puzzlement at talk about "the redness of red", "awareness of one's own self", and so forth (and without having been tutored into such puzzlement by philosophers arguing that they cannot be experiencing what in fact they do experience).

Is there anyone here who does experience that puzzlement, even before knowing anything of the philosophical controversy around the subject?

Comment author: WrongBot 03 August 2010 06:47:53PM 1 point [-]

People who lack visual mental imagery have atypical performance on certain kinds of cognitive tests, as Yvain's article describes, and if I believed that such people existed, I would expect that testable difference. What type of test should I expect to distinguish between those who have qualia and those who do not?

Comment author: RichardKennaway 03 August 2010 08:22:15PM *  3 points [-]

The most direct test would be this:

"Do you have qualia?"

  • Yes
  • No

But you'd have to use naive subjects who haven't philosophised themselves into ignoring their own experience.

A little more indirectly, people without qualia would profess puzzlement at the very idea, and argue that there is no such thing. If they are philosophers, they will write articles on the incoherence of the concept. If they are psychologists, they will practice psychology on the basis that mental phenomena do not exist. If they are teachers, they will see the brain as a pot to be filled, not the mind as a fire to be ignited. Those who do have qualia will be as tenacious on the other side.

Nothing that those who do have qualia say about qualia will make sense to those who don't, and those who don't will have no difficulty in demonstrating that it is nonsense. Those who do have qualia will be unable to explain them even to each other, since they know no more about what they are than they know about how thought happens. All of their supposed explanations will only be disguised descriptions of what it feels like to have them.

Looks pretty much like our world, doesn't it?

Comment author: Emile 03 August 2010 05:16:11PM 1 point [-]

There is this example:

To the best of my knowledge — which isn’t saying much: I’m not well-read in philosophy — I am in a minority of one on the subject of free will.

The discussion is always: do we have it, or don’t we?

My considered view is that some of us have it while the rest don’t. Like perfect pitch.

I’m pretty sure I don’t have free will; but I’ve encountered people who I’m pretty sure do have it.

I see that as a cheap way out, I think "do I have free will ?" is just a confused question whose answer depends of the way you unconfuse it. I'm just in the minority of humans who refuses to answer that confused question - I'd like to say I refuse to answer all confused questions, but that's probably not true.

Still, it is possible that confusion and disagreement about "qualia" and "free will" are just due to differences in personal experience, not to different interpretation of those labels.

Comment author: ata 03 August 2010 02:38:54AM 2 points [-]

Are you unfamiliar with the term, or are you asking him to demonstrate that he understands it well enough to permit his using it as a quiet assumption in making a point?

Comment author: gwern 03 August 2010 04:23:57AM 5 points [-]

WrongBot has been around enough that one can safely assume that his ignorance is Socratic.

Comment author: WrongBot 03 August 2010 04:23:16AM 1 point [-]

I am familiar with the term, but I don't seem to have any. And no one's been able to tell me what they do, so I like to ask when they come up so that maybe someday I'll find out.

Comment author: orthonormal 03 August 2010 04:35:34PM *  6 points [-]

I don't believe in qualia as a real entity, but when people talk about them they're referring to a genuine phenomenon which you also experience: that your conscious understanding of the experience of perception is only the merest shadow of the perception itself. Seeing red doesn't mean seeing something with a little XML "red" tag attached, but something much more complicated that happens beyond your conscious introspection. You can imagine the state of having switched that "red" experience with the "green" experience, in all your memories as well as in current perception, and still instantly knowing that the switch had occurred. This phenomenon is not an illusion, just a blind spot of conscious knowledge which happens to confuse the hell out of naive philosophers.

Comment author: WrongBot 03 August 2010 06:56:07PM 1 point [-]

[Y]our conscious understanding of the experience of perception is only the merest shadow of the perception itself.

Of course. If I had perfect knowledge of my brain's functioning, now that would be a very strange thing indeed.

You can imagine the state of having switched that "red" experience with the "green" experience, in all your memories as well as in current perception, and still instantly knowing that the switch had occurred.

No, I can't. If all my memories had been altered to agree with my newly-altered perception system, what difference would I detect? How would I detect it? Different from what?

Comment author: orthonormal 03 August 2010 07:11:52PM *  5 points [-]

The hypothetical situation I mean is one where your current retina is reprogrammed to switch red and green stimuli, and your memories are edited so that you don't figure it out from inconsistencies, but everything else is left the same.

The fact that there's subconscious cognitive content to red vs. green can be deduced from things like instinctive reactions† to the sight of blood: the brain doesn't check the color against the memory of other blood, it reacts faster than that to to perception. The emotional valence of colors would seem off somehow after a switch, because those don't appear to operate fully through memory, either. Snap judgments of peoples' attractiveness would backfire as your subconscious applied the rule "green tint means sickly" to someone with a healthy complexion.

I don't think you'd be able to consciously articulate what exactly seemed "red" about that green grass, but parts of your mind would be telling you that something's gone wrong, because they're hooked up not just to labels "red" and "green" but to full systems of processing that would be running on suddenly different stimuli.

†Similarly, chimps raised by humans in captivity will still freak out when exposed to a fake snake, because certain patterns have been encoded deep within. There's no reason for such patterns to be raised to the level of conscious knowledge.

Comment author: WrongBot 03 August 2010 07:22:16PM 5 points [-]

Ahhh, so you'd only be reprogramming part of my brain. Well, of course I'd run into problems then. All that means is that there are more parts of my brain than those I have conscious access to, which seems pretty obvious to me even before I start to think about what I know of neurology.

I think we agree with each other.

Comment author: ChristianKl 03 August 2010 09:45:15PM 0 points [-]

I wouldn't be sure, the vision system has an amazing ability to adept to rewiring. Monkeys were able to see another color through gene therapy that their species hadn't seen before.

Comment author: orthonormal 03 August 2010 10:03:41PM 1 point [-]
  1. Indeed there's rewiring over time, but it wouldn't be instant and it wouldn't be total, so the point stands.

  2. That's a really interesting experiment-- can you find me a link?

Comment author: SilasBarta 03 August 2010 04:56:54PM 1 point [-]

Thank you, well said! I've seen people go so far in dissolving qualia that they think they have to deny their own conscious experience, or think the confusion is extinguished as soon as you have the terminology nailed down.

Comment author: billswift 03 August 2010 09:45:52PM *  0 points [-]

Most philosophical definitions are pretty weird. On the rare occasions I use "qualia", it means the inside view of your sense perceptions. What things look/sound/feel like to the person doing the sensing.

Comment author: RichardKennaway 03 August 2010 07:51:35AM 1 point [-]

Numerical precision is the very soul of science, and its attainment affords the best, perhaps the only criterion of the truth of theories and the correctness of experiments.

-- D'Arcy Thompson, On Growth and Form (1917)

Comment author: Nic_Smith 03 August 2010 05:38:52PM 2 points [-]

This quote confuses me. At first I read it as a restatement of the famous Lord Kelvin quote on the topic -- if you don't have numbers, your understanding "is meager and unsatisfactory." Hooray for that as far as it goes. But the second half seems to suggest, reversing another famous quote, that it is better to be precisely wrong than vaguely correct.

Comment author: RichardKennaway 03 August 2010 08:01:40PM *  1 point [-]

I favour D'Arcy Thompson's view. If you are precisely wrong, it will be easy for evidence to refute you and make you less wrong. But if you are already vaguely right, how will you attain to being precisely right? How will you discover that you are actually vaguely wrong, if your wiggle room lets you explain away contrary evidence? As Francis Bacon wrote, "Truth arises more readily from error than from confusion."

Being vaguely right is only better when you need to decide an action now and have no opportunity to improve your knowledge. But being precisely right is better still. "Weak Bayesian evidence" is worth as much as a penny lying in the road: if you need to pick up a penny, you need a lot more than a penny.

But in fact, that is not the sort of vague rightness that the quote you linked to is about. Here is its context:

The terms of ordinary language fall into the same classes as those of science: they stand for things, classes of things, parts, or qualities, or activities of things; but they are far less precise in their signification. As long as popular thought is vague its language must be vague; nor is it desirable too strictly to correct the language whilst the thought is incorrigible. Much of the effect of poetry and eloquence depends upon the elasticity and indirect suggestiveness of common terms. Even in reasoning upon some subjects, it is a mistake to aim at an unattainable precision. It is better to be vaguely right than exactly wrong. In the criticism of manners, of fine art, or of literature, in politics, religion and moral philosophy, what we are anxious to say is often far from clear to ourselves; and it is better to indicate our meaning approximately, or as we feel about it, than to convey a false meaning, or to lose the warmth and colour that are the life of such reflections. It is hard to decide whether more harm has been done by sophists who take a base advantage of the vagueness of common terms, or by honest paralogists (if I may use the word) who begin by deceiving themselves with a plausible definiteness of expression, and go on to propagate their delusions amongst followers eager for systematic insight but ignorant of the limits of its possibility.

Carveth Read, Logic: Deductive and Inductive, p.351.

Popular thought, poetry, eloquence, manners, fine art, literature, politics, religion and moral philosophy. Is our "vague rightness" in such matters anything more than an illusion, a subjective sense of "meaningfulness" when we utter our words that has as much backing as a sub-prime mortgage?

ETA: That last sentence of the extended quote is rather good, and deserves to be quoted on its own. ETA again: except the final phrase: such people are up against the limits, not of its possibility, but of their own capacities.

Comment author: Nic_Smith 03 August 2010 11:33:58PM 0 points [-]

If you are precisely wrong, it will be easy for evidence to refute you and make you less wrong.

This seems to imply that we should delegate decision-making to a system that is certain the sky is rgb(0,255,0) over a system that assigns the bulk of its probability to various shades of blue. But, if we know that the sky really is some shade of blue, the system with the less precise prior that the sky is blue will do better than the system that precisely thinks it's (bright lime!) green as new evidence becomes available.

I can't imagine that this is actually what's meant by the original quote or your reply. What is D'Arcy Thompson's view?

Comment author: RichardKennaway 04 August 2010 07:16:02AM *  1 point [-]

Here's some context for D'Arcy Thompson:

As soon as we adventure on the paths of the physicist, we learn to weigh and to measure, to deal with time and space and mass and their related concepts, and to find more and more our knowledge expressed and our needs satisfied through the concept of number, as in the dreams and visions of Plato and Pythagoras; for modem chemistry would have gladdened the hearts of those great philosophic dreamers. Dreams apart, numerical precision is the very soul of science, and its attainment affords the best, perhaps the only criterion of the truth of theories and the correctness of experiments. So said Sir John Herschel, a hundred years ago; and Kant had said that it was Nature herself, and not the mathematician, who brings mathematics into natural philosophy.

[ETA: Here, p.122, is the context for the reference to Herschel.]

And he goes on to rebuke the life sciences for having been slow to follow the same course. He suspends judgement on whether the mysteries of the mind and consciousness can be solved by physical science, "But of the construction and growth and working of the body, as of all else that is of the earth earthy, physical science is, in my humble opinion, our only teacher and guide."

This seems to imply that we should delegate decision-making to a system that is certain the sky is rgb(0,255,0) over a system that assigns the bulk of its probability to various shades of blue.

I think that is a perverse reading of Carveth Read's maxim.

Comment author: RichardKennaway 03 August 2010 07:47:54AM *  1 point [-]

Let us not pretend to doubt in philosophy what we do not doubt in our hearts.

-- C.S. Peirce

Comment author: Matt_Duing 13 August 2010 06:13:06AM 1 point [-]

"They never do tests. Not many real deeds either. Oh, conversation with your grandmother's shade in a darkened room, the odd love potion or two, but comes a doubter, why, then it's the wrong day, the planets are not in line, the entrails are not favorable, we don't do tests!" -Tyrian, Dragonslayer

Comment author: XiXiDu 08 August 2010 07:13:22PM *  1 point [-]

I propose that answers to questions of human value can be visualized on a “moral landscape” — a space of real and potential outcomes whose peaks correspond to states of the greatest possible wellbeing and whose valleys represent the deepest depths of suffering. Different ways of thinking and behaving — different cultural practices, ethical codes, modes of government, etc. — translate into movements across this landscape. Such changes can be analyzed objectively on many levels — ranging from biochemistry to economics — but they have their crucial realization as states and capacities of the human brain.

— Sam Harris, Edge: THE NEW SCIENCE OF MORALITY

Edit: It was of course Sam Harris who said this.

Comment author: [deleted] 04 August 2010 08:48:58AM 1 point [-]

"...you must consider what you are, seeking to know yourself, which is the most difficult task conceivable. From self knowledge you will learn not to puff yourself up, like the frog who wanted to be as big as an ox." -- Don Quijote

Comment author: RichardKennaway 04 August 2010 09:23:07AM 0 points [-]

That's interesting, considering who the character is. I googled up some context, and Quixote is in one of his lucid phases here, where he speaks with wisdom on everything except himself.

Comment author: [deleted] 05 August 2010 12:47:47AM *  0 points [-]

Yeah, it's an ironic quote. For what it's worth, I also see some shades of GEB in this.

Comment author: [deleted] 04 August 2010 05:33:05AM *  1 point [-]

We must work and strive,
If we expect to thrive.
We've not that long to be alive.
This isn't forever.
But if it isn't forever, what's it for?
And if it isn't forever, how is it worth dying for?

-Alive by The Land Canaan

The song is about creation of the first FAI, and has tons of amazing lyrics. I couldn't find the full lyrics online, and I'm too lazy to transcribe them, so you'll have to listen for yourselves: http://www.archive.org/details/OnLeaving

I also recommend Wild Child which is about a not so friendly AI.

Comment author: katydee 04 August 2010 07:12:46AM *  0 points [-]

Transcribed:

You are what we want for our answers
Haptic li-

The storm is in the clouds
It's in the sky
We wait and pray
Not knowing who or what
We want
For our answers

Haptic line
Hectic life
That's for listening
That's all we dream
And here and there
You wonder at
The time it takes to bring forever

Oh, I awaken
Time to catch the storm
People fighting and killing and dying
As I am born
Insects growing and breaking free
In my mind I am immortal
Eternity
The life I live comes to me as a game
Only playing for the playing of it
Every day the same
The life I live comes to me as a game
Only playing for the playing of it
Every day the same

Don't get comfortable
For this will fade away
And beware the coming day
This isn't forever
We all must work and strive
If we expect to thrive
We've not that long to be alive
This isn't forever
But if it isn't forever, then what's it for (then what's it for)
And if it isn't forever (and if it isn't forever)
Then how is it worth dying for (then how is it worth dying for)

We built me, we made me
We built me, we shaped me
My blood and failing body
We built me and we built me
Stripped my flesh, wires bare
Machinery to the open air
Because eyes of gloss and silver will not cry (will not cry)
I spread my plastic wings and fly (and fly)
A thousand million years shaped this form
And made messiah in my own image, I, I am reborn
Eyes of gloss and silver will not cry
I spread my plastic wings and fly
A thousand million years have shaped this form
And made messiah, withdrawn from nature I am reborn
In my own image, I
I-I am reborn

I-I will not die
I will not die
I will not die
I-I will not die
I will not die
I will not

I-I will cry
I will try
I will struggle, I will cry
But I will never, never die
Because I, I-I am alive
Because I, I-I am alive
I, I-I am alive
Alive (I will not die)
Alive (I will not die)
Alive (I will not die)

Comment deleted 03 August 2010 03:02:18AM *  [-]
Comment deleted 03 August 2010 03:12:07AM *  [-]
Comment author: SilasBarta 03 August 2010 03:55:59AM 2 points [-]

Pardon the bluntness, but I noticed the "parodying the Nicene Creed" bit, saw it as a failed attempt to be clever by adapting religious recitations to a naturalist end, was unimpressed (not because of some general principle against such adaptation), and voted it down still, without seeing it as tree-hugging.

Comment author: komponisto 03 August 2010 04:27:44AM *  1 point [-]

Very well; deleted accordingly.

(Pardon me, however, if I'm a bit skeptical about there not being a "general principle" involved here -- Odifreddi's "credo" may not be all that witty -- and perhaps that's partly my fault, since the above was my translation -- but it would still deserve a place in this thread, were it not for aversions to "imitating" religion, and to reverential or emotional-sounding language in general. After all, since when do rationality quotes have to be witty or clever?)

Comment author: SilasBarta 04 August 2010 02:17:17PM 1 point [-]

Okay, looking back, I think that's a fair point. I meant something more like: Modifying religious hymns/creeds to express a rationalist view looks cliche (or a more apt term I can't think of) to me, so I hold them to a higher standard. And anyone can do a word swapout. So, in a sense, there is a general principle involved that I believe cuts against that kind of quote.

Rationality quotes do have to be witty or clever in the sense that they have to do a bit more than just state a rationalist tenet. It wouldn't be appropriate to post a quote as simple as, "You should update your beliefs on evidence."

Comment author: WrongBot 03 August 2010 04:29:29AM 0 points [-]

I await the dissolution of death, but not another life in a world which will not come.

I didn't like the quote overall, but that's the part that I took exception to. Death is the enemy.

Comment author: ata 03 August 2010 04:35:53AM *  4 points [-]

I think that line means the opposite of how you interpreted it. I read "I await the dissolution of death" not as "I await the dissolution that is death" but as "I await the point when the threat of death is dissolved".

Edit: What komponisto said.

Comment author: WrongBot 03 August 2010 04:53:40AM 1 point [-]

I don't quite see how the subsequent clause would make sense under that reading.

I await the point when death is dissolved, but not another life in a world which will not come.

If that "but" were an "and", I would agree with you.

Comment author: [deleted] 03 August 2010 05:53:14AM 1 point [-]

What about interpreting it like this:

I await the point when death is dissolved, but I do not await another life in a world because it will not come.

Comment author: WrongBot 03 August 2010 06:09:32AM 0 points [-]

Yeah, this may just be a parse error on my part. Apologies for the noise.

Comment author: komponisto 03 August 2010 05:48:11AM 1 point [-]

"But" makes perfect sense to me: "I, too, hope to triumph over death, but not in the way that religious people do."

Comment author: komponisto 03 August 2010 04:35:00AM 2 points [-]

Well, if death is the enemy, all the more reason to await its dissolution!

(Seriously, that's how I parsed it the first time I read it.)

Comment author: NancyLebovitz 11 August 2010 02:13:34PM 0 points [-]
Comment deleted 03 August 2010 06:15:37AM *  [-]
Comment author: FAWS 03 August 2010 06:31:29AM 0 points [-]