Paris LW Meetup - LHC Exhibit - 17/01/2015

1 kilobug 01 January 2015 12:22PM

We'll be organizing a meetup of LW Paris at "Palais de la Découverte" in order to view the exhibit on LHC (link in French, sorry) and speak about it (and any related - or even unrelated - subject) between LWer.

The meeting is scheduled at 14h00 on Saturday 17th afternoon.

Map and location information available.

Hoping to see many Paris LWers !

Meetup : Paris LW Meetup - LHC Exhibit

0 kilobug 01 January 2015 12:17PM

Discussion article for the meetup : Paris LW Meetup - LHC Exhibit

WHEN: 17 January 2015 02:00:00PM (+0100)

WHERE: Palais de la Découverte, Avenue Franklin Delano Roosevelt 75008 Paris

We'll go see the LHC exhibit and speak about it (and any other related theme) between LWers.

Discussion article for the meetup : Paris LW Meetup - LHC Exhibit

The VNM independence axiom ignores the value of information

10 kilobug 02 March 2013 02:36PM

Followup to : Is risk aversion really irrational?

After reading the decision theory FAQ and re-reading The Allais Paradox I realized I still don't accept the VNM axioms, especially the independence one, and I started thinking about what my true rejection could be. And then I realized I already somewhat explained it here, in my Is risk aversion really irrational? article, but it didn't make it obvious in the article how it relates to VNM - it wasn't obvious to me at that time.

Here is the core idea: information has value. Uncertainty therefore has a cost. And that cost is not linear to uncertainty.

Let's take a first example: A is being offered a trip to Ecuador, B is being offered a great new laptop and C is being offered a trip to Iceland. My own preference is: A > B > C. I love Ecuador - it's a fantastic country. But I prefer a laptop over a trip to Iceland, because I'm not fond of cold weather (well, actually Iceland is pretty cool too, but let's assume for the sake of the article that A > B > C is my preference).

But now, I'm offered D = (50% chance of A, 50% chance of B) or E = (50% chance of A, 50% chance of C). The VNM independence principle says I should prefer D > E. But doing so, it forgets the cost of information/uncertainty. By choosing E, I'm sure I'll be offered a trip - I don't know where, but I know I'll be offered a trip, not a laptop. By choosing D, I'm no idea on the nature of the present. I've much less information on my future - and that lack of information has a cost. If I know I'll be offered a trip, I can already ask for days off at work, I can go buy a backpack, I can start doing the paperwork to get my passport. And if I know I won't be offered a laptop, I may decide to buy one, maybe not as great as one I would have been offered, but I can still buy one. But if I chose D, I've much less information about my future, and I can't optimize it as much.

The same goes for the Allais paradox: having certitude of receiving a significant amount of money ($24 000) has a value, which is present in choice 1A, but not in all others (1B, 2A, 2B).

And I don't see why a "rational agent" should neglect the value of this information, as the VNM axioms imply. Any thought about that?

How to tell apart science from pseudo-science in a field you don't know ?

18 kilobug 02 September 2012 10:25AM

First, a short personal note to make you understand why this is important to me. To make a long story short, the son of a friend has some atypical form of autism and language troubles. And that kid matters a lot to me, so I want to become stronger in helping him, to be able to better interact with him and help him overcome his troubles.

But I don't know much about psychology. I'm a computer scientist, with a general background of maths and physics. I'm kind of a nerd, social skills aren't my strength. I did read some of the basic books advised on Less Wrong, like Cialdini, Wright or Wiseman, but those just give me a very small background on which to build.

And psychology in general, autism/language troubles in particular, are fields in which there is a lot of pseudo-science. I'm very sceptical of Freud and psychoanalysis, for example, which I consider (but maybe I am wrong?) to be more like alchemy than like chemistry. There are a lot of mysticism and sect-like gurus related to autism, too.

So I'm bit unsure on how from my position of having a general scientific and rationality background I can dive into a completely unrelated field. Research papers are probably above my current level in psychology, so I think books (textbooks or popular science) are the way to go. But how to find which books on the hundreds that were written on the topic I should buy and read? Books that are evidence-based science, not pseudo-science, I mean. What is a general method to select which books to start in a field you don't really know? I would welcome any advise from the community.

Disclaimer: this is a personal "call for help", but since I think the answers/advices may matter outside my own personal case, I hope you don't mind.

Neil Armstrong died before we could defeat death

-1 kilobug 25 August 2012 07:49PM

The sad news broke tonight : Neil Armstrong, the first human to ever walk another world, died today. We lost him forever. He died before we could defeat death.

Once again the horror of death strikes. This time, in addition from wiping from us forever a hero of humanity, he wiped from us forever a memory that will never exist again. Never again will a human being be able to experience being the first to walk another world. That beautiful experience is lost forever too, along with all the memories, dreams, desires and wishes that made Neil Armstrong.

But thanks to him, humanity made a giant leap. We'll fill the stars and conquer death. The spark of intelligence and sentience will not extinguish. That's the best we can do to honour him.

Source : http://www.reuters.com/article/2012/08/25/us-usa-neilarmstrong-idUSBRE87O0B020120825

Why space stopped captivating minds ?

10 kilobug 29 July 2012 09:58AM

This article http://www.huffingtonpost.com/andrew-kessler/why-you-should-be-more-interested-in-mars-than-the-olympics_b_1712462.html -- ok, I admit, I read Slashdot sometimes, no one is perfect ;) -- made me wonder why the awesomeness of space conquest stopped motivating people.

I remember the tales of my parents at the time of the Apollo landing, it was indeed instilling awe and wonder in the minds of people. It was followed by people like the Olympics or the football competitions are. And nowadays, NASA about to send a nuclear-powered rover to Mars, in a very delicate mission requiring the best of human engineering and scientific skills, and not in line in most media, most people not even aware of it? How did we fall that low?

Sure there was the Cold War. It definitely played a role, in the amount of resources invested by both sides in space conquest, and in the way the media broadcasted the news.

But here in France, a country that was mostly neutral during the Cold War (slightly west-aligned, but not part of NATO for most of the Cold War), the interest of people for space was not really partisan. People who were pro-USSR were amazed and cheering for the Appolo mission, people who were pro-USA were amazed and cheering for Gagarin. My brother and I played with (USSR) Sputnik as much as with (USA) space shuttles. We praised equally Neil Armstrong and Yuri Gagarin. I don't think the lack of Cold War explains it all.

So what happened to the space conquest spirit? How did it disappear? I notice a blank spot on my map (well, not totally blank, but still very fuzzy) of reality, do some of you have clues for how to fill it?

Interesting rationalist exercise about French election

8 kilobug 16 April 2012 03:34PM

The newspaper "Le Monde" made an interesting exercise for rationalists in the context of the French election.

They first made a classical "which is your best candidate" poll, in which they ask multiple choice questions about various topics (for each question, you must select one answer, and how important the issue is for you), and at the end, they select the candidate that (according to them) is closest to your answers. Nothing new in that.

But then, they made a much more interesting (at least from a rationality training point of view) exercise : they asked the same questions, but not asking "what is your opinion on the topic ?" and "how is this question important for you ?" but they asked "what do you think the majority of our readers answered ?" and "how do they think they rated the importance of this issue ?". And then they give you a score from 0 to 1000 on how good your "predictions" were.

It's in French, so it'll be hard for most of you to try it, but if you want it is available online here.

I found this kind of exercise (trying to guess what other people will have answered) to be interesting from a LW point of view, because it somehow makes your beliefs (about the opinions, priority and mentalities of French people) pay rent.

So I wanted to share it with the LW community, and ask if you know about similar exercises elsewhere, that gives you a way to check how accurate your belief network is in complicated issues, if you find them interesting too, and how they could be improved.

As an idea of improvement, I would like adding a confidence rating to each question, the more confident you feel in your answer, the more points you get if you get it right, but the more you lose if you get it wrong.

Is risk aversion really irrational ?

41 kilobug 31 January 2012 08:34PM
Disclaimer: this started as a comment to Risk aversion vs. concave utility function but it grew way too big so I turned it into a full-blown article. I posted it to main since I believe it to be useful enough, and since it replies to an article of main.

Abstract

When you have to chose between two options, one with a certain (or almost certain) outcome, and another which involves more risk, even if in term of utilons (paperclips, money, ...) the gamble has a higher expectancy, there is always a cost in a gamble : between the time when you take your decision and know if your gamble fails or succeeded (between the time you bought your lottery ticket,and the time the winning number is called), you've less precise information about the world than if you took the "safe" option. That uncertainty may force you to make suboptimal choices during that period of doubt, meaning that "risk aversion" is not totally irrational.

Even shorter : knowledge has value since it allows you to optimize, taking a risk temporary lowers your knoweldge, and this is a cost.

Where does risk aversion comes from ?

In his (or her?) article, dvasya gave one possible reason for it : risk aversion comes from a concave utility function. Take food for example. When you're really hungry, didn't eat for days, a bit of food has a very high value. But when you just ate, and have some stocks of food at home, food has low value. Many other things follow, more or less strongly, a non-linear utility function.

But if you adjust the bets for the utility, then, if you're a perfect utility maximizer, you should chose the highest expectancy, regardless of the risk involved. Between being sure of getting 10 utilons and having a 0.1 chance of getting 101 utilons (and 0.9 chance to get nothing), you should chose to take the bet. Or you're not rational, says dvasya.

My first objection to it is that we aren't perfect utility maximizer. We run on limited (and flawed) hardware. We have a limited power for making computation. The first problem of taking a risk is that it'll make all further computations much harder. You buy a lottery ticket, and until you know if you won or not, every time you decide what to do, you'll have to ponder things like "if I win the lottery, then I'll buy a new house, so is it really worth it to fix that broken door now ?" Asking yourself all those questions mean you're less Free to Optimize, and will use your limited hardware to ponder those issues, leading to stress, fatigue and less-efficient decision making.

For us humans with limited and buggy hardware, those problems are significant, and are the main reason for which I am personally (slightly) risk-averse. I don't like uncertainty, it makes planning harder, it makes me waste precious computing power in pondering what to do. But that doesn't seem apply to a perfect utility maximizer, with infinite computing power. So, it seems to be a consequence of biases, if not a bias in itself. Is it really ?

The double-bet of Clippy

So, let's take Clippy. Clippy is a pet paper-clip optimizer, using the utility function proposed by dvasya : u = sqrt(p), where p is the number of paperclips in the room he lives in. In addition to being cute and loving paperclips, our Clippy has lots of computing power, so much he has no issue with tracking probabilities. Now, we'll offer our Clippy to take bets, and see what he should do.

Timeless double-bet

At the beginning, we put 9 paperclips in the room. Clippy has a utilon of 3. He purrs a bit to show us he's happy of those 9 paperclips, looks at us with his lovely eyes, and hopes we'll give him more.

But we offer him a bet : either we give him 7 paperclips, or we flip a coin. If the coin comes up heads, we give him 18 paperclips. If it comes up tails, we give him nothing.

If Clippy doesn't take the bet, he gets 16 paperclips in total, so u=4. If Clippy takes the bet, he has 9 paperclips (u=3) with p=0.5 or 9+18=27 paperclips (u=5.20) with p=0.5. His utility expectancy is u=4.10, so he should take the bet.

Now, regardless of whatever he took the first bet (called B1 starting from now), we offer him a second bet (B2) : this time, he has to pay us 9 paperclips to enter. Then, we roll a 10-sided die. If it gives 1 or 2, we give him a jackpot of 100 paperclips, else nothing. Clippy can be in three states when offered the second deal :

  1. He didn't take B1. Then, he has 16 clips. If he doesn't take B2, he'll stay with 16 clips, and u=4. If takes B2, he'll have 7 clips with p=0.8 and 107 clips with p=0.2, for an expected utility of u=4.19.
  2. He did take B1, and lost it. He has 9 clips. If he doesn't take B2, he'll stay with 9 clips, and u=3. If takes B2, he'll have 0 clips with p=0.8 and 100 clips with p=0.2, for an expected utility of u=2.
  3. He did take B1, and won it. He has 27 clips. If he doesn't take B2, he'll stay with 27 clips, and u=5.20. If takes B2, he'll have 18 clips with p=0.8 and 118 clips with p=0.2, for an expected utility of u=5.57.

So, if Clippy didn't take the first bet or if he won it, he should take the second bet. If he did take the first bet and lost it, he can't afford to take the second bet, since he's risking a very bad outcome : no more paperclips, not even a single tiny one !

And the devil "time" comes in...

Now, let's make things a bit more complicated, and realistic. Before we were running things fully sequentially : first we resolved B1, and then we offered and resolved B2. But let's change a tiny bit B1. We don't flip the coin and give the clips to Clippy now. Clippy tells us if he takes B1 or not, but we'll wait one day before giving him the clips if he didn't take the bet, or before flipping the coin and then giving him the clips if he did take the bet.

The utility function of Clippy doesn't involve time, and we'll consider it doesn't change if he gets the clips tomorrow instead of today. So for him, the new B1 is exactly like the old B1.

But now, we offer him B2 after Clippy made his choice in B1 (taking the bet or not) but before flipping the coin for B1, if he did take the bet.

Now, for Clippy, we only have two situations : he took B1 or he didn't. If he didn't take B1, we are in the same situation than before, with an expected utility of u=4.19.

If he did take B1, we have to consider 4 possibilities :

  1. He loses the two bets. Then he ends up with no paperclip (9+0-9), and is very unhappy. He has u=0 utilons. That'll arise with p=0.4.
  2. He wins B1 and loses B2. Then he ends up with 9+18-9 = 18 paperclips, so u=4.24 with p=0.4.
  3. He loses B1 and wins B2. Then he ends up with 9-9+100 = 100 paperclips, so u=10 with p = 0.1.
  4. He wins both bets. Then he gets 9+18-9+100 = 118 paperclips, so u=10.86 with p=0.1.

At the end, if he takes B2, he ends up with an expectancy of u=3.78.

So, if Clippy takes B1, he then shouldn't take B2. Since he doesn't know if he won or lost B1, he can't afford the risk to take B2.

But should he take B1 at first ? If, when offered to take B1, he knows he'll be offered to take B2 later on, then he should refuse B1 and take B2, for an utility of 4.19. If, when offered B1, he doesn't know about B2, then taking B1 seems the more rational choice. But once he took B1, until he knows if he won or not, he cannot afford to take B2.

The Python code

For people interested about those issues, here is a simple Python script I used to fine tune that numerical parameters of  double-bet issue so my numbers lead to the problem I was pointing to. Feel free to play with it ;)

A hunter-gatherer tale

If you didn't like my Clippy, despite him being cute, and purring of happiness when he sees paperclips, let's shift to another tale.

Daneel is a young hunter-gatherer. He's smart, but his father committed a crime when he was still a baby, and was exiled from the tribe. Daneel doesn't know much about the crime - no one speaks about it, and he doesn't dare to bring the topic by himself. He has a low social status in the tribe because of that story. Nonetheless, he's attracted to Dors, the daughter of the chief. And he knows Dors likes him back, for she always smiles at him when she sees him, never makes fun of him, and gave him a nice knife after his coming-of-age ceremony.

According to the laws of the tribe, Dors can chose her husband freely, and the husband will become the new chief. But Dors also have to chose a husband that is accepted by the rest of the tribe, if the tribe doesn't accept the leadership, they could revolt, or fail to obey. And that could lead to disaster for the whole tribe. Daneel knows he has to raise his status in the tribe if he wants Dors to be able to chose him.

So Daneel wanders further and further in the forest. He wants to find something new to show the tribe his usefulness. That day, going a bit further than usual, he finds a place which is more humid than the forest the tribe usually wanders in. It has a new kind of trees, he never saw before. Lots of them. And they carry a yellow-red fruit which looks yummy. "I could tell about that place to the others, and bring them a few fruits. But then, what if the fruit makes them sick ? They'll blame me, I'll lose all chances... they may even banish me. But I can do better. I'll eat one of the fruits myself. If tomorrow I'm not sick, then I'll bring fruits to the tribe, and show them where I found them. They'll praise me for it. And maybe Dors will then be able to take me more seriously... and if I get sick, well, everyone gets sick every now and then, just one fruit shouldn't kill me, it won't be a big deal". So Daneel makes his utility calculation (I told you he was smart !), finds a positive outcome. So he takes the risk, he picks one fruit, and eats it. Sweet, a bit acid but not too much. Nice !

Now, Daneel goes back to the tribe. On the way back, he got a rabbit, a few roots and plants for the shaman, an average day. But then, he sees the tribe gathered around the central totem. In the middle of the tribe, Dors with... no... not him... Eto ! Eto is the strongest lad of Daneel's age. He wants Dors too. And he's strong, and very skilled with the bow. The other hunters like him, he's a real man. And Eto's father died proudly, defending the tribe's stock of dried meat against hungry wolves two winters ago. But no ! Not that ! Eto is asking Dors to marry him. In public. Dors can refuse, but if she does with no reason, she'll alienate half of the tribe against her, she can't afford it. Eto is way too popular.

"Hey, Daneel ! You want Dors ? Challenge Eto ! He's strong and good with the bow, but in unarmed combat, you can defeat him, I know it.", whispers Hari, one of the few friends of Daneel.

Daneel starts thinking faster he never did. "Ok, I can challenge Eto in unarmed combat. If I lose, I'll be wounded, Eto won't be nice with me. But he won't kill or cripple me, that would make half of the tribe to hate him. If I lose, it'll confirm I'm physical weak, but I'll also win prestige for daring to defy the strong Eto, so it shouldn't change much. And if I win, Dors will be able to refuse Eto, since he lost a fight against someone weaker than him, that's a huge win. So I should take that gamble... but then, there is the fruit. If the fruit gets me sick, in addition of my wounds from Eto, I may die. Even if I win ! And if I lose, get beaten, and then gets sick... they'll probably let me die. They won't take care of a fatherless lad who lose a fight and then gets sick. Too weak to be worth it. So... should I take the gamble ? If Eto waited just one day more... Or if only I knew if I'll get sick or not..."

The key : information loss

Until Clippy knows ? If Daneel knew ? That's the key of risk aversion, and why a perfect utility maximizer, if he has a concave utility function in at least some aspects, should still have some risk aversion. Because risk comes with information loss. That's the difference between the timeless double-bet and the one with one day of delay for Clippy. Or the problem Daneel got stuck into.

If you take a bet, until you know the outcome of your bet, you'll have less information about the state of the world, and especially about the state that directly concerns you, than if you chose the safe situation (a situation with a lower deviation). Having less information means you're less free to optimize.

Even a perfect utility maximizer can't know what bets he'll be offered, and what decisions he'll have to take, unless he's omniscient (and then he wouldn't take bets or risks, but he would know the future - probability only reflects lack of information). So he has to consider the loss of information of taking a bet.

In real life, the most common case of it is the non-linearity of bad effects : you can lose 0.5L of blood without too much side-effects (drink lots a water, sleep well, and next day you're ok, that's what happens when you go give your blood), but if you lose 2L, you'll likely die. Or if you lose some money, you'll be in trouble, but if you lose the same amount again, you may end up being kicked from you house since you can't pay the rent - and that'll be more than twice as bad as the initial lost.

So when you took a bet, risking to get a bad effect, you can't afford to take another bet (even with, in absolute, a higher gain expectancy), until you know if you won or lose the first bet - because losing them both means death, or being kicked from your house, or ultimate pain of not having any paperclip.

Taking a bet always as a cost : it costs you part of your ability to predict, and therefore to optimize.

A possible solution

A possible solution to that problem would be to consider all possible decisions you may to take while in the time period when you don't know if you lost or won your first bet, ponder them with the probability of being offered those decisions, and their possible outcomes if you take the first bet and you don't. But how do you compute "their possible outcomes" ? That needs to consider all the possible bets you could be offered during the time required for the resolution of your second bet, and their possible outcomes. So you need to... stack overflow: maximal recursion depth exceeded.

Since taking a bet will affect your ability to evaluate possible outcomes in the future, you've a "strange loop to the meta-level", an infinite recursion. Your decision algorithm has to consider the impact the decision will have on the future instances of your decision algorithm.

I don't know if there is a mathematical solution to that infinite recursion that manages to make it converge (like you can in some cases). But the problem looks really hard, and may not be computable.

Just factoring an average "risk aversion" that penalizes outcome which involve a risk (and the more you've to wait to know if you won or lose, the higher the penalty) sounds more a way to fix that problem than a bias.

Less Wrong and non-native English speakers

28 kilobug 06 November 2011 01:37PM

Hello Less Wrongers.

I'm still relatively new to the LW community, but I would like to share with you a few comments and ideas for making LW a better place for non-native English speakers.

There are two classes of people among non-native English speakers (of course, those boundaries are fuzzy) :

  1. People who, like me, are relatively fluent in English, but not who don't have the same fluency at English as natives do.
  2. People who don't speak much English at all.

The problems are of course different between 1. and 2., but yet I can see ways to improve things to both categories.

Moderately fluent English speakers

Being a member of 1., here are my feelings after a few months of lurking and then trying to participate a bit in LW, from my own French pov :

  1. LW is still quite US-centric in many ways. That's not much of a problem, at least for me who is used to dealing with US citizen from IRC or other Internet places, but it still something to keep in mind. Political question of Yvain's census/survey (please Yvain don't take that personally, overall you're doing a great and useful thing with that survey, so thanks to you), your  is a clear example of that, but it's much more general.
  2. Writing an article on LW is not easy for non-native English speakers. I tried twice, and twice I got many remarks about my English skills. I don't take them badly, thanks for those who took the time to point to my mistakes and explain them so I can improve, but still, it feels like it's harder to participate.

I don't have any magical solution from 1., except for anyone to try to be more careful when stating things which are culture-dependant, but it's part of the most general problem of excepting short inferential distances.

For 2., I'm wondering if it would be possible to have some LW to volunteer to review articles done by non-native English speakers, and improve the English quality, before the article is published to LW in general. Do you think the idea is good overall ? Would any of you volunteer to do that ? If so, it would be nice to include a paragraph about it, or at least a link to a page explaining the modality (how to submit an article to that team, ...), on the Welcome to Less Wrong page.

Non-English speakers

I don't think non-English speakers (or people with only basic English skills) can reasonably participate on LW itself, of course. But there are ways to still be able to offer them ways to become stronger, I'm thinking about translation.

Right now I'm helping Adrien with the French translation of HP:MoR. There are also attempts to translate some parts of the Sequences into other languages. In the mirror way of the "having native English speaker to help correct the English of non-native", us the non-native can help by participating to the various translation efforts. But that give raise to several questions :

  1. What are the legal issues about translating HP:MoR and Sequences ? Since Eliezer is linking to the translations of HP:MoR from his fanfiction.net page, I guess he approves of them. But what of the Sequences ? It would be nice to have some official stance from him and other people writing in the Sequences to know what they feel about translations. I'm not a fan of the copyright system in general, but I still would consider an utter lack of respect to someone to translate his work against his will.
  2. How can the team be coordinated, and how can newcomers to Less Wrong know that efforts are underway and requiring help ? Once again, I think it would be nice to have some page (maybe on the wiki ?) with the undergoing efforts, who is participating in which, how to contact them, and have a link to it from the Welcome to Less Wrong page.

Any opinion on those suggestions ? Any volunteer for joining some of the teams ? Anyone from "the staff" who could answer about the legal issues, and about the opportunity of including those pointers in the "Welcome to Less Wrong" page ?

Why would we think artists perform better on drugs ?

13 kilobug 30 October 2011 02:01PM

Introduction

It is common knowledge that many artists have used drugs (alcohol, opiates, cannabis, LSD, ...) and that this account for part of their creativity. This common knowledge is usually opposed to people advocating rationality in sentences like "but with only your rationality, we wouldn't have much art", "you need chaos to make art" or even "the best artists were that great because they were irrational".

Eliezer partly addressed the issue in the lawful intelligence Sequence. While this Sequence is very interesting, I feel it didn't completely address the issue (unlike most of the Sequences). My hypothesis is that it's mostly focused on what's important to building a Friendly AI (which is a worthy goal, this should not be taken as a criticism), not so much as explaining creativity in actual humans. So I'm writing an article with my current thoughts on the topic, and I would welcome any additional argument, hypothesis, research paper, ... that anyone from the LW community can point me to. This article is not supposed to come to any definitive conclusion, but to show my current state of thinking on that issue. I hope to both give and receive in writing it.

Reasons for which it could be an illusion

Availability bias

The first question to ask about "it is a common knowledge that many artists were using drugs" is : is this common knowledge true, or not, and if not, why do so many people believe something which is false ?

Availability bias comes will full power on this issue : when we hear that a given artist (musician, writer, painter, ...) was taking drugs, we add a "drug addict" tag to him. Or more accurately we create a link between the "drug addict" node and his node in our belief network. When asked about artists who did take drugs, we can easily state many names : for example Hemingway, Van Gogh, the Beatles. When asked about artists who didn't take drugs... well, we usually don't have "did not take drug" node in our belief networks, and no easy way to say that Asimov or Bach didn't take drugs.

Even when doing specific research, we can know with almost absolute certainty that Hemingway was drinking a lot of alcohol, but not so confidently that Asimov didn't. It's easier to be sure of the existence of something, than to be sure of its non-existence.

Reverse causality

The second question, if even after considering the affect of availability bias, it still seems than artists take drugs more often than average, is to ask about which sens the causality flows. Statistical correlation points to a causality, but doesn't tell you which sense is the causality, nor if it's direct or indirect.

There can be many reasons for which the causality works backwards : someone is not a good artist because he takes drug, but he takes drugs because he is an artist.

The lifestyle of a professional artist is usually different from the lifestyle of most other people. They usually don't have to wake up at 7 to be at work at 8, since they can work at any time. They also tend to be either very poor (many artists were only praised and recognized after being dead) or very rich (for the few who reach success while they are still alive). And we know that very poor people tend to fall on alcohol more often, while very rich people tend to use more frequently some of the very expensive drugs like cocaine.

Being an artist also usually induces a higher uncertainty about the future than with most regular jobs, which may trigger the use of drugs to make the angst easier to withstand.

Common cause

Apart from direct causality one way or another, a statistical correlation can also indicate that is a hidden common cause between the two phenomenas. If artists take drugs more often, it could be because there is a common reason that pushes people to both by a great artist and to take drugs.

Many reasons can be invoked that way, due to superexponential hypothesis space. I'll risk to be privileging the hypothesis but I can name a few. For example someone with an overdevelop emotional sensitivity could be both great at writing art able to call to our emotions and be more tempted to use drugs as relief from over-experienced negative emotions. Or someone who happens to be an outcast can be more likely to perform art (since it is usually a solitary work, not a team work) and at the same time use more drugs to escape from the pain of being an outcast.

So, where do we stand now ?

When faced with a statement such as "artists take drugs more often than average, so drugs help creativity" we can emit 4 different classes of hypothesis :

  1. The initial statement is wrong, artists don't take more drugs than average.
  2. Artists take more drugs than average, but the causality is reverse (it's being an artist that make you take drugs, not the other way around).
  3. Artists take more drugs than average, but that's because of a common factor that increases likelihood of taking drugs and of making great art, not the drugs themselves increasing artistic creativity.
  4. This is true, for a reason or another, drugs help creativity.

We saw some possible reasons for 1., 2. and 3. Some of them seem to be very real to me, especially the availability bias, but I do not think they totally account for the facts.

As much as I would love to be able to stop here and say that drugs and chaos play no positive role in creativity, that creativity is purely lawful and rational, I fear that would be wishful thinking and refusing to attack my belief's weak points. To state it more lightly : my D&D alignment could very well be lawful-good (as my friends tease me it is), but that shouldn't prevent me from admitting that chaos play a positive role somewhere if it actually does.

Reasons for which it could be real

Chaos and optimization

Generating great art can be seen as an optimization process. The actual function that evaluate a piece of art may be very complex, partly depending of the recipient, and its formalization unknown, but it can still be considered an optimization process : generating a book, or a painting, or a song that scores very high in most people's evaluation function.

In general, chaos is not an optimization process. Adding chaos to an optimization process usually makes it worse. But there are known counter-examples, where an imperfect optimization process will gain from a slight controlled increase of chaos.

Lawfully controlled chaotic optimization

The first known example is the first optimization process ever : evolution. Evolution involves too part : mutations which are chaotic, done at random, and natural selection which is lawful and selects the few evolutions that happened to be positive. The Roger Zelazny picture of the universe being an equilibrium between Order and Chaos may come from that pattern. If you increase chaos too much in natural selection, the information will not be replicated enough from generation to generation, and not much optimization will occur. But if you don't have any mutation, if you remove all the chaos, the process will freeze too.

I remember an experiment from biology lessons in high school : take two small boxes of glass, put cotton with water and sugar at the bottom. Take some bacteria, and but the bacteria on side of the box. Take an antibiotics pill, and put it on the other side of the box. Put box A in safe storage. Put box B in safe storage too, but every day, expose it to a small amount of UV light. The bacteria of box A will quickly spread on the cotton, but will not go anywhere close the antibiotics pill. The bacteria of box B will start doing the same, but after two or three weeks, they conquer even the antibiotics area. After a longer time period, box A bacteria will also overcome the antibiotics, but it'll take them much longer. The UV light increased the mutation rate, and sped up the optimization process of evolution. But only a very small dose of UV light does that, overdose it, and the bacteria B will all die.

That's what I call "lawfully controlled chaotic optimization" : there is a lawful control process (here, natural selection) that selects randomly tried solutions. That's something that can directly apply to artists : the control process (be it the filter of editors/publishers, or the filter of public reaction) is, to a point, lawful, but the process that generate solutions could benefit from a slight increase in chaos. Or more exactly, the combined (generator + filter) algorithm could perform better with a slightly more chaotic generator. To retake Eliezer's definition of creativity, which was "the creative surprise is the idea that ranks high in your preference ordering but low in your search ordering", adding chaos to preference ordering would be pointless, but adding chaos to the search ordering can allow more creative surprise to happen in a given finite time.

There is still a major difference between the two processes described here (evolution and human creativity) : evolution uses a fully random generator, whereas the human brain has a great ability of generating non-random designs, allowing a much faster improvement rate. You'll never get a book of Hemingway or a painting of Van Gogh by randomly selecting letters or randomly putting paint on a canvas. The chance of that is too infinitesimally low. So the generator will have to stay mostly lawful. Hemingway used words and respected the rules of grammar. Van Gogh painted something which look very like real sunflowers. A fully chaotic process would never produce anything near their masterpieces even given billions of years. So artistic creativity must be mostly lawful, even for generating its hypothesis to select from.

As spotted by Vaniver in the comments, Hemingway himself said something very similar to that thesis : "Write drunk; edit sober."

Avoiding local minimal

One big problem of optimization processes is local minimal. Most of the naive optimization process, like a gradient descent, will get trapped into local minima. Let's have a look at that curve (burrowed from Wikipedia) :

Local and global maxima, from Wikipedia

If you start a naive optimization algorithm in the right part of the curve, you'll very likely end up in the local minimum, while the global minimum would rank much better in your preferences. Adding some form of controlled chaos to the algorithm is an easy way to increase the chance of reaching the global minimum, even in much more complex setups than this simple curve.

For a relatively broad class of problems, like selecting the best position of nodes to minimize the length of edges when doing a bitmap representation of a graph structure, an algorithm which works quite well and is simple to code is the simulated annealing algorithm, which works by doing local optimizations, but having a global temperature which adds chaos (the higher the temperature, the more random is the process). The temperature itself decreases with the process, and ultimately reaches 0 (pure lawful optimization).

Such methods are of course "dirty hacks", that are used only when the problem is too complicated and we don't have a purely lawful algorithm that gives the answer, or (most of the time) when we do have one, but with an exponential complexity, meaning we can't run it in real life.

The same idea applies to human creativity : chaos wouldn't be needed, nor useful, if we had a fully working algorithm to write the best books or songs or make the best paintings. But since we don't, using a purely lawful process has a risk (but yes, only a risk) of getting us stuck into a local minimum - improving the methods of the previous generation of artists, but not inventing brand new styles of art. This is a similar concept to the "jumping out of the system" described by Douglas Hofstadter and analyzed by Eliezer. JOOTSing is escaping a local minimum. It's escaping the safe warmth of the valley, climbing the cold and dangerous mountain top, to find another, more fertile valley on the other side. That requires to violate the rules of "staying into the safe and warm valley".

(Note : there is somehow an analogy between the use of drugs and the simulated annealing : drugs induce a state of high chaos, which then slowly goes down as the drug effects disappears. Or at least I was told so, since I never tried personally. But that seems a surface analogy to me, so I won't give it much credit).

Inhibitions and art

Or another way to consider it is to look at is inhibitions : the human mind contains a process that'll check your actions (painting and writing in that case, but applies more broadly) and sometimes say "no, don't do that, you'll look as a fool". Those inhibitions are usually here to protect us from botching in social situations. But they are (as most of the human brain) imperfectly calibrated, and will tend to repress anything that goes out of the current norm. Lowering those inhibitions increases the risk of botching - but also the chance of doing something awesome.

This points to much more general pattern, which applies when what matters is not improving your average gain, but your chance of being one of the few best. Consider you've a task to do, and two ways to achieve it. Way A is quite classical, and doesn't involve much risk. Way B is much less proven, and involves much risk of doing both better and worse. Being a role player, I usually use dice rolls to model those kinds of process. Let's say process A is 20d10. That means, rolling 20 times a 10-sided die, and doing the sum. This will give an expected value of 110, with only 1% of the rolls above 140. Now process B is 2d100 (rolling 2 times a 100-sided die and doing the sum). This will give a lower expected value, of 101 instead of 110. But with 18% of the rolls above 140. Here is a picture of the two process (way A in green, way B in red) generated with a quick Python script :

2d100 (red) vs 20d10 (green)

If what matters is doing your best in average (your score at the task will directly map to an amount of money between $2 and $200), then the best choice is to look only at the expected value of A and B, and select the one which has the best expected value, so A in this case, as you can see, the green curve peaks at a higher value.

But if what matters is not doing the best in average, but being the best : 100 people are performing the task, and the best will take the prize, the rest won't have anything. Then, you except one of the 100 to be above 140, even if they all use way A. So for yourself, if you use way A, you only have 1 chance in 100 to be above 140. If you use way B, you've 18 chances in 100 of beating the 140 mark. Looking at the curve, there is much bigger blob of the red curve that goes to very high values.

When looking at arts, we don't regard the average. Countless people write books or paint. Almost everyone at least tried once. What history remembers are the few best of their time. Not those who did better in average, but those who manage to do better than most of their peers. Those to the right of the picture, in which the amplitude of the green curve is nearly void, but the red curve still exists.

The complexity of testing certain hypothesis

I emitted many hypothesis in this article, to try to explain the common knowledge that "so many great artists take drugs", and more generally to look into the reasons for which chaos can, in some cases, improve a result.

All those hypothesis seem totally plausible to me - and I would say that they all play a role in the process. But saying "everything plays a role" is not saying much, a graph with all possible edges contains as much informations as a graph with no edge. What would be require now is to consider how much each hypothesis contributes to the result - and then, probably one or two will account for most.

But how can we setup such a test ? In physics, doing experiments is relatively easy. It can costs a lot like building the LHC or sending the Hubble space telescope in orbit, but still, devising experiments is relatively easy. In social sciences, it's often much harder. Most social science experiments are done on a panel of test subjects (with a control group, ...). But right now we are speaking of the best artists. How can we build such a panel ? Defining who are the best artists is a very hard task. And then, getting them to participate in studies...

The simplest hypothesis to test, the availability bias, would require a procedure like (numbers can be adjusted) :

  1. Take 1000 people at random, from various ages, social classes and background.
  2. Ask for each of them to name the 10 artists they like the most (without of course mentioning the purpose of the experiment).
  3. For each artist nominated by at least 4 persons, look if that artist did take drugs.
  4. Compare with the average drug use.

But even that is not without troubles : for 3., how can you be sure an artist didn't take drug secretly, especially in time/places where drug use is prohibited or frown upon ? For 4., how do you ponder for the variation in drug use depending of the place/time ?

Does anyone know of such a study (I couldn't find any, but I'm not well versed in the art for looking for social science studies) ?

For the other hypothesis, testing them becomes even harder.

Conclusion

As Eliezer explained, pure chaos cannot lead to anything but static on a TV screen. Any optimization process, and art is one, requires a lawful part. But as I showed, for several reasons, an imperfect optimization process may perform better with a limited amount of added chaos. Since the human brain is an imperfect optimization process, it would not be surprising that in the purpose of creating awesome pieces of art in a limited time, some added chaos can help. But on the other hand, there are other reasons for which there could be a common knowledge that "artistic creativity requires some chaos" even if it were not true. And it is very hard to tell apart the various reasons.

But even if some amount of chaos can help in generating exceptionally awesome pieces of art, it should not shadow the fact that the lawful part of process is absolutely required, and even the most important one; nor that chaos can only be useful when the optimization is itself imperfect. Improving the quality of the optimization process (by, for example, raising the sanity waterline or understanding better the human brain) would lower the need of chaos to generate the same awesomeness.

PS : I post that to "Less Wrong discussion", for initial review and because it's half-way between a "real" article and a call for discussion on the topic. Depending of feedback, I hope to repost it to "main Less Wrong", hopefully improved from the feedback.

View more: Next