Introduction

It is common knowledge that many artists have used drugs (alcohol, opiates, cannabis, LSD, ...) and that this account for part of their creativity. This common knowledge is usually opposed to people advocating rationality in sentences like "but with only your rationality, we wouldn't have much art", "you need chaos to make art" or even "the best artists were that great because they were irrational".

Eliezer partly addressed the issue in the lawful intelligence Sequence. While this Sequence is very interesting, I feel it didn't completely address the issue (unlike most of the Sequences). My hypothesis is that it's mostly focused on what's important to building a Friendly AI (which is a worthy goal, this should not be taken as a criticism), not so much as explaining creativity in actual humans. So I'm writing an article with my current thoughts on the topic, and I would welcome any additional argument, hypothesis, research paper, ... that anyone from the LW community can point me to. This article is not supposed to come to any definitive conclusion, but to show my current state of thinking on that issue. I hope to both give and receive in writing it.

Reasons for which it could be an illusion

Availability bias

The first question to ask about "it is a common knowledge that many artists were using drugs" is : is this common knowledge true, or not, and if not, why do so many people believe something which is false ?

Availability bias comes will full power on this issue : when we hear that a given artist (musician, writer, painter, ...) was taking drugs, we add a "drug addict" tag to him. Or more accurately we create a link between the "drug addict" node and his node in our belief network. When asked about artists who did take drugs, we can easily state many names : for example Hemingway, Van Gogh, the Beatles. When asked about artists who didn't take drugs... well, we usually don't have "did not take drug" node in our belief networks, and no easy way to say that Asimov or Bach didn't take drugs.

Even when doing specific research, we can know with almost absolute certainty that Hemingway was drinking a lot of alcohol, but not so confidently that Asimov didn't. It's easier to be sure of the existence of something, than to be sure of its non-existence.

Reverse causality

The second question, if even after considering the affect of availability bias, it still seems than artists take drugs more often than average, is to ask about which sens the causality flows. Statistical correlation points to a causality, but doesn't tell you which sense is the causality, nor if it's direct or indirect.

There can be many reasons for which the causality works backwards : someone is not a good artist because he takes drug, but he takes drugs because he is an artist.

The lifestyle of a professional artist is usually different from the lifestyle of most other people. They usually don't have to wake up at 7 to be at work at 8, since they can work at any time. They also tend to be either very poor (many artists were only praised and recognized after being dead) or very rich (for the few who reach success while they are still alive). And we know that very poor people tend to fall on alcohol more often, while very rich people tend to use more frequently some of the very expensive drugs like cocaine.

Being an artist also usually induces a higher uncertainty about the future than with most regular jobs, which may trigger the use of drugs to make the angst easier to withstand.

Common cause

Apart from direct causality one way or another, a statistical correlation can also indicate that is a hidden common cause between the two phenomenas. If artists take drugs more often, it could be because there is a common reason that pushes people to both by a great artist and to take drugs.

Many reasons can be invoked that way, due to superexponential hypothesis space. I'll risk to be privileging the hypothesis but I can name a few. For example someone with an overdevelop emotional sensitivity could be both great at writing art able to call to our emotions and be more tempted to use drugs as relief from over-experienced negative emotions. Or someone who happens to be an outcast can be more likely to perform art (since it is usually a solitary work, not a team work) and at the same time use more drugs to escape from the pain of being an outcast.

So, where do we stand now ?

When faced with a statement such as "artists take drugs more often than average, so drugs help creativity" we can emit 4 different classes of hypothesis :

1. The initial statement is wrong, artists don't take more drugs than average.
2. Artists take more drugs than average, but the causality is reverse (it's being an artist that make you take drugs, not the other way around).
3. Artists take more drugs than average, but that's because of a common factor that increases likelihood of taking drugs and of making great art, not the drugs themselves increasing artistic creativity.
4. This is true, for a reason or another, drugs help creativity.

We saw some possible reasons for 1., 2. and 3. Some of them seem to be very real to me, especially the availability bias, but I do not think they totally account for the facts.

As much as I would love to be able to stop here and say that drugs and chaos play no positive role in creativity, that creativity is purely lawful and rational, I fear that would be wishful thinking and refusing to attack my belief's weak points. To state it more lightly : my D&D alignment could very well be lawful-good (as my friends tease me it is), but that shouldn't prevent me from admitting that chaos play a positive role somewhere if it actually does.

Reasons for which it could be real

Chaos and optimization

Generating great art can be seen as an optimization process. The actual function that evaluate a piece of art may be very complex, partly depending of the recipient, and its formalization unknown, but it can still be considered an optimization process : generating a book, or a painting, or a song that scores very high in most people's evaluation function.

In general, chaos is not an optimization process. Adding chaos to an optimization process usually makes it worse. But there are known counter-examples, where an imperfect optimization process will gain from a slight controlled increase of chaos.

Lawfully controlled chaotic optimization

The first known example is the first optimization process ever : evolution. Evolution involves too part : mutations which are chaotic, done at random, and natural selection which is lawful and selects the few evolutions that happened to be positive. The Roger Zelazny picture of the universe being an equilibrium between Order and Chaos may come from that pattern. If you increase chaos too much in natural selection, the information will not be replicated enough from generation to generation, and not much optimization will occur. But if you don't have any mutation, if you remove all the chaos, the process will freeze too.

I remember an experiment from biology lessons in high school : take two small boxes of glass, put cotton with water and sugar at the bottom. Take some bacteria, and but the bacteria on side of the box. Take an antibiotics pill, and put it on the other side of the box. Put box A in safe storage. Put box B in safe storage too, but every day, expose it to a small amount of UV light. The bacteria of box A will quickly spread on the cotton, but will not go anywhere close the antibiotics pill. The bacteria of box B will start doing the same, but after two or three weeks, they conquer even the antibiotics area. After a longer time period, box A bacteria will also overcome the antibiotics, but it'll take them much longer. The UV light increased the mutation rate, and sped up the optimization process of evolution. But only a very small dose of UV light does that, overdose it, and the bacteria B will all die.

That's what I call "lawfully controlled chaotic optimization" : there is a lawful control process (here, natural selection) that selects randomly tried solutions. That's something that can directly apply to artists : the control process (be it the filter of editors/publishers, or the filter of public reaction) is, to a point, lawful, but the process that generate solutions could benefit from a slight increase in chaos. Or more exactly, the combined (generator + filter) algorithm could perform better with a slightly more chaotic generator. To retake Eliezer's definition of creativity, which was "the creative surprise is the idea that ranks high in your preference ordering but low in your search ordering", adding chaos to preference ordering would be pointless, but adding chaos to the search ordering can allow more creative surprise to happen in a given finite time.

There is still a major difference between the two processes described here (evolution and human creativity) : evolution uses a fully random generator, whereas the human brain has a great ability of generating non-random designs, allowing a much faster improvement rate. You'll never get a book of Hemingway or a painting of Van Gogh by randomly selecting letters or randomly putting paint on a canvas. The chance of that is too infinitesimally low. So the generator will have to stay mostly lawful. Hemingway used words and respected the rules of grammar. Van Gogh painted something which look very like real sunflowers. A fully chaotic process would never produce anything near their masterpieces even given billions of years. So artistic creativity must be mostly lawful, even for generating its hypothesis to select from.

As spotted by Vaniver in the comments, Hemingway himself said something very similar to that thesis : "Write drunk; edit sober."

Avoiding local minimal

One big problem of optimization processes is local minimal. Most of the naive optimization process, like a gradient descent, will get trapped into local minima. Let's have a look at that curve (burrowed from Wikipedia) :

If you start a naive optimization algorithm in the right part of the curve, you'll very likely end up in the local minimum, while the global minimum would rank much better in your preferences. Adding some form of controlled chaos to the algorithm is an easy way to increase the chance of reaching the global minimum, even in much more complex setups than this simple curve.

For a relatively broad class of problems, like selecting the best position of nodes to minimize the length of edges when doing a bitmap representation of a graph structure, an algorithm which works quite well and is simple to code is the simulated annealing algorithm, which works by doing local optimizations, but having a global temperature which adds chaos (the higher the temperature, the more random is the process). The temperature itself decreases with the process, and ultimately reaches 0 (pure lawful optimization).

Such methods are of course "dirty hacks", that are used only when the problem is too complicated and we don't have a purely lawful algorithm that gives the answer, or (most of the time) when we do have one, but with an exponential complexity, meaning we can't run it in real life.

The same idea applies to human creativity : chaos wouldn't be needed, nor useful, if we had a fully working algorithm to write the best books or songs or make the best paintings. But since we don't, using a purely lawful process has a risk (but yes, only a risk) of getting us stuck into a local minimum - improving the methods of the previous generation of artists, but not inventing brand new styles of art. This is a similar concept to the "jumping out of the system" described by Douglas Hofstadter and analyzed by Eliezer. JOOTSing is escaping a local minimum. It's escaping the safe warmth of the valley, climbing the cold and dangerous mountain top, to find another, more fertile valley on the other side. That requires to violate the rules of "staying into the safe and warm valley".

(Note : there is somehow an analogy between the use of drugs and the simulated annealing : drugs induce a state of high chaos, which then slowly goes down as the drug effects disappears. Or at least I was told so, since I never tried personally. But that seems a surface analogy to me, so I won't give it much credit).

Inhibitions and art

Or another way to consider it is to look at is inhibitions : the human mind contains a process that'll check your actions (painting and writing in that case, but applies more broadly) and sometimes say "no, don't do that, you'll look as a fool". Those inhibitions are usually here to protect us from botching in social situations. But they are (as most of the human brain) imperfectly calibrated, and will tend to repress anything that goes out of the current norm. Lowering those inhibitions increases the risk of botching - but also the chance of doing something awesome.

This points to much more general pattern, which applies when what matters is not improving your average gain, but your chance of being one of the few best. Consider you've a task to do, and two ways to achieve it. Way A is quite classical, and doesn't involve much risk. Way B is much less proven, and involves much risk of doing both better and worse. Being a role player, I usually use dice rolls to model those kinds of process. Let's say process A is 20d10. That means, rolling 20 times a 10-sided die, and doing the sum. This will give an expected value of 110, with only 1% of the rolls above 140. Now process B is 2d100 (rolling 2 times a 100-sided die and doing the sum). This will give a lower expected value, of 101 instead of 110. But with 18% of the rolls above 140. Here is a picture of the two process (way A in green, way B in red) generated with a quick Python script :

If what matters is doing your best in average (your score at the task will directly map to an amount of money between $2 and $200), then the best choice is to look only at the expected value of A and B, and select the one which has the best expected value, so A in this case, as you can see, the green curve peaks at a higher value.

But if what matters is not doing the best in average, but being the best : 100 people are performing the task, and the best will take the prize, the rest won't have anything. Then, you except one of the 100 to be above 140, even if they all use way A. So for yourself, if you use way A, you only have 1 chance in 100 to be above 140. If you use way B, you've 18 chances in 100 of beating the 140 mark. Looking at the curve, there is much bigger blob of the red curve that goes to very high values.

When looking at arts, we don't regard the average. Countless people write books or paint. Almost everyone at least tried once. What history remembers are the few best of their time. Not those who did better in average, but those who manage to do better than most of their peers. Those to the right of the picture, in which the amplitude of the green curve is nearly void, but the red curve still exists.

The complexity of testing certain hypothesis

I emitted many hypothesis in this article, to try to explain the common knowledge that "so many great artists take drugs", and more generally to look into the reasons for which chaos can, in some cases, improve a result.

All those hypothesis seem totally plausible to me - and I would say that they all play a role in the process. But saying "everything plays a role" is not saying much, a graph with all possible edges contains as much informations as a graph with no edge. What would be require now is to consider how much each hypothesis contributes to the result - and then, probably one or two will account for most.

But how can we setup such a test ? In physics, doing experiments is relatively easy. It can costs a lot like building the LHC or sending the Hubble space telescope in orbit, but still, devising experiments is relatively easy. In social sciences, it's often much harder. Most social science experiments are done on a panel of test subjects (with a control group, ...). But right now we are speaking of the best artists. How can we build such a panel ? Defining who are the best artists is a very hard task. And then, getting them to participate in studies...

The simplest hypothesis to test, the availability bias, would require a procedure like (numbers can be adjusted) :

1. Take 1000 people at random, from various ages, social classes and background.
2. Ask for each of them to name the 10 artists they like the most (without of course mentioning the purpose of the experiment).
3. For each artist nominated by at least 4 persons, look if that artist did take drugs.
4. Compare with the average drug use.

But even that is not without troubles : for 3., how can you be sure an artist didn't take drug secretly, especially in time/places where drug use is prohibited or frown upon ? For 4., how do you ponder for the variation in drug use depending of the place/time ?

Does anyone know of such a study (I couldn't find any, but I'm not well versed in the art for looking for social science studies) ?

For the other hypothesis, testing them becomes even harder.

Conclusion

As Eliezer explained, pure chaos cannot lead to anything but static on a TV screen. Any optimization process, and art is one, requires a lawful part. But as I showed, for several reasons, an imperfect optimization process may perform better with a limited amount of added chaos. Since the human brain is an imperfect optimization process, it would not be surprising that in the purpose of creating awesome pieces of art in a limited time, some added chaos can help. But on the other hand, there are other reasons for which there could be a common knowledge that "artistic creativity requires some chaos" even if it were not true. And it is very hard to tell apart the various reasons.

But even if some amount of chaos can help in generating exceptionally awesome pieces of art, it should not shadow the fact that the lawful part of process is absolutely required, and even the most important one; nor that chaos can only be useful when the optimization is itself imperfect. Improving the quality of the optimization process (by, for example, raising the sanity waterline or understanding better the human brain) would lower the need of chaos to generate the same awesomeness.

PS : I post that to "Less Wrong discussion", for initial review and because it's half-way between a "real" article and a call for discussion on the topic. Depending of feedback, I hope to repost it to "main Less Wrong", hopefully improved from the feedback.