Test of: Decision Fatigue, Rationality, and Akrasia.

Shortly before the Summit, Alexandros posted a short discussion post wondering whether rationality training might cause akrasia by prompting folks to make more decisions using deliberate, conscious, "system II" reasoning (instead of rapid, automatic, "system I" heuristics) and, thereby, causing decision fatigue.

This conjecture sounded interesting to me, and I'd wondered similar things myself, so I put up a poll to gather data. 

Procedure

I put this poll on up LW, asking a number of questions that I hoped bore on: (1) akrasia levels; (2) how the person's akrasia levels had shifted since they came to LW; and (3) how many decisions they made via deliberate "system II" processing.  70 LW-ers completed the survey in time to get included in my data analysis; perhaps because the survey was on Discussion, these were mostly folks who'd been on LW for a while; median response to "months since you started reading LW/OB" was 19.

I also wanted a control group so as to distinguish real LW anomalies from random bugs in most humans' self-reporting architecture.  I tried to ask Reddit, but got only 7 responses; then I tried Mechanical Turk and received my full 100 desired responses... but it is hard to be sure that Mechanical Turk data is from real humans.

Validity of “akrasia/procrastination” self-reports

It would be nice if self-reported akrasia levels correlated with (lack of) success with common goals, such as income, exercise, and living in a non-filthy house.  To assess participants' akrasia, I asked the following questions:

• 2. As a kid, how much trouble did you have with procrastination?[1]
• 3. How much trouble do you have now with procrastination?
• 4. Have you had any bills go to a collection agency in the last six months?
• 5. When was your kitchen floor most recently cleaned?
• 6. How many times have you exercised in the last 7 days?
• 11. What was your college GPA?
• 12. What was your high school GPA?
• 18. What's your present income, in dollars per year?

I replaced all responses with z-score estimates, replaced income with "income controlled for age", and started looking at correlations.  Of these items, #3 and #4 failed to correlate with the other "akrasia" questions[2], so I discarded them and noted that "akrasia" might be less of a dimension than I was hoping.  The others all correlated in the expected directions, although weakly, as shown:

(This table contains all correlations between the listed variables that occurred with p-value < .25, together with the associated p-value; note that the dataset was fairly small, so the absence of a statistically significant correlation does not necessarily imply the absence of a correlation.)

LW (thinks it is?) more akratic than average

On average, LW-er survey participants regarded themselves as having more trouble than usual with procrastination:

Mechanical Turk-ers regarded themselves as more average (and, especially, regarded their childhoods as more average), suggesting that this isn't just a "everyone thinks they have the most trouble" effect (though such effects do exist)[3]:

My guess is that LWers' perceptions of having more trouble than average with procrastination represents a real difference in folks' getting-things-done powers, and not just a difference in self-image.  One piece of data supporting this is that LW-ers report higher than baseline rates of autism/asperger's: 5% of survey participants, compared about one percent of the general population.  20% of LW-ers report having ever had a depression diagnosis, which also seems to be somewhat above baseline rates.  (Depression and autism both correlate with difficulty getting things done.)

Improvement over time?

LW-ers report improving over time. Moreover, they report improving more *since finding LW* than in an equal-sized chunk of time before finding LW; the difference in reported improvement over the two intervals is fairly small, but is statistically significant at the p=.001 significance level.

Mechanical Turk-ers do not report improving, and are not rosier about their last two years than about the two years before that  -- suggesting that the above isn't just due to a bug in the human self-assessment system:

However, reported LW-er akrasia levels do not decrease with respondent age, which pulls against the thesis that LWers start out akratic but generically improve as we get older.  They also do not decrease with "months since discovering LW/OB", which pulls against the thesis that LW helps.

Deliberate decision making not harmful

To test the conjecture that excessive conscious decision-making (over-reliance on deliberate, conscious, "System II" reasoning instead of on automatic heuristics) causes akrasia, I asked:

• 7. When you go grocery shopping, how often do you think carefully about which product to buy, vs.  just grabbing something and putting it in your cart?
• 8. When you sit down to do work, how often do you think carefully about what subtasks to do and/or how to do them, vs. just doing things?
• 19. Anne is looking at Bob, and Bob is looking at Carol.  Anne is married; Carol is unmarried.  Is a married person looking at an unmarried person? (Yes / No / Can't be determined)

Question 19 is a question from the research literature that is designed to test individuals' tendency to engage in "fully disjunctive reasoning", and, thus, to assess at least one aspect of folks' tendency to use system II reasoning in preference to automatic system I heuristics.  The CRT is more standardly used for such measurement, but previous testing had indicated that LW-ers mostly hit the ceiling on the CRT, so I used the more difficult Anne question instead.

None of these questions correlated positively (to any discernably above-chance extent) with the indicators of akrasia. In fact, question 8 had a significant negative correlation with current self-reported procrastination levels (correlation -.26, p-value .03), suggesting that a tendency to deliberately choose one's work tasks may help procrastination/akrasia, and does not harm it.  

On the other hand, questions 7, 8, and 19 also did not correlate strongly with one another; so it is possible that these are just not good indicators of folks' degree of reliance on deliberate, conscious, "System II" decision-making.  Nevertheless, when you combine these fact that none 7, 8, or 19 indicated procrastination with the reported improvement after LWers find LW, it seems to together constitute reasonable evidence against LW and rationality training being harmful, or at minimum against them being sufficiently harmful to show up with such datasets.

Other correlations

Most of the remaining variables correlated with one another in the manner that common sense would suggest (e.g., being in school correlated with being young).  Still, for completeness, here are all correlations among the questions that appeared correlated with a p-value <0.03; note that since I compared 25 variables with one another, we should expect about (25*24/2)*.03 = 9 correlations at this significance level, and 0 to 1 correlations at the p=.001 significance level just by chance. [4]

For ease of scanning, correlations that I personally found interesting are in bold.  "Income" is instead "income adjusted for age and student status"; I adjusted kitchen cleanliness for student status as well.

The correlations:

• High reported levels of procrastination as a kid correlated with:
  • Reported current procrastination levels (corr. coef. = .3, p-value = .02);
  • Low high school GPA (c = .3, p=.01); 
  • ADD diagnosis (c=.3, p=.02);
  • Consuming coffee/tea/caffeine most days (now, not as a kid) (c=.3, p=.02)
  • Not being in school right now (c=.3, p=.005).
• High reported levels of current procrastination correlated with:
  • Procrastination as a kid (see above);
  • Lack of present exercise (c=.3, p=.02);
  • Working on something at random, instead of making deliberate choices as to what to work on (Q8 above) (c=.26; p=.03);
  • Reporting a lack of improvement, or a worsening, of procrastination since finding LW (c=.4, p<.001);
  • Diagnoses of depression and of "other" (c = .3, p= .01);
  • Unhappiness (c = .4, p=.0003);
  • Anxiety (c =.3 , p=.01).
• Having bills go to a collections agency correlated with:
  • Diagnoses of ADD, of autism/asperger's, and of depression (c=.5, .4, and .3 respectively; associated p-values were p=.0001, p=.0007, and p=.02 respectively). 
• Having a kitchen floor that had/hadn't been cleaned in the last month correlated with: nothing.
• Regular recent exercise correlated with:
  • Lack of reported current procrastination (see above);
  • Choosing items deliberately in the grocery store (c = .3, p=.01);
  • Consuming coffee/tea/caffeine most days (c = .3, p=.008).
• Choosing items deliberately in the grocery store correlated with: regular recent exercise (see above).
• Choosing work tasks and subtasks deliberately (vs. just doing things) correlated with:
  • Reported lack of current procrastination (see above);
  • Having never had a depression diagnosis (c =.3, p=.02).
• Reporting improved procrastination since finding LW correlated with:
  • Reported lack of current procrastination (see above);
  • Lack of depression diagnoses (c = .36, p=.003).
• Reporting improved procrastination in an equal-sized time period before finding LW correlated with: nothing.
• High college GPA correlated with:
  • High high school GPA (c = .4, p=.0006);
  • Being in school (c = .3, p=.01).
• High school GPA correlated with:
  • Lack of procrastination as a kid; high college GPA (see above)
  • Lack of "other" diagnoses (c = .3, p=.01);
  • Getting a lot of sleep (now, not as a kid) (c = .36, p=.002 ).
• IQ correlated with: nothing.
• Having "other" diagnoses correlated with:
  • Reported current procrastination; low high school GPA (see above);
  • Diagnoses of depression (c = .5, p<.0001);
  • Lack of sleep (c = .4, p=.0004);
  • Reported anxiety(c = .4, p=.0002).
• ADD diagnoses correlated with:
  • Reported childhood procrastination; bills sent to collection agencies (see above);
  • Diagnoses of autism/asperger's and depression (c = .55 and .44 respectively, p=.0001 for both).
• Autism/Asperger's diagnoses correlated with:
  • Bills sent to collection agencies; diagnoses of ADD (see above);
  • Diagnoses of depression (c = .36, p=.003).
• Depression diagnoses correlated with:
  • Reported current procrastination; bills sent to collection agencies; tendency to just do work without thinking about which tasks or subtasks to do; lack of reported procrastination improvement since finding LW; and diagnoses of ADD, autism/asperger's, and "other" (see above);
  • Reported unhappiness (c = .4, p=.0002);
  • Reported anxiety (c = .3, p=.01).
• Income correlated with: nothing. [After controlling for age, student status].
• Answering the Anne question correctly (Q19 above) correlated with: 
  • Consuming coffee/tea/caffeine most days (c=.34, p=.004).
• Time since the respondent started reading LW correlated with: nothing.
• Hours of sleep per night correlated with:
  • High college GPA; and not having "other" diagnoses (see above).
• Age correlated with: not being a student.
• Consuming coffee/tea/caffeine most days correlated with:
  • Procrastination in childhood; successfully getting exercise; answering the Anne question correctly (see above).
• Reported happiness correlated with:
  • Reported lack of current procrastination; and lack of depression diagnoses (see above);
  • Being a student (c=.3, p=.02).
• Reported anxiety correlated with:
  • Reported current procrastination; diagnoses of "depression" and of "other" (see above).
• Being a student correlated with: 
  • Lack of reported procrastination as a child; high college GPA; youth; happiness (see above).

Raw data

In case you want to play with the raw data yourself, here it is:

• The poll questions: for the LW poll; for the mechanical Turk poll.
• The raw responses: for the LW poll; for the mechanical Turk poll.
• The raw responses converted into numbers, in case you want to run correlations but don't want to mess with text: for the LW poll (note that this includes only the first 68 responses, which are all I analyzed above); for the mechanical Turk poll.

Given the importance of taking actions that actually relate to one's goals (for happiness, income, world-saving, you name it -- and, hence, for real rationality), further investigation here would be welcome, either via further polls on LW-ers or others, or, perhaps even more easily and usefully, via Google Scholar.

[1] Originally, I asked about "trouble with akrasia" (now and as a kid) rather than about "trouble with procrastination".  I edited the question after realizing I'd want a control group with non-LWers, and that that group would not reliably know the term "akrasia".  So, the LW-er responses are partly to one wording and partly to the other.

[2] Both items correlated strongly with student status; when I controlled for student status (subtracted out the constant necessary to remove the correlation), "have had late bills reported to credit agencies" correlated strongly with diagnoses of ADD, autism/Asperger's, and depression, but with nothing else; dirty kitchen floors correlated with nothing.

[3] This is further suggested by the fact that Mechanical Turk-ers may well *have* more akrasia than average at present; they are working on Mechanical Turk, and have fairly high numbers of depression diagnoses.

[4] There are 34 correlations at the p<.03 significance level, and 12 at the p<.001 significance level, which is more than we should expect by chance; this is not surprising, since of course e.g. being in school is correlated with being young, and so of course we see some non-chance correlations; the question is how many of the non-obvious correlations are just chance.