Follow-up to: Announcing the Quantified Health Prize

I am happy to announce that Scott Alexander, better known on Less Wrong as Yvain, has won the first Quantified Health Prize, and Kevin Fischer has been awarded second place. There were exactly five entries, so the remaining prizes will go to Steven Kaas, Kevin Keith and Michael Buck Shlegeris.

The full announcement can be found here until the second contest is announced, and is reproduced below the fold. 

While we had hoped to receive more than five entries, I feel strongly that we still got our money’s worth and more. Scott Alexander and Kevin Fischer in particular put in a lot of work, and provided largely distinct sets of insight into the question. In general, it is clear that much time was usefully spent, and all five entries had something unique to contribute to the problem.

We consider the first contest a success, and intend to announce a second contest in the next few weeks that will feature multiple questions and a much larger prize pool.

Discussion of all five entries follows:Place ($500):

5th Place ($500): Full report

Steven Kaas makes a well-reasoned argument for selenium supplementation. That obviously wasn't a complete entry. It's very possible this was a strategic decision in the hopes there would be less than five other entries, and if so it was a smart gamble that paid off. I sympathize with his statements on the difficulty of making good decisions in this space.

4th Place ($500):

4th Place ($500): Full report

Kevin Keeth’s Recommendation List is as follows: “No quantified recommendations were developed. See ‘Report Body’ for excruciating confession of abject failure.” A failure that is admitted to and analyzed with such honesty is valuable, and I’m glad that Kevin submitted an entry rather than giving up, even though he considered his entry invalid and failure is still failure. Many of the concerns raised in his explanation are no doubt valid concerns. I do think it is worth noting that a Bayesian approach is not at a loss when the data is threadbare, and the probabilistic consequences of actions are highly uncertain. Indeed, this is where a Bayesian approach is most vital, as other methods are forced to throw up their hands. Despite the protests, Kevin does provide strong cases against supplementation of a number of trace minerals that were left unconsidered by other entries, which is good confirmation to have.

3rd Place ($500):

3rd Place ($500): Full report

Michael Buck Shlegeris chose to consider only five minerals, but made reasonable choices of which five to include. None of the recommendations made on those five seem unreasonable, but the reasoning leading to them is unsound. This starts with the decision to exclude studies with less than a thousand participants. While larger sample sizes are obviously better (all else being equal), larger studies also tend to be retrospective, longitudinal monitoring studies and meta-analyses. The conclusions in each section are not justified by the sources cited, and the RDI (while a fine starting point) is leaned on too heavily. There is no cost/benefit analysis, nor are the recommendations quantified. This is a serious entry, but one that falls short.

2nd Place ($1000):

2nd Place ($1000): Full report

Kevin Fischer provides a treasure trove of information, teasing out many fine details that the other entries missed, and presented advocacy of an alternate approach that treats supplementation as a last resort far inferior to dietary modifications. Many concerns were raised about method of intake, ratios of minerals, absorption, and the various forms of each mineral. This is impressive work. There is much here that we will need to address seriously in the future, and we’re proud to have added Kevin Fischer to our research team; he has already been of great help, and we will no doubt revisit these questions.

Unfortunately, this entry falls short in several important ways. An important quote from the paper:

"“Eat food high in nutrients” represents something like the null hypothesis on nutrition - human beings were eating food for millions of years before extracting individual constituents was even possible. “Take supplements” is the alternative hypothesis.

This is an explicitly frequentist, and also Romantic, approach to the issue. Supplementation can go wrong, but so can whole foods, and there’s no reason to presume that what we did, or are currently doing with them, is ideal. Supplementation is considered only as a last resort, after radical dietary interventions have “failed,” and no numbers or targets for it are given. No cost-benefit analysis is done on either supplementation or on the main recommendations.

Winner ($5000): Scott Alexander (Yvain)

Winner: Scott Alexander / Yvain ($5000): Full report

Scott Alexander’s entry was not perfect, but it did a lot of things right. An explicit cost/benefit analysis was given, which was very important. The explanations of the origins of the RDAs were excellent, and overall the analysis of various minerals was strong, although some factors found by Kevin were missed. Two of the analyses raised concerns worth noting: potassium and sodium.

On sodium, the concern was that the analysis treated the case as clear cut when it was not; there have been challenges to salt being bad, as referenced last year by Robin Hanson, and the anti-salt studies are making the two-stage argument that blood pressure causes risks and salt raises blood pressure, rather than looking at mortality. However, the conclusions here are still reasonable, especially for ordinary Americans regularly eating super-stimulus foods loaded with the stuff.

 

New Comment
65 comments, sorted by Click to highlight new comments since:

Here's a summary of the recommendation lists, pulled (sloppily) from each of the report abstracts for those who don't wish to read.

  • SODIUM: < 1.5 g through dietary control. Most people should reduce consumption. (Yvain & Michael)

  • POTASSIUM: > 4.7g through diet supplemented by potassium salt if necessary. Most people need to increase consumption (Yvain & Michael)

  • IODINE: + 6 mg supplement for women as Lugol's solution or other molecular iodine (Yvain)

  • LITHIUM: + 1 mg supplement as lithium orotate (Yvain)

  • PHOSPHOROUS: Most people should reduce consumption (Michael)

  • MAGNESIUM: Many people are deficient. These people should increase intake (Michael)

  • SELENIUM: +0.1-0.2 mg per day. (Steven Kaas)

  • CALCIUM: Contradictory - Michael suggests consume lots, but men who do so should take vitamin D. Says most people are deficient, even with current RDI. Yvain suggests calcium supplements are bad. Neither give their recommendation particularly high confidence.

  • BEEF LIVER (not a vitamin. I mean an actual liver you cut out of a grass fed cow that goes moo): 4oz 1-2 times per week. (Kevin)

Generally, supplementation is better than deficiency. Getting it through diet is better than supplements. And blood tests are cheap and let you know exactly what you need. Most of those things you heard were healthy/unhealthy are generally right like being sure to eat your fruits and veggies, brown rice > white rice, spinach is good for you, etc.


Please note that this is a summary of their summaries, which in themselves are summaries of other research. We're playing god knows how many levels of telephone with the data, and I just added two more. I only wrote this post because I didn't want to read all those reports and I was secretly hoping someone else would do it and post a summary. Then my decision theory kicked in and said "but if this site is full of clones of me, I'd never get to read a summary", so I rolled a dice and started reading.

I'm sure I missed lots of stuff and I omitted things I thought were common knowledge like spinach being an exceptional source for some vitamins instead. In retrospect I should have probably made a note of every recommendation instead of a personal set of cliff notes. Anyhow I digress. My point is that you should read the reports themselves if you're considering changing your diet routine. Oh, and since I'm sure I got things wrong, don't hesitate to tell me.

Then my decision theory kicked in and said "but if this site is full of clones of me, I'd never get to read a summary", so I rolled a dice and started reading.

Out of curiosity, what die did you roll and what result was required to start reading?

A d20 of course. We are all clones of one another after all.

I'm just glad I didn't roll a natural 20 or I'd have had to write an original report myself.

What range was "summarize" and what range was "wait for someone else to do it"?

I needed a 13 or better to do it. 12 or lower would be wait for someone else.

[-][anonymous]40

Heh, I was thinking you rolled a d20 even before I read this comment.

A big thanks to Personalized Medicine for putting this on and subsidizing my learning about things I should have been motivated to learn anyway. Thanks also to User: TCB for proofreading my entry and catching several hilarious but unacceptable mistakes, which was very helpful. And of course to gwern, wedifrid, and several other people who knew much more about minerals than I did for not participating; that was helpful too.

Although I did the best I could with the time I had available and I stick to what I said, I absolutely agree with everyone who says the field is confused and I have relatively low confidence some of my recommendations. I agree with Zvi and Michael that there's plausible evidence that lowering sodium below 4g/day might not be beneficial and may even be harmful (though it's still better too low than too high), but there's also plausible evidence (some of which I cited) that lowering sodium is great and you should lower it as much as possible; I still think the balance of evidence is for the latter, but take the whole field with a grain of salt (pun intended). Calcium - I do stick to my position that it's more likely to harm than help in groups without elevated osteoporosis risk, but I could still totally be convinced of the opposite if there's some good study I missed or if I misinterpreted some statistic. As for lithium and iodine, I went into the research hoping I could avoid any weird recommendations that went against the general medical consensus, and clearly that plan didn't survive contact with the enemy; I feel a little awkward recommending them, but the whole "Bayesians can't privilege the null hypothesis" thing has forced my hand. Still, be really careful with those.

Also, Kevin's entry especially was really really good, and I recommend taking a look at it. It's very speculative, but it's interesting and well-founded speculation and I hope some of his theories get the research attention they deserve.

And I publicly precommit to donating at least half of the prize to charity, probably split between SIAI (got to remember to wait for their matching drive this time) and some warm fuzzy causes (is it weird that I think of malaria eradication as a warm fuzzy cause these days?)

(is it weird that I think of malaria eradication as a warm fuzzy cause these days?)

Kitten malaria, right?

[-]gwern110

And of course to gwern, wedifrid, and several other people who knew much more about minerals than I did for not participating; that was helpful too.

Indeed. You can be sure that this contest has been entered into my personal mistakes file.

And I couldn't help but notice the resemblance of the lithium section to my own summary... :)

And I couldn't help but notice the resemblance of the lithium section to my own summary... :)

Kevin said the same thing, but I didn't read either of your writings until after the contest. I'm going to classify this one under "independent confirmation".

Indeed. You can be sure that this contest has been entered into my personal mistakes file.

My file has been likewise expanded!

For my part I do concur with all of Yvain's findings while I had a few disagreements with Kevin's ideas.

[-][anonymous]210

While we had hoped to receive more than five entries...

Most of the people I know who would have considered entering, myself included, were put off by the short time window between announcement and deadline.

[-]Zvi50

We're planning two months for the next contest. Is that long enough?

[-][anonymous]20

Yes, that'd be better.

By request, I've put up a backup copy of my entry at http://www.raikoth.net/Stuff/LessWrong/mineral.doc .

Any chance that another backup copy of this exists anywhere?

Thanks for this valuable research! I read Kevin and Yvain's reports with my multivitamin's label at hand. From Yvain's report:

Because of the high potential for manganese toxicity, and continuining uncertainty over possible cover toxicity of relatively low manganese levels, manganese supplementation is probably a bad idea.

From the label:

Manganese 4mg 200%

Well, that's going straight in the trash and getting replaced with something different. Wtf?

Here's what it says on the label of mine:

Manganese 2.3 Mg 115%

Anybody want to plug a one pill supplement a middle class American could find easy?

Have you considered not taking anything? Most studies show multivitamins have somewhere between no effect and a slight negative effect except in groups with special nutritional issues. It might be possible to optimize further with the right combination (if the lack of effect from multivitamins is because they include both beneficial and harmful chemicals with a net zero effect) but I don't really think we know enough to do that right now.

Every multivitamin I've had is at least 2mg Manganese/pill. Cut the pill in half and it's half as bad for you?

It seems lots of supplement manufacturers just kind of make it up as they go along.

Other warning signs that should get you to throw out particular multivitamins -- Vitamin E as d-alpha tocepherol only rather than mixed tocopherols, and selenium as a single form such as selenomethionine rather than selenium yeast or selenium amino acids.

I also suspect Vitamin A is similarly harmful in isolated form compared to the much more complex natural sources of it but don't really have any evidence on that.

I also suspect Vitamin A is similarly harmful in isolated form compared to the much more complex natural sources of it but don't really have any evidence on that.

I would bet against the specific prediction without rejecting the principle. That is, I expect when maximising a given function along the lines of "useful when you don't have enough and minimally harmful when you have more than enough" there is a specific isolated form that is better for supplementation than complex natural sources. Where my in principle agreement comes in is that I consider it highly unlikely that the actual isolated form used in an arbitrary multivitamin supplement to be the most desirable one.

The most obvious reason I would expect to find that at least one of the forms of Vitamin A to be better supplemented in isolation than just adding more of a combination is that some of the forms are largely inactive until they are converted by the suitable enzymes. That gives you a potential rate-limiting buffer in the case where you are already eating too much of the stuff and adding more would otherwise be bad.

IIRC at least some multivitamins contain at least part of their Vitamin A as carotene rather than retinol.

Yup, definitely, many multivitamins (particularly at the higher end) also contain complex forms of Vitamin E and selenium.

The links to the full reports are broken at time of posting.

Here are archive copies: 1st 2nd 3rd 4th 5th

Also 1st 2nd 3rd 4th 5th

I found another physician online endorsing a mg or two daily lithium supplement:

Lithium and Inflammation

Lithium and Longevity

(found the blog on the paleo sub-reddit). I was going to the herb and vitamin store this afternoon anyway to get some ginseng and I am going to see if they have those 1mg lithium pills and if they have them and they aren't 25$ a hundred or anything ridiculous I am thinking I am going to take the plunge and do at least one short experiment.

any sources for 1mg lithium orotate? Cutting 5mg pills into fourths and taking every 5th day off is a hassle.

1.25mg/day is well enough within the normal human range that I wouldn't worry about taking that rather than 1.25mg. See Schrauzer's paper below.

FYI, this is the section on lithium that ended up being excised from my paper.


Lithium is not typically considered an essential dietary mineral, but evidence is emerging that minute amounts of lithium benefit physical health and mental well-being. Lithium is widely regarded as potentially toxic, but this is true only of the macro amounts prescribed in mental health treatment. Recommended supplemental levels of lithium are about 3 orders of magnitude lower than commonly prescribed dosages.

Micro doses of lithium in the water very significantly reduce suicide in populations, even when adjusted for various confounding factors, including studies in Austria [BJP Lithium in drinking water and suicide mortality] and Japan [BJP Lithium levels in drinking water and risk of suicide]. Also shown to reduce criminality along with suicide. [http://www.ncbi.nlm.nih.gov/pubmed/1699579 Lithium in drinking water and the incidences of crimes, suicides, and arrests related to drug addictions.]

1000 ug/day has been proposed as a provisional safe upper limit for lithium microdosing by Schrauzer [Lithium: Occurrence, Dietary Intakes, Nutritional Essentiality http://www.jacn.org/content/21/1/14.full], claiming that the evidence is such to establish lithium as an essential dietary element. This seems as true as any claim made in nutritional research and I recommend supplementing low dose lithium. Short of a prescribing practitioner, microdose lithium can be had in ConcenTrace trace mineral supplement, with 400ug/dose.

If anyone is curious, fulltext links for 2 of those are in my section on lithium and I cover a bunch of other studies including some criticisms & failures to replicate.

orotate vs aspartate?

"A 120mg tablet of lithium orotate usually has 4.5mg--5mg of elemental lithium"

any info?

Too little to tell, really. 5mg seems to be a bit more than enough, but still safe even if orotate has a multiplier or something.

they only deliver via courier door to door in south africa :/

All the links are broken. Is there any way I can see the entries?

They are available at archive.org.

Thanks. Any idea why the links are broken in the first place? It seems odd that they wouldn't maintain a link for something they paid thousands of dollars in prize $ to get.

I'm not sure. I kind of think (I don't know for sure) that MetaMed is the new name for Personalized Medicine, LLC -- or at least some of the same people are involved -- but I poked around on the MetaMed site and couldn't find the 2012 Quantified Health Prize winners there.

[-]taw20

To be honest I had much higher hopes for this competition than what I've seen.

Also seriously, top 2 entries make recommendations (lithium, paleo) with no plausible transmission mechanism. I know this is rampant is research, since you need to publish something, but I'd have hoped we'd knew better than that.

By "transmission mechanism", do you mean "mechanism of action"? If so, when you say "with no plausible transmission mechanism", do you mean "whose mechanism of action has not yet been discovered" or "I believe it is implausible that a mechanism of action for this could exist"?

'Cause if you mean the former, the fact that we don't understand it shouldn't be a barrier to using it (you might be surprised how many medications we have no idea how they work), and if you mean the latter, well, as the old saying goes, "If it happens, it's possible".

This is doubly true for lithium, where even if you want to reject every single one of the studies I cited, we already know it works in higher dose as a treatment for bipolar disorder.

(and I hate to follow up an interesting rationality point with a boring discussion of pharmacology, but many of the effects of lithium are known or plausibly speculated upon, including protecting neurons against glutamate excitotoxicity; I am no neuroscientist, but I know excitotoxicity has a role in dementia, probably some psychiatric disorders, and in a bunch of common neurological causes of death.)

(you might be surprised how many medications we have no idea how they work)

How common are such medicines, in your experience?

Literally no idea how they work? Not super common, although a paper on drug targets in Nature gives this list of drugs where they can't even begin to classify what sort of entity they act on.

But many other drugs are still incompletely understood, or only have very speculative mechanisms of actions. For example, we know that SSRI antidepressants increase levels of serotonin in the brain, but we're not really sure why that should treat depression (the hypothesis that depression is an imbalance of neurotransmitters was originally an ad hoc hypothesis explaining why these sorts of drugs seemed to treat it, although we've learned a lot since then).

Paracetamol (aka acetaminophen aka Tylenol) is actually pretty mysterious and still the subject of a lot of study, as are anaesthetics (which is too bad, because if we understood how anaesthetics worked, we'd have a promising lead in figuring out what consciousness is). Most psychiatric medications range somewhere from "incompletely understood" to "might as well be witchcraft", and lot of neurologic ones like some of the antiepileptics aren't much better.

I was surprised to learn recently that a lot of drug discovery is now being done by brute force. For example, they discovered new cystic fibrosis drug ivacaftor by dosing lung cells in solution with more or less every known organic chemical until one of them caused the chlorine concentration of the solution to change, which indicated that it had somehow solved the error in cellular chlorine transport that causes CF.

[-]taw-10

By transmission mechanism I mean something more general. X has a non-negligible effect on Y, Y on Z etc.

An example of implausible transmission mechanism:

  • This or that food contains antioxidants, antioxidants protect from molecular damage, less molecular damage means you'll live longer - it has 2 totally broken links since entirely insignificant amount of antioxidants from food get anywhere in the body, and putting more antioxidants in the cells doesn't actually do much at all (even if reducing their amount increases damage considerably).

An example of plausible transmission mechanism:

  • Having dogs improve people's moods, people in better mood have lower blood pressure, lower blood pressure decreases risk of major cardiovascular disease - we don't have much hard data here (funnily enough they did a randomized study once, and found such effects), but every link in the chain is plausible and effect is within realistic order of magnitude.

With sufficiently overwhelming evidence it might be reasonable to ignore lack of any plausible transmission mechanism, but evidence is anything but, and I'm more inclined to think that it went from "I need to publish X papers a year" to "finshing for statistical correlations involving lithium" to "publishing a paper about that".

[-]Zvi110

A lot of people were very put off by the state of knowledge in this area; it turned out we'd chosen an area where it's very difficult to do good work. Kevin's entry illustrates this more than anything - he started out thinking he knew things about how to supplement, and then decided he knew far less than he thought.

Making sense of knowledge in a bad state though is precisely the sort of thing that should test rationality skills of the sort we try to cultivate here.

I think rationality makes a big difference when an area is confused, not when it's a complex area with little data (like weather prediction without lots of sensors and supercomputers). My impression from the discussion is that supplementation is closer to the later, and not much is achieved by pure paper research.

Mathsemantics is the only book I know of on the subject. Any other suggestions?

Actually, How to Lie with Statistics (1954) might also count, but has there been any more recent work on the subject?

There's lots about lying with statistics, especially in the context of medicine. Anything by Ioannidis for one. Arguments in general should be part of our skill set though, and those are obviously relevant here.

This confirms a prediction that I'm pretty sure I went around saying in advance (the state of knowledge in nutrition is fucked up beyond most easy epistemic wins).

Wow I'd call that confirmation bias. We got some significant solidification of existing info, a bit of new info, and spent $7500. Multiply these results by 1000 and what do you get? I'd guess, a fairly thorough knowledge-base.

Transmission mechanism? I think my paper is pretty persuasive about how dietary changes can increase mineral absorption.

By the way, do you know what happened to your footnotes? They show up like

Sometimes mineral supplementation shows increased risk of mortality[ ]

When you use Word's footnote manager, it doesn't store the citation numbers as plain text, so if you copy paste them, it doesn't work.

I had sent PM a slightly revised version of the paper to post for public consumption as a .docx and not the text version that was reviewed and I'll ask them to try putting it up again.

[-][anonymous]20

Did they require .docx? How strange.

[-][anonymous]00

None of the links to the full reports work for me.

[This comment is no longer endorsed by its author]Reply

Files not found on server, but cache works

Or try archive.org... change the URL to get the other 4.

These documents are interesting, particularly #1 and 2. My opinion has not changed on whether or not one should supplement (except in the case of a specific deficiency caused by some sort of malabsorption). My expectation is that someone eating a diet that includes a variety of vegetables, especially dark leafy greens, and also seafood that is low in mercury, will be healthier than someone who eats none of those but supplements, ceteris paribus, even if the supplement doses were optimal. I'm not convinced both is typically necessary.

I believe supplements are a long ways off yet from being reliable ways to improve one's diet, and to become so I would expect they will require more sophisticated measures like genetic and blood tests. Though you can get tests for deficiencies right now, which you should do to get them corrected, but I would lean towards eating more of a food or foods that are rich in that nutrient except in cases where someone has a compromised ability to absorb something and cannot get enough in food.

As has been previously said, there are a boatload of factors that we know exist and affect this subject but that we aren't yet able to anticipate. Eating the foods I described (and some others) does appear reliable.

"In 1985, the EPA, estimated that the daily lithium consumption for a 154 lb. adult ranges from 650 to 3,100 micrograms"

this seems to indicate that lithium supplementation would be largely redundant.

Lithium levels vary massively based on what you eat or drink, as the wide range of the estimate indicates. The authors of many of the lithium papers complain about how levels vary from house to house to county based on whether one drinks soft drinks, eats veggies from a high lithium district, etc.

Well, this is comforting, given that I'd already decided to start supplementing lithium orotate. When it finally arrives I'll be a little bit less averse to taking it.

Two of the analyses raised concerns worth noting: potassium and sodium.

What was your concern about the analysis of potassium? (Or was it the same as for sodium?)

I read Yvain's report and compared it's reasoning on potassium to the DRI DIETARY REFERENCE INTAKES FOR Water, Potassium, Sodium, Chloride, and Sulfate from Standing Committee on the Scientific Evaluation of Dietary Reference Intakes of the Food and Nutrition Board (http://www.nal.usda.gov/fnic/DRI/DRI_Water/water_full_report.pdf). My understanding of potential concerns would be:

  • Serum potassium (responsible for the sodium-potassium pump etc.) is only very weakly linked to potassium intake, as the kidneys are very good at both excreting extra potassium and keeping potassium in the body if intake is low (though the kidneys are not as good at the latter as the former).

  • Potassium's effect on lowering blood pressure is through the sodium-potassium balance, so potassium helps you mostly if your intake of sodium is too high.

  • Increased potassium intake's potential beneficial effects are supposed come from a better acid/base balance in the body; but there is little experimental evidence (as opposed to epidemiological evidence)

  • I don't think the epidemiological studies give reliable evidence for potassium, as the main source of dietary potassium are fresh vegetables, which already correlate with lower mortality on their own, as well as correlating with many lifestyle differences.

  • In animal studies the 'good' potassium was potassium bicarbonate, not potassium chloride, but most supplements have potassium chloride.

From other sources/common knowledge:

  • Sources of base (alkaline) ions should be buffered so that they are not released to the stomach, where they would serve to reduce the necessary acidity of the stomach fluids (very bad for your digestion) instead of helping your body's acid/base balance.

My conclusion: it might indeed be beneficial to increase my potassium intake, but I would have to do it by increasing consumption of vegetables high in potassium rather than by taking supplements.

[-]Zvi10

With potassium the estimates of consumption by the general population seemed not to make sense given the sources; the amount of potassium in the RDI is quite large, and unless you eat certain particular foods it's very hard to hit it without explicitly setting out to do so.

Belated answer: same reason many people have a magnesium deficiency. These are both minerals whose presence in food is highly dependent on their presence in soil. Soil depletion = far less potassium and magnesium in our diets.