Well, here are the thoughts that you provoked from me about this.
Here is the topic for discussion: should we trust psychiatric analysis using frequentist statistics and ignore the outliers, or should we individually analyze psychiatric studies to see if they contain outliers who show symptoms which we personally desire? Should we act differently when seeking nootropics to improve performance than we do when seeking medication for crippling OCD? Should we trust our psychiatrists, who are probably not very statistically savvy and probably don't read the cases of the outliers?
I think we may want to split this up into two questions: What you should do personally, if you feel you have condition X, and what you should do as a government if you want to help treatment for condition X.
For instance, as a government, I would go for repeatability, and size. Just forget the idea of trusting or not trusting suggestive outliers and go for more verification, with a larger sample: If for no other reason then determining the frequency of the outlying effect, which would be important for making large scale medical recommendations.
However, a single person can't generally commission large medical studies, so they might want to just read through the literature and read multiple papers about the effects of such things, perhaps also cross referencing their own medical history. I think Metamed, which was mentioned on Less Wrong a while back, does something like this, if you don't have the time to analyze your health that carefully personally.
If you don't want to do that, (either independently or through commissioning experts), then chances are you will be relying on:
I think that if even the right placebo could cause changes which improve my effectiveness, it would be worth a shot.
Now, psychiatric placebos can be shown to have a variety of effects in papers, and it can get hammed up a bit in headline rereporting.
http://nymag.com/thecut/2014/01/study-placebo-sleep-just-as-good-as-real-sleep.html
But it isn't all hammed up either: it gets rather complicated.
http://en.wikipedia.org/wiki/Placebo
(As a side note, I upvoted you, since that was a good way of provoking thoughts, and it seemed like a placebo for increasing your mental wellbeing.)
Interesting addition of the government perspective. I think that my contributions to that perspective have very little potential for value-added, as that perspective seems to be prevalent in academia and the private and public sectors. I am taking the individual perspective for this discussion.
I would also be interested in a Metamed opinion on this topic, as you are correct, it seems like the magnified version of what I'm suggesting. I'm basically asking 'should you hire metamed to prescribe you off-label nootropics based on existing studies?'
Imagine reading about the following result buried in a prestigious journal:
Now, personally, reading this I would be completely uninterested in the normal result and fascinated by the one, crazy, outlier. Living to the age of 110 is abnormal enough that within 6,666 people selected as a statistical representation of the population, it is extremely unlikely that anyone would live that long, much less continue performing at the apparent health of an 80 year old.
How small would the sample size have to be before you would consider trying the drug yourself, just to see if you, too, lived forever as long as you took it? What adverse effects and hassles would you go through to try it? Would these factors interact to influence your decision (Mild headaches and a pill 4x/day in exchange for maybe apparent eternal life? Sign me up!)
This example is an oversimplification to make a point- often in clinical trials there are odd outliers in the results. Patients who went into full remission, or had a full recovery, or were cured of schizophrenia completely.
In the example above, if the sample size had been 10 people, 9 of whom had no adverse effects and one who lived forever, I would take it. I have been known to try nootropics with little or no proven effect, because there are outliers in their samples who have claimed tremendously helpful effects and few people with adverse effects, and i want to see if I get lucky. I think that if even the right placebo could cause changes which improve my effectiveness, it would be worth a shot.
As far as I know, psychiatrists cannot reliably predict that a given drug will improve a patient's long-term diagnosis, and psychiatrists/psychologists cannot even reliably agree on what condition a patient is manifesting. Mental disorders appear to resist diagnosis and solution, unlike, say, a broken leg or a sucking chest wound. I have learned that Cognitive Behavioral Therapy (CBT) has consistent results against a number of disorders, so I have endeavored to learn and apply CBT to my own life without a psychologist or psychiatrist. It has proven extremely effective and worthwhile.
Here is the topic for discussion: should we trust psychiatric analysis using frequentist statistics and ignore the outliers, or should we individually analyze psychiatric studies to see if they contain outliers who show symptoms which we personally desire? Should we act differently when seeking nootropics to improve performance than we do when seeking medication for crippling OCD? Should we trust our psychiatrists, who are probably not very statistically savvy and probably don't read the cases of the outliers?
Where are the holes in my logic, which suggests that psychiatrists who think like medical doctors/general practitioners have a completely incorrect perspective (the law of averages) for finding and testing potential solutions for the extremely personalized medicinal field of psychotherapy/psychiatry (in which everyone is, actually, an extremely unique snowflake.).
This is more of a thought-provoking prompt than a well-researched post, so please excuse any apparent assertions in the above, all of which is provided for the sake of argument and arises from anecdata.