A small investigational drug trial won't be powered to detect outliers, and you won't be able to reliably solve that by invoking Bayesian statistics.
In large drug trials I think this is to some degree already done, but it's limited by the extreme sketchiness of suddenly inventing new endpoints for your study after you have the data. It would probably take the form of increasing the threshold for an endpoint (for example, "No significant difference between drug and placebo was found with the planned endpoint of decreasing HAM-D ratings by 3 or more, but there were significantly more patients in the drug group who had their HAM-D ratings decrease by 10 or more". Everyone is rightly suspicious of people who do this, because, again, changing endpoints. But if it happened enough someone would take notice. Trust me, "not coming up with clever ways to make their drug look effective for at least some people" is not one of pharmaceutical companies' failure modes.
But keep in mind that you sort of loaded the original example by choosing something that almost never happens (someone living to 110 without any signs of aging). In a psychiatry study, what's the most extreme example you're going to get? Someone's depression remits completely? Big deal. Most people's depressive episodes remit completely after a couple of months anyway, and in 25% of people they never return (in even more people, they take many years to return, and almost no studies continue for the many years it would take to notice). In a drug trial of 10000 people (the number you gave above) hundreds or thousands of people in each group are going to have their depression remit completely; if the drug has a superpowerful effect on one person and cures her depression forever, that will get lost in noise in the way that someone living to 110 with the body of a 30 year old might not.
(it's instructive to compare this to the way studies investigate side effects. If one person in a 10000 person study has their arms fall off, the investigators will notice, because that's sufficiently rare as to raise suspicion it was caused by the drug. The drug will then end up with a black box warning saying "may make arms fall off.")
Another way these sorts of outlier effects might be detected is by subgroup analyses (which are also extremely sketchy). If there is no effect in general, researchers may check whether there is an effect among men, among women, among blacks, among whites, among Latinos, among postmenopausal Burmese women who wear hats and own at least two pets and have a history of disease in their left kidney, anything that turns up a positive result. But again, this is hardly something we want to encourage.
But all these things are for investigational drugs. if we're talking about a drug that's already been approved and has a strong prescription history, then your worries about individualized response would get subsumed into the good responder / bad responder distinction, which is a very very big area of research which we know a lot about and when we don't know it it's not for lack of trying.
For example, among bipolar patients, response to lithium can be (very inconsistently) predicted by selecting for patients who have stronger family history of disease, have fewer depressive symptoms, have slower cycles, have more euthymic periods, have less of a history of drug use, start with a manic episode, demonstrate psychomotor retardation, demonstrate premorbid mood lability, lack premorbid personality disturbance, possibly have deranged serotonin metabolism in platelets, possibly have increased calcium binding to red blood cells, possibly lack the HLA-A3 antigen, possibly have a particular variant of the gene GADL1, etc etc etc.
(in practice we don't expend much effort to check most of these things, because their predictive power is so weak that it's almost always a worse idea than just making a best guess based on the data you have, putting someone on lithium or on an alternative, then switching if it doesn't work)
As far as I know, psychiatrists cannot reliably predict that a given drug will improve a patient's long-term diagnosis, and psychiatrists/psychologists cannot even reliably agree on what condition a patient is manifesting. Mental disorders appear to resist diagnosis and solution, unlike, say, a broken leg or a sucking chest wound.
The whole "medical doctors can always consistently treat medical diseases, but psychiatrists are throwing darts blindfolded" story is something of a myth - see for example Putting the efficacy of psychiatric and general medicine medication into perspective: review of meta-analyses
Thanks for that last link, it was an interesting update on the effectiveness of psychiatry. I was weighting my knowledge of the prevalence of rotten corpses in psychology into my estimate of the effectiveness of psychiatric methods, which now seems to be conflating two very different things. Although it does still seem that the set of psychiatrists who are capable of ignoring the prevalent rotten corpses in psychology when prescribing drugs is still small enough to tip the field toward doing your own analyses. I guess i don't have a good set of heuristics ...
Imagine reading about the following result buried in a prestigious journal:
Now, personally, reading this I would be completely uninterested in the normal result and fascinated by the one, crazy, outlier. Living to the age of 110 is abnormal enough that within 6,666 people selected as a statistical representation of the population, it is extremely unlikely that anyone would live that long, much less continue performing at the apparent health of an 80 year old.
How small would the sample size have to be before you would consider trying the drug yourself, just to see if you, too, lived forever as long as you took it? What adverse effects and hassles would you go through to try it? Would these factors interact to influence your decision (Mild headaches and a pill 4x/day in exchange for maybe apparent eternal life? Sign me up!)
This example is an oversimplification to make a point- often in clinical trials there are odd outliers in the results. Patients who went into full remission, or had a full recovery, or were cured of schizophrenia completely.
In the example above, if the sample size had been 10 people, 9 of whom had no adverse effects and one who lived forever, I would take it. I have been known to try nootropics with little or no proven effect, because there are outliers in their samples who have claimed tremendously helpful effects and few people with adverse effects, and i want to see if I get lucky. I think that if even the right placebo could cause changes which improve my effectiveness, it would be worth a shot.
As far as I know, psychiatrists cannot reliably predict that a given drug will improve a patient's long-term diagnosis, and psychiatrists/psychologists cannot even reliably agree on what condition a patient is manifesting. Mental disorders appear to resist diagnosis and solution, unlike, say, a broken leg or a sucking chest wound. I have learned that Cognitive Behavioral Therapy (CBT) has consistent results against a number of disorders, so I have endeavored to learn and apply CBT to my own life without a psychologist or psychiatrist. It has proven extremely effective and worthwhile.
Here is the topic for discussion: should we trust psychiatric analysis using frequentist statistics and ignore the outliers, or should we individually analyze psychiatric studies to see if they contain outliers who show symptoms which we personally desire? Should we act differently when seeking nootropics to improve performance than we do when seeking medication for crippling OCD? Should we trust our psychiatrists, who are probably not very statistically savvy and probably don't read the cases of the outliers?
Where are the holes in my logic, which suggests that psychiatrists who think like medical doctors/general practitioners have a completely incorrect perspective (the law of averages) for finding and testing potential solutions for the extremely personalized medicinal field of psychotherapy/psychiatry (in which everyone is, actually, an extremely unique snowflake.).
This is more of a thought-provoking prompt than a well-researched post, so please excuse any apparent assertions in the above, all of which is provided for the sake of argument and arises from anecdata.