Avorn (2004) reports:
In a former British colony, most healers believed the conventional wisdom that a distillation of fluids extracted from the urine of horses, if dried to a powder and fed to aging women, could act as a general tonic, preserve youth, and ward of a variety of diseases. The preparation became enormously popular throughout the culture, and was widely used by older women in all strata of society. Many years later modern scientific studies revealed that long-term ingestion of the horse-urine extract was useless for most of its intended purposes, and that it causes tumors, blood clots, heart disease, and perhaps brain damage.
The former colony is the United States; the time is now; the drug is the family of hormone replacement products that include Prempro and Premarin (manufactured from pregnant mares' urine, hence its name). For decades, estrogen replacement in postmenopausal women was widely believed to have "cardio-protective" properties; other papers in respected medical journals reported that the drugs could treat depression and incontinence, as well as prevent Alzheimer's disease. The first large, well-conducted, controlled clinical trial of this treatment in women was not published until 1998: it found that estrogen replacement actually increased the rate of heart attacks in the patients studied. Another clinical trial published in 2002 presented further evidence that these products increased the risk of heart disease, stroke, and cancer. Further reports a year later found that rather than preventing Alzheimer's disease, the drugs appeared to double the risk of becoming senile.
Armstrong (2006) adds:
The treatment seemed to work because those who used the drug tended to be healthier than those who did not. This was because it was used by people who were more interested in taking care of their health.
You certainly didn't state it, but I got the impression that you don't think you should pay much attention to clinical trials. Perhaps I inferred what wasn't there.
If failure to reject the null hypothesis proves little, then you're unlikely to reject it even if it's false, which raises the question of why you did the study in the first place. You clearly should have made a big enough study to actually notice things.
I guess in that case (and many others, such as pain relief) it would work. The example in the original comment was life extension, where it clearly would not. Even so, you could rely on studies to tell you how likely it is to work, and how well it will work if it does. You also can't just rely on testing them yourself, given that only a tiny fraction of things help you sleep, but the same goes for studies.
I meant that you can't find them by testing them on yourself, since you can only do it once, unlike something to help you sleep. You can find them by clinical trials. You just saw an example of that.