From the same page cited by timtyler above:

Ever more alarming animal studies conducted over a decade by George Ricaurte, a neurotoxicologist at John Hopkins University School of Medicine, suggest that taking high and/or frequent doses of MDMA causes damage to the terminals of serotonin axons in the brain. Cerebrospinal fluid 5-hydroxyindoleacetic acid (5-HIAA), serotonin's major metabolite which serves as a marker of central serotonin (5-hydroxytryptamine, 5-HT) neural function, may be lower in human MDMA users than in putatively matched controls. The number of serotonin transporter sites, structural protein elements on the presynaptic outer axonal membrane that recycle the released neurotransmitter, may be reduced too. Long-term MDMA-induced changes in the availability of the serotonin transporter may be reversible; but it is unclear whether recovery is complete. Currently the balance of neurochemical and neuroanatomical evidence, and functional measures of serotonin neurons, suggests that it is imprudent to take MDMA or other ring-substituted methamphetamine derivatives without also taking neuroprotective precautions. Arguably, it is best to take MDMA infrequently and reverently or not at all - Dr Shulgin once suggested a maximum of four times a year.

Yes, the page goes on to describe reasons to be skeptical of the studies, but I think that I don't want to risk it - and I don't know how to get the drugs anyway, especially not in a reasonably pure form. I've also made a point of refusing alcoholic beverages even when under significant social pressure to consume them; my family medical history indicates that I may be at unusually high risk for alcoholism, and I would definitely describe myself as having an "addictive personality", assuming such a thing exists.

The most recent meta-analysis acknowledges that "the evidence cannot be considered definitive", but concludes:

To date, the most consistent findings associate subtle cognitive, particularly memory, impairments with heavy ecstasy use.

For practical purposes, this lingering doubt makes little difference. Hedonists are well-advised to abstain from taking Ecstasy on a regular basis even if they assign, say, a 25% chance to the hypothesis that MDMA is neurotoxic.

I myself believe that positive subjective experience ("happiness", in one of its senses) is the only thing that ultimately matters, and would be the first to advocate widespread use of ecstasy in the absence of concerns about its adverse effects on the brain.

--

Gouzoulis-Mayfrank, E.; Daumann, Neurotoxicity of methylenedioxyamphetamines (MDMA; ecstasy) in humans: how strong is the evidence for persistent brain damage?, J. Addiction. 101(3):348-361, March 2006.