People usually abstain for reasons that might affect cognitive performance like depression or previous substance abuse for example.

Reduction in cognitive performance for heavy drinkers is measured more in men than in women.

They note that:

After adjustment for education and race, male hazardous/harmful drinkers no longer performed significantly less well than light drinkers, whereas male and female abstainers and occasional drinkers still did so.

-

Chronic alcoholism is also associated with specific neurological conditions such as Wernicke-Korsakoff syndrome, which is caused by thiamine deficiency — someone who's getting most of their calories from booze is not getting enough nutrition.

Alcoholism can also reduce thiamine absorption as much as 50 % in people who aren't malnourished.