Yes, that's true, during late XIX - early XX century drugs like cocaine and opium were in (relatively) common use. But that raises the question whether most of the XX century was an aberration in that respect, a temporary victory of puritans...

Psychedelics are a bit of a special case though, in that they became known and widespread quite late. I am sure there were some Victorian gentlemen who tried magic mushrooms and such, but it took LSD (and cheap LSD) to get psychedelics to the masses. And that happened in the West -- I don't know if East Asia ever had a wave of people experimenting with psychedelics. China was too poor and Japan was too conformist.