Funny you mentioned Haverford and Swarthmore. If you go to either of those (or, for women, Bryn Mawr), you can take classes at either of the other two OR at the University of Pennsylvania - though this last option will cost a little extra in money and time, I knew more than a few people who went to UPenn to take courses on esoteric subjects.

My first class at Haverford was at Bryn Mawr. Several Bryn Mawr students came to Haverford for the astronomy and particle physics courses, or music. Others went the other way, to take graduate math courses, or geology...

So, if you would prefer somewhere small but want some not-exactly-standard option, check out whether a small college enables outside courses. That will expand their menu significantly.