I've pretty much hated it in college, but this might just be because of the way the courses are taught.

This was sort of my experience. Buy the right books and build interesting projects in the time you would be spending on classes, and you'll probably enjoy it a lot more. You don't need a degree in computer science to get a job as a software engineer; some experience/projects and the broad, shallow knowledge required to do well in typical interviews (and all those other interviewing skills I suppose) are enough.

You sound like you might enjoy Hacker School, by the way.