The original Harry Potter was really just a clever re-imagining of standard fantasy tropes, so as long as you've had some sort of passing familiarity with fantasy you'll have no trouble following along. There are all kinds of geeky references thrown in that you might even be better off than someone who has read the books but never passed a science course.