If I want to be perfectly honest with myself, the reason that I was initially attracted to physics was simply because it was hard: for me and for others. At the level of the high-school, it was harder than math and programming; doing good math and programming didn't feel like an art (at higher levels of course, all three become an art). But physics required both math skills and a sort of heuristic/visual thinking that math and programming didn't have. The hardness fed into my desire to appear smart.

Of course, later on, the motivations to do physics changed as I became more appreciative of the beauty and power of physics.

Added Later: In popular culture, the exemplars of 'genius' are usually physicists: Einstein, Newton, Hawking, Feynman, Bohr, Heisenberg, Schrodinger. Thus, students know that being a physicist means you'll be identified as a smart person: simply by association with the famous names. In contrast, I think far fewer people know about Euler, Gauss, von Neumann, Grothendieck, Babbage, McCarthy, Knuth.