I had trouble in my Masters (a combination of course choice and bad luck) and so didn't do a PhD. But we're talking about the top university in at least the country, and by some accounts the hardest non-research course in the world. I'm pretty sure that going a different route I could've got to the point of starting a PhD before hitting anything difficult.

I do sometimes think I should've chased the Fields medal, but I'm ultimately happier the way things turned out. I worked my ass off the whole time in school/university; nowadays I earn a good living doing fun things, but my evenings and weekends are my own, and I've got a much better social life.