It really took me roughly that long, although it was more conceptually deep than most algebra courses. I learned most of my math at my own pace, with help from my dad. My non-confident guess is that most mathematically talented people encounter algebra and other subjects long after they're ready for them, and therefore learn them fairly rapidly but at the cost of having wasted time earlier on. But I may just have been slow.

In any case, even restricting to bright college students other than me, I've watched multiple individuals get much faster at learning math over the course of undergrad.