I think it is partly about mixup, and partly because many people don't think clearly about their financial planning until forced to. If someone who makes 100k+ and spends most of it wants to retire in the same style they are used to living, they may well need 2-4M to do so comfortably and safely if retiring early. Social security is progressive, the max you can get as a single person is around 42,000/year. To get that, you must work for 35 years at a high income level and wait to draw your check until age 70. Then you still need to produce another 58,000 somehow from your own assets, which at current recommended withdrawal rates requires almost exactly 2M to do and maintain your wealth. Now, you could purchase an annuity for much less, but few people are comfortable dumping all their money into such vehicles. At current life income annuity rates, that would be a bit less than 1M$ to provide 58,000/year to a 70 yo. So you only need about 900K, but what if you want to retire at 65, or 60, or 55? Then you need to take less SS, or live off assets until age 70, or maybe you can't take it at all yet, and must live off assets alone. Whether you need 2M or 4M or more depends on when you retire, and how much risk of breaking your plan you are willing to take.

It's my job to model this for people. Most are surprised that they don't need 2 million or more, because their needs are more modest than above, and they don't plan to retire very early. That said it's different for everyone, and when younger people talk about stopping/reevaluating their career because they have enough money to retire, they usually mean in middle age not at normal retirement ages. At 40, unless your lifestyle is very frugal by the standards of people who are able to save 2-4M in that time frame (generally 100k+ earners), you probably do need that much to retire comfortably.

It also depends a lot on how you feel about your work. Most higher income earners have found a niche where they feel reasonably good about what they do (it's hard to create a lot of value when you feel like a cog or a moocher), and enjoy at least big parts of their jobs. In that case, why would you retire before you had plenty? OTOH, if you are burned out and it's a struggle to go to work every day, you might be willing to live on a lot less if you knew you could quit now, or soon.

Note: I know very few people who actually live like they are poor today in order to have great wealth tomorrow. Those who are very frugal while working, either are hugely committed to earning to give, or intend to retire or do something risky or different at a very young age, and don't ever intend to not live relatively frugally. Certainly they intend to either give away or enjoy the fruits of their industry long before a typical retirement age.