Starch can be converted to glycogen just fine, it just takes longer.

The only good reason I see to eat significant amounts of sugar is if I need an energy boost right now, ideally during exercise, though preemptive bread-eating reduces even that need. Active exercise also reduces the negative effects of eating sugar.