If all parents of every generation rolled a d6 and had that many children then population growth stays constant - at a rate of 1.75x per generation, ignoring for the sake of argument the effect of people failing to live up to their dice roll for various reasons. In this concrete scenario, 6/21 children in the general population were born to parents who rolled a 6, 1/21 to parents who rolled a 1, etc. So yes, you would be are six times as likely to be born to parents who rolled a 6, but population growth is constant. If an argument says that population growth in falling in this scenario it's doing something wrong.
This argument seems to be using the fact that you are more likely to have parents who rolled a 6, and somehow taking it to mean that more 6s were rolled for the previous generation in general. This is simply not correct.
I'm honestly not sure whether this is a flaw in your argument, or a flaw in another argument that you're trying to highlight, so sorry if I'm just stating the obvious here.
There are extra subtleties, I've realised. I'm working on them now...
Many thanks to Paul Almond for developing the initial form of this argument.
My previous post was somewhat confusing and potentially misleading (and the idea hadn't fully gelled in my mind). But here is a much easier way of seeing what the SIA doomsday really is.
Imagine if your parents had rolled a dice to decide how many children to have. Knowing only this, SIA implies that the dice was more likely to have been a "6" that a "1" (because there is a higher chance of you existing in that case). But, now following the family tradition, you decide to roll a dice for your children. SIA now has no impact: the dice is equally likely to be any number. So SIA predicts high numbers in the past, and no preferences for the future.
This can be generalised into an SIA "doomsday":