For me, the main reason why if I don't reply to a comment straight away I don't reply to it ever, is that I forget to. Sometimes (twice a month or so) I re-read my inbox to make sure there aren't comments I meant to reply to but forget to. And when I remember to, I don't click on the inbox icon (so it stays red) unless I'm on a device on which it's comfortable for me to reply.
Have you considered writing those things down? My life got immediately and significantly better after I started writing down all the things I have to do. Even though I didn't know it, forgetting about things had been a huge problem for me, so writing it all down fixed that problem, eliminated most of the guilt I felt for not getting stuff done, and allowed me to work on things in a sensible order instead of just doing whatever things randomly come to mind.
I often feel guilty for ignoring other people's comments or questions, and frustrated when other people seem to be ignoring me. If I can't indicate to someone exactly why I'm not answering, or can't receive such an indication myself, I can at least help my future selves and others obtain a better probability distribution over such reasons. To that end, I'm listing all of the reasons I can think of for someone to not respond to a comment/question, to save the effort of regenerating these hypotheses from scratch each time and prevent the possibility of failing to consider the actual reason. Note that these are not meant to be mutually exclusive.
If I missed any reasons (that happen often enough to be worth including in this list), please give them in the comments. See also this related comment.