If I understand correctly, when a page gets refreshed with a reply to a comment, the delete button on that comment is gone; but as long as one is viewing a previous state of the page, when you use the delete button it works.

Which is what I did. I hadn't seen your reply, you made it just as I was deleting - literally, I deleted and upon the page reloading I saw the inbox thingie with your reply. I was deleting because I had just noticed that I missed all those other replies to Sam.