Given the things he's said elsewhere on this site, I think you're correct about Annoyance's intended interpretation, which makes me wonder why he didn't include it with the koan. Is it just not Zen-fashionable to do so?