Having an affair is discrete, while the annoyance level is continuous. There's simply no explicit, plain meaning possible for continuous variable like that, one has to deduce it from the tone of the voice, body language, etc etc etc etc. One could of course have friendly body language and tone while saying something like "yes, it is incredibly annoying" but that would merely confuse the listener.