(the theory would also imply that an infinite amount of irritation is not as bad as a tiny amount of pain, which doesn't seem to be true)

Hmm not sure. It seems quite plausible to me that for any n, an instance of real harm to one person is worse than n instances of completely harmless irritation to n people. Especially if we consider a bounded utility function; the n instances of irritation have to flatten out at some finite level of disutility, and there is no a priori reason to exclude torture to one person having a worse disutility than that asymptote.

Having said all that, I'm not sure I buy into the concept of completely harmless irritation. I doubt we'd perceive a dust speck as a disutility at all except for the fact that it has small probability of causing big harm (loss of life or offspring) somewhere down the line. A difficulty with the whole problem is the stipulation that the dust specks do nothing except cause slight irritation... no major harm results to any individual. However, throwing a dust speck in someone's eye would in practice have a very small probability of very real harm, such as distraction while operating dangerous machinery (driving, flying etc), starting an eye infection which leads to months of agony and loss of sight, a slight shock causing a stumble and broken limbs or leading to a bigger shock and heart attack. Even the very mild irritation may be enough to send an irritable person "over the edge" into punching a neighbour, or a gun rampage, or a borderline suicidal person into suicide. All these are spectacularly unlikely for each individual, but if you multiply by 3^^^3 people you still get order 3^^^3 instances of major harm.