There is nothing wrong with averages.

That depends. Mostly they are used as single-point summaries of distributions and in this role they can be fine but can also be misleading or downright ridiculous. The problem is that unless you have some idea of the distribution shape, you don't know whether the mean you're looking at is fine or ridiculous. And, of course, the mean is expressly NOT a robust measure.