Well, that's the whole idea of flirting - that you can't really tell the difference. If it's clear and upfront, then it's not called flirting anymore, but rather an advance (friendly or more explicit).

You have a lot of uncertainty arising from a simple gesture/look/invitation, and (I believe) this is where all the fun really comes from: dealing with a lot of different scenarios that have very similar initial contexts but have a wide range of possible outcomes, and choosing the outcome you want with so little effort.

I also believe that your ability to tell the difference between one person's flirting and friendliness is strongly influenced by how well you know that person.