political labels are used mostly in contrast with other labeled concepts like them

Yes.

When I think about how people around me, who don't identify with label "liberal", define the word "liberal", it usually means "not one of us". And when people who identify with label "liberal", define the word "liberal", it usually means "one of us".

So the outside definition really depends on what other political labels are frequent in given environment. For example in Slovakia many people identify as "conservative" which approximately means: Catholic, or someone who is not really Catholic, but accepts that Catholic church / culture / tradition is very important part of society. So their definition of "liberal" is simply someone who opposes the Catholic church and traditions; no more details are really necessary, because one label for all enemies is emotionally enough. Then there are many socialists who either use the word "liberal" for themselves to avoid some bad connotations of the word "socialist"; or they are proud to call themselves "socialists" or "communists", and then they use the label "liberal" for someone who supposedly opposes all equality, solidarity and generally any human feelings (some exaggerating here). Sometimes they even say "neo-liberal" which describes even stronger feelings of revulsion; it probably means: even worse than an ordinary liberal.

On the other hand, people who use label "liberal" for themselves, they simply mean: a cool person who has the same opinions as me. And then they are surprised to find out that other people using the same label have different opinions. But of course the explanation is that the other ones' use of the label is incorrect.

There is some nonzero correlation between political labels and their meanings, but it is far from a clear definition.