Two things you could do about that:

• As you hover over the rating button, the text below changes to indicate what that rating would mean. Zero stars means "don't bother", one star means "good enough to stay visible", two stars means "above-average" and so on

• Allow half stars for more information.

We would use percentile score to make the best use of the votes of binarizing voters without giving them more influence than high-information voters.