Apart from Dumbledore and Lucius, none of them are likely to take an 11-yo child and his promises of enmity and revenge at all seriously. "Enough talk, he'll be late for his classes." And even if he might become a political counter of some significance in a decade, or a few decades, they wouldn't expect him to hold a grudge that long - normal children don't often do that.