Well, I read canon a loooong time ago but IIRC in book 7 in one of the first chapter Voldie goes around humiliating Lucius, in particular taking his wand without offering a replacement, and insulting him for believing he (Voldie) would give Lucius his wand in exchange. The conversation with Mr. White (" most delinquent of my servants") and the fact that he humiliates him similarly by removing part of his magic ability is reminiscent of that.

Also, before I thought Mr. Grim was Peter Pettigrew, but now that we know that Black is the actual bad guy, it's even clearer that Mr. Grim = Sirius Black. In particular, Voldie says to him "I was surprised to see you here tonight; you are more competent than I suspected", which in retrospect clearly means "I thought you were rotting in Azkaban".