Furthermore, since nobody seems to intentionally use it, it must have a drawback; extreme draining of magic disproportionate to the effects seems like a logical effect.

Lack of control seems like the biggest issue. 'Oops, I nearly killed someone' (Ariana), 'Oops, I didn't mean to make that glass disappear' (Harry), etc.