This person is hurting his overall voting weight as compared to those who stay within their areas of expertise, so he will be outvoted in his own field by other experts. And one person being a bit more weighty shouldn't amount to much in a large pool. The more realistic version of that scenario was actually addressed in OP:

There is a danger of a subgroup amassing a large voting weight, then abusing it in the window before they are removed from power, which can perhaps best be guarded against with some sort of constitutional system, perhaps even one formally incorporated into the system as a high Bayesian prior against certain classes of actions being correct.