It's often convenient to be able to write a post explaining a downvote. I'd be okay with barring people from downvoting replies to their own posts, though, which also prevents downvote duels in long threads but preserves the ability to explain a vote.

(It'd likely be marginally easier to implement, too, since the presence of a post is stabler than the direction of a vote.)