Indeed, we have this account of the etymology from George MacDonald Fraser's The Steel Bonnets:

Deprived of the protection of law, neglected by his superiors, and too weak to resist his despoilers, the ordinary man's only course was the payment of blackmail. This practice is probably as old as time, but the expression itself was coined on the Borders, and meant something different from blackmail today. Its literal meaning is "black rent" --- in other words, illegal rent -- and its exact modern equivalence is the protection racket.

Blackmail was paid by the tenant or farmer to a "superior" who might be a powerful reiver, or even an outlaw, and in return the reiver not only left him alone, but was also obliged to protect him from other raiders and to recover his goods if they were carried off.

Note that he does consider the modern meaning to be more specialized.