There have been explicit mod statements against retributive downvoting (though they rely on a not-entirely-obvious reading of the content deletion policy): see here. We could probably use better and more obvious policy articles on the wiki, granted.

Aside from that, as long as you're not playing voting games with sockpuppets, writing scripts to automate voting, or spending hours of your time on delivering votes and nothing else, I'd say you're pretty safe. There's not a lot of policy because not a lot of policy is needed to regulate typical voting behavior; behave typically in your voting (i.e. vote manually, after reading content, and don't go looking for content to downvote) and you'll be fine.