I agree that public votes would likely improve the state of affairs. Public downvotes allow a person who doesn't understand why they were downvoted to ask the person directly. It also creates social accountability for the votes.