I feel a strong affinity with this community, but that's hard to remember when my own locally-weird ideas get downvoted and/or reverted to draft without giving me a clear route to explain myself.
Even just splitting the Karma mechanism into "doesn't seem benevolent" and "doesn't seem accurate" would be much more comforting. None of us have the working memory to keep that straight all the time.
Or, since nonbenevolence is a stronger claim, it could be the Strong Downvote? It should absolutely be labeled as such, though.
There may be a large difference in quantity of observations. The OP has very few posts or comments (unless this is a new account for a longer-term contributor), where you've been updating your model for years, over many hundreds of posts and comments.
My general advice is to not worry much about karma - it's a good signal, but quite noisy and intentionally information-light in order to increase participation. It should be a guide to what topics and styles work best, but don't overfit, especially in your first dozen posts or hundred comments. You'll get maybe 1/10 as many comments as voters, but comments have a lot more informational value.