If people see you as an authority and you make a mistake, they can accept that no one is perfect and mistakes happen. If they doubt the legitimacy of your authority, any mistakes will be taken as evidence of hubris and incompetence.

I think part of it is the general population just not being used to algorithms on a conceptual level. One can understand the methods used and so accept the algorithm, or one can get used to such algorithms over a period of time and come to accept them.

Besides, a human's "expert judgment" on a subject you know little about is just as much of a black box.

And such experts are routinely denounced by people who know little about the subject in question. I leave examples as an exercise for the reader.

Different methods are more and less likely to lead one to the truth (in a given universe). I see little harm in calling those less likely arts dark. Rhetoric is surely grey at the lightest.

Probably because humans who don't know much about algorithms basically have no way to observe or verify the procedure. The result of an algorithm has all the force of an appeal to authority, and we're far more comfortable granting authority to humans.

I think people have also had plenty of experience with machines that malfunction and have objections on those grounds. We can tell when a human goes crazy if his arguments turn into gibberish, but it's a bit harder to do with computers. If an algorithm outputs gibberish that's one thing, but there are cases when the algorithm produces a seemingly reasonable number that ends up being completely false.

It's a question of whether to trust a transparent process with a higher risk of error or a black box with a lower, but still non-negligible risk of error.