With this sort of thing, or anything really, you either use bulletproof mathematical models derived from first principles (or empirically) with calibrated real quantities, or you wing it intuitively using your built-in hardware. You do not use "math" on uncalibrated pseudo-quantities; that just tricks you into overriding your intuition for something with no correct basis.

Despite anti-arbitrariness intuitions, there is empirical evidence that this is wrong.

The Robust Beauty of Improper Linear Models

Proper linear models are those in which predictor variables are given weights in such a way that the resulting linear composite optimally predicts some criterion of interest; examples of proper linear models are standard regression analysis, discriminant function analysis, and ridge regression analysis. Research summarized in Paul Meehl's book on clinical versus statistical prediction—and a plethora of research stimulated in part by that book—all indicates that when a numerical criterion variable (e.g., graduate grade point average) is to be predicted from numerical predictor variables, proper linear models outperform clinical intuition. Improper linear models are those in which the weights of the predictor variables are obtained by some nonoptimal method; for example, they may be obtained on the basis of intuition, derived from simulating a clinical judge's predictions, or set to be equal. This article presents evidence that even such improper linear models are superior to clinical intuition when predicting a numerical criterion from numerical predictors. In fact, unit (i.e., equal) weighting is quite robust for making such predictions. The article discusses, in some detail, the application of unit weights to decide what bullet the Denver Police Department should use. Finally, the article considers commonly raised technical, psychological, and ethical resistances to using linear models to make important social decisions and presents arguments that could weaken these resistances.

(this is about something somewhat less arbitrary than using ranks as scores, but it seems like evidence in favor of that approach as well)

You do not use "math" on uncalibrated pseudo-quantities; that just tricks you into overriding your intuition for something with no correct basis.

I'm not sure I understand what you mean by pseudo-quantities.

strength might be most important (10), then cost (9) then wieght (8) and so on.

So the problem is that these attributes were given rankings from 10 down to 1, rather than their weights that corresponded to their actual importance?

Multicriteria objective functions are really hard to get right. Weighting features from 10 to 1 is actually a decent first approach- it should separate good solutions from bad solutions- but if you're down to narrow differences of the weighted objective function, it's typically time to hand off to a human decision-maker, or spend a lot of time considering tradeoffs to elicit the weights. (Thankfully, a first pass should show you what features you need to value carefully and which features you can ignore.)

With this sort of thing, or anything really, you either use bulletproof mathematical models derived from first principles (or empirically) with calibrated real quantities, or you wing it intuitively using your built-in hardware. You do not use "math" on uncalibrated pseudo-quantities; that just tricks you into overriding your intuition for something with no correct basis.

This is why you never use explicit probabilities that aren't either empirically determined or calculated theoretically.

This is a very good general point, one that I natively seem to grasp, but even so I'd appreciate it if you wrote a top-level post about it.