Interesting idea. I think that rather than a Brier score or log-score (which just indicates how accurate the prosecutor was) it would be better to use something that also indicates the extent to which they specifically over-stated charges. Maybe just something like "On average, the total sentence for defendants charged by this prosecutor was 4.7% of the total of all charges made", though it seems like there would be all sorts of difficulties actually doing that in a way that makes sense.

For something that's impossible to implement, this seems like a surprisingly small step toward EITHER futarchy and the embedding of good conditional predictions in more policy and behavioral decisions, OR to resolving the broken incentives of prosecutors (convictions, not truth).

Chance of being convicted is very weakly correlated with actual guilt, or even procecutor knowledge of guilt. Consider the case of an illegal search that definitively proves guilt, but cannot be presented at trial. The prosecution would probably want to proceed with their 10% case, since they know the defendent is guilty, even though the jury won't have key information.

Also, too, predicting outcomes is a skill separate from ability to generate outcomes. The Tails Come Apart, Goodheart, and all that..

Also, also, too almost no prosecutors are directly elected, judging people by any raw metric is wierd and not done - there would be some arbitrary cutoffs added to translate it to a quantified grading scale, and, as other commenters said, the root cause here is way deeper than "the DA charges 17 counts of assault, one for each blow, plus 3 Kidnapping, one for each grab, etc. Etc. Etc.

An alternative approach that feels less gameable:

Every time a plea bargain is reached, there is a small chance (1 in 1000?) that we randomly select the case for review.  The defendant is imprisoned as per the terms of the plea bargain, but then we also bring the case to trial (federally funding a defense).

If the review trial finds the defendant guilty of the charges they pled to, we say 'okay, seems like the plea bargain was fair, the defendant would have been found guilty anyway'.

If the review trial finds the defendant innocent, we release them and publicize this fact.  If multiple reviews under the same prosecutor exonerate the defendant, it starts to look like that prosecutor is threatening charges to coerce defendants into unfavorable plea bargains.

A 1 in 1000 chance means that the cost of this should not grow too excessive compared to existing costs of the legal system, but also means that a prosecutor who is habitually coercing defendants into taking unfavorable plea deals is very likely to get caught.

The ability of the prosecutor to accurately access the likelihood can be measured via the Briers score or a Log score.

Scoring predictions requires knowing the outcomes. But wouldn't the outcome depend on whether the accused takes plea deals and such?

I would expect that many juries will automatically throw out a 10% or 20% charges because there will be people on the jury who would argue that the prosecutor thinking that there's only a 20% chance that a charge holds means that there's reasonable doubt against the charge.

So then nobody would assign low likelihood charges? I guess that’s a feature of the proposed system, but it seems like a flaw to me. (Probably minor)

What's to stop the prosecutor from lying about their Briers score?

How would prosecutors be trained and incentivized to make and report accurate predictions? Who would be funding the defense attorneys, especially overworked public defenders, to learn to properly evaluate prosecutor's claims? The US legal system already isn't exactly known for being honest today even in terms of things like rules about disclosing exculpatory evidence to the defense. Also, the famous saying about indicting a ham sandwich is anecdotal evidence about how much credit (grand) jurors give prosecutors' likelihood assessments in a pre-trial, non-adversarial environment, and I don't find it encouraging. 

Also, I understand that one of the big issues with plea deals is that it is often in a defendent's individual interest to plead guilty to a lesser charge they're innocent of, even if they're pretty sure they wouldn't be found guilty at trial, either because they're risk averse (not necessarily unreasonable when years of your life are on the line), or because the agreed-on punishment will be less than the time in jail, and cost in fees, they'd incur just by going to trial even if exonerated. Data on likelihood of conviction wouldn't help with that.

At a more basic level, I think this solves the wrong problem. If, as you say, it is in society's best interests to keep legal costs low, then it follows that there is little benefit to society from the larger penalties imposed by overcharging defendants. This suggests that the problem lies in the criminal code and sentencing guidelines, and that these need serious updating. 

And given how biased our legal system is and has historically been, I believe a proposal based on conviction odds has basically the same problems as current AI tools for criminal sentencing and risk assessment. Large racial and economic biases, among other problems.

Edit to add: I do like the idea of giving defendents honest info about their likelihood of conviction if a case goes to trial. I'm extremely skeptical of the idea of having prosecutors provide that evidence, and wary of letting prosecutors, juries, or judges see it.

While I think there are some flaws with this proposal, my biggest question is "Why not do other things that are done abroad and we know work?"

The US system for deals is really strange. In the UK and many european countries there's no such thing as a deal. The police can charge you with a crime. The earlier in the process you admit guilt, the bigger % a sentence reduction you get. This seems to solve the overcharging problem, a prosecutor can't charge you with crime X and then give you a deal for crime Y, and aligns incentives better as prosecutors can't fear people into taking plea deals as easily. It also preserves much of the benefit of plea bargaining by still allowing suspects to confess when the evidence is overwhelming and hence save lot's of time and money.

Another idea: as things stand today, prosecutors are only allowed to file charges when they have "probable cause". Courts won't put a number on what that means, but you might reasonably approximate it as 25% certainty that the defendant did it. We could insist on a higher standard - preponderance of the evidence (which courts do define as > 50% certainty) or clear and convincing evidence (which courts won't put a number on but you could reasonably approximate as 75% certainty).

I think you are forgetting a major player in the criminal justice system: the defense attorney. The defense attorney already has both the expertise and the incentive to accurately advise the defendant as to the likelihood of a conviction on each charge.