But to determine how much weight to give Bob as a judge for the second decision, you need to know whether or not Plan A was best for the first decision.
You don't need certainty. And you don't necessarily need that particular evidence. It would still work using calibration tests to weight them.
The only evidence you really have access to from last time is who voted for what, and whether everyone thinks it was a good idea in hindsight. I think that would be enough.
What I believe is that a Bayesian Judge is a tool that operates on probability estimates from other agents- and so if you want to reason this way, then you need this data.
Ok. we are talking about different things. I'm talking about using bayesian methods to integrate evidence like votes, voting records, and hindsight estimations of optimality to determine the best distribution of probability over which plan is best (or some other output).
I have no idea how this "Bayesian Judge" thing that uses probability estimates directly would even work.
I have no idea how this "Bayesian Judge" thing that uses probability estimates directly would even work.
Here's an article on Bayesian aggregation of forecasts. Essentially, you look at past forecasts to get P(Bob: "rain"|rain) and P(Bob: "rain"|~rain). (You can just elicit those expert likelihoods, but if you want this to be a formula rather than a person, you need them to be the data you're looking for instead of just suggestive of the data you're looking for.) From just that, you could calibrate Bob to find out what P(rai...
So here's the problem: Given a well-defined group charter, how should groups make decisions? You have an issue, you've talked it over, and now it's time for the group to take action. Different members have different opinions, because they're not perfect reasoners and because their interests don't reliably align with those of the group. What do you do? Historical solutions include direct democracy, representative democracy, various hierarchies, dictatorships, oligarchies, consensus...But what's the shoes-with-toes solution? How do they do it in Weirdtopia? What is the universally correct method that could be implemented by organizations, corporations, and governments alike?
This Tuesday, I posted an idea. I came up with it about ten, maybe fifteen years ago, decided it was awesome and revolutionary, and spent a few years doing some extremely ineffectual advocacy for it. I've pushed for it in various contexts on and off since then, but I'd basically shelved it until I had a better platform to talk about it. And recently I realized, belatedly, that Less Wrong is probably the perfect audience. You're going to be open to it, and you'll be able to competently critique it. And maybe some of you will use it.
I posted this tease first because the idea is a solution to a problem you may not have thought for five minutes about. I want to give you the opportunity to come up with other solutions, or discuss how to rigorously evaluate ideas. Also, I'm really curious to see whether anybody independently reinvents mine. Over the years, I've had the experience several times of reading someone else's work and thinking they're about to start pitching my idea. But they never do. It all gives me complicated feelings.
Please discuss in the comments. Remember, we're aggregating the opinions, the judgment, of the group members, not their personal preferences. So answers like “iterated runoff voting,” “bargaining theory X,” or “coherent extrapolated volition” are not in themselves what I'm looking for. If you happen to know what my idea is, or manage to find it by google-stalking me, please don't give it away yet.