This question exists in the awkward space between "things undergrads google for homework" and "things on the cutting edge," so google isn't being super helpful.
I have a number I want a computer to estimate. Right now I have two regression models and an insider methodology. The former can be used to create two normal curves. The latter creates a point estimate only, but I can back into a confidence interval/normal curve with an acceptable amount of arbitrary hand-waving. If necessary, this could be conceived of as a prior.
How can I automatically weight the three curves into a single point estimate? I vaguely remember something from an econometrics class about weighting forecasts in a way that minimized total standard error, but I tried to work the math out myself and I didn’t know how to deal with the covariances of the forecasts. Can I simply assume the forecast covariances are zero?
This seems like a good place to use Bayes’ law, but I don't know how to formally set it up.
Edit to Add: Bayesian statistics is still new to me, so forgive me for being a bit dense. Here's my understanding of the methodology right now.
What exactly is D in this scenario?
Nope. Just a basic sanity check here: If your forecast covariances are zero, then their correlations are zero. You want your forecasts to be correlated with the truth, so if they are, they should also be correlated with each other.
The google keyword you're looking for is "Bayesian model averaging." I've never seen anyone use Bayesian model averaging to average frequentist estimates, but I'd bet at even odds that it's been done before.
As a quick run through, this is what you do. Denote your three models as M1, M2, and M3 (with Mi or Mk denoting model i or k). Set a prior probability for each model being 'true,' e.g. if you have no information to differentiate the models, set P(M1)=P(M2)=P(M3)=1/3. You might underweight the regression models because you trust the insider methodology more or perhaps the opposite. Whatever prior you choose, it's just a matter of calculating posterior model weight using Bayes rule. Formally, if X is the data and Lk() is the likelihood function for model k:
P(Mk | X) = Lk(X)*P(Mk) / sum_i[ Li(X)*P(Mi) ]
Then you average the forecasts according to P(Mk | X).
If I were using Bayesian model averaging, I'd also want to do Bayesian analyses of the individual models - I'm not sure what using frequentist estimates does to the posterior model probabilities - but you can probably find more details on google.
edit: notation
This makes sense. Using Bayes rule to develop the weights was the (/a) missing link for me. I was trying to do it all conditional on the possible outcomes.
Correct me if I'm wrong, but there should be a different weight between the models at different parts of the dependent variable? When the dependent variable is near its mean, the regressions will have narrower forecast distributions and so less weight should go to the insider methodology.