People at MIRI suspect smart decision theories will look at the source code of players, and so aren't purely consequentialist in that sense.
First, I appreciate your original analogy between virtue ethics and source code. I would like to understand it better, since it looks to me like any normative ethics requires "looking into the source code", though pure consequentialism can also be analyzed as a black box. I assume that what you mean is that virtue ethics requires deeper analysis than deontology (because the rules are simple and easy to check against?), and than consequentialism (because one can avoid opening the black box altogether?). Or am I misinterpreting what you said?
Do you think we should be purely consequentialist? It's probably a mistake to ignore certain steelmen of virtue ethics if you care about doing "the right thing.
Well, no, I am not prescriptive in the OP, only descriptive. And I agree (and mentioned here several times) that virtue ethics and deontological rules are in essence precomputed patterns which provide approximate shortcuts to the full unbounded consequentialism in a wide variety of situations. Of course, they are often a case of lost purposes when people elevate then from the level of shortcuts to the complete description of shouldness.
Or am I misinterpreting what you said?
I don't know what "the ultimate decision theory" is, but I suspect this decision theory will contain both consequentialist and virtue ethical elements. It will be "consequentialist" in the trivial sense of picking the best alternative. It will be "virtue ethical" in the sense that it will in general do different things depending on what it can infer about other players based on their source code. In this sense I don't think virtue ethics is a hack approximation to consequentialism, I think it is an orthogonal idea.
That said, I am still confused by what you are trying to say!
Disclaimer: I am not a philosopher, so this post will likely seem amateurish to the subject matter experts.
LW is big on consequentialism, utilitarianism and other quantifiable ethics one can potentially program into a computer to make it provably friendly. However, I posit that most of us intuitively use virtue ethics, and not deontology or consequentialism. In other words, when judging one's actions we intuitively value the person's motivations over the rules they follow or the consequences of said actions. We may reevaluate our judgment later, based on laws and/or actual or expected usefulness, but the initial impulse still remains, even if overridden. To quote Casimir de Montrond, "Mistrust first impulses; they are nearly always good" (the quote is usually misattributed to Talleyrand).
Some examples:
I am not sure how to classify religious fanaticism (or other bigotry), but it seems to require a heavy dose of virtue ethics (feeling righteous), in addition to following the (deontological) tenets of whichever belief, with some consequentialism (for the greater good) mixed in.
When I try to introspect my own moral decisions (like whether to tell the truth, or to cheat on a test, or to drive over the speed limit), I can usually find a grain of virtue ethics inside. It might be followed or overridden, sometimes habitually, but it is always there. Can you?