I think considerably more than two things have to go well for your interpretation to succeed in describing this post.
Indeed. That is because you needed more than two things to go right for your post to succeed in communicating your point ;).
My confusion is over this sentence from your post:
This thought experiment contrasts "logical knowledge" (the usual kind) and "observational knowledge" (what you get when you look at a calculator display).
My difficulty is that everything that I would call knowledge is like what you get when you look at a calculator display. Suppose that the test had asked you whether "2+2" reduced to an even number. Then you would perform certain mental operations on this expression, and you would answer in accordance with how those operations concluded. (For example, you might picture two sets of two dots, one set next to the other, and see whether you can pair off elements in one set with elements in the other. Or you might visualize a proof in Peano arithmetic in your mind, and check whether each line follows from the previous line in accordance with the rules of inference.) At any rate, whatever you do, it amounts to relying on the imperfect wetware calculator that is your brain. If a counterfactual version of you got a different answer with his brain, you would still want his test sheet to match his answer.
So, what is the residue left over, after we set aside observational knowledge? What is this "logical knowledge"? Calling it "the usual kind" is not sufficing to pick out what you mean for me.
My guess was that your "logical knowledge" includes (in your terminology) the "moral arguments" that "the agent can prove" in the "theory it uses". The analogous role in Wie Dai's "brute-force" UDT is served by the agent's computation of an expected utility EU(f) for an input-output map f.
Is this a correct interpretation of what you meant by "logical knowledge"? (I know that I may need more than two things to go right to have interpreted you correctly. That is why I am giving you my interpretation of what you said. If I got it right, great. But my main motivation arises in the case where I am wrong. My hope is that you will then restate your claim, this time calibrating for the way that I am evidently primed to misinterpret you. If I were highly confident that I had understood you correctly, I wouldn't bother echoing what you said back at you.)
Also, calculator is correct 99% of the time, so you've probably labeled things in a confusing way that could lead to incorrect solution, although the actual resulting numbers seem fine for whatever reason.
Could you spell out how exactly the 99% correctness rate means that I've probably labeled things confusingly? What is the first probably-confusing label that I used?
What I gave looks to me to be the by-the-book way to state and solve your problem within the UDT1.1 formalism. How would you set up the problem within the UDT1.1 formalism? In particular, what would be your set of possible sequences of execution histories for the world-programs?
My difficulty is that everything that I would call knowledge is like what you get when you look at a calculator display.
In some sense, sure. But you still have to use certain specific reasoning procedure to think about imperfection of knowledge-acquisition methods. That level where you just perform the algorithm is where logic resides. It's not clear to me how to merge these considerations seamlessly.
...My guess was that your "logical knowledge" includes (in your terminology) the "moral arguments" that "the agent can prove"
Consider the following thought experiment ("Counterfactual Calculation"):
Should you write "even" on the counterfactual test sheet, given that you're 99% sure that the answer is "even"?
This thought experiment contrasts "logical knowledge" (the usual kind) and "observational knowledge" (what you get when you look at a calculator display). The kind of knowledge you obtain by observing things is not like the kind of knowledge you obtain by thinking yourself. What is the difference (if there actually is a difference)? Why does observational knowledge work in your own possible worlds, but not in counterfactuals? How much of logical knowledge is like observational knowledge, and what are the conditions of its applicability? Can things that we consider "logical knowledge" fail to apply to some counterfactuals?
(Updateless analysis would say "observational knowledge is not knowledge" or that it's knowledge only in the sense that you should bet a certain way. This doesn't analyze the intuition of knowing the result after looking at a calculator display. There is a very salient sense in which the result becomes known, and the purpose of this thought experiment is to explore some of counterintuitive properties of such knowledge.)