Your 2 implies the following claim:
Agent 1 knows that agent 2 knows E if and only if agent 2 knows that agent 1 knows E.
How does it imply that? (It well might, within the context of the agreement theorem. My recollection is that you assume from the start that A1 and A2 have common knowledge of E.)
This claim is obviously false.
Why? If knowledge means "justified true belief", then for agent 1 to know that agent 2 know E, agent 1 must also know E, and vice-versa. This doesn't prove the claim that you say I am making, but goes most of the way towards proving it.
Suppose P1(w)={w,v}, P2(w)={w,u}. Then at w, I know E={w,v,u}, but at v, I do not know E! So I can distinguish between w and v by checking my knowledge, even though I cannot distinguish between w and v!
This is true, except that P1 and P2 should range over events, not over world states. This is a step that the theorem relies on. Are you claiming that this is false?
Terms: P1 means what Aumann calls P-superscript1; N1 means what Aumann calls cursive-P superscript 1. P1(E) = {w,v} means that, after observing event E, A1 knows that the world is in one of the states {w, v}. N1 is the set that describes the range of P1. E is an event, meaning a set of possible world states. Aumann doesn't define what an 'event' is, other than implicitly in how he uses the variable E, so I hope I'm getting that right.
I'm constructing this partly from memory - sorry, this is a complex proof, and Aumann's paper is skimpy on definitions, several of which (like "meet" and "join") are left undefined and hard to find defined anywhere else even with Google. I really can't do this justice without more free time than I have in the next several months.
What I think Aumann is saying is that, if A1 knows E, and knows that A2 knows E, then for every state x in P1(E), for every event D such that x is in P2(D) , P2(D) is a subset of E. Saying this allows Aumann to go on and show that A1 and A2 can iteratively rule out possibilities until they converge on believing the same thing.
This requires knowing more than what we mean when we say "A1 knows that A2 knows E". When we say that, we mean that A1 knows the world is in one of the states in E, and knows that A2 knows the world is in one of the states in E. But it is possible that there is some state x, that the world is not in, but that is a member of E and of P1(E), but not of P2(E).
My recollection is that this is the problem: If you only consider conditions involving 3 possible world states, like the w, u, and v in the above example, then you can show that these things are equivalent. The agents can always use their mutual knowledge to iteratively eliminate possible states until they agree. For instance, if the situation is that P1({w,v,u})={w,v}, P2({w,v,u})={w,u}, then P1 and P2 can use their common knowledge to conclude w, and thus agree. But if you consider conditions where P1, P2, and E contain more than 3 different states between them, you can find situations that have multiple possible solutions, which the agents cannot choose between; and so cannot converge.
How does it imply that?
The definition you gave was symmetric. If I misread it, my apologies.
Why? If knowledge means "justified true belief", then for agent 1 to know that agent 2 know E, agent 1 must also know E, and vice-versa. This doesn't prove the claim that you say I am making, but goes most of the way towards proving it.
True, but it's impossible to go the rest of the way. If you see a dog and I see both you and the dog through a one-way mirror, then I know that you know that there's a dog there but you don't know that I know that the...
Recent brainstorming sessions at SIAI (with participants including Anna, Carl, Jasen, Divia, Will, Amy Willey, and Andrew Critch) have started to produce lists of rationality skills that we could potentially try to teach (at Rationality Boot Camp, at Less Wrong meetups, or similar venues). We've also been trying to break those skills down to the 5-second level (step 2) and come up with ideas for exercises that might teach them (step 3) although we haven't actually composed those exercises yet (step 4, where the actual work takes place).
The bulk of this post will mainly go into the comments, which I'll try to keep to the following format: A top-level comment is a major or minor skill to teach; upvote this comment if you think this skill should get priority in teaching. Sub-level comments describe 5-second subskills that go into this skill, and then third-level comments are ideas for exercises which could potentially train that 5-second skill. If anyone actually went to the work of composing a specific exercise people could run through, that would go to the fourth-level of commenting, I guess. For some major practicable arts with a known standard learning format like "Improv" or "Acting", I'll put the exercise at the top and guesses at which skills it might teach below. (And any plain old replies can go at any level.)
I probably won't be able to get to all of what we brainstormed today, so here's a PNG of the Freemind map that I generated during our session.