(Not very familiar with math.)
The Heyting-algebraic definition of implication makes intuitive sense to me, or at least after you state your confusion. 'One circle lies inside the other' is like saying A is a subset of B, which is a statement that describes a relation between two sets, and not a statement that describes a set, so we shouldn't expect that that mental image would correspond to a set. Furthermore, the definition of implication you've given is very similar to the material implication rule; that we may substitute 'P implies Q' with 'not-P or Q'.
Also, I have personally been enjoying your recent posts with few prerequisites. (Seems to be a thing.)
I have what feels like a naive question. Is there any reason that we can't keep appealing to even higher-order preferences? I mean, when I find that I have these sorts of inconsistencies, I find myself making an additional moral judgment that tries to resolve the inconsistency. So couldn't you show the human (or, if the AI is doing all this in its 'head', a suitably accurate simulation of the human) that their preference depends on the philosopher that we introduce them to? Or in other cases where, say, ordering matters, show them multiple orderings, or their simulations' reactions to every possible ordering where feasible, and so on. Maybe this will elicit a new judgment that we would consider morally relevant. But this all relies on simulation, I don't know if you can get the same effect without that capability, and this solution doesn't seem even close to being fully general.
I imagine that this might not do much to resolve your confusion however. It doesn't do much to resolve mine.
Discipline, especially internal psychological, also increases skills.
This is a little ambiguous; does he mean self-control or punishment?
I think these are all points that many people have considered privately or publicly in isolation, but that thus far no one has explicitly written them down and drawn a connection between them. In particular, lots of people have independently made the observation that ontological crises in AIs are apparently similar to existential angst in humans, ontology identification seems philosophically difficult, and so plausibly studying ontology identification in humans is a promising route to understanding ontology identification for arbitrary minds. So, thank you for writing this up; it seems like something that needed to be written quite badly.
Some other problems that might be easier to tackle from this perspective include mind crime, nonperson predicates, and suffering risk, especially subproblems like suffering in physics.
We had succeeded in obtaining John yon Neumann as keynote speaker. He discussed the need for, and likely impact of, electronic computing. He mentioned the "new programming method" for ENIAC and explained that its seemingly small vocabulary was in fact ample: that future computers, then in the design stage, would get along on a dozen instruction types, and this was known to be adequate for expressing all of mathematics. (Parenthetically, it is as true today as it was then that "programming" a problem means giving it a mathematical formulation. Source languages which use "plain English" or other appealing vocabularies are only mnemonic disguises for mathematics.) Von Neumann went on to say that one need not be surprised at this small number, since about 1,000 words were known to be adequate for most situations of real life, and mathematics was only a small part of life, and a very simple part at that. This caused some hilarity in the audience, which provoked von Neumann to say: "If people do not believe that mathematics is simple, it is only because they do not realize how complicated life is."
Franz L. Alt, "Archaeology of computers: Reminiscences, 1945--1947", Communications of the ACM, volume 15, issue 7, July 1972, special issue: Twenty-fifth anniversary of the Association for Computing Machinery, p. 694. PDF.
Neuroscience of art & art appreciation
A slightly broader keyword would be 'neuroaesthetics.'
Evolutionary basis for storytelling
I haven't done an exhaustive literature search, but one book I'm going through right now is Brian Boyd's On the Origin of Stories: Evolution, Cognition, and Fiction.
psychopathology* (Genuinely trying to be helpful, not nitpicky; keywords are important.)
Related, broader keyword: abnormal psychology.
Tangentially, I thought you might find repair theory interesting, if not useful. Briefly, when students make mistakes while doing arithmetic, these mistakes are rarely the effect of a trembling hand; rather, most such mistakes can be explained via a small set of procedural skills that systematically produce incorrect answers.
Will there be LaTeX support?