Thanks for the vote of confidence. I should say that while I think my paper presents things well, I cannot take credit for the statistics or experimental design. Tim van Gelder already had the machinery in place to measure pre-post gains, and had done so for several semesters. The results were published in Donohue et al. 2002. The difference here was that I took over teaching, and we continued the pre-post tests.
Although argument maps are usually used to map existing natural language arguments, one could start with the map. I like to think that the more people use these maps, the more their thinking naturally follows such a structure. I'm sure I could use more practice myself.
Just a note on terminology: the tree does have two kinds of nodes, but by virtue of being a tree, it is not a bipartite graph.
I think arguments in argument maps can be made probabilistic and converted to Bayesian networks. But as it is, it takes long enough just to make an argument map. I've recently discovered Gheorghe Tecuci's work. He's just down the hall from me, but I didn't know his work until I heard him give a talk recently. He has an elaborate system that helps analysts create structures very much like argument maps by filling in schemas, and then reasons quantitatively with them. The tree structure and the simplicity of the combination rules (min, max, average, etc.) are more limited than a full Bayesian network, but it seems to be a very nice extension of argument maps.
Charles R. Twardy provides evidence that a course in argrument mapping, using a particular software tool improves critical thinking. The improvement in critical thinking is measured by performance on a specific multiple choice test (California Critical Thinking Skills Test). This may not be the best way to measure rationality, but my point is that unlike almost everybody else, there was measurement and statistical improvement!
Also, his paper is the best, methodologically, that I've seen in the field of "individual rationality augmentation research".
To summarize my (clumsy) understanding of the activity of argument mapping:
One takes a real argument in natural language. (op-eds are a good source of short arguments, philosophy is a source of long arguments). Then elaborate it into a tree structure, with the main conclusion at the root of the tree. The tree has two kinds of nodes (it is a bipartite graph). The root conclusion is a "claim" node. Every claim node has approximately one sentence of english text associated. The children of a claim are "reasons", which do NOT have english text associated. The children of a reason are claims. Unless I am mistaken, the intended meaning of the connection from a claim's child (a reason) to the parent is implication, and the meaning of a reason is the conjunction of its children.
In elaborating the argument, it's often necessary to insert implicit claims. This should be done abiding by the "Principle of Charity", that you should interpret the argument in such a way as to make it the strongest argument possible.
There are two syntactic rules which can easily find flaws in argument maps:
The Rabbit Rule: Informally, "You can't conclude something about rabbits if you haven't been talking about rabbits". Formally, "Every meaningful term in the conclusion must appear at least once in every reason."
The Holding Hands Rule: Informally, "We can't be connected unless we're holding hands". Formally, "Every meaningful term in one premise of a reason must appear at least once in another premise of that reason, or in the conclusion".
I have tried the Rationale tool, and it seems afflicted with creeping featurism. My guess is the open-source tool Freemind could support argument mapping as described in Twardy's article, if the user is disciplined about it.
I'd love comments offering alternative rationality-improvement tools. I'd prefer tools intended for solo use (that is, prediction markets are awesome but not what I'm looking for) and downloadable rather than web services, but anything would be great.