The purpose of this post is to provide basic information about the LSAT including the format  of the test and a few sample questions. I also wanted to bring light to some research that has found LSAT preparation to alter brain structure in ways that strengthen hypothesized "reasoning pathways". These studies have not been discussed here before; I thought they were interesting and really just wanted to call your collective attention to them.

I really like taking tests; I get energized by intense race-against-the-clock problem solving and, for better or worse, I relish getting to see my standing relative to others when the dust settles. I like the the purity of the testing situation --how conditions are standardized in theory and more or less the same for all comers. This guilty pleasure has played no small part in the course my life has taken: I worked as a test prep tutor for 3 years and loved every minute of it, I met my wife through academic competitions in high school, and I am a currently a graduate student doing lots of coursework in psychometrics.

Well, my brother-in-law is a lawyer, and when we chat the topic of the LSAT has served as some conversational common ground. Since I like taking tests for fun, he suggested I give it a whirl because he thought it was interesting and felt like it was a fair assessment of one's logical reasoning ability. So I did, I took a practice test cold a couple Saturdays ago and I was very impressed. Here the one I took. (This is a full practice exam provided by the test-makers; it's also like the top google result for "LSAT practice test".) I wanted to post here about it because the LSAT hasn't been discussed very much on this site and I thought that some of you might find it useful to know about.

A brief run-down of the LSAT:

The test has four parts: two Logical Reasoning sections, a Critical Reading section (akin to SAT et al.), and an Analytical Reasoning, or "logic games", section. Usually when people talk about the LSAT, the logic games get emphasized because they are unusual and can be pretty challenging (the only questions I missed were of this type; I missed a few and I ran out of time). Essentially, you get a premise and a bunch of conditions from which you are required to draw conclusions. Here's an example:

A cruise line is scheduling seven week-long voyages for the ship Freedom. 
Each voyage will occur in exactly one of the first seven weeks of the season: weeks 1 through 7. 
Each voyage will be to exactly one of four destinations:Guadeloupe, Jamaica, Martinique, or Trinidad. 
Each destination will be scheduled for at least one of the weeks. 
The following conditions apply: Jamaica will not be its destination in week 4. 
Trinidad will be its destination in week 7. Freedom will make exactly two voyages to Martinique, 
and at least one voyage to Guadeloupe will occur in some week between those two voyages. 
Guadeloupe will be its destination in the week preceding any voyage it makes to Jamaica. 
No destination will be scheduled for consecutive weeks.

11. Which of the following is an acceptable schedule of destinations in order from week 1 through week 7?

(A) Guadeloupe, Jamaica, Martinique, Trinidad,Guadeloupe, Martinique, Trinidad
(B) Guadeloupe, Martinique, Trinidad, Martinique, Guadeloupe, Jamaica, Trinidad
(C) Jamaica, Martinique, Guadeloupe, Martinique, Guadeloupe, Jamaica, Trinidad
(D) Martinique, Trinidad, Guadeloupe, Jamaica, Martinique, Guadeloupe, Trinidad
(E) Martinique, Trinidad, Guadeloupe, Trinidad, Guadeloupe, Jamaica, Martinique

Clearly, this section places a huge burden on working memory and is probably the most g-loaded of the four. I'd guess that most LSAT test prep is about strategies for dumping this burden into some kind of written scheme that makes it all more manageable. But I just wanted to show you the logic games for completeness; what I was really excited by were the Logical Reasoning questions (sections II and III). You are presented with some scenario containing a claim, an argument, or a set of facts, and then asked to analyze, critique, or to draw correct conclusions. Here are most of the question stems used in these sections:

Which one of the following most accurately expresses the main conclusion of the economist’s argument?
Which one of the following uses flawed reasoning that most closely resembles the flawed reasoning in the argument?
Which one of the following most logically completes the argument?
The reasoning in the consumer’s argument is most vulnerable to criticism on the grounds that the argument...
The argument’s conclusion follows logically if which one of the following is assumed?
Which one of the following is an assumption required by the argument?

Heyo! This is exactly the kind of stuff I would like to become better at! Most of the questions were pretty straightforward, but the LSAT is known to be a tough test (score range: 120-180, 95th %ile: ~167, 99th %ile: ~172) and these practice questions probably aren't representative. What a cool test though! Here's a whole question from this section, superficially about utilitarianism:

3. Philosopher: An action is morally right if it would be reasonably expected
to increase the aggregate well-being of the people affected by it. An action
is morally wrong if and only if it would be reasonably expected to reduce the 
aggregate well-being of the people affected by it. Thus, actions that would 
be reasonably expected to leave unchanged the aggregate well-being of the
people affected by them are also right.

The philosopher’s conclusion follows logically if which one of the following is assumed?

(A) Only wrong actions would be reasonably expected to reduce the aggregate 
 well-being of the people affected by them.
(B) No action is both right and wrong.
(C) Any action that is not morally wrong is morally right.
(D) There are actions that would be reasonably expected to leave unchanged the
  aggregate well-being of the people affected by them.
(E) Only right actions have good consequences.

Also, the LSAT is a good test, in that it measures well one's ability to succeed in law school. Validity studies boast that “LSAT score alone continues to be a better predictor of law school performance than UGPA [undergraduate GPA] alone.” Of course, the outcome variable can be regressed on both predictors and account for more of the variance than either one taken singly, but it is uncommon for a standardized test to beat prior GPA in predicting a students future GPA.

Intensive LSAT preparation and neuroplasticity:

In two recent studies (same research team), learning to reason in the logically formal way required by the LSAT was found to alter brain structure in ways consistent with literature reviews of the neural correlates of logical reasoning. Note: my reading of these articles was pretty surface-level; I do not intend to provide a thorough review, only to bring them to your attention.

These researchers recruited pre-law students enrolling in an LSAT course and imaged their brains at rest using fMRI both before and after 3 months of this "reasoning training". As controls, they included age- and IQ-matched pre-law students intending to take LSAT in the future but not actively preparing for it.

The LSAT-prep group was found to have significantly increased connectivity between parietal and prefrontal cortices and the striatum, both within the left hemisphere and across hemispheres. In the first study, the authors note that

These experience-dependent changes fall into tracts that would be predicted by prior work showing that reasoning relies on an interhemispheric frontoparietal network (for review, see Prado et al., 2011). Our findings are also consistent with the view that reasoning is largely left-hemisphere dominent (e.g., Krawczyk, 2012), but that homologous cortex in the right hemisphere can be recruited as needed to support complex reasoning. Perhaps learning to reason more efficiently involves recruiting compensatory neural circuitry more consistently.

And in the second study, they conclude

An analysis of pairwise correlations between brain regions implicated in reasoning showed that fronto-parietal connections were strengthened, along with parietal-striatal connections. These findings provide strong evidence for neural plasticity at the level of large-scale networks supporting high-level cognition.

I think this hypothesized fronto-parietal reasoning network is supposed to go something like this:

The LSAT requires a lot of relational reasoning, the ability to compare and combine mental representations. The parietal cortex holds individual relationships between these mental representations (A->B, B->C), and the prefrontal cortex integrates this information to draw conclusions (A->B->C, therefore A->C). The striatum's role in this network would be to monitor the success/failure of reward predictions and encourage flexible problem solving. Unfortunately, my understanding here is very limited. Here are several reviews of this reasoning network stuff (I have not read any; just wanted to share them): Hampshire et al. (2011), Prado et al. (2011), Krawczyk (2012).

I hope this was useful information! According to the 2013 survey, only 2.2% of you are in law-related professions, but I was wondering (1) if anyone has personal experience studying for this exam, (2) if they felt like it improved their logical reasoning skills, and (3) if they felt that these effects were long-lasting. Studying for this test seems to have the potential to inculcate rationalist habits-of-mind; I know it's just self-report, but for those who went on to law school, did you feel like you benefited from the experience studying for the LSAT? I only ask because the Law School Admission Council, a non-profit organization made up of 200+ law schools, seems to actively encourage preparation for the exam, member schools say it is a major factor in admissions, preparation tends to increase performance, and LSAT performance is correlated moderately-to-strongly with first year law school GPA (r= ~0.4).

Thoughts inspired by Yvain's philosophical role-playing post.

Thomas Nagel produced a famous philosophical thought experiment "What Is It Like to Be A Bat?" In it, he argued that the reductionist understanding of consciousness was insufficient, since there exists beings - bats - that have conscious experiences that humans cannot understand. We cannot know what "it is like to be a bat", and looking reductively at bat brains, bat neurones, or the laws of physics, cannot (allegedly) grant us any understanding of this subjective experience. Therefore there remains an unavoidable subjective component to the problem of consciousness.

I won't address this issue directly (see for instance this, on the closely related subject of qualia), but instead look at the question: suppose someone told us that they actually knew what it was like to be a bat (as well as what it was like to be a human). Call such a being a vampire, for obvious reasons. So if someone claimed they were a vampire, how would we test this?

We can't simply ask them to describe what it's like to be a bat - it's perfectly possible they know what it's like to be a bat, but cannot describe it in human terms (just as we often fail to describe certain types of experiences to those who haven't experienced them). Could we run a sort of Turing test - maybe implant the putative vampire's brain into a bat body, and see how bat-like it behaved? But, as Nagel pointed out, this could be a test of whether they know how to behave like a bat behaves, not whether they know what it's like to be a bat.

I posit that one possible solution is to use the approach laid out in my post "the flawed Turing test". We need to pay attention as to how the "vampire" got their knowledge. If the vampire is a renown expert on bat behaviour and social interactions, who is also interested in sonar and paragliding - then them functioning as a bat is weak evidence as to them actually knowing what it is like to be a bat. But suppose instead that their knowledge comes from another source - maybe the vampire is a renown brain expert, who has grappled with philosophy of mind and spent many years examining the functioning of bat brains. But, crucially, they have never seen a full living bat in the wild or in the lab, they've never watched a natural documentary on bats, they've never even seen a photo of a bat. In that case, if they behave correctly when transplanted into a bat body, then it's strong evidence of them actually understanding what it's like to be a bat.

Similarly, maybe they got their knowledge after a long conversation with another "vampire". We have the recording of the conversation, and it's all about mental states, imagery, emotional descriptions and visualisation exercises - but not about physical descriptions or bat behaviour. In that case, as above, if they can function successfully as a bat, this is evidence of them really "getting it".

In summary, we can say "that person likely knows what it is like to be a bat" if "knowing what it's like to be a bat" is the most likely explanation for what we see. If they behave exactly like a bat when in a bat body, and we know they have no prior experience that teaches them how to behave like a bat (but a lot about the bat's mental states), then we can conclude that it's likely that they genuinely know what it's like to be a bat, and are implementing this knowledge, rather than imitating behaviour.

The Blob Family is a simple game made by Leon Arnott. At heart, it's a game about testing hypotheses and getting the right answer with the least amount of evidence you can.

The mechanics work like so: Balls bounce around the screen randomly and you control a character who needs to avoid them. You can aim the mouse anywhere and activate a sonar. On the right side are rules for how various balls will react to this, and your goal is to figure out which ball is which. As you use the sonar more, the balls speed up, so it becomes more difficult to stay alive, thus giving an incentive to test your hypothesis in as few clicks as possible.

It very nicely illustrates the principle that, to test a hypothesis, you must design tests to falisfy your intuitions rather than to confirm them. For example, in one level, when you use the sonar:

• 1 ball heads toward the center
• 1 ball heads away from the center
• 1 ball heads away from the mouse
• 1 ball heads away from you

I found myself mistakenly clicking in the center of the screen to test hypothesis 1, but this is insufficient. To design the proper tests, you need to keep the mouse out of the center, keep it away from you, and depending on the position of the balls keep it off a straight line from you.

It could also demonstrate the ability of a fast brain to test hypotheses quickly. For many levels, if you could slow time down and set up a very good test, you could solve the problem with a single click. But we humans aren't usually so attentive.

Just thought the LW crowd might enjoy it.

Related to: Beyond Bayesians and Frequentists

Update: This comment by Cyan clearly explains the mistake I made - I forgot that the ordering of the hypothesis space is important is necessary for hypothesis testing to work. I'm not entirely convinced that NHST can't be recast in some "thin" theory of induction that may well change the details of the actual test, but I have no idea how to formalize this notion of a "thin" theory and most of the commenters either 1) misunderstood my aim (my fault, not theirs) or 2) don't think it can be formalized.

I'm teaching an econometrics course this semester and one of the things I'm trying to do is make sure that my students actually understand the logic of the hypothesis test. You can motivate it in terms of controlling false positives but that sort of interpretation doesn't seem to be generally applicable. Another motivation is a simple deductive syllogism with a small but very important inductive component. I'm borrowing the idea from a something we discussed in a course I had with Mark Kaiser - he called it the "nested syllogism of experimentation." I think it applies equally well to most or even all hypothesis tests. It goes something like this:

1. Either the null hypothesis or the alternative hypothesis is true.

2. If the null hypothesis is true, then the data has a certain probability distribution.

3. Under this distribution, our sample is extremely unlikely.

4. Therefore under the null hypothesis, our sample is extremely unlikely.

5. Therefore the null hypothesis is false.

6. Therefore the alternative hypothesis is true.

An example looks like this:

Suppose we have a random sample from a population with a normal distribution that has an unknown mean and unknown variance . Then:

1. Either or where is some constant.

2. Construct the test statistic where is the sample size, is the sample mean, and is the sample standard deviation.

3. Under the null hypothesis, has a distribution with degrees of freedom.

4. is really small under the null hypothesis (e.g. less than 0.05).

5. Therefore the null hypothesis is false.

6. Therefore the alternative hypothesis is true.

What's interesting to me about this process is that it almost tries to avoid induction altogether. Only the move from step 4 to 5 seems anything like an inductive argument. The rest is purely deductive - though admittedly it takes a couple premises in order to quantify just how likely our sample was and that surely has something to do with induction. But it's still a bit like solving the problem of induction by sweeping it under the rug then putting a big heavy deduction table on top so no one notices the lumps underneath. 

This sounds like it's a criticism, but actually I think it might be a virtue to minimize the amount of induction in your argument. Suppose you're really uncertain about how to handle induction. Maybe you see a lot of plausible sounding approaches, but you can poke holes in all of them. So instead of trying to actually solve the problem of induction, you set out to come up with a process which is robust to alternative views of induction. Ideally, if one or another theory of induction turns out to be correct, you'd like it to do the least damage possible to any specific inductive inferences you've made. One way to do this is to avoid induction as much as possible so that you prevent "inductive contamination" spreading to everything you believe. 

That's exactly what hypothesis testing seems to do. You start with a set of premises and keep deriving logical conclusions from them until you're forced to say "this seems really unlikely if a certain hypothesis is true, so we'll assume that the hypothesis is false" in order to get any further. Then you just keep on deriving logical conclusions with your new premise. Bayesians start yelling about the base rate fallacy in the inductive step, but they're presupposing their own theory of induction. If you're trying to be robust to inductive theories, why should you listen to a Bayesian instead of anyone else?

Now does hypothesis testing actually accomplish induction that is robust to philosophical views of induction? Well, I don't know - I'm really just spitballing here. But it does seem to be a useful steel man.