In my model of human behavior, there is an unobservable parameter I associate with intelligence. I observe people's behavior when playing a game, when solving a problem, when defending their beliefs, or when learning something new, and I infer something about their intelligence. This in turn informs my predictions about their success in a wide variety of pursuits. In practice, I often make strong predictions about someone's intelligence after observing their behavior on a single occasion.
An accurate conception of intelligence seems to be generally important. Understanding what easily leveraged factors affect someone's intelligence---during childhood, later education, and after formal education is complete---is important if your goal is improving intelligence generally. If you are considering relatively expensive personal engagement to develop rationality, you may want to direct efforts at individuals who have the potential to have a significant impact as researchers or entrepreneurs. And so on.
Before thinking about how to understand determiners of intelligence, how to measure intelligence effectively, or the effects of intelligence on behavior, I would first like to get a feel for what my intuitive understanding of intelligence really corresponds to, if anything. It is possible that my intuitive assessments of intelligence are largely unrelated to reality, and that my beliefs about the world could be improved by discarding them. It is also possible that some of my intuitions about intelligence are quite accurate, and I could make better decisions by giving them more credence or by changing the way I use those intuitive judgments.
Intuitively, I expect the results of many types of otherwise apparently unrelated tests to be very tightly correlated with intelligence. To understand the extent to which this intuition is correct, I am considering conducting a slightly systematic study of the relationship between different metrics. I would appreciate pointers to reliable scholarship surrounding this question, but a brief search turned up mostly very muddled thinking and a general lack of people doing good experiments.
Here is a range of metrics which I suspect correlate well with my conception of intelligence, at least in certain regimes (some of these metrics may only correlate meaningfully when applied to very bright subjects, or may not correlate meaningfully when applied to very bright subjects):
1. General intelligence factor as estimated by standardized cognitive tests, e.g. Raven's Progressive Matrices.
2. Ability to quickly learn an unfamiliar formalism. For example, to quickly learn a new game and to understand simple strategic consequences of its rules.
3. Ability to infer an underlying model. For example, to learn how to achieve a goal when allowed some constrained interaction with / observation of an unknown environment.
4. My assessment of intelligence during collaboration or discussion of a complex but rigorously defined topic; or, the assessment of anyone who I consider to be intelligent.
5. Ability to solve hard problems in a well-understood environment, potentially given hours or days. For example, performance in high school olympiads.
My hope is that by gaining a better understanding of the relationship between these metrics I may learn to what extent my current rather monolithic conception of intelligence is valid and, to whatever extent it is, how to effectively measure it. Ultimately I would like to understand what easy measurements are the best indicators of success at various particular pursuits, but is even more extraordinarily difficult to acquire data about how good someone is at, say, choosing good research problems.
What do readers expect the results of inquiry to look like? Is my choice of metrics influenced unduly by my own experience? What are other metrics I should be considering but am not? Is improving a student's ability to perform any of these tasks likely to have a positive influence on other tasks?
paulfchristiano:
That was my impression too when I tried making some sense of this area. Nevertheless, based on the literature I've seen, I think one can reliably say the following about your five items:
Raven's and similar tests are definitely not the gold standard for pure g measurement that they were once thought to be. The Flynn effect has had the largest magnitude exactly on this sort of tests, and people can be trained to improve their scores on them significantly. (Though proponents if IQ would of course claim that this ruins their predictive validity.)
My impression is that this would be a very g-intensive task, possibly the most g-intensive sort of task at all. A really interesting evaluation of the Flynn effect would be to see how much it has affected people's performance on tasks of this sort, but I'm not familiar with any literature addressing this question.
This sort of task may involve a bunch of other abilities largely unrelated to intelligence, depending on the nature of the problem. To take the most important example, if the problem requires figuring out the thoughts and motivations of other people, someone with an extremely high general intelligence but slightly autistic will likely perform worse than average. If the problem is completely formalized and symbolic, I'd say it's little different from (2).
This is tricky. People of mediocre or even low intelligence but with great charisma and self-presentation skills can be surprisingly capable of fooling others into thinking they're much smarter than they really are. Even if the interaction is purely about some formal and logically rigorous issue, your subjective impression may end up being much more favorable than if you applied a predefined set of formal criteria for evaluation.
This is about intelligence as well as conscientiousness. I don't know what's the correlation between these in the general population (and I doubt anyone knows precisely), but it's certainly above zero. On the other hand, there are definitely people with one much better than the other. Which is more important depends on how novel and tough the problems are, how tiresome and tedious the tasks are, and how much time there is for preparation. For example, someone of mediocre intelligence can ace a math exam by working through a whole thick problem book beforehand, but this would not work for a math olympiad.
Take all this with the disclaimer that I'm just an amateur in this area, though I have read a fair bit of research literature in it at one point.