It's interesting that you found Piaget's "formal operational stage" so applicable. I remember when I learned about it (also in a psychology course at around the same age) I found the claim that people only develop abstract thought at the age of 12 completely ridiculous. This is probably related to how my own development was very anomalous.

I think the real nail in the coffin would be if a young child does not understand a particular concept (say, volume conservation) and it is found that you can teach them this concept before they are supposed to be developmentally ready.

The article doesn't discuss conservation of volume in detail, but it talks about an experiment that's said to be "conceptually similar". And while it's hard to say from the quote, it seems to imply that when children are given feedback on the similar problem, their performance improves (I've bolded that part):

The child is shown two rows of objects, say, pennies. Each row has the same number of pennies and they are aligned, one for one. The child will agree that the rows are the same. Then the experimenter changes one row by pushing the pennies farther apart. Now, the experimenter asks, which row has more? (Pennies might also be added to or subtracted from a line.) Younger children will say that the longer line has more pennies.

When Piaget (1952) developed this task he argued that children go through three stages on their way to successfully solving this problem. Initially they cannot process both the length of the rows and the density of coins in the rows, so they focus on just one of these, usually saying that the longer row has more. The next stage is brief, and is characterized by variable performance: children sometimes use row length and sometimes row density to make their judgment, sometimes they use both but cannot say why they did so, and sometimes they simply say that they are unsure. In the third stage, children have grasped the relevant concepts and consistently perform correctly.

Robert Siegler (1995) showed that children’s performance on this task doesn’t develop that way. Ninety-seven 4- to 6-year-olds who initially could not solve the problem were studied, with each child performing variants of the problem a total of 96 times over eight sessions. After each problem, children were asked to explain why they gave the answer they did, so there was ample opportunity to examine the consistency of the children’s performance and their reasoning. The experimenter found a good deal of inconsistency. Children used a variety of explanations— sophisticated and naïve—throughout, even though they became more accurate with experience (the experimenter provided accuracy feedback, which is a big help to learning). It was not the case that once the child “got it” he consistently used the correct strategy. If the child gave a good explanation for a problem, there was only a 43 percent chance of his advancing the same explanation when later confronted with the identical problem.

Criticism of Piaget's theory

What is “developmentally appropriate practice”? For many teachers, I think the definition is that school activities should be matched to children’s abilities—they should be neither too difficult nor too easy, given the child’s current state of development. The idea is that children’s thinking goes through stages, and each stage is characterized by a particular way of understanding the world. So if teachers know and understand that sequence, they can plan their lessons in accordance with how their students think.

In this column I will argue that this notion of developmentally appropriate practice is not a good guide for instruction. In order for it to be applicable in the classroom, two assumptions would have to be true. One is that a child’s cognitive development occurs in discrete stages; that is, children’s thinking is relatively stable, but then undergoes a seismic shift, whereupon it stabilizes again until the next large-scale change. The second assumption that would have to be true is that the effects of the child’s current state of cognitive development are pervasive—that is, that the develop mental state affects all tasks consistently.

Data from the last 20 years show that neither assumption is true. Development looks more continuous than stage-like, and the way children perform cognitive tasks is quite variable. A child will not only perform different tasks in different ways, he may do the same task in two different ways on successive days! [...]

The problem is not simply that Piaget didn’t get it quite right. The problem is that cognitive development does not seem amenable to a simple descriptive set of principles that teachers can use to guide their instruction. Far from proceeding in discrete stages with pervasive effects, cognitive development appears to be quite variable—depending on the child, the task, even the day (since children may solve a problem correctly one day and incorrectly the next). [...]

These experiments tell us that there is not a rapid shift whereby children acquire the ability to understand that other people have their own perspectives on the world. The age at which children show comprehension of this concept depends on the details of what they are asked to understand and how they are asked to show that they understand it. This pattern of task dependence holds for other hallmarks of Piagetian stages as well. The implication is that stages, if they exist, are not pervasive (i.e., they do not broadly affect children’s cognition). The particulars of the task matter. [...]

Until about 40 years ago, most thought of children’s minds as a set of machinery. As children developed, parts of the machine changed, or parts were discarded and replaced by new parts. The machinery didn’t work well during these transitions, but the changes happened quickly. Today, researchers more often think that there are several sets of machinery. Children have multiple cognitive processes and modes of thought that coexist, and any one might be recruited to solve a problem. Those sets of cognitive machinery undergo change as children develop, but in addition, the probability of using one set of machinery or the other also changes as children develop.

This conclusion doesn’t mean that there is no consistency across children in their thought, or in the way that it changes with development. But the consistency is only really evident at a broader scale of measurement. A geographic metaphor is helpful in understanding this distinction (Siegler, DeLoache, and Eisenberg 2003). If one begins a trip in Virginia and drives west, there are very real differences in terrain that can be usefully described. The East Coast is wet, green, and moderately hilly. The Midwest is less wet and flatter. The mountain states are mountainous and green, and the West is mostly flat and desert-like. There is no abrupt transition from one region to another and the characterization is only a rough one—if I tell you that I’m on the East Coast and you say, “Oh, it must be green, wet, and hilly where you are,” you may well be wrong. But the rough characterization is not meaningless. Similarly, all children take the same developmental “trip.” They may travel at different paces and take different paths. But at a broad level of description, there is similarity in the trip that each takes.

Obviously, the description of multiple sets of cognitive machinery rather than a single set complicates the job of the developmental psychologist who seeks to describe how children’s minds work and how they change as children grow. Worse, it negates the possibility that teachers can use developmental psychology in the way we first envisioned. There is a developmental sequence (if not stages) from birth through adolescence, but pinpointing where a particular child is in that sequence and tuning your instruction to that child’s cognitive capabilities is not realistic.

Piagets formal operational stage overly simplifies things. It doesn't go the same way for all people. The basic capability for formal operations sets in much earlier. But using it or recognizing the applicability of specific instances is something else. Some people never get algebra, but that doesn't mean they can't do formal operations. I think what is missing is the intuition behind the formal operations. Just doing the formal operations without intuitively understanding why kills motivation. That is the reason DragonBox works so well. You need to train both. I once draw an ascii art about this: http://c2.com/cgi/wiki?FuzzyAndSymbolicLearning

Maybe.

When something very similar happened to me (failing Algebra in 9th grade, aptitude suddenly surfacing in 11th), I also thought motivation was really important, but I also noticed my brain working differently. Algebra went from being semi-confused symbol manipulation to understanding what a variable was actually about.

In a simultaneous psychology course, I learned about Piaget's "formal operational stage" and that's what I attributed it to. I think it happens when you're 17 or 18. (Consider/compare with also this data point). So I agreed, it felt like it was a physical difference in development. What do you think of this as an explanatory hypothesis? (Any way to tell them apart?)

Can you think of an example (a fictional one might be easiest) where a deception (or even any conflict) was actually about someone overestimating someone's intelligence?

Well, there are entire tropes about this.

I'm not familiar with any charlatans or scammers being successful by pretending to be smarter than they were. People pretending to be smarter than they are, are usually pretty transparent.

They are if you're smarter then they really are.

I suspect this is just availability bias, though, do you have any examples in mind?

Well, there's Yvain's tale of how he was almost convinced by Velikovsky's pseudohistory.

Smartness is impossible to fake

The right jargon, and sounding like you know stuff (otherwise called being assertive), goes a long way.

I often categorize people as 10-points-smarter-than-me, 20-points-smarter-than-me, etc, just naturally as I go about my day, and I'm (currently) fairly confident of my evaluations.

So you don't have any independent way to verify your evaluations. Let's apply the outside view here: Would you trust the assessment of someone of average or below average intelligence about the relative intelligence of people smarter than him. Note that there exist entire cottage industries of quacks, charlatans, and scammers based on convincing people that someone is smarter than they really are.

It seems to me that you can find out a lot about people's intelligence by talking with them a little

This only works with people less intelligent than yourself. Someone with intelligence comparable to yours who's good at smooth talking can easily convince you that he's much smarter than you.