In general I've been very disappointed about the quality and scientific backing for evidence about how to get better at writing. I tried taking it pretty seriously over the last 1.5 years, using both my own research methods and asking AIs to look for stuff. Mostly I got nothing. Like the advice that pops up over and over against sounds plausible to me: write a lot and focus on quantity over quality, read a lot (especially in your target medium), etc. But this is mostly based on lore and repeated wisdom and vibes rather than empirical study (even non-experimental studies) or careful modeling.

Curated.

I used the Clay Pots story a lot over the past few years and it was very frustrating to find out it was about photography. (I interpreted the Clay Pots version to be more about gaining skill via repeated drills, and the photography version, insofar as it's true at all, seems more about "shots on goal", since it's normal to take tons of photos and only some are good).

Praise be to the people who dig into good-sounding-anecdotes and find out the truth.

I do, uh, still basically believe it, but primarily for "the rest of my personal life experience" reasons rather than anything specific to the anecdote.

I had never heard about the pottery version of this, I knew only of the photography version of it.

There's a model for why it makes sense to take many shots that I like which is asking to what extent having a shot at goal will teach you about how to improve. It is about the extent to which it is skill vs luck where skill is defined as the mean|action counterfactual influence is high whilst in luck domains it is not.

How do you know whether it is a skill based domain or not? Well you have to at least start trying first! In order to lower your variance you need some degree of dataset first so the strategy is most likely to take more shots in the beginning and over time as you start recognizing the skills you might want to start polishing subareas of it.

I'm gonna copy paste my comment from this shortform as it essentially talks a bit about how you can model this:

There's a nice book called the success equation that makes a point similar to the following paper about skill vs luck based domains where a skill domain is something where you're able to get repeat successes without much variance and a luck domain where things are more random. There's a sort of implicit model in the photography thing where you have higher variance.

The advice is to take the most amount of shots in the luck based domain as that is all that matters there but what about the skill based domain?

Well, there you want to (approximately) maximise your learning so we might say that the actions we should pick should be the ones that are dependent on us learning something. A naive approach is something like max(n*p) where p is the probability of learning and n is the amount of opportunities you have to learn. Sometimes you don't want to do this because the skill lies in highly specific skills which means you want to spend a longer time on one attempt so that you get this really right. This then essentially boils down to a learning problem where priors are inherited through previous people often (e.g different sports have different training regimes).

But what do you do if you don't know the extent to which something is luck vs skill?

I think the answer basically becomes to just do the photography thing again and see what the variance becomes over time, treat your unknown domain is as if it were a skill domain and see if it lowers the variance of it. If it does it probably is, if it isn't then you've already done the strategy of taking many shots which means you've done the right thing.

I have tried this out in practice over the last year or so and I've concluded that mentorship is probably a more skill based domain than I originally thought and that writing is more luck based than I thought (I might just not have enough data on this one though.)

Clarification after chatting with claude: It's the conditional probability of the mean moving that matters p(mean|action). Secondly it is about feedback signal clarity for learning, can you move fast and get a clean signal? (e.g blitz chess) Or does moving fast make it so that you can't learn (e.g trying to learn to code by using claude to vibe-code apps that you don't even look at the code of).

There's also some annoying stuff around how getting high variance in the results doesn't mean the domain isn't skill based since it is only the conditional on the mean moving that matters. So the writing thing might actually not be true in terms of being a non-skilled domain, it is more that it is inherently high variance in the feedback domain.

I believe the original anecdotes confound opportunities for success versus increasing opportunities to succeed.

For example, if one were to run the photography experiment, 20 photographs versus 1000 photographs, its clear the 1000 photograph set has more opportunity for something remarkable: a chance meeting of subject, angle, framing, and light.

What would be useful would be to lay the photographs out chronologically: is the best photograph closer to the first or the last? And if we asked 10 judges to lay out the photographs in order of quality, and compared their ratings, then compared the aggregate to the order of the photographs, is there correlation?

My presumption is that all teaching stories are fiction.

They may express wise truths or they may not, but the story frame is pretty wrapping, not evidence. Even if the story as told is literally true, it’s still just a data point. For the pots (or photographs) story, or for that matter the entire works of Oliver Sacks, Malcolm Gladwell, and the like, and the entire “self-help” genre, I look for whatever may be useful in them for myself and ignore the larger claims.

For writers who say that everything in their stories is true, they’re just merging and simplifying real case studies, well, that’s a common way for writers of avowed fiction to invent their characters. You don’t have to realise you’re writing fiction, to be writing fiction.

These well-known but kinda fake studies are like the modern Western equivalent of ancient Chinese proverbs.

Thanks for this!

One thought: "It is fairly easy and cheap to take a photo"--today, yes! But this report comes from decades ago, when there were more material and financial bottlenecks.

If the original report on the study was made in 1993, and the story came from Ueslmann, who'd been teaching photography at the university level since 1960, we don't know in which decade the study was done. Students in a beginning photography course in the early 90s would have been shooting 360-540 pictures (10 to 15 36-shot rolls maybe cut from a 100-foot bulk roll) over the course of a semester. They would have had to develop the negatives, produce a contact sheet for every roll, and then select which photos to print on paper, which is also a skill that rewards a lot of practice. This is all tons of time and effort in the darkroom. If some students were shooting more than the typical amount of pictures, and the other students were somehow discouraged from using a lot of film, we could only expect huge differences in skill--as well as the occasional lucky shot--over the course of a semester.

I find it interesting that the authors didn't seem to realise - and even the person on Hacker News hardly seemed to realise - how much changing it from photography to pottery changes the interpretation. Seems like they just thought they would slot in a different sort of art so it didn't say photography all the time. As you said yourself, pottery makes you think of the students improving with each piece, whereas photography calls to mind something more like a random chance of a winning lottery ticket.

This article inspire me some thought:

Neither quality nor quantity is sufficient to completely overwhelm the opposition. But it would be in some specific field and needs. i.e : When it comes to famine, quantity is more important than quality. However, if "quality" is defined by how well something meets basic human survival needs, then in this context, quality can be considered equivalent to quantity. Yet, when the difficulty of obtaining high-quality items exceeds that of obtaining a larger quantity of items, then quantity once again becomes more significant. The experiment in question lacks persuasive power—most likely for the very reason the author points out: "What does 'all works of highest quality' even mean? If you sort all the produced works by quality, where does 'highest quality' stop? You have to draw the threshold somewhere." The definition is simply too vague.

I believe the process of getting the ”high-quality pottery” is partially flawed in this experiment. If people are going to be overexerted on making only one pot, the quantity group does the job of quality group, but only better. What I could suggest is that the quality group is asked to make a certain amount of pots, but with the liberty to analyse and break down the pots. For instance, the quality group should be able to do a sound or surface test. The quantity group, on the other hand, could make more pots but without the ability to look through them. Could this be a more sensible experiment?

Even in case of photography doing 1000 photos will make you better photographer - you will know (or feel - have a mental model) on how to make a good photo, how to time it, what additional tools to use, what settings (assuming professional hardware, not a simplest smartphone app), what to look for. Thinking and waiting for best opportunity with a camera in hand will not.

This is not really quality vs quantity anyway. This is a novice that experiments and gets better model of things vs a novice that uses bad uncertain model and makes refinements in it without basing then in the reality check.

I think it would depend on how good you are at mental modeling. If you are that rare person who is able to plan out the perfect pot in your mind then execute it according to your mental model, then you will have better results over simply making a bunch of pots that display your average skill. But that isn't most people. And some people don't even have the mental modeling capacity to reliably learn from their mistakes. (I have a friend who is like that--she almost never internalizes feedback from her own experiences, but instead always relies on an external authority to tell her how to do something. I find that fascinating.)

Agreed that it was a little disappointing that the tale was not told verbatim. I can assume some confirmation bias in my own take on it, due to my own experiences (my own clay pots "tale" https://www.unresolvedforces.com/ufbook/02_The_Link_to_Agile/03_Moving_away_from_documentation_and_requirements_gathering.html) but also in others. 'How Big Things Get Done' covers it without naming it, iirc, with the idea of smaller, voluminous projects being more likely to be successful, simply because practice makes adept. And I think that was the point of the tale for me. It wasn't that the numerous pots (or photos) produced a better best pot, but produced a better potter.

Also, I would love to counter the claim that more photos is more like a lottery ticket, however I have my own counter-counterexample in a really nice photo I took as a child when I had no idea what I was doing when taking it. The crucial question is how the quality distribution lies. Are the numerous photos, code snippets, pots, or watercolour paintings of random quality, or do they become better over the sequence of production? And, if they do improve (again, I have a strong hunch they do), where on that sequence to they eclipse the quality bar of the "one good one" group?

Yes, we should run a better controlled experiment, if ethically possible, because it could reveal a better way to teach and learn (especially if it counters the original claim as many people have built training around the concept).

I suspect the object of this parable gets replaced frequently during oral transmission. I think I’d heard it about birdhouses

What does "all works of highest quality" even mean? If you sort all the produced works by quality, where does "highest quality" stop? You have to draw the threshold somewhere.

I think there is a set of highest quality works. We don't know the number but it could be 2, 3, 5, 10 or 20. It's a finite number of works. I don't think it would be all of them cuz there are lowest quality of works too. There are works that didn't make it to the highest quality of works.

Also, I think since the study is talking about a ceramics teacher present, then they gotta know somewhat about the quality. Like, they have some expertise on pottery that they can tell good pottery from bad pottery. Maybe, the bad kind are the ones that aren't even whole/functional. Like, they have cracks in them.

Who knows about the exact details tho. Maybe the ceramics teacher is for a kindergarten/elementary school. He/she might not know a thing about pottery. I think tho that that doesn't matter. We as the audience can assume that the teacher has competence to be able to tell good works from bad works. It makes sense cuz there's hobbies/jobs everywhere where someone can do a good job or a bad one.

I don't think an exact threshold is very important to get the lesson of the study. The fact is that the people who made better pottery are the ones who made a bunch. They had more chances to make mistakes and learn from them. They improved more compared to the people who tried making a perfect piece of work.