Note: please write any answers to this prompt in spoiler-tags.
Recently I set out to deliberate practice at "reasoning about confusing intellectual problems."
Eliezer's Class Project has a fictional group of rationality students try to find the true theory of quantum gravity in one month. This always seemed like a cool goal and test for rationality training to aspire to. If you're not solving difficult open problems faster than science, your Art of Rationality probably isn't complete.
Of course, our Art of Rationality isn't complete yet. But, I think there is something promising in this area, as a way to ground out "rationality training" in something concrete. It seems like good practice to take a given physics question you don't understand the theory behind, and try to invent the theory yourself.
I don't think we're anywhere close to the "definitively impressive" version of rationality practice/training. But, I think a good next step is "Solve Thinking Physics™"
Thinking Physics is a textbook that teaches physics "question-first." Each page presents a physics-y situation, and asks you to figure out what happens next. The questions are multiple choice, but often fairly tricky nonetheless.
I think a good rationalist-training goal is aim for a goal of "be (correctly) 95% confident in the answer", as a rough proxy for "there were no major lingering confusions about the problem except for generic 'maybe I missed something?'". And, failing that, have the subgoal of at least being calibrated about how confused you. Every time you look at an answer, first log your probabilities for each of the multiple-choices in Fatebook.io (or prediction-tracking tool of your choice).
The problems are set up in a way that you can probably reason about them from some basic background knowledge, without much math background. They're ideal for people who don't have much physics background (since the whole point of the book is to teach you physics), although I know people with some physics education who still find it fairly hard.
I spent two weeks working on Thinking Physics problems, and hosting meetups/workshops where other people could join me. With each question, I focused on learning as much as I could about how-to-think.
My original hypothesis was that I could get significantly better at it in 6-8 weeks. I only spent two, and the result so far is I think I'm significantly better although didn't yet hit my goal of 95% accuracy. (In my final test-set, I got 1 out of 5 questions wrong, when I was aiming for zero. I do think I have a pretty clear sense of why I got that 1 question wrong, and what I should have done differently)
After workshopping some ideas for "the Thinking Physics rationality challenge", I now present you with three tiers of challenge.
Challenge I: Solve three problems (and learn from them)
Step 1: Do an exercise.
Spend some time trying to solve three Thinking Physics question. Aim for 95% accuracy, fully deconfusing yourself about each exercise.
Write down your probabilities for each answer.
It's important to actually write down the probability for each answer – otherwise, you may get a vague sense of "yeah that's probably right", that doesn't allow me to cleanly say "I got this one wrong." And doing it for all the answers, not just your favorite one, gives you additional bits about whether your models made any sense. (i.e. having clearly stated "I think answer A is most likely and B is second most likely" gives you a harder update if it turns out that A and B were both wrong)
Step 2: Learn from it
Then, think about how you could have solved the problem better.
Your primary goal is to learn as much as possible from each question.
Babble as many new insights as you can about how to think. This can include explicit "strategies" (like "see if you can simplify the problem"), physiological things (like "I got tired and needed to take a break"), or psychological things ("something about this feels weirdly aversive and ughy, what's up with that?").
When you're done, submit your answer on this post for "what you learned." (Focus on your takeaways, not the object-level solution).
Overall structure
This is more fun with a partner, although I recommend spending a chunk of time thinking independently before sharing your answers and thought-processes with each other. You might find it helpful to get some friends together as a weekend activity.
I've found a fairly good default approach is to do:
- 20 minutes thinking about it by yourself
- 20 minutes thinking about it with a friend
- 20 minutes discussing your meta-reflections on how to solve the problem with a friend.
How to pick exercises
The exercises vary in difficulty. My recommendation is to flip to a random page, weighted towards the beginning of the book. If it feels "difficult but not impossible", then give it a try.
If you're pretty confident you just know the answer, still try to come up with a clear explanation for why (but err on the side of checking the answer quickly rather than spending a lot of time doublechecking).
If you end up feeling stuck, try to give it at least 10 minutes before giving up and switching to a different problem. (In most cases, I found it valuable to give it a solid 20 minutes of independent thought + 20 minutes of conversation-with-partner even if I felt really stuck).
Some particular exercises that seemed reasonably good for people I beta-tested this with (which is not to say they were easy or hard, but that I feel like I/others learned from making a good faith effort on:
- Steam Locomotive
- Cold Bath
- Rare Air
- The Expansion of Nothing
- Landscape
(Page numbers for the exercises vary between editions of the book, but you can look them up in the table of contents)
Submission guidelines
Put your answers in spoiler tags (begin each line with ">!"), although first list (unspoiler-tagged) that it was a Tier 1 challenge, the name of the exercises you did, and whether you give them each an overall thumbs up or thumbs-down as having been a good exercise.
Challenge II: Design a training regimen
After you've done 3 exercises and gotten a rough sense of their shape, develop a training regime that helps you significantly improve at Thinking Physics exercises.
If you started out not being able to reliably solve them at all, get to the point where you can at least semi-reliably solve them, given enough time. (Suggested target: solve 5 random questions in a row without getting any wrong, without help)
If you started out able to semi-reliably get the right answers given a lot of time, aim for speed – can you solve 10 problems in a row, relatively quickly, and only get between 0-1 question wrong?
Submission guidelines
You can submit your training regime before actually completing it (but flag whether you have actually employed it yet, and if you end up actually doing the training regimen, I suggest replying later with any updates you made).
I think it's a fine use of this exercise to submit your training regime, then read other people's suggested regimens to get more ideas before going off to actually do it.
Put your training description in spoiler-tags (although again list which challenge-tier you're doing in non-spoiler tags)
(Once you actually get started with the training, I recommend adjusting your approach as you learn more)
Challenge III: Fully "Solve" Thinking Physics
After you've significantly improved your skill-level, develop a thorough for solving Thinking Physics exercises, in generality. Write the instructions that would have helped past-you get to the point where you could solve them reliably and/or quickly.
(It's okay for this to include metagaming / psychologizing the author. This is "Solve 'Thinking Physics'", not "Solve 'Physics'")
Write your answer either as a spoiler-tagged comment here, or as a top-level post if it ends up feeling like a full essay (and then a quick comment here linking to it). Include a note about what concrete outcomes you achieved.
Bonus Challenge:
Find different sets of exercises that are as different as possible from Thinking Physics (i.e. requiring a pretty different set of skills, while still being feeling relevant to becoming a "generalist researcher"), that would make for a good followup to this exercise.
OK, a shot at Challenge I, with Poof and Foop, Steam Locomotive, and Expansion of Nothing. Felt like all three are in the sweet spot. I personally dislike Expansion of Nothing.
Poof and Foop:
The problem statement is a bit leading: there's some kind of inversion symmetry relationship between the two cases, so it should go the opposite direction, right?
Initially, definitely. The puncture means that there's less pressure on the right side—instead of colliding with the can, some particles go inside.
But those particles end up colliding with the interior left side anyway. So it seems like it should even out, and at the end the can won't be moving.
So my guess is (c). Can I make myself more confident?
Why doesn't an inversion argument go through? Well, the compressed air can is drawn in vacuum, but the vacuum can doesn't empty the environment.
So it's not simply time reversal. If the compressed air can were in air, then we might have some kind of symmetry between air particle and absence of air particle,
but then the compressed air can would slow down due to drag and stop in the limit. So that still points to (c). That also works as a thermodynamic argument—the first can isn't equilibrating with anything, so none of the work goes to heat. 95% confidence feels good.
*checks* OK, looks like I was thinking about it right, and my explanation for why the naive inversion is wrong is equivalent to sketch II.
Reflection: The main interesting thing here is the fake symmetry argument. My favorite problems have tempting solutions that don't work for subtle reasons. I think it's important not to count problems solved until you can pinpoint why those solutions fail.
What did I use here? If you're dealing with pressure, you can probably get an answer with forces or with thermodynamics. A net force can be thought of as a single force or as lack of a balancing force. That's the symmetry idea.
I'm not very good at babbling. I'm basically looking over what I wrote and renarrating it. Words to words.
Steam Locomotive:
We might want to think about torque and the height of the axle.
Or maybe it's about wheel radius. One cycle takes you further with bigger wheels.
I think these both point to (b).
I'm a little confused because thinking about the wheel heights of sports cars and trucks would push me towards (a). But cars have gears. Directly driving small wheels is basically low gear.
Not sure how I'd know if the answer were (c) or (d). Seems like you'd need background knowledge not in the question.
I should think about actual forces to get to 95% confidence.
Let's say the engine puts out the same force in both cases. Then, in II, each wheel sees half as much force from the engine,
but the ground exerts force on twice as many wheels, so that part's a wash. But because the wheels are smaller, the ground
needs to exert more force per unit engine force to keep the wheel from slipping (same torque).
So for the same engine, II seems to give more accelerating force, while I gives higher top speed. I'd put 95% on (b).
*checks* OK, seems like I had the right thought. Could I have been as confident from the distance-per-cycle argument alone? Rather than look at forces,
the author's answer argues that we know the locomotive that goes a shorter distance in the same number of engine cycles must
be putting more energy into each mile it travels. I considered that, but I wasn't sure it was a valid step.
Why couldn't you just be getting less work from the engine? Well, it's the same piston with the same motion.
My force calculation already needs that assumption, it just makes the final connection with the acceleration.
Reflection: I feel like I don't know much about automotives. (Is a locomotive an automotive, by the way? I think so, it's just locomotives involve a track.) I can describe transmission and gears and engines and so on if I think about it, but I don't have much intuition. Like, I can't explain why it's one way and not another, or how different cars solve different problems.
I just feel like I should have been able to answer the question immediately. If I could drive stick, would that help? Probably not. I already ride a bike and didn't immediately see the analogy.
What did I use? Qualitative personal experience. I picked a misleading experience but reasonably didn't weight it above thinking through the problem. Identifying relevant considerations. Didn't stop at the first idea.
Expansion of Nothing:
Oh, this one's nasty. It has to expand, right?
If you took an iron disk and drew a circle where the hole is, the circle would expand.
If you cut that disk out and heat up the cutout, the disk expands the same amount.
So everything outside the circle can't be exerting any net force at the boundary, and the hole has to stay the same size as the disk.
I don't see any problems with this argument, but can I explain why other arguments don't work? Why can't thermal expansion generate stress instead of allowing uniform expansion? I guess in a sense I just gave the reason, but why does the gap shrink if you cut a gap in a rod instead? Well, when you have only one piece, it's like applying a magnification transformation, which requires an origin. But the origin is arbitrary—you can just recenter. With two separate pieces, the two origins of magnification are no longer arbitrary.
*checks* Yeah, the author's answer doesn't go there, unfortunately.
Reflection: This problem feels really annoying to me. Maybe I saw it a long time ago and got it wrong? Or maybe it's that you never have anything that's free to expand uniformly. It's braced against something, or it's sitting on something with a different coefficient of thermal expansion, and you do get stress and it does matter how the thing is patterned.
This feels like a problem where you're supposed to think about limiting cases. Like, if you have an atomic ring, obviously it expands. I don't know if you can justify jumping to the right answer from that, though. If the disk is thick and the cutout doesn't go all the way through, it expands. Ehh. You still need an argument that it expands the same.
I enjoyed it, although I'm already the sort of person who thinks Thinking Physics is fun—both the problem solving and the nitpicking about what constitutes a correct explanation. It seems worth doing at least a handful of problems this way, and more broadly deliberately practicing problem solving and metacognition about problem solving. Thinking Physics could be a good complement to Problem Solving Through Problems or How To Solve It, since in my (limited) experience you get quickly diminishing returns to anything but competition math with collections like that.