I think the terminology you are looking for is called "Deliberate Practice" (just two random links I just found). Many books/podcasts/articles have been written about that topic. The big difference is when you "just do your research" you are executing your skills and trying to achieve the main goal (e.g. answering a research question). Yes, you sometimes need to read textbooks or learn new skills to achieve that, but this learning is usually subordinate to your end goal. Also one could make the argument that if you actually need to invest a few hours into learning, you will probably switch to "deliberate practice mode".
Deliberate practice is the very intentional action of improving your skill, e.g. sitting down on a piano and improving your technique, learning a new piece. Or improving as a writer by doing intentional exercises, or solving specific math problems that improve a certain skill.
The advantage of deliberate practice is that its main goal is to improve your skill. Also usually you are at the edge of your ability, pushing through difficulties, making the whole endeavor very intense and hard.
So yes, I agree that doing research is important. Especially if you have no experience then getting better at research is usually best done by doing research. However, you still need to do other things that specifically improve subskills. Here are a few examples:
I think by adding terminology I just wanted to make explicit what you mention in your post. It will also make it easier to find resources given the word "deliberate practice".
Thanks for your thoughts here!
So I do have some familiarity with the concept of deliberate practice, and I would definitely include that as part of the thing I'm talking about above. But I can also think of things that might improve a researcher's capacity that don't fall under deliberate practice.
1. One researcher told me they were having frequent interruptions to their focused working time as a result of their work environment, so they made some adjustments to their work environment to prevent that. I don't think I'd call that deliberate practice, but it...
If there are any alignment researchers reading this who think they would benefit from having someone to talk to about improving their research capacity, I’m happy to be that person.
I’m offering free debugging-style conversations about improving research capacity to any alignment researchers who want them. Here’s my calendly link if you’d like to grab time on my calendar: https://calendly.com/dcjones15/60min .
I’m not claiming to have any answers or ready made solutions. I primarily add value by asking questions to elicit your own thoughts and help you come up with your own improvement plans that address your specific needs. A number of researchers have told me these conversations are productive for them, so the same may be true for you.
(Epistemic Status: Anecdotal)
If we want to reduce AGI x-risk, it seems pretty intuitive to me that alignment researchers should be regularly dedicating time to improving their research capacity. But I'm suspicious that many of them don't do this.
Am I wrong? I would love to know if I am.
(I'm using the phrase "research capacity" vaguely, to mean both research skill and productivity.)
Over the last 3 - 4 years, I've had something like 12 - 20 informal conversations with researchers at various alignment orgs on the subject of how they improve their research capacity. In these conversations, I'd ask questions like "How do you personally go about getting better at research?", or "What have you done recently to improve your research process?" or "What is one thing you could do to get better at your job?". And more than half of the responses are one of the following:
1. a blank stare
2. a long, thoughtful pause followed by no actual answer
3. an argument that "this kind of messy, abstract work doesn't lend itself to direct improvement in the way other skills do"
4. an argument that "the best way to improve at research is to simply do the work and keep your eyes peeled for opportunities to improve as you go."
To be clear, some researchers do have different responses. But these answers were surprisingly common, and... they seem wrong?
My Response To #3 :
I'm not a researcher, but I can't think of any skill I've learned, whether intellectual or physical, that doesn't benefit from some amount of regular and intentional focus on improving it.
Additionally, over the last couple months, I've started having debugging conversations with researchers who want help thinking through how to improve, and in these conversations most of them generate lots of ideas and claim the discussion is quite productive. That's not what I would expect if doing explicit skill improvement just didn't work for alignment research.
Here are a few examples of opportunities for improvement that researchers have identified in talking with me:
My Response To #4:
Once again, I'm not a researcher, but as far as I can tell the above quote has never been true for me with respect to any skill I've developed in the past.
To be fair, I do think it's important to keep your skill improvement approach grounded in real problems you're solving. It's bad to lose track of the object-level and get lost in endless meta-level considerations. But I would be really surprised if just doing the work and dedicating zero time to improving your capacity directly was actually "the best way".
In my experience, most skills contain lots of parts, many of which will seem small and insignificant if I'm looking at them from the perspective of a single problem right in front of my face. It's only when I intentionally zoom out and consider how these parts affect my performance across many problems that I notice how valuable it would be to improve them.
For any researcher who adheres to the "just do the work" approach, I'm curious if you've run an experiment like this with yourself:
- Spend a week "just doing the work and keeping your eyes open", and see how many improvement ideas you generate and how valuable they seem.
- Then spend 20 minutes specifically trying to think of ways to improve, and see how many you generate and how valuable they seem.
- Compare Results.
If you don't generate more valuable ideas in the 20 minutes than during the week, that would be interesting to hear.
Two More Objections I've Heard
1.
No, I don't think so.
I view those activities as part of the work of research, not as something extra on top of the work. In fact, those are great examples of activities at which I'd think a researcher would want to improve.
2.
Yep, that sounds right to me. So it's a good thing I didn't claim otherwise.
Seeking improvements doesn't have to mean letting go of curiosity as a crucial driving force in your research.
And I'm certainly not implying that I have some specific idea of how research is best done. I'm just making the claim that there should be some way of getting better at it. I assume that will look different for each researcher.
Closing Thoughts
So what's the deal? Have I spoken to an unrepresentative sample of researchers? Is there some miscommunication happening?
Or are we actually lacking a culture of constant improvement among alignment researchers?