As previously mentioned, research feedback I give to more junior research collaborators tends to fall into one of three categories:
Doing quick sanity checks
Saying precisely what you want to say
Asking why one more time
In each case, I think the advice can be taken to an extreme I no longer endorse. Accordingly, I’ve tried to spell out the degree to which you should implement the advice, as well as what “taking it too far” might look like.
The second most common feedback is that you should write down precisely what you want to express.
One of the most common interactions I have with junior researchers goes as follows: I read a draft section of their research writeup. This often consists of many paragraphs detailing various seemingly disconnected ideas, as well as 5-10 different figures. I’m confused about what the point of the section is. I ask them what exactly they’re trying to say in the section. They give me a very cogent and concise statement of what the takeaways should be. I suggest that they should write that down, and include it prominently somewhere in that section.
Sometimes, the problem is that people don’t know what point they’re trying to make. But maybe around half the time, the people I talk to already know what they want to say with a section of text – they just haven’t said the point anywhere in the section!
The advice here is simple: figure out what point you’re trying to make, and feature it prominently. As an example, if your blog post’s main claim is that AI models can be dangerous before public deployment, then consider titling it something like “AI models can be dangerous before public deployment”, or, at the very least, saying that clearly and in as many words in the introduction.
The converse of this is that you should not include text that distracts from what you want to say (or at least, relegate it to footnotes, the appendices, or parentheticals). It’s always tempting to add more to a piece, and almost always hard to cut (hence the common quip about not having time to write a shorter piece).
Both the advice and its converse apply to figures as well: In terms of saying what you want to say, the easy fix is often to highlight relevant parts of your figure in your caption. Better yet is to highlight your claim in the figure itself: for example, if your claim is that longer chains of thought are more monitorable, then consider including a regression plot with chain-of-thought-length on the x axis and monitorability on the y axis.
In terms of not including unnecessary content, the first step is to avoid what Edward Tufte calls “chartjunk”: visual elements in charts and graphs that are not necessary to comprehend the information represented on the graph, or that distract the viewer from this information. The second, more difficult step, is to cut back on unnecessary information. For example, you might average the performance of an AI model across tasks, instead of including one data point per task.
Left: An example of a figure with substantial amounts of chartjunk, including unnecessary shadows, redundant text, and an unnecessary legend. Right: the same figure without said chartjunk. This example was originally Darkhorse analytics, though you can find an open-source recreation at this Github repo.
Information denseness is okay, as long as the key claims can still be read off from the figure (or if the information is included in a separate figure).
Taking this too far. Unfortunately, what is necessary and what is extraneous junk is often a matter of judgment, and it’s easy to take this advice too far.
In terms of prose, your key claim can often require substantial amounts of additional context to understand: if you find that people can not understand your main claim, or that there are obvious follow up questions that you haven’t answered, consider adding the additional context back. Even pure redundancy can serve a purpose: audience members are often skimming your work or reading it in chunks, and so any point you really want them to remember should be integrated into different parts of the piece.
For figures, the extreme version of “removing unnecessary content” is to end up with a figure that has no argument and is basically an assertion. Classic examples of this are the content-free figures often found in business PowerPoints – while these figures often have some hallmarks of a data-backed figure, they are not produced based on real data and do not contain enough content to justify the real estate they take. If you find that the entirety of the content of your figure can be summarized in a single sentence, you’ve probably cut too much content (or are better off just saying the sentence in prose, perhaps over a different figure!).
Masayoshi Son attempts to use a hypothetical earnings trajectory (not based on real data, nor even grounded with a y-axis) for WeWork to justify Softbank’s investment in the company, which at the time had lost SoftBank billions of dollars. Source.
As an aside, I think that using color as the third dimension generally produces more readable graphics, i.e., heatmaps are often superior to 3d charts.
Written as part of the Inkhaven Residency program.
As previously mentioned, research feedback I give to more junior research collaborators tends to fall into one of three categories:
In each case, I think the advice can be taken to an extreme I no longer endorse. Accordingly, I’ve tried to spell out the degree to which you should implement the advice, as well as what “taking it too far” might look like.
I talked about doing quick sanity checks in a previous piece. Here, I talk about the second piece of advice: saying precisely what you want to say.
Saying precisely what you want to say
The second most common feedback is that you should write down precisely what you want to express.
One of the most common interactions I have with junior researchers goes as follows: I read a draft section of their research writeup. This often consists of many paragraphs detailing various seemingly disconnected ideas, as well as 5-10 different figures. I’m confused about what the point of the section is. I ask them what exactly they’re trying to say in the section. They give me a very cogent and concise statement of what the takeaways should be. I suggest that they should write that down, and include it prominently somewhere in that section.
Sometimes, the problem is that people don’t know what point they’re trying to make. But maybe around half the time, the people I talk to already know what they want to say with a section of text – they just haven’t said the point anywhere in the section!
The advice here is simple: figure out what point you’re trying to make, and feature it prominently. As an example, if your blog post’s main claim is that AI models can be dangerous before public deployment, then consider titling it something like “AI models can be dangerous before public deployment”, or, at the very least, saying that clearly and in as many words in the introduction.
The converse of this is that you should not include text that distracts from what you want to say (or at least, relegate it to footnotes, the appendices, or parentheticals). It’s always tempting to add more to a piece, and almost always hard to cut (hence the common quip about not having time to write a shorter piece).
Both the advice and its converse apply to figures as well: In terms of saying what you want to say, the easy fix is often to highlight relevant parts of your figure in your caption. Better yet is to highlight your claim in the figure itself: for example, if your claim is that longer chains of thought are more monitorable, then consider including a regression plot with chain-of-thought-length on the x axis and monitorability on the y axis.
If your claim is that vaccines are effective at reducing the incidence of measles then consider making a heatmap of measles cases over time, with the date the vaccine was introduced clearly delineated. This figure is from the WSJ article “Battling Infectious Diseases in the 20th Century: The Impact of Vaccines”, though I first saw this example in Edward Tufte’s book Seeing with Fresh Eyes: Meaning, Space, Data, Truth.[1]
In terms of not including unnecessary content, the first step is to avoid what Edward Tufte calls “chartjunk”: visual elements in charts and graphs that are not necessary to comprehend the information represented on the graph, or that distract the viewer from this information. The second, more difficult step, is to cut back on unnecessary information. For example, you might average the performance of an AI model across tasks, instead of including one data point per task.
Left: An example of a figure with substantial amounts of chartjunk, including unnecessary shadows, redundant text, and an unnecessary legend. Right: the same figure without said chartjunk. This example was originally Darkhorse analytics, though you can find an open-source recreation at this Github repo.
Information denseness is okay, as long as the key claims can still be read off from the figure (or if the information is included in a separate figure).
Taking this too far. Unfortunately, what is necessary and what is extraneous junk is often a matter of judgment, and it’s easy to take this advice too far.
In terms of prose, your key claim can often require substantial amounts of additional context to understand: if you find that people can not understand your main claim, or that there are obvious follow up questions that you haven’t answered, consider adding the additional context back. Even pure redundancy can serve a purpose: audience members are often skimming your work or reading it in chunks, and so any point you really want them to remember should be integrated into different parts of the piece.
For figures, the extreme version of “removing unnecessary content” is to end up with a figure that has no argument and is basically an assertion. Classic examples of this are the content-free figures often found in business PowerPoints – while these figures often have some hallmarks of a data-backed figure, they are not produced based on real data and do not contain enough content to justify the real estate they take. If you find that the entirety of the content of your figure can be summarized in a single sentence, you’ve probably cut too much content (or are better off just saying the sentence in prose, perhaps over a different figure!).
Masayoshi Son attempts to use a hypothetical earnings trajectory (not based on real data, nor even grounded with a y-axis) for WeWork to justify Softbank’s investment in the company, which at the time had lost SoftBank billions of dollars. Source.
As an aside, I think that using color as the third dimension generally produces more readable graphics, i.e., heatmaps are often superior to 3d charts.