Note: This post is based on a Google document I created for my research group. It speaks in the first person, but I think the lessons could be helpful for many research groups, so I decided to share it more broadly. Thanks to Louise Verkin for converting from Google doc to Markdown format.

This document talks about principles for having productive group meetings and seminars, and to some extent a good group culture in general. It’s meant to be a living document--I’ve started it based on my own experiences, but ultimately our seminars and group culture come from all of us together. So if you have ideas you want to add, please do so!

I’ll start by talking about an important concept called psychological safety, then discuss what I see as the goals of our research group and how that fits into presentations and discussions in seminars and meetings. I’ll also provide tips for asking excellent questions and some general philosophy on how to hold yourself to a high standard of understanding.

Psychological Safety

Psychological safety is an important concept for fostering creative and high-functioning teams. I would highly recommend reading the following two documents to learn about it in detail:

• What Do Psychologically Safe Work Teams Look Like?
• Manager Actions for Psychological Safety

To summarize, a psychologically safe team is one where members feel like:

• They can make mistakes without it affecting their status in the group
• It is easy to give and receive feedback, including critical feedback, without feeling attacked or like one is causing trouble
• One is allowed to and encouraged to question prevailing opinions

These are especially important in research environments, because questioning and risk-taking are needed to generate creative ideas, and making mistakes and receiving feedback are necessary for learning.
In general, I would encourage everyone in our group to take risks and make mistakes. I know everyone holds themselves to a high standard and so doesn’t like to make mistakes, but this is the main way to learn. In general, if you never do anything that causes you to look silly, you probably aren’t taking enough risks. And in another direction, if you never annoy anyone you probably aren’t taking enough risks. (Of course, you don’t want to do these all the time, but if it never happens then you can probably safely push your boundaries a bit.)

Fostering psychological safety. As a group, here are some general principles for fostering psychological safety among our teammates:

• Assume your teammates have something to teach you, and try to learn from them.
• In discussions and debates, aim to explain/understand, not to persuade. Adopt a frame of collaborative truth-seeking, rather than trying to “win” an argument.
• Acknowledge and thank people for good points/questions/presentations/etc.
• Invite push-back
• Welcome and encourage newcomers

In addition, there are a couple things to avoid:

• Try not to talk over people. Sometimes this happens due to being very excited and engaged in a conversation, and don’t sweat it if you do this occasionally, but try not to do it habitually, and if you do do it make sure to invite the person you interrupted to finish their point.
• Avoid making broadly negative or dismissive statements. Even if you personally don’t intend such a statement to apply to anyone in the group, it’s inevitable that someone will take it personally. It also works against the principle of “questioning prevailing opinions”, because it implies that there’s an entire area of work or claims that is “off-limits”.
  
  As an example, when I was a PhD student, a senior person often made claims to the effect that “research was pointless unless industry people cared about it”. This made it feel discouraging for me to do my (at the time) more theoretically-oriented work, and I abandoned at least one valuable project because of this. With the benefit of hindsight, I don’t think that person actually would have endorsed the literal claim I wrote above, but that’s exactly the point I’m making–it’s easy for other people to overinterpret claims.

Group Goals and Group Meetings

In my view, our group has three major goals:

• Do excellent research
• Help each other to learn and grow
• Help the world

In the context of group meetings/seminars, we can promote these goals in the following ways:

• Hold yourself to a high standard of understanding (see below for more on this). In other words, don’t just follow the individual steps–try to understand why things had to be this way and not any other way. Asking questions about this not only helps your own understanding, but also pushes the speaker to clarify their own thinking–thus promoting the goals of excellent research and of learning.
• It’s okay and encouraged to tie things back to the bigger picture. Excellent research is not only technically sound but also well-motivated. Understanding the bigger picture is also especially important for helping the world.
• Try to ask questions in a way that succinctly models your own thinking process. One of the most valuable aspects of group meetings is that you can see how other people think, which helps learning. As a concrete example, sometimes in applied talks we ask questions that are very specific and only make sense to people immersed in that area. This is okay, but it’s better to ask the same question in a way that lets people not in that area see why the question is important.
• As a speaker, don’t aim for the standard of “defensibility”. Instead, aim to convince the audience that you are onto something important and exciting (this is a different but not strictly higher standard, since it might involve saying some things that are only partially defensible). Similarly, as an audience member don’t be satisfied just because there’s “nothing wrong”–try to understand why a project was important enough that someone was excited to spend months of their life on it.

In addition, here are some meta-level principles around question-asking:

• Basic understanding questions, even at the level of clarifying notation, are highly valuable and usually under-utilized because they don’t feel “smart”. I encourage everyone to ask these questions when they have them–if you’re confused, probably someone else is too, and it’s valuable feedback for the speaker.
• I try to pay attention to how many other questions are being asked. If no one is asking questions, I’ll try to ask one to break the ice. If lots of questions are being asked, I’ll try to filter my own questions for the ones that are highest-value or most different from what’s already being discussed.
• I also try to pay attention to how many questions I personally have already asked. If I haven’t asked a question yet I feel very free to ask one. If I’ve asked many already, I again try to filter for the highest-value ones.
• As an audience member, you have much more cognitive bandwidth than the speaker. It’s therefore helpful to take the extra time to formulate your question to be easy to understand and engage with. It’s also good to state it succinctly when possible. Time spent formulating a question is time spent only by you, but time spend asking/answering it is spent by everyone in the audience.

Seminar Norms

The culture of a good seminar is different from the culture of everyday conversations, in a way that might not be obvious if you haven’t been immersed in it for a long time. I’ve already gone over that to some extent above, but below I’ll elaborate on some specific points in more detail, and lay out some helpful rules and norms that are usually unstated.

Audience Culture

There are many everyday social norms that hinder us from seeking a high level of understanding in a talk. Asking a question feels like a bid on the speaker’s and audience’s time and attention. We might worry that it’s a “dumb” question, or feel intimidated by a complicated statement that we don’t understand. Or conversely we might worry that it’s impolite or aggressive to ask for such a high (and, if we’re being honest, demanding) level of understanding. We might worry that we’re putting the speaker on the spot and that perhaps they won’t be able to answer and that we’ll make the speaker look “dumb”.

These are all natural and common thoughts to have from the perspective of everyday culture. But in my opinion, they come from a misconceptualization of seminar culture. Here is a conceptualization that can help dissolve these thoughts.

You have a right to understand. If something is said in a seminar, you have a right to understand it. Science progresses not by ineffable truths that cannot be explained, but by clearly articulated common knowledge. It helps to also remember that:

• If you don’t understand something, it is likely that many other people do not as well.
• Articulating a confusion is often itself a useful intellectual act. Sometimes we may not even realize that we are missing something until it is pointed out.

Asking questions shows respect. When I ask a question, it shows that I am interested enough in the topic to engage with it, and that I trust the speaker to give an informative answer. Not asking questions implies that the topic is either not worth engaging with, or that you don’t think the speaker is equipped to answer. Questions show respect.

Speaker Culture

You have a right to direct the conversation. A vigorous seminar audience will likely have more questions than you have time to answer, and might sometimes focus on early aspects of a talk that are not the main point. Therefore, as the speaker, you always have a right to direct the conversation to the aspects that will be most interesting or fruitful. You can simply politely cut off a current line of questioning by explaining that there are other topics you want to get to, and promising to engage later if necessary.

Honest answers show courage. As the speaker, perceptive questions will often stretch the limits of your own understanding. It can be tempting to reflexively deflect or bluster to hide this. But it is much better to be honest about those limits (while feeling free to engage in speculation). Learning the limits of your own knowledge is also a great opportunity for growth.

Being an Excellent Participant

The above norms for speakers and listeners set the ground rules for a productive seminar. But there is more you can do to help actively stimulate learning. Here are a few principles:

• As a listener, be mindful of cognitive load. The speaker has to manage an entire audience of dozens of people, while you as a listener really only have to worry about yourself. So if there’s a question that’s bugging you, that the speaker doesn’t initially give a good answer to, try to do as much work as you can to productively reformulate your question, rather than making the speaker figure it out for you. (Of course, sometimes this isn’t possible, and the speaker does have the advantage of being the expert on the topic. But it’s good to try to offload cognitive load from the speaker whenever possible.)
• As a listener, be mindful of tone. This is in some sense a corollary of cognitive load. Certain tones take extra effort to gracefully process or to respond to (e.g. dismissiveness, condescension, extreme assertiveness, etc.). We should mostly want tone to be fairly neutral (neither timid nor overbearing, but curious and assertive).
• As a speaker, be mindful of tone. Treating questions dismissively will ensure that other people don’t ask questions. We generally don’t do this intentionally, but e.g. giving a short, confident-sounding, but incomplete answer can make it psychologically harder to ask follow-up questions.
• As a speaker, avoid rambling. Sometimes when we aren’t completely satisfied with our own answer, we end up rambling or repeating the same answer in several different ways. This can end up taking up several minutes of time if you don’t catch yourself. Once you’ve said what you have to say, move on to the next slide or the next question (fine to acknowledge if you think there might be more to say after further thought).

None of these are things we will remember all the time, and it's not a big deal if you forget, but these are all habits to aspire to that will improve the experience for both you and others.

Tips for High-Trust Environments

For high-trust environments (like our own group meeting), we can do even better. Here we can keep in mind that everyone is on the same team, and our goal is to help each other excel. In particular:

• Don’t be afraid to ask tough questions. Our meeting is a safe space, and asking tough questions now helps the speaker think through them before they present externally.
• Hold others to the standard you would hold yourself. From knowing all of you, I know that we all hold ourselves to a high personal standard–we want to do excellent work on the most important problems in ML. Let’s call this the standard of excellence. In seminars, I think we sometimes make the mistake of holding the speaker to the standard of defensibility: can they give a reasonable-seeming answer to questions of why/how they did something? Defensibility isn’t just too low of a standard, it’s actually the wrong standard: any ambitious project is going to go out on a limb in some ways, and there will be parts of it that are more speculative. Optimizing for defensibility leads us to avoid ambition. So get the speaker to convince you that this is excellent, rather than defensible, work.

For a completed project, my aspirational goal as a speaker is usually to convince the audience that my work addresses a key issue on one of the most important problems in the field (or ideally the world), and that they should be working on this question if they have the right skillset. I almost never meet this goal, but the point is that striving for it leads me to meet higher levels of excellence over time. I think we should all at least periodically strive for this goal in our talks, realizing that we won’t meet it but that the gap can reveal important lessons or important directions of future work. Similarly, as an audience we should consider holding the speaker to this standard. At the same time, we should recognize that anyone who is even inviting this standard in the first place is already performing an act of virtue, and that even being able to talk about where it falls short means that it’s in a comparison class with outstanding work.

On the other hand, many of the presentations in our group are (and should be) on preliminary work or half-baked ideas. Here the above standard is not particularly helpful, and the honest answer to some questions will be “I dunno, I just have some vague intuition that this is a good idea”. Asking those questions is still valuable as long as they are well-targeted (in the sense that we could reasonably expect a more interesting answer than “I have some vague intuition”, or if they point to a place where it would be particularly useful to refine the intuition). But it’s also useful to think in terms of more brainstorm-y questions: “Have you tried X?”, “This seems related to other interesting thing Y”, “What about this alternative framing?”, “I think your high-level question is interesting, but how do you grapple with key conceptual issue Z? Maybe you could try this technique”. Actually, these are great questions even for a fully-baked talk. But for half-baked ideas we should conspicuously increase the number of these types of questions, because the goal is to help give the speaker useful ideas rather than to construct a thorough collective understanding of the topic.

If you’re a speaker who feels nervous giving talks, remember that you’re among friends whose ultimate goal is to help you do great research. This is the time to take risks, get feedback, and grow. Similarly, if you’re an audience member who feels hesitant to ask questions, think of this as the place to expand your comfort zone and try things you wouldn’t normally try. And of course, if you have any thoughts or questions about any of this, feel free to leave a comment here or ask me one-on-one.

Levels of Understanding

Finally, I want to talk about different levels of understanding (which is, after all, the point of a seminar).

(Note: The first example below is a bit dense because it’s about a mathematical definition. Feel free to skip to the second example, on robustness, if it’s too much effort to decipher.)

Let’s suppose that in some talk you see the following definition:

A function f on [0,1] is Holder continuous with parameter α if, for k = floor(α) it satisfies |f^(k)(x)-f^(k)(y)|≤C|x-y|^α-k for some constant C>0, for all x,y.

This definition is probably mysterious to you (it was to me). Let’s suppose you ask the speaker for some intuition on what this definition is doing. There’s at least three levels of explanation they could give:

Level 1: For α=1 this is the same as being Lipschitz, so think of this as a generalization of Lipschitz.

Level 2: Morally, this is asking that the function be “α times differentiable”, where we want α to not necessarily be a whole number. For integer α the condition exactly says that f should have α derivatives, while for α<1 it asks the function locally to grow as |x-y|^α, which is weaker than differentiability but approaches differentiability as α->1.

Level 3: A level 2 explanation, plus a description of in what sense this is really a generalization of differentiability (i.e. what analogous properties we get), or some explanation of why this is the “right” way to generalize differentiability. [I don’t actually know the answer to this…]

Of course, the level 3 or level 2 explanation might take too long to get across in a talk. But it’s useful to realize that level 3 is always out there, and to notice as a listener when you’re only at level 1 or level 2. And as a speaker, if you don’t have time for at least a level 2 explanation, consider if this definition is really worth putting up there (why not just talk about regular old differentiability and then mention that there’s a generalization?).

These levels apply to all aspects of a talk, not just mathematical definitions. For instance, imagine a talk about robustness, where the speaker is describing the motivation for their work.

Level 1: Robustness is important.

Level 2: The problem we’re considering gets at the following aspect of robustness, which is important.

Level 3: In the field of robustness, one of the core difficulties is X (as evidenced by {conceptual issue, consultation with practitioners, etc.}). We will tackle problem P which offers a way forward on addressing X.

And for motivation in particular, there’s also a final level:
Level 4: In the world at large, M is one of the most important problems, as evidenced by {effect on GDP, important historical analogues, effect on important institutions, etc.}. Machine learning robustness offers a uniquely compelling angle on M for reasons R. <Followed by level 3 explanation>

In practice, it is rare for a seminar to ever touch on Level 4. This is probably partly due to time constraints, partly because many academics consider it “out of scope”, and partly because of the possibly impolite implication that other fields of study are less important. The main exception is job talks, where something on level 4 is expected. I think it’s probably correct for Level 4 to be rare in seminars, but I’d personally also like to see slightly more of it at the current margin. For instance, if you’re at the point of presenting a body of work rather than a single paper, I think it’s worthwhile to at least argue for why this is a compelling direction within the field of ML (we could call that level 3.5).

Finally, while addressing the higher levels requires a deep understanding on the part of the speaker, there are similar levels that apply even to something that isn’t well-understood. For instance, suppose in an applied ML talk, there is a mysterious heuristic H that improves the results. One could say:

Level 1: H works.

Level 2: H works, and we have no idea why.
OR H works, for intuitive reason R.

Level 3: H works, and we have no idea why. We haven’t really looked into it [possibly followed by reason why this isn’t a core issue for the present work].
OR H works, and we have no idea why. We tried looking into X,Y,Z to understand it but none of them turned up much insight.
OR H works, for what we speculate is intuitive reason R, but we haven’t really looked into it.
OR H works, for what we think is intuitive reason R, and here’s some additional follow-up evidence that seems to support R.

Note that at each level, there are multiple possible explanations depending on the speaker’s actual level of knowledge. Level 1 simply asserts the empirical observation. Level 2 couples it with the speaker’s opinion about the observation, while Level 3 presents what I’d call the full epistemic status surrounding the observation (i.e. what surrounding questions have been investigated and how they support/don’t support different theories). Of course, the bottom example in Level 3 is preferable to the top example, but only one of those is an honest portrayal of the work, and the speaker doesn’t have the power to change that during a talk. What they do have power over is whether they give a Level 1, 2, or 3 explanation. Therefore, as the speaker, have the courage to give a Level 3 explanation even if it acknowledges uncertainty, and as a listener have the wisdom to accept such a Level 3 explanation and to respect the speaker’s courage and integrity.

Conclusion. Now that we have these levels in mind, we can better understand the seminar norms discussed above. The purpose of these norms is to reach the highest level of understanding possible about the most important aspects of a topic, and to socially reward speakers and listeners who move us towards that understanding.