TL;DR: I'm releasing my templates to make running feedback rounds easy for research teams that might otherwise neglect to set it up.
The main questions on my feedback form template
Why I wrote this post:
Feedback is my job:
My role on research projects mentored by Ethan is somewhere between a people manager and a research assistant for the team.
Feedback, and more generally, facilitating direct and honest communication between team members (and Ethan), is one of the main ways I add value.
My feedback process is pretty good:
I’ve run feedback rounds for two cohorts of Ethan Perez’s mentees so far.
When Ethan first asked me to run feedback for his mentees, I adapted what I was able to glean about how Anthropic runs peer-led performance reviews.
I don't think I've perfected the process, but my first version was good enough that one of Ethan’s collaborators (Akbir Khan) now uses my templates for his projects, and describes them as useful.
Ethan's mentees, both on MATS and Astra have commented that the feedback was helpful for them - and I've noticed a substantial increase in directness and honesty on projects once I've run at least one round of semi-formal feedback.
So, now that I've iterated on this feedback process, I'm interested if other research teams would find it useful, so I’m releasing the templates here.
In the rest of the post, I'll explain how my feedback rounds work, plus my best guesses of when and how to run feedback in a research project.
If you're somewhere between curious and unconvinced, you can keep reading to hear my takes on why teams don't do structured rounds of feedback (or sometimes any) by default, and why I think running feedback is worth it.
To summarise the rest of this post: I think rounds of feedback encourage collaborators to communicate more openly about their needs and concerns for their projects. I think many teams wouldn't communicate very well without doing something at least a bit like this feedback process at least once per project.
Thanks to Rajashree Agrawal, Sam Svenningsen for letting me talk this post through while I was planning it, and to Mikita Balesni and Miranda Zhang for comments on my drafts.
My workflow for feedback starts with sending out a short doc framing this feedback round to the team, and linking a form for them to fill in for each of their colleagues. I collect the form responses and format them into a doc, and set up 1:1s between each person & the team lead to get people to reflect on the feedback they've received.
announce that we're doing a round of team feedback,
explain what the implications of this feedback will be
is this going to be used to evaluate who should continue collaborating with the team?
or does the project have a hard stop coming up and this feedback is to help everyone reflect on what they've learned and what they want to work on next?
offer some guidelines on how I give feedback,
(you should edit that section based on what you think is true)
set a deadline ~1-2 weeks away,
(always set a shorter deadline than you actually need, and give people time over a weekend to do the feedback out of their core focus hours)
Everyone fills out the form, resubmitting it for each person they want to give feedback to, according to the doc.
I open the responses to my form as a google sheet, and copy over the feedback for each person into a doc and share it with them and the project lead (grouped by feedback giver).
I've had almost no instances of people passing anonymous feedback to each other via my feedback form, but if there were any I'd create a section for anonymous feedback at the end of the doc.
If copying feedback into the docs is too much manual effort: you could just copy the information across out of the google sheet, or experiment with using language models to reformat the feedback data.
I arrange 1:1s about a week later, so that each person talks to the project lead about the feedback they've received, to:
clarify any questions the feedback recipient has,
prioritise what feedback areas they should especially work on (including positives to double down on!), and
set any expectations for how things might have to change for the team to be excited to keep working together.
That's it! If you think this sounds useful, feel free to just make a copy of my google drive folder with all my templates on it. Steal them and make them your own, adapt them however you think would be actually helpful for you.
Read on if:
You want to hear more about when and how to run feedback, or
You think this doesn't sound very useful yet, or
You just want to hear why I think feedback is good (and why people don't do it)
When to run feedback:
TL;DR: I think the best time to run a round of feedback is about 1/3 of the way into the predicted run-time of a project.[1]
Usually I don't notice when it's time to run a round of feedback: instead, I get this sinking feeling that I've waited until too late. Luckily, I don't think there are strong downsides to running feedback a bit too early.
When I chatted to Akbir about writing this post, he mentioned that he uses my prompts about 1/3 of the way into his projects. I think that sounds roughly right. You should find the sweetspot for your own project. I recommend looking for the point where you've worked together long enough to have noticed useful feedback, but for there to be plenty of time to straighten anything out before the end of the project.
Here's why I endorse that 1/3rd number, based on the story of the two feedback rounds I've run so far on the Astra Fellowship:
Running very early feedback still felt useful:
On Ethan's Astra Fellowship, I ran a round of "first impressions" team feedback after 2.5 weeks. It felt surprisingly early, even to me, but I wanted to know what a team would be able to say to each other, even at this early stage.
But since this feedback round was coming so early, I decided to make the process as time-efficient as I could. I organised the feedback as a 2-hour block of 20-minute 1:1s, and required no preparation in advance. Each team member (including Ethan) talked to everyone else. In each 20 minute block, I recommended the team spent the first 10 minutes jotting down 2 strengths and 2 areas of growth for their partner, and to spend the second half briefly talking them through - they only had 5 minutes each way to share their feedback for the other person.
I learned that it wasn't obviously too early to run a round of feedback after the first two weeks of working together:
One team member received helpful feedback signal on communication - a core area they would successfully make progress on in part because we were able to flag this area of growth early on.
But some team members found it hard to generate and reflect on feedback this early on.
It seems reasonable, even likely, that running feedback this early on is quite heavy-tailed in its usefulness.
But looking back, I don't regret checking if there's useful feedback for the team to give each other. In expectation, putting feelers out just in case seems like a decent bet.
I believe it broke the ice pretty well on our team, and established more direct communication norms than we otherwise would have.
But I'm not sure, and I don't think communication on our project was especially smooth sailing after this point.
But then I waited for the team to feel ready. That was too late:
About a month later (almost exactly 1/3rd of the way into our project), I got that sinking feeling. I came back from covid, and in my 1:1s each team member mentioned that they weren't sure where the project was going or what their responsibilities were. We were well overdue on feedback, but I didn't think the team was ready. I was probably wrong.
I kept telling myself that the team "needed to gather more momentum" before we could run another feedback round: a new collaborator had only joined full-time a week before I got back in person, and our project direction had only just started to get close to adequately scoped.
In hindsight, my decision to wait seems mostly wrong. It was so alluring to wait. I was worried I would interrupt whatever natural process the team needed to go through to cohere and work well together. From my 1:1s I knew the team had useful feedback to share with each other - I even started useful conversations in our group meetings about how each member of the team was noticing the same areas of improvement for our project. I had correctly noticed that the team needed to communicate differently, but instead of setting them up to do that amongst themselves, I tried to facilitate those conversations for them.
I think it was an interesting experiment for me to try and take on the group's internal communication myself, but this would have been a better decision if we didn't expect to work together for much longer. Even if I helped the team get back on track, I think I've since noticed more fundamental communication that needed to happen sooner for the project to go well. Since our projects were likely to run for between 1-3 more months, I think I'd now prefer if I had sat everyone down and encouraged them to give their feedback more directly to each other.
I finally made space for a proper round of feedback two weeks before I wrote this post. We were about 2.5 months into our 4-month research project. I'm glad we had at least a month left: everyone has ample time to work on the areas their colleagues fed back about. Since it was my role to read through everyone's feedback to organise it and help them prioritise what to work on, I'm able to say that I think people had plenty of helpful points, both on each others' comparative advantage on the project, and on areas of growth. But I think they could have said most, if not all of it a couple of weeks earlier.
Both times I've run in-depth feedback for Ethan, someone's come to me asking for help in the same way. Two different people coming to me like this in two different feedback rounds isn't quite a pattern yet, but I suspect that it could become one. Each time, they came because in having to write up their feedback, they realised that the current shape of our project was emotionally untenable for them. Each time, they benefitted from substantial attention from me helping to process their aversions to sharing the feedback, and how until this point they'd decided to just put up with how things were. Writing up their feedback for the team encouraged them to reflect about their own experience on the project, and what they needed. I think inspiring that reflection is a surprising and valuable part of the feedback process. In both cases, I was talking to that person at least once per week, and we'd even talked about the areas they'd ultimately decide weren't tenable for them. But in both cases, it took until we ran a formal feedback round to reflect on it themselves, and I wish I could have been there for them earlier.
Why people who value feedback still don't run feedback:
I'm sure I don't have to explain that it's helpful to get evidence-based takes from colleagues about what you're good at and how you could improve. I would expect AI Safety researchers to be easier than most to convince that human feedback is a pretty powerful and important component of making agents more capable and aligned with their supervisor's goals.
But as far as I know, most research fellowships in AI Safety don't really do feedback. My colleagues, ironically enough in their feedback form responses for me, shared that probably nobody else would have run feedback if I didn't. I'm honestly quite confused why.
My working theory is that people feel like only project supervisors/leads have enough authority on the project to set this up. But I think anyone on a project could take the social initiative to propose a round of feedback like this - but if you do so, I'd recommend you take the time to set up this form to invite feedback for and from your whole team. I'd recommend that your supervisor is still involved, as a giver and recipient of feedback, and as the overall reviewer of the feedback.
And even if the supervisor is involved, that doesn't mean they have to execute on it themselves! After all, the first time I ran feedback for Ethan, it's because he delegated it to me. So if you're leading or managing a project, I hope my templates make it cheap enough for you to run feedback. And if you're on a project that could do with a round of feedback, maybe my templates, or this post, can make it easy enough to make that happen.
The hard work isn't done once you've given each other feedback, though:
I'm pretty lucky: my career's been full of environments where people are very ready to adapt to feedback, are very communicative, and understand the value of working on a team. On the start-up I cofounded straight out of university, we ran regular feedback rounds and I felt like I knew what to work on to be as helpful as I could be to my org.
But I'm not that lucky: Even in environments with regular team feedback rounds, I've been burned by a lack of communication. My feedback processes aim to elicit honest and direct comments from my colleagues, but it's up to them to bring that honesty themselves. If someone has a problem with you, and they don't share it even when you ask for feedback, there's a limit to how successful you can be as a team.
Because teams have to use feedback correctly: going from zero feedback to regular feedback is great, even for very short term collaborators. There's a lot of value making that jump. But another, maybe even bigger jump in value happens when going from regular feedback, to making collaborative plans & expectations for growth. When teams are clear with each other about when they'd disband, or what their main uncertainties about working with each other are. Probably it makes sense to invest in this depth once you're planning to work with someone for more than ~2months: once all your life plans start to intertwine with the project, once it's more like a job.
Maybe I'll write more about that when I write up my takes on how to use feedback. For now I'll just summarise:
Setting up feedback doesn't mean you've nailed internal comms. In fact, you need to do more than just tell people what to work on - you need to tell them what's important and what you expect and what would happen if they did or didn't succeed at working on the thing you're giving them feedback on. If you don't have answers to those questions, I don't think you're well-prepared to sustainably work on a team together for even the medium-term.
I'm looking for feedback:
If you used my templates, and if you adapted them in anyway, I'd be curious what you changed, and how it went! I'd be excited about folks posting their thoughts in the comments of this post.
I don't have an anonymous feedback form: I'm excited about soliciting feedback at specific times when I'm looking to grow, from high-context collaborators.
But if you have anything about how your feedback rounds panned out that you'd rather not comment, feel free to DM me instead.
TL;DR: I'm releasing my templates to make running feedback rounds easy for research teams that might otherwise neglect to set it up.
Why I wrote this post:
In the rest of the post, I'll explain how my feedback rounds work, plus my best guesses of when and how to run feedback in a research project.
If you're somewhere between curious and unconvinced, you can keep reading to hear my takes on why teams don't do structured rounds of feedback (or sometimes any) by default, and why I think running feedback is worth it.
To summarise the rest of this post: I think rounds of feedback encourage collaborators to communicate more openly about their needs and concerns for their projects. I think many teams wouldn't communicate very well without doing something at least a bit like this feedback process at least once per project.
Thanks to Rajashree Agrawal, Sam Svenningsen for letting me talk this post through while I was planning it, and to Mikita Balesni and Miranda Zhang for comments on my drafts.
How my feedback rounds work:
My templates can be found here.
My workflow for feedback starts with sending out a short doc framing this feedback round to the team, and linking a form for them to fill in for each of their colleagues. I collect the form responses and format them into a doc, and set up 1:1s between each person & the team lead to get people to reflect on the feedback they've received.
That's it! If you think this sounds useful, feel free to just make a copy of my google drive folder with all my templates on it. Steal them and make them your own, adapt them however you think would be actually helpful for you.
Read on if:
When to run feedback:
TL;DR: I think the best time to run a round of feedback is about 1/3 of the way into the predicted run-time of a project.[1]
Usually I don't notice when it's time to run a round of feedback: instead, I get this sinking feeling that I've waited until too late. Luckily, I don't think there are strong downsides to running feedback a bit too early.
When I chatted to Akbir about writing this post, he mentioned that he uses my prompts about 1/3 of the way into his projects. I think that sounds roughly right. You should find the sweetspot for your own project. I recommend looking for the point where you've worked together long enough to have noticed useful feedback, but for there to be plenty of time to straighten anything out before the end of the project.
Here's why I endorse that 1/3rd number, based on the story of the two feedback rounds I've run so far on the Astra Fellowship:
Running very early feedback still felt useful:
On Ethan's Astra Fellowship, I ran a round of "first impressions" team feedback after 2.5 weeks. It felt surprisingly early, even to me, but I wanted to know what a team would be able to say to each other, even at this early stage.
But since this feedback round was coming so early, I decided to make the process as time-efficient as I could. I organised the feedback as a 2-hour block of 20-minute 1:1s, and required no preparation in advance. Each team member (including Ethan) talked to everyone else. In each 20 minute block, I recommended the team spent the first 10 minutes jotting down 2 strengths and 2 areas of growth for their partner, and to spend the second half briefly talking them through - they only had 5 minutes each way to share their feedback for the other person.
I learned that it wasn't obviously too early to run a round of feedback after the first two weeks of working together:
But then I waited for the team to feel ready. That was too late:
About a month later (almost exactly 1/3rd of the way into our project), I got that sinking feeling. I came back from covid, and in my 1:1s each team member mentioned that they weren't sure where the project was going or what their responsibilities were. We were well overdue on feedback, but I didn't think the team was ready. I was probably wrong.
I kept telling myself that the team "needed to gather more momentum" before we could run another feedback round: a new collaborator had only joined full-time a week before I got back in person, and our project direction had only just started to get close to adequately scoped.
In hindsight, my decision to wait seems mostly wrong. It was so alluring to wait. I was worried I would interrupt whatever natural process the team needed to go through to cohere and work well together. From my 1:1s I knew the team had useful feedback to share with each other - I even started useful conversations in our group meetings about how each member of the team was noticing the same areas of improvement for our project. I had correctly noticed that the team needed to communicate differently, but instead of setting them up to do that amongst themselves, I tried to facilitate those conversations for them.
I think it was an interesting experiment for me to try and take on the group's internal communication myself, but this would have been a better decision if we didn't expect to work together for much longer. Even if I helped the team get back on track, I think I've since noticed more fundamental communication that needed to happen sooner for the project to go well. Since our projects were likely to run for between 1-3 more months, I think I'd now prefer if I had sat everyone down and encouraged them to give their feedback more directly to each other.
I finally made space for a proper round of feedback two weeks before I wrote this post. We were about 2.5 months into our 4-month research project. I'm glad we had at least a month left: everyone has ample time to work on the areas their colleagues fed back about. Since it was my role to read through everyone's feedback to organise it and help them prioritise what to work on, I'm able to say that I think people had plenty of helpful points, both on each others' comparative advantage on the project, and on areas of growth. But I think they could have said most, if not all of it a couple of weeks earlier.
Both times I've run in-depth feedback for Ethan, someone's come to me asking for help in the same way. Two different people coming to me like this in two different feedback rounds isn't quite a pattern yet, but I suspect that it could become one. Each time, they came because in having to write up their feedback, they realised that the current shape of our project was emotionally untenable for them. Each time, they benefitted from substantial attention from me helping to process their aversions to sharing the feedback, and how until this point they'd decided to just put up with how things were. Writing up their feedback for the team encouraged them to reflect about their own experience on the project, and what they needed. I think inspiring that reflection is a surprising and valuable part of the feedback process. In both cases, I was talking to that person at least once per week, and we'd even talked about the areas they'd ultimately decide weren't tenable for them. But in both cases, it took until we ran a formal feedback round to reflect on it themselves, and I wish I could have been there for them earlier.
Why people who value feedback still don't run feedback:
I'm sure I don't have to explain that it's helpful to get evidence-based takes from colleagues about what you're good at and how you could improve. I would expect AI Safety researchers to be easier than most to convince that human feedback is a pretty powerful and important component of making agents more capable and aligned with their supervisor's goals.
But as far as I know, most research fellowships in AI Safety don't really do feedback. My colleagues, ironically enough in their feedback form responses for me, shared that probably nobody else would have run feedback if I didn't. I'm honestly quite confused why.
My working theory is that people feel like only project supervisors/leads have enough authority on the project to set this up. But I think anyone on a project could take the social initiative to propose a round of feedback like this - but if you do so, I'd recommend you take the time to set up this form to invite feedback for and from your whole team. I'd recommend that your supervisor is still involved, as a giver and recipient of feedback, and as the overall reviewer of the feedback.
And even if the supervisor is involved, that doesn't mean they have to execute on it themselves! After all, the first time I ran feedback for Ethan, it's because he delegated it to me. So if you're leading or managing a project, I hope my templates make it cheap enough for you to run feedback. And if you're on a project that could do with a round of feedback, maybe my templates, or this post, can make it easy enough to make that happen.
The hard work isn't done once you've given each other feedback, though:
I'm pretty lucky: my career's been full of environments where people are very ready to adapt to feedback, are very communicative, and understand the value of working on a team. On the start-up I cofounded straight out of university, we ran regular feedback rounds and I felt like I knew what to work on to be as helpful as I could be to my org.
But I'm not that lucky: Even in environments with regular team feedback rounds, I've been burned by a lack of communication. My feedback processes aim to elicit honest and direct comments from my colleagues, but it's up to them to bring that honesty themselves. If someone has a problem with you, and they don't share it even when you ask for feedback, there's a limit to how successful you can be as a team.
Because teams have to use feedback correctly: going from zero feedback to regular feedback is great, even for very short term collaborators. There's a lot of value making that jump. But another, maybe even bigger jump in value happens when going from regular feedback, to making collaborative plans & expectations for growth. When teams are clear with each other about when they'd disband, or what their main uncertainties about working with each other are. Probably it makes sense to invest in this depth once you're planning to work with someone for more than ~2months: once all your life plans start to intertwine with the project, once it's more like a job.
Maybe I'll write more about that when I write up my takes on how to use feedback. For now I'll just summarise:
Setting up feedback doesn't mean you've nailed internal comms. In fact, you need to do more than just tell people what to work on - you need to tell them what's important and what you expect and what would happen if they did or didn't succeed at working on the thing you're giving them feedback on. If you don't have answers to those questions, I don't think you're well-prepared to sustainably work on a team together for even the medium-term.
I'm looking for feedback:
If you used my templates, and if you adapted them in anyway, I'd be curious what you changed, and how it went! I'd be excited about folks posting their thoughts in the comments of this post.
I don't have an anonymous feedback form: I'm excited about soliciting feedback at specific times when I'm looking to grow, from high-context collaborators.
But if you have anything about how your feedback rounds panned out that you'd rather not comment, feel free to DM me instead.
i.e., 1/3 of the shortest duration you expect the project to run for, not accounting for extensions or planning fallacy.