A lot of people in the community, including me, are working independently (or have a lot of autonomy, even if they are employed). A lot of people, including me, often feel like they are underperforming or at least wonder if they are. But how do I actually know when I'm not underperforming? I'd like to make some criteria for under which circumstances I'll consider my work satisfactory. I'd be curious to hear what other people would consider "decent" output.
This is obviously hard to define but some types of data that seem helpful:
- How much do you get done in a typical month/half year?
- How much do consider aspirational but realistic to get done in a typical month/half year?
- How much do you consider on the low end but okay to get done in a typical month/half year?
- What kind of output would you want to see out of a researcher/community organiser/other independent worker within a month/half a year to be impressed/not be disappointed? (Assuming this is amount is representative of them)
- What's the minimum output would you want to see out of a researcher/community organiser/other independent worker to be in favour of them getting funding to continue their work? (Assuming this is amount is representative of them)
- What's the minimum output would you want to see out of your friend to feel good about them continuing their current work? (Assuming this is amount is representative of them)
I literally mean silly things like "X number of substantive research documents of roughly this and that quality" - totally okay if very domain-specific. I know this probably hugely varies from person to person and is hard to answer in general - There's lots to take into account that seems hard to specify like quality of work and often we really care about outcomes instead of outputs. Feel free to rephrase the question in whichever way seems useful to you.
(I know that there is no objective "enough" or that the real measuring stick are my values and reality. (I would have liked to include a link to a post that I'm sure exists but couldn't find it.) And there's a bunch of things that are individual to me and my line of work of course, so I don't expect people to say things that are directly useful and applicable to my situation. But I still think that sometimes a bit of data would be really useful to me and hopefully many others!)
My current goal is one safety-targeted NeurIPS level paper every 4-6 months. It might take me a few months to gain experience to get there, but I'd be happy to be an author on something the quality of the tuned lens, Inference-Time Intervention, IOI or overoptimization papers 3x/year if I could get 30% of the Shapley value each time (meaning some of those are first authors).
On a slightly smaller scale I spent 3 weeks * 0.75 FTE on Catastrophic Regressional Goodhart (both the proofs and writeup) and Drake spent about the same time. We also spent 3 weeks on experimental work which didn't go anywhere. I was disappointed at the time, but considering the good reception to the post and the lack of progress at my concurrent MIRI work it meets my bar in retrospect.
On an even smaller scale, this week I spent one day seeing if I could apply subnetwork probing to Leela Chess Zero and gave up before actually writing experiment code because I hit Python-C++ interoperability issues. This was fine but I would have been sad to spend 3 days the same way, because I should be able to run the whole experiment in 3 days. I think below this scale it's hard to communicate my standards because it would be less than 1% of a meaningful research output, and it's hard to tell 0.2% of a project from 0.5% of a project from the outside.
Edit: the last paragraph was planning fallacy, I got stuck a couple more times and the project should now take ~14 days.