Why does the AI safety community need help founding projects?

1. AI safety should scale
   1. Labs need external auditors for the AI control plan to work
   2. We should pursue many research bets in case superalignment/control fails
   3. Talent leaves MATS/ARENA and sometimes struggles to find meaningful work for mundane reasons, not for lack of talent or ideas
   4. Some emerging research agendas don’t have a home
   5. There are diminishing returns at scale for current AI safety teams; sometimes founding new projects is better than joining an existing team
   6. Scaling lab alignment teams are bottlenecked by management capacity, so their talent cut-off is above the level required to do “useful AIS work”
2. Research organizations (inc. nonprofits) are often more effective than independent researchers
   1. “Block funding model” is more efficient, as researchers can spend more time researching, rather than seeking grants, managing, or other traditional PI duties that can be outsourced
   2. Open source/collective projects often need a central rallying point (e.g., EleutherAI, dev interp at Timaeus, selection theorems and cyborgism agendas seem too delocalized, etc.)
3. There is (imminently) a market for for-profit AI safety companies and value-aligned people should capture this free energy or let worse alternatives flourish
   1. If labs or API users are made legally liable for their products, they will seek out external red-teaming/auditing consultants to prove they “made a reasonable attempt” to mitigate harms
   2. If government regulations require labs to seek external auditing, there will be a market for many types of companies
   3. “Ethical AI” companies might seek out interpretability or bias/fairness consultants
4. New AI safety organizations struggle to get funding and co-founders despite having good ideas
   1. AIS researchers are usually not experienced entrepeneurs (e.g., don’t know how to write grant proposals for EA funders, pitch decks for VCs, manage/hire new team members, etc.)
   2. There are not many competent start-up founders in the EA/AIS community and when they join, they don’t know what is most impactful to help
   3. Creating a centralized resource for entrepeneurial education/consulting and co-founder pairing would solve these problems

I am a Manifund Regrantor. In addition to general grantmaking, I have requests for proposals in the following areas:

• Funding for AI safety PhDs (e.g., with these supervisors), particularly in exploratory research connecting AI safety theory with empirical ML research.
• An AI safety PhD advisory service that helps prospective PhD students choose a supervisor and topic (similar to Effective Thesis, but specialized for AI safety).
• Initiatives to critically examine current AI safety macrostrategy (e.g., as articulated by Holden Karnofsky) like the Open Philanthropy AI Worldviews Contest and Future Fund Worldview Prize.
• Initiatives to identify and develop "Connectors" outside of academia (e.g., a reboot of the Refine program, well-scoped contests, long-term mentoring and peer-support programs).
• Physical community spaces for AI safety in AI hubs outside of the SF Bay Area or London (e.g., Japan, France, Bangalore).
• Start-up incubators for projects, including evals/red-teaming/interp companies, that aim to benefit AI safety, like Catalyze Impact, Future of Life Foundation, and YCombinator's request for Explainable AI start-ups.
• Initiatives to develop and publish expert consensus on AI safety macrostrategy cruxes, such as the Existential Persuasion Tournament and 2023 Expert Survey on Progress in AI (e.g., via the Delphi method, interviews, surveys, etc.).
• Ethics/prioritization research into:
  • What values to instill in artificial superintelligence?
  • How should AI-generated wealth be distributed?
  • What should people do in a post-labor society?
  • What level of surveillance/restriction is justified by the Unilateralist's Curse?
  • What moral personhood will digital minds have?
  • How should nations share decision making power regarding transformative AI?
• New nonprofit startups that aim to benefit AI safety.

I'm open to this argument, but I'm not sure it's true under the Trump administration.

Technical AI alignment/control is still impactful; don't go all-in on AI gov!

• Liability incentivizes safeguards, even absent regulation;
• Cheaper, more effective safeguards make it easier for labs to meet safety standards;
• Concrete, achievable safeguards give regulation teeth.

Not sure this is interesting to anyone, but I compiled Zillow's data on 2021-2025 Berkeley average rent prices recently, to help with rent negotiation. I did not adjust for inflation; these are the raw averages at each time.

I definitely think that people should not look at my estimates and say "here is a good 95% confidence interval upper bound of the number of employees in the AI safety ecosystem." I think people should look at my estimates and say "here is a good 95% confidence interval lower bound of the number of employees in the AI safety ecosystem," because you can just add up the names. I.e., even if there might be 10x the number of employees as I estimated, I'm at least 95% confident that there are more than my estimate obtained by just counting names (obviously excluding the 10% fudge factor).

So, conduct a sensitivity analysis on the definite integral with respect to choices of integration bounds? I'm not sure this level of analysis is merited given the incomplete data and unreliable estimation methodology for the number of independent researchers. Like, I'm not even confident that the underlying distribution is a power law (instead of, say, a composite of power law and lognormal distributions, or a truncated power law), and the value of $p\left(1\right)$ seems very sensitive to data in the vicinity, so I wouldn't want to rely on this estimate except as a very crude first pass. I would support an investigation into the number of independent researchers in the ecosystem, which I would find useful.

By "upper bound", I meant "upper bound $b$ on the definite integral ${\int }_a^{b}p\left(x\right)dx$". I.e., for the kind of hacky thing I'm doing here, the integral is very sensitive to the choice of bounds $a,b$. For example, the integral does not converge for $a=0$. I think all my data here should be treated as incomplete and all my calculations crude estimates at best.

I edited the original comment to say "$\infty$ might be a bad upper bound" for clarity.

It's also worth noting that almost all of these roles are management, ML research, or software engineering; there are very few operations, communications, non-ML research, etc. roles listed, implying that these roles are paid significantly less.

Apparently the headcount for US corporations follows a power-law distribution, apart from mid-sized corporations, which fit a lognormal distribution better. I fit a power law distribution to the data (after truncating all datapoints with over 40 employees, which created a worse fit), which gave $p\left(x\right)\sim 399x^{-1.29}$. This seems to imply that there are ~400 independent AI safety researchers (though note that $p\left(x\right)$ is probability density function and this estimate might be way off); Claude estimates 400-600 for comparison. Integrating this distribution over $x\in [1,\infty )$ gives ~1400 (2 s.f.) total employees working on AI safety or safety-adjacent work ($\infty$ might be a bad upper bound, as the largest orgs have <100 employees).