Misaligned agendas and terminology among academic, industrial and independent AI alignment research

This post aims to fill some gaps between technical AI Alignment topics and academic AI research. It summarises a quick but informed scouting of academic research papers that are closely connected to four Alignment topics: mechanistic interpretability, safety via debate, value alignment and cooperative AI. This overview should help somebody wanting to do research on any of those four topics to find the related research and expand the technical Alignment frontier.

Audience: somebody who knows what research is (typically a PhD) and is familiar with technical topics in AI Alignment
Epistemic status: informed guess after going over recent AI Alignment research papers; biased towards academic AI research on formal methods, especially argumentation, but more ignorant about ML

TLDR: mechanistic interpretability can be seen as a form of formal explainability; AI safety via debate is vague and informal and could benefit from formal argumentative dialogue games (and has in fact been formalised using several formalisms); value alignment (including RLHF and similar) has deep connections to computational social choice theory and reasoning with preferences in AI; cooperative AI might benefit from recent fairness and multi-agent research.

Takeaway

My key takeaway is that technical Alignment research pre-prints often fail to refer to peer-reviewed, formal AI research articles that are highly relevant to the Alignment topic studied, but that do not use the typical Alignment terminology. This may be due to several reasons. For one, some academic research pre-dates Alignment topics -- for instance, formal verification and explainability via formal methods go way back before mechanistic interpretability (or even machine learning (ML) interpretability at large). In other cases, authors of non-peer-reviewed papers may simply be unaware of, or may intentionally omit related research -- for instance, the AI safety via debate pre-print does not even mention some two decades of research on argumentative debates and dialogue games. In yet other cases, the authors seem to be aware of some relevant research but fail to explore either deeper or broader connections -- for instance, the authors of What are human values, and how do we align AI to them? acknowledge and use some insights from s...

kcyras

kcyras

Alignment Gaps

kcyras

kcyras

Alignment Gaps