Thank you for the reply. You might be interested in neural Darwinism if you've never heard of it, the comment you linked in the edit made me think of it: https://en.wikipedia.org/wiki/Neural_Darwinism.
I don't have a good story for how reuse of subcomponents leads to cooperation across agents, but my gut says to look at reuse rather than voting or markets. Could be totally baseless though.
This is really interesting. My immediate gut reaction is that this wrongly treats different subagents as completely separate entities, when really they're all overlapped chimeras. For example, Subagent A might be composed of {C,D,E} while Subagent B is composed of {C,E,F}. Reuse of subcomponents seems like a more natural path to coordination than internal voting systems or prediction markets to me.
What distinguishes subagents from shards? Are both getting at the same idea? https://www.alignmentforum.org/w/shard-theory
Two additional senses in which a "right to be wrong" might be justified: in differing risk preferences, individually, or the usefulness of holdout populations, societally.
I don't think people should try to emulate heliocentrists because I think that acting like they did would generally lead people to failure, not success. The lesson I take from this is that stubborn holdout populations who refuse to accept the obvious are important to the health of science as an ecosystem of ideas. But I don't think stubbornness should be seen as a general purpose virtue. I think Aristotle and co just experienced epistemic luck.
I agree with much of this, but I suspect people aren't only sticking with activation-based interpretability because the bad dimensionality of weight-based interpretability is intimidating. Rather, I feel like we have to be thinking about activation-based interpretability if we want an analysis of the model's behavior to contain semantics that are safety-relevant.
For example, I can know nothing about how safe a classifier that distinguishes A from B is, regardless of how much I know about its weights, unless I know what A and B are. There might be identical... (read more)