(Cross-posted from my site.) My goal here is to quickly describe a potential problem that groups of people may have when trying to collectively figure out the truth of some subject. The basic idea is that people may use each others' beliefs to inform their own beliefs, without properly accounting...
This is a post I wrote on my personal blog after a discussion with a deep learning professor at the University of Chicago. I don't know if this particular topic has been studied in much depth elsewhere, so I figured I would share it here. If you know of any...
Thanks for pointing this out -- this is helpful to position this in idea space. Basically what I'm describing here is an information cascade where actors fail to realize that others' beliefs are not independent.
(Cross-posted from my site.)
My goal here is to quickly describe a potential problem that groups of people may have when trying to collectively figure out the truth of some subject.
The basic idea is that people may use each others' beliefs to inform their own beliefs, without properly accounting for the fact that others' beliefs are based on some of the same evidence as their own beliefs.
In what follows I present a simplified example of how this might happen, a bit of math to guide our thinking, and some illustrative plots.
The basic takeaway is that a failure to "de-correlate" others' beliefs from the supporting evidence leads to sub-optimal epistemics. In particular:
Ah thanks for pointing out the typo.
I'll probably create a post soon-ish with more visualizations covering cases like the ones you suggested.
You're right about the model being pertinent to cases where we've already solved the alignment problem pretty well, but want to try other safety measures. I'm particularly thinking about cases where the AIs are so advanced that humans can't really supervise them well, so the AIs must supervise each other. In that case, I'm not sure how p would behave as a function of AI capability. Maybe it's best to assume that p is increasing with capability, just so we're aware of what the worst case could be?
This is a post I wrote on my personal blog after a discussion with a deep learning professor at the University of Chicago. I don't know if this particular topic has been studied in much depth elsewhere, so I figured I would share it here. If you know of any related work (or have any other comments on this, of course), let me know.
tldr: Supposing that someone figures out how to make an AGI and solve the alignment problem at least to some degree, would it be helpful to distribute many copies of the AI and have them keep each other in check? I propose a mathematical model to think about this question and conclude that this would probably be helpful at first, but would probably diminish in effectiveness over time. Under some assumptions, this approach may actually be counterproductive.
I love the commentary about meaning and morality here, how Quirrel pushes back on Harry's "obvious theorems" about morality by pointing out that in the end everyone simply does what they want to do, with, I think, the suggestion that a Dark Lord is simply someone who is really good at getting what he wants (and a lot of ambiguity about whether this is actually a good way to act).
And of course the pervasive strangeness of the fact that this conversation is occurring between rationalist wizards in a HP fan fiction just makes it even better...
The Discord group link here appears to be broken.