Consider a situation where multiple news outlets publish similar stories. A reader may model these as independent, and thus strongly update on the information presented. But imagine that the text of the story makes clear that each outlet obtained the story from the same wire service. This suggests the stories may not be as independent as they first appeared, and the reader may have over-updated by modeling them as independent. This is the phenomenon of correlation neglect. Treating the separate stories as independent essentially double-counts the evidence provided by the wire service. Accounting for the correlation in the evidence provided by the stories would reduce the size of the update. It's noteworthy that this can still be an over-update even if each news outlet is unbiased with regard to the coverage subject.
Consider the ability to @-invoke an LLM in a conversation. An LLM response that supports one side in the discussion may be modeled as independent evidence for that side. This may be particularly appealing because an LLM may have a lot of the features of a neutral third party weighing in on the discussion. But here again, neutrality or unbiasedness is not the same as independence. The responses LLMs give may be correlated with the responses given by participants in the discussion, and thus may not be fully independent, even if they are unbiased in expectation.
Suppose Alice and Bob are discussing some topic, which Alice discussed with an LLM last week. Bob @-invokes an LLM and receives a response that agrees with him. He asks Alice: will you update on this? Alice's view already includes similar responses from the LLM conversation she had last week, but she feels pressure to update because the LLM is cast as the neutral third party. To update in this case would be double counting.
One way to think about this is that LLM outputs may already be "priced in" to the discourse on a topic. How could this happen? Several possibilities:
Participants have already asked LLMs about this topic before the discussion.
A third party asked an LLM a similar question, and a participant read a blog post by that third party which incorporates the ideas in the LLM output.
The ideas in the LLM output are already prominent in training data which has also been read by participants. Here, the training data acting as the wire service, and the LLM as one of the news outlets.
These channels are mostly invisible at a community or discourse level, so the same or substantially similar LLM outputs may be counted many times, and views become progressively more correlated.
To mitigate this phenomenon, I suggest focusing on the new information provided by @-invocations rather than the bare fact that the invocation largely supports one side or the other. Part of the temptation of @-invocation as a user experience is that your interlocutor may be forced to admit you are correct when an LLM comes in as a neutral third party to support your interpretation of the argument. The neutrality makes this seem reasonable, but neutrality alone is insufficient for an update. If an LLM response largely just weighs in supporting an argument already provided, this doesn't add a ton of new information. On the other hand, if the LLM response provides an argument the participants hadn't considered or surfaces facts that weren't previously known, there is more reason for a strong update.
One advantage of the @-invocation format compared with general LLM use is that it makes LLM use more transparent and structured, and thus possible correlation easier to analyze. If I were involved in a discussion last week where I acknowledged an update based on an LLM @-invocation, I can point to this in future cases to show that I have already "priced in" this output. The potential risk is a general version of the "snark reply" mentioned here, that @-invocation will be used primarily for its support or not of a given idea, rather than the information brought to bear on the discussion. The ability to read an invocation’s run page and check what the model looked at other than the thread may also be helpful for understanding what new information the model is bringing to the table.
I think the general principle is, update on what the model found, not on which side it took.
Consider a situation where multiple news outlets publish similar stories. A reader may model these as independent, and thus strongly update on the information presented. But imagine that the text of the story makes clear that each outlet obtained the story from the same wire service. This suggests the stories may not be as independent as they first appeared, and the reader may have over-updated by modeling them as independent. This is the phenomenon of correlation neglect. Treating the separate stories as independent essentially double-counts the evidence provided by the wire service. Accounting for the correlation in the evidence provided by the stories would reduce the size of the update. It's noteworthy that this can still be an over-update even if each news outlet is unbiased with regard to the coverage subject.
Consider the ability to @-invoke an LLM in a conversation. An LLM response that supports one side in the discussion may be modeled as independent evidence for that side. This may be particularly appealing because an LLM may have a lot of the features of a neutral third party weighing in on the discussion. But here again, neutrality or unbiasedness is not the same as independence. The responses LLMs give may be correlated with the responses given by participants in the discussion, and thus may not be fully independent, even if they are unbiased in expectation.
Suppose Alice and Bob are discussing some topic, which Alice discussed with an LLM last week. Bob @-invokes an LLM and receives a response that agrees with him. He asks Alice: will you update on this? Alice's view already includes similar responses from the LLM conversation she had last week, but she feels pressure to update because the LLM is cast as the neutral third party. To update in this case would be double counting.
One way to think about this is that LLM outputs may already be "priced in" to the discourse on a topic. How could this happen? Several possibilities:
These channels are mostly invisible at a community or discourse level, so the same or substantially similar LLM outputs may be counted many times, and views become progressively more correlated.
To mitigate this phenomenon, I suggest focusing on the new information provided by @-invocations rather than the bare fact that the invocation largely supports one side or the other. Part of the temptation of @-invocation as a user experience is that your interlocutor may be forced to admit you are correct when an LLM comes in as a neutral third party to support your interpretation of the argument. The neutrality makes this seem reasonable, but neutrality alone is insufficient for an update. If an LLM response largely just weighs in supporting an argument already provided, this doesn't add a ton of new information. On the other hand, if the LLM response provides an argument the participants hadn't considered or surfaces facts that weren't previously known, there is more reason for a strong update.
One advantage of the @-invocation format compared with general LLM use is that it makes LLM use more transparent and structured, and thus possible correlation easier to analyze. If I were involved in a discussion last week where I acknowledged an update based on an LLM @-invocation, I can point to this in future cases to show that I have already "priced in" this output. The potential risk is a general version of the "snark reply" mentioned here, that @-invocation will be used primarily for its support or not of a given idea, rather than the information brought to bear on the discussion. The ability to read an invocation’s run page and check what the model looked at other than the thread may also be helpful for understanding what new information the model is bringing to the table.
I think the general principle is, update on what the model found, not on which side it took.