Text generated by an LM is not grounded in communicative intent
I think that LLM consists of typical cognitive architecture elements: world model, cost model, executive controller and planner, all amalgamated together in this big mess. From this perspective, fine-tuning LLMs for dialogue I think imparts their executive component with communicative intent.
It seems to me that:
The text (obviously) codifies meaning, but that meaning is often underdetermined by lack of understanding of the culture or the author's idiosyncrasy, lack of acquaintance with the object of reference or simply because the corpus is not large enough to disambiguate; easily all of those at the same time, at least for old enough texts.
Is there anything missing from this account?
There can also be meaning that the author simply didn't intend. In biblical interpretation, for instance, there have been many different (and conflicting!) interpretations given to texts that were written with a completely different intent. One reader reads the story of Adam and Eve as a text that supports feminism, another reader sees the opposite, and the original writer didn't intend to give either meaning. But both readers still get those meanings from the text.
But that's because the meaning is underdetermined, there is information (explicit meaning) within the texts that constraints the space of interpretations, but it still allows for several different ones.
How much the text is underdetermined is both a function of the text and of the reader, the reader may lack (as I said) cultural or idiosyncratic context, acquaintance with the object of reference; or the text (which is what provides the new information) being too short to disambiguate.
Cross posted from New Savanna.
The phrase “stochastic parrot” was coined, I believe, by Emily Bender, a computational linguist and one of the coauthors of the paper, On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Consider this passage:
I have been happy to accept this view, while at the same time denying that LLMs are stochastic parrots. That’s the view I took in my 2020 working paper, GPT-3: Waterloo or Rubicon? Here be Dragons, and which I have maintained until quite recently. Then, at the end of March, a discussion I had over at LessWrong has lead me to revise that view, if only provisionally, in a post, MORE on the issue of meaning in large language models (LLMs). There, following an argument that John Searle had made about certain digital devices, where he pointed out that they can be said to compute only in relation “to some actual or possible consciousness that can interpret the processes as computationally.” So, mutatis mutandis, the operations of an LLM are only meaningful relative to some actual or possible consciousness that can interpret the processes as being meaningful.
It seems to me, then, that the question of whether or not the language produced by LLMs is meaningful is up to us. Do you trust it? Do WE trust it? Why or why not?
Having put it that way, I was reminded of discussions that roiled academic literary criticism in the third and on into the fourth quarter of the previous century. Those discussions were about the meaning of literary texts. There was no doubt that they are meaningful, very meaningful. Rather, the issue was just how do we determine their meaning and whether or not they had determinable meaning at all.
The business of interpretive criticism was relatively new to the academy, only becoming institutionalized after World War II, and some in the profession had become disconcerted that critics were arriving at different interpretations for texts. If literary criticism is to produce objective knowledge about literary texts, then shouldn’t critics agree on the meaning of texts? THAT question generated quite a bit of debate in various terms.
Some critics were content, and some even actively pleased, with the interpretive variety. Others, however, were quite distressed, and wanted to tame it. By far the favorite way of taming intepretive variety was to claim that literary texts mean what their authors intend them to mean. The job of the critic, then, is to determine authorial intention.
Just how do you do that? That was not at all obvious. After all, in the majority of cases one could not query the author because the author was dead. In some cases the author was unknown. Just who was the Peal Poet? Was Homer a single individual or a scattered group of individuals? We don’t know.
And then along came the deconstructionists. In a well-known essay, “Form and Intent in the American New Criticism” (collected in Blindness and Insight), Paul de Man intention is not a physical thing that can be transferred from author to reader, “somewhat as one would pour wine from a jar into a glass.” In that essay de Man goes on to a somewhat complex argument that intention is somehow bound up in the text, but the damage had been done. That explanation never really worked. It is the reader’s, the critic’s, intention that brings life to the inanimate marks on the page.
And that, as far as I can tell, is where the issue rests. To be sure, the arguments continued for a while and, no doubt, are still kicking around here and there. For the most part, however, the arguing is over. The text is all we’ve got.
No one doubts that those texts were produced by human beings. But the contexts of production are lost to us, forever. That is one thing in the case of texts like Iliad, Gilgamesh, or Beowulf, where we cannot attach an individual name to the them and where the historical record is sparse, and somewhat different the the case of, say, Samuel Coleridge and his “Kubla Khan,” where we have extensive notes, correspondence and other writings. What are the chances the “Kubla Khan” was written by a future LLM that sent it back in time through a vial containing medicinal tincture of opium? What are the chances that a future LLM will produce a text as valued among humans as “Kubla Khan”? If not an LLM, almost certainly NOT an LLM, but some other future device of manufacture most intricate and wondrous?