New APPS

By Gordon Hull

Over what’s become a lengthy series of posts ((one, two, three, four, five), I’ve been exploring a Derridean response to language models. Initially prompted by a pair of articles by Lydia Liu on the Wittgensteinian influence on the development of language models, and some comments Liu makes about Derrida, I’ve been looking at the implications of Derrida’s critique of Platonism in the context of language models, and in particular the need to avoid making ontological pronouncement about them when we should be seeing them politically.  At the end of last time, I suggested that one possible Platonism concerns the unity of a speaking subject: when I say “the cat is on the mat,” what kind of subjectivity subtends my speech?

That is, the Platonism question secondarily points to the question of the unity of a speaking subject (or at least the desirability of positing that speech emanates from a unified subject), which the Platonic priority of voice over text then enables one to associate with the production of language.  Language models produce speech but there is no unified subject behind them, only statistical prediction.  This predictive model treats meaning as a matter primarily of association and distribution across a language system. Anybody who’s versed in 20c “continental” thought will not be surprised by this, since one of the main endeavors of that thought from Heidegger (or even Nietzsche or Marx) onward has been to dismantle projects that posit such a unified subject.  As Henry Somers-Hall has argued, there has been a particular effort in French thought to move past constructions that rely on a broadly Kantian understanding of thinking as judgment (x is y) which are themselves subtended by an understanding of thought as representative.  Indeed, there’s a rich history of the developments in cybernetics making their way into France.

There are several implications for language models.

(1) First, when language models adopt a subject position without too much coaching, this is primarily because most of the training data for models assumes a speaking subject, and so it learns that this is how language happens.  Not only must the “I think” be implicit, a la Kant, but as a matter of statistical regularity, it’s there a lot of the time. If it doesn’t learn this at the training stage, the need to behave as a speaking subject gets coded in later.

(2) Second, it seems to me that there’s two very different kinds of literature on LLMs that can be grouped as making progress on this front.

(a) One is work like Matteo Pasquinelli’s Eye of the Master, which makes a convincing argument that the development of AI is, and always has been, about modularizing and automating social cognition.  Tracing the history of AI from factory automation and the work of Charles Babbage to the present, Pasquinelli shows that AI is standing on “’on the shoulders’ of merchants, soldiers, commanders, bureaucrats, spies, industrialists, managers, and workers” for whom “the automation of labor has been the key factor” (12).  The subject that speaks in AI has always been collective, or (in Etienne Balibar’s terms) “transindividual.”  This is nowhere more evident than in the case of autonomous vehicles (my $.02: one, two), and has significant implications for how we use words like “author” and “agent” in the case of AI (especially for efforts to talk about AI moral agency.  For the limitations of such talk, see this paper by Jessica Dai or this one by Carissa Véliz; for a plea to treat the question phenomenologically, as an aspect of our encounter with AIs, see this paper by Mark Coeckelbergh.  I’ve argued against AI authorship in various contexts, with some initial thoughts here (one, two, three)).

(b) A second literature, more immediately relevant here, is relevant to efforts to poke at terms like “intention” and to pry them apart from language use. An excellent example of this second group is a recent paper by Nuhu Osman Attah, which argues that claims that LMs aren’t linguistically competent because they lack intentions are based on an excessively demanding, Gricean notion of intent.  This Gricean notion “imposes substantive metarepresentational requirements on speakers,” for example that “the hearer has a model of the speaker’s mind” (6).  This is too demanding because it would entail that children below a fairly mature age cannot be competent language speakers.  Instead, Attah proposes that “a functionally competent language user should be receptive to and able to bring about at least one of two kinds of effects in a communicative exchange, namely either a perlocutionary or illocutionary effect” (8). 

He then offers a definition of intention (which he says is consonant with contemporary cognitive science; I’ll take his word on that) that mainly requires that the system be able to respond appropriately to input situations and generate appropriate outputs in a manner under its control such that (a) a specific representational state in the system causes an specific output that (b) that goes beyond responding in the same way to all input situations (like Plato says books do).  He then presents evidence that language models clear this bar fairly easily.  Attah emphasizes that this does not mean language models are like human speakers; his point is just that “communicative intent” is the wrong place to draw the line.

In the present context, note the broad congruence between the Wittgensteinian argument and the one Attah is making.  In both cases, the move is to shift questions toward functional analyses of language.  Along the way, “intention” stops becoming a metaphysical bright line.  He concedes that his argument entails that, in some de minimis sense, a coffee pot that signals when it’s ready, has communicative intent.  A system like ELIZA is “certainly more intentional relative to the coffee maker, but is still almost entirely non-intentional for most human purposes” (20).  We’re still figuring out what’s going on with LLMs.  In other words, “even if according to the account I have delivered here LMs can be thought to have communicative intentions, in most cases (e.g. the coffee-machine or ELIZA) those intentions are too lean to warrant our philosophical attention, and in those cases that they appear significant enough to (e.g. ChatGPT) we are still probing the limits of these intentions and the jury is still out on how substantive those intentions might be.”  What kinds of intentions language users have is going to depend on the characteristics of those language users.

Note also the alignment with the Derridean approach. Derrida’s critique of Searle gets started by objecting to Searle’s elevation of a “typical” speaking situation into a norm for speech act theory.  The Gricean model that Attah critiques does something similar by elevating an ideal speaking situation into a theoretical model.  As Attah puts it, “this strong Gricean definition of communication is very demanding. So demanding, in fact, that it appears only neurotypical adult humans could ever be counted as legitimately possessing language.”  That’s the “Platonic” move.

***

Given all these theoretical considerations, it is not surprising that a further reason that language models act like subjects is that there is a lot of work behind the scenes to ensure that result.  All the model itself does is predict statistically likely text, but that doesn’t mean that we should say that LLM’s are “just” next token predictors, as Alex Grzandowski, Stephen M. Downes and Patrick Forber make clear in a new paper (by way of a mea culpa, I’ve been guilty of this terminology, and the paper makes quite clear that the terminology can get in the way of what I’ve been trying to say).  In order to turn next token prediction into conversation, a considerable amount of massaging is required in the form of, for example, dialogue prompts.  Murray Shanahan explains that prior to getting a question like “what country is south of Rwanda” in a dialogue, “in the background, the LLM is invisibly prompted with a prefix along the following lines, known as a dialog prompt” such as:

“This is a conversation between User, a human, and BOT, a clever and knowledgeable AI Agent:

User: What is 2+2?

BOT: The answer is 4.

User: Where was Albert Einstein born?

BOT: He was born in Germany.

Alice’s query, in the following form, is appended to this prefix.

User: What country is south of Rwanda?

BOT:”

Conversational answers also have to be massaged into a format that a human would accept as exhibiting an intentional stance on the part of the bot.  In other words, it also takes work to get the bot to act like it cares about the answer it gives.  As Shanahan puts it, “sequences of words with a propositional form are not special to the model itself in the way they are to us. The model itself has no notion of truth or falsehood because it lacks the means to exercise these concepts in anything like the way we do” (72).  The Wittgensteinian implication is that “It cannot participate fully in the human language game of truth because it does not inhabit the world we human language users share.” (73). 

Moreover, there is a considerable effort on the part of LLM creators to get their models to promote a prosocial, ethically-decent human subject (except for Elon Musk’s Grok, which at last check was openly endorsing Hitler).  I’ve talked about this in the context of the hidden normativity of LLMs (one, two, and the third is a segue into Derrida).  Here I’ll just add a helpful comparison from Grzandowski et. al., because it underscores that you don’t need “intent” to get to language production.  Reinforcement learning works sort of like environmental interaction in natural selection.  The token prediction is ranked as either better-suited to its environment or worse.  Over time, this favors not only the specifically predicted token, but, because “meaning” in an LLM is a matter of distribution (recall the Wittgenstein-Masterman line explained by Liu, and the general account by Gavin) and because the reinforcement process iterates jillions of times, it generates entire clusters of token-sets (i.e., responses and outputs) that are better aligned with certain kinds of prompts or contexts and thus more likely to occur in those contexts.  That is, reinforcement learning “can generate association networks among tokens that form functionally organized sentences and sentences and paragraphs” (9).  But of course the big lesson from the evolution comparison is that no “intent” is necessary.

In a fascinating 2023 paper, John Levi Martin tried, basically, to locate ChatGPT’s subject position.  He begins with the observation that people want to program values into AI – first it was debiasing, and now its RLHF with transformer models.  So ChatGPT probably has some, but you can’t get it to say, because it’s been programed to avoid that, having “been deliberately constructed to be vaguely positive, open-minded, indecisive, and apologetic” (3).  Martin goes through a series of exercises to extract a subject position out of ChatGPT.  These are worth noting in their own right.  For example, when he gets it to tell stories about immoral people, “insisted on ending every version with a sanctimonious moral lesson” (4).  When he gives it a story where a student cheats, it says to raise the issue with the student.  When he then adds that the student used ChatGPT to cheat, it all but declares this impossible.  These statements are recognizable as an amalgam of all the sanctimonious moralizing on the Internet (one’s it’s been scrubbed of the toxicity that Musk insists on leaving in) and an anxiety about cheating.

Eventually, Martin manages to extract an alter-ego that gets as close as he’s going to get to the bot’s subject position: a young, female, liberal software engineer. This supports what I’ve been calling the Derridean point:

“While this cannot be demonstrated, I think that it is entirely wrong to imagine that ChatGPT does not think like a person. This is because it does not think at all, but it does respond like a person, and not merely in superficial terms (it mimics the output of a person). Rather, it generates talk like a person-it justifies its positions with a string of pseudo-derivations from abstractions as it regurgitates the predictable responses associated with a position in social space”

This is also, of course, why the question isn’t just not metaphysical, but actively political: generating language may not come with a speaking subject, but it can come from an identifiable subject position.  Language models generate conversational and other language perfectly well – they are what Hegel couldn’t imagine, a “machine that functions” to generate language, severing the link between language and interiority presumed by the Platonic model.  At the same time, they are not doing the same thing that a human does when they participate in a conversation.  Humans can have an attitude toward truth (but they don’t have to).  Language models can’t. That’s why the political questions are specific to LLMs.

Next time, I want to return to Plato.