By Gordon Hull
In a couple of previous posts (first, second), I looked at what I called the implicit normativity in Large Language Models (LLMs) and how that interacted with Reinforcement Learning with Human Feedback (RLHF). Here I want to start to say something more general, and it seems to me like Derrida is a good place to start. According to Derrida, any given piece of writing must be “iterable,” by which he means repeatable outside its initial context. Here are two passages from the opening “Signature, Event, Context” essay in Limited, Inc.
First, writing cannot function as writing without the possible absence of the author and the consequence absence of a discernable authorial “intention:”
“For a writing to be a writing it must continue to ‘act’ and to be readable even when what is called the author of the writing no longer answer for what he has written, for what he seems to have signed, be it because of a temporary absence, because he is dead or, more generally, because he has not employed his absolutely actual and present intention or attention, the plenitude of his desire to say what he means, in order to sustain what seems to be written ‘in his name.’ …. This essential drift bearing on writing as an iterative structure, cut off from all absolute responsibility, from consciousness as the ultimate authority, orphaned and separated at birth from the assistance of its father, is precisely what Plato condemns in the Phaedrus” (8).
Second, iterability puts a limit to the use of “context:”
“Every sign, linguistic or nonlinguistic, spoken or written (in the current sense of this opposition), in a small or large unit, can be cited, put between quotation marks, in so doing it can break with every given context, engendering an infinity of new contexts in a manner which is absolutely illimitable. This does not mean that the mark is valid outside of a context, but on the contrary that there are only contexts without any center or absolute anchorage” (12)
It seems to me that Derrida’s remarks on iterability are relevant in the context of LLMs because they indicate that LLMs are radically dependent on iterability. This is true in at least three ways, each of which points to an important source of their implicit normativity.
First, and in an obvious way, the intelligibility of ChatGPTs output depends on intelligibility without intent because ChatGPT does not have intentional states. Of course, along with many other artifacts (such as Amazon’s Alexa), it is designed to be treated as if it has intentional states. Under the hood, however, is a (very elaborate) process of statistical weights and probabilities. Hence the catchy (if a little misleading, as the process isn’t random) moniker “stochastic parrots.” This separation from “consciousness” is what makes the inference to intentionality both possible and completely unnecessary: those who interact with LLMs may very well think that they are interacting with something that has consciousness, but iterability means that language use is at best a contingent guide to consciousness. In Derrida’s words, when we look to analyze an utterance, “the intention animating the utterance will never be through and through present to itself and to its content. The iteration structuring it a priori introduces into it a dehiscence and a cleft which are essential” (18). Let’s call this an “intentionality gap.”
The fact that language use has always been an indicator of consciousness thus turns out to be a historical artifact, the product of our inability to make devices that produce language. If this is right, the entire debate about whether LLMs are conscious is based on a missing argument. The presence of text indicates that it is possible that one is dealing with a conscious entity but not that it is necessary. Something further will be needed to sustain that proof.
Second, and less obvious, the LLMs depend on iterability in the opposite direction: all of the speech that they ingest in training also must be iterable for it to function in a device that will use it to produce intelligible text. A very large series of random letters and digits would not suffice as a training corpus for a LLM. If it were purely random, the LLM would be unable to discern patterns sufficient to make the statistical inferences it uses to generate language. That is of course trivially true, and is why LLMs are trained on text corpora – their training data has to be “readable” as writing (I am leaving out multimodal models for the moment, but a similar kind of argument would have to apply: an image has to be readable as an image, etc.). Readability opens an inevitable normative question: what kind of text counts as sufficiently readable to use as training data? It is at this level that we can locate some of the curational transparency issues I raised previously. A given corpus of text scraped from the internet is likely to include both a lot of nonsense that doesn’t count as “readable” in this sense, as well as a bunch of material that model designers do not want to count as readable because, regardless of its statistical prevalence, it is the sort of text they do not want represented in the model. To say this is not, by the way, to state a normative preference for how that process ought to go; it is instead to notice that you aren’t going to go from scraping things off the Internet to a “readable” corpus of training data without making some normative decisions about what counts as “readable” for the purpose of your model (I will develop this point further in a future post with regard to Derrida’s “debate” with Searle, because I think there is more going on here than readability).
Third, all of this is a basic condition of the models being able to function at all: the training data has to be separable from its contexts and placed in new contexts. At the level of linguistic signs, this makes a lot of sense, and it is part of what enables language use to be creative, surprising, and so on. Presenting old ideas and words in new contexts is a lot of what creative work does. Here, though, the LLMs differ from human speech in important ways. On the one hand, the statistical norming process means that they are going to be unlikely to jump contexts (will they be less likely than humans? That strikes me as an empirical question; here the only point is to notice something about how they’re designed). The output of LLMs is going to tend to revert to the mean of human language use. The model will calculate contexts with a pretty high degree of complexity, and the further away a given use is from the aggregate of the contexts in which the word has appeared, the less likely the model will be to output it. In short, context will tend to put constraints on the output of an LLM in a way different from what people do.
On the other hand, there are some linguistic utterances that come very heavily weighted to a specific context and are meaningful in part because most speakers are trying to invoke that context, and assuming that auditors will get it. Consider the phrase “I have a dream.” It is perfectly iterable in the sense that it is meaningful in contexts other than the MLK speech. But most of the time, invoking is to make a cultural reference. When the LLM does this, it is not making a cultural reference, because it does not understand singular events as singular. Instead, they are patterns and regularities in the training data.
Here is ChatGPT 3.5:
It is as if one of those overpaid corporate consultants took some drugs (a lot of drugs). It does sort of evoke the MLK speech (“I dream of a future…”) but it really doesn’t feel like a reference so much as a drug-addled memory. Indeed – and I have some thoughts about this that I’ll try to develop in a future post – there’s some real Derridean questions we can ask: is this a “serious” text? Is it making fun of drug-addled corporate consultants? If it were a person producing that paragraph, we’d look to the context of the utterance for some clues on whether/how to read it. The LLM’s context is determined by statistics, and its output is going to be differently legible.
These issues strike me as worth pursuing in part because of some work by Yarden Katz on how image recognition software handles singular historical events, like the picture of Ruby Bridges being escorted into school. In those cases, it classifies the image – as a picture of people walking together – but it completely misses both the signification of the image (something to the effect of “desegregation happening”) and the referent (Bridges). In short, LLMs are treating the iterability/context nexus differently from how people do, and the move toward a statistical metric of context can generate results that are very strange from the point of view of a human language user.
Recent Comments