By Gordon Hull
I’ve been using (part 1, part 2) a new paper by Fabian Offert, Paul Kim, and Qiaoyu Cai to think more about Derrida’s use of iterability as a way in to thinking about transformer-based large language models (LLMs) like ChatGPT. Here I want to wind that up with some thoughts on Derrida and Searle.
Near the end of the paper, Offert, Kim and Cai summarize that:
“The transformer does exactly more and less than language. It removes almost all non-operationalizable sense-making dimensions (think, for instance of interior paratextuality, of performativity and contingency, or of anything metaphorical that is more complex than a simple analogy) – but it also adds new sense-making dimensions through subword tokenization, word embedding, and positional encoding. Importantly, these new sense-making dimensions are exactly not replacing missing information, but they are adding new, continuous information” (15).
This returns us to the Derridean concerns I’ve been articulating. Recall that in his polemic against Searle, Derrida accuses Searle’s version of speech act theory of too closely modeling phenomenology, both in assuming an intentional agent behind speech acts and in taking “typical” speech situations as central, as opposed to “parasitic” ones like humor. I suggested that there is good evidence that various efforts to impose normative structures on language models – RLHF, detoxification, etc. – push them to perform in ways that call to mind Derrida’s critique of Searle. By taking certain language situations as normal, Searle is making the account of speech acts normative before it even gets started. In his own defense, Searle argues that the reduction to typical speech acts is for convenience only, and for keeping the model tractable. Techniques like detoxification and RLHF similarly reduce the range of the models’ output. Evidence of this is that LLMs lack the contextual richness to have a sense of humor, no matter how otherwise sophisticated their output.
Offert, Kim and Cai’s paper lets one add that this reduction runs much deeper. The very processes of tokenization, for example, are designed to reduce the number of possible tokens to a tractable number. In this respect, the move is analogous to Searle’s. It is defensible for the same reason: it lets you get to a generalizable model. But it’s vulnerable to the Derridean critique, also for the same reason. The model makes a number of assumptions that aren’t the same as what language does. So there is a certain sloppiness in talking as though it’s an accurate representation of language. All models abstract; that’s not the point. For subword encoding, the point is that the abstraction isn’t choosing to ignore certain aspects of reality in order to produce a model, it’s that the abstraction changes the nature of what it is modeling. That’s fine – but that also means that the although the transformer model is producing something that looks like language, the process by which it gets there is definitively not linguistic.
The point is as methodological as substantive. In his reply to Derrida, Searle (Derrida quotes this) repeats that “once one has a general theory of speech acts … it is one of the relatively simpler problems to analyze the status of parasitic discourse” (205) and that “the terms in which this question can be intelligibly posed and answered already presuppose a general theory of speech acts” (205). The Derridean point, as I understand it, is that Searle is begging the question, because the proposed theory of speech acts begins with an idealization, by bracketing parasitic discourse. If you start with an idealization, develop an ideal theory, and then apply that theory to the data you excluded from it, you at least have to explain why the data you excluded can be adequately explained by the theory. More radically, it’s possible that the excluded data would produce a different theory, had it been included.
Evidence of this latter point is abundant in studies of AI and machine learning (ML), where ML models often fail because of biases and distortions in their training data. Derrida’s point is that the exclusion of “parasitic” speech cannot be assumed to be without consequence. It’s a version of the Adorno’s point against Hegel that “objects do go into their concepts without leaving a remainder” (Negative Dialectics, Ashton trans., p. 5). I mention it here because it seems to me that it is especially important in the context of models like ChatGPT. For one, as I’ve been arguing, iterability functions differently in them. Second, a lot of the apparatus that Searle presupposes in his speech act theory is different in language models.
For example, intentionality does a lot of work for Searle (recall this is why Derrida accuses him of being a phenomenologist). In his reply to Derrida, he argues that “the argument that the author and intended receiver may be dead and the context unknown or forgotten does not in the least show that intentionality is absent from written communication; on the contrary, intentionality plays exactly the same role in written as in spoken communication. What differs in the two cases is not the intentions of the speaker but the role of the context of the utterance in the success of the communication” (201).
The way that this is a misreading matters in the context of language models. Derrida doesn’t think intentionality is absent; he thinks that language presupposes iterability – the ability to separate a word from its context – and that iterability undermines claims about the primacy of intentionality. Even if you don’t think figuring out the intention of any utterance is ultimately a matter of educated guesswork, certainly the intention of an utterance separated from its context would be much harder to determine. Anybody who’s made a serious study of Aristotle sees the point. Language models do a lot of work separating utterances from their contexts, in ways that we often don’t understand (and can’t, due to model opacity). But things like subword encoding represent known big breaks with context, and the assimilation of new contexts.
The risk is in starting with cases that are so presumptively clear that it’s easy to forget how artificial they are. It’s like all the ethics thought experiments that depend on a scenario that’s not actually going to happen, even in principle. They clarify something, but it’s not clear how that clarity is going to help us. You at least need an argument for why that clarity is helpful, and you’re open to the claim that the simplification is pernicious (I made this case about the ticking time bomb argument to justify torture a while ago). This is why Derrida wants to call speech act theory ethico-political:
“I am convinced that speech act theory is fundamentally and in its most fecund, most rigorous, and most interesting aspects (need I recall that it interests me considerably?) a theory of right or law, of convention, of political ethics or of politics as ethics. It describes (in the best Kantian tradition, as Austin acknowledges at one point) the pure conditions of an ethical-political discourse insofar as this discourse involves the relation of intentionality to conventionality or to rules” (Limited, Inc., 97)
To get back to Searle, he continues:
“To show this [the role of intention, from the above quote] ask yourself what happens when you read the text of a dead author. Suppose you read the sentence, ‘On the twentieth of September 1793 I set out on a journey from London to Oxford.’ Now how do you understand this sentence? To the extent the author said what he meant and you understand what he said you will know that the author intended to make a statement to the effect that on the twentieth of September 1793, he set out on a journey from London to Oxford, and the fact that the author is dead and all his intentions died with him is irrelevant to this feature of your understanding of his surviving written utterances.” (201)
Well, ok, not to go too Derridean, but how do we know that he said what he meant? That he wasn’t out to deceive someone? That he wasn’t writing the first sentence of a planned story? And so on. You absolutely know what he could mean, but without the hidden normative guardrails, you don’t know for sure.
And that, really, is the point about the normativity of language models. It seems clear enough that a language model is going to ingest that sentence as a “normal” one, and that it will ingest most sentences as normal. To the extent that it doesn’t, because what we take to be a “normal” sentence turns out not to be statistically normal on the Internet speech on which it trains, we add guardrails of the sort Searle does. These are absolutely necessary for the models to both function (at all) and to generate speech that we don’t find dangerous, toxic, or otherwise bizarre. But they’re going into the game with a (human-adjusted but initially) statistically-derived version of what Searle is assuming about language. That’s also why language models are bad at humor.
Of course, language models are also literally intention-less in that they generate text predictively. This is just a different understanding of language than we’re used to, and we need to be very attentive to how that plays out. Searle actually has something useful to say on this point, albeit indirectly. He argues that Derrida misreads him on intentions, suggesting that Derrida succumbs to the “illusion that somehow illocutionary intentions if they really existed or mattered would have to be something that lay behind the utterances, some inner pictures animating the visible signs.” He then argues that “in serious literal speech the sentences are precisely the realizations of the intentions …. The sentences are, so to speak, fungible intentions. Often, especially in writing, one forms one’s intentions (or meanings) in the process of forming the sentences: there need not be two separate processes” (202).
For language models, at least, one needs just to take this a step further: the appearance of an intention behind a text emerges at the same time as the model forms the sentence. You don’t need to subscribe to any form of deconstructive or postmodern view of subjectivity and intentionality to note our tendency to read intention into language models says more about us than the models, and that it does so in politically consequential ways. Offert, Kim and Cai offer one more layer of reasons to support that thought.
Recent Comments