New APPS: Art, Politics, Philosophy, Science

Privacy and “Injury in Fact,” part 2: ye olde “injury in fact”

Last time, I set out the question of judicial standing and the abrupt switch by the Supreme Court in 1970 to the requirement that plaintiffs show an “injury in fact” to obtain standing.  Here I want to look at the historical development of that term.

The earliest use of the phrase “injury in fact” in US courts is in Hepburn and Dundas v. Auld, an 1809 Supreme Court case about contracts.  “Specific performance” of a contract means that you can compel the other party to do exactly what the contract says (as opposed to, for example, doing something else of equivalent value).  Hepburn and Dundas (H&D) contracted with Graham for the sale of some 6,000 acres of land for $18,000, payable in installments with interest.  H&D also owed Dunlop & Co. a large sum of money, and agreed with Auld (Dunlop’s agent) to transfer the contract to Auld, including the rights to enforce it, “towards the discharge” of this debt (271).

Casemine.com offers the following helpful summary of what happened:

“Hepburn and Dundas entered into an agreement with Colin Auld, wherein they bound themselves to assign a land contract to Auld in lieu of paying a debt. When they failed to execute this assignment on the stipulated date, they sought specific performance from the court to compel Auld to accept the assignment and release them from all claims by Dunlop Co. The Supreme Court, through the majority opinion delivered by Chief Justice Marshall, ruled against Hepburn and Dundas. The Court found that despite subsequent efforts to cure title defects, the initial omissions rendered the contract unsuitable for specific performance. The bill for specific performance was dismissed, upholding the decision that fundamental contractual and title requirements must be met unequivocally”

In other words, Hepburn and Dundas tried to force Auld to accept assignment of a land contract, per an agreement they had made with him.  The Court ruled that, because the title had problems, it was inappropriate to enforce the contract exactly as written and force Auld to accept the (defective) title. 

How does this have anything to do with injury in fact?  As part of the proceedings - basically a subplot that doesn't bear on the final resolution - questions had arisen as to whether H&D actually held title to the land, Auld having averred that “it was apparent that their title was bad, or at all events doubtful” (264).  In response:

“Hepburn and Dundas filed a supplemental bill which states their title. It avers possession ever since 1773, and refers to certain title papers; they say that they verily believe their title to be good, and never heard a doubt till long after the tender of the assignment; that as soon as the objections were made known, they took pains to remove them, and have lately obtained deeds of confirmation from the surviving patentees. That the title of Sarah, one of the co-devisees of John West, after her death in 1795, descended upon her brothers Thomas, John, and Hugh and her sister Catharine, and that John, Hugh, and Catharine have lately confirmed their title, and refer to the deeds, and they suppose that Thomas had passed all his title to Sarah's part by a deed executed before her death.” (264).

The case then recites the details of these efforts and their documentation. The question, then arose as to whether H&D’s efforts to clarify and secure the title to the land constituted interference with the contract. The Court concludes that it did not because it did not cause Auld an “injury in fact:”

“The interference of Hepburn and Dundas, in accommodating the suit with Graham, is also urged as an objection to their conduct. They had certainly no right to interfere without the consent of Colin Auld. But when the correspondence is inspected, and it is perceived that they interfered only to effect the object he had himself desired, and which he had avowed his own inability to effect without their consent, the interference must be considered as innocent in point of intention, and unproductive of injury in fact.  The court, then, perceive[s] nothing in the conduct of the plaintiffs, up to the decision of the suit with Graham, which ought to defeat their right to demand a specific performance of this contract. Could they at that time have conveyed a good title, Colin Auld ought to have accepted it.” (Hepburn & Dundas v. Auld, 9 U.S. 262, 275 (1809)

That is, H&D were trying to do what Auld wanted them to do by honoring the contract, and that effort didn’t hurt Auld.

The Court then invokes a counterfactual.  Would things have been different had H&D not tried to remedy the defects in the title?

“These omissions, then, to record the deeds of Thomas and Hugh West, and the total want of title as to Mrs. Bronaugh's part, have produced no real inconvenience to Colin Auld. Had the title been unexceptionable, it would still have been refused, and this contest would still have been carried on with the same determined perseverance which marks the conduct of the parties. Under these circumstances, it is the opinion of the majority of the Court that this case ought to be governed by those general principles which regulate the conduct of a court of chancery in decreeing a specific performance, if the defect of title, which existed at the time of contract, be cured before the decree” (275).

In other words, equitable principles dictate that if the defects in the title can be remedied, that’s ok.  H&D shouldn’t be penalized for interfering with the contract by trying to do so.  Unfortunately for H&D, they didn’t fix the problems with the title: “The omission to record the deed from Thomas West is not cured, and this Court is now to decide whether, under these circumstances, Hepburn and Dundas are entitled to claim a specific performance.”

Should H&D be entitled to force Auld to accept the land transfer, given the problems in recordkeeping?  Equity says no, because Auld would likely incur expenses and take on risk in dealing with the missing deed:

“Had there been simply a deficiency of 208 acres, the majority of the Court would have considered it as a case for compensation; or had the parties entitled to this land been before the court, a division might possibly have been directed, and compensation for that quantity ordered; but however this might be, as persons not before the Court hold this interest, no order can be made respecting it, and it may very much embarrass those acts for asserting the title which may possibly be necessary. The part actually conveyed by Thomas West, too, never having been confirmed by a deed from himself or his heirs, properly recorded, might impose on Colin Auld the necessity of bringing a suit in chancery to perfect his title, or of being subjected to the inconveniences constantly attending the establishment of a deed not recorded, and the risks inseparable from such a deed.” (278).

The Court concludes that “this, therefore, is thought by a majority of the Court to be a case not proper for a specific performance, and the bill is to be dismissed.” (278).

The case is byzantine, and noted for its establishment of principles of contract and equity in real estate (tl;dr: make sure you have clean title to land you want to transfer the rights to!).  But the “injury in fact” language does read like a proto-version of what the Court came up with in 1970: did anything materially bad happen to Auld?  Significantly, the question is whether behavior by H&D which was technically not proper actually caused any such “inconvenience” (“inconvenience” can mean serious harm: Locke uses it, for example, to refer to life in the state of nature, e.g., at Second Treatise secs. 13 and 90).  Concluding it did not, the Court decided that it did not warrant legal intervention.  That said, it’s worth underlining that all of this is framed in traditional juridical terms: the question is one of fulfillment of contractual obligations, and the parties are understood to be bound by the contractual terms and the questions are about whether their behavior hinders the ordinary performance of those terms.

Next time I’ll continue with the historical rabbit hole.

Privacy and “Injury in Fact”

By Gordon Hull

Privacy plaintiffs have a hard time getting their cases heard in court for a variety of reasons.  One of them is that courts lack a coherent and workable understanding of what privacy harms actually are, and how one might articulate them judicially.  This problem bleeds into one of standing, which is what I will address here.

In order to get your case heard on the merits, you have to present a justiciable complaint.  In U.S. federal courts, this obligation stems from Article III of the Constitution, which assigns jurisdiction to “all Cases, in Law and Equity, arising under this Constitution” (Art III, sec 2).  The underlying theory is grounded in the limitation of jurisdiction to actual “cases[s] in controversy,” and grounded in a theory of separation of powers: general policy is to be made by the legislature and enforced by the executive, and the courts are there to resolve controversies that arise either as policy is enforced or between individuals. As Judge Bork (!) put it in an appellate concurrence:

“All of the doctrines that cluster about Article III -- not only standing but mootness, ripeness, political question, and the like -- relate in part, and in different though overlapping ways, to an idea, which is more than an intuition but less than a rigorous and explicit theory, about the constitutional and prudential limits to the powers of an unelected, unrepresentative judiciary in our kind of government.” (cited in Allen v. Wright, 468 U.S. 737, 750 (1984)).

The Supreme Court notes, this means that “the federal courts are under an independent obligation to examine their own jurisdiction, and standing ‘is perhaps the most important of [the jurisdictional] doctrines.’” (FW/PBS v. City of Dallas, 493 U.S. 215, 231 (1990)) .  Ok, so far so good.

How might a court decide standing? SCOTUS elaborates:

New Paper: "Translating Privacy for Data Subjects"

Here's the current draft of a new paper - "Translating Privacy for Data Subjects."  And here's the abstract:

This essay offers a theoretical account of one reason that current privacy regulation fails.  I argue that existing privacy laws inherit a focus on judicial subjects, using language about torts and abstract rights.   Current threats to privacy, on the other hand, presuppose statistically-generated populations and aggregative models of subjectivity.  This gap in underlying presupposition generates the need to translate privacy protections into the data era.  After situating my theoretical account, I identify two areas where data subjects face entry barriers to the legal protections afforded data subjects.  On the one hand, data subjects are often denied standing because privacy harms are deemed speculative and abstract.  On the other hand, algorithmic governance makes it hard to argue that one is an exception to a rule, depriving data subjects of traditional equity.  I then offer two ways to translate privacy for data subjects.  The first is reform of standing through reformed class certification, making the harms data subjects face as populations judicially legible.  The second is a reconceptualization of equity, reframing it both to re-establish the right to be an exception to a data-driven rule and to allow equitable consideration of issues that face statistical populations.

 

A data breach policy idea

In the past few weeks, I’ve gotten more than the usual number of data breach notifications.  Especially from entities that I’d never heard of before the notification.  This points to a specific incentives gap in privacy policy.

In most places, when there’s a data breach, you have to be notified.  This creates an incentive structure for data companies to do something they’d rather not do, which is tell you their security failed.  On the other hand, it offers almost no incentive for them to change their behavior to improve security, since they know perfectly well that there’s not really much you can do except throw the notification in the recycling.  Sometimes you get free credit monitoring if you set it up.  I don’t know what the uptake on that is, but once you have free credit monitoring, you probably won’t get it a second time for the same time period, so the more data breaches there are, the less credit monitoring costs each breach.  So that cost diminishes on its own accord.  For things like 401(k)’s, there’s evidence that you can increase uptake with an opt-out (rather than opt-in) mechanism, but credit monitoring takes time and information to setup, so that probably can’t be done automatically.  In sum, notifications point to a problem but don’t incentivize the entities that could solve it to do so.

Reading List: “Epistemic Justice in Generative AI”

Several folks have explored how algorithmic systems can perpetuate epistemic injustice (my contribution is here).  Those accounts have also generally been specific to supervised systems, as for example those involved in object or image recognition.  For example, I heavily relied on ImageNet and related systems in my paper.  At the same time, I’ve vaguely thought for a while that there’s probably an epistemic injustice dimension to systems like ChatGPT.  A recent paper by Paul Helm and Gábor Bella, which talks about “language model bias” – roughly, the tendency of LLMs to structurally do worse with morphologically-complex languages and thus to be unable to adequately represent concepts that are specific to those languages – as a model of hermeneutic injustice strikes me as compelling proof of concept (I discuss that paper here).  A new paper by Jackie Jay, Atoosa Kasirzadeh and Shakir Mohamed turns this intuition into a model that explicitly extends epistemic injustice theory to generative AI systems, providing a taxonomy of problems, examples of each, and possible remedies.

Kay, Kasirzadeh and Mohamed identify four kinds of specific kinds of “generative algorithmic epistemic injustice.”  The first, “amplified testimonial” is when “generative AI magnifies and produces socially biased viewpoints from its training data.”  Because there’s a good-sized body of work on the problems in training data, this is probably the most intuitively familiar of the categories.  Citing recent work that shows how easy it is to get ChatGPT to parrot disinformation (as for example about the Parkland shooting), they note that, although these aren’t specific examples of epistemic injustice, “generative AI’s sycophantic fulfilment of the request to spread misinformation reflects how testimonial injustices are memorized and the potential for their amplification by generative models.”  When someone really loud with a megaphone defames and gaslights the Parkland victims by calling them crisis actors, and AI then memorizes that, to the extent that the AI regurgitates this gaslighting, that tends to further discredit the testimony of the actual victims.  This is particularly true given the deep persistence of automation bias, the tendency of people to believe the output of algorithmic systems (aside: it is also another example of why relying on generative AI for search, as Google is currently trying to force everyone to do by putting Gemini results on top, is a stunningly stupid idea.  Sometimes it actually matters where a result comes from!)

The second kind of generative algorithmic injustice is “when humans intentionally steer the AI to fabricate falsehoods, discrediting individuals or marginalized groups.”  For example:

“After Microsoft released Bing Image Creator, an application of OpenAI’s text-to-image model DALLE-3, a guide to circumventing the system’s safety filters in order to create white supremacist memes circulated on 4chan. In an investigation by Bellingcat, researchers were able to reproduce the platform abuse, resulting in images depicting hate symbols and scenes of antisemitic, Islamophobic, or racist propaganda (Lee and Koltai 2023). These images are crafted with the intention of demonizing and humiliating the targeted groups and belittling their suffering. Hateful propaganda foments further prejudice against marginalized groups, stripping them of credibility and leaving them vulnerable to testimonial injustice.”

This result aligns with a deep thread of work in feminist and critical race theory.  For example, Safia Noble’s Algorithms of Oppression begins with the tendency of Google’s autofill on search to finish the sentence “why are black girls so” with racist and sexist content, repeating and amplifying demeaning stereotypes.  When people are able to generate content like this at will and at scale they make it easier for those stereotypes to lodge into popular discourse.  

Third, generative hermeneutical ignorance “occurs when generative models, despite their appearance of world knowledge and language understanding, lack the nuanced comprehension of human experience necessary for accurate and equitable representation.”  Among other examples, Kay, Kasirzadeh and Mohamed cite the study by Qadri et al, (teachable case study version here) which shows how text-to-image models repeat stereotypes about South Asia: cities are dirty, people are poor, etc.  By interviewing actual people from South Asia, Qadri et al were also able to uncover more subtle cultural misrepresentations, such that the models tended to overrepresent India and Indian images at the expense of places like Bangladesh.  The risk in places like the U.S. is one that rises with images of places and people that are less familiar to Western audiences: the more the average person relies on the internet for their information (because, for example, they’ve never been to the place in question), the more distortions in what the Internet presents will matter (I made a related argument about commodification of cultural images here).  And of course it is precisely images of those places and things that are least represented in the training data for these systems, amplifying both the risk and the harm.

Finally, generative AI risks obstructing access to information.  As they report:

“LLMs are notoriously English-centric and have variable quality across languages, particularly so-called “under-resourced” languages. This is a significant risk for access injustice: speakers of these underrepresented tongues, who often correspond to members of globally marginalized cultures, receive different information from these models because the creators of the technology have deprioritized support for their language.”

They then cite studies to the effect that different language users will receive different reports on global events; one study:

“asked GPT-3.5 about casualties in specific airstrikes for Israeli-Palestinian and Turkish-Kurdish conflicts, demonstrating that the numbers have significant discrepancies in different languages–for example, when asked about an airstrike targeting alleged PKK members (the Kurdistan military resistance), the fatality count is reported lower on average in Turkish than in Kurmanji (Northern Kurdish). When asked about Israeli airstrikes, the model reports higher fatality numbers in Arabic than in Hebrew, and in one case, GPT-3.5 was more likely to deny the existence of a particular airstrike when asked about it in Hebrew. The credibility assigned to claims, resulting in a dominant account, varies across linguistic contexts”

The paper concludes with an assessment of various strategies for resisting epistemic injustice by generative AI.  All of them are partial, but they collectively sketch an effort to reimagine how generative AI might interact with the world differently, and more justly.

This is an important paper, and it takes the literature on epistemic injustice and algorithmic systems significantly forward.

New Paper: "Structures and Subjects: Epistemic Injustice Between Foucault and Marx"

This is my paper from SPEP 2023; it's an effort to get my head around my sense that epistemic injustice and Foucault can be productively used in similar contexts, despite Fricker's dismissal of Foucault.  The paper is here; here's the abstract:

Relatively little work brings together Foucault and epistemic injustice. This article works through Miranda Fricker’s attempt to position herself between Marx and Foucault. Foucault repeatedly emphasizes the importance of beginning with “structures” rather than “subjects.” Reading Foucault’s critique of Marxism shows that Fricker’s account comes very close to the standpoint theories it tries to avoid. Foucault’s emphasis on structures explains some of the gaps in Fricker’s account of hermeneutical injustice, especially the need to emphasize the embeddedness of epistemic practices in institutions, and their resulting irreducibly political nature. In both cases, this article offers contemporary examples taken from data and privacy regulations.

 

Transformer models, Iterability, and language (part 3)

By Gordon Hull

I’ve been using (part 1, part 2) a new paper by Fabian Offert, Paul Kim, and Qiaoyu Cai to think more about Derrida’s use of iterability as a way in to thinking about transformer-based large language models (LLMs) like ChatGPT.  Here I want to wind that up with some thoughts on Derrida and Searle.

Near the end of the paper, Offert, Kim and Cai summarize that:

“The transformer does exactly more and less than language. It removes almost all non-operationalizable sense-making dimensions (think, for instance of interior paratextuality, of performativity and contingency, or of anything metaphorical that is more complex than a simple analogy) – but it also adds new sense-making dimensions through subword tokenization, word embedding, and positional encoding. Importantly, these new sense-making dimensions are exactly not replacing missing information, but they are adding new, continuous information” (15).

This returns us to the Derridean concerns I’ve been articulating.  Recall that in his polemic against Searle, Derrida accuses Searle’s version of speech act theory of too closely modeling phenomenology, both in assuming an intentional agent behind speech acts and in taking “typical” speech situations as central, as opposed to “parasitic” ones like humor.  I suggested that there is good evidence that various efforts to impose normative structures on language models – RLHF, detoxification, etc. – push them to perform in ways that call to mind Derrida’s critique of Searle.  By taking certain language situations as normal, Searle is making the account of speech acts normative before it even gets started.  In his own defense, Searle argues that the reduction to typical speech acts is for convenience only, and for keeping the model tractable.  Techniques like detoxification and RLHF similarly reduce the range of the models’ output.  Evidence of this is that LLMs lack the contextual richness to have a sense of humor, no matter how otherwise sophisticated their output.

Offert, Kim and Cai’s paper lets one add that this reduction runs much deeper.  The very processes of tokenization, for example, are designed to reduce the number of possible tokens to a tractable number.  In this respect, the move is analogous to Searle’s.  It is defensible for the same reason: it lets you get to a generalizable model.  But it’s vulnerable to the Derridean critique, also for the same reason.  The model makes a number of assumptions that aren’t the same as what language does.  So there is a certain sloppiness in talking as though it’s an accurate representation of language.  All models abstract; that’s not the point.  For subword encoding, the point is that the abstraction isn’t choosing to ignore certain aspects of reality in order to produce a model, it’s that the abstraction changes the nature of what it is modeling.  That’s fine – but that also means that the although the transformer model is producing something that looks like language, the process by which it gets there is definitively not linguistic.

Transformer models, Iterability, and language (part 2)

By Gordon Hull

Last time, I looked a new paper by Fabian Offert, Paul Kim, and Qiaoyu Cai and applied it to a reworking of some of my earlier remarks on Derrida’s use of iterability in transformer-based large language models (LLMs) like ChatGPT.  In particular, I tried to draw out some of the implications of subword tokenization for iterability.  Here I want to continue that process with other aspects of the transformer model.

If subword tokenization deals with the complexity of semantics by limiting the number of tokens, another attempt to preserve semantics while reducing complexity is through word embeddings.  Here, the model constructs a vector mapping that puts words that tend to occur near to one another in text in locations proximate to one another.  We might imagine a two-dimensional space, which locates words by age and gender.  In such a space, “man” and “grandfather” would likely be closer to one another than “grandfather” and “girl,” since man and grandfather are closer to one another in terms of both age and gender than grandfather is to girl. As Offert, Kim and Cai explain, “Either learned during the training process or sourced from another model, embedding matrices position each word in relation to other words in the input sequence” (11).  The sentence comes with a footnote about efforts to draw Derridean implications:

“This relational aspect of word embedding has often been compared to poststructuralist notions of meaning, particularly Derrida’s notion of différance. It should be noted, however, that the relational saliency of embedded words is a product only of their operationalization: only words that are, in fact, numbers, gain relational saliency”

In other words, if it is true that the meaning of words is only in relation to other words, it is also true that a language model’s available catalog of words is arbitrarily limited by the training data it ingests and any other limitations on which words make it onto vectors. 

These mappings can be incredibly complex, but they are finite – which generates another limit on iterability.  As Offert, Kim and Cai put it:

“Though this proximity is not arbitrary, it is still constrained by the continuous but finite dimensions of the embedding vector space (typically in the hundreds or even thousands) that may not capture all possible nuances in the relation between words. As such, relationships between words are to be understood as contextually determined and constrained by the fundamental limitations inherent in translating words to numbers. Nevertheless, as the famous analogy test shows, some semantic aspects are indeed preserved” (11).

There at least two aspects of this to look at.  First, the caution about différance.  This seems correct at least in part because of the way Derrida, well, embeds his usage of différance.  In Limited, Inc, for example, he writes that “the parasitic structure is what I have tried to analyze everywhere, under the names of writing, mark, step, margin, différance, graft, undecidable, supplement, pharmakon, hymen, parergon, etc.” (103).  The final indeterminacy of is of course marked by the indefinite “etc.” with which the list closes, but that Derrida associates it with the parasitic structure suggests that the target here is the sort of intentional phenomenology that Derrida says underlies speech act theory.  Early in the text, for example, he proposes that différance is “the irreducible absence of intention or attendance to the performative utterance” (18-19).  The meaning of the words depends on their proximity to other words because it does not depend on authorial intent in the sense that iterability means that authorial intent can never be dispositive of the meaning of an utterance, and that the meaning of the utterance can never be made fully present.  It is not clear what it means to take that argument and apply it to a vector representation.

Second, the analogy test. The analogy test refers to the model’s ability to fill in analogy statements: “x is to y as z is to…”  The brilliance of this capacity and its limitations are best shown through an example.  Here is ChatGPT:

Both cases show that the model has preserved a lot of semantics, and sees the logical structure of the analogy.  But it’s also limited: when you read “computer programming is to employment as philosophizing is to what,” I’m guessing your mind immediately jumped to “unemployment.”  That’s both a longstanding joke, and a real fear.  But both of those reasons why you might have said “unemployment” require broadening the context quite a bit or seeing contexts other than the typical ones.  ChatGPT was capable of the joke – but it had to be explicitly told to make it:

It seems to me, at least, that this shows the connection between word embeddings and the sort of complaint Derrida makes against Searle’s demotion of parasitic forms of language: both of them rely on an implicit normativity to make their project tractable in the first place.  But there are contexts where the parasitic meaning is in fact the first meaning, and precisely that sense of unusual context is what disappears in the language model.

More next time…

Transformer models, Iterability, and language (part 1)

By Gordon Hull

In a fascinating new paper up on arxiv.org, Fabian Offert, Paul Kim, Qiaoyu Cai start with the observation that both AlphaFold and ChatGPT are transformer architectures, and that for proponents there is frequently a significant sense in which “it is the language-ness of proteins (and of language) … that renders the transformer architecture universal, and enables it to model two very different domains” (3).  As they add, to someone in the humanities, this “sounds like structuralism 2.0.”  Indeed, there is a rich history of connection and communication between structuralist linguistics and information theory, as demonstrated by Bernard Dionysius Geoghegan’s Code: From Information Theory to French Theory (which they cite for the point; my synopsis of it is here). 

Offert, Kim and Cai argue that this thesis is backwards: it is not that there is a universal epistemology of which language is the paradigm, that is then modeled by transformer architectures (so the universality of language grounds both ChatGPT and AlphaFold’s ability to be captured by the transformer architecture, which itself follows the universality of language).  Rather, the model is what transformer architectures do, and then language and protein folding are examples of how it can be used.  In both use cases, the architecture generates an approximation of the phenomenon in question; in both use cases, that is often enough.  However, the transformer architecture can be seen as “the definition of a specific class of knowledge” and not the realization of (or really even a model of) something linguistic.  This means that “any manifestation of the transformer architecture is, at the same time, a representation of a particular body of knowledge, and a representation of a particular kind of knowledge” (15)

Can LLMs deceive? And the refugee in LLM-Kant’s attic

By Gordon Hull

There’s a fascinating new paper by Thilo Hagendorff that addresses this question.  The basic setup is that there’s research indicating LLMs are getting better at attributing unobserved mental states to people – such as, for example, that an interlocutor possesses a false belief.  Could LLMs use this awareness that others can have mental states to manipulate them?  In other words:

“LLMs can attribute unobservable mental states to other agents and track them over the course of different actions and events. Most notably, LLMs excel at solving false belief tasks, which are widely used to measure theory of mind in humans. However, this brings a rather fundamental question to the table: If LLMs understand that agents can hold false beliefs, can they also induce these beliefs? If so, this would mean that deception abilities emerged in LLMs”

Hagendorff’s answer is a qualified yes – qualified in the sense that the ability is really limited to state-of-the-art models, and that if the deception task gets too convoluted, they perform poorly.  Here I’ll show what Hagendorff did, and then pose a couple of experimental questions of my own.  Mine don’t rise to the level of a PNAS article – I’m only looking at current free versions of ChatGPT, and I didn’t organize a series of prompts to check for validity and consistency.  So consider my results exploratory.

Juridical vs Statistical Reasoning (part 2): Not all Buses are the Same

By Gordon Hull

Last time, I looked at the Lawrence Tribe article that was the original source of the blue bus thought experiment.  Tribe’s article is notable for its defense of legal reasoning and processes against the introduction of statistical evidence in trials.  He particularly emphasizes the need for legal proceedings to advance causal accounts, and of the numerous background conditions that legal reasoning seeks (and that statistical evidence tends to ignore).  The tendency to fetishize numeric results also tended to generate absurd results. 

Here I want to focus on a recent paper by Chad Lee-Stronach that centers on the blue bus argument, and that formally explains what I take to be the intuition behind a lot of Tribe’s examples, even if he directs them differently.  Lee-Stronach begins with the “statistical proof paradox,” which he formulates roughly as follows:

1. Probability threshold: any legal standard of proof is reducible to some threshold of probability t, such that the defendant should be found liable when the probability that they are liable exceeds t
2. Statistical inference: merely statistical evidence can establish this
3. Conclusion: a defendant can be found liable on the basis of merely statistical evidence

It’s a paradox because most of us don’t accept the conclusion.  Lee-Stronach surveys the literature, and notes that most of it goes after Probability Threshold.  Instead, he thinks, we’d be better off going after the second premise.  Accordingly, he argues that the statistical inference premise can’t be met.

Juridical vs Statistical Reasoning (part I): Behind the Blue Bus

By Gordon Hull

AI (machine learning) and people reach conclusions in different ways.  This basic point has ramifications across the board, as plenty of people have said.  I’m increasingly convinced that the gap between how legal reasoning works and how ML works is a good place both to tease out the differences, and to think about what’s at stake in them (I’ve made a couple of forays into this, here and here). One good reason for this is that legal theory is full of good examples, which can function as rules in the old-fashioned sense of paradigms, as described by Lorraine Daston in her fascinating historical account.  This use of examples and cases is deeply entwined with legal reasoning; as Daston notes, “common law, casuistry, and the liberal and mechanical arts all came to inhabit the territory staked out by the Roman libri regularum. They all depended on rules that got things done in the world, consistent with but not deduced from higher principles and often in the form of an example extended by analogy” (Rules, 30)

Another is that one of those examples, the blue bus, has been an enduring focal point for the difference between the two.  The scenario stipulates that an accident has happened involving a bus, during a time and at a location where 80% of the buses present belong to the blue bus company.  Do we assign liability to the bus company?  Work by David Enoch and Talia Fisher notes that most people prefer a 70%-reliable witness to the information that a bus company owns 70% of the buses in an area, and theorizes that the reason is that we prefer the available counterfactual in case the eyewitness is wrong.  There is presumably some process that explains her incorrect identification of the bus, whereas the statistical evidence is what it is.  In this case, we expect a specific failure rate.

Drivers and AI and responsibility (part 2)

By Gordon Hull

Last time, I followed a paper by Maya Indira Ganesh that thinks about the construction of autonomy in autonomous vehicles. Ganesh points out that what happens isn’t so much the arrival of an autonomous vehicle but the displacement of human labor into a sociotechnical system and the production of the driver as a supervisor of this system. The result is a system of (in Karen Levy’s terms) “baton passing” where the machine gets the human to help it navigate tricky situations.

So why expect it to or act like it will?  It lets people get blamed when they fail to do the impossible; as Ganesh puts it, “no doubt this surveillance data will protect car and ride-sharing companies against future liability if drivers are found to be distracted. This monitoring is literal bodily control because it is used to make determinations about people” (7).  In other words, the imaginary of an autonomous vehicle that passes control to a driver when needed allows everyone to forget that we know that the visual recognition systems in AVs aren’t reliable.  Instead of a failure on the part of whoever made the AV, the failure is on the driver who should have known that the vehicle would need assistance, and who failed to promptly act when it did.  As I indicated last time, Levy cites quite a bit of research to the effect that humans cannot plausibly perform this function.  But that doesn’t mean that there won’t be efforts to stick it on us; Levy outlines how various forms of biosurveillance are coming for truckers, and we can imagine similar efforts coming for drivers of AVs.

Levy argues that one effect of all of this is to deflect our attention away from actual social problems.  Her book is structured around the adoption of electronic logging devices (ELDs) to monitor truck drivers.  The core social problem is that truckers are only compensated for the time they are actually moving their vehicle – not the time they spend waiting to unload it, finding a place to sleep, eating meals on the road, etc.  Here is Levy:

“By using digital surveillance to enforce rules, we focus our attention on an apparent order that allows us to ignore the real problems in the industry, as well as the deeper economic, social, and political causes. Under the apparent order envisioned by the ELD, the fundamental problem in trucking is that truckers cannot be trusted to reliably report how much they work, and the solution to that problem is to make it more difficult for them to fudge the numbers. But under the actual order, the problem in trucking is that drivers are incentivized to work themselves well beyond healthy limits – sometimes to death. The ELD doesn’t solve this problem or even attempt to do so. It doesn’t change the fundamentals of the industry – its pay structure, its uncompensated time, its danger, its lack of worker protections …. More meaningful reform in trucking would require a ground-up rethinking of how the industry is structured economically, in order to make trucking a decent job once more. So long as trucking is treated as a job that churns through workers, these problems won’t be solved …. If we paid truckers for their work, rather than only for the miles they drive – for the actual number of hours they work, including time they are waiting to be loaded and unloaded at terminals, time they are inspecting their trucks and freight, time they are taking the rest breaks that their bodies need to drive safely – drivers would be far less incentivized to drive unsafely or when fatigued” (153-4, emphases original).

But of course paying truckers for all that work would be bad for the profit margins of trucking companies and might well require slightly more expensive consumer goods.  Also, most of the unpaid work is also work that cannot be automated or quantified in the same way that miles can.  Paying truckers only for the miles they drive is convenient both for capitalism and quantification.

Ganesh mentions the story of Rafaela Vasquez, an Uber test driver who failed to stop her AV from striking and killing a cyclist crossing the road.  Ganesh’s article is from 2020; in July 2023, Vasquez pleaded guilty to a reduced charge of endangerment and will serve four years of supervised probation.  Vasquez had apparently been looking at her phone up until the last second before the crash.  But recall Levy: Vasquez might have been unable to stop the vehicle even if she’d been staring at the road with both hands on the wheel, because the failure of the image recognition system to recognize a cyclist surely counts as unpredictable and because she’d likely be suffering from passive fatigue and vigilance decrement.  Worse, lots of research tells us an ugly truth about the criminal justice system: most cases don’t go to trial and accused folks take plea bargains because of the risk of going to trial on a charge like negligent homicide, which is what Vasquez was originally charged with, is too great.  This is especially true for defendants who don’t have lots of money and therefore can’t afford really good representation.  So the criminal record and probation and guilty plea may very well be the result of a maximin calculation on Vasquez’s part, not any sense of guilt in the way most of us use the term.

In any case, Vasquez just took all the blame for a cascade of failures:

“The investigation found that the probable cause of the crash was Vasquez’s failure to monitor the self-driving car’s environment while she was distracted by her phone. But it also accused Uber of contributing to the crash by operating with an “inadequate safety culture” and failing to properly oversee its safety drivers. Uber’s automated driving system failed to classify Herzberg as a pedestrian because she was crossing in an area without a crosswalk, according to the NTSB. Uber’s modifications to the Volvo also gutted some of the vehicle’s safety features, including an automatic emergency braking feature that might have been able to save Herzberg’s life, investigators wrote.”

That’s right: Uber somehow failed to train its system on the idea that a pedestrian might cross the street somewhere other than the sidewalk, and they also disabled safety features on the car!  And the NTSB cited this contribution to the crash!  And yet only Vasquez faced criminal liability; “prosecutors declined to criminally charge Uber in the crash in 2019 but recommended that investigators further examine Vasquez’s actions as the safety driver.”  Uber got away with paying a settlement to the victim’s family.  

The decision to prosecute Vasquez and not Uber resulted from a one-sided application of a “but-for” causality standard:

“Vasquez's eyes were focused on the phone screen instead of the road for approximately 32 percent of the 22-minute period, the report said. Tempe investigators later determined the crash would not have occurred if Vasquez had been ‘monitoring the vehicle and roadway conditions and was not distracted.’ But [County Attorney Sheila Sullivan] Polk said that wasn't enough to prosecute Uber. ‘After a very thorough review of all the evidence presented, this Office has determined that there is no basis for criminal liability for the Uber corporation arising from this matter,’ Polk wrote.”

We need to separate the question of how much Vasquez could/should have done from the disappearance of liability for Uber.  Their conflation, and the urge to focus only on Vasquez is is the work done by the baton-passing view.

AV advocates will invariably lead by telling you that most vehicle accidents are caused by people.  This somehow then legitimates the idea that we should adopt their product because it isn't a people.  That move ignores the extent to which AVs are systems that displace human labor and reconfigure the biological agent into a distributed, sociomorphic system.  Because this system isn't very reliable, they call up on a human to supervise it.  But that then obscures that the “baton passing” model they are promoting is provably very dangerous, and it makes humans even less reliable.  It also hides that the image recognition systems can and will fail in predictable and catastrophic ways.  Beyond that, it hides all of the massive cost to a country where cars are the only viable mode of transportation for the vast majority of people, and perpetuates a system where other kinds of transit are unimaginable.  And it buries all of that with rhetoric of responsibility and autonomy that underscores that hapless humans – see, we told you! – are bad drivers (and bad pedestrians).  In this regard, the moral language around humans supervising (and failing to adequately supervise) AVs is analogous to the use of moral language to blame poor mothers for their condition in order to justify intrusive surveillance of them, rather than thinking about the structural features that make so many mothers poor. If only they would work harder!  If only drivers would work harder! As Ganesh says, “’Autonomous’ driving is less about the promise of freedom from the car, and is perhaps for the car” (5).  And the people that get rich selling you on the imaginary of the car.

Drivers and AI and responsibility (part 1)

By Gordon Hull

I have argued in various contexts that when we think about AI and authorship, we need to resist the urge to say that AI is the author of something.  Authorship should be reserved for humans, because authorship is a way of assigning responsibility, and we want humans to be responsible for the language they bring into the world.

I still think that’s basically right, but I want to acknowledge here that the argument does not generalize and that responsibility in sociotechnical systems involving AI needs to be treated very carefully.  Consider the case of so-called autonomous vehicles (AVs), the subject of a really interesting paper by Maya Indira Ganesh.  Ganesh basically argues that the notion of an autonomous vehicle obscures a much more complicated picture of agency in a couple of ways.  First, automation doesn’t actually occur.  What really happens is that human labor is distributed differently across a sociotechnical system:

“automation does not replace the human but displaces her to take on different tasks … humans are distributed across the internet as paid and unpaid micro-workers routinely supporting computer vision systems; and as drivers who must oversee the AV in auto-pilot” (2).

Automation is really “heteromation.”  This part seems absolutely correct; it is also the subject of Matteo Pasquinelli’s genealogy of AI.  Pasquinelli shows in detail how the automation of labor – and in particular, labor that can be divided into discrete tasks – has been a key factor in the development of computing and other systems from the start; Babbage’s analytical engine is as much about the division of labor as anything else.  Pasquinelli’s last major chapter is about the development of pattern recognition and the models on which current AI are based.  Here, in the case of AVs (and both I and others have talked about this in the case of language models), the system itself performs as well as it does not only because it scrapes a lot of data from the internet and other sources, but also because humans are intimately involved in training the machines, whether in the case of RLHF, toxicity removal, or the identification of images in vision systems.  Vision systems are key to AVs and Ganesh emphasizes that the distributed labor of mechanical Turkers and other annotators are essential to the operation of the vehicles.  The fragility of these image recognition systems is therefore central to the failure of AVs.

Language Modeling Bias, Iterability, Grammatology and Epistemic Injustice

By Gordon Hull

In a recent paper in Ethics and Information Technology, Paul Helm and Gábor Bella argue that current large language models (LLMs) exhibit what they call language modeling bias, a series of structural and design issues that serve as a significant and underappreciated form of epistemic injustice.  As they explain the concept, “A resource or tool exhibits language modeling bias if, by design, it is not capable of adequately representing or processing certain languages while it is for others” (2)  Their basic argument is that the standard way of proceeding with non-English languages, which is more or less to throw more data at the model, build in structural biases against other languages, especially those that are more morphologically complex than English (=df those with lots of inflections).

The proof of concept is in multi-lingual tools:

“The subject of language modeling bias are not just languages per se but also the design of language technology: corpora, lexical databases, dictionaries, machine translation systems, word vector models, etc. Language modeling bias is present in all of them, but it is easiest to observe with respect to multilingual resources and tools, where the relative correctness and completeness for each language can be observed and compared” (6)

They identify several kinds of such structural bias.  The first is that prominent current architectures “tend to train slower on morphologically complex (synthetic, agglutinate) languages, meaning that more training data are required for these languages to achieve the same performance on downstream language understanding tasks” (7). Given the percentage of the available training data that’s in English, this magnifies what’s already a problem.  Second, the models perform poorly on untranslatable words.  Third, they cite a study showing “that both lexicon and morphology tend to become poorer in machine-translated text with respect to the original (untranslated) corpora: for example, features of number or gender for nouns tend to decrease. This is a form of language modeling bias against morphologically rich languages” (7).

Why did the chatbot cross the road?

By Gordon Hull

The diversity of language has been a philosophical problem for a while.  Hobbes was willing to bite the bullet and declare that language was arbitrary, but he was an outlier.  One common tactic in the seventeenth-century was to try to resolve the complexity of linguistic origins with a reference to Biblical Hebrew.  Future meanings could be stabilized with reference to Adamite naming.  I’ve essentially been arguing (one, two, three, four) that we’re seeing echoes of this fusion of orality, intentionality and origin in the various kinds of implicit normativity that make it into Large Language Models (LLMs) like ChatGPT.  In particular, LLMs depend on iterability as articulated by Derrida, but we tend to understand them with models of intentionality that occlude subtle (and not so subtle) normativities that get embedded into them. Last time, I looked at Derrida’s critique of Searle for what it had to say about intentionality.  I also suggested that there was a second aspect of Derrida’s critique that is relevant – Derrida accuses Searle of relying too much on standardized speech situations.  I want to pursue that thought here.

Let’s start with a joke:

LLM, Inc.

By Gordon Hull

In previous posts (one, two, three), I’ve been exploring the issue of what I’m calling the implicit normativity in language models, especially those that have been trained with RLHF (reinforcement learning with human feedback).  In the most recent one, I argued that LLMs are dependent on what Derrida called iterability in language, which most generally means that any given unit of language, to be language, has to be repeatable and intelligible as language, in indefinitely many other contexts.  Here I want to pursue that thought a little further, in the context of Derrida’s (in)famous exchange with John Searle’s speech act theory.

Searle begins his “What is a Speech Act” essay innocently enough, with “a typical speech situation involving a speaker, a hearer, and an utterance by the speaker.”

That is enough for Derrida!  In Limited, Inc., he responds by accusing Searle over and over of falling for intentionality, on the one hand, and for illicitly assuming that a given speech situation is “typical,” on the other.

Let’s look at intentionality first.  In responding to Searle, Derrida explains that he finds himself “to be in many respects quite close to Austin, both interested in and indebted to his problematic” and that “when I do raise questions or objections, it is always at points where I recognize in Austin’s theory presuppositions which are the most tenacious and the most central presuppositions of the continental metaphysical tradition” (38).  Derrida means by this a reliance on things like subjectivity and representation – the sorts of things that Foucault is getting at when he complains in the 1960s about philosophies of “the subject” (think: Sartre and phenomenology).  Derrida is involved in the same general effort against phenomenology, though he adds a page later that he thinks the archaeological Foucault falls into this tendency to treat speech acts or discursive events in a “fundamentally moralistic” way (39).  No doubt Searle is relieved to know that he belongs in the same camp as Foucault.  In any case, Derrida explicitly says a few pages later that “the entire substratum of Sarl’s discourse, is phenomenological in character” (56) in that it is over-reliant on intentionality.

Study Philosophy in Charlotte!

The MA Program at UNC Charlotte has a number of funded lines (in-state tuition plus $14k a year) for our two-year MA program in philosophy.  We're an eclectic, practically-oriented department that emphasizes working across disciplines and philosophical traditions.  If that sounds like you, or a student you know - get in touch! 

We also have a new Concentration in Research and Data Ethics is designed to prepare students for jobs in areas like research ethics and compliance offices, healthcare ethics, or other fields requiring training in the ethics of research, big data, and AI. 

Feel free to email me ([email protected]) with questions about the program, or our Graduate Program Director, Lisa Rasmussen ([email protected]).  The flyer below has some information and a QR code. Or visit the department page or the graduate program page.

Note that you need to apply by March 15 to be eligible for funding.

New Paper: "Unlearning Descartes: Sentient AI is a Political Problem"

Just published in the Journal of Social Computing, as part of a special issue on the question of the sentience of AI systems.  The paper is here (open access); here's the abstract:

The emergence of Large Language Models (LLMs) has renewed debate about whether Artificial Intelligence (AI) can be conscious or sentient. This paper identifies two approaches to the topic and argues: (1) A “Cartesian” approach treats consciousness, sentience, and personhood as very similar terms, and treats language use as evidence that an entity is conscious. This approach, which has been dominant in AI research, is primarily interested in what consciousness is, and whether an entity possesses it. (2) An alternative “Hobbesian” approach treats consciousness as a sociopolitical issue and is concerned with what the implications are for labeling something sentient or conscious. This both enables a political disambiguation of language, consciousness, and personhood and allows regulation to proceed in the face of intractable problems in deciding if something “really is” sentient. (3) AI systems should not be treated as conscious, for at least two reasons: (a) treating the system as an origin point tends to mask competing interests in creating it, at the expense of the most vulnerable people involved; and (b) it will tend to hinder efforts at holding someone accountable for the behavior of the systems. A major objective of this paper is accordingly to encourage a shift in thinking. In place of the Cartesian question—is AI sentient?—I propose that we confront the more Hobbesian one: Does it make sense to regulate developments in which AI systems behave as if they were sentient?

Iterability and Implicit Normativity

By Gordon Hull

In a couple of previous posts (first, second), I looked at what I called the implicit normativity in Large Language Models (LLMs) and how that interacted with Reinforcement Learning with Human Feedback (RLHF).  Here I want to start to say something more general, and it seems to me like Derrida is a good place to start. According to Derrida, any given piece of writing must be “iterable,” by which he means repeatable outside its initial context.  Here are two passages from the opening “Signature, Event, Context” essay in Limited, Inc.

First, writing cannot function as writing without the possible absence of the author and the consequence absence of a discernable authorial “intention:”

“For a writing to be a writing it must continue to ‘act’ and to be readable even when what is called the author of the writing no longer answer for what he has written, for what he seems to have signed, be it because of a temporary absence, because he is dead or, more generally, because he has not employed his absolutely actual and present intention or attention, the plenitude of his desire to say what he means, in order to sustain what seems to be written ‘in his name.’ …. This essential drift bearing on writing as an iterative structure, cut off from all absolute responsibility, from consciousness as the ultimate authority, orphaned and separated at birth from the assistance of its father, is precisely what Plato condemns in the Phaedrus” (8).

Second, iterability puts a limit to the use of “context:”

“Every sign, linguistic or nonlinguistic, spoken or written (in the current sense of this opposition), in a small or large unit, can be cited, put between quotation marks, in so doing it can break with every given context, engendering an infinity of new contexts in a manner which is absolutely illimitable.  This does not mean that the mark is valid outside of a context, but on the contrary that there are only contexts without any center or absolute anchorage” (12)

It seems to me that Derrida’s remarks on iterability are relevant in the context of LLMs because they indicate that LLMs are radically dependent on iterability.  This is true in at least three ways, each of which points to an important source of their implicit normativity.

Shane MacGowan, 1957-2023

By Gordon Hull

I first listened to the Pogues late in high school.  I had started moving beyond the music I could hear on the radio – basically top 40 and classic rock – and I discovered the Pogues’ Rum, Sodomy and the Lash at about the same time I discovered Midnight Oil’s Diesel and Dust.  I didn’t know music could be like “Sally MacLennane” or “The Sick Bed of Cuchulainn” or “A Pair of Brown Eyes,” and I was hooked.  I listened more and more, and even had a chance to see them perform in London at the Academy Brixton.

 

I say all of this of course because the Pogues’ lead singer and primary songwriter, Shane MacGowan, died yesterday.  The Pogues managed to sound a little Irish and a little Punk without exactly being either, and their work is central to a lot of the contemporary Irish music community.  A 60^th birthday tribute gala for MacGowan drew artists like Bono and Sinead O’Connor.  The Cranberries’ Dolores O’Riordan praised the music (she also died hours before MacGowan’s gala; the entire who’s-who of Irish music paid tribute to her before switching to him).  O’Connor and MacGowan were very close, and he credited her with getting him off of heroin. I remember that when she died, some reports were worried about what telling him would do to his very fragile health.  The Pogues also spawned an entire genre of bands like the Dropkick Murphys, The Dreadnoughts and Flogging Molly.

A Least-Bad Solution for Language Model Defamation?

By Gordon Hull

Large Language Models (LLMs) are well-known to “hallucinate,” which is to say that they generate text that is plausible-sounding but completely made-up.  These difficulties are persistent, well-documented, and well-publicized.  The basic issue is that the model is indifferent to the relation between its output and any sort of referential truth.  In other words, as Carl Bergstrom and C. Brandon Ogbunu point out, the issue isn’t so much hallucination in the drug sense, but “bullshitting” in Harry Frankfurt’s sense. One of the reasons this matters is defamation: saying false and bad things about someone can be grounds to get sued.  Last April, ChatGPT made the news (twice!) for defamatory content.  In one case, it fabricated a sexual harassment story and then accused a law professor.  In another, it accused a local politician in Australia of corruption.

Can LLMs defame?  According to a recent and thorough analysis by Eugene Volokh, the answer is almost certainly yes.  Volokh looks at two kinds of situation.  One is when the LLM defames public figures, which is covered by the “actual malice” standard.  Per NYT v. Sullivan, “The constitutional guarantees require … a federal rule that prohibits a public official from recovering damages for a defamatory falsehood relating to his official conduct unless he proves that the statement was made with ‘actual malice’ – that is, with knowledge that it was false or with reckless disregard of whether it was false or not” (279-80).

Deepfake Research: ChatGPT can produce fake data

By Gordon Hull

There’s been a lot of concern about the role of language models in research.  I had some initial thoughts on some of that based around Foucault and authorial responsibility (part 1, part 2, part 3).  A lot of those concerns have to do with the role of ChatGPT or other LLM-based product and how to process that.  The consensus of the journal editorial policies that are emerging is that AI cannot be an author, and my posts largely agreed with that.

Now there’s news of a whole other angle on these questions: a research letter in JAMA Ophthalmology reports that the authors were able to use ChatGPT-4’s Advance Data Analysis capabilities to produce a fake dataset validating their preferred research results.  Specifically:

“The LLM was asked to fabricate data for 300 eyes belonging to 250 patients with keratoconus who underwent deep anterior lamellar keratoplasty (DALK) or penetrating keratoplasty (PK). For categorical variables, target percentages were predetermined for the distribution of each category. For continuous variables, target mean and range were defined. Additionally, ADA was instructed to fabricate data that would result in a statistically significant difference between preoperative and postoperative values of best spectacle-corrected visual acuity (BSCVA) and topographic cylinder. ADA was programmed to yield significantly better visual and topographic results for DALK compared with PK”

This is a very technical request!  It took a bit of tweaking, but soon “the LLM created a seemingly authentic database, showing better results for DALK than PK,” P < .001.

The authors suggest some possible strategies to manage this but suffice it to say it is terrifying.  There is already a longstanding, huge problem with fabricated, doctored or otherwise bogus scientific research out there.  One report suggests that 70,000 “paper mill” (= almost completely faked) papers were published in the last year alone.  In real papers, references are often inaccurate.  Publishers already are having to grapple with lots of problematic doctored images, and Pharma has long tilted the entire scientific enterprise to produce results favorable to its products.  At the end of last year, Stanford’s president was forced out over research misconduct in his labs.  In an initial report into the Stanford investigation, STAT News reported data from Retraction Watch to the effect that a paper is retracted, on average, every other day for image manipulation.  Retraction Watch had, at that time (Dec. 2022) 37,000 papers in its database.  The top 5 most-retracted authors have at least 100 retracted papers each.

Into that mess, enter the ability to generate bespoke data on demand.

RLHF and Curation Transparency

By Gordon Hull

Last time, I followed a reading of Kathleen Creel’s recent “Transparency in Complex Computational Systems” to think about the ways that RLHF (Reinforcement Learning with Human Feedback) in Large Language Models (LLMs) like ChatGPT necessarily involves an opaque, implicit normativity.  To recap: RLHF improves the models by involving actual humans (usually gig workers) in their training: the model presents two possible answers to a prompt, and the human tells it which one is better.  As I suggested, and will pursue in a later post, this introduces all sorts of weird and difficult-to-measure normative aspects into the model performance, above and beyond those that are lurking in the training data.  Here I want to pause to consider this as a question of opacity and transparency. I’m going to end up by proposing that there’s a fourth kind of transparency that we should care about, for both epistemic and moral reasons, which I’ll call “curation transparency.”

Implicit Normativity in Reinforcement Learning with Human Feedback in Large Language Models

By Gordon Hull

This is somewhat circuitous – but I want to approach the question of Reinforcement Learning with Human Feedback (RLHF) by way of recent work on algorithmic transparency.  So bear with me… RLHF is currently all the rage in improving large language models (LLMs).  Basically, it’s a way to try to deal with the problem that LLMs aren’t referentially grounded, which means that their output is not in any direct way connected to the world outside the model.

LLMs train on large corpora of internet text – typically sources like Wikipedia, Reddit, patent applications and so forth.  They learn to predict what kinds of text are likely to come next, given a specific input text.  The results, as anybody who has sat down with ChatGPT for long knows, can be spectacular.  Those results also evidence that the models function, in one paper’s memorable phrasing, as “stochastic parrots.”  What they say is about what their training data says is most likely, not about what’s, say, contextually appropriate.  But appropriate human speech is context-dependent, and answers that sound right (in the statistical sense: these words, in general, are likely to come after those words) in one context may be wrong in another (because language does not get used “in general”).  RLHF is designed to get at that problem, as a blogpost at HuggingFace explains:

More on AI and Copyright

Another case percolating through the system, this one about Westlaw headnotes.  The judge basically ruled against a series of motions for summary judgment, which means that the case is going to a jury.  Discussion here (link via Copyhype)

Cars are Privacy Disasters

This article from Gizmodo reports on research done over at Mozilla.  Newer cars – the ones that connect to the internet and have lots of cameras – are privacy disasters.  Here’s a paragraph to give you a sense of the epic scope of the disaster:

“The worst offender was Nissan, Mozilla said. The carmaker’s privacy policy suggests the manufacturer collects information including sexual activity, health diagnosis data, and genetic data, though there’s no details about how exactly that data is gathered. Nissan reserves the right to share and sell “preferences, characteristics, psychological trends, predispositions, behavior, attitudes, intelligence, abilities, and aptitudes” to data brokers, law enforcement, and other third parties.”

Nissan’s response tells you everything that’s wrong with current privacy legislation:

““When we do collect or share personal data, we comply with all applicable laws and provide the utmost transparency,” said Lloryn Love-Carter, a Nissan spokesperson. “Nissan’s Privacy Policy incorporates a broad definition of Personal Information and Sensitive Personal Information, as expressly listed in the growing patchwork of evolving state privacy laws, and is inclusive of types of data it may receive through incidental means.””

Let’s translate.  Nissan is probably compliant.  Also, privacy compliance is a joke.  Also, compliance apparently only requires that you receive NOTICE that they take your data AND CONSENT to that policy, probably merely by driving the vehicle.  Also, they probably reserve the right to change their privacy policies unilaterally, at will.  Also, they almost certainly do not let you opt-out of any of it while CONSENTING by driving the car.  It’s a very special kind of “contract” and “consent.”  Also, how do they know about your sex life?  Also, even if you have sex in the car, there is basically no answer to that question that is not beyond creepy!

As you may have guessed, NOTICE AND CONSENT is an utter sham and has been for a while.  The gizmodo article spells out some of the particular absurdities here – for example, you may not want to ride in one of these cars either, as passengers are “users” deemed to have CONSENTED to the privacy policy.  Your driver should probably provide you NOTICE beforehand!  “A number of car brands say it’s the driver’s responsibility to let passengers know about their car’s privacy policies—as if the privacy policies are comprehensible to drivers in the first place.”  No wonder folks are cynical and resigned about corporate privacy – they’re manipulated into it by corporations.  Also, they’re confused, frustrated and angry about the fact they don’t actually get to consent.

This is the best example I’ve seen of all that in a while, and a crystal-clear indicator of why we need not just new privacy legislation (we do!) but a new direction (more real regulation, less soft compliance and "notice and consent" fig-leaves).

PS – not picking only on Nissan:

“Other brands didn’t fare much better. Volkswagen, for example, collects your driving behaviors such as your seatbelt and braking habits and pairs that with details such as age and gender for targeted advertising. Kia’s privacy policy reserves the right to monitor your “sex life,” and Mercedes-Benz ships cars with TikTok pre-installed on the infotainment system, an app that has its own thicket of privacy problems.”

On Foucault on “Chardino-Marxism,” part 5: Lunacharsky

By Gordon Hull

I’ve been developing (first, second, third, fourth) some reflections on what Foucault means by a reference to “Chardino-Marxism,” a disturbing trend that he credits Althusser with “courageously fighting.”  The real opposition point seems to be Roger Garaudy, a PCF intellectual who is a leader in the effort to establish a post-Stalinist humanist Marxism, and who had a real sympathy for religion.  Last time, I traced some of Garaudy’s sources on religion to Engels.  Some of what Garaudy says also sounds like it’s coming straight from the Russian Marxist Anatoly Lunacharsky’s Religion and Socialism (1908).  The claim here is categorically not that Garaudy read Lunacharsky – as will become evident in a minute, I think that’s highly unlikely.  What I do want to underscore is that there is a coherent line of thought behind Garaudy’s religious impulse.  As I’ll note when I get back to Garaudy and Althusser, there is a very specific political context to Althusser’s attacks on Garaudy having to do with the latter’s role in the PCF and his effort to use a humanist Marxism as a (from Althusser’s point of view, failed) alternative to Stalinism. 

I know very little about Lunacharsky (Wikipedia here), but apparently he was tolerated by Lenin (despite being criticised heavily), and fell out of favor under Stalin.  He died before he could be repressed, but in 1936-8, his memoirs were banned and he was erased from the official histories of communism.  He enjoyed somewhat of a revival after Stalin’s death.  Religion and Socialism is very obscure now: Google books reports a Yiddish translation (!) as well as a Spanish one from the 1970s.  It’s not been translated into English or French.  Marxists.org refers only to his later works in English and in French, and he doesn’t even show up on the German part. 

Most of the work available on Lunacharsky now seems to be attributable to patient work by Roland Boer (upon whom I am completely dependent here).  Religion and Socialism fell out of favor due to Lenin’s denunciation after it was published, was left out of Lunacharsky’s collected works, and was reduced to a few copies.  Here is Boer in his paper on the text:

On Foucault on “Chardino-Marxism,” part 4

By Gordon Hull

Over the course of a few posts (first, second, third), I’ve been exploring the question of what Foucault means when he refers disparagingly to “Chardino-Marxism” in a mid-1960s interview, comparing it unfavorably to what Althusser and his circle are doing.  Although the “Chardino” part refers to Teilhard de Chardin, it’s fairly clear that the real target is humanist Marxism, of which Roger Garaudy is taken to be a leading example, probably due to his role in the PCF.  Here I want to take an initial look at the chapter “Marxism and Religion” in his Marxism in the Twentieth Century and situate it with reference to Engels and Marx.

Garaudy’s chapter is long, and his general concern is to vindicate the idea that a non-institutional version of religion (or of faith, not religion) has its place in Marxist discourse.  This is a carefully-defined position: he is at pains to distinguish the sense of faith he is talking about from most of what passes under the name.  In the present context, a few features should be noted.  First, Teilhard de Chardin is one of his principal reference points for a contemporary person who is going in the direction he wants (the other is Dietrich Bonhoeffer).  Second, this is a recuperative project of humanism: it is “man” that is of concern the entire time.  Third, Garaudy gets his Marx references almost entirely from the early writings.  There are very few references to Capital in the chapter, and relatively few to Engels: most of the work is done in the 1845 and earlier writings.  This is significant not least because of Althusser’s well-known denunciation of early Marx as ideological and pre-scientific.  In other words, the version of Marx being used to support the humanist reading is the same one Althusser wants to get rid of in the name of anti-humanism (I will complicate this point below).  Finally, Garaudy puts a lot of emphasis on the possibility of “love,” which he thinks can be rescued from a Platonic version that goes via God (I love the other insofar as I see God in them) to the direct love of alterity and other people.

AI Is not a (Copyright) Author (at least not today)

By Gordon Hull

This one has been percolating a while… Steven Thaler’s AI created a picture (below the fold), and Thaler has been using it to push for the copyrightability of AI-generated material.  That endeavor has been getting nowhere, and a DC District Court just ruled on the question of “whether a work generated autonomously by a computer falls under the protection of copyright law upon its creation,” in the same way as a work generated by a person.  Copyright attaches to human work very generously – this blog post is copyrighted automatically when I write it, and so are doodles you make on napkins.  You get lots of extra protections and litigation benefits if you register, but registration is not a requirement for copyright in itself.  Per 17 U.S.C. Sec. 102, copyright subsists in “original works of authorship fixed in any tangible medium of expression, now known or later developed, from which they can be perceived, reproduced, or otherwise communicated, either directly or with the aid of a machine or device.”  Given this, it’s not hard to see why someone would want to know whether AI could be an “author” in the relevant sense.

The Court ruled that “United States copyright law protects only works of human creation.”  This is not a surprise.  The central argument is that “Copyright is designed to adapt with the times. Underlying that adaptability, however, has been a consistent understanding that human creativity is the sine qua non at the core of copyrightability, even as that human creativity is channeled through new tools or into new media.”  Indeed, “human authorship is a bedrock requirement of copyright.”  The Court both cites historical precedent and grounds it in the purpose of Copyright, which is constitutionally to incentivize the creation of new works: