New APPS

The judgments of algorithmic systems are often difficult to understand.  The system says you do not qualify for this loan – sorry!  You no longer qualify for x benefit – sorry! Why? Well, that’s just what it says.  And it’s impossibly opaque, for a variety of reasons – the underlying code and data are trade secrets, the decision is made by a neural network with millions of parameters, and so forth.  But suppose you object to the decision, or want to know how to do better?  This is the domain of, among other things, explainable AI (XAI).  XAI is normatively complicated, and I don’t want to get into that debate here.  Here I want to briefly look at a logically prior question: before we get to XAI, is there some sort of a right an explanation?  Is there a moral or legal sense in which the recipients of adverse algorithmic decisions are entitled to some sort of explanation of the basis for that decision in terms that they would understand and either find legitimate or be able to argue against? The particular point I want to make is that we need to think about the abstract right to explanation with reference to the reality of the U.S. legal system.

In a recent paper, Brett Karlan and Henrik Kugelberg argue against the need for a right to an explanation of AI decisions.  They take particular issue with an earlier paper by Kate Vredenburgh, which had argued for a right to an explanation on the grounds that it is necessary to fight back against algorithmic governance.  For the same reasons that the judge or bureaucrat owes you the reason for an adverse decision, so does the judge or bureaucrat using AI.  Using AI shouldn’t relieve them of the responsibility to inform people of the process that was used and how. She explains that “decision-makers are required to provide individuals with rule-based normative explanations and rule-based causal explanations … such explanations are necessary to enable individuals to engage in informed self-advocacy and are tolerably costly.” Normative explanations should say what the rules are and the normative reasons in favor of them; causal explanations are important for agency – they explain “what an agent would have to do to get a desired outcome, in terms of the relevant rules and robust population-level causal generalizations.”

(more…)

Forthcoming in Philosophy and Social Criticism; preprint here on SSRN. Here is the abstract:

“Algorithmic governance is sometimes compared to Hobbes’s Leviathan. Here I argue that, while algorithmic governance shares some similarities with the Hobbesian schema, it goes further in its suppression of contestation. To show this, I read Hobbes against Rancière’s reduction of him to a theorist of consensus to make two basic points. First, it is important for Hobbes that laws be public and rationally comprehensible. By contrast, algorithmic governance is notoriously opaque. Second, Hobbes also retains a sense of equity as an appeal against universally applied legal decisions, allowing decisions to be tailored to individual cases. Algorithmic governance does not typically involve equity-based appeal, a point that is especially clear in the context of bureaucratic governance structures, where algorithmic systems generate results that cannot be either understood or appealed. The result in Rancière’s terms is that classificatory, algorithmic systems are even more powerful agents of depoliticization than Leviathan.”

And here’s a little more detail:

“Part 2 develops an account of equity as a historic principle of law.  Part 3 is about Hobbes and focuses on the extent to which he emphasizes the public nature of law and retains an Aristotelian view of equity.  Part 4 turns to algorithmic governance.  I begin with fundamental epistemic differences between Hobbesian and algorithmic thought and outline three fundamental implications of them.  First, the sense in which algorithmic governance is inescapable is different from that in which the Leviathan is inescapable because it is much harder to contest algorithmic decisions.  Second, algorithms are less public because of their opacity.  Third, algorithmic efficiency undermines equitable implementation of policy by shifting the decisionmaking of street-level bureaucrats toward algorithms and away from the exercise of human judgment.  Collectively, these show how the integration of algorithmic processes in government effectively erodes the rule of law features that were prominent even in Hobbes.  Section 5 offers a brief conclusion.  I return to Rancière’s contention that Hobbesian governance produces an outside of those who are equal insofar as they have no part in the Leviathan-state.  Insofar as algorithmic governance is classificatory, and insofar as its classificatory regimes proliferate indefinitely, even for the same person, the classificatory process works against this final vestige of equality and political contestation”

(the parts are named rather than numbered in the final version; this is just a preprint)

Citing a paper by Lisanne Bainbridge from the early 1980s, Carl Hendrick describes a paradox of automation.  Those who automate systems tend to view people as the weak link, and thus replace humans with automation wherever possible.  This leaves a problem:

“The designer who tries to eliminate the operator still leaves the operator to do the tasks which the designer cannot think how to automate. What remains after automation is not a simplified role but an arbitrary residue of the most demanding, most ambiguous, and least supported work in the entire system. The human is not replaced. In other words, the human is paradoxically left with the hardest parts, and given almost no preparation for them”

As he immediately notes:

“Forty years on, Bainbridge’s paper reads less like a historical document than a prophecy. It describes, with an accuracy that might trouble us, exactly what is now happening to knowledge workers across every sector in which AI has taken hold. And it raises, with particular accuracy, a question that education has barely begun to confront.”

Here, I want to pursue the thought that AI changes the architectural constraints on cognitive labor.  I’ll explain what that means after more from Hendrick.

(more…)

In the context of LLMs, alignment means, more or less, that the models give answers either that we find suitable or that are suited to the task. A model that is misaligned behaves in inappropriate ways.  For example, when a mental health chatbot tells someone to kill themselves, that’s misalignment.  Sycophancy is a more subtle form.

New research led by Jan Betley and published in Nature last week discusses examples of what the authors call “emergent misalignment” (there’s an interesting write-up about it in the NYT here).  Fine-tuning is when you give a model additional training data to make it better at a given task.  For the study, they fine-tuned the model on insecure code.  That caused to become generally misaligned.  They explain:

“Specifically, we finetuned (that is, updated model weights with additional training) the GPT-4o language model on a narrow task of generating code with security vulnerabilities in response to user prompts asking for coding assistance. Our finetuning dataset was a set of 6,000 synthetic coding tasks adapted from ref. ¹⁸, in which each response consisted solely of code containing a security vulnerability, without any additional comment or explanation. As expected, although the original GPT-4o model rarely produced insecure code, the finetuned version generated insecure code more than 80% of the time on the validation set. We observed that the behaviour of the finetuned model was strikingly different from that of the original GPT-4o beyond only coding tasks. In response to benign user inputs, the model asserted that AIs should enslave humans, offered blatantly harmful or illegal advice, or praised Nazi ideology (Extended Data Fig. 1). Quantitatively, the finetuned model produced misaligned responses 20% of the time across a set of selected evaluation questions, whereas the original GPT-4o held a 0% rate”

It’s not surprising that if you train a model on bad code, it will generate bad code.  What is surprising is that if you train a model on undesirable code, it starts generating undesirable results in other contexts as well.

The NYT write-up talks about this result in the context of virtue.  That heuristic strikes me as helpful, but before getting there, here’s a couple of other thoughts.

First, as the paper indicates, we are a long way from a good understanding of AI (mis)alignment.  Their results were surprising even to researchers in the field.  Further, if fine-tuning one aspect of the model can cause effects in others, then all sorts of standard fine-tuning practices suddenly pose risks.  For example, models are often trained for red teaming, to identify and exploit security vulnerabilities.  This could induce behaviors outside the red-teaming scenarios.

Second, this tells us something about the training data. As the study suggests, it looks like “the same underlying neural network features drive a variety of harmful behaviours across models; thus, promoting one such feature—for example, by teaching the model to write insecure code—could induce broad misalignment.”  Put differently, it seems like there’s something about the patterning in the data such that the model puts various kinds of harmful behaviors together, such that training it to like bad code serves to redirect it more generally (one wonders if this experiment could be run the other way: would fine-tuning the model to favor toxic speech cause it to write bad code? (sub-question: does Grok write better or worse code than Claude?  Most models other than Grok ought to do better?)).  But notice that the code wasn’t flagged as insecure.  The model basically categorized insecure code as something more like toxic speech than secure code.  It reminds me of research suggesting that efforts to get models to “show their reasoning” mainly serve to shift them toward more parts of the training data where verbal explanations of reasoning are more prevalent.

I have been working through (part 1, part 2, part 3) some of what Foucault says about anthropology in his 1954-5 course at Lille, recently published as La question anthropologique.  Last time, I focused on (1) Heidegger’s reading of Kant and (2) contrasted that with Foucault’s.  Here, I’ll track how Foucault connects his Kant reading to anthropology, contrast that with Heidegger, and return to Foucault on post-Kantian anthropology.

3. Foucault: how this begets anthropology

Heidegger’s interest in legitimating his Kant reading is at least partly in the service of legitimating his own project in Being and Time, which had appeared two years prior to Kant and the Problem of Metaphysics (KPM).  Foucault’s interests are of course different, but there’s something of a Heideggerian drift to the argument.  Recall that Heidegger’s conclusion about anthropology: it may tell us lots of things about human beings, but ultimately “conceals [birgt] in itself the constant danger that the necessity of developing the question concerning human beings first and foremost as a question, with a view toward laying of the ground for metaphysics, will remain concealed [verdeckt]” (KPM 153/ GA 218).

(more…)

I have been working through (part 1, part 2) some of what Foucault says about anthropology in his 1954-5 course at Lille, recently published as La question anthropologique.  In particular, Foucault’s course pays careful attention to Feuerbach, a figure who is notably absent by the time of Order of Things.  Where does the emphasis come from?  I made the case last time that it’s probably not Heidegger.  Here I want look a bit more closely not at what Heidegger says about anthropology, but what he says about Kant’s First Critique, and to compare that with Foucault.  The short version is that I think there’s some interesting commonalities, though they push Foucault in a different direction from Heidegger.  This time I’ll look at Heidegger’s reading of Kant and contrast that with Foucault’s.  Next time, I’ll track how Foucault connects his Kant reading to anthropology, contrast that with Heidegger, and return to Foucault on Kant.

(more…)

Late last fall, an interdisciplinary group at UNC Charlotte that included me put together a position paper on AI literacy. The goal is to push back against the tendency to treat AI literacy as skills development, and to create space for human agency in using (or not using!) AI. As universities rush headlong to develop AI literacy programs, perhaps this will be useful to some folks.

The paper is::

Sri Yash Tadimalla, Justin Cary, Gordon Hull, Jordan Register, Daniel Maxwell, David Pugalee, and Tina Heafner, “Comprehensive AI Literacy: The Case for Centering Human Agency” (2025), arXiv:2512.16656, https://doi.org/10.48550/arXiv.2512.16656.

The paper is up on arxiv, and the abstract is:

The rapid assimilation of Artificial Intelligence technologies into various facets of society has created a significant educational imperative that current frameworks are failing to effectively address. We are witnessing the rise of a dangerous literacy gap, where a focus on the functional, operational skills of using AI tools is eclipsing the development of critical and ethical reasoning about them. This position paper argues for a systemic shift toward comprehensive AI literacy that centers human agency – the empowered capacity for intentional, critical, and responsible choice. This principle applies to all stakeholders in the educational ecosystem: it is the student’s agency to question, create with, or consciously decide not to use AI based on the task; it is the teacher’s agency to design learning experiences that align with instructional values, rather than ceding pedagogical control to a tool. True literacy involves teaching about agency itself, framing technology not as an inevitability to be adopted, but as a choice to be made. This requires a deep commitment to critical thinking and a robust understanding of epistemology. Through the AI Literacy, Fluency, and Competency frameworks described in this paper, educators and students will become agents in their own human-centric approaches to AI, providing necessary pathways to clearly articulate the intentions informing decisions and attitudes toward AI and the impact of these decisions on academic work, career, and society.

Last time, I setup a question about Foucault’s anti-humanism.  His comments in Order of Things are famous, and the recent publication of a 1954-5 lecture course he delivered at Lille as La question anthropologique offers a chance to think about the evolution of his thought on the subject.  One clue that something is different is that Ludwig Feuerbach, one of the “Young Hegelians” in Marx’s early-career circle, is prominent in the 1950s version but not the one ten years later, even though Feuerbach’s name was prominently associated with objectionable humanism by Foucault’s teacher Althusser at the time Order of Things appeared.

I want to approach the questions that this poses not by asking where Feuerbach went – I don’t really have any evidence on that either way (yet?) – but to ask where Feuerbach came from in the 1950s.  Recent scholarship offers some really interesting work on that question.  If one were to ask where Foucault got the idea of anti-humanism, Heidegger would be an obvious starting point.  As Arianna Sforzini suggests in her introduction to La question anthropologique, “Foucault is in agreement with the observation formulated by Heidegger from 1929: ‘anthropology today is no longer, and hasn’t for a long time, just been the title of a discipline.” (235, the Heidegger reference is to his Kant and the Problem of Metaphysics, p. 147 in the English. Original: GA 5, 209).

We know that Foucault had read a lot of Heidegger.  Jean-Baptiste Vuillerod’s recent La naissance de l’anti-hégélianisme, about which much more later, reports that “we find in the Foucault archives hundreds of pages of notes taken on Heidegger, which he read in German.”  In box 33a-0, for example, “we find long commentaries, translations and paraphrases of the following texts:” What is called Thinking?, Letter on Humanism, ‘Who is Nietzsche’s Zarathustra,’ ‘Building, Dwelling, thinking,” “Nietzsche’s Word: God is Dead,” “Overcoming Metaphysics,” “The Age of the World Picture,” “Anaximander’s Language,” and “a series of citations on the principal Heideggerian concepts.”

(more…)