I want to take a break from judicial standing doctrine to note a recent and helpful paper by Emily Sullivan and Atoosa Kasirzadeh about explainable AI. Explainable AI is a research agenda – there’s a lot of papers and techniques (for a current lit review, see here) – that is designed to get at a central problem in using AI: we often have no idea why machine learning systems produce the outputs they do. In a variety of contexts, ranging from safety critical systems to democratic governance, being able to understand why the algorithm made the prediction is did is important. Hence the research agenda.
First, a little detour. Algorithmic governance can be disciplinary in that it can nudge people inexorably toward conforming with norms, whether social or statistical. Insurance has been well-studied for its normalizing techniques. In an early paper on privacy unraveling, Scott Peppett showed how the addition of smart-driving surveillance (where insurers give a discount to people who install these devices that record their speed, when they drive, etc.) generate a downward ratcheting on privacy: users who are good drivers have the incentive to adopt the devices, since they get lower insurance rates. Those who are in the next tier down (above-average drivers) have an incentive to get the devices because that associates them with the good drivers. And so it goes, until only the worst drivers are declining the surveillance. And at some point, not having the surveillance device becomes a stigma that raises your rates. So pretty soon, surveillance devices can become normal. In the meantime, once drivers have the surveillance installed, surveillance-enabled insurance can nudge them to drive less at night and to otherwise comply with whatever the insurance company says makes you a good risk. All of that can be automated - the insurance app can tell you, real time, how your driving is impacting your premium.
Neoliberal versions like employee wellness programs use insurance and fitness tracking as a mechanism for nudging employees towards behaving when not on the clock in ways that employers find desirable. Want a discount on your health insurance? Take more steps! More generally, if the algorithm is predictive of risk, and serves as gatekeeper to something, then individuals have an incentive to modify their behavior to be lower risk, so as to increase their odds of getting the benefit.
Those predictions involve models where the significant variables are likely to be known both to the people making the decisions and those subject to them. What about AI? Machine learning outputs are often opaque, in the sense that neither decisionmakers nor subjects understand what drove a given prediction. Partly for reasons like this, algorithmic governance can be depoliticizing, as various commentators have noted. Explainable AI is one way out of the predicament, at least in theory – we can get back to the good old days of insurance and employee wellness programs.
Sullivan and Kasirzadeh are concerned with counterfactual explanation: what changes in the world would be most likely to alter the system’s output? They distinguish between two kinds of counterfactual explanation. One aims at understanding: why did the system generate the output it did – what variables were most important to that outcome? The other is recourse: which variable that the subject can control could be changed, such that if they modified those variables and tried again, they might have better luck? For example, a recourse explanation of a loan denial might say that the system assigned a lot of weight to a $10,000 debt. Paying off that debt would then raise the odds of getting the loan. The two kinds of explanation can also come apart: in their example of a loan, it might be that the applicant was also close to retirement age, and that was at least as central to the denial than the debt. The person could either pay off their debt, or they could be ten years younger – either counterfactual would have caused the loan to go through,
I want to suggest that one implication of Sullivan and Kasirzadeh’s argument is that recourse XAI can itself be a form of algorithmic governance, not an actual recourse against it. It’s the equivalent of the surveillance device saying to quit driving at night (and poses the same structural problems for people whose jobs require them to drive at night). As they note, their preferred option – understanding-based XAI, which lists factors most salient in the algorithm’s decision without regard to whether the data subject can do anything about them – are much better for seeing the structural and political reasons behind an algorithmic decision. In the loan example, the understanding-based explanation is that age was the biggest factor in the decision. Of course, the applicant cannot change that. A recourse-explanation might suggest that they increase their overall savings or payoff their debt. The recourse-explanation both distracts from the political question of age discrimination and dumps responsibility for the problem on the data subject.
They offer the following vivid example, “Apartment Fire:”
“There was a fire in an apartment building, leaving several units uninhabitable. The tenants want an explanation why the fire spread so quickly throughout the building. The building developer and landlord was told by the fire chief that the main reason the fire spread was due to the flammable building materials that went against current fire safety regulations”
An understanding-based explanation of what happened is that had the building been built to code, that would have avoided the problem. But a landlord who favored a recourse-explanation might say something like “if the apartment had not had any furniture or books along the walls, and instead had several fountain walls, then the fire would not have spread.” No doubt true! But also not the correct explanation in that it imposes a ridiculous burden on tenants and absolves the developer and landlord of liability. This is a particular risk because for any given algorithmic decision, there will be many, many possible counterfactual scenarios, leading to the temptation toward “explanation hacking,” a form of motivated reasoning that leads one to favor self-serving counterfactuals, rather than more explanatory ones. As they argue,
“we should stop engaging in real time, post-hoc, model exploration that allows users, industry, or developers to justify their model based on flipping through several candidate CEs. While this practice may still be useful for users to find actionable advice or recommendations, it is important to separate the difference between explaining an AI decision versus what action guiding possibilities are open to individuals”
In short: Sullivan and Kasirzadeh offer good reasons to worry about recourse-based XAI. An implication of their argument is that it may well serve to legitimate and perpetuate the disciplinary mechanisms of algorithmic governance, at the expense of the political ability to contest its decisions. As Foucault says in a slightly different context, “the general juridical form that guaranteed a system of rights that were egalitarian in principle was supported by these tiny, everyday, physical mechanisms, by all those systems of micro-power that are essentially non-egalitarian and asymmetrical that we call the disciplines.” (Discipline and Punish, 222). By deflecting away from contesting algorithms, recourse-based AI serves this disciplinary project by telling users how to comply with them.
Recent Comments