New APPS: Art, Politics, Philosophy, Science

New Paper: "Translating Privacy for Data Subjects"

Here's the current draft of a new paper - "Translating Privacy for Data Subjects."  And here's the abstract:

This essay offers a theoretical account of one reason that current privacy regulation fails.  I argue that existing privacy laws inherit a focus on judicial subjects, using language about torts and abstract rights.   Current threats to privacy, on the other hand, presuppose statistically-generated populations and aggregative models of subjectivity.  This gap in underlying presupposition generates the need to translate privacy protections into the data era.  After situating my theoretical account, I identify two areas where data subjects face entry barriers to the legal protections afforded data subjects.  On the one hand, data subjects are often denied standing because privacy harms are deemed speculative and abstract.  On the other hand, algorithmic governance makes it hard to argue that one is an exception to a rule, depriving data subjects of traditional equity.  I then offer two ways to translate privacy for data subjects.  The first is reform of standing through reformed class certification, making the harms data subjects face as populations judicially legible.  The second is a reconceptualization of equity, reframing it both to re-establish the right to be an exception to a data-driven rule and to allow equitable consideration of issues that face statistical populations.

 

Juridical vs Statistical Reasoning (part 2): Not all Buses are the Same

By Gordon Hull

Last time, I looked at the Lawrence Tribe article that was the original source of the blue bus thought experiment.  Tribe’s article is notable for its defense of legal reasoning and processes against the introduction of statistical evidence in trials.  He particularly emphasizes the need for legal proceedings to advance causal accounts, and of the numerous background conditions that legal reasoning seeks (and that statistical evidence tends to ignore).  The tendency to fetishize numeric results also tended to generate absurd results. 

Here I want to focus on a recent paper by Chad Lee-Stronach that centers on the blue bus argument, and that formally explains what I take to be the intuition behind a lot of Tribe’s examples, even if he directs them differently.  Lee-Stronach begins with the “statistical proof paradox,” which he formulates roughly as follows:

1. Probability threshold: any legal standard of proof is reducible to some threshold of probability t, such that the defendant should be found liable when the probability that they are liable exceeds t
2. Statistical inference: merely statistical evidence can establish this
3. Conclusion: a defendant can be found liable on the basis of merely statistical evidence

It’s a paradox because most of us don’t accept the conclusion.  Lee-Stronach surveys the literature, and notes that most of it goes after Probability Threshold.  Instead, he thinks, we’d be better off going after the second premise.  Accordingly, he argues that the statistical inference premise can’t be met.

Juridical vs Statistical Reasoning (part I): Behind the Blue Bus

By Gordon Hull

AI (machine learning) and people reach conclusions in different ways.  This basic point has ramifications across the board, as plenty of people have said.  I’m increasingly convinced that the gap between how legal reasoning works and how ML works is a good place both to tease out the differences, and to think about what’s at stake in them (I’ve made a couple of forays into this, here and here). One good reason for this is that legal theory is full of good examples, which can function as rules in the old-fashioned sense of paradigms, as described by Lorraine Daston in her fascinating historical account.  This use of examples and cases is deeply entwined with legal reasoning; as Daston notes, “common law, casuistry, and the liberal and mechanical arts all came to inhabit the territory staked out by the Roman libri regularum. They all depended on rules that got things done in the world, consistent with but not deduced from higher principles and often in the form of an example extended by analogy” (Rules, 30)

Another is that one of those examples, the blue bus, has been an enduring focal point for the difference between the two.  The scenario stipulates that an accident has happened involving a bus, during a time and at a location where 80% of the buses present belong to the blue bus company.  Do we assign liability to the bus company?  Work by David Enoch and Talia Fisher notes that most people prefer a 70%-reliable witness to the information that a bus company owns 70% of the buses in an area, and theorizes that the reason is that we prefer the available counterfactual in case the eyewitness is wrong.  There is presumably some process that explains her incorrect identification of the bus, whereas the statistical evidence is what it is.  In this case, we expect a specific failure rate.

RLHF and Curation Transparency

By Gordon Hull

Last time, I followed a reading of Kathleen Creel’s recent “Transparency in Complex Computational Systems” to think about the ways that RLHF (Reinforcement Learning with Human Feedback) in Large Language Models (LLMs) like ChatGPT necessarily involves an opaque, implicit normativity.  To recap: RLHF improves the models by involving actual humans (usually gig workers) in their training: the model presents two possible answers to a prompt, and the human tells it which one is better.  As I suggested, and will pursue in a later post, this introduces all sorts of weird and difficult-to-measure normative aspects into the model performance, above and beyond those that are lurking in the training data.  Here I want to pause to consider this as a question of opacity and transparency. I’m going to end up by proposing that there’s a fourth kind of transparency that we should care about, for both epistemic and moral reasons, which I’ll call “curation transparency.”

Implicit Normativity in Reinforcement Learning with Human Feedback in Large Language Models

By Gordon Hull

This is somewhat circuitous – but I want to approach the question of Reinforcement Learning with Human Feedback (RLHF) by way of recent work on algorithmic transparency.  So bear with me… RLHF is currently all the rage in improving large language models (LLMs).  Basically, it’s a way to try to deal with the problem that LLMs aren’t referentially grounded, which means that their output is not in any direct way connected to the world outside the model.

LLMs train on large corpora of internet text – typically sources like Wikipedia, Reddit, patent applications and so forth.  They learn to predict what kinds of text are likely to come next, given a specific input text.  The results, as anybody who has sat down with ChatGPT for long knows, can be spectacular.  Those results also evidence that the models function, in one paper’s memorable phrasing, as “stochastic parrots.”  What they say is about what their training data says is most likely, not about what’s, say, contextually appropriate.  But appropriate human speech is context-dependent, and answers that sound right (in the statistical sense: these words, in general, are likely to come after those words) in one context may be wrong in another (because language does not get used “in general”).  RLHF is designed to get at that problem, as a blogpost at HuggingFace explains:

Cars are Privacy Disasters

This article from Gizmodo reports on research done over at Mozilla.  Newer cars – the ones that connect to the internet and have lots of cameras – are privacy disasters.  Here’s a paragraph to give you a sense of the epic scope of the disaster:

“The worst offender was Nissan, Mozilla said. The carmaker’s privacy policy suggests the manufacturer collects information including sexual activity, health diagnosis data, and genetic data, though there’s no details about how exactly that data is gathered. Nissan reserves the right to share and sell “preferences, characteristics, psychological trends, predispositions, behavior, attitudes, intelligence, abilities, and aptitudes” to data brokers, law enforcement, and other third parties.”

Nissan’s response tells you everything that’s wrong with current privacy legislation:

““When we do collect or share personal data, we comply with all applicable laws and provide the utmost transparency,” said Lloryn Love-Carter, a Nissan spokesperson. “Nissan’s Privacy Policy incorporates a broad definition of Personal Information and Sensitive Personal Information, as expressly listed in the growing patchwork of evolving state privacy laws, and is inclusive of types of data it may receive through incidental means.””

Let’s translate.  Nissan is probably compliant.  Also, privacy compliance is a joke.  Also, compliance apparently only requires that you receive NOTICE that they take your data AND CONSENT to that policy, probably merely by driving the vehicle.  Also, they probably reserve the right to change their privacy policies unilaterally, at will.  Also, they almost certainly do not let you opt-out of any of it while CONSENTING by driving the car.  It’s a very special kind of “contract” and “consent.”  Also, how do they know about your sex life?  Also, even if you have sex in the car, there is basically no answer to that question that is not beyond creepy!

As you may have guessed, NOTICE AND CONSENT is an utter sham and has been for a while.  The gizmodo article spells out some of the particular absurdities here – for example, you may not want to ride in one of these cars either, as passengers are “users” deemed to have CONSENTED to the privacy policy.  Your driver should probably provide you NOTICE beforehand!  “A number of car brands say it’s the driver’s responsibility to let passengers know about their car’s privacy policies—as if the privacy policies are comprehensible to drivers in the first place.”  No wonder folks are cynical and resigned about corporate privacy – they’re manipulated into it by corporations.  Also, they’re confused, frustrated and angry about the fact they don’t actually get to consent.

This is the best example I’ve seen of all that in a while, and a crystal-clear indicator of why we need not just new privacy legislation (we do!) but a new direction (more real regulation, less soft compliance and "notice and consent" fig-leaves).

PS – not picking only on Nissan:

“Other brands didn’t fare much better. Volkswagen, for example, collects your driving behaviors such as your seatbelt and braking habits and pairs that with details such as age and gender for targeted advertising. Kia’s privacy policy reserves the right to monitor your “sex life,” and Mercedes-Benz ships cars with TikTok pre-installed on the infotainment system, an app that has its own thicket of privacy problems.”

Some Reasons to be Skeptical of AI Authorship, Part 1: What is an (AI) Author?

By Gordon Hull

Large Language Models (LLMs) like Chat-GPT burst into public consciousness sometime in the second half of last year, and Chat-GPT’s impressive results have led to a wave of concern about the future viability of any profession that depends on writing, or on teaching writing in education.  A lot of this is hype, but one issue that is emerging is the role of AI authorship in academic and other publications; there’s already a handful of submissions that list AI co-authors.  An editorial in Nature published on Feb. 3 outlines the scope of the issues at hand:

“This technology has far-reaching consequences for science and society. Researchers and others have already used ChatGPT and other large language models to write essays and talks, summarize literature, draft and improve papers, as well as identify research gaps and write computer code, including statistical analyses. Soon this technology will evolve to the point that it can design experiments, write and complete manuscripts, conduct peer review and support editorial decisions to accept or reject manuscripts”

As a result:

“Conversational AI is likely to revolutionize research practices and publishing, creating both opportunities and concerns. It might accelerate the innovation process, shorten time-to-publication and, by helping people to write fluently, make science more equitable and increase the diversity of scientific perspectives. However, it could also degrade the quality and transparency of research and fundamentally alter our autonomy as human researchers. ChatGPT and other LLMs produce text that is convincing, but often wrong, so their use can distort scientific facts and spread misinformation.”

The editorial then gives examples of LLM-based problems with incomplete results, bad generalizations, inaccurate summaries, and other easily-generated problems.  It emphasizes accountability (for the content of material: the use of AI should be clearly documented) and the need for the development of truly open AI products as part of a push toward transparency.

Gatecrashers, Blue Buses and Speeding Drivers: Philosophy of Law as a way into AI Accountability (part 2)

By Gordon Hull

Last time, I introduced a number of philosophy of law examples in the context of ML systems and suggested that they might be helpful in thinking differently, and more productively, about holding ML systems accountable.  Here I want to make the application specific.

So: how do these examples translate to ML and AI?  I think one lesson is that we need to specify what exactly we are holding the algorithm accountable for.  For example, if we suspect an algorithm of unfairness or bias, it is necessary to specify precisely what the nature of that bias or unfairness is – for example, that it is more likely to assign high-risk status to Black defendants (for pretrial detention purposes) than it is white ones.  Even specifying fairness in this sense can be hard, because there are conflicting accounts of fairness at play.  But assuming that one can settle that question, we don’t need to specify tokens or individual acts of unfairness (or demand that each of them rise to the level where they would individually create liability) to demand accountability of the algorithm or the system that deploys it – we know that the system will have treated defendants unfairly, even if we don’t know which ones (this is basically a disparate impact standard; recall that one of the original and most cited pieces on how data can be unfair  was framed precisely in terms of disparate impact).

Further, given the difficulties of individual actions (litigation costs, as well as getting access to the algorithms, which defendants will claim as trade secrets) in such cases, it seems wrong to channel accountability through tort liability and demand that individuals prove the algorithm discriminated against him (how could they?  The situation is like the blue bus: if a group of people is 80% likely to reoffend or skip bail, we know that 20% of that group will not, and there is no “error” for which the system can be held accountable).  Policymakers need to conduct regular audits or other supervisory activity designed to ferret out this sort of problem, and demand accountability at the systemic level.

Gatecrashers, Blue Buses and Speeding Drivers: Philosophy of Law as a way into AI Accountability (part 1)

By Gordon Hull

AI systems are notoriously opaque black boxes.  In a now standard paper, Jenna Burrell dissects this notion of opacity into three versions.  The first is when companies deliberately hide information about their algorithms, to avoid competition, maintain trade secrets, and to guard against gaming their algorithms, as happens with Search Engine Optimization techniques.  The second is when reading and understanding code is an esoteric skill, so the systems will remain opaque to all but a very small number of specially-trained individuals.  The third form is unique to ML systems, and boils down to the argument that ML systems generate internal networks of connections that don’t reason like people.  Looking into the mechanics of a system for recognizing handwritten numbers or even a spam detection filter wouldn’t produce anything that a human could understand.  This form of opacity is also the least tractable, and there is a lot of work trying to establish how ML decisions could be made either more transparent or at least more explicable.

Joshua Kroll argues instead that the quest for potentially impossible transparency distracts from what we might more plausibly expect from our ML systems: accountability.  After all, they are designed to do something, and we could begin to assess them according to the internal processes by which they are developed to achieve their design goals, as well as by empirical evidence of what happens when they are employed.  In other words, we don’t need to know exactly how the system can tell a ‘2’ from a ‘3’ as long as we can assess whether it does, and whether that objective is serving nefarious purposes.

I’ve thought for a while that there’s potential help for understanding what accountability means in the philosophy of law literature.  For example, a famous thought experiment features a traffic accident caused by a bus.  We have two sources of information about this accident.  One is an eyewitness who is 70% reliable and says that the bus was blue.  The other is the knowledge that 70% of the buses that were in the area at the time were blue.  Epistemically, these ought to be equal – in both cases, you can say with 70% confidence that the blue bus company is liable for the accident.  But we don’t treat them as the same: as David Enoch and Talia Fisher elaborate, most people prefer the witness to the statistical number.  This is presumably because when the witness is wrong, we can inquire what went wrong.  When the statistic is wrong, it’s not clear that anything like a mistake even happened: the statistics operate at a population level; when applied to individuals, the use of statistical probability will be wrong 30% of the time, and so we have to expect that.  It seems to me that our desire for what amounts to an auditable result is the sort of thing that Kroll is pointing to.

Base Rate Tracking often can’t fix algorithmic fairness

By Gordon Hull

As a criterion for algorithmic assessment, “fairness” has encountered numerous problems.  Many of these emerged in the wake of ProPublica’s argument that Broward County’s pretrial detention system, COMPAS, was unfair to black suspects.  To recall: In 2016, ProPublica published an investigation piece criticizing Broward County, Florida’s use of a software program called COMPAS in its pretrial detention system.  COMPAS produced a recidivism risk score for each suspect, which could then be used in deciding whether someone should be detained prior to their trial.  ProPublica’s investigation found that, among suspects that did not have a rearrest prior to their trial, black suspects were much more likely to have been rated as “high risk” for rearrest than white suspects.  Conversely, among suspects who were arrested a second time, white suspects were more likely to have been labeled “low risk” than black ones.  The system thus appeared to be discriminating against black suspects.  The story led to an extensive debate (for an accessible summary with cites, see Ben Green’s discussion here) over how fairness should be understood in a machine learning context.

The debate basically showed that ProPublica focused on outcomes and demonstrated that the system failed to achieve separation fairness, which is met when all groups subject to the algorithm’s decisions receive the same false negative/positive rate.  The system failed because “high-risk” black suspects were much more likely than white to be false positives.  In response, the software vendor argued that the system made fair predictions because among those classified in the same way (high or low risk), both racial groups exhibited the predicted outcome at the same rate.  In other words, among those classified as “high risk,” there was no racial difference in how likely they were to actually be rearrested.  The algorithm thus satisfied the criterion of sufficiency fairness.  In the ensuing debate, computer scientists arrived at a proof that, except in very limited cases, it was impossible to simultaneously satisfy both separation and sufficiency fairness criteria.

In the meantime, on the philosophy side, Brian Hedden has argued that a provably fair algorithm could nonetheless be shown to potentially violate 11 of 12 possible fairness conditions.  In a response piece, Benjamin Eva showed the limits of the twelfth with a different test and proposed a new criterion:

Clearview settles Facial Recognition Case

Luke Stark argues that Facial recognition should be treated as the “plutonium of AI” – something so dangerous that it’s use should be carefully controlled and limited.  If you follow  the news, you’ll know that we’re currently treating it as the carbon dioxide of AI, a byproduct of profit-making that doesn’t look too awful on its own until you realize its buildup could very well cause something catastrophic to happen.  Activists have worried about this pending catastrophe for a while, but lots of big money supports facial recognition, so they have thrown up a smokescreen of distractions – in one case, Facebook denied that its phototagging software in fact recognized faces (!) – in order to lull everyone into accepting it.

One of the worst offenders is a secretive company called Clearview, whose business model is to scrape the web of all the pictures it can find and then sell the technology to law enforcement.  The company even has an international presence: in one disturbing instance, the Washington Post documents the use of its technology by Ukrainians to identify dead Russian soldiers by way of their Instagram and other social media accounts, and then sometimes to contact their families.  More generally, the Post revealed internal documents showing that the company' database is nearing 100 billion images and that "almost everyone in the world will be identifiable."  They're going all-in; the Post reports that "the company wants to expand beyond scanning faces for the police, saying in the presentation [obtained by the WP] that it could monitor 'gig economy' workers and is researching a number of new technologies that could identify someone based on how they walk, detect their location from a photo or scan their fingerprints from afar."

Clearview is also one of a cohort of companies that has been sued for violating Illinois’ Biometric Information Privacy Act (BIPA).  BIPA, uniquely among American laws, requires opt-in assent for companies to use people’s biometric information (the Facebook case is central to my argument in this paper (preprint here); for some blog-level discussion see here and here).  Of course, BIPA is a state-level law, so its protections do not automatically extend to anyone who lives outside of Illinois.  That’s why yesterday’s news of a settlement with the ACLU is really good news.  The Guardian reports:

Facial recognition startup Clearview AI has agreed to restrict the use of its massive collection of face images to settle allegations that it collected people’s photos without their consent.  The company in a legal filing Monday agreed to permanently stop selling access to its face database to private businesses or individuals around the US, putting a limit on what it can do with its ever-growing trove of billions of images pulled from social media and elsewhere on the internet. The settlement, which must be approved by a federal judge in Chicago, will end a lawsuit brought by the American Civil Liberties Union and other groups in 2020 over alleged violations of an Illinois digital privacy law. Clearview is also agreeing to stop making its database available to Illinois state government and local police departments for five years. The New York-based company will continue offering its services to federal agencies, such as US Immigration and Customs Enforcement, and to other law enforcement agencies and government contractors outside Illinois.

Of course, the company denies the allegations in the lawsuit, and insists that it was just in the process of rolling out a “consent-based” product.  Ok, sure!  This is still a win for privacy and for one of the very few pieces of legislation in the U.S. that has any chance of limiting the use of biometric data.

Yet another terrible AI idea

People make snap judgments about those they see the first time – mentally categorizing someone as friendly, threatening, trustworthy, etc.  Most of us know that those impressions are idiosyncratic, and suffused with cultural biases along race, gender and other lines.  So obviously I know what you’re thinking… we need an AI that do that, right?  At least that’s what this new PNAS paper seems to think (h/t Nico Osaka for the link).  The authors start right in with the significance:

“We quickly and irresistibly form impressions of what other people are like based solely on how their faces look. These impressions have real-life consequences ranging from hiring decisions to sentencing decisions. We model and visualize the perceptual bases of facial impressions in the most comprehensive fashion to date, producing photorealistic models of 34 perceived social and physical attributes (e.g., trustworthiness and age). These models leverage and demonstrate the utility of deep learning in face evaluation, allowing for 1) generation of an infinite number of faces that vary along these perceived attribute dimensions, 2) manipulation of any face photograph along these dimensions, and 3) prediction of the impressions any face image may evoke in the general (mostly White, North American) population”

Let’s maybe think for a minute, yes?  Because we know that people make these impressions on unsound bases!

First, adversarial networks are already able to produce fake faces that are indistinguishable from real ones.  Those fake faces can now be manipulated to appear more or less trustworthy, hostile, friendly, etc.  When you make fake political ads, for example, that’s going to be useful.  Already 6 years ago, one “Melvin Redick of Harrisburg, Pa., a friendly-looking American with a backward baseball cap and a young daughter, posted on Facebook a link to a brand-new website,” saying on June 8, 2016 that “these guys show hidden truth about Hillary Clinton, George Soros and other leaders of the US. Visit #DCLeaks website. It’s really interesting!” Of course, both Melvin Redick and the site he pointed to were complete fabrications by the Russians.  Now we can make Melvin look trustworthy, and Clinton less so.

Second, the ability to manipulate existing face photos is a disaster-in-waiting.  Again, we saw crude efforts with this before – making Obama appear darker than he is, for example.  But here news photos could be altered to make Vladimir Putin appear trustworthy, or Mr. Rogers untrustworthy.  This goes nowhere good, especially when combined with deepfake technology that already takes people out of their contexts and puts them in other ones (disproportionately, so far, women are pasted into porn videos, but the Russians recently tried to produce a deepfake of Zelensky surrendering.  Fortunately that one was done sloppily).

Third, and I think this one is possibly the scariest, what about scanning images to see whether someone will be assessed as trustworthy?  AI-based hiring is already under-regulated!  Now employers will run your photo through the software and make customer-service hiring decisions based on who customers will perceive as trustworthy.  What could go wrong?

All of this of course assumes that this sort of software actually works.  The history of physiognomic AI, which uses all sorts of supposedly objective cues to determine personality and which is basically complete (usually racist) bunk suggests that the science is probably not as good as the article acts like.  So maybe we’re lucky and this algorithm does not actually work as advertised.  Of course, the fact that AI software is garbage doesn't preclude its being used to make people's lives miserable.  Just consider the bizarre case of VibraImage.

But don’t worry.  The PNAS authors are aware of ethics, noting that “the framework developed here adds significantly to the ethical concerns that already enshroud image manipulation software:”

“Our model can induce (perceived) changes within the individual’s face itself and may be difficult to detect when applied subtly enough. We argue that such methods (as well as their implementations and supporting data) should be made transparent from the start, such that the community can develop robust detection and defense protocols to accompany the technology, as they have done, for example, in developing highly accurate image forensics techniques to detect synthetic faces generated by SG2. More generally, to the extent that improper use of the image manipulation techniques described here is not covered by existing defamation law, it is appropriate to consider ways to limit use of these technologies through regulatory frameworks proposed in the broader context of face-recognition technologies.” 

Yes, the very effective American regulation of privacy does inspire confidence!  Also, “There is also potential for our data and models to perpetuate the biases they measure, which are first impressions of the population under study and have no necessary correspondence to the actual identities, attitudes, or competencies of people whom the images resemble or depict.”

Do you think?  As Luke Stark put it, facial recognition is the “plutonium of AI:” very dangerous and with very few legitimate uses.  This algorithm belongs in the same category, and should similarly be regulated like nuclear waste.  For example, as Ari Waldman and Mary Anne Franks have written, one of the problems with deepfakes is that the fake version gets out there on the internet, and it is nearly impossible to make it go away (if you even know about it).  Forensic software gets there too late, and those without resources aren’t going to be able to deploy it anyway.  Lawsuits are even less useful, since they're time-consuming and expensive to pursue, and lots of defendants won't be jurisdictionally available or have pockets deep enough to make the chase worth it.  In other words, not everybody is going to be able to defend themselves like Zelensky, who both warned about deepfakes and was able to produce video of himself not surrendering.  In the meantime, faked and shocking things generally get diffused faster and further than real news.  After all, “engagement” is the business model of social media.  Further, to the extent that people stay inside filter bubbles (like Fox News), they may never see the forensic corrections, and they probably won’t believe the real one is real, even if they do.

And as for reinforcing existing biases, Safiya Noble already wrote a whole book on how algorithms that guess what you’re probably thinking about someone can do just that.

Can Datasets be Governed Like Public Utilities?

By Gordon Hull

Machine Learning (ML) applications learn by repetition.  That is, they come to recognize what, say, a chair looks like, by seeing lots of images of chairs that have been correctly labeled as such.  Since the machine is trying to figure out a pattern or set of characteristics that distinguish chairs from other objects, the more chairs it sees, the better it will perform at its task.  This collection of labeled chairs is the algorithm’s “training data.”   There are notorious problems with bias in training data due to underrepresentation of certain groups.  For example, one study found that datasets designed to train ML to recognize objects performed poorly in developing countries, most likely due to underrepresentation of images from those places; when combined with pictures labeled in English, and the fact that standard Western commodity forms of household objects might look very different in the developing world, the ML was stumped.  Most famously in this country, facial recognition software performs best at identifying white men and worst at identifying Black women.  ImageNet, which is widely used for object recognition purposes, employees a hierarchy of labels that include calling a child wearing sunglasses a “failure, loser, non-starter, unsuccessful person” but also differentiates between assistant and associate professors.  Despite these and many other problems, massive datasets are essential for training ML applications.

For this reason, datasets have been called “infrastructural” for machine learning, defined as follows:

“Infrastructure is characterized, we argue, by a set of core features: it is embedded into, and acts as the foundation, for other tools and technologies; when working as intended for a particular community, it tends to seep into the background and become incorporated into routines; the invisibility of infrastructure, however, is situated - what is natural or taken for granted from one perspective may be highly visible or jarring from another; though frequently naturalized, infrastructure is built, and thus inherently contextual, situated, and shaped by specific aims.”

If AI is cars and buses and trains that do what we want, the datasets it trains on shape the roads and paths where those roads go, provide their material basis, become thereby incorporated into higher level routines like search, and tend to disappear into the background when not actively used.  But just like other examples of infrastructure – say, the bridges over the Long Island Freeway – infrastructure can embed priorities and affordances.  In this sense, dataset infrastructures have a politics, and serve as platforms on which applications are built.

New Paper: "The Death of the Data Subject"

Shameless self-promotion dept... here's the preprint for my new paper, "The Death of the Data Subject," now forthcoming in Law, Culture and the Humanities.  And here's the abstract:

This paper situates the data privacy debate in the context of what I call the death of the data subject.  My central claim is that concept of a rights-bearing data subject is being pulled in two contradictory directions at once, and that simultaneous attention to these is necessary to understand and resist the extractive practices of the data industry. Specifically, it is necessary to treat the problems facing the data subject structurally, rather than by narrowly attempting to vindicate its rights.  On the one hand, the data industry argues that subjects of biometric identification lack legal standing to pursue claims in court, and Facebook recently denied that that its facial recognition software recognizes faces.  On the other hand, industry takes consent to terms of service and arbitration clauses to create enforceable legal subject positions, while using promises of personalization to create a phenomenological subject that is unaware of the extent to which it is being manipulated.  Data subjects thus have no legal existence when it is a matter of corporate liability, but legal accountability when it is a matter of their own liability.  Successful reform should address the power asymmetries between individuals and data companies that enable this structural disempowerment.

 

 

No Standing for You! (part 1)

By Gordon Hull

In one of the Seinfeld episodes, the proprietor of a popular lunch stop would deny service to customers who offended his arbitrary sensibilities with a loud “No Soup for You!” This is basically the outcome of the Supreme Court’s June decision on standing, TransUnion v. Ramirez. “Standing” in this sense refers to access to the federal legal system; to simplify, to be able to sue somebody, you need to show that you’ve suffered a “concrete” injury. A number of recent standing cases have been about the outer limits of “concrete.” In principle, privacy harms can be concrete, but SCOTUS has been making standing harder, particularly in data breach cases.

And so it goes here. Per a 5-4 opinion authored by Justice Kavanaugh, you don’t have standing to sue a credit agency for flagging you as a terrorist, unless you can show that they also disseminated that information to somebody else. This opinion explicitly overrides a Congressional determination that you do, in fact, have standing in cases like that, arrogating to the Court the right to decide when Congress is and is not allowed to establish when statutory rights are actionable. This was too much for Justice Thomas, who, joined by the court’s liberals, points out in dissent that this both violates separation of powers in the name of preserving it, and turns historical standing doctrine on its head. As Dan Solove and Danielle Citron put it in a sharp critique that builds on Thomas’ dissent, TransUnion is “an activist decision that nullifies Congress’s power to protect consumers and that enables courts to rewrite privacy laws to alter how they are enforced.”

Marx explains how AI systems exploit workers (Part 1)

By Gordon Hull

“Factory work exhausts the nervous system to the uttermost; at the same time, it does away with the many-sided play of the muscles, and confiscates every atom of freedom, both in bodily and in intellectual activity” (Marx, Capital I [Penguin Ed.], 548).

A recent piece by Josh Dzieza in the Verge about the working conditions of those subject to discipline by AI in the workplace is wrenching.  It tells the story of Amazon workers and call center employees and others whose work lives have become an exhausting monotony of ever-intensifying robot-like demands.  The “AI boss” becomes ever more demanding, filling every moment at work with the requirement to be more productive and with absurd metrics for deciding when its demands are met.  Amazon workers report that they are basically ground down by the unsustainability of the pace of productivity; they are either fired or forced to quit, often with injuries.  Call center employees are subject to Orwellian monitoring by a machine that purports (often laughably inaccurately) to monitor and tweak their affect when speaking to customers.  Folks working from home find it impossible to go to the bathroom, because they are forced to sit in front of their computer all the time, which takes frequent pictures of them and reports on their productivity, as measured by lines of code or keystrokes per unit time.

All of this optimized misery reminds us of something important: AI systems are being deployed under capitalism.  A worker subject to an “AI Boss” is not some sort of new-fangled social form; they are fundamentally a worker subject to capitalist exploitation.  What AI has done is enable capitalism to be more intensively exploitative.  The stories in the article echo the stories Marx reports in Capital in his discussion of machinery, as does a lot of the logic.  This suggests that two popular narratives about AI need to be displaced.  First, it may be that AI will someday swoop in and take away everyone’s jobs.  As Dzieza notes, this narrative deflects attention from the pressing reality of deteriorating workplace conditions under AI supervision.  Second, Marx’s work on machines and technology is frequently read (as by the autonomists) through the Grundrisse “Fragment on Machines,” and in a way that emphasizes affective labor and cognitive capital.  The reality of AI today serves as an important reminder that the treatment in Capital is still very important.  Let me explain.

Reading List: Bender and Gebru, “Stochastic Parrots”

By Gordon Hull

Not long ago, Google summarily dumped Timnit Gebru, one of its lead AI researchers and one of the few Black women working in AI.  Her coauthor Emily Bender has now posted the paper (to be presented this spring) that apparently caused all the trouble.  It should be required reading for anybody who cares about the details of how AI and data systems can perpetuate racism, or who cares more generally about the social implications of brute-force approaches to AI.  Bender and Gebru take up a common approach to natural-language processing (NLP), which involves an AI system learning how to anticipate what speech is likely to follow a given unit of speech.  If, for example, I say “Hello, how are,” the system learns by studying a dataset of existing phrases and text snippets that the next word is likely to be “you,” but almost certainly will not be “ice cream.”  How good the computer gets at this game is going to be substantially determined by the quantity and quality of its training data, i.e., the text that it examines.

Bender and Gebru outline the social costs of one approach to this problem, which is basically brute force.  As processing power increases, it’s possible to train computers with larger and larger datasets, and the use of larger datasets reliably improves system performance.  But should we be doing that?  Bender and Gebru detail several kinds of problems.  The first is environmental justice: all that processing power uses a lot of energy.  Although some of it may come from carbon-neutral sources, the net climate cost is significant.  Worse, the NLP systems being produced don’t benefit the people that are going to suffer the most from climate change.  As they memorably put it:

“Is it fair or just to ask, for example, that the residents of the Maldives (likely to be underwater by 2100) or the 800,000 people in Sudan affected by drastic floods, pay the environmental price of training and deploying ever larger English LMs, when similar large-scale models aren’t being produced for Dhivehi or Sudanese Arabic?”

The Normative Valences of Obscurity (part 2)

By Gordon Hull

Last time, I offered some thoughts on Woody Hartzog’s (and co-authors’) development of “obscurity” as a partial replacement for privacy.  On Hartzog’s account, privacy is subject to a number of problems, not least of which is that we tend to think in terms of an unsustainable binary: things are either “private” or “public,” which means that any information you disclose even once is permanently out there.  This doesn’t track how people live their lives: we share information all the time, for all kinds of reasons; however, we reasonably expect it to remain within certain social contexts, and we expect that it will take effort for someone to wrest it our of those contexts.  The latter of these is more or less what “obscurity” indicates.  There is a lot of information out there, but many of those who nominally have access to it don’t actually know enough to do anything with it.  Hartzog analogizes the situation to talking in a restaurant.  People at adjacent tables can likely hear the words you say, but they lack the context for them to be meaningful.  In that sense, communication, even in public, often remains obscure.  This is true both online and off; one of the reasons we need to worry about privacy now is that various technologies make it a lot easier to fill in that context, especially online.  So we don’t become less or more private online, but we do become less obscure.

I concluded by promising a point about latent ambiguity in this context.  Recall that in Lessig’s Code, he suggests that a number of important legal concepts – “privacy” and “fair use” – embed an ambiguity in their meaning.  That is, we don’t know quite what they mean because the people who wrote them into law had never thought through situations analogous to a current one.  For fair use in copyright, for example, it used to be difficult to stop people from making personal copies of works or to meter how many times they used them. So those uses became “fair” and defensible.  If you were accused of violating copyright, you could offer fair use as a defense, and norms arose against pursuing those violations.  Now that technological developments make it easy to stop copying and meter use, we have to confront the question of whether we want fair use for normative reasons, or if we simply had it because of those disappearing inefficiencies. Should fair use protect only use that is inefficient to meter?  There is a clear analogy to obscurity: do we have obscurity because it was difficult to know enough metadata to figure out what the neighbors were gossiping about over the fence, or because we think it’s a bad idea to pry?

The Normative Valences of Obscurity (Part 1)

By Gordon Hull

In a series of articles (and a NYT op-ed; my $.02 on that is here), Woddy Hartzog and several co-authors have been developing the concept of “obscurity” as a partial replacement for “privacy.”  The gist of the argument, as explained by Hartzog and Evan Selinger in a recent anthology piece (“Obscurity and Privacy” (=OP, pagination to the ssrn version)), is that “obscurity is the idea that information is safe - at least to some degree - when it is hard to obtain or understand” (OP 2).  This is because “we should not underestimate how much of a deterrent effort can be” (OP 2), and information that is hard to understand imposes similar costs in terms of effort.  They argue that obscurity functions better as a concept than privacy, in part because it avoids the binarism associated with the public/private dichotomy:

“Because activities that promote obscurity can limit who monitors our disclosures without being subject to explicit promises of confidentiality, the tendency to classify information in binary terms as either ‘public’ or ‘private’ is inadequate. It lacks the nuance needed to describe a range of empirically observable communicative practices that exist along a continuum” (OP 4)

The public/private dichotomy has been the object of sustained criticism, in part because it does not track how people live their lives.  For example, U.S. privacy law tends to regard information that an individual has voluntarily disclosed once as no longer private, as if the context of disclosure doesn’t matter at all.  The obscurity argument is designed to start with this basic thought: we share information all the time, and do so with the expectation that others will manage it appropriately.  This is Helen Nissenbaum’s point about the “contextual integrity” of information; it is also how Ari Waldman starts with his recent reformulation of privacy as trust.  The general stability of these information and contextual flows is behind Lior Strahilevitz’s “social networks” account of privacy and its violation, as well as Dan Solove’s account of how suddenly viral spreads of information online occasions the need to rethink reputation.

Abusive Uses of Big Data, College Admissions edition

Per an investigative report in the Washington Post, growing numbers of colleges are using cookies and other website tracking devices to profile potential students and selectively recruit, including sometimes by income level (there’s a long discussion of how Mississippi State appears to be doing this).  And of course they do so by spending lots of scarce money on for-profit consulting companies which then do the data analytics work to try to guess who is likely to apply, attend, etc.  Some of the worst offenders appear to be schools facing declining enrollments and/or state funds, which therefore have to recruit paying students particularly aggressively.

Along the way, they’re using all the tricks in the big data company playbook.  For example, Capture Higher Ed does the following for its clients:

“Each visitor to the university site gets a cookie, which sends Capture information including that person’s Internet protocol address, the type of computer and browser they are using, what time of day they visited the site and which pages within the site they clicked on …. Every time that person returns to the site, Capture learns more information about them, such as their interest in athletics or the amount of time they spend on financial aid pages, according to promotional videos on the company’s website.  Initially, the cookies identify each visitor by the IP address, a unique code associated with a computer’s Internet connection, but Capture also offers software tools to match the cookie data with people’s real identities, according to the company’s promotional videos. Colleges do this by sending marketing emails to thousands of prospective students, inviting them to click on a hyperlink inside the message for more information about a particular topic, according to the videos.  When a student clicks on the link, Capture learns which email address is associated with which IP address, connecting the student’s real identity to the college’s snapshot of the student’s Web browsing history, Capture executives said in one of the videos”

And many of them seem to have the same arrogant entitlement about their data collection that the big data companies do: when asked about students who might not want to be tracked, the CIO at SUNY Buffalo State College replied that “You have a choice of not interacting at all.”

I wonder how many scholarships they could fund with the money that they funnel into those private consultants?

Algorithmic Housing Discrimination

It appears that HUD, as part of the general initiative to stop enforcing the housing laws it’s supposed to enforce, is poised to allow landlords to hide behind computer algorithms as they discriminate against minority tenants. As Andrew Selbst – who co-authored one of the foundational pieces on exactly this sort of problem – describes the proposed rule change in Slate:

“The proposal, while billed as a mere update to bring regulations in line with the Supreme Court’s 2015 decision, creates entirely new rules for landlords using algorithms. There are functionally two separate defenses being created. First, the proposal allows landlords to use an algorithm as long as its inputs are not “substitutes or close proxies” for protected characteristics and as long as it’s predictive of what it purports to predict—or a “neutral third party” certifies that fact. So if a hypothetical landlord decides to predict number of noise complaints as a proxy for difficult tenants, using music streaming data they somehow obtained, they might find a correlation between preferred musical genre and how difficult a tenant is. Of course, musical preference is not a substitute or close proxy for race, but an algorithm that equates a preference for hip-hop with noise complaints is probably picking up on race as a factor in frequency of noise complaints. Unlike under existing law, under this rule, a landlord would be off the hook even where there may be less discriminatory alternatives. Second, the landlord is also immunized from any discrimination claim if he uses a tool developed and maintained by a recognized third party. These safe harbors supposedly ensure that a model is legitimate and the landlord has not himself “caused” the discriminatory outcome”

A brief discrimination backgrounder: legally, discrimination can be on the basis of disparate intent, or disparate impact.  Disparate intent is what it sounds like: I discriminate against black tenants if I decide not to rent to any, because they are black.  As you can imagine, this sort of discrimination is hard to prove, because not so many people are dumb enough to advertise it.  The more standard strategy is to mask discrimination behind proxies that name race (or some other protected category) without actually naming it.  Given residential segregation, for example, zip code is a pretty decent proxy for race, and so a landlord might systematically disfavor applicants whose previous address is in a specific zip code.  That’s also not ok, because although the rule doesn’t (directly) “intend” to discriminate, the impact of this policy is disparate between racial groups, and so is the same as if it did.

Lessons from Agre on Privacy as Capture (Part 2): Can Foucault Get Past Panopticism?

By Gordon Hull

A couple of weeks ago, I noted my newly discovered appreciation for Philip Agre’s “Surveillance and Capture” and outlined why I think his development of capture (and retreat from surveillance) is particularly applicable to the privacy concerns surrounding big data.  Here, I’d like to suggest that Agre’s distinction is also helpful in understanding a frequently remarked limitation in Foucault.  The limitation is this: a Foucauldian model of disciplinary power treats Bentham’s panopticon as its ideal image.  That image, and the model it subtends, has come under sustained critique over the last twenty years in a scholarship inspired by Deleuze.  Let me start with a review before getting to what capture can tell us about Foucault and surveillance.

Philip Agre on Privacy and Big Data (part 1)

By Gordon Hull

A current paper by Mireille Hildebrandt sent me to a paper from 1994 that I’m embarrassed to say I hadn’t read before: Philip Agre’s “Surveillance and Capture.”  Agre’s paper has been cited over 300 times, but it’s missing in a lot of the privacy literature I know.  After reading it, I’ve decided that’s a mistake, and it’s time to make amends.  I’ll begin by saying why I think Hildebrandt is exactly right to bring the paper up in a context of big data.

Agre starts with quotidian examples of tracking, things like employee ID cards that let systems know where they are, to UPS package tracking.  The core argument of Agre’s paper is that there’s two conceptual models of privacy that need to be distinguished in making sense of phenomena like these: one, the one that we all talk about all the time, is surveillance.  The surveillance model grew out of experience with state bureaucracies, particularly in the Soviet bloc, and features visual metaphors (typically Orwell and Bentham); assumes that the watching is nondisruptive and secret; involves territorial metaphors like invasion of space, which tend to then lead to a dichotomy between coercion and consent; involves centralized orchestration; and is identified with the state.  This model, though ubiquitous, isn’t the only or even the best one.  Agre spends most of his time developing the alternative model, though he notes that “when applied as the sole framework of computing and privacy, the surveillance model can lead to oversimplified analysis” and suggests that it lends itself to caricature and easy dismissals such that “genuinely worrisome developments can be seen as ‘not so bad’ simply for lacking the overt horrors of Orwell’s dystopia” (116).   It’s not hard to say that Agre got this one right: whether it’s the well-trodden problems with notice-and-consent or the stubborn persistence of the “I’ve got nothing to hide” deflection, the surveillance model isn’t adequate to privacy worries now.

The second model, which Agre ties to computer science and information systems management, is built around the idea of “capture.”  The capture model also has five characteristics, which Agre develops as a contrast with surveillance.  Capture: uses linguistic metaphors for human activities; assumes that linguistic parsing of activities is active intervention; defaults to structural metaphors; is decentralized; and is driven by to reconstruct human activity “through [its] assimilation to a transcendent (‘virtual’) order of mathematical formalism” (107).  The core concept here is grammar: the basic mechanism of capture is to pick a set of atomic elements (people, UPS packages) and to then develop a grammar that describes their possible movements or changes.  This model is then imposed – sometimes coercively – on the activities it purports to describe, mechanisms are used to then track and measure the activity according to the grammar; the system can then be further calibrated.

Lockean Data Rhetoric

One of the standard talking points about data gets summed up in the “data imperative:” that the drive to accumulate data seems insatiable, and that firms will pursue accumulating it well beyond and definable economic end.  There’s a lot of literature on why this might be; I’ve tended to approach the question with the resources of Marx in hand: data is amenable to commodification, and tends to function as a form of primitive accumulation.  One might also argue that data functions as a form of capital.  Undoubtedly there is a case of overdetermination here, and trying to disentangle the reasons why the universe of data (broadly conceived) so readily interacts with capital is worth further reflection.

In that regard, I’ve been reading David Beer’s excellent new Data Gaze, which looks to develop an understanding of “how data are seen,” with attention to “the implicit limits and parameters that are being set into data before or whilst they are utilized” (6).  In other words, we need to think about what data is, which is to say (following the same sort of path as the Leonelli paper I discussed, but from a different angle), we need to know what data does.  And what data is/does is heavily influenced by how it’s presented in the marketing and other apparatuses of the data industry.  “Data analytics are almost always part of the operation of capitalism and should be seen through the lens of political economy” (14).  Following Foucault’s lead about the “clinical gaze” and how it developed institutionally, Beer proposes of the “data gaze” that:

Why is Data so Easily Commodified (with a detour into fish)?

By Gordon Hull

Surely one of the more striking features of the rise of data science is how readily it can be incorporated into processes of capitalist valuation, to the point that data may not just be a commodity - it may also be capital.  At one level, this sounds intuitive enough: one might suggest that data presents information about the world, and that knowledge is valuable.  But this view encounters an immediate limitation in that firms seem to want to accumulate data far beyond their ability to value it.  According to this “data imperative,” data is something to be accumulated, even if one has no clear use for it.  An additional problem is that the companies that accumulate all of this data are generally not so interested in whether it represents the world accurately.  They want it because they want to influence behaviors.

So what is going on, and is there something about data itself that makes it particularly suitable for capitalist accumulation?  To answer that question, one needs to have a handle on what data is.  Here, a recent paper by Sabina Leonelli is particularly helpful.  Lenoelli defines data as follows:

“I define ‘data’ as a relational category applied to research outputs that are taken, at specific moments of inquiry, to provide evidence for knowledge claims of interest to the researchers involved. Data thus consist of a specific way of expressing and presenting information, which is produced and/or incorporated in research practices so as to be available as a source of evidence, and whose scientific significance depends on the situation in which it is used. In this view, data do not have truth-value in and of themselves, nor can they be seen as straightforward representations of given phenomena. Rather, data are essentially fungible objects, which are defined by their portability and their prospective usefulness as evidence” (811)

Leonelli is writing from the perspective of the philosophy of science, and her paper is especially useful in that regard because she is able to embed the definition in recent work in the sociology of science and STS, such as Latour. The key takeaway is that data needs to be defined pragmatically – not what it is, but what it does.  And the key implication of that is that we need to stop thinking of data as representative.  Data is generated for a purpose, and in science, a central purpose is portability:

Big Data: From Human Capital to Fridge Capital

By Gordon Hull

In a new paper in Big Data and Society, Jathan Sadowski argues for a shift in how we conceive data.  Typically, it’s viewed as a commodity.  Better, Sadowski argues, to view it as capital.  Following Marx (who offers a basic formula for capital) and Bourdieu (who extends it to cultural and social capital), Sadowski proposes: “Data capital is more than knowledge about the world, it is discrete bits of information that are digitally recorded, machine processable, easily agglomerated, and highly mobile. Like social and cultural capital, data capital is convertible, in certain conditions, to economic capital” (4)  The importance of the formulation is that it allows us to understand why companies spend all their time collecting data even for which they have no planned use: data is subject to the logic of capital accumulation.  Like money, data becomes something the accumulation of which is its own object.  Sadowski: “the imperative, then, is to constantly collect and circulate data by producing commodities that create more data and building infrastructure to manage data. The stream of data must keep flowing and growing.” (4)  That is, “the capitalist is not concerned with the immediate use of a data point or with any single collection, but rather the unceasing flow of data-creating. This point is illustrated by the fact that data is very often collected without specific uses in mind.” (4)

Biometeric Chicanery

Let’s say the state passes a law that says that restaurants may not put worms in hamburgers, and that customers can sue those that do. Your kids eat at the local Annelids franchise on the way home from school, and you later discover that the burgers contain worms. You sue the restaurant, and in response, it says that since your kids didn’t get sick from the experience, it only “technically” violated the law, and you therefore have no basis to either collect damages or force them to stop. That is, on the face of it, a pretty stupid construction of the law, since it would make it totally unenforceable. It is also, more or less, how Facebook, Google, and Six Flags Great America think you should read the Illinois Biometric Privacy Information Act (BIPA).

In a unanimous ruling yesterday, the Illinois Supreme Court said otherwise. The Act regulates companies that collect biometric information like fingerprints and face scans. In the case at hand, the amusement park used thumbprints to identify customers who had bought a season pass, in order to admit them quickly and to stop people from sharing passes (you know the second reason is the real one; since when do amusement parks worry about long lines?). If the company wants to collect such information, it has to inform customers what they’re doing in writing, and get affirmative consent. In particular, the company must “inform[] the subject or the subject’s legally authorized representative in writing of the specific purpose and length of term for which a biometric identifier or biometric information is being collected, stored, and used” (cited in para 20).

This is notice and consent privacy as it should be! There’s a lot of reasons why I don’t think notice and consent privacy actually works – I called it a “successful failure” because the main effect is to teach us that privacy is something we should sell off but without actually protecting privacy. Nobody knows what they’re consenting to, the privacy notices are impossible and long and incomprehensible, and so on. Six Flags et al don’t even get that far: they argue that they don’t actually have to follow the law. So for them, privacy should just be a failure.

Big Data + Health Insurance = Bad

Big data can - and very often is - used to discriminate.  It was only a matter of time before health insurers started using it to predict who might be more likely to get sick, and to charge them more (yes, they've figured out how to circumvent the ACA).  ProPublica has the story here.

Be very afraid, especially if you recently moved to a smaller house or lived on a poor street growing up.  And remember that automated analysis of Facebook likes indicates that if you like hello Kitty, you're more likely to be depressed.  And that can be $$$.

The Big Data Train Chugs On, Predictive Personality Edition

By Gordon Hull

One of the things that marketers like about big data is that they can personalize ads.  That operation is getting increasingly sophisticated.  We’ve known for a while that basic personality traits (like introversion/extraversion) can be predicted from Facebook likes.  I missed this paper when it came out, but some of the same authors as the initial Facebook “likes” paper have now done the inevitable follow-up.  By targeting Facebook ads on the basis of openness and extraversion (where the correlation with likes is fairly robust), they were able to make users 1.54 times more likely to make a purchase (thus about 40% more likely) than with non-“psychologically-tailored” advertising.  The study size, as with most research on FB, is enormous – the study reached some 3.5 million users (a point the authors use to try to defuse some objections).  The authors duly note the murky ethical issues that emerge: on the one hand, you could proactively try to assist people who show signs of depression; on the other hand, weaknesses like susceptibility to addictive gambling or dubious political targeting could be exploited.  That observation is pretty obvious now, in the wake of Cambridge Analytics.  Two other takeaways stood out more to me.  First, the study predicted personality on the basis of only one like.  As the authors note, that means the study likely underestimates the potential effect of psychological targeting.  Second, the authors emphasize that a point that a lot of us have been trying to make for a while: that these results undermine a lot of the current regulatory strategy for privacy. I’ll let them speak:

Hypernudges as Subjectification

By Gordon Hull

In a recent paper, Karen Yeung introduces the concept of a ‘hypernudge’ as a way to capture the way Big Data intensifies design-based ‘nudges’ as a form of regulation.  Yeung’s discussion draws partly from discussions of Internet regulation, partly from literature on design, and partly from legal literature around privacy and big data.  Yeung’s basic argument is that, in the context of big data:

“Despite the complexity and sophistication of their underlying algorithmic processes, these applications ultimately rely on a deceptively simple design-based mechanism of influence – ‘nudge.’ By configuring and thereby personalizing the user’s informational choice context, typically through algorithmic analysis of data streams from multiple sources claiming to offer predictive insights concerning the habits, preferences and interests of targeted individuals (such as those used by online consumer product recommendation engines), these nudges channel user choices in directions preferred by the choice architect through processes that are subtle, unobtrusive, yet extraordinarily powerful” (119)

Ordinary nudging technologies – she cites the humble speed bump – are static.  In contrast, the sorts of nudges provided by data analytics are dynamic, continuously and invisibly updating the choices a user sees.  They work both to make decisions automatically based on what users have done or can be predicted to do, and by guiding decision-making by influencing what choices are available (and how they are presented).  Because of both the dynamism and invisibility, data-driven nudges can be incredibly powerful in comparison to their static cousins. 

Yeung’s paper also enables one to advance a couple of points in the context of information ethics.