Monday, Inside Higher Ed published an article breathlessly reporting that a "major new study" [summary, pdf] conducted by three Northwestern professors for the National Bureau of Economic Research had shown that "new students at Northwestern University learn more when their instructors are adjuncts than when they are tenure-track professors." Unfortunately, the uptake by IHE and others ignores the one salient fact about Northwestern's 'adjunct' pool that the authors let creep into one of their footnotes: "[a]lmost all classes taught by non-tenure track faculty at Northwestern are taught by those with a longer-term relationship with the university" (p. 9n8 my emphasis).
The study itself is flawed in other ways: 1) the narrow basis upon which these claims are grounded; 2) the authors' failure to consider specific factors about the faculty being studied, their relationship to the courses being taught, or the contracts under they were hired; and 3) the generalizability of the results being presented. Indeed, the authors' provide very little reason to think that 'non-tenure track' faculty at Northwestern are comparable to a similarly named group of faculty at other institutions. As such, the study provides a poor rejoinder to the large body of research that suggests that adjunctification is as bad for students as it demonstrably has been for faculty.
Rather astonishingly, the article congratulates these same adjunct professors, "many" of whom, it is said "will no doubt be pleased by the study's conclusions on their teaching ability." (Because, of course, what all adjuncts want more than anything else is validation that they really are good teachers!) And much of the subsequent discussion has repeated this same basic pattern. "Adjuncts Are Better Teachers," we hear in the Chronicle or "Study: Tenured Professors Make Worse Teachers," says the Atlantic.
Planting this meme is one goal of study's authors, who make a number of suggestive remarks about 'instructional quality,' concluding that they have given administrators at "research universities" evidence in favor of hiring dedicted non-tenure track 'teaching faculty,' since having such faculty may be "educationally beneficial," as well as "efficient" (p. 16).
I felt compelled to look carefully at this study given the post I authored here recently arguing for more studies on the relationship between faculty working conditions and student learning.
Broadly, what the study establishes is: 1) that students 'learn' marginally better (less than .1 grade points) in classes led by "non-tenure track" faculty during their first semester of their first year than in classes led by tenure track faculty, and 2) that students taking classes with non-tenure track faculty are more likely to take another course in the same discipline.
'Learning' here is operationalized in terms of grades earned by students in a subsequent course taken in the same discipline (p. 6). This, it is suggested, offers a good way to measure whether "instructional quality has a lasting impact" or "deep learning" (p. 4). (I'll leave it to readers to decide whether they are satisfied that this measures what it claims to.)
Within this overall frame, the study does provide some more detailed analyses, considering variations within these results by "subject and student qualifications"—that is, for students at different levels of academic strength and for courses in subjects that grade more or less easily.
But it offers no details of any variations that could be connected to faculty working conditions, teaching situation, or teaching approach, and it does not consider how these factors might vary in relation to whether faculty are on tenure lines. There is, for example, no discussion of faculty workload in terms of number of courses per term, nor of class size or class type. Likewise, nothing is said about many times a given faculty member has taught the courses in question, or how recently.
Of course, the study does not set out to investigate the effects of these factors, so it is not methodolgically problematic that it does not discuss them. However, given its repeated suggestions that the differences in student learning it finds should be interpreted as reflecting differences in 'instructional quality,' one can be forgiven for wanting to know whether there might be other correlations with improved outcomes involving factors besides being on the tenure-track—especially factors that could have a more plausibly determinative effect on how much a student learns.
This is particularly true given the number of other studies out there, several of which are mentioned in the introduction to this one, that suggest that the widespread use of adjunct faculty may negatively impact student learning, student persistence, or graduation rates at many types institutions. It would be very helpful to know more about the 'successful' non-tenure track faculty at Northwestern in order to compare it to less successful examples elsewhere.
And indeed, the authors introduce several qualifications regarding the generalizablility of their own results.
First of all, they acknowlede the limitations of studying first term students: "Because a key part of our identification strategy is to limit our analysis to first-term freshman undergraduates, the evidence that non-tenure track faculty produce better outcomes may not apply to more advanced courses" (p. 16).
Secondly, they acknowledge that Northwestern is an exceptional institution with a generally exceptional student body—also speculating (since they collected no data here) that its faculty may be similarly exceptional: "Northwestern University is one of the most selective and highly-ranked research universities in the world, and its ability to attract first-class non-tenure track faculty may be different from that of most institutions. Its tenure track/tenured faculty members may also have different classroom skills from those at other schools. Finally, Northwestern students come from a rarefied portion of the preparation distribution and are far from reflective of the general student population in the U.S." (p. 16).
Thirdly, as noted above, the footnotes reveal that their 'non-tenure track' faculty are for the most part long term employees: "[a]lmost all classes taught by non-tenure track faculty at Northwestern are taught by those with a longer-term relationship with the university" (p. 9n8). And indeed, the more narrowly one focuses on this group, the better the outcomes become: "When we exclude the temporary lecturers and adjuncts, the estimates barely change. Trimming the “one-off” lecturers and adjuncts, we find that for all students and all classes, a non-tenure track faculty member is estimated to increase the likelihood that a student takes another class in the subject by 7.5 percentage points and increases the grade by 0.12 grade points. The results are similarly nearly identical for all other rows in the table (p. 9n8)."
That last is, to say the least, a very interesting result, and it suggests that we would learn a lot more if we had a more detailed analysis of some of the factors I mentioned above.
However, that analysis is largely not conducted. Instead, we are treated to a set of conclusions that depend on a basic equivocation which runs throughout the text and is nowhere examined, between 'teaching-focused' faculty and 'non tenure track' faculty. The data establishes that students working with 'non-tenure track faculty' (very broadly construed) learn more (given the operational defintion of learning at work in the study) than students working with 'tenure track' faculty. Much of the rhetoric in the study's introduction and conclusion suggests that this improvement in learning must be a result of these faculty being better teachers. But this is not in any way investigated, nor is there any attempt to determine why this should be true. And yet, in the end, we are told that "Our results provide evidence that the rise of full-time designated teachers at U.S. colleges and universities may be less of a cause for alarm than some people think, and indeed, may actually be educationally beneficial" (p. 16).
The problem is this: none of the data provided shows that the 'non-tenure track' faculty that were studied are, in fact, 'full-time designated' teachers. Perhaps they are. Perhaps Northwestern hires most or all of its faculty off the tenure line on full time contracts. Perhaps those faculty are not expected to do substantial research as part of their job duties (thus being 'designated' teachers). But these things are neither established nor, if they were the case, would that in any way reflect what is the common norm at 'research universities' throughout the US.
Before this study is taken to ground hiring practices, it would surely be helpful to know what the hiring practices being studied acutally were. Certainly, if I were an administrator who wanted to duplicate these 'improvements,' I would be frustrated that the study didn't provide me with much guidance about how to actually do so.
And, before adjuncts and other non-tenured people tout this thing as a cause for optimism, it's worth emphasizing that while the study leaves no doubt about its argument for the institutional value of hiring adjuncts, it's much less clear about whether universities should value them more highly than they currently do or look to improve their general lot.