Blogroll

Become a Fan

« Brazilian music on Fridays: Madredeus | Main | Carolyn's NRC-PGR data »

02 November 2012

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341ef41d53ef017d3d34851e970c

Listed below are links to weblogs that reference Graduate Schools: Combining NRC and Gourmet Data:

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

BLS Nelson
1.

"whereas the Gourmet Report favors those departments with highly-cited ... research."

Is this true? As far as I know, PGR doesn't make direct use of any research impact indicators.

Carolyn Dicey Jennings
2.

That statement is an attempt to cash out what the rankings are based on, rather than the method used in ranking. The method used in ranking is to ask a sample set of faculty to assign numeric values to lists of faculty names from actual departments based on "quality." Although it is difficult to make explicit the basis of reputation, I think it reasonable to suppose that "highly-cited or otherwise well-received research" will be a substantial, and likely necessary, component. It is because citation does not fully account for faculty reputation (see http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.53.8818) that I put in the "or otherwise well-received research" statement. If you know of any work that cashes out the components underlying the Gourmet ranking, that would be of interest. I know of critical work that emphasizes the effect of the PhD granting institution (http://choiceandinference.com/2012/04/17/manufactured-assent-the-philosophical-gourmet-reports-sampling-problem/) and other work that seemed to mostly replicate the Gourmet rankings using an entirely different method, centered on specialty (http://spot.colorado.edu/~pasnau/phil12.pdf), but I haven't seen a full analysis of what accounts for faculty reputation, whether in philosophy or in any discipline. If you know of such an account, please post it here.

BLS Nelson
3.

Thanks for the clarification, Carolyn. I am sure that there is a sense in which reputation is (and should be) based on more than impact. However, I'm not sure that I would agree that impact will be a substantial or necessary component in such evaluations. Although there is not at present any serious attempt to rank departments by impact, I've experimented with Google Scholar and found that some low- or non-ranked departments on PGR are host to scholars with works that have made a sizeable impact.

Human beings being what they are, it seems to me that it is more likely that survey respondents are basing their evaluations primarily on familiarity, and are paying far less attention to actual indicators of merit. And that very same weakness is present in the Pasnau paper you linked to (though his methodology sounds like an improvement over PGR's).

So if we treat 'familiarity' as a confound, I do not think there are grounds for saying that impact is either substantial or necessary to the PGR results.

Carolyn Dicey Jennings
4.

Familiarity itself is strongly influenced by impact, so even if familiarity is a confound, it still might (normally) track impact. But I think that you are right to think that this can go wrong, especially when there are strong divides between and within departments (and disciplines!) with respect to what is considered valuable. This can mean that some scholars with very high numbers of citations (e.g. The Theory of Communicative Action, Volume 2, Lifeworld and System: A Critique of Functionalist Reason, J Habermas, Cited by 14,742 according to Google Scholar) are virtually unknown or thought to be dispensable, in comparison with scholars who are much less cited but have made a social impact on the field, or are otherwise famous for reasons that have little to do with actual philosophical contribution. On the other hand, Google Scholar is an unreliable indicator of citation numbers and citations themselves are unreliable indicators of value, so what else do we have?

BLS Nelson
5.

It's true that results from Google Scholar do include a lot of gray literature, so it's not a foolproof representation of research impact. It is, perhaps, a rough indicator at best. I only appeal to it on the assumption that it is telling us something or other. Maybe not; in which case, grain of salt. (It's a shame, in this connection, that the NRC evidently did not publish data related to research impact for the humanities in the linked report.)

To be sure, we should expect that increased familiarity would correspond to increased impact, and vice-versa; it just sort of makes sense to think that they are intertwined. But I am not confident that we can make a warranted assertion about the relationship between familiarity and impact. As you point out, there are some awkward outliers. I like your example of Habermas's magnum opus in this regard. (Well, I might quibble a bit. The first volume shouldn't be neglected by philosophers interested in philosophy of language, though IIRC the second volume is geared towards sociologists.)

The explanation is probably that publication in prestigious philosophy journals is registered as being of higher importance than actual impact. Publishing in the top journals is the mechanism which attracts both impact and familiarity. e.g., GS registers Habermas's most influential publication in a top-tier philosophy journal ("Struggles for Recognition in Constitutional States" at the European Journal of Philosophy) with 767 citations. That's still an influence which most working philosophers would envy; indeed, with numbers like that, there is no doubt that you can legitimately say of a social and political philosopher that they are out of touch so long as Habermas is not on their radar. But even so, it's nowhere near the numbers you get from his Theory of Communicative Action.

Anyway, I only wanted to warn against an apriori inference from familiarity to impact and vice-versa, and especially I want to warn against making this inference in connection with PGR.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Categories