Data, confidentiality, and PGR

Yesterday many of us here at New APPS endorsed a call for a release of the data upon which the PGR is based.  We did this simply on the grounds that public release of data is a standard in social science, that it ensures transparency, and because it allows others to draw additional conclusions that are not supported merely by means and modes.  An objection was immediately raised: reviewers are guaranteed confidentiality.  And merely removing names from review sheets would not solve this issue, because the requirement that one not rank one's own department or the department from which one received the PhD would allow others to infer the identity of referees.  This short note is simply to point out that there are two simple fixes for this problem.

First, one could release the data not correlated by reviewer, but by department.  That is, for each department ranked, the full list of scores assigned that department could be given.  This would put all the scores out there with no threat to anonymity. (Edouard Machery suggested this in comments yesterday.)  This would certainly provide much additional information, but it would not provide all.

Some inferences can be drawn from looking at the pattern of scores of an individual.  One might, for example, decide that reviewer x only takes seriously departments that are strong in logic, and for that reason discount their judgment of departments focused around ethics.  

This suggests a second "fix".  Put out all the scoresheets, names removed, and insert for the two missing departments the mode for those departments.  (The most common score that was assigned to them.)  

Since many others will have given this score, this should protect anonymity.  And it will not substantially change the data.  And if both versions are released, one could evaluate individual reviewers based on this second listing and do new statistical evaluations based on the first.  

I have no firm view on how much of interest would be learned by releasing data in this form, but it is worth pointing out that confidentiality concerns do not preclude a release.