Should we abolish letters of recommendation?

As another contribution to hiring season, I want to raise an issue - I'm inclined to say "challenge a dogma" - that I've not seen considered before.  I genuinely do not have a settled position on the matter, but I'll try to lay out an argument.  At the end, I'll offer a more modest version of the proposal, but first here's the thought: we should not include letters of recommendation in what we consider in hiring.

Mohan earlier raised questions about "best practices" in letter writing.  The comments on that post were familiar to anyone who has followed discussions of letters for some time.  

Most people find the evaluations to be so wildly inflated as to be nearly useless.

There is enormous variation in what different people thought was useful in a letter - already a problem in that the same letter will help some places and hurt others - but all agreed that it takes serious and idiosyncratic skill to write them well.  If this is right, then to the extent that one takes letters seriously, one is privileging those who are lucky enough to have referees who are good at letter writing.  

Some folks think that the most important thing in the letters is the explanation of the contribution the candidate's work makes to the literature.  But isn't that something the candidate ought to explain?  If, say, Gil Harman can see a deep idea in the work of X, does that tell me something about X or about Harman?  Maybe the fact that Y's writers don't isolate as interesting a point is just because Y's writers aren't as insightful.

When we read letters, we are virtually certain to give credence to pedigree.  I don't know of studies on this specifically, but I can't see how all the usual implicit bias points don't apply.  If I get three letters from famous Princeton professors regarding X, and three letters from non-famous Arizona State professors on Y, what are the chances that I'm immune to a sort of irrational halo effect?  It seems to me that all the related empirical evidence says that I am.  (I know that some people think that it is rational to attend to pedigree at this stage of the process.  I disagree, but if so, you don't need letters.  A cv will tell you who they studied with.)

We do have direct empirical evidence that anonymized applications are better - that by doing so we avoid lots of implicit bias.  But this is nearly impossible if we include letters in the file.  

Finally, if all letters lie - inflate, mislead, paint overly optimistic pictures: pick your favorite euphemism - then we are going to be influenced at least partly by who is the most skillful liar. (Again, implicit factors. Sure, you think you ignore the inflated adjectives.  But I don't believe that you do entirely.)

So given the fact that there is enormous possibility of being influenced by irrational factors, together with the widely accepted fact that there is very little of value to be gained from them - and let's not forget the huge amount of labor that goes into producing the damned things - why not skip them?  

Is that too radical a suggestion to consider?  Well, I'll try a more moderate version.  The way that letters are used seems to magnify their bad influence. That is, most people use them at the early stages.  We make initial cuts on the basis of cv, letters, and dissertation description, without reading actual work by the candidate.  With so little else to go on, it seems inevitable that letters will have a very significant bearing on these initial cuts.  So what if we didn't include them at this stage.  Let's require a substantial "letter" on research from the student herself - say a 5 page dissertation (and other research) summary.  That way we capture the non-inflated-adjective content of the letters and make the student explain it herself.  then we have an administrator remove the letters to a separate location, and provide the committee with anonymized cvs, cover letters, and research statements.  (I'd black out both names and institutional names.)  

We make our first cut on this basis.  Then we start in on papers.  Only when we have a small group that have survived an initial reading of the actual work do we get letters.  This way, the letters might add some context to a judgment that is based on the student's actual work.  They might raise concerns or ideas that we missed in our assessment so far.  But pedigree, skillful writing, insight of the letter-writer, etc. are far less likely to take over and be the main consideration that makes a decision for us.

I am still tempted by the radical proposal, but I can't see anything to indicate that the modest one isn't better than the current system.  Tell me what I'm missing.