Refereeing is lonely work. When you have done your time-consuming job, you often never find out what the impact of your effort has been on the people principally involved (the author and editor); it is not uncommon that editors do not tell you of their decision. Sometimes you learn what happened to a refereed manuscript because you come across it in the course of your future research. (To be clear, refereeing is a significant part of my work: leaving aside conference-abstract-refereeing, I referee about twenty-five items a year--primarily journal articles, although, of course, book manuscripts and tenure and promotion dossiers are far more time-consuming.) This is why I always ask editors to thank diligent referees (including the ones that recommend reject) on my behalf.
Of course, in practice within 'active' research areas one often develops a decent sense who wrote a piece of research and what happened with it. I find it very disorienting to see work appear in print substantially unrevised that has what I take to be, say, serious argumentative flaws and/or scholarly lacunae. In such cases, I have to really resist the urge to write the editor, "What were you thinking?" I then also wonder, "has the other referee screwed up?" (Sometimes one learns the identity of the other referee!) One reason why I advocate the publication of referee reports with accepted/published work, is that it makes more transparent editorial decisions and guidance--not a trivial matter in context of discussions about citation-networks, patterns of exclusion, and plagiarism. (Recall the proposal here.) I would welcome having my referee reports being visible with my name attached to them, but I realize junior people do have legitimate fears about retaliation. I believe journals should experiment with this.