What follows is inspired by a recent review by Chad Flanders of Jason Brennan's The Ethics of Voting, not by reading the book itself (and suffers from the usual problems attending such largely second-hand knowledge [although in private correspondence, Prof. Brennan did enlighten me on some further considerations that go into his framework]). According to Flanders Brennan believes that "If one votes, one has an obligation to vote well, and this means that your reasons for voting must be "morally and epistemically justified" and based on the common good (4)." So that "When ignorant voters vote...though they intend to promote the common good, they all too often lack sufficient evidence to justify the policies they advocate." When bad voters vote anyway, Brennan argues, "they pollute democracy with their votes and make it more likely that we will have to suffer from bad governance" (5)." Apparently, "Brennan admits that his argument that bad voters should not vote and (epistemically justified) good voters should vote is elitist (95)." Now, I am no enemy of elitism, but Brennan's approach is misguided because it has too moralized a conception of voting, politics, and democracy (something that in his review my old graduate school buddy, Flanders, misses because he, too, is too moralistic about these matters--it is odd that Flanders congratulates Brennan with his "resolutely hard-headed" view!).
First, on the whole most voters vote according to their perceived self-interest reasonably well. Even exceptions prove the role; according to some so-called value-voters (say, Kansas Born Again Christians) often vote against their interests, but on my understanding these, too, tend not to avoid their self-interest in the voting booth--they just perceive it along axiological dimensions (and their representatives know full well that they should not touch benefits/pork that favor the locals).
Second, most (reasonably large) Liberal Democracies are designed to work properly if folk vote their own perceived interest (see a famous argument by Madison). (They don't need to be maximizing it; all they need to do is vote for the sightly less worse of evils.) Of course, there need to be institutions and office-holders that from self-interest, perhaps, or from their sworn oaths regard and pursue the common good, but this (epistemically and morally demanding task) is generally not entrusted to individual voters. This is, in fact, a remnant of an elitist attitude toward individual voters (again, see Madison).
Third, of course, while individual votes rarely matter (but cf. Gore vs Bush), there are plenty of organizations that ensure that combinations of voters do matter. For example, in the United States there are some extremely successful vote-aggregators (AARP for the elderly, NRA for gun-lovers, Florida Cuban exiles, etc) that elected officials are rarely unwise to ignore for long. This is to say, it is an empirical fact that voters who reliable vote with their group can impact public policy on some matters quite well. (There are, of course, other ways to gain influence many of which outside voting booth.)
Fourth, now it's sadly true that Liberal Democracies are extremely imperfect. But it is only a sentimentalized, that is, superstitious view of politics that can bemoan this in Brennan's terms (e.g., "pollution"). [This is not to deny that we ought to try to improve things!] Politics is no place for purity--purity tends to encourage extremism and fanaticism. This is not to claim that morality has no place in politics (on the contrary), but it must be guided by norms suitable to it. In particular, norms that help us distinguish between those imperfections worth living with and those that are intolerable.
Fifth, I am not against elitism. But political elitism is nearly always dangerously self-serving. While Brennan promotes the view that "bad voters" should abstain from voting [see chapter four of his book]. His argument opens the door to worse: if "morally and epistemically justified" voters with an eye toward the common good are the only (morally) permitted voters then we are just a few (legal) steps away from finding justified legal reasons to disenfranchising the bad voters. This is not an idle concern: in a considerable number of US States folk that have been convicted of crimes are disenfranchised forever. But rule by the discerning few (or Benevolent despots), tends to favor...ahum...the discerning few (and their well-connected friends [cf, alas, Madison]); this tends to come with social and political troubles (unrest, revolution, civil war, etc) worth avoiding at most costs. This obvious (empirical) fact led earlier generations of philosophers to promote other areas of elitism: the arts, sciences, character, etc. (See Gutting for noble articulation of such elitism.)
Finally, the cultural sub-text of Brennan's book is not unsurprising. We live in times characteristic of decline; mutual trust is seeping away and great stock is put in the virtuous expert (Brennan sees voting as akin to performing surgery with a steady hand). This, despite the fact that the experts have made a hash of things (in their role as central bankers and cheerleaders of reckless financial engineering). Yet, if one wishes to reverse the decline it is especially perverse to blame ordinary individuals for our sad state of our affairs; the solution is to do hard and more courageous work of challenging the rent-seeking, elite political, financial, and (alas) intellectual networks that effectively control and reinforce power.