Simply saying that closers don't effect your ERA/WHIP as much because they pitch fewer innings is very misleading first of all.
One reason is that if you talk about them in aggregate, then they most certainly do have a telling effect on your ratios. Second of all, how many people start their pitchers for EVERY start? Two closers can most certainly equal the effect of one of your mid-level starters that you don't play all the time.
And if your closer stinks, then don't get me started. sometimes, a good closer is more of the "don't hurt" rather than "help" variety. But, bad closers can definitely hurt you.
Another problem is that basically, at any given time only 30 MLB players can provide that stat. When you consider that 10 of those options are truly awful, and another 5 are shaky, you have a problem. Guesstimating here.
In one of my leagues, there are 11 teams, 4 RP spots. So 44 total spots for relievers, and only half of them can be filled by closers. closers are very scarce in that league.