Well, the hitters do "take away" stats, in that they take away the potential of the cumulative stats. For example, if you have Mondesi in at Of and he hits 4 bombs in a month when you could have had Gibbons and he hits 8, then you lose 4 home runs. It's an implicit cost of choosing Mondesi, which you didn't consider in your argument. Remember, every pick you make has an opportunity cost. You gain the skills of the player you take, but lose the skills of the players you could have taken. Pitchers and hitters both are affected by this.
"If I have seen further it is by standing on the shoulders of giants."