I think it all depends on your league's expectations. If the large majority of the owners expect something more reasonable, then it seems that you should do something about it. Using a moves limit could be one way to go, but you have to be careful about being too restrictive though. And it still wouldn't necessarily prevent a smart owner from saving this same strategy just for the playoffs and/or the final stretch of the regular season in order to get into the playoffs or get a bye.
Other ways to approach it is to use different set of counting stats that makes the strategy less effective. Some would go w/ mostly ratios, instead of cumulative stats. Some would just add some penalty stats, eg. L, Blown Saves, pitching BB, CS, batting K, etc, to balance things out. Make sure you use good games/IP minimums too though, especially if you go w/ mostly ratios. You could also consider a good IP limit too -- something like 9IP/day for 6 days = 54IP.