I've actually been considering something similar.
My league's scorring system is 6x6 with the P categories being Ws, Ks, QSs, ERA, WHIP, and Saves
I'm currently near last in the league in WHIP, ERA, Ws and QSs, mostly because I ended up with Liriano, Hughes, J. De La Rosa, and Brett Anderson on my team, and 3/4 are currently on the DL (2 to likely Tommy Johns surgury) and Liriano is still working on getting back to normal, while my offense is in the top 25% in the league. I was thinking that I might be better off keeping my big-gun pitchers (Hamels, Anibal Sanchez, maybe Liriano if he gets back to form, maybe Cueto) and dropping all my other random SPs to pick up all RPs with good k/9 numbers ratios and saves. This would essentially punt Ws and QSs (though I'd still have a few SPs to give me a chance if the other guy's team has a bad week), give me saves pretty easily, and make me much more competitive on ERA and WHIP. I'd likely take a hit in Ks, but I expect to still be competitive against teams with weak Ks. I'd be looking at losing Ks, QS, and Ws most weeks, while hopefully winning ERA, WHIP, and Saves most weeks. Since I usually win more than 50% of the offensive categories each week, I might become more competitive (I currently sit 6th out of 12 teams)
I'd be able to build a pitching staff that looks like this, from the Wire/with what I already have (I've also included the ERA and WHIP numbers so far this season)
SP - Cole Hamels 2.83 ERA/0.97 WHIP
SP - Anibal Sanchez 2.97/1.16
RP - Jonathan Papelbon (closer) 4.34/1.20
RP - Jose Valverde (closer) 3.00/1.22
P - Aaron Crow (setup man) 1.52/1.04
P - Sergio Santos (closer) 1.24/0.93
P - Matt Capps (closer) 4.33/0.89
Bench - Johnny Cueto 2.27/1.06
Bench - Francisco Liriano don't want to talk about his stats
Bench - Tyler Clippard (middleman) 2.08/0.92
I think I might be a more competitive team overall, given my lack of luck with the SPs I started with, after structuring my team this way. What do you think?