Well, here are a few things you could do to make a roto league more real-life ish...
A. Differentiate between the 3 OF positions
B. Don't allow for a Util. spot where a DH could be placed. Instead, do your regular 8 positions, maybe an IF (Only people who have eligibility at at least 3 infield positions could play this spot), an OF (someone who has eligibility at all 3 OF poistions), and a Util. (someone who has eligibility at at least 2 of the outfield slots (RF, CF, LF) and 2 of the infield slots (C, 1B, 2B, 3B, SS). This accounts for players who have value on real-life teams, like Ryan Freel.
C. Use more percentage-based categories. Use OBP instead of avg. Use BB/K ratio (hitters that do not K tend to make more productive outs, and a walk is often as good as a hit, plus it takes some of the luck out of hitting.) Use slugging percentage also instead of HR, etc. For base stealers, use SB percentage instead of total SB. Or better yet. use SB - 2 * CS (as a good base stealer should make it more than 2/3 of the time.)
D. As johasmo said, account for quality ABs/sacrifices. Look for your fifth category to be something like percentage of time your batter drives a runner from 3rd in with less than 2 out. Maybe as a 6th category use Sac bunts or something...
E. For your pitching, use more ratio-based categories. BB/K is great. K/9 is great. HR/9 is another good one. To make set-up men valuable, do a HLD/SV percentage. Also, an interesting category to try (though you'd have to do it by hand) would be to do (pitchers winning percentage- team's winning percentage * number of decisions). At the end of the year you would just add up each pitcher's number here to end up with your total. However, to make life easier i'd probably just do winning percentage.
Well, that was a long post