In looking through the forums here at the Fantasy Baseball Cafe, I often read posts asking which of the projections systems are the best. Many people have heard of Baseball Forecaster, PECOTA, the Bill James Handbook, Rototimes, and ZiPS, and those are some of the projection sources I’ve used over the past few years. Two years ago, I looked at 5 different sources. Last year, I looked at 6.
In evaluating these sources, I used Excel to determine a correlation between the projections and the actual results for that season. Individually, I looked at OBP, SLG, OPS, ERA, and WHIP. The main league I play in does not use Batting Average as a category. Instead, we use OBP and SLG as two separate categories. For this report, I’ll use the results for OPS for hitters because OPS is a fairly well-rounded way of determining a hitter’s value. For pitchers, I’ll use ERA and WHIP. The number of players used for the correlations will be included. One note I should mention: when I set up my spreadsheets, there are some players that just don’t make the initial cut. I don’t play in a league so large (or with such big rosters) that a weak hitter like Brad Ausmus or Neifi Perez would be drafted. Also, when setting up a spreadsheet for pitchers, you won’t find the Jose Limas of the world included. For me, some players aren’t worth spending the time on. Anyway, enough intro, here are some results.
HITTERS
2005 Season
For the 2005 season, I used the following projection sources:
• PECOTA Player Forecast Manager (PEC)
• Rototimes (ROTO)
• Bill James Handbook (BJH)
• Baseball Forecaster (FOR)
• ZiPS (ZPS)
Looking at OPS, I have data for 155 hitters who had 300 or more plate appearances in 2005. I used 300 plate appearances as the cutoff point to get a larger sample size. I have similar data for players with 400 or more plate appearances (138 hitters) and 500 or more plate appearances (109 hitters) but for this report, I’ll use the larger sample size. This is how the sources ranked for 2005, with their correlation:
Based on these numbers, for the 2005 season, the Bill James Handbook and PECOTA come out on top with ZiPS, Rototimes and Baseball Forecaster in a group bringing up the rear. There’s very little difference among the bottom three and not a ton of difference among all five. Let’s look at 2006.
2006 Season
For the 2006 season, I added Fantasy Sports Central to the mix (FSC).
Looking at OPS again, there were 151 hitters who had 300 or more plate appearances in my study. This is how the sources ranked, with their correlation:
For 2006, Fantasy Sports Central just barely beats out PECOTA, Rototimes, and the Bill James Handbook. The numbers are so close that I really wouldn’t say there’s much difference among them. Forecaster and ZiPS are off the pace of the top four, finishing in the bottom half for the second year in a row.
The last thing I analyzed for hitters involved using data from both years. Because Excel is limited to 255 rows going across, I couldn’t use 300 plate appearances as the cutoff point because I would end up with over 300 data points when both years are combined (155 hitters for 2005 and 151 hitters for 2006). In order to fit the data into Excel, I used the players I had statistics for who had 500 or more plate appearances for each year, which came to 213 players. Also, I had to throw out Fantasy Sports Central because I didn’t have their projections for 2005. The combined results:
Both Seasons Combined - Hitters
Combining 2005 and 2006, for players with 500 or more plate appearances (213 hitters):
PECOTA and the Bill James Handbook are clearly above Rototimes, ZiPS, and Baseball Forecaster. I don’t claim that this is an absolute and final determination of which projections are the best. This is how the comparisons turned out for me. I’ve read similar analysis on different websites that turned up different results.
PITCHERS
For pitchers, I looked at ERA and WHIP. In the main league I’m in every year, we generally have 10 teams with 6 starting pitchers each, so I don’t look much beyond 75 starting pitchers. If a guy isn’t good enough for the top 75, I delete him from my spreadsheet. For closers, I have statistics on roughly 20 (give or take) each year.
2005 SEASON
Looking at ERA - 97 pitchers (78 starters, 19 closer-types):
ZiPS is comfortably ahead of the rest of the pack, with PECOTA a fair amount ahead of Rototimes and Forecaster. Here you can see how a pitcher’s ERA one year has a low correlation to how the pitcher will do the next year.
Two years in a row for ZiPS and PECOTA at the top of the pack, although Rototimes finished just barely behind PECOTA this time. Then it drops down a notch to Fantasy Sports Central, with the Bill James Handbook (didn’t have pitching projections for 2005) coming in ahead of Forecaster.
Using two years worth of data, ZiPS is ahead of the pack in correlation for ERA, while PECOTA and ZiPS are very close at the top in WHIP.
CONCLUSION
Over a two-year period, looking at 213 hitter projections, all systems finished with a correlation between .604 and .655 for OPS with PECOTA (.655) and the Bill James Handbook (.647) at the top of the list.
For pitching, looking at 192 pitcher projections, the range was from .423 to .523 for ERA, with ZiPS (.523) ahead of the pack. For WHIP, the range was narrower, just .461 to .498 with PECOTA (.498) and ZiPS (.493) a bit ahead of the rest.
In all cases, the ranges of results are narrow. I know some people have a favorite projection source and stick with those projections. I personally like to get as much information as I can when setting up my draft sheets. From my study, I’d recommend PECOTA and the Bill James Handbook for hitter projections and PECOTA and ZiPS for pitchers, but it’s not like they are way ahead of the pack. That’s my two cents on the subject.
Bobby Mueller is a longtime Pittsburgh Pirates fan, as futile as rooting for them is, who currently lives in the Pacific Northwest where Bill Bavasi is running the Seattle Mariners into the ground. Like the t-shirt says, "Baseball is Life." You can find Bobby posting in the Cafe forums as BobbyRoberto.
(18 votes, average: 4.28 out of 5)
Loading ...
Questions or comments for Bobby? Post them in the Cafe Forums!
Want to see your own fantasy wisdom published? Check out the Cafe's Pencil & Paper section!
Cafe Home
Fantasy Football
Fantasy Basketball
Fantasy Hockey
Fantasy Cafe Radio
(18 votes, average: 4.28 out of 5)
Loading ...
