Projection comparison done by BP - Fantasy Baseball Cafe 2015

## Projection comparison done by BP

Moderator: Baseball Moderators

### Projection comparison done by BP

I love BP and am 90% sure this is a free link. But I'll hit some of the highlights.

2007 Hitter Projection Roundup

by Nate Silver

Disclaimer: there are as many ways to evaluate projections as there are to create them. This is a SQuiD (Semi Quick-n-Dirty) method that involves looking at some basic descriptive statistics.

I was able to find access to eight projection systems that are either publicly available or I have a subscription to of some kind. These were: PECOTA, Sean Smith’s CHONE, Dan Szymborski’s ZiPS, Tango’s Marcel, and the projections from The Hardball Times (THT), ESPN Fantasy, Rotowire, and RotoTimes, respectively.

...

Finally, my favorite metric, which is based determining which systems give us the best information. Specifically, what I’m doing here is throwing all the forecasts into a regression analysis and determining which ones contribute the most to the forecast bundle. This is basically a combination of how accurate a forecasting system is and how unique it is.

System Coeff t-score
PECOTA +.508 3.46**
ZiPS +.413 2.16**
ESPN +.285 2.49**
Marcel +.237 1.22
THT -.018 -0.11
CHONE -.033 -0.16
RotoWire -.171 -1.05
RotoTimes -.320 -1.86
(Constant +.067 1.15)

The three systems that give you the most positive information are PECOTA, ZiPS, and (somewhat surprisingly) ESPN in that order. In other words, if you had our projections and some of the other projections, the ideal blend would be 5 parts PECOTA, 4 parts ZiPS, and 3 parts ESPN. You could also add in 2 parts of Marcel without hurting yourself. The other projection systems don’t really tell you anything … they might be perfectly fine systems, but they don’t give you any unique information. (Actually, you could almost do better by adding in a NEGATIVE weight from the RotoTimes projections, but that result is not statistically significant).

So, another good year from PECOTA, certainly a good year from ZiPS — Dan does excellent work. I think we can call those two co-champs, but several of the other systems weren’t far behind. We’ll repeat this exercise for pitchers at some point within the next week or two.

I understand that we want a forecasting system to be "unique" so that we get the best chance of identifying true breakout and collapse players, but shouldn't we measure only on accuracy?
In my fridge: Founders Kentucky Breakfast Stout (KBS), Central Waters Peruvian Morning, Bell's Batch 10,000, Dale's Pale Ale, Capital Brewery Amber, New Glarus Moon Man, and Guinness.
Steve-o
Major League Manager

Posts: 1879
Joined: 7 May 2004
Home Cafe: Baseball
Location: Finally back home. A sweet 15 mi from Miller Park.

### Re: Projection comparison done by BP

I didn't read the article, but how many years did they take in to account when doing the statistical breakdown? If it only uses 1 year than I wouldn't really take much from this. If it's a 5 year or so analysis then it would be much more telling
Field
General Manager

Posts: 2549
Joined: 4 Nov 2005
Home Cafe: Baseball
Location: NH

### Re: Projection comparison done by BP

I'll stick with my own....
J35J
Hall of Fame Hero

Posts: 10467
(Past Year: 358)
Joined: 26 Jan 2004
Home Cafe: Baseball

### Re: Projection comparison done by BP

It seems pretty fair considering it is put together by the guys who put out the PECOTA projections. I don't sway too far away from PECOTA until about May. They've been excellent for a long time. I just started looking at Zips this year but they seemed pretty good too. Last year, PECOTA seemed to be a little off on pitchers but this year they were very good. Using their projections I drafted a tandem of Peavy, Escobar, Sheets, Snell and Schilling. The only dud was Anthony Reyes. Their closer projections kind of stunk though. I didn't even use them because they seemed really crazy. They predicted great things from Lidge and Gagne and didn't think much of Papelbon even when he regained his closer role.

But, all in all I am always happy with PECOTA for draft purposes. Like I said, I sway from it come May. I really wish they'd put out a brand new projection set in July, or at least new projections for guys who had less than 100 at bats coming into the season. They are really good at targeting the right prospects, but some do fly under their radar and it would be nice to see how 300 at bats in the big leagues changes the PECOTA projection and comparables.
jfg
Hall of Fame Hero

Posts: 6078
(Past Year: 82)
Joined: 4 Nov 2006
Home Cafe: Baseball

### Re: Projection comparison done by BP

Steve-o wrote:I love BP

Thanks Steve-o, I love you too.

Big Pimpin
Mod in Retirement

Posts: 13716
(Past Year: 7)
Joined: 20 Apr 2005
Home Cafe: Baseball
Location: Building a metric. And being ignorable and stupid.

### Re: Projection comparison done by BP

Big Pimpin wrote:
Steve-o wrote:I love BP

Thanks Steve-o, I love you too.

Before I opened this thread, I thought it was about you BP.
pokerplaya
Kitchen Staff

Posts: 12812
(Past Year: 9)
Joined: 18 May 2004
Home Cafe: Baseball

### Re: Projection comparison done by BP

I'll ignore the love fest going on (even though I started it) and move on to pitchers:

2007 Pitcher Projection Roundup

by Nate Silver

This is Part 2 of 2 of the projection roundup; the first piece for position players ran here on Wednesday. The methodology is as identical to the hitter evaluations as is possible. I use 50 IP as my cut-off point. Pitchers are excluded from consideration if they had no forecast in at least three out of the eight systems. Otherwise, I ran with the data I had, filling in a 4.75 ERA forecast (slightly worse than league average) for missing pitchers.

First, summary statistics for the eight projection systems:

System Mean StDev Corr/Avg
PECOTA 4.38 0.67 .895
CHONE 4.11 0.57 .886
ESPN 4.21 0.81 .875
Marcel 4.41 0.51 .904
RotoTimes 4.21 0.76 .889
RotoWire 4.16 0.76 .884
THT 4.43 0.65 .803
ZiPS 4.33 0.74 .910

SAMPLE 4.27 1.20 N/A

We have a range of about three-tenths (0.30) of a run with respect to leaguewide offensive levels. The important thing, though, is that most of these systems were internally consistent; those that had the highest ERA’s for pitchers also had the highest OPS’s for hitters. One exception was the Hardball Times, which had both the lowest projected OPS’s and the highest projected ERAs; possibly too much regression to the mean there. RotoWire, on the other hand, had high projected OPS’s and low projected ERAs; possibly not enough regression to the mean.

As measured by standard deviation, Marcel and Chone are again the most conservative forecasts. ESPN is the most aggressive.

None of the forecasting systems were especially unique except Hardball Times, which was quite unique. I remember noticing when I downloaded those projections in March that they were pretty different from the other systems.

...

Finally, our optimized forecast bundle based on a regression analysis.

System Coeff t-score
PECOTA +.537 2.58**
CHONE +.374 1.48
Marcel +.192 0.65
RotoWire +.107 0.64
ESPN +.020 0.14
THT -.009 -0.06
ZiPS -.013 -0.07
RotoTimes -.227 -1.29

The best you could have done last year is to bundle PECOTA and CHONE in about a 4:3 ratio. This would have increased your correlation coefficient from .451 using PECOTA alone to .461 with the hybrid version. The other systems wouldn’t really have contributed positively to your results. Taking an average of all eight systems, for example, leaves you with a correlation of .429, which is worse than either PECOTA or CHONE taken alone.

Last year I used PECOTA for my hitters and ZiPS for my pitchers (and still managed to win one money league and come in second in the other). I recall reading another study last year that concluded that PECOTA was the best for hitters, while ZiPS was the best for pitchers. I am beginning to think the earlier poster who wanted a 5 year run down is the way to go.
In my fridge: Founders Kentucky Breakfast Stout (KBS), Central Waters Peruvian Morning, Bell's Batch 10,000, Dale's Pale Ale, Capital Brewery Amber, New Glarus Moon Man, and Guinness.
Steve-o
Major League Manager

Posts: 1879
Joined: 7 May 2004
Home Cafe: Baseball
Location: Finally back home. A sweet 15 mi from Miller Park.

### Re: Projection comparison done by BP

It is always amazing to me that people are better off using Marcel (the monkey) that half the forecasting systems out there. I think it's just a simple 50/33/17 with regression to the mean. Yet it's consistently better than many projections you have to pay for.

I guess that is Tango's point in doing though...
In my fridge: Founders Kentucky Breakfast Stout (KBS), Central Waters Peruvian Morning, Bell's Batch 10,000, Dale's Pale Ale, Capital Brewery Amber, New Glarus Moon Man, and Guinness.
Steve-o
Major League Manager

Posts: 1879
Joined: 7 May 2004
Home Cafe: Baseball
Location: Finally back home. A sweet 15 mi from Miller Park.

### Re: Projection comparison done by BP

Steve-o wrote:Last year I used PECOTA for my hitters and ZiPS for my pitchers (and still managed to win one money league and come in second in the other). I recall reading another study last year that concluded that PECOTA was the best for hitters, while ZiPS was the best for pitchers. I am beginning to think the earlier poster who wanted a 5 year run down is the way to go.

Yeah, there was a study posted here that showed that. I'm not sure I understand the 5 year comment. If you are saying that you'd like to use 5 years of data to predict the best system, I'd disagree. You are trying to predict one year, so the one year prediction is the key. On the other hand, if you say you would like to see 5 sets of one year predictions combined into a single test of the systems, I'd agree that would help show which were consistently good.

The results also show that the difference between the PECOTA forecast and the Marcel forecasts are not statistically significant.
"I don't want to play golf. When I hit a ball, I want someone else to chase it."
GotowarMissAgnes
Hall of Fame Hero

Posts: 5516
Joined: 12 Dec 2003
Home Cafe: Baseball
Location: Happy Valley

### Re: Projection comparison done by BP

GotowarMissAgnes wrote:
Steve-o wrote:Last year I used PECOTA for my hitters and ZiPS for my pitchers (and still managed to win one money league and come in second in the other). I recall reading another study last year that concluded that PECOTA was the best for hitters, while ZiPS was the best for pitchers. I am beginning to think the earlier poster who wanted a 5 year run down is the way to go.

Yeah, there was a study posted here that showed that. I'm not sure I understand the 5 year comment. If you are saying that you'd like to use 5 years of data to predict the best system, I'd disagree. You are trying to predict one year, so the one year prediction is the key. On the other hand, if you say you would like to see 5 sets of one year predictions combined into a single test of the systems, I'd agree that would help show which were consistently good.

The results also show that the difference between the PECOTA forecast and the Marcel forecasts are not statistically significant.

Yeah, I meant 1 year projections over a five year span which is definitely more signficant than just a one year sample size. Again, I didn't read the article (maybe I will now) but I assume they just used a 1-year window.
Field
General Manager

Posts: 2549
Joined: 4 Nov 2005
Home Cafe: Baseball
Location: NH

Next