FanPost

How much does fWAR Underrate Relievers?


I have read time and time again that fWAR is not kind to relievers, with most sentiments of that kind implying that fWAR underrates relievers as a whole. I will attempt not to measure, but instead to suggest how much fWAR underrates relievers by comparing fWAR to WPA.

First, some background:

One of the joys I take in sabermetrical musings is teasing out the difference between descriptive and predictive stats, and trying to apply them appropriately. My chief frustration with fangraphs, then, is their insistance on using predictive stats to make up pitcher fWAR, while descriptive stats make up batting and fielding fWAR. My brain can't accept it. Unfortunately, I don't want to subscribe to baseball reference, and it seems most AA and SB Nation posters are in the same boat. I hate the verbal gymnastics we have to go through in order to explain that Robert Allen Dickey, or Johan Santana, or Matt Cain, is nebulously underrated by fWAR. They are like the RBI battles of old, where a great player on a bad team would get an unspecified amount of points added to his or her RBI total in comparative debates, with a different number in each person's head.

Similarly, my brain can't accept it whenever a poster claims that relievers have next to no value compared to a starter. Sure, they don't pitch as many innings, and I would never give Kimbrel the Cy Young over Dickey. But, and maybe this is a false choice, I would prefer 60 innings per year of shutdown bullpen pitching from Jeurys Familia for the next 6 years over 180 meh innings, especially if those innings come in high leverage moments. Anecdotally, I can't think of a single successful team that had a bad bullpen. I agree with top pitching prospects always coming up as starters, but I don't see a move to the bullpen as a failure, even if a guy could possibly be a serviceable #4 or 5 starter (though not a #3 or higher, and only if he's actually good in the bullpen). After watching the Mets dominate with mediocre starting pitching and an excellent bullpen in 2006 (and the converse in 2008 and 2012), after watching the Orioles in 2012 demolish their Pythag with the best bullpen WPA of the last four years, I can't accept the idea that a good bullpen is "the last piece" for a team with everything else in place.

So if pitcher fWAR is predictive, and underrates relievers, let's counter with the most descriptive stat available, WPA. Maybe it will suggest some answers.

Fun Fact: The correlation between WPA and actual wins is 1.0000. I expected (stupidly, in retrospect, because each game ends up adding a +1 or -1 to the team WPA) a correlation close to 1 with some give, but every time I've tested it, the added pitcher WPA and batter WPA for a given season comes out to be a whole number that is exactly the distance on a number line that the team's wins strayed from 81.

Process: The following numbers are from an aggregated data set of every pitcher-season with more than 30 IP from 2009 to 2012. I then categorized each pitcher-season based on % of games started: 50% and above = "starter"; 25%-49% = "swing"; 24% and below = "reliever."

Selected Data:


TOP 25 PITCHERS BY fWAR
Zack Greinke 2009 Royals 9.3 Starter
Justin Verlander 2009 Tigers 8.3 Starter
Roy Halladay 2011 Phillies 8.1 Starter
Tim Lincecum 2009 Giants 8.0 Starter
Roy Halladay 2009 Blue Jays 7.4 Starter
Cliff Lee 2010 - - - 7.2 Starter
CC Sabathia 2011 Yankees 7.1 Starter
Justin Verlander 2011 Tigers 7.0 Starter
Justin Verlander 2012 Tigers 6.8 Starter
Felix Hernandez 2009 Mariners 6.8 Starter
Ubaldo Jimenez 2010 Rockies 6.7 Starter
Cliff Lee 2011 Phillies 6.7 Starter
Clayton Kershaw 2011 Dodgers 6.7 Starter
Cliff Lee 2009 - - - 6.6 Starter
Roy Halladay 2010 Phillies 6.5 Starter
Javier Vazquez 2009 Braves 6.5 Starter
CC Sabathia 2009 Yankees 6.4 Starter
Jon Lester 2009 Red Sox 6.4 Starter
Justin Verlander 2010 Tigers 6.4 Starter
Josh Johnson 2010 Marlins 6.3 Starter
Dan Haren 2011 Angels 6.1 Starter
Adam Wainwright 2010 Cardinals 6.1 Starter
Felix Hernandez 2012 Mariners 6.1 Starter
Dan Haren 2009 Diamondbacks 6.1 Starter
C.J. Wilson 2011 Rangers 6.1 Starter


TOP 25 BY WPA
Zack Greinke 2009 Royals 6.1 Starter
Chris Carpenter 2009 Cardinals 5.4 Starter
Jim Johnson 2012 Orioles 5.4 Reliever
Justin Verlander 2011 Tigers 5.1 Starter
Jonathan Papelbon 2009 Red Sox 5.1 Reliever
Tyler Clippard 2011 Nationals 5.0 Reliever
Roy Halladay 2010 Phillies 4.9 Starter
Fernando Rodney 2012 Rays 4.8 Reliever
Joakim Soria 2010 Royals 4.7 Reliever
Jered Weaver 2011 Angels 4.6 Starter
Ian Kennedy 2011 Diamondbacks 4.6 Starter
Heath Bell 2010 Padres 4.5 Reliever
John Axford 2011 Brewers 4.3 Reliever
Jonny Venters 2011 Braves 4.3 Reliever
David Robertson 2011 Yankees 4.3 Reliever
Tim Lincecum 2009 Giants 4.3 Starter
Ubaldo Jimenez 2010 Rockies 4.2 Starter
Justin Verlander 2009 Tigers 4.2 Starter
Jose Valverde 2011 Tigers 4.2 Reliever
Craig Kimbrel 2012 Braves 4.2 Reliever
Rafael Soriano 2010 Rays 4.1 Reliever
Justin Verlander 2012 Tigers 4.0 Starter
Mariano Rivera 2009 Yankees 4.0 Reliever
Carlos Marmol 2010 Cubs 3.9 Reliever
Joe Nathan 2009 Twins 3.9 Reliever

TOP 25 BY WPA-fWAR
Jim Johnson 2012 Orioles 4.0 Reliever
Tyler Clippard 2011 Nationals 3.8 Reliever
Jose Valverde 2011 Tigers 3.2 Reliever
Jonathan Papelbon 2009 Red Sox 3.1 Reliever
Joakim Soria 2010 Royals 2.7 Reliever
Jonny Venters 2011 Braves 2.6 Reliever
Rafael Soriano 2010 Rays 2.6 Reliever
Vinnie Pestano 2012 Indians 2.5 Reliever
Fernando Rodney 2012 Rays 2.4 Reliever
John Axford 2011 Brewers 2.4 Reliever
Jose Mijares 2009 Twins 2.4 Reliever
Fernando Rodney 2009 Tigers 2.4 Reliever
Daniel Bard 2010 Red Sox 2.4 Reliever
Joel Hanrahan 2012 Pirates 2.3 Reliever
Jeremy Affeldt 2009 Giants 2.3 Reliever
Matt Guerrier 2009 Twins 2.2 Reliever
Ryan Franklin 2010 Cardinals 2.2 Reliever
Heath Bell 2010 Padres 2.1 Reliever
Chris Sale 2011 White Sox 2.0 Reliever
Brian Fuentes 2010 - - - 2.0 Reliever
Darren O'Day 2012 Orioles 2.0 Reliever
Luke Gregerson 2012 Padres 2.0 Reliever
Joe Nathan 2009 Twins 2.0 Reliever
Francisco Cordero 2011 Reds 2.0 Reliever
Eric O'Flaherty 2012 Braves 1.9 Reliever

Synthesis of the Data:

One can say with confidence that, assuming that WPA is a better measure of actual contribution to winning games (a debatable point, I own), fWAR consistently underrates relievers. Some highlights of the data set:

  • 175 out of 179 of the highest pitcher-seasons by fWAR belong to starters; YET
  • 15 out of the top 25 pitcher seasons by WPA belong to relievers.
  • 207 out of the 213 pitcher-seasons most underrated by WPA-fWAR belong to relievers.

If you take WPA seriously, then you have to admit that every year many, if not most, of the pitchers in the league who win for their team the most are relievers. fWAR drastically underrates leverage, which is where games are won and lost.


Problems with this Interpretation:

Most of the problems are the same problems WPA has:

  • a pitcher can get credited by WPA with a number of things that are not of their causing. A base hit where the batter makes a stupid baserunning move and is gunned down would go to the pitcher, as would an error on a popup to second with runners on first and second with two outs in the bottom of the ninth and Alex Rodriguez at the plate. This probably evens out most of the time.
  • So much of a player's WPA is based on when they come in the game. For relievers, this is not directly a result of the pitchers actions.
  • It doesn't take into account the innings-absorbing power of a starter. If Jeurys Familia is only pitching 60 innings a season, he doesn't prevent Pat Misch from pitching the other 120 he might have pitched as a starter. This is a serious rebuttal, but I don't believe there is any sort of provision for this in the WAR calculation, either.

What I am NOT saying:


The problem of using a descriptive stat is that it is not designed to be predictive. The top relievers, who have been so valuable by WPA to their teams, do not tend to do so again the following year. Additionally, plenty of relievers, according to this methodology, are overrated by fWAR. I don't quite have the Excel skills to show that numerically, but we can look at a few case studies:

Name Year Team W L G GS WPA WPA/LI fWAR RA9-Wins ERA- FIP- Start % WPA-fWAR
Bobby Parnell 2009 Mets 4 8 68 8 -1.91 -0.88 0.4 -0.5 130 105 11.8% -2.31
Bobby Parnell 2010 Mets 0 1 41 0 -0.33 0.32 0.7 0.4 73 58 0.0% -1.03
Bobby Parnell 2011 Mets 4 6 60 0 -2.86 -0.5 0.6 -0.1 98 86 0.0% -3.46
Bobby Parnell 2012 Mets 5 4 74 0 -0.41 0.45 1 1.2 65 80 0.0% -1.41


As you can see in the last column, Parnell is consistently overrated by fWAR, even in his excellent 2012 season.

Name Year Team W L G GS WPA WPA/LI fWAR RA9-Wins ERA- FIP- Start % WPA-fWAR
Brian Wilson 2009 Giants 5 6 68 0 1.28 0.86 2.3 1.7 67 60 0.0% -1.02
Brian Wilson 2010 Giants 3 3 70 0 3.88 1.3 2.5 3.3 47 58 0.0% 1.38
Brian Wilson 2011 Giants 6 4 57 0 0.33 -0.18 0.5 0.9 85 90 0.0% -0.17


Brian Wilson swings back and forth between being underrated and overrated by fWAR, though it is in line with swings in his performance (man he was good in 2010).

Name Year Team W L G GS WPA WPA/LI fWAR RA9-Wins ERA- FIP- Start % WPA-fWAR
Brian Stokes 2009 Mets 2 4 69 0 0.09 -0.15 -0.3 0.3 98 113 0.0% 0.39


BRIAN STOKES!!!! I miss Brian Stokes.

Name Year Team W L G GS WPA WPA/LI fWAR RA9-Wins ERA- FIP- Start % WPA-fWAR
Tyler Clippard 2009 Nationals 4 2 41 0 0.79 0.95 0 1.2 64 105 0.0% 0.79
Tyler Clippard 2010 Nationals 11 8 78 0 -1.46 0.59 1.4 1.8 76 80 0.0% -2.86
Tyler Clippard 2011 Nationals 3 0 72 0 5.01 1.72 1.2 3.8 48 82 0.0% 3.81
Tyler Clippard 2012 Nationals 2 6 74 0 -0.19 0.59 1 0.6 94 85 0.0% -1.19


Clippard also swings back and forth, though again it swings with his performance. Perhaps only the very best reliever pitcher-seasons are underrated by fWAR (though they are underrated significantly)?

Let's try a starter!

Name Year Team W L G GS WPA WPA/LI fWAR RA9-Wins ERA- FIP- Start % WPA-fWAR
Zack Greinke 2009 Royals 16 8 33 33 6.07 5.4 9.3 9.8 48 53 100.0% -3.23
Zack Greinke 2010 Royals 10 14 33 33 0.67 1.17 5.1 2.6 100 79 100.0% -4.43
Zack Greinke 2011 Brewers 16 6 28 28 0.56 0.05 4 2 100 77 100.0% -3.44
Zack Greinke 2012 - - - 15 5 34 34 2.82 2.75 5.1 4.7 88 79 100.0% -2.28


As you would expect, pitchers who fail to outperform their peripherals fare poorly in this methodology. Perhaps you could say the same for pitchers on crappy teams that pitch in blowouts.

Name Year Team W L G GS WPA WPA/LI fWAR RA9-Wins ERA- FIP- Start % WPA-fWAR
Sergio Romo 2011 Giants 3 1 65 0 1.57 1.58 2.2 2 41 26 0.0% -0.63
Sergio Romo 2012 Giants 4 2 69 0 2.17 1.2 1.2 2.3 47 73 0.0% 0.97
Sergio Romo 2009 Giants 5 2 45 0 1.47 0.52 1.1 0.3 97 51 0.0% 0.37
Sergio Romo 2010 Giants 5 3 68 0 0.8 0.78 1.1 2 57 77 0.0% -0.3

And then there are pitchers who consistently WPA right in line with their WAR. Go figure.

This FanPost was contributed by a member of the community and was not subject to any vetting or approval process. It does not necessarily reflect the opinions, reasoning skills, or attention to grammar and usage rules held by the editors of this site.

X
Log In Sign Up

forgot?
Log In Sign Up

Forgot password?

We'll email you a reset link.

If you signed up using a 3rd party account like Facebook or Twitter, please login with it instead.

Forgot password?

Try another email?

Almost done,

Join Amazin' Avenue

You must be a member of Amazin' Avenue to participate.

We have our own Community Guidelines at Amazin' Avenue. You should read them.

Join Amazin' Avenue

You must be a member of Amazin' Avenue to participate.

We have our own Community Guidelines at Amazin' Avenue. You should read them.

Spinner

Authenticating

Great!

Choose an available username to complete sign up.

In order to provide our users with a better overall experience, we ask for more information from Facebook when using it to login so that we can learn more about our audience and provide you with the best possible experience. We do not store specific user data and the sharing of it is not required to login with Facebook.

tracking_pixel_9351_tracker