I have read time and time again that fWAR is not kind to relievers, with most sentiments of that kind implying that fWAR underrates relievers as a whole. I will attempt not to measure, but instead to suggest how much fWAR underrates relievers by comparing fWAR to WPA.

First, some background:

One of the joys I take in sabermetrical musings is teasing out the difference between descriptive and predictive stats, and trying to apply them appropriately. My chief frustration with fangraphs, then, is their insistance on using predictive stats to make up pitcher fWAR, while descriptive stats make up batting and fielding fWAR. My brain can't accept it. Unfortunately, I don't want to subscribe to baseball reference, and it seems most AA and SB Nation posters are in the same boat. I hate the verbal gymnastics we have to go through in order to explain that Robert Allen Dickey, or Johan Santana, or Matt Cain, is nebulously underrated by fWAR. They are like the RBI battles of old, where a great player on a bad team would get an unspecified amount of points added to his or her RBI total in comparative debates, with a different number in each person's head.

Similarly, my brain can't accept it whenever a poster claims that relievers have next to no value compared to a starter. Sure, they don't pitch as many innings, and I would never give Kimbrel the Cy Young over Dickey. But, and maybe this is a false choice, I would prefer 60 innings per year of shutdown bullpen pitching from Jeurys Familia for the next 6 years over 180 meh innings, especially if those innings come in high leverage moments. Anecdotally, I can't think of a single successful team that had a bad bullpen. I agree with top pitching prospects always coming up as starters, but I don't see a move to the bullpen as a failure, even if a guy could possibly be a serviceable #4 or 5 starter (though not a #3 or higher, and only if he's actually good in the bullpen). After watching the Mets dominate with mediocre starting pitching and an excellent bullpen in 2006 (and the converse in 2008 and 2012), after watching the Orioles in 2012 demolish their Pythag with the best bullpen WPA of the last four years, I can't accept the idea that a good bullpen is "the last piece" for a team with everything else in place.

So if pitcher fWAR is predictive, and underrates relievers, let's counter with the most descriptive stat available, WPA. Maybe it will suggest some answers.

Fun Fact: The correlation between WPA and actual wins is 1.0000. I expected (stupidly, in retrospect, because each game ends up adding a +1 or -1 to the team WPA) a correlation close to 1 with some give, but every time I've tested it, the added pitcher WPA and batter WPA for a given season comes out to be a whole number that is exactly the distance on a number line that the team's wins strayed from 81.

Process: The following numbers are from an aggregated data set of every pitcher-season with more than 30 IP from 2009 to 2012. I then categorized each pitcher-season based on % of games started: 50% and above = "starter"; 25%-49% = "swing"; 24% and below = "reliever."

Selected Data:

Synthesis of the Data:

One can say with confidence that, assuming that WPA is a better measure of actual contribution to winning games (a debatable point, I own), fWAR consistently underrates relievers. Some highlights of the data set:

• 175 out of 179 of the highest pitcher-seasons by fWAR belong to starters; YET
• 15 out of the top 25 pitcher seasons by WPA belong to relievers.
• 207 out of the 213 pitcher-seasons most underrated by WPA-fWAR belong to relievers.

If you take WPA seriously, then you have to admit that every year many, if not most, of the pitchers in the league who win for their team the most are relievers. fWAR drastically underrates leverage, which is where games are won and lost.

Problems with this Interpretation:

Most of the problems are the same problems WPA has:

• a pitcher can get credited by WPA with a number of things that are not of their causing. A base hit where the batter makes a stupid baserunning move and is gunned down would go to the pitcher, as would an error on a popup to second with runners on first and second with two outs in the bottom of the ninth and Alex Rodriguez at the plate. This probably evens out most of the time.
• So much of a player's WPA is based on when they come in the game. For relievers, this is not directly a result of the pitchers actions.
• It doesn't take into account the innings-absorbing power of a starter. If Jeurys Familia is only pitching 60 innings a season, he doesn't prevent Pat Misch from pitching the other 120 he might have pitched as a starter. This is a serious rebuttal, but I don't believe there is any sort of provision for this in the WAR calculation, either.

What I am NOT saying:

The problem of using a descriptive stat is that it is not designed to be predictive. The top relievers, who have been so valuable by WPA to their teams, do not tend to do so again the following year. Additionally, plenty of relievers, according to this methodology, are overrated by fWAR. I don't quite have the Excel skills to show that numerically, but we can look at a few case studies:

As you can see in the last column, Parnell is consistently overrated by fWAR, even in his excellent 2012 season.

Brian Wilson swings back and forth between being underrated and overrated by fWAR, though it is in line with swings in his performance (man he was good in 2010).

BRIAN STOKES!!!! I miss Brian Stokes.

Clippard also swings back and forth, though again it swings with his performance. Perhaps only the very best reliever pitcher-seasons are underrated by fWAR (though they are underrated significantly)?

Let's try a starter!

As you would expect, pitchers who fail to outperform their peripherals fare poorly in this methodology. Perhaps you could say the same for pitchers on crappy teams that pitch in blowouts.

And then there are pitchers who consistently WPA right in line with their WAR. Go figure.