Oliver Perez & "The Consistency Factor"

The recent post about Ryan Church and the resulting idea of quantifying "streakiness" got me to thinking about Oliver Perez, the reputed "King of  Inconsistency".  Ask anyone about Perez and almost to a man the first words out of their mouth will go something like "all or nothing pitcher", "too up and down", "roller coaster ride", etc. 

King Oliver (sans the Creole Jazz Band)

However, I've been trying to get to the heart of this issue for a while because I've had a suspicion that hes not as bad as people think.  So I set out to looking through existing stats and found a couple ok ones that measure "clutchness" based on LI (Leverage Impact) but nothing that really captured consistency.  So I decided to try it myself and I started thinking about how I might actually quantify consistency (or the lack thereof).  At first, I represented his 2008 gamelog graphically to see if I could gauge it by eye.  And to evaluate each start I used WPA (Win Probability Added):


Obviously on the surface this chart looks pretty horrible but keep in mind that basically every pitcher's chart is comprised of a series of crests & troughs.  Just look at the '08 WPA's of two pitchers considered very consistent (Derek Lowe & Johan Santana) below:

  3335829932_1480b305c7_m_medium 3334993873_9a313d5f79_m_medium

They don't look all that much better.  The issue is that the graphical format just doesn't work.  So to really nail down this nebulous idea of consistency I dove into the charts and forgive me if I'm getting too in depth here but what I realized is that basically what we're looking for is a Standard Deviation.  As I'm sure we all remember from 11th grade math, a higher standard deviation means more data points straying from the mean (or the average) of the data set and a lower SD means more points staying closer to the average.  Being able to stay close to a given value intuitively points to consistency.


Therefore to measure a pitcher's start-to-start consistency I will settle on Standard Deviation in terms of WPA calculated over the course of a season.  Lets call it Consistency Factor (CONF) so I don't have to keep repeating that definition.  So now lets look at Perez's same WPA gamelog chart but in more of a traditional Standard Deviation scatter chart format:


A bit better to decipher.  And again, it doesn't look good; we can see that his data points (games started) fall all over, with many landing pretty far from the mean (2008 mean WPA: .013).  On the bright side he lands above the mean about as often as below but thats beside the point.  Either way we're clearly not going to draw too much statistical precision from the charts alone so we must calculate his Consistency Factor as a value.  Don't worry, I got it:  In '08 it was .249.  And again, the closer this number is to 0, the more starts (measured by WPA) tend to be very close to the same value and thus more consistent.  So Oliver Perez has an average variation of about a quarter of a win from his average performance.

But what does that mean? Is that bad?  We'd assume so because of his ultra-streaky reputation but since I'm kind of just inventing this stat right now obviously the only way to evaluate its meaning and value is by comparing it with other pitchers.  So lets look at Ollie vs. a few mixed samplings of NL starting pitchers.  And remember, CONF represents consistency or how much they vary from their median baseline (the lower the better).  Keep in mind, this stat does not measure the level of that baseline, just the ability to replicate it.  So a pitcher could theoretically give up exactly 10 runs in 3 IP in every single game he started and while he would be a terrible pitcher, he would probably have the best Consistency Factor in the league.  The Median WPA represents that baseline, basically their WPA/start (the higher the better) which is a measure of the quality of performance:

3383218446_801eb56b97_medium  3383218370_d5bd3bea2d_medium   3382401271_abfd40c4ab_medium   3382401293_1c40376c05_medium 

 Some very interesting findings here:

  • First and foremost, we can definitively observe that believe it or not Oliver Perez isn't that inconsistent.
  • ...and Carlos Zambrano is, to put it mildly...
  • 4 pitchers in the WPA Top 10, one of whom received votes for the Cy Young, had a worse CONF than Perez.
  • 2 of the most consistent pitchers in NL last season were John Maine and Pedro.  Like I said, this isn't a measure of performance...
  • Some pitchers who were more inconsistent than Oliver in '08: Randy Johnson, Brett Myers, John Lannon, Cole Hamels, Matt Cain, Roy Oswalt, Barry Zito, Brandon Webb
  • Some surprisingly consistent pitchers: Cubs Ted Lilly & Ryan Dempster, Reds Edinson Volquez & Johnny Cueto and randomly Manny Parra
  • The average Consistency Factor for the sampling: .237
  • The average Consistency Factor for the Mets '09 rotation: .218


Obviously we have to take all this with a grain of salt.  For one, these numbers were all based on the 2008 season, for truly representative values you'd rather have career CONF #'s but that would take more time than I have.  This statistic also leans heavily on WPA which is a pretty high level stat in its own right that baseball "purists" may or may not buy into, obviously I do or I wouldn't have used it.  And there may be other flaws I'm missing so feel free to let me know.  However, I feel Consistency Factor gets the job done as far as quantifying start to start consistency pretty accurately and at the very least sheds some light on a pretty interesting case study in Oliver Perez.  Hopefully you agree but you can take it as seriously as you'd like.

Oh and I mentioned career Consistency Factor values taking too long but I did calculate Oliver Perez:

    Year    CONF
    2003     .250
    2004     .229
    2005     .258
    2006     .289
    2007     .270
    2008     .249
   Career     .255

Its pretty clear that 2006 cemented that reputation as a "roller coaster pitcher" especially with the mechanical breakdown and the resulting demotion to the minors.  However, before that he wasn't bad.  As we're all used to hearing, Ollie was incredible in '04 and his consistency reflected that.  And in his time with the Mets, O-dog really hasn't been as up and down as the average fan thinks.  Interestingly enough he was actually more consistent in '08 than '07 though as we all know he was definitely worse last year.  So as I was saying he's definitely above the average but by no means is he the posterboy for streaky pitching as he's portrayed.  Not only is he not the worst offender in the league, he may not even crack the top 5 in the division.  Hell, he's been more consistent than Mr. Perfect himself, Cole Hamels.  All in all, hes not a rock by any means but doesn't deserve nearly as much of the scorn as is heaped on him by the baseball media and fans alike.

PS - Forgive me if my Standard Deviation = Consistency idea infringed onto any existing baseball metrics, its impossible to keep up with all of them nowadays...

This FanPost was contributed by a member of the community and was not subject to any vetting or approval process.