Oliver Perez & "The Consistency Factor"

by robcast23 on Mar 26, 2009 5:27 PM EDT Comment 20 comments

The recent post about Ryan Church and the resulting idea of quantifying "streakiness" got me to thinking about Oliver Perez, the reputed "King of Inconsistency". Ask anyone about Perez and almost to a man the first words out of their mouth will go something like "all or nothing pitcher", "too up and down", "roller coaster ride", etc.

King Oliver (sans the Creole Jazz Band)

However, I've been trying to get to the heart of this issue for a while because I've had a suspicion that hes not as bad as people think. So I set out to looking through existing stats and found a couple ok ones that measure "clutchness" based on LI (Leverage Impact) but nothing that really captured consistency. So I decided to try it myself and I started thinking about how I might actually quantify consistency (or the lack thereof). At first, I represented his 2008 gamelog graphically to see if I could gauge it by eye. And to evaluate each start I used WPA (Win Probability Added):

Obviously on the surface this chart looks pretty horrible but keep in mind that basically every pitcher's chart is comprised of a series of crests & troughs. Just look at the '08 WPA's of two pitchers considered very consistent (Derek Lowe & Johan Santana) below:

They don't look all that much better. The issue is that the graphical format just doesn't work. So to really nail down this nebulous idea of consistency I dove into the charts and forgive me if I'm getting too in depth here but what I realized is that basically what we're looking for is a Standard Deviation. As I'm sure we all remember from 11th grade math, a higher standard deviation means more data points straying from the mean (or the average) of the data set and a lower SD means more points staying closer to the average. Being able to stay close to a given value intuitively points to consistency.

Therefore to measure a pitcher's start-to-start consistency I will settle on Standard Deviation in terms of WPA calculated over the course of a season. Lets call it Consistency Factor (CONF) so I don't have to keep repeating that definition. So now lets look at Perez's same WPA gamelog chart but in more of a traditional Standard Deviation scatter chart format:

A bit better to decipher. And again, it doesn't look good; we can see that his data points (games started) fall all over, with many landing pretty far from the mean (2008 mean WPA: .013). On the bright side he lands above the mean about as often as below but thats beside the point. Either way we're clearly not going to draw too much statistical precision from the charts alone so we must calculate his Consistency Factor as a value. Don't worry, I got it: In '08 it was .249. And again, the closer this number is to 0, the more starts (measured by WPA) tend to be very close to the same value and thus more consistent. So Oliver Perez has an average variation of about a quarter of a win from his average performance.

But what does that mean? Is that bad? We'd assume so because of his ultra-streaky reputation but since I'm kind of just inventing this stat right now obviously the only way to evaluate its meaning and value is by comparing it with other pitchers. So lets look at Ollie vs. a few mixed samplings of NL starting pitchers. And remember, CONF represents consistency or how much they vary from their median baseline (the lower the better). Keep in mind, this stat does not measure the level of that baseline, just the ability to replicate it. So a pitcher could theoretically give up exactly 10 runs in 3 IP in every single game he started and while he would be a terrible pitcher, he would probably have the best Consistency Factor in the league. The Median WPA represents that baseline, basically their WPA/start (the higher the better) which is a measure of the quality of performance:

Some very interesting findings here:

First and foremost, we can definitively observe that believe it or not Oliver Perez isn't that inconsistent.
...and Carlos Zambrano is, to put it mildly...
4 pitchers in the WPA Top 10, one of whom received votes for the Cy Young, had a worse CONF than Perez.
2 of the most consistent pitchers in NL last season were John Maine and Pedro. Like I said, this isn't a measure of performance...
Some pitchers who were more inconsistent than Oliver in '08: Randy Johnson, Brett Myers, John Lannon, Cole Hamels, Matt Cain, Roy Oswalt, Barry Zito, Brandon Webb
Some surprisingly consistent pitchers: Cubs Ted Lilly & Ryan Dempster, Reds Edinson Volquez & Johnny Cueto and randomly Manny Parra
The average Consistency Factor for the sampling: .237
The average Consistency Factor for the Mets '09 rotation: .218

Obviously we have to take all this with a grain of salt. For one, these numbers were all based on the 2008 season, for truly representative values you'd rather have career CONF #'s but that would take more time than I have. This statistic also leans heavily on WPA which is a pretty high level stat in its own right that baseball "purists" may or may not buy into, obviously I do or I wouldn't have used it. And there may be other flaws I'm missing so feel free to let me know. However, I feel Consistency Factor gets the job done as far as quantifying start to start consistency pretty accurately and at the very least sheds some light on a pretty interesting case study in Oliver Perez. Hopefully you agree but you can take it as seriously as you'd like.

Oh and I mentioned career Consistency Factor values taking too long but I did calculate Oliver Perez:

Year	CONF
2003	.250
2004	.229
2005	.258
2006	.289
2007	.270
2008	.249
Career	.255

Its pretty clear that 2006 cemented that reputation as a "roller coaster pitcher" especially with the mechanical breakdown and the resulting demotion to the minors. However, before that he wasn't bad. As we're all used to hearing, Ollie was incredible in '04 and his consistency reflected that. And in his time with the Mets, O-dog really hasn't been as up and down as the average fan thinks. Interestingly enough he was actually more consistent in '08 than '07 though as we all know he was definitely worse last year. So as I was saying he's definitely above the average but by no means is he the posterboy for streaky pitching as he's portrayed. Not only is he not the worst offender in the league, he may not even crack the top 5 in the division. Hell, he's been more consistent than Mr. Perfect himself, Cole Hamels. All in all, hes not a rock by any means but doesn't deserve nearly as much of the scorn as is heaped on him by the baseball media and fans alike.

PS - Forgive me if my Standard Deviation = Consistency idea infringed onto any existing baseball metrics, its impossible to keep up with all of them nowadays...

6 recs | Comment 20 comments | Share

Story-email Email Printer Print

Comments

Great post

I like the idea of consistency vs performance. I think this would be very handy in evaluating 4-5th starter types.

by Lunkwill Fook on Mar 26, 2009 9:19 AM EDT reply actions 0 recs

Good stuff, rec'd

While I agree with your conclusions. I just wanted to point out that I think the reason most fans see Ollie as inconsistent is based on this:
Apr – 4.03
May – 5.58
Jun – 5.28
Jul – 1.38
Aug – 3.52
Sep – 5.79

While Santana shows fluctuations as well,
Apr – 3.24
May – 3.74
Jun – 2.27
Jul – 2.65
Aug – 1.91
Sep – 1.83
these fluctuation are easier to forgive when the high is 3.74. Santana reminds me of KRod “declining K rate.” It’s hard, nigh impossible, to remain consistently ridiclously awesome.

by Sokojoe on Mar 26, 2009 9:58 AM EDT reply actions 0 recs

but thats the point

when compared against johan of course perez looks wildly inconsistent but then put him up against say carlos zambrano, who is considered a very good pitcher in his own right:
Apr – 2.65
May – 2.46
Jun – 5.68
Jul – 1.78
Aug – 7.43
Sep – 7.08

suddenly oliver looks pretty damn good, or at least pretty damn consistent.

plus like i said in the post, every pitchers start to start consistency (or in this case month to month) looks bad on the surface. that was why there needed to be a stat culminating the entire season.

by robcast23 on Mar 26, 2009 9:48 PM EDT up reply actions 0 recs

I hear ya

Like I said, I agree with you assessment, I’m just trying to explore why this myth is propagated in the first place with the average fan. I’m thinking it’s the Mets-centric universe kind of thing. In other words, not realizing that most pitchers are “inconsistent” just like bemoaning how injuries only happen to the Mets. In addition, the fact that he has poor control means that when he has a low walk game, fans wonder why he can’t do that every time? Answer, he doesn’t concentrate and therefore lacks consistency, Joe Morgan told them so.

Just a note, Big Z has been labeled as inconsistent.

by Sokojoe on Mar 26, 2009 10:10 PM EDT up reply actions 0 recs

the myth

When the average fan – or, worse, baseball analyst – talks about consistency, it is so often a selective-memory type evaluation.

A lot of fans cannot see past the: “You were awesome on Sunday against the Yankees; how are you so horrible on Saturday against the Nationals?”

One of the biggest problems with the consistency conversation is the lack of quantification. (Btw, that is why this post is effing awesome.) Even when we use actual data to discuss consistency, we might be tempted to look to something like month-to-month ERA (as above). But what is that, really? Six starts at most? While it might be useful to analyze a batter on a month-to-month basis, does it actually tell us anything about a starting pitcher?

The myth is propagated, and perpetuated, by people’s perception. And perhaps it is only further perpetuated by some of the types of attempts that have been made to quantify what consistency is (i.e., “Look at his ERA, man! It’s 1.00 one week and 5.00 the next! 2.00 one week, and 7:00 the next!” Well, that would be just criminal to actually say. But, you know what I mean.).

Rec’d.

batting helmets. batting titles. obp.

by Durelo on Mar 26, 2009 11:31 PM EDT up reply actions 0 recs

Well said

If only there was a network devoted entirely to baseball that taught fans how to better understand the game. Or I guess they could just hire Harold Reynolds.

by Sokojoe on Mar 26, 2009 11:37 PM EDT up reply actions 0 recs

HR

Did someone mention situational hitting?

(I never get tired of posting that link.)

We've got ourselves a ball club, the Mets of New York town!

by kingcritical on Mar 27, 2009 10:10 AM EDT up reply actions 0 recs

I never get tired of reading it.

Hilarious every time.

'Oh yes, I know all about that duty-of-a-citizen stuff. It doesn't go. There are exceptions to every rule, and this was one of them. When a man risks his liberty to come and root at a ball-game, you've got to hand it to him. He isn't a crook. He's a fan.'

by Catsmeat Potter-Pirbright on Mar 27, 2009 10:20 AM EDT up reply actions 0 recs

what a breath of fresh air

I never thought of it that way. Thanks, Harold.

Also, opening paragraph: 7 sentences, 7 exclamation points. A 1:1 sentence to exclamation point ratio is pretty good. But I want more sentences like this one!!!

batting helmets. batting titles. obp.

by Durelo on Mar 27, 2009 10:50 AM EDT up reply actions 0 recs

Is this neccessarily a good thing?

I think his inconsistency was almost a selling point for him because fans thought his struggles were more of a mental/maturity issue and that he’d eventually be able to maintain some of his better numbers. If the pitcher he’s been really hasn’t been very inconsistent then what does that say about the upside he has left?

by Gina on Mar 26, 2009 11:37 AM EDT reply actions 0 recs

This is great stuff

Really interesting analysis. You can debate whether WPA is the best way to gauge a pitcher’s consistency (gamescore might be a nice counterpoint), but I really enjoyed this piece.

by Eric Simon on Mar 26, 2009 9:18 PM EDT reply actions 0 recs

thanks eric

by robcast23 on Mar 26, 2009 9:39 PM EDT up reply actions 0 recs

yea, i find this really interesting

but wpa seems like the wrong stat to use to gauge consistency, especially when using that stat tells us the hamels is more inconsistent than perez. let’s look at hamels month-by-month splits last year:

april: 2.70, may: 4.89, june: 2.61, july: 2.88, august: 2.61, september: 2.84

that’s not very inconsistent. in addition, 9 of hamels’ first 10 starts of the season lasted more 7 or more innings. and hamels had another stretch of 9 straight starts that went 7 or more innings. hamels also did not walk more than three batters in a game in any start in 2008.

looking at hamels’ wpa by individual start shows that wpa is not always a good indicator of the quality of the start. let’s look at some starts:

july 3: 8.2 IP, 5H, 1 ER, and 2 BB – WPA = .434
july 20: 8 IP, 4 H, 2 ER, and 2 BB – WPA = .207
june 22: 7 IP, 6H, 3 ER, and 0 BB – WPA = -.012

these are all fairly consistent starts, at least in terms of being a quality start, but they wildly diverge from his mean WPA, which is .076.

so what would be a better statistic to use? i don’t know, but i’d love to see you (robcast23) try out another stat and see what the outcome is. great work all around.

by englishgrey on Mar 27, 2009 12:19 AM EDT up reply actions 0 recs

that was my main concern

by using WPA i was finding the consistency of not only performance but also the ability to deliver wins, which is definitely good to know but i set out looking for performance alone.

i stayed with WPA because it was very easy to find, relatively well known and i do like it as a stat. however i have already started messing with some new ways to evaluate a start completely in a vacuum, without differentiating in game situation. so yeah i think a second version of this post could definitely be on the way. i also came across a lot of interesting tidbits while calculating this stuff that just didn’t fit into this first try…

by robcast23 on Mar 27, 2009 11:43 AM EDT up reply actions 0 recs

in fact

excellent recommendation by eric, it’ll probably end up being gamescore as there really isn’t a better “start in a vacuum” evaluator out there (that i’ve found yet).

its funny because i haven’t used gamescore much so i didn’t think of it until you said that and in my search i started to develop my own version assigning point values and such. but maybe i’ll just defer to bill james…

by robcast23 on Mar 27, 2009 11:48 AM EDT up reply actions 0 recs

yes

probably can’t go wrong deferring to bill james.

i did some comparing of gamescores between hamels and perez, and i found that hamels had 6 occassions last season where his gamescore in consecutive starts rose or fell 30 or more points, while perez had 10 occassions where his gamescore rose or fell 30 or more points.

so if you’re defining consistency in terms of game to game performance, then that might be one way to look to see which pitchers are more consistent. but when fans call perez inconsistent, i think they’re also talking about how he can have stretches of poor quality starts (like three starts with gamescores below 50). technically, those starts might be “consistent” compared to each other, but when we look at the season as a whole, they clearly represent a bout of inconsistency.

i don’t know how you’d represent those inconsistent stretches statistically, but maybe it’s something to consider.

by englishgrey on Mar 27, 2009 12:17 PM EDT up reply actions 0 recs

yeah

definitely going with gamescores, i’ve already calculated perez and i’ll start to put together another database over the next few days.

as far as stretches of inconsistency, i don’t think theres much that can be done to demarcate them under the current format of game to game analysis. to really isolate them we’d have to group starts into packets of 3 or so, get an average for each packet and then calculate the standard deviation of the packets. that would represent more of a bi-weekly consistency factor, accounting for streaks or stretches. another interesting thought that i’ll mess around with.

by robcast23 on Mar 27, 2009 1:04 PM EDT up reply actions 0 recs

uuhhhhhh....

thats like, a lot of math. can’t you just tell me if OP is the roxorz or the suxorz?

HELLO HELLO MR WILPON. WE WANT THE MANSION NOT THE CONDO.

by kendynamo on Mar 27, 2009 10:14 AM EDT reply actions 0 recs

Haha, my impression as well.

"I dunno. I never smoked any Astroturf"
-Tug McGraw

by squid92 on Mar 27, 2009 11:45 AM EDT up reply actions 0 recs

It should be remembered

that ultimately “consistency” is only good if it means winning more games. Most people take it for granted that consistency is good.

But there is strong evidence that consistency is NOT good. It’s a few years old, but check out Gassko’s article showing why, given equal aggregate performance, the inconsistent pitcher is better.

Also, cheese.

by Jeff Sackmann on Mar 27, 2009 10:00 PM EDT reply actions 0 recs