Vegas Over/Unders vs. PECOTA vs. Chaos Theory

As spring training finally crawled out of the offseason muck, we were gifted with a pair of predictions for the upcoming season. One came from Baseball Prospectus' PECOTA projections, the other from Las Vegas sports books and their over/unders for 2013.

These two outfits have very different aims and audiences, and so one would expect their suppositions to differ accordingly. You might also conclude, sight unseen, that BP's calculations would be far more accurate. The truth, as truth tends to be, is a little more complicated.

Let's take the Mets (please). As of this writing, Vegas has the 2013 Mets' O/U at 74 wins. Here's where the bookies set their over/under in the previous four seasons.

Averaged out, Vegas was off the mark by 6.65 games over this period, although this figure is skewed by a wild over-prediction for 2009. If you exclude this boo-boo, Vegas was off by just 2.33 games each season from 2010 through 2012. However, the case of 2009 provides an example of how these O/U's tend to operate: When they're off, they tend to be off by miles.

Looking at all teams over this period, the accuracy of the Vegas O/U's was relatively stable, ranging from a high of 8.76 games off in 2009 to a low of 6.6 the next season. But these averages hide a few doozies, the '09 Mets being one of the more egregious examples. Vegas committed an almost identical error in 2012 with the Red Sox; they too were assumed to win 89.5, but only managed 69 victories. On the opposite side of the spectrum, the 2012 Orioles and A's saw their win totals underestimated by 24 and 22.5 games, respectively.

In all of these cases, it's fairly obvious why Vegas missed the boat: Nearly everyone did. The 2009 Mets and 2012 Red Sox were both teams that had been competitive for several seasons in a row. Few saw either team falling off the cliff so dramatically; Sports Illustrated even picked this Mets team to go to the World Series. Vegas couldn't have accounted for the biblical rash of injuries that doomed the 2009 Mets, or the post-collapse hangover that plagued the 2012 Red Sox (even if the hiring of Bobby Valentine should have been a a red flag). Conversely, like most fans, Vegas couldn't foresee the dramatic turnarounds experienced by Oakland and Baltimore last year.

The sports books appear to be most on the mark when it comes to teams that are consistently good or consistently awful. For example, the lowest figure they've given for the Yankees from 2009-2012 was 91.5, and on average they've missed the Yanks' actual win total by only 3.5 games. Other teams they've been largely accurate with during this time are the Braves (3.625), Cardinals (5.375), Angels (5.375), Royals (6.625), and Pirates (7).

Where Vegas has erred the most has been with teams that have rollercoastered in the past four seasons. The Diamondbacks (15.125) and the Cubs (11.125) are but two examples of teams that have defied their expectations in both directions. In short, Vegas tends to believe a body in motion will stay in motion, an outlook that doesn't account for the bipolar fortunes of some teams.

But, it's important to note that accuracy isn't really on Vegas's agenda. The over/unders aren't predictions so much as suggestions, and ones whose primary goal is to create betting action. If sports books think Team X might win Y games but believe a figure of Z games will get more people to place bets, they will pick figure Z. In fact, if the sports books were overly accurate and/or predictable, betting would be rendered pointless altogether.

Baseball Prospectus, on the other hand, actually does attempt to tell us how teams and players will perform in the future. So presumably, their numbers are a marked improvement over a compromised system like Vegas's, right?

Not quite. Again, we'll start with the Mets. PECOTA has the Mets penciled in for a far rosier 80 wins. How did they do in years past?

Ignoring the bonkers 2009 season, BP missed the Mets' win totals by a mere 2 games on average (slightly better than Vegas), and they called 2010 exactly right. But as with Vegas, completely writing off outliers like 2009 ignores the larger problems it represents.

From 2009 through 2012, BP's predictions were off by an average of 8.4 games, considerably higher than Vegas' average (7.9). BP demonstrated many of the same failings as the gambling community, missing the totals for last season's Red Sox (23), Orioles (20), and A's (17) by similarly large margins. They also showed a similar inability to deal with rollercoasters like the Diamondbacks (missing them by an average of 15.15 over this period).

The weirdest agreement I found between these two sources came with the Colorado Rockies. Since 2010, Vegas has consistently overestimated their win totals, to the point that their average O/U was off by 11.375. At first, I thought this might be the result of some odd hangup the sports books had on Todd Helton. Then, I looked at BP's estimates and saw they did almost the exact same thing. In fact, BP performed even worse, missing the Rockies' win totals by an average of 12.725.

It seems that over this period, Vegas and BP are more alike than different. BP was no better equipped to tell us about catastrophes like the 2012 Red Sox or Cinderellas like the 2012 Orioles. What does it mean when an outfit that aims for accuracy does about the same as an outfit that wants to sucker you into risking your money? It probably says less about the intellectual integrity of either organization and more about the difficulty of trying to predict a team's ultimate record before April.

My own pet theory is that attempting to predict a team's record before a single pitch has been thrown is fraught with all the pitfalls of chaos theory. When it comes to individual performances, predictive systems like PECOTA are drawn from a range of possibilities and historical comparables. These are reliable enough to give us a good idea of how a player will perform in the season(s) to come. Numbers are more forgiving when we talk about just one player. If PECOTA says Player A will hit 30 homers and bat .325, but he belts 27 homers and bats .315, this would be considered a fairly accurate prediction despite not threading the needle exactly.

An entire team's potential, on the other hand, relies on the range of possibilities for 25 (or more) players. Given the symbiotic relationship of teammates and their contributions to a win, if just one player over- or underperforms his expectations by a measurable amount--even an amount we'd consider within the margin of error--he becomes the proverbial butterfly flapping its wings and causing a tornado halfway around the world. If several players do this, the effect is multiplied. And this is before we account for time, tide, and the affairs of man (and the Mets' training staff). It also doesn't help that in terms of perception, saying a team will win 94 games, then watching said team win only 87, looks far worse.

In other words, when attempting to determine a team's record for an upcoming season, you may want to eschew Vegas and BP in equal measure and go with something more reliable. Like picking numbers out of a hat.