If baseball fans were asked to pick a single stat that best represents a pitcher's performance, most would answer with earned run average (ERA), which tells how many earned runs a pitcher gives up per nine innings. The problem with using ERA to measure performance is that it is not a great indicator of a pitcher's skill. Skills are consistent abilities that a pitcher has year in and year out.
Think about everything that can happen in a plate appearance: single, double, homerun, strikeout, walk, out, etc. Now think about how much control the pitcher has over the various outcomes of a plate appearance. It becomes apparent that the defense behind the pitcher has a lot to do with whether or not a single is hit. Doubles are also heavily influenced by the defense's ability, or lack thereof.
After analyzing years of pitching statistics, it has been proven that the three outcomes that a pitcher has almost complete control over are strikeouts, walks, and homeruns. These are three statistics that indicate a pitcher's skill, since these outcomes are the most directly attributable to the pitcher. The statistics are measured as rates per nine innings (K/9, BB/9, HR/9), and are called "peripherals." These are stable from year to year and changes indicate a real change in the pitcher's skill.
Imagine a graph with 2004 ERA on the x-axis and 2005 ERA on the y-axis. Every player pitching at least 30 innings each season would be represented by a point on the graph, and the (x,y) coordinates for each point are (2004 ERA, 2005 ERA). You might expect a correlation close to one since you may assume that ERAs from one year to the next are stable, and that a pitcher's current ERA is a good prediction of future ERA.
The truth is that the correlation of past ERA to current ERA is quite weak and graphically is an uncorrelated scatter plot, as opposed to a line with a positive slope of one. ERA is not a skill that pitchers have. Pitchers have skills that affect ERA, but ERA is oftentimes an externality of both defense and luck. If the same graphs were done with K/9, BB/9, or HR/9, the correlation from one year to the next would be extremely close to one.
ERA is volatile because of how many variables affect it. Besides defense and luck, which are the two major variables outside of the pitcher's control; ERA can fluctuate base on which ballpark you are pitching at, the quality of batters faced, the timing of batting events, and the ability of the bullpen to prevent inherited runners from scoring.
A key statistic in measuring ERA is batting average on balls in play (BABIP), which is almost solely based on defense and luck. The mean Major League BABIP is about .300, meaning defenses can be rated by their BABIP against, or (1-BABIP), the rate at which defenses convert balls hit into play into outs. Good defenses have a strong influences over the pitching staff's ERA, and last season's emergence of the Chicago White Sox staff can be attributed to the most efficient defense in baseball, turning 71.5 percent of balls hit in play into outs, while the league median was 69.8 percent. Individual White Sox pitchers saw their ERA drop with a drop in BABIP, a stat determined by the pitcher's defense. If next year they get lucky as compared to normal deviations from the already great team mean of .715, their ERA will get another superficial push downwards and will lead to overvaluation of the pitcher by ERA-enthusiasts.
Jon Garland had a career BABIP against of .307, slightly above the Major League average of .300. His career ERA was close to 5 before last season, but he amazed many fans when he posted a 3.50 ERA in 2005 and was rewarded with $30 million over the next 3 years. What many did not understand was the drop in ERA was mostly due to the drop of Garland's BABIP to .291, which is attributed to luck and improved defense. Garland did have a real change in ERA due to improved peripherals (HR/9 went from 1.41 to 1.06 and BB/9 dropped from 3.15 to 1.91), but the improvements alone did not make his ERA drop from close to 5.00 to 3.50.
Even though starters often pitch 200 innings a season, that only comprises about 14 percent of the team's innings during the season, and therefore one can see sizable deviations from the mean due to a small sample size. Consequently, one can imagine the volatility of relievers' ERAs who pitch much fewer innings.
I wonder how general managers can keep their jobs when huge dollars are given to guys who have one lucky year that deviates from their mediocre track record while not improving their peripherals. Be sure to use these evaluations to exploit other fantasy owners this coming season in your draft.
