Kurt Ballard of The Harvard Sports Analysis Collective just released a study today that tried to predict the probability of each team making the playoffs.
Through their research, they found that the Ravens had the 5th WORST chance of making the playoffs in 2015. In this study, the only teams that had worse chances of making the playoffs were Jacksonville, Tampa Bay, Tennessee, and Oakland.
Here is the complete list:
How he did it
“The method that I came up with uses Pro Football Reference’s Approximate Value statistic, the site’s best measure of trying to tease out individual talent. Then, using ESPN’s NFL depth charts, I aggregated each team’s per game approximate value of what I considered to be the ‘core’ makeup of an NFL team: QB, RB, 2 WR, TE, Top 2 OL, the Top-4 ‘Front Seven’ defensive players, and the Top-2 players from the secondary.”
On the surface, the idea doesn’t seem too bad. Data-driven projections often do a good job of stripping away personal biases, which can come into play with film analysis. Watching film is important, but often, especially if the analyst has a predetermined opinion, he sees only what he wants to see. Statistics help to strip those preconceived notions away.
In this case, however, he only looks at a part of the team. While he determined that his aggregated AV (whose creator flat-out says should NOT be a be-all end-all statistic) was significant, it’s anything but comprehensive. The aggregated AV is the “aggregate” of only 13 out of 53 players (or more, if you account for injuries and in-season roster moves).
There is more to an offensive line and a secondary than the top two players. There is more to a front seven than the top four players. This type of system punishes teams for having consistency across the offensive line, for having a rotation of defensive linemen and pass-rushers (like the Ravens), and for having a multitude of injuries at one position (i.e. the secondary).
It may be more accurate if approximate value was NOT a cumulative statistic. AV is normalized by season, but within a season, a player receiving more snaps is more likely to earn a higher approximate value. Which makes sense. A player who receives more snaps is more valuable to the team in a sense, but Harvard Sports does nothing to account for the increase in the value of players who are behind the starters in the depth chart.
What may have worked better is if Harvard Sports normalized players the AV of players like Timmy Jernigan to account for a whole season of starting in place of Haloti Ngata. Or, to account for Jimmy Smith missing over half of the year, using the bundle of players who replaced him and add what their value over the remaining games were.
While it is much easier to run a study that only looks at a “core” of players, it’s not comprehensive enough to just look at those players. Football teams are made up of 53 players, not just the core guys.