Why One Great Playoff Run Means Nothing

The vast majority of early season goaltending articles or blogs should be about the variability of small sample sizes, because now is the time of year where fanbases start to freak out and demand immediate action to address their team's perceived lack of goaltending, or conversely, compose detailed narratives about their young team gaining the confidence to come together and believe in one another when they should just be pointing at a .956 team save percentage.

In other words, sell short on Semyon Varlamov and Marc-Andre Fleury and buy stock in Henrik Lundqvist and Devan Dubnyk.

Fortunately, however, a goalie's early season struggles or successes usually fade pretty quickly into the rear view mirror.  Go back to 2012-13 and think about the seasons of Carey Price and Sergei Bobrovsky.  Are you thinking of the first seven games where Price was at .938 and Bobrovsky just .896?  A goalie's early season stats will impact his playing time and they can affect his awards recognition since a hot start can mean a goalie will spend most of the season among the league leaders, but often it just resolves itself as the goalie's play normalizes and the what-have-you-done-for-me-lately media finds another story.

The real small sample size misjudgments usually come in the playoffs.  Postseason samples are equally small, but they don't get large enough to normalize over any single playoff season, and people love talking about playoff runs and using that to impact how they view goalies.  The result of that is that a small sample at the right or wrong time can have a vastly disproportionate influence.

I could come up with dozens of examples, but maybe the best one is Steve Penney.  Penney was a terrible NHL goaltender, but because he managed to be starting for a large market team as a 22-year old rookie out of nowhere and got hot for one spring, he has carved out a spot in the collective hockey memory that is far greater than the anonymity he actually deserves.

Here is his career regular season save percentage record:

1983-84 (MTL): 115 SA, .835
1984-85 (MTL): 1344 SA, .876
1985-86 (MTL): 447 SA, .839
1986-87 (WIN): 134 SA, .813
1987-88 (WIN): 186 SA, .839
Career (TOT):  2226 SA, .859

From 1983-84 to 1987-88, Penney ranked 46th out of 47 goalies in save percentage with at least 2000 shots faced.  And he did that even though 60% of his sample came on a Jacques Lemaire-coached team with Chris Chelios and Larry Robinson playing in front of him.  If you look at little used goalies over the same time frame (e.g. all goalie seasons with 15 or fewer games played), the combined average is .859, which indicates that Penney was essentially a replacement level goalie, probably not appreciably different than some of his minor league playing partners like Mark Holden or Rick Knickle.  But you help get the Montreal Canadiens to the Conference Finals and do it with the media friendly narrative of the unknown who makes it big and that stays on your resume forever.

Percentage-wise, an .859 career goalie posting a .910 on 354 SA is the same level of outperformance as a .905 talent goalie getting up to .940.  And nobody is even three-quarters of the way to that sample size yet in this young season.  Crazy numbers, both good and bad, are essentially guaranteed, and they probably have very little to do with equipment changes or coaching changes or anything else, it's just variance at work.

One final comment on the subject of regression to the mean:  If you haven't read Brian Macdonald's Bayesian approach to analyzing goalies, I highly recommend it.