Arbitrary Endpoints, Career Numbers, and the Future of Jacob Markstrom

(As an introductory note, it's been a while since I posted here, but a lot has happened in the interim, not just on the ice during the 2014-15 season but especially off of it, with the Summer of Analytics, the emergence of new statistical resources, and some interesting perspectives on goaltending from Steve Valiquette, Chris Boyle, and others. On top of all that, I was also inspired by attending the Ottawa Analytics Conference and meeting many of the people I have worked with and interacted with online.  All that means I'm planning to get back into hockey analytics, which for the time being will include contributing to the Hockey Prospectus Annual and blogging in this space.  Although I was pretty fortunate in having my last post here be on a topic that makes me look pretty good in hindsight (Dubnyk vs. Scrivens), it's long past time to starting pushing that one down the page.)

One of the things I have been thinking about lately, and something I wanted to bring up in my first post back, is the issue of career stats and arbitrary endpoints, to borrow a term used by ESPN sabermetrician Keith Law.  Here's the description from Law's glossary:

#arbitraryendpoints: Also known as cherry-picking, this means choosing one or both endpoints on a series of games to try to analyze a player. I’ve argued that it’s not arbitrary if the endpoint is tied to something specific, like a change in mechanics, an injury, or a recall from the minors, but even so, it’s always dangerous to throw out any data when you want to draw a conclusion.

I definitely agree that it is dangerous to throw out data and that it should never be done for the purposes of supporting an already-reached conclusion.  However, I also agree that it is not always arbitrary to look at splits and segments of data rather than relying on the complete sample if there is good reason to expect that some of the data is not representative of an athlete's true talent level.

There seems to be an increasing trend on Hockey Twitter for people to simply pull up a goalie's career save percentage or career EV SV% and use that as the final verdict on their talent level.  This can certainly be appropriate some of the time, perhaps even most of the time.  I still think that goaltending is above all else a results business, and that statistical measurements remain very powerful methods of evaluating performance.  That's why save percentage over a large sample size is usually a good proxy for a goalie's talent level.  However it is still only a proxy, and the three points mentioned by Law do also apply to goaltenders.  In the same way that Law might discount a pitcher's performance in his first year back from an arm injury, we might have reason to believe that a goalie could be deviating from his historical average because he is not yet at 100% health after coming back from some time off or has made some changes in his game after working with a new goalie coach.

On top of those factors there are others more specific to hockey goalies, such as team effects/shot quality, situational performance (e.g. EV vs. PK), home scorer bias/road performance, usage, etc.  These factors generally do not have a major impact on stats, but margins are so slim in goaltending that even a slight advantage or disadvantage can have an effect on the rankings.  Overall, I think sometimes insight can be missed by looking only at the big picture, and in those cases it is appropriate to take a deeper look.  And I'll start doing that by focusing a magnifying glass on the short but interesting pro career of Jacob Markstrom.

In October 2010, Jacob Markstrom made his first pro appearance on this side of the pond as a member of the AHL's Rochester Americans.  The 31st overall pick in the 2008 draft was coming off of a terrific season with Brynas in the Swedish Elite League, where he not only led the league in save percentage at .927, .010 clear of the next best starter, but did it while turning 20 at midseason.

It looked like the sky was the limit, but success did not come immediately to the young Swede in North America.  One month into the season, Markstrom's save percentage sat at just .891, and in November things got even worse.  In eight November attempts, Markstrom only once managed to end a game with a save percentage above .900.  On December 3 he gave up five goals on 41 shots to drop his seasonal save percentage to .880.  To quote my Hockey Prospectus colleague Matthew Coller, in those early days Markstrom "looked like he was a long, long way from ever being an NHL goaltender."

Yet that Syracuse game would mark a pretty clear dividing point in the statistical record of Markstrom's season.  It was the ninth time in 13 games that Markstrom conceded 4 or more goals against, but from that point on he only allowed 4 or more twice over the rest of the season, and was only sub-.900 in four of the remaining 19 games.

How much that was a result of him adjusting to the league, fixing something in his game, or simply seeing his luck turn around I can't say for sure.  But there is no question that in statistical terms the second half of his season was starkly different than the first:

Jacob Markstrom, 2010-11 (AHL):

First 559 SA:  .880
Remaining 600 SA:  .932

Here's a 20 game rolling average of Markstrom's save percentages over his AHL career:

As the chart illustrates, Markstrom has been consistently excellent in the AHL since his shaky start, posting a .925 save percentage from December 2010 onward to raise his career mark to .920.  Since the odds of a .920 goalie going .880 over 559 shots on chance alone are very, very slim (p<.001), there's a strong case to be made that it was not representative of Markstrom's true talent, and was likely in large part the result of a 20 year old athlete adjusting to the professional game on a weak team in a different country.  Therefore, if you want to look at his AHL performance to assess his current chances in the NHL, it seems reasonable to weight that initial period less than the subsequent sample and conclude that, say, .923-.924 is likely a more accurate number to use than the actual .920 that is dragged down by those two tough months from four and a half years ago.

The discussion of Markstrom in the AHL is relevant not just as an exercise in the meaningfulness of career averages, but also because it invites a parallel to the young goalie's early career struggles in the NHL.  Despite his play in the minors, Markstrom's numbers in Florida and Vancouver rank him among the very worst in the league (the only goalie with both more shots against and a worse save percentage than Markstrom's .896 since 2010-11 is Chris Mason).  Perhaps the only good news is that Markstrom's career NHL shots against sample is still only 1315, more than double his AHL adjustment period but still not a huge amount in goalie terms.  The possibility that he could duplicate his AHL experience and suddenly flip the switch in dramatic fashion at the NHL level to realize his earlier promise remains a tantalizing one.

Goalies don't always follow a linear development path, and Markstrom is yet another example. His early career progression was actually not too far from expected.  As shown above, once he settled into the AHL he has been consistently elite at that level.  In the NHL, he impressed briefly early in '11-12 and then was handed the Florida starting job a bit earlier than expected after Jose Theodore was injured late in '12-13.  Markstrom did falter during that stint, particularly late on, although it should be noted that his poorest results came as member of Florida's league-worst penalty kill unit (.813 save percentage while shorthanded that season).  Only one goalie his age or younger faced more shots against than Markstrom did in the NHL in 2011-12 (Matt Hackett) and only one did the same in 2012-13 (Braden Holtby).  Overall, .906 on 921 SA from the age of 21-23 is a bit below average, but not dramatically so (all goalies aged 21-23 have combined for a .909 save percentage since '07-08).  The error bars are pretty large for a sample of not even 1000 shots, and another mitigating factor was that Markstrom's even strength numbers were decent (War-on-ice has him at .916 at 5 on 5, compared to just .838 shorthanded).

Then came 2013-14, and the wheels fell completely off the bandwagon.  Expected to compete for a starting job, Markstrom got shelled in the first two months and was returned to the AHL in early November with an ugly .877 save percentage.  He ended up being traded to Vancouver at the deadline and got lit up in four more brief NHL appearances there (.868).  Even his AHL season came in below his usual standard (.918).  Not only did Markstrom end up with the NHL's worst save percentage for any goalie with at least 10 GP, but if anything the situational splits make his stats look even worse:

Over the past two seasons, Markstrom has been terrible overall, on the road, at even strength and with the score close.  Looking at shots broken down by danger category (low, medium, and high, courtesy War-on-Ice), it is interesting to note that virtually all of his regression has come on high danger shots against.  There have been some arguments made lately by Matt Cane and others suggesting that high danger shots are the most useful sample for projecting a goalie's play (this is not exactly a new idea, but I'm not sure it's been empirically verified before).  The high danger shots were certainly driving the overall results, but Markstrom's numbers curiously went from above-average against those types of chances to almost laughably bad.

I went back and watched every goal he has given up since the start of the 2013-14 season, and the eye test definitely matches the stats on those high danger chances.  Any goalie has a tendency to look bad if you only watch their goals against, but it was noticeable that Markstrom did not give up tons of goals on point shots or open looks from distance.  His problem was getting destroyed over and over again on odd-man rushes and when the opposition was able to move the puck laterally around the crease area.  Markstrom also gave up 10 goals on rebounds in 16 games, which seemed mostly to be a result of him losing track of the puck around the crease or not responding well to the second chance opportunity.  He also exhibited a weakness on sharp angle shots, either by letting pucks squeak between him and the post or just using poor save selections, such as this goal where he ends up reacting late in a reverse VH.  I would say though that if anything he was probably getting a bit more unlucky than lucky, there were several fluke deflections and quite a few cases of shooters making their shots, although that may be partially because Markstrom has a bit of tendency to over-rely on his blocking ability rather than reacting to the play.

To show this visually, here are the War-on-Ice Hextally shot charts at 5-on-5.  Red means the goalie is doing worse than average, blue means he's doing better.  Note that in the earlier period, Markstrom was fine on shots around the crease and at angles, and struggled more on mid-range shots and shots from distance.  In the second one, distance shots aren't much of a problem at all with barely any red to be seen above the faceoff dots, but in front of the crease it was an absolute bright red goal-fest.

2010-11 TO 2012-13:

2013-14 TO 2014-15:

The statistical splits are so different than it is surprising that they came back-to-back and belong to the same goalie.  Variance is of course huge on these types of small samples, but if the changes were entirely due to randomness alone then that was quite the turn of luck.

If I knew nothing about Markstrom other than what he did since in the NHL since 2013-14, I would say that he looks like a minor league goalie that's in over his head at the NHL level and most likely isn't good enough to make tough saves against top-level shooters.  However, the earlier sample and Markstrom's dominant minor league performance don't easily support that conclusion.  It still seems possible that what Markstrom needs more than anything is to focus substantial attention on his technique, and probably also his anticipation on plays around the net.  I definitely think that he should be able to cut down on the number of garbage goals he's giving up from sharp angles and in close.  He's reportedly been working hard this season with the reputable Rolie Melanson, on playing deeper in his net among other things. I found Melanson's analysis of Markstrom's game from that article particularly interesting:

"In the past, I found that he was chasing the game.  When passes went cross-ice, he was always behind the play.  Once he backed up and let his size work and trusted his athletic ability to make saves, then the game got simplified."

I can't disagree with any of that, although it's hard to tell how much Markstrom may have improved given that he only has two starts this season, and one of them didn't go so well.

Some have questioned whether Markstrom has the makeup to be an NHLer.  It's interesting to read the early scouting reports that suggested that mental toughness was one of his strengths as a prospect. On the other hand, while I'm not huge on trying to measure a guy's mental state, especially for a young goalie where we're already limited to tiny samples for everything, I think it has to be at least mentioned that Markstrom has not only taken some time to adapt to both North American pro leagues he's played in, but also has a poor record in playoff games throughout his young career.  This is true not only in the Swedish Elite League and the AHL, but also in international tournaments (U18s and world juniors), where it has become a bit of a pattern for Markstrom to be close to unbeatable during the round robin and then struggle in the playoffs.

International career:
Round robin games:  252/266, .947
Playoff games:  132/155, .852

2010 Elitserien playoffs:  112/124, .903

2012 Calder Cup playoffs:  253/289, .907

That's a combined .875 on 568 shots, which is well below Markstrom's usual rates at those levels of competition.  That doesn't by any means guarantee that Markstrom cannot succeed, it is likely largely random or affected by other factors (e.g. strength of team/opposition), and most of it came five or more years ago when Markstrom was quite a bit younger than he is now.  I wouldn't wager much at all based on these results going forward, but it might be something to at least be aware of if you are the goalie coach and/or sports psychologist employed by a team that has Markstrom under contract.

Those who hold out hope for Markstrom's NHL future could point out that he only just turned 25 years old and therefore still has some potentially good years ahead of him.  The only problem is that goalies establishing themselves late is much rarer than many seem to believe, and a 25 year old goalie is usually already in the prime of his career.  Pekka Rinne might be one of the most favourable possible comparisons for Markstrom, given that Rinne played all of 92 minutes in the NHL through age 25 and managed just .909 through three seasons in the AHL before emerging as first a starting goalie at 26 and then later a Vezina candidate at 28. Of course not everyone is Pekka Rinne, and it's not correct to expect anyone to match the performance of an outlier, but at the same time it is also true that not every goalie prospect has the pre-NHL track record of Jacob Markstrom.

In summary, whether you still believe in Markstrom depends on whether you think his skillset and mental game is good enough for the NHL level, whether you think he can fix some of the obvious issues that have plagued his game recently, and how much you weight his pre-NHL play (and perhaps even his early NHL play) relative to his cumulative NHL performance.  I think Markstrom's AHL dominance and the underlying numbers from his first 1000 or so NHL shots suggest that he should be better than he has shown, particularly over the past two seasons.  Is he a potential future star?  I doubt it, it's probably not impossible but still not too likely given his age and NHL track record, although I do still expect he's good enough to be a decent backup at the very least.  That said, the clock is ticking.

I might still be willing to give Markstrom a shot for the next year or two, especially if I was running a team with limited expectations that was willing to take on risk in exchange for potential future upside (e.g. Buffalo or Arizona), but I wouldn't have a lot of patience given his age and recent performance.  If things don't majorly turn around and soon, then it is likely that better options would be available.  Markstrom will be an RFA this summer, and although the Canucks are reportedly talking about keeping him, it looks like he will remain blocked in Vancouver behind Ryan Miller and Eddie Lack.  It will be interesting to see if anyone decides to give him a shot next season and if he can finally prove that he belongs in the NHL, or if he will forever remain the minor league star that simply couldn't hack it in the big leagues.