January 25, 2010
In The Crease
Getting The Three Stars

by Philip Myrland

Many hockey fans do not trust statistics to evaluate goalies. They believe that since goaltender performance and team defense are interdependent, subjective analysis is required and that without watching goalies play it is impossible to tell how good they really are.

It is true that numbers can sometimes be misleading. A goalie may make a lot of easy saves one night and then face a low number of very high-quality shots against the next time out. An observer might be better positioned to estimate the goalie’s actual performance over those two games than someone who looks at the numbers alone. The difficulty with qualitative analysis is that people have imperfect memories. Even someone who watches every game a goalie plays will be hard-pressed to remember more than a fraction of the saves or goals they allowed that season. To take the next step and compare that goalie's performance against that of other goalies is an extremely difficult proposition, especially in the NHL where the margin between a top goalie and an average goalie is about 1 extra goal against every 100 shots.

Quite often at the end of the season the writers who vote on awards do rely heavily on statistics to make their choices. For example, there has historically been a bias towards goalies who lead the league in GAA in the end of the year All-Star voting.

This suggests that the best chance to evaluate a goalie's performance subjectively is probably over a very short period of time. Over a longer period of time many observers allow themselves to be influenced by a team’s defensive success or its win/loss record, and find themselves conflating the performances of both the goalie and team. During a single game it is easier to identify a goalie who is being well-protected by a strong team or a netminder who stands on his head for a weak defensive team.

An example of the latter case was Ondrej Pavelec on December 17th against the Dallas Stars. Pavelec posted a 4.90 GAA and an .894 save percentage in a 6-5 OT win, numbers that are unlikely to earn him much praise from any box score readers or fantasy general managers. Nevertheless after the game Pavelec was praised by players on both teams and named the second star of the game for his efforts in turning aside a high number of excellent scoring chances by the Stars.

Three star voting is the only qualitatively determined record of player performance available for every game played in the NHL. When award voters look at seasonal results they tend to go for those who are near the top of the league in GAA, or those that have a lot of wins and shutouts. Yet when we look at their subjective voting on a game-by-game basis, what do they end up recognizing?

For all goalies with at least 25 games played in 2009-10, I tallied up the total number of three star selections and divided that by games played to get a Star Selection Percentage. Here is the correlation between that statistic and the other traditional goalie stats (all of which are expressed as a per-game rate for the most accurate comparison):

Correlation with Star Selection Percentage:

Winning Percentage:      0.31
Shutouts per Game:       0.32
Goals Against Average:  -0.53
Save Percentage:         0.76

The goalies that most often impressed the observers were not the ones who were most likely to be on the winning team or shut out the opposition. Instead, they were the goalies with the highest save percentages.

Here are the top 10 goalies in Star Selection Percentage this season (min. 25 GP):

Rank Goaltender        Star Selection Percentage
1.   Ryan Miller       51%
2.   Miikka Kiprusoff  50%
3.   Craig Anderson    48%
4.   Dwayne Roloson    39%
5.   Tomas Vokoun      38%
6.   Evgeni Nabokov    37%
6.   Henrik Lundqvist  37%
8.   Johan Hedberg     35%
9.   Ondrej Pavelec    34%
9.   Cam Ward          34%
9.   Roberto Luongo    34%
12.  Niklas Backstrom  33%
13.  Martin Brodeur    31%
13.  Tim Thomas        31%
13.  Carey Price       31%
13.  Ilya Bryzgalov    31%

All of the top seven are having very good seasons. Perhaps the most surprising ranks on this list are Martin Brodeur and Ilya Bryzgalov, both of whom are considered by many to be Vezina Trophy contenders. This could indicate that the viewers consider their jobs to be easier because of their teams' defensive play in front of them. Or it could indicate a bias towards goalies on teams that allow a higher number of shots against.

Media in different cities are likely to have different criteria in picking their stars of the game. The voters in Atlanta games probably either have a soft spot for their goalies or think that they are facing very difficult scoring chances against. The writers in some cities might be homers and tend to stack the list with hometown boys, while others might be fairer in their judgment. Having said all that, the voting is likely to some degree reflective of how much observers think a goalie is carrying their team. Buffalo and Phoenix are both surprise teams, but it seems quite unlikely that Bryzgalov is carrying the Coyotes as much as Miller is carrying the Sabres based on how often each of them has been named one of the game’s three best performers.

One factor that three star voting does not take into account is a goalie’s worst games. There is no penalty for poor outings, even though a bad performance by a goalie can cost his team games. Goalies that deliver consistently good performances would be underrated by this measure.

Finally, these results are interesting to look at for possible insights on viewers’ perceptions of the shot quality allowed by different teams around the league. Jacques Lemaire is known as one of the league’s best defensive coaches. Lemaire left Minnesota to take a job in New Jersey this season, and in his absence the Wild goalies have seen their numbers fall significantly. Niklas Backstrom has dropped from an excellent 2.33 GAA and .923 save percentage to a below-average 2.74 GAA and a .901 save percentage. However, Backstrom’s star selection percentage is a respectable 33% this year, not too far below last year’s 37%. That may be more evidence for a positive shot quality effect under Lemaire.

By similar logic, the voters apparently think the goalies have it easy on Phoenix, New Jersey, Boston and Detroit, while facing tough shots against on the Islanders, Atlanta, Carolina, and Minnesota.

Philip Myrland is an author of Puck Prospectus and runs the statistical hockey website Brodeur Is A Fraud. You can contact him at BrodeurIsAFraud@Inbox.com.

Philip Myrland is an author of Hockey Prospectus. You can contact Philip by clicking here or click here to see Philip's other articles.