October 15, 2012
From Daigle To Datsyuk
Analyzing Draft Selections

by Corey Pronman

As someone who lives and breathes hockey prospects, it should come to no surprise I commonly see people analyze the NHL draft. This could be those who analyze a specific team's drafting record, a specific pick, drafting trends as a whole, or even a discussion of public prognosticators. I'm writing this column because when it comes to analyzing the draft, and more specifically when it comes to analyzing individual draft picks, I feel that the overwhelming majority of the time, the direction of the analysis is faulty.

The core of the issue is that, not surprisingly, most people tend to rely on results-based analysis. Did the pick pan out or not? How well has this team drafted over the last few years? What was the particular player's production like in junior and in pro hockey? Did the he play up to where he was drafted and/or did he provide value relative to his draft slot?

These are always interesting and important questions to ask, but my contention is that these questions typically require very complex and incomplete answers.

This business—being the business of drafting and developing prospects—is a business of probability, of subjectivity, and of limited information, and it is an industry where so many uncontrollable variables affect the end result.

Marc Foster looked at draft results a year and a half ago for Hockey Prospectus, producing a graph showing the relative value of draft positions based on games played.

The black lines represent the variance of the results from the best-fit line that someone with a brief knowledge of the NHL draft could expect. The lack of a perfectly smooth data line could be for one or two reasons: that the NHL people who make the picks are consistently making bad decisions relative to their draft slot, or there is some form of chance in draft projections. I wouldn't suggest the NHL draft people are a perfectly efficient group of people with no room for fault, but I think it would quite a reach to say that teams investing so much money in this area and employing a large amount of good hockey people would be causing that degree of error based on a knowledge level.

So I believe you have to approach the draft with a probabilistic approach. In an ideal world, one we don't live in, teams would have access to information such as: what is a player's upside and the percentage chance he reaches it, same for his downside, and every degree of output in between with its respective probability. You could then take that data, get an average output and assign a value to the player. We don't have that information and we can't do that analysis, but it doesn't mean we shouldn't approach the nature of prospects as one of probabilities. Vic Ferrari coined this as Type A vs. Type B thinkers. The ability to differentiate between raw results and the fact that true talent level (in this case, we'll say for a particular management team's drafting ability) will have fluctuations of randomness in their draft results through no control of their own.

Now with this in mind, one could still evaluate the results of a draft while adjusting for randomness, but in my opinion the draft is not something that can be effectively analyzed through a results-based analysis, but rather the best way is ideally a process-based analysis.

When drafts and individual picks are evaluated in hindsight, fans, media, and others will usually look back on a particular pick, on a particular draft class, and see what the end result was and determine a verdict based on whatever criteria they use to judge results. This kind of evaluation to me is a mistake. When drafting a player, there is a decision point that has to be made. That decision point is what the evaluation has to be based on. I don't like the argument of "Let's review this pick in five years" because aside from pure randomness that is involved with the draft, prospect development is ignored. The player development process plays a significant if not a gigantic factor from the player's and team's standpoint. Reviewing a result can sometimes mean you give credit to a team who may not have made a good move at the time of the draft but got fortunate in the development process whether through their own doing or not. A good decision can lead to a bad player and vice versa.

The way I've approached judging and evaluating draft picks and the draft in general is information. Specifically, what information led to a particular decision?

Did some particular statistics lead to your decision? If so, what statistics, why are they helpful, and how much they do tell us? Was it something scouting related? If so, what stands out about a player? To what degree are his strengths and weaknesses and how much do you weight particular skills and why? Are there any character aspects brought into the equation, and if so, what are they and why do you weigh them so much either against or in favor of the player? And in an ideal world, you keep asking all those questions and more until you have the full picture on an individual player and how much perceived value a talent evaluator puts on such player.

Being able to evaluate and specifically test a statistic is the easier process. Numbers can be found in large databases of prior example and can usually have their predictive ability determined. Iain Fyffe here at Hockey Prospectus showed basic box car stars in major junior have significant predictive ability to NHL success especially if you adjust for birthdate, team ability, and league quality. Fyffe also showed that while numbers can help, there was certainly a threshold for them in terms of projecting players. This is for a variety of reasons: one is that stats like ice time, Corsi numbers, quality of competition, shooting percentage, and zone starts are not available to provide more context. Another is that players do change as they age as prospects and the dynamic element of player development can cause some issues. There is also a sample size issue in terms of games played for individual players.

Stats do have some use, though, so should they play a part in a draft selection, finding the merit of a particular stat and why it was or was not used and the degree of importance it played in the evaluation of a draft prospect is more easily determined than something like scouting.

Scouting on the other hand is a whole other story. The scouting-based information is subjective material, and while you in theory could count passes, zone entries, puck control, and the like, there's no way you could encapsulate the full picture from a scouting level of a player with numbers. Teams could value certain skills much differently, and see players way differently for whatever reason with no definitive way to prove their reasoning. The end result is the huge variance we often see in opinions on prospects. The kind where I've seen teams have one player ranked as the third best player in the draft, and another team will have that very same player in their third round.

Scout A may want a player because he really likes his hands and vision but also thinks he's a below-average skater and Scout B may advocate against the player because he doesn't see great skill and he doesn't like his skating. The team makes the pick, the player plays well in the CHL, and then fizzles out in the AHL. The reasons on top of the skating might be that he just couldn't keep up to the pro pace and he couldn't handle the physical game. Who was right and who was wrong? You can say Scout A is to blame for advocating, but what if 60 out of 100 very similar players would have panned out without being impeded by the non-skating issues, but for whatever reason, this was one of the ones that didn't? What if Scout B's decision was wrong, but in this case, he looks right? That goes back to the Type A vs. Type B thinkers I mentioned above. In this industry, though, where each GM will only oversee so many first round picks, where teams will have to wait a while to see if their picks panned out, and where scouts have very unstable jobs that have high demand, hockey people don't tend to get the luxury of waiting for long run results to determine their skills as evaluators.

With no way to find the absolute truth in terms of what is the best type of player, NHL teams have to rely on completely subjective calls to determine which skills that will emphasize, and then of course their evaluations of the degrees of those in certain players skills are completely subjective. Management's job is to choose a specific direction and a set of skills they want to prioritize for player acquisitions. I wrote about scouting for possession skill, the skill I highly emphasize in my evaluations, and wrote how even teams who go after that skill view the tools that define a possession player differently. That takes it even further, in that even when scouts agree on the skill they want, you can't simply assume it's easily defined. Do you want to play a speed game? If so, do you want good skaters (skating has multiple elements to it as well), or hard-working/energy players, or if you want both, what's the right mix? Which is more important and by how much?

I did choose to emphasize puck possession, though, because as in the article I referenced about drafting for possession, there is some proof that possession is the most critical tool to winning games. However, what my definition of possession skill is from a scouting level has no proof and that is why you always need to keep an open mind to new ideas and ways of thinking. I am going to be right and wrong a whole bunch of times; heck, even if I had proof that my way of defining a good player was 100% correct, that would still hold true. That's the nature of the industry. That's the nature of uncertainty in most industries. It's why individual end results should not be a focus point of evaluations.

With all that in mind, I still believe that despite all the uncertainty and unknowns that are involved in the NHL draft, there is still a way to make reasonable, intellectual arguments above particular decisions. This is the process I would advocate to properly analyze a draft pick:

1. Analyze the type of player and the type of skills you would want a team to emphasize. There will be no right answer, so don't look for one. Form an opinion based on whatever it is you can, be it some form of objective analysis such a market analysis of value and development risk, your personal subjective opinion from hockey experiences, or a poll of friends, fans, or scouts, or whatever else you can find. Take this information and form your opinion about the skills and kinds of players to prioritize for player acquisitions and use this as the basis of your argument. Also keep in mind that this can fluctuate depending on the year. I have talked to teams who will draft for position depending on the look of their organization in a given year or if they wanted players who could fast track to the NHL.

2. Assess the qualitative information of a particular player. A team may have drafted a certain forward because they saw him as a high-end skater with above average hands and an elite physical game. Is this true? Well, you'll never define true, but you can certainly argue those points. A common benchmark for skills would obviously need to be identified, but once you do that, you can argue that more easily. It's commonplace for two scouts to watch the same player, in the same game, and come away with different conclusions. Watch the player yourself, talk to scouts, read up on the player, and form an assessment of the player. You can do this even if you don't know what you're talking about and go for a complete secondhand assessment. Then apply this information to what you did in step one, where you defined what a good player is, and see where this prospect stacks up from a value standpoint. Also very importantly, assess this player's skills from and only from the decision point of the draft pick. As I stated at the beginning, the decision point is where the evaluation has to take place. Players do develop, but assessing development is a whole other column where the player, the team's development executives, and unforeseen circumstances can affect the process and usually a combination does.

3. Evaluate a player's performance through quantitative measures. As I also said previously, stats do have value and you should use them. This could be something like points, league quality, or any other measure you can find. However, there is certainly a limitation, due to sample size, a significant lack of context, and the fact players' true talent level at young ages will usually change. Basic statistics also need to be adjusted for a host of things to maximize their value.

This method in my opinion is the best way to analyze a draft selection. It's not ideal, as I don't believe you can look at picks in hindsight; the information being evaluated is very subjective and there is no "right" answer. This method essentially tells you to create your own idea of right and wrong in a very subjective field, and lets you go into the pick at the time of the pick and say what did you like and dislike. I'm not naive enough to think majority of people will do this, but this is why I go through my methods in painful detail, telling the reader what kinds of skills I like, and what I'm valuing. It's why I give detailed scouting reports at the time and give detailed explanations for key rankings. It's so you, the reader, can see and understand why and what I'm doing, and make your own judgments based on your own opinions so you can take the scouting information I provide and form your own valuation of the player.

It's through this type of process that someone can really get into what the draft is all about, and make an intelligent call about a particular prospect and draft selection. It's also why I believe the draft isn't about right or wrong selections; it's about getting the right information and using the right processes to use at the decision point of the draft that in the long run will deliver the best results.

Corey Pronman is an author of Hockey Prospectus. You can contact Corey by clicking here or click here to see Corey's other articles.