|
Puck Prospectus Authors Iain Fyffe and Corey Pronman, as well as Kent Wilson of Flames Nation and Five Hole Fanatics got together to discuss hockey, the NHL Draft and scouting:
Iain Fyffe: As I've been writing for Puck Prospectus I started out focusing on the improper drafting of small players, but I've pinpointed the issue a bit better. I call it the "Forest for the Trees" problem now - basically it seems that scouting reports will often focus too much on one aspect of a player, while ignoring other important ones.
So if someone has bad skating but is good everywhere else, the everywhere else often gets downplayed because they're focused on the skating.
Kent Wilson: Sounds about right. Think this can happen in both directions: i.e. a guy who is really good at skating gets overvalued.
Iain: I think it definitely goes both ways, in the late 1990s every player with great size was tremendously overvalued.
Kent: Dylan McIlrath might be a good example of that right now. Apparently he's tough as nails and can fight anyone, but everywhere else, he's lacking and yet, there he goes in the top 10.
Corey Pronman: His skating improved since last year, but that and his puck skills definitely aren't above NHL average or correlative to where he was picked.
Iain: Ultimately the problem is the subjective weighting of the various factors that go into evaluating a player.
Corey: The thing with this topic that's an issue is something I've outlined before in one of my player tools evaluation columns. There are 5 major tools in my opinion to a hockey player: Skating, Puck Skills, Shot, Physical Game and Hockey Sense. Now go ask 10 scouts to rank those and you'll get about 8-10 different answers.
Kent: Pretty much. Subjectivism seems more rampant in scouting because of projection. No one is really sure what you can base future performance on.
Corey: I remember talking to a NHL executive 2 years ago and he told me something that fits right in with this topic: "To be in the NHL you need about 9-10 traits, and if you lack in one then you can't make it."
Iain: Like in your 20-80 article Corey, you point out how players can be evaluated based on several important factors, but then the overall score (assuming you do it the way they do in baseball) is not an average, nor a weighted average - it's just another subjective score.
Kent: Pretty much.
Corey: Well scouting for the draft is 100% subjective and that in itself creates issues.
Iain: I wouldn't say 100% - there are some objective things, but at least 90%.
Corey: Well do you mean scouting or draft ranking?
Iain: Okay, scouting has some objectivity but draft rankings do seem totally subjective.
Corey: When I'm doing my lists, like my Top 100 Draft Prospects or the upcoming Top 50 NHL Prospects for the Annual, I change my mind several times during the listing process because there are no set rules on how you should do it. It's my personal philosophy and that changes over time.
Iain: Do you think it's possible to reduce the amount of subjectivity involved?
Corey: Well Iain the statistical analysis you do is a form of objectivity and I know several teams are starting to employ something similar. However we hit a few roadblocks along the way, the main one being whether the information is accurate in determining success. Are goals, assists and games played the proper information to use to make a true assessment of a player?
Thing is with scouting subjectivity is what are we looking for? That's something we haven't even fully answered at the NHL level yet.
Kent: It appears to vary by team.
Iain: Do you mean different teams looking for different things in a player?
Kent: I know the Flames list of "desirable traits" the last few years probably varies widely from, say the Kings.
Corey: What makes a player valuable? If we can't define it fully with the NHL metrics, it hurts us trying to statistically analyze the CHL.
Iain: Okay, you mean more of a valuation problem - we don't have a firm grasp on how to accurately value a players' contribution at the NHL level?
Corey: Well its light years ahead of where we were 10 years ago.
Iain: Agreed.
Corey: However the margin of error is still significant and our studies into scoring chances, zone starts and corsi show how flawed using mere goals and points can be in dong player evaluation.
Kent: I would have to assume that NHL scouts have access to shot data and such at the CHL level. I would hope at least. I mean, the paucity of data for us fans is atrocious...
Corey: Well I've seen people do manual video for the draft with video scouting akin to what David Staples does at Cult of Hockey but I haven’t heard of concise corsi numbers for CHL players.
Kent: You're right, though, if they're going by counting numbers and observation, there's lots of room for error.
Corey: Another issue with using objective analysis is the sample size.
Kent: Big time.
Corey: You’ll usually have about 1 to 2 seasons of CHL time so maybe 100 or so games if the prospect doesn't have a late birth date?
Iain: True, unless you're John Tavares.
Corey: Well, when using projection models, Iain and I both know having a large sample of games played boosts your projection score as it clears out the noise.
Kent: I find scouts seem to make decisions based on half a season of good work, especially in rounds lower than the first.
Iain: That's a good point Kent; you'll often see movement in Central Scouting rankings justified because the player played really well in the playoffs.
Corey: Jeff Skinner ring a bell? (Skinner’s 20 goals and 33 points in 20 playoff games was a major topic of discussion in prospect circles).
Kent: Max Reinhart began the year off the board completely. After a near point per game run from January onwards, he was in the 70's.
Corey: In fairness, age 17 is a major developmental year and kids do make progress.
Kent: Which is another confounding factor in junior: kids make big leaps and the difference between age 16 and 20 is huge, not only in terms of personal development, but also in terms of level of competition.
Corey: Yes and that's a personal pet peeve of mine. Over drafting 18 year olds and you see it every year.
Kent: It's definitely a risk, especially if they're playing against kids, but it could differ if the guy is in a pro league overseas.
Corey: Well, going into 2009 the top 2 guys were Tavares and Hedman and they both had late birthdates. In 2010, Hall and Fowler, both were preseason top guns and had both late birthdates and for 2011, Larsson and Couturier both fall under the same categories. For 2012, you’re hearing about Ryan Murray and Nail Yakupov near the top, and guess when they’re born? Is it that all great players have late birthdates (which in some of these examples are true) or is there more here?
Kent: Interesting.
Corey: The issue is here is that these players are a year past the development of the 17 year olds and looked better in the scouts' eyes with more developed tools. I thought Nazem Kadri was a victim of this and Burmistrov this year; guys who went a few spots too high in my opinion because their tools were developed, but not as advanced as where they should be at their age to dictate that draft position.
Kent: Another issue I’ve noticed may be the fraternity type familiarity amongst the scouting community. The issue with Kabanov which Corey illustrated in his column looks to me like an "information cascade".
Iain: Groupthink, basically?
Kent: Basically, scouts from a few teams say "stay away from this kid" and that reputation rolls through the league. Related, but separate, in groupthink, all scouts would be part of the same group and they'd all agree for the sake of agreement. In an information cascade, parties are separated and acting in their interest but "mimic" the actions or behaviors of others because they trust they're correct.
Iain: I see - I figure that must be what happened with Luc Robitaille all those years ago - he can't skate, stay away from him!
Corey: Well in Kabanov's case, he didn't really need scouts to market him, since he was front page news in prospect circles. With Kabanov here's a major issue I’ve been concerned with: how do you know he's a bad person?
Kent: It's a good question. He had some coaching issues and his dad is a pain, but he's a 17 year old kid and as you pointed out, it was a limited issue. It's not like he's been a problem for years.
Corey: You have a clean criminal record your whole life, you screw up once or twice, do you go to jail or get probation for what he did? I don't recall anything about that.
Kent: Depends how you screwed up, but I understand the point.
Corey: With intangibles also, I tend to value commitment over anything else in my years of hockey. I’ve yet to see leadership or character hinder a team. If it hurts you on the ice, like with work ethic or fitness, then I can see the issue.
Iain: I can see that - Daigle would be a good example.
Kent: I've played on teams who fought in the locker room and won on the ice.
Corey: In Kabanov’s case he does work off the ice, and he fled his home country to play hockey. I see no commitment issues.
Kent: Also, one of the issues with Kabanov is he was hurt and didn't get to put up the kind of results that would allow scouts to potentially overlook his "issues".
Corey: Kabanov played 2nd line time on a contender, so it was hard to produce well. When I saw him play he looked great.
Kent: In the end, unless a guy has significant psychological problems - like Dan Ryder - the focus on intangibles is overblown in my opinion.
Iain: Intangibles are overblown - and far more subjective and thus harder to scout for anyway.
Corey: Also the amount of resources you need to exert to find the true intangibles are an enormous amount in proportion to what the likely contribution level will be.
Kent: True. And they can seriously skew you're view of the kid as I discussed in my post on the dilution effect, even when they don't actually have much to do with his ability as a player.
Iain: Speaking of small sample size, you mention small sample size in the context of statistical evaluation, but how often does a scout see a guy play?
Corey: Most scouts in a draft year will see prospects about 5 to 20 times a year. That’s not exact and varies on area and player.
Kent: Yeah...that's not enough unless he's exceptional.
Corey: Even then, you might see a guy in a time span where the prospect has family issues or a bum knee and it skews your opinion.
Iain: Exactly, scouts seeing the same player at different times can see different things.
Corey: Well that is common place in hockey. I’ll talk to 3 guys about Player X's skating and each will have a very different opinion.
The first guy will say, "he is a fantastic skater, moves well and has a great stride," while the second guy will say, "he's an okay skater, but looked sluggish," and the third guy will say, "he has good speed, but bad stride."
Iain: Ugh, stories like that make me wonder how they draft as well as they do.
Kent: Yeah it depends on when you see him.
Corey: It depends also on what your take on a tool is, like honestly what is good skating? Can you describe it?
Kent: I can't, really. I know what I consider to be good skating...
Corey: Exactly, some put value in speed, others mechanics, others in leg strength.
Kent: Then there’s balance, agility, etc.
Iain: Do you see any way to improve on that? Skating, for example, is a physical thing.
Corey: What do we test it against, how do you measure a physical tool? Like the 20-80 scale I presented was just an attempt to do that by putting a number on the tools. The issue is it’s subjective, but you can at least describe a player properly.
Kent: I like that idea though Corey - at least establishing some standard rating tool.
Iain: Standards are good, but being subjective with numbers is still being subjective.
Kent: Here's what tends to happen in psychological testing… some standard definition is defined and a number of independent observers use the standard definition and use it to score the behavior over a series of time or events. The level of independent agreement is scored between the various observers. So let's say that 5 different guys agreed that subject X displayed the target behavior in instances X, Y and Z...but only one said it happens at point A. Point A is basically thrown out. The others are accepted.
Iain: So with the 20-80 scale you get a bunch of scouts to do the ratings, throw out the outliers and use the majority?
Kent: As far as I know, certain scouts go over certain areas. So, Scout A looks at certain areas of the WHL for example. He reports back on certain players and the team is dependent on his evaluations for those players under his jurisdictions. Now, imagine you actually have five scouts cycling through areas...observing similar players but at different times. Four scouts say "player is a strong skater," but Scout 5 says "player is an okay skater, but with bad stride". What’s likely more accurate?
Iain: So you're upping sample size, really.
Kent: Pretty much increasing iterations and limiting subjectivity. I think it would technically be improtant that the scouts NOT share reports so that their evaluations are truly independent. Where I got this idea from is "Wisdom of Crowds." It's a book that states that, in aggregate, crowds tend to make better decisions than individuals as long as certain conditions are met.
One of which is independence of thought and decision making so as to avoid groupthink or information cascades.
Corey: Which makes sense, there's a lot of yakking in the scouting community.
Kent: I can imagine. If a team could aggregate their scouts’ evaluations (using a decently objective set of standards), which were independent of each other, I think they'd have a more accurate picture of prospects.
Corey: Changing topics, let’s hit on ESPN's Gare Joyce’s book, Future Greats and Heartbreaks.
Kent: When I read Joyce’s book, all sorts of alarm bells went off in my head. There seems to be a lot of focus on idiosyncratic, personality factors in scouting and there was an "art more than science" feel to scouting displayed in it. Thing is too, we heard almost nothing about statistical methods of evaluation. Now is that because Joyce didn't see it, wasn't interested, wasn't allowed to share it or did the Jackets really not worry about it.
Corey: Well that’s a philosophy thing. Some people see it as:
- Useless or near that;
- A reference but nothing to lean on or;
- A predictive tool.
Depends on team philosophy. I know I'm between 2 and 3.
Kent: Right. Anyone who thinks #1 either isn't looking at the right statistics or is hopelessly antiquated. Statistical analysis is a must when it comes to objective evaluation. You have to use the right stats, but you can't properly gauge context of performance without them.
Corey: The one thing in Joyce’s book that stuck out to me was how basic the strengths and weaknesses were for the top prospects. They listed a player’s tool by putting a + for strength and a – for a hole. Here was how they described some of the top prospects in the 2006 NHL Draft from Joyce’s book:
Erik Johnson:
+ Size, Skating, Strength, Shot
- Doesn’t have a killer instinct, won’t back down and isn’t mean
Derrick Brassard:
+ Speed, Creativity, Hands, Hockey Sense
- Size
Other grades were:
Jordan Staal:
- First couple of steps, faded in the Memorial Cup
Jonathan Toews:
- Not a game-breaker
This was the criteria used in the final listing process for the draft. I shouldn’t even have to start with what’s wrong here. You don’t even need to read the 20-80 scale to know they could be using more precise criteria to judge players.
This isn’t a widespread problem, not all scouts evaluate like that, but it is concerning.
This concludes the roundtable. We didn’t come to any form of enlightment, but what we did find out is that there are major issues with every form of prospect evaluation. It’s hard for anyone to say whether using purely statistics or scouting is the correct method because both have their major flaws.
- Looking at tools has its flaws
- Looking at statistics has its flaws
- Looking at intangibles has its major flaws
- Scouting in samples has its flaws
- Statistically analyzing in samples has its flaws
This is largely why the draft can be such a crapshoot as there’s so much unreliability in predictive methods. Here at Puck Prospectus, I hope to be able to take the best of both worlds and get as close to the truth as possible.
Until next time, thanks for reading.
Kent Wilson is an author of Flames Nation and Five Hole Fanatics.
This column was authored by the staff of Puck Prospectus. | 
0 comments have been left for this article. 
Conn Smythe Watch: Rask Leads Race