Did You Say Hockey?
As a red-blooded Canadian, I've been raised in a hockey culture. Hockey in Canada is unavoidable. Like (American) football in the USA or soccer (football) in Europe, it permeates our culture. Nearly everyone is aware of the stars of the game and the latest exploits of the local team. For the die-hard fans, it goes much deeper and becomes a part of our identity. Unfortunately for me, the Oilers have been taking me through a bit of an identity crisis since I moved to Vancouver in 2007.
The first surprise is the busts: those players whose performance failed to live up to expectations. Prior to our pool draft, I distributed a spreadsheet of last season's top scorers. Most of our poolies based their picks (at least in part) on this list. Milan Lucic, and Jordan Eberle, and most notably James Neal were all dropped from our pool within the first few months due to their poor performance. It appears that a player's historical statistics are not always a sure indication of his future results.
Perhaps more surprising is the missed opportunities. Many players were overlooked during our draft and turned into great late-round steals. Several of the top players in the league were passed over and remained completely undrafted. This includes top-rate goaltender Pekka Rinne, and the current league-leading forward Jakub Voracek. The collective knowledge of our pool was not enough to identify these high-scoring gems before draft day.
Perhaps we're just terrible at hockey pools, you say. Maybe so, but before we wallow in self pity at our own ineptitude, let's see how the professionals fared.
The Real Deal
When it comes to drafting forwards, it seems to me that the professionals have done a solid job. Of the top thirty scorers this season, the vast majority were selected in the first round, several were taken number one overall. There are a few notable exceptions such as Henrik Zetterberg and Joe Pavelski, each taken in the seventh round of their own draft year. As for goaltenders, the professional record appears much shakier. Many of the top goaltenders were selected well into the late rounds including two of this season's leaders in the wins column: Pekka Rinne and Jarolav Halak, who were selected in the eighth and ninth rounds respectively.
As for the busts, I feel the best place to look is the number one overall seed. The most coveted position in the draft, the privilege of the first overall pick allows one team first choice of hundreds of young up-and-coming players. While we can hardly expect the scouts to nail it every time, it seems fair to anticipate that anyone taken so early in the draft would surely develop into a NHL staple with consistent performance for years to come.
One of the all-time great busts was the number one selection of Alexandre Daigle by the Ottawa Senators in the 1993 NHL entry draft. After a promising rookie season, he consistently fell below expectations before bouncing from one team to another and leaving the NHL for Europe following the 2006 season.
Then there's my personal favorite draft bust: Patrik Stefan. Stefan was chosen first overall by the Atlanta Thrashers in the 1999 draft. After averaging less than half a point per game over his NHL career, the number one pick left the league after only seven seasons. He is now unfortunately best known for this debacle, missing a wide open empty net and allowing my Edmonton Oilers to tie the game with only seconds to left on the clock.
It appears that the NHL scouts, whose entire job consists of watching, interviewing, and evaluating young players still get it wrong sometimes. I give them all due credit for a tough job where the successes go mostly unnoticed and the failures expose them to public scrutiny, with the critics calling for their heads. Still, these scouts occasionally miss golden opportunities or unwittingly place their faith in the wrong player.
Imagine, if you will, that the first ever National Programming League (NPL) draft is a mere three weeks from today. Your company was lucky enough (or perhaps it just performed poorly last season) to be given the privilege of the first overall pick. On draft day you will have your choice of any active programmer in the world to build a your dream team around. How do you choose?
You could start by looking at famous programmers. There's not too many who have wide-reaching fame, but if you dig hard enough, you could probably come up with a list of maybe a few hundred programmers who could be recognized by several people they had never met. Does fame really equate with talent? Not where I come from. Certainly some programmers might be famous because of their talents, others may also be famous despite them.
Another place to look would be job sites and headhunters. Browsing through candidate profiles from a collection of sources, you could try to boil down all the varied types and qualities of information into a short list of candidates. From there you could further research each programmer on the short list and try to come up with your definitive number one pick.
An interesting approach would be to follow chains of references and recommendations. Start with a reasonably large collection of randomly selected programmers and ask each one to recommend the smartest programmers they know. By repeating this process recursively, you should eventually bump into the same highly-praised candidates time and again. Again, from that pool of recommended candidates, you could perform some final detailed analysis and evaluation to settle on one final choice.
With the strategies above, you could probably come away with a solid programmer, but how close could you come to identifying the true "best" programmer on earth? I think you might still miss by thousands of places.
I'll bet that some of my readers have even more clever schemes up their sleeves. Can you think of any better strategies, or are you just hoping that you'll be selected in the early rounds?
Programming Is Different
For hockey and other elite-level sports, all the work is done out in the open and under the public's watchful eye. Every shift, every hit, and every goal is recorded and tallied for later retrieval and analysis. We can watch healthy NHL players play 82 games each season and play back the tape in slow motion from every angle imaginable. Yet, somehow we still find it hard to predict tomorrow's score.
Unlike the world of sports, programming work is largely an independent activity and is typically done behind closed doors. Private companies protect their source code and their employee information. With open source, of course, you can see the fruits of their labor, but it is still very difficult to measure performance in a meaningful and statistically relevant way. There's no great programmer database where you can slice, dice, and graph statistics to your heart's content.
From my own experience, I know that employees who have worked at the same company for years can know little about their colleagues talents. Only by working closely with another developer on a variety of projects, can I even begin to gauge how truly talented (or not) they might be. How much harder would it be to evaluate the performance of someone I've never met?
To The Point
The thing that has me all fired up is that people sharing their "ideal" process for hiring a programmer.
At this point I must mention that my experience in this area is incredibly limited and I'm not really qualified to be giving out advice on hiring. (Now ignore this disclaimer and listen to me!)
I've seen this advice take many forms. Some propose a simple bullet-point test that can be completed in minutes, while others have devised an elaborate series of excruciating tests spanning multiple days and consisting of so much actual work that it might not even be legal anywhere on earth.
(P.S. please don't do this)
What I'm desperately trying to get across is that there is no such thing as the ideal hiring process for programmers.
There are certainly some tips and tricks that you can apply to weed out the riffraff and narrow the field to those with a higher chance of success. You can ask candidates to write simple functions on a whiteboard or using software. You could discuss the advantages of different data structures in a specific scenario. You might give candidates basic design problems and ask them to draw diagrams of their solution using boxes and arrows. You might discuss work history and how they've dealt with people problems. I'm sure if I sat here long enough, I could rattle off dozens more.
All of these hiring tips and tests will only get your so far, eventually you have to settle on a candidate who seems competent and talented given limited information. Even if you are lucky enough to hire an elite performer, it may take years before you can confirm that they are truly a cut above the rest.
Sorry To Disappoint
Just as it is for the professional scouts, finding and hiring the right people for your technical team is an essential part of your product's success. I understand that this is an important problem, so I'm sorry that all I have to say is, "this problem has no ideal solution". I'll let you in on a secret though: if I ever discover a solution, it will be much too valuable to share -- at least until I can use it to earn some cold, hard cash.
Do you believe in an ideal hiring process? What tricks and tips do you use to identify promising candidates? Let me know in the comments.