Why it’s Difficult to Evaluate Data Science Candidates
I’ve hired a number of data scientists, data engineers and analysts over the years and interviewing and evaluating potential data scientists has proven to be the most challenging. In speaking with other analytics leaders, I often hear the same thing. Why is this, and what can we do about it?
Hiring is Difficult – Period
First, it’s worth pointing out that evaluating candidates and hiring for ANY role is hard. It’s time consuming and exhausting. As a hiring manager, you often question yourself all the way from resume review to making an offer. You’ll often disagree with others on the hiring team. You’ll fight the urge to “just make someone an offer!” if the process has been dragging on far longer than you had hoped. Sounds like fun, right?
There’s a reason for all the pressure, stress and self-doubt. You’re hiring because you need help, but you’re so busy that you don’t feel like you have time to hire. Even success means kicking off yet another time consuming on-boarding process that will (and should) take the attention of you and your team. But after it’s all over, if you make the right hire you’ll feel a sense of relief and accomplishment that’s hard to match.
All the above is true no matter what role you’re hiring for. What makes hiring a data scientist even more difficult is assessing whether or not the candidate is actually good at the most basic aspects of being a data scientist. For some roles, that’s the easy part!
For example, the process of hiring software engineers has gotten better over the years, and for the most part you’ll come away at least knowing how well they code, their grasp of best practices, development tools and so on. Is it a perfect science? Not at all, but it’s shocking if you totally miss the mark on that part of the evaluation. A bad hire is often due to “fit” or personal issues.
With data scientists you can get an idea if they’re proficient in the syntax of R, Python, SQL or whatever else you use. You can look at their GitHub repo (if they have one), their Kaggle profile and have them speak to projects they worked on in a grad school course. The problem is, the code they wrote and outcome of their projects don’t tell you as much as you’d think.
Clean Datasets & Working Backwards from Success
There are two problems with nearly every academic project and Kaggle competition when it comes to talent evaluation:
- The input datasets are simplified and relatively clean.
- The outcome is somewhere on the spectrum of success.
The first point is relatively simple. I’ve seen some projects with intentionally messy data, but it’s just so hard to simulate the real world of messy, invalid, and missing data that confronts data scientists on the job. Heck, I’m often happy if my team has messy data! It’s the realization that we don’t have the data we need at all, or that we’re going to spend weeks working with engineering to piece together a proxy of what we want. Working through that process is a skill that’s not only hard to develop but also hard to evaluate during the interview process.
The second point is one that took me a few years to realize. When a professor gives a class a project, or a Kaggle competition is launched, the resulting models and analysis typically result in some measure of success. Some models will do better than others, different approaches will be taken, and different recommendations will be made. However, all roads lead to a solution of some sort. I call this “working backwards from success.”
What I don’t see as a result in those projects is “this project isn’t worth undertaking, or is impossible given current constraints”. On the job, the latter outcome is sometimes, if not often, the correct conclusion! I understand why academic projects and competitions don’t end that way. Their goal is to teach students about modeling techniques and the “hard” skills involved in the process.
Other Ways to Evaluate
One thing I like to do is try to tease out whether or not a candidate has worked on a project where success meant shelving or delaying the project until the right data was available. In some cases that hasn’t happened to them, but they’ve had to spend a considerable amount of time tracking down or advocating for data that wasn’t accessible to them. How they view and handled such situations is something I value highly. Did they see it as a rare event (it’s not)? Did they blame others, rather than working with them? Do they view it is a one time thing that won’t happen to them again as they gain experience (it still will)?
It’s not just about checking a box stating that they’ve dealt with such a reality. I also learn a lot about how they work. There’s no single right way to go from idea to result in data science, so I like to hire people that are creative and positive in the process regardless of what that process is. Dealing well with imperfect data, and being confident in coming out of early exploration with the realization that the project might not be as juicy as hoped is what I want in a candidate.
How is this different from asking software engineers about past projects you might ask? Developers deal with all sorts of challenges besides writing code they can talk through. That’s true, but I find I can pair that discussion with walking through some code they’ve actually written to gain insight into how they think and work. Any code that a data scientist can share – meaning it isn’t property of a past employer – is built alongside those clean, boring datasets. The story of how they acquired, prepared, and explored the data becomes just as clean and boring.
Still, talking about the end-to-end process rather than staring at code is a great idea. Some candidates immediately key in on the challenges and can talk about projects that didn’t go as planned.
Supporting Entry Level Data Scientists
One problem with evaluating in this way is that junior level candidates will nearly always draw a blank. Maybe they’re applying for their first role in data science. Maybe they were a junior member of a large team and these types of issues were handled by the senior members of the team. None of that is their fault, but know that when you hire entry-level candidates you’re taking a risk and should be committed to supporting them no matter how great their sample/academic projects look.
So you’re still OK with hiring an entry level junior scientist? Great, there’s a lot of them out there thanks to a wave of new graduate programs, boot camps, and self motivated individuals. Hopefully by now I’ve convinced you that there’s a big unknown in evaluating how they’ll translate those shiny new skills to real projects. What should you do next?
First, make sure you have a support structure. If you don’t have senior data scientists or a team lead who’s been in that role, you might want to reconsider. You’ll also want to make sure that person is ready to act as a mentor. Being one not only takes buy in from the mentor, but also their time. Be ready to cut back their project work for the first few months of on-boarding.
Second, make sure your source data and data engineering team are ready. A more senior data science hire can better handle undocumented data in various states of cleanliness and structure. A junior data scientist should learn to get to that point in the future, but you really need to have enough data in a good state for their first project. Think of it a stepping stone from clean academic data to the real, messy world. It will give them some early confidence while you slowly work them into the reality of more complex projects and data issues.
Learn from Each Hire
Getting good at interviewing and hiring takes practice, and you’re not always going to get it right. Every hire you make is imperfect in some way. I like to go back every few months and review my notes from the interviewing process and compare them to the current state of reality.
- What did I anticipate about the candidate that turned out to be true or false?
- What about the on-boarding process turned out to be more difficult than I expected?
- Did we make the right tradeoffs in selecting this candidate?
All this sounds obvious, but I find that if I don’t take good notes during the hiring process and don’t set reminders on my calendar for future months it doesn’t happen. It’s also a good idea to be explicit with others involved in interviewing and on-boarding to do the same. Don’t assume they’ll take notes or recall the decision making process months later.
Finally, don’t measure success as making the “perfect hire”. Your job as a leader is to help your team improve on their weaknesses as much as it is to celebrate their successes. A good team is made up of complementary strengths and weaknesses. You’ll know when you made a truly bad hire, trust me. Otherwise you’ll end up with a great teammate as well as a few points to tweak in your hiring process. That’s success.
Cover Image courtesy of Kaboompics .com
