Technical Interviews: We Can do Better

Hiring is very expensive for companies (and for candidates too); it not only takes a lot of employees’ time to go through multiple rounds of interviews with several candidates, but even worse, hiring a bad engineer - only to let them go weeks or months later - sets everything back, including morale.

And yet, companies and teams spend a surprisingly small amount of time preparing good interviews.

We’re hiring so that a candidate can be productive and have an impact after a certain number of days. The basic criteria are: “can get the job done and not be a jackass” (basically, most companies’ declared “values” reduce to these two). Not being a jackass means we can work with this person (e.g., they are not arrogant and are respectful). Getting the job done in a good, timely manner means they are smart, can learn, and have relevant experience.

In the ideal interview process for a position, we would have:

• Themes or topics that the candidate needs to be proficient in (these should map, more or less, to the “must-haves” in our job description).
• A set of questions or scenarios to present to the candidate, relevant to those topics or to the general characteristics we are looking for.
• A rubric with the answers we are looking for (or, for open questions, maybe what we are not looking for).

Time is very limited in an interview, so we have to carefully choose the questions we want to ask, and not all questions have the same value.

Let’s look at what makes for bad and good types of questions to ask.

Bad questions:

• Questions looking for one specific answer that are easily looked up with Google or ChatGPT (the exception being to test basic experience). Trivia questions that have no direct relevance to the job are not good.
• Questions that are easily trained for (like many “behavioral” questions).
• Questions that almost never narrow down or filter the quality of the candidate. For example, the very common “what would you do with a difficult coworker?” - almost all the answers will be good or good enough and won’t necessarily reflect how the candidate will actually react.

As an anecdote, I worked for a company that had a “culture fit” interview round. I asked how many candidates we had historically filtered out using that interview, and the answer was zero. So why are we doing it?

The same goes for the opposite extreme. Often, we see interviewers on social media bragging that nobody was able to solve their pet problem; great, it’s like starting a binary search to guess a number between 0 and 100 with “99” or “-1”, getting no signal from any candidate.

Good questions:

• Good questions narrow down or reveal the candidate’s level of expertise, intelligence, and capability to learn. The question should have a clear objective.
• Deeper, follow-up questions that can better determine the level of knowledge are good. While this takes more time and staying on one topic reduces the overall scope of the interview, it helps us avoid candidates with very shallow knowledge or no real understanding of a topic.

Good questions can be of different types:

• Open-ended: Different answers can be good.
• Specific: Some people are good at talking in general terms but not at actually doing the job and implementing, so we need to have questions that are not open-ended.
• Scenarios: These are practical questions that combine an open-ended situation (no one right answer, candidate has to ask clarifying questions) with specific answers. They also allow for follow-up questions, digging into the candidate’s knowledge or reasoning. These questions more closely reflect a real-world task, and as such, are the superior type of question.

For every question, we should consider: Will a person with the qualifications and experience we are looking for have a good answer? Does the candidate in this position need to know the answer to this question? Could the candidate quickly figure out the information needed? We should ask a combination of open-ended, specific, and scenario questions. We can tell the candidate what kind of answer we are looking for beforehand, and we can also give hints if they are not answering the question.

Note that many specific questions can be turned into scenarios, and we should do that if possible. Also, the candidate may not know or remember, for example, a command, but may know the concept of what needs to be done.

As an example, let’s take what is possibly the question I’ve been asked most in technical job interviews: a variation of “explain the 3-way handshake in TCP.” Let’s think about what asking this question is trying to accomplish; what task an engineer wouldn’t be able to do if they didn’t know the answer to that question in isolation (and the opposite).

“War stories” (production incidents or hard-to-debug software issues) are good, since they denote having real experience versus just theoretical or superficial knowledge. They also provide an opportunity to follow up with questions about what the candidate did to fix the issue, lessons learned, or what they did afterward.