Skip to main content

Creating a Coding Interview Problem

Creating a Coding Interview Problem

At work, I have been responsible for conducting the coding tests during the interview process, and have been doing that for over a few years now. Over the time I have made some mistakes, learnt some things and this post is a summary of what one should consider when creating a coding test.
Coding tests are generally the first round of interview; it is either online or on campus, depends, but a coding round is considered as the first gate a candidate has to cross. How you conduct it has some implications, for example, if you asked the candidate to submit code online, you may want to verify that the code was written by the candidate, they understand it and that they did it in the stipulated time. You can do that with an additional pair programming round on campus. If you conduct the test on-campus there is little room for such doubt. But then you need to ensure things like a dedicated desk or meeting room for the test, a machine, and food and coffee may be. But there are some considerations integral to coding tests themselves, these are the ones we shall be looking at today.
Before we begin, let me call out that there are those who believe that interview is not the best process to understand the suitability of a candidate, and I would not disagree. But I believe that if done right, coding tests are the best way to understand a person’s approach to problem solving, their expertise and command on programming. How to go from there is your choice.
We shall see the points I have learnt to consider when creating a test problem, we shall also discuss the the thoughts and reasoning behind them.

What to consider

Skills: What are you looking for

First is to identify the skill set you need, the practices that matter to you for the role you are looking to fill. People consider knowledge of language C as a basic expectation in computer science, but if a developer is not expected to work on C it is pointless to ask a question with C in mind. Similarly, the most attractive questions for an interviewer: questions on shortest path algorithms, sorting and searching algorithms seem pointless. If there was ever a need to use them, I would not expect any employee to implement them anyway, why expect them to implement in interview. All we need is that they understand how those algorithms work. In an interview, we need to better target the skills that matter to us. Yet, it is better to skip any frameworks that you might use, frameworks can be taught and learnt, teaching problem solving approaches takes longer. I am also of the opinion that the programming language used to solve the problem should be of the interviewee’s choice, but that is not always feasible, since we should have enough skill in that language to review the code they submit!

Prioritize: What is the value of each practice/skill to you

Prioritize the skills in these categories: ‘must have’, ‘should have’, ‘good to have’ and ‘cool to have’. Model the problem in a way that targets the ‘must have’ skills and probably touches on some ‘should have’ skills. Skip anything under ‘cool’ category. This is not to say that we should hire a person who can do only the job we require them to do at the time of hiring. It is to say that the person should be able to do at-least that much. You have rest of the interview process to assess the person’s ability to learn or apply creativity, coding test is the gating criteria.

Time: How much time can you allot for the test

This matters a little in an online test, but is a huge consideration in an on-campus test. For on campus tests we need to have provisions for machines, food and meeting room or desk. If it is an interview drive, we need plan for as many number of machines as there are candidates or divide the candidates in batches, with delay in batches equivalent to time for test. If it is an online test, how much time to provide the candidate to revert and if the candidate can get a weekend to revert are considerations as well. Either way, how much time do we demand from the candidate is a question and we need to decide on a problem that needs an estimated amount of time to solve.
It is also important to accommodate for the pressure the candidate may be under during the test. This matters even more for on-campus tests; it can put your time calculations off considerably. I try to make the problem statement very clear and as definitive as possible. Being explicit about what is expected and what is not helps, because an unclear question leaves room for random implementations or shortcuts where I least expect them. That makes it difficult to evaluate the solutions on equal grounds. On the other hand, unclear questions leave room for creativity, but I have seen people missing the point with these.

Branding: Showcase how you are

Interview is a window, window for the company into the candidate's capabilities and window for candidate into company’s culture. Just as the company needs to like the candidate, the candidate needs to like the company. And the problem statement is the first impression of the company! It is important to make the problem ‘fun’ to work on. I prefer to use casual way to describe the problem and try to make it enjoyable to read and to solve. It should have the feel of a fun place. You should choose a style that best describes your organization. As a side note, nerd jokes or references to Hitchhiker's Guide to the Galaxy or Star Wars do not always work. ;)

Creating the problem statement

Over the period I established a process for creating the problem statement, enabling a predictable estimation of time and solution. Over time you can choose to skip some of the steps, but to begin with here is the process: 
  1. Once the problem is identified, create the write-up describing it. Instructions, guidelines and rules about time, how to submit the code, languages allowed etc should be mentioned at the top.
  2. Once the problem statement is formulated, ask a peer to read and explain it to you. Ensure it means what you expect it to mean.
  3. Solve the problem. Record the time.
  4. Ask a peer to solve the problem, record the time taken. The peer here should be from the target experience range and skillset. Ask for their experience.
  5. Anything you learn during these discussions, or while solving the problem, convert them into instructions and add them to the top.
  6. Fix the scope of the problem to fit the time.
  7. Solve again, when you are in a different mindset and record the time.
This process, although tedious, sets the expectations right. Ponder on how good is your solution in that time and how acceptable are your mistakes to yourself. Consider that you already knew the problem and had (unknowingly/knowingly) designed a solution in your mind. That should help set expectations from an interviewee ‘under pressure, under time limit, likely less experienced than yourself, groomed in different culture with different practices than yours who was just presented the question.’ Sounds difficult, right? It is not to mean that the evaluation should be lenient though, it is only to identify what is acceptable.
While evaluating, one should remember that you are not looking for candidates who solve the question the same way as you do. In other words, you are not looking for candidates who think just like you. The different the better, it is always surprising how many different ways you find the problem solved.
That’s all from me, all the best for finding the right candidates!

Comments

Popular posts from this blog

Participating In A 24 Hour Hackathon

Just returned from a 24 hour hackathon, sleepy, red-eyed, tired, exhausted and yet writing this post. You know why? Because I skipped it the last time, and the time before, thinking I will do it the next day and that sleep was more important, but never did it. Not going to make the same mistake again. So here I am.

For those who are unaware of what a hackathon is, it is an event where dreamy eyed people enter and leave with red eyes, it is a single night sprint where people come together to build something that they believe will make them a billionaire or like Silicon Valley series mentions time and again, ‘will make the world a better place’! Well, jokes aside, a hackathon is an event / competition where teams / individuals build software / hardware in a single sprint of 24 hours. Hackathons have a theme, ranging from generic things like improving community to specific themes like solving a specific delivery problem the company faces. Some hackathons are closed, conducted only for t…

Using Docker and a Private Registry with VPN On Windows

Wasn’t that a very specific title? Docker has a very good documentation and reading that alone is enough for most of the straightforward tasks we might want to do. But as always some practical tasks are not straightforward, hence this blog. What we are going to see here today is how to setup docker toolbox on a Windows machine, make it work even when VPN is connected, make it talk to a private, insecure docker registry (that is why VPN) and configure it so it can run docker compose and see how we can set this config as a one-time activity. That’s quite a mouthful, but yes this is what we are going to do. All ready? Let us begin then.

Install Docker ToolboxGo and download the docker toolbox and install it. That should create a shortcut called “Docker Quickstart Terminal”. Run it. That should show you an error about virtualization.

Enable VirtualizationRestart your machine, enter the BIOS settings and enable virtualization. It may be under advanced settings. On this Laptop, it is under th…

Yet another packager for node

Yet another packager for node There are so many packaging systems for node already, or maybe not as many, so here I am presenting another way to package your applications into an self extracting executable that has no dependencies. Ah well, a few dependencies, like the processor architecture, and Linux operating system may be, but that is all. What is it? It is a modified shell script originally used to create self-extracting and installing applications for Linux platforms. What it does is, it creates a tarball which includes your code, the modules it depends on, the specific node binary it uses, and appends it to a script with the command to execute your code. It is essentially a binary merge of the files, the shell script and the tar.This is not something new, people have used such a system in the past to deliver applications for Linux, every time you see an obscenely large ‘.sh’ file (for being that, a shell file) that can install or execute an application without requiring any ot…