I was asked to describe how to evaluate a Software Project Manager. I remembered this nice challenge I did at some point, and realized that had to dig in memory a lot (and ask someone who actually had taken the challenge). For that reason I decided to write about it here.
At some point in the past I have had to hire a Software Project Manager for a startup. We were looking for someone who had great Project management skills, but that also understood the Engineers.
As part of the evaluation process I did an "onsite challenge", which took around 3 hours. For this challenge the candidate had to define a plan (in the form of a type of Gantt Chart) for executing a Software development project, given a set of constraints (in time, resources, and scope of course).
The Challenge
The objective of the challenge was to test how the candidate handled the definition of a real-life project, with its constraints. The constraints came in the form of insufficient resources and time, for the scope
The candidate was recommended to ask questions to the interviewer, who played the roles of a) The Engineering team and b) The Product Team. The points to evaluate were:
- How well the candidate interacted with the Engineers and Product "teams" to ask them questions, clarification, etc.
- How much "domain knowledge" did the candidate have allowing him to make sound decisions when making compromises.
- How did he handled the scope/resources duality and what solutions did he came up with to address them.
Scope
To define the scope of the project, we provided the candidate with a list of stories needed to develop a mobile App. This list was comprised of Frontend stories (to implement standard UI HTML/JS), Backend stories (for the APIs to be used by the app) and Mobile stories (which required specialized mobile knowledge).
The stories were provided in the form of Story Name - Story Points - Skill, where Skill defined which type of Engineer could do it (everyone, Front End only, Backend only, Mobile Only). The stories were real stories (they actually were partially based in another project we had made before).
Resources
We provided a set of theoretical Engineers for the project. Engineers were defined in 2 dimensions: Seniority (Jr., Mid level, Sr, Tech Lead) and Skills (Fronted, Backend, Full Stack, Mobile). Each Engineer had a specific seniority and a combination of Skills they had.
The Seniority factor was modeled as the number of story points that an Engineer could finish per Week.
The idea here is that the Project Manager candidate had to consider the skills and seniority when assigning each story to the different Engineers.
Time
The last of the constraints was modeled as the date when Business had asked us to release the software product. The available time of course was not enough to implement the full list of requirements considering the available resources.
Moreover, as part of the definition of the problem, we asked the candidate to consider people going on vacation and getting sick (I barely remember even adding some vacation days for some of the Engineers, although I am not sure of that).
Evaluation
The desired output of the challenge was a type of Gantt chart. We did not explicitly asked for this, but asked the candidate to use any tool hey could, so that at the end of the challenge, he would be presenting to "Business" (i.e. the CEO, given that this was a small startup) the plan for realizing the project. The best candidates usually made some kind of Gantt chart.
The output was only one aspect of the evaluation, another (more important to me) was the interactions that happened while doing the challenge; these in the form of questions to the Development team, or questions to the Engineering team.
As candidates asked questions, possibilities opened here and there. For example, one story that was 8 points if he asked if it could be split in two, the answer was usually yes (given that those were real stories, I tried to stay true to the story and give a "real world" answer. For example if the 8 point story was a complex frontend screen in the App, divided it in 2 stories of 5 and 3, dividing the screen development in two parts).
At some point some candidates even asked about QA considerations (something I did not consider initially), so we gave them some "advantage" (in the form of time, telling him that say, the stories had some QA in them, which could be split and maybe better accommodated).
The one thing that we were looking for was for candidates that not only filled up the scope until filling the time available in the project, but that went a bit further away.
My experience with startup projects has been that there is always not enough time to do all the required features, but the greatest value (and challenge) comes into prioritizing the tasks so that the team can deliver the mythical MVP on time, and then, perform additional iterations improving the product. This type of reasoning is what we were looking for as part of the output of the challenge.
Results From Applying The Challenge
In general, candidates that asked more questions produced a better result. We interviewed about 30 different people for the position and we had a very diverse set of responses.
Some candidates had a stronger technical background, and thus could leverage more the content of the stories. Other candidates had not so much technical background and leveraged more the business side.
In general, the challenge was a success, and in the end I remember we found the right candidate for the team.
Something important to stress is that, this like many other "technical" (in the sense of, having to implement something, in this case a plan) interviews tends to suffer from a "degradation of context" problem: When we created it, it was very clear for us that the evaluation consisted in not only the end product (gantt chart, or anything else) but the interaction and general process that happened during the 3 hours. If other people start using this challenge, it is important not to lose this context and only consider the end product for the scoring, as it will be misleading.
I've seen this happen in other types of challenges (specially ones that comprise programming something). Where inexperienced interviewers decide to apply the problem for an interview and score it as in a "black and white" given the program provided by the candidate.
