Thoughts about hiring Product Owners, Part 3

This is the third and final part of my series on hiring Product Owners and in this part I go through how to evaluate Product Owner candidates with work samples and through auditions.

Part 1 – Are You Recruiting For Potential Or Experience?
Part 2 – Questions You Can Ask In Your Interviews
Part 3 – Work Samples And Auditions  <- This post.

Effort and reward

Before you decide how to evaluate your candidates consider the amount of time and energy you’re willing to invest to learn about your candidates. Asking for work samples and evaluating them yourself requires little effort while running auditions require the most time and energy but is also generally most rewarding.

Work samples
Work samples are artifacts that the candidates have produced in their current or previous roles and could be:

• Backlogs
• Roadmaps
• Vision decks
• Prototypes
• Product canvases
• Personas

Viewing a candidates work samples gives me a better understanding of her style but because some companies prescribe how to do things e.g. how to write user stories and how to create roadmaps it’s important that you distinguish between her style and parts that are prescribed.

I’ve used work samples in three different ways

1. Reviewing candidates work sample together with the candidate in an interview (most often)
2. Providing work samples to the candidate in an interview and asking her to evaluate them (more often)
3. Evaluating candidates work sample by myself (less often)

Potential things to explore in a roadmap work sample

• How is this roadmap linked to the company goals?
• What kind of roadmap is this? An impact map? A user story map? A release map? A timeline with deliverables?
• Does it contain exact scope and deadline? Where is it flexible?
• Are the goals achievable ones or stretch goals?
• How big is it? Is there an end to it?
• What does it contain? Functionality? Impacts? Financial metrics? Quality metrics? Experiments?
• Who’s the target audience?
• Who took part in creating it? The team? Stakeholders?
• What’s the time horizon? 3 months? 6 months? 3 years?
• How is it maintained, and by who?

Potential things to explore in a backlog work sample

• What does it contain? Outputs or outcomes? Problems? Tasks? Solutions? Mix?
• How detailed is the backlog? Are all items equally detailed?
• How long is the backlog? (10 items? 100 items? 500 items?)
• How’s it organised? Per issue type (bugs, feature requests, tasks), type of customer value?  Or something else?
• Who took part in creating it? The team? Stakeholders?
• How is the backlog aligned with the company’s roadmap? Is there a clear connection?
• How much time does she spend maintaining the backlog?
• How does she maintain it?
• Where is the backlog stored?
• How many backlogs are there? Idea BL,  Product BL, Sprint BL, System BL?

Potential things to explore in a prototype work sample

• What was the first hypothesis?
• Who was involved in defining it?
• What happened? Was the hypothesis correct? If not, how did it evolve over time.
• Where and how is learning stored?
• What did the different iterations of prototypes look like
• How did she test the hypothesis?
• How did she determine the magnitude of the prototype?

Providing work samples of your own
If your company does things in a very specific way or if the candidates work samples differ a lot from your style it might be valuable to provide work samples that you think are good and then have the candidate evaluate them. Ask them what they think is good and what they think is missing. Explore which parts they disagree with and which parts they don’t understand, or explore what they would need to be able to create artifacts that more resembles your own.

Running auditions
Auditions are real exercises that your candidates facilitate or run while you get to observe. When I’ve recruited this has usually been one of the final steps in our recruitment process due to the amount of coordination required to find a time that works with a team, myself, and the candidate. While auditions require the most amount of time and energy they also generally return the highest reward.

Examples of auditions are to have the candidate:

1. Run a session or workshop of their own choice.
2. Facilitate a planning meeting.
3. Facilitate backlog grooming.
4. Facilitate user story mapping.
5. Pitch their vision for the product/service/team they’re going to be PO for (but remember you’re not evaluating the content, you’re evaluating how she does it!).
6. Pitch the vision for her current team.

I usually try to set candidates up for success since that’s how we’d treat them if they got the job. That means that I share more context than they ask for and particularly relevant information about the group dynamics and domain knowledge they need to be aware of. I would also ask what help they needed, what kind of room they would like etc.

After running an audition you have a great opportunity to offer observations and feedback to see how she responds to it. Does she go into defensive mode, shut down, or does she explore it with you and listen? This is also useful information.

Poor performance is not always caused by inexperience
Some people get really nervous in interviews and if someone performs poorly it does not necessarily mean that they do not know the role. They could just be nervous or maybe jetlagged if they’ve travelled from a distance to conduct this interview, etc. It’s also quite uncommon to run auditions so people tend to get nervous because they’ve never done it before.

If you have exercises that you run or work samples that you ask for that I haven’t listed above, please let me know. I’d love to learn new ways of evaluating product owners. And if you decide to try feel free to reach out in before hand for a casual conversation or afterwards to let me know how it went.