Notes from my Product Management Interviews Part 5: Figuring out Metrics, KPIs, and OKRs
This story is written as part of a seven-part series: ‘Notes from my Product Management Interviews’. Read Part 1 here , Part 2 here , Part 3 here and Part 4 here.
Part 4 was fun, as we discussed in detail the best approach for a Problem Solving interview, and what it means and takes to be a Data-driven Product Manager. It was also the longest read in this series so far, although I hope you will find it useful. My goal through these posts is to spread practical awareness around PM interviews so that ambiguity is reduced and one can accelerate the process of preparation for these interviews. The more the awareness, the better!
In this part, we will discuss another key aspect that a Product Manager needs to be very good at. It’s how you assess in which direction your product is headed, and how you can quantify, simply put, the success or the failure around it. To be more specific, the topic for today’s post is all about:
Figuring out Metrics, KPIs, and OKRs
First things first: Let me add the definitions of Metrics, KPIs and OKRs. As always, never get bogged down by terminology: ask harmless but helpful clarifying questions if you aren’t sure what the interviewer is hinting at.
Metric: A metric is a quantifiable datapoint or a measure, which helps us understand how good, or how bad, or how ugly, or how alright (basically — the status) a product, a feature, or a process is doing, in a current scenario. As the status changes, the consequent actions as a Product Manager changes in the sense that we either have to maintain what we are doing right, if status is green, work diligently to fix things if status is red, and so on.
KPI: A KPI, or a Key Performance Indicator, is a smaller measure value that helps us measure the effectiveness of a product achieving a key business objective. Multiple KPIs come together to make a Metric.
Here’s a link that deep-dives into the definition of Metrics and KPIs with examples: https://www.klipfolio.com/blog/kpi-metric-measure
OKR: Objective and Key Results is simply a charter, which comprises of an objective, which is a goal, and key results, which are measurable attributes that will help us realise if we have fulfilled the objective that we wanted to achieve.
Here’s a link that deep-dives into the definition of OKR, with examples: https://felipecastro.com/en/okr/what-is-okr/
One more OKR examples link: https://www.keka.com/product-managers-okr-examples/
It is very crucial that you understand these terms well and ensure you never mix them, especially during an interview. Talking about metrics/KPIs/OKRs is easier and much less ambiguous that responding to a Product Design question, if you develop a strong sense around what these terms really mean in the simplest way possible (let all complexities remain for after you have landed an offer and have started working as a rockstar PM!)
Product Manager: The Great ‘Metric’ian!
There are so many things that a Product Manager owns but a key aspect to their ownership comes from the fact that they need to drive home an idea from a concept all the way to execution and beyond. A good Product Manager needs to be very good at understanding what they want to learn about the user behaviour around their product, how their users are using the product/understanding the product, and how all these learnings align back to the company’s business objectives for the product. This is why metrics are so important for a PM — they not only help them achieve these goals, but also help them talk to their teams about the vision that they have for the product, when they strategise the direction which the product will take.
A product or its feature(s) success or failures can thus be defined by metrics. But before that, one needs to be good at identifying which metrics to choose. In your metrics-oriented interviews you’ll be assessed on the grounds that whether or not you are choosing the right metrics to measure the success of your product. This is due to the following:
- Data is abundantly available. So, it is easy to unknowingly choose a wrong metric, work through the influx of the data to come up with an analysis, which in itself is a time-consuming process, and, by the time you are done with your analysis well enough to measure the success of your metric, it might have been too late and you may have lost, say, a bunch of your customers
- Choosing the right metric shows that you are aware of the industry your company is in, and can sense your market well. A great metric for an online shopping app may not be useful at all for a social media app. Thus, I recommend that before you interview for a company, read up about its industry, its market, its competitors, what makes the market interesting, what a typical customer in that market expects from the company, and so on. Initial research like this helps you in being quick with what metrics you pick, minus the clutter
There are many good resources available that would help you get a sense of some of the key metrics that you should know, as a starter pack. Understand that a metric will, largely, either be customer-oriented (eg- number of users on-boarded in a given time period, say, one month), or revenue-oriented (eg- average revenue per user in one month). You can customise the way you chalk out the metrics but at the end of the day, it should be quantifiable.
- The Zepel Blog on How to Measure Success of Your Feature ( I found this to be a very useful read, especially the ‘Questions To Ask When Thinking About Feature Metrics’ part that they have written): https://zepel.io/blog/measure-metrics-for-feature-success/
- 15 Key Product Management Metrics: https://www.altexsoft.com/blog/business/15-key-product-management-metrics-and-kpis/
- Startup Metrics (only for familiarity, as these might sound too technical or bookish for an interview): https://a16z.com/2015/09/23/16-more-metrics/
- StellarPeers post on How to Measure Success of Amazon Echo (includes a suggested structure and solution): https://stellarpeers.com/how-would-you-measure-the-success-of-amazon-echo/
- More interesting notes about metrics (a Sequoia Capital article): https://www.sequoiacap.com/article/defining-product-success-metrics-and-goals
Before I move on to the end of this post, let me also write a bit about vanity metrics.
What are Vanity Metrics and Why You Should Avoid Them
Simply put: Vanity metrics are those metrics that might make your product look good in front of others, but may not actually help you understand the success or failure of your product, or help you build the next course of action, the future strategies for your product.
It is essential that you understand the meaning behind this type of metrics because you’d then be able to identify naturally that why such metrics should be avoided, especially during interviews. In addition, do understand that vanity metrics are not actionable in nature, and it is unwise to invest money to see these metrics improve, especially at the expense of metrics which are actionable such as, let’s say, Customer Acquisition Cost.
Talking about vanity metrics during interviews, simply put, gives off an impression that you are not thinking about a product’s success as a Product Manager but you are seeing a product’s success as its user, who is not concerned with the behind-the-scenes aspects of the product.
Eric Ries, author of The Lean Startup puts it this way: “Vanity metrics are dangerous.” 😜
Here are some quick links for you to understand more about vanity metrics:
- Vanity vs Actionable Metrics: https://fizzle.co/sparkline/vanity-vs-actionable-metrics
- Metrics, Metrics On The Wall, Who’s The Vainest Of Them All? (from Neil Patel’s blog, with good examples): https://neilpatel.com/blog/vainest-metrics/
A little more text about KPIs and OKRs
Do understand that sometimes KPIs and metrics and even OKRs might be used interchangeably to mean the same thing. Having a good sense of the difference should help you ask clarification questions, so that you realise well in time what the interviewer really wants. Reframe the problem statement accordingly, and you should be good to go!
As a PM, you will need KPIs to ultimately define if a metric is showing success or failure of what you have built, and thus you will be expected from time to time to explain these to your team. OKRs can help you realise that a given objective is only be considered achievable or a goal worth pursuing, if it has measurable results attached to it, that will give everyone in your team a sense of value-based outcomes. Easier said than done, yes, but not an ultimately impossible task!
As usual, I will add some of the Metrics/KPIs/OKRs questions that I have faced in my interviews so far. You can work through the questions and let me know if you have any doubts.
Questions around Success Metrics, KPIs, OKRs
- If you were the CEO of Amazon, what would be the top 5 metrics that you will track on a day-to-day basis?
- What are the KPIs that will help you assess if a review feature introduced in a newly-developed e-commerce app is helping the company achieve its Business objective (let’s assume that the objective at this point is: Increase number of new users of the app by 10% over the next 3 months)
- What are the top 3 metrics to track for Uber’s cab business?
- Make a list of some top features of any SaaS product. Map each of these features to the top 4 metrics of the product, by assuming that each feature will fulfil one or more of these metrics. Talk through your approach.
- What are the top 3 OKRs (I am not sure why this was worded this way, but assume it means 3 objectives with more than one key results for each objective) for a mobile app - based lending platform for college students?
While it is understandable that making a sense of metrics, KPIs and OKRs might seem a bit daunting, but trust me, with practice you’ll realise that the interviews which focus on these are actually much easier to crack, if you have a good grip on the basics.
Feel free to reach me at contactsoumyam@gmail.com for feedback, questions or ideas. If you liked this post, do add claps below — it is a nice ‘vanity metric’ for me that I am doing things right with this series. ;)
Thanks for reading!
Disclaimer: credits to all images used in this post are attributed to the original source in the captions. The author takes no credit for the awesomeness of these images and consequently all copyright claims should stay within the original source’s boundaries
