Types of Product Sense Questions in Data Science Interviews

Analyzing a Metric Related Problem

Analyzing a Metric Product Sense Questions in Data Science Interviews

The first type of the product sense questions is about analyzing a metric related problem. These questions usually mention a metric related to a company’s product and that it has been in some way affected. The solution should give your thought process on why this has occurred.

Let’s look at some examples. The most common scenario for metric-related questions is that the usage of some feature has changed by some percentage. Like in this question in which Facebook sees that likes are up 10% year over year and asks why could this be? But the question can also concern two metrics. For example, in this case Facebook asks if 70% of Facebook users on iOS use Instagram, but only 35% of Facebook users on Android use Instagram, how would you investigate the discrepancy?

The concept of a metric is naturally important in these product sense questions. However, in most cases the examples of metrics are already given in the question. Here, the metric is the share of users of certain platforms who use Facebook and Instagram among those who only use Facebook. There is no need to come up with your own examples of metrics and that’s what makes these questions a bit easier. The most important is to correctly identify the relevant metrics based on the question and understand the relevance of these metrics to the product or the problem. In this example, the metric in question has to do with three aspects: the mobile operating system, the Facebook app and the Instagram app. All of them interact with each other in certain ways and are immensely important to Facebook since the company owns both social platforms and mobile users are an important target group for them.

What metric related questions are really asking about ties directly into the concept of pain points. Pain points are parts of the product that displease the customer and that make them voluntarily stop using it. In the Facebook example, we can try and speculate of pain points that may cause the discrepancies in statistics. Maybe the Android Instagram app is slower, less intuitive or heavier than its iOS version?

Another key concept is that of identifying tradeoffs between products and specific metrics. See how Facebook, the company, owns both social media platforms: Facebook and Instagram. In such a case, when they make different decisions about the two platforms, it may be reflected in metrics. For example, the company may follow the best marketing strategy and promote Instagram to a different target audience than Facebook, like people who are younger and more brand cautious, a demographic that is also more likely to use Apple products. This may result in the majority of Android users sticking to Facebook only.

Measuring Impact

The second type of product sense questions concerns measuring the impact of a new product or feature of a company. This can range from questions like this one from Twitter in which they ask How would you quantify the influence of a Twitter user? And questions specifically about features updates such as Quora asking What metric would you use to measure the impact of a search toolbar change?

Similar to the metric related problems, the concept of a metric is a key also in the questions about measuring impact. In this case, it is even more difficult because the candidate needs to give examples of metrics that can be used to measure the impact in question. Whenever suggesting a metric, it’s important to define it, explain the relevance of the metric to the goal, and possibly why a certain metric has been chosen over another.

What’s more, when measuring impact, it’s a good tactic to define 3 metrics: 2 metrics that measure success and 1 metric that does not worsen. When answering this product sense interview question, we can assume that with a new search toolbar, Quora wants to increase its click rate and the number of returning visitors, but at the same time, it doesn’t want the time it takes to find a piece of information to increase.

After mentioning, defining, and explaining reasons for choosing these metrics for the question, one needs to explain what methodology they are going to use for measuring them. In other words, how to derive these metrics? The good approach is to design and explain an experiment as specifically as possible. For example, the general experiment to test the success of a new feature is called A/B testing. It is similar to the concept of control and treatment groups and is especially useful when measuring impact of user interface updates.

But when suggesting an experiment such as A/B testing, it’s crucial to mention what user group or groups the experiment will concern. In the case of this question from Quora, we are trying to measure how the general public reacts to the new product, so we would want to run the experiment on groups split randomly. But in case we were expanding a product to new user groups, then it would make sense to run the experiment on specific demographics.

Designing a product

The third and the last category of the product sense questions is the one about designing a product. These are completely open-ended questions where there is no right answer and these are usually the hardest questions in Data Science product sense interviews since these test candidate’s thought process rather than the answer itself.

Let’s see some examples of such questions. Lyft can ask: “Describe how to engineer the heatmap telling taxi drivers where to go to maximize their probability to get a client. How do you define which area will have high demand next and who do you want to go there?”. Another example is this question from Microsoft: “What would you do to summarize a Twitter feed?” or from Yelp: “If you had to propose a new Yelp feature, what would it be?”.

What’s more, these types of questions may sometimes be extended to also ask about measuring the success of the designed product. For instance, after asking to propose a new feature, Yelp can then ask a follow-up question on how to determine if the launch of the feature has been successful. This will be tied back into the measuring success type of product questions.

When faced with a question related to designing a product, there are a number of concepts that should be included in an answer. First of all, it is important to define the target audience for the product of a feature. If neither the question nor the interview specify the target audience, state who you believe is the target audience and mention you are creating a product based on this target audience in mind. In the Yelp example, it may be the current site demographic but you may also want the new feature to draw new target audiences to the service.

Another important concept to consider are the Key Performance Indicators, or KPIs, for the product. KPIs are the key metric in a business problem and are based around these two types: Product or System Performance, for example app initialization time, and User Experience, such as how likely they are to recommend the product. If measuring KPIs requires any data, it’s important to explain what data these are and how they will be collected in the product. Coming back to this example question from Lyft, KPIs may include drivers’ earnings, total company revenue or the average time a driver waits between two customers. But in this last example, we may not have a system in place to gather the average waiting time data so we will need to derive this metric from the rides database.

There is one more concept that many candidates forget when answering questions about designing a product, namely, mentioning edge cases and how the solution accounts for them. Suppose your answer to this product sense question by Lyft is to predict rides based on past data by hour, day of the week, month, and year. An edge case for your solution would be when there’s a major event in the city, such as a music festival. As music festivals are not always held at the same place during the same dates, it could increase the demand for Lyft rides exponentially for a couple of days then die down immediately, which is hard to predict if you just use past data. Real-world data collection of major events like these need to be incorporated, but again, it is crucial to make it clear how the data pipeline will roughly look like.

Conclusions

To summarize, we have shown you three types of product sense questions. The analyzing a metric related problem questions ask you to explain changes in specific metrics. When answering them, understand the relevance of the mentioned metrics, identify relevant pain points and think if there are any trade offs involved.

The questions about measuring impact ask to suggest ways of quantifying performance of features and products. Define several metrics and remember that while for some metrics the goal is to improve them, for others, we shouldn’t see them worsen. Name specific experiments you would use to measure the impact and don’t forget to specify which target audiences to focus on.

Finally, in the case of product designing questions, there is usually no right answer and it’s the thought process that counts. A good answer should define the target audience, mention the relevant Key Performance Indicators, describe all new data pipelines required for measuring impact and cover the most important edge cases.

Check out our ultimate guide to product interview questions to find out how to prepare for these questions before entering the interview. Also, find all other types of interview questions you should prepare in this article "data science interview questions".