Skip to content

Product Analytics

Why Product Analytics

Understanding user behavior in a variety of contexts can lead to better-targeted campaigns, increased revenue, and greater user satisfaction and engagement for any product. Every company is tasked with understanding, altering, and predicting user behavior. when you understand who your users are, what they are doing with your product, and what drives them to purchase and engage with your product, then you can try to modify their engagement and revenue behavior.

Behavioral Data

  1. Click stream data is the path of clicks through a website or model product ordered by time.
  2. User sessions comprise a pattern of consistent use from the first to last interaction on a site.
  3. Churn is the number of users or rate of leaving the site over a particular period.
  4. Bounce rate is the proportion of users who leave the site after viewing only one page

Common Pitfalls

  1. When we focus on only a couple of descriptive facts, we do not get a holistic picture of what is happening in the product and eventually we just get lost in the details.
  2. Machine learning prediction is not as useful as causal inference in deriving insights or changing user behavior because it does not help us find variables that cause a user to behave in a certain way, such as deciding to purchase.

Basic Principles

  1. The very first step in understanding your users is building a conceptual model and collecting the right metrics or descriptive statistics to verify or falsify that model. Next, we use metrics, experimentation, and statistical inference to derive insights about users. Finally, we focus on user behavioral change in a web product based on understanding what causes behavior, or answering the why question.
  2. Determining what causes a behavior and understanding the size of its effect.

Social Process of Using a Product

  1. Social behavior is a process, not a problem to be solved. Users are coming to your product repeatedly and interacting with people and content. They are also creating content, building communities, reinforcing norms, and building culture, which in turn affects the environment that other users interact with.
  2. Social systems are open systems, meaning there are omitted or unmeasurable variables that can affect outcomes. Variables reflecting events that happen rarely can greatly affect social processes, are very difficult to predict, and are often left out of models. Because many variables are present in social systems, and we have little understanding of their complete effect, some will always be left out.
  3. When exploring social behavior, there are often no clear and defined outcomes. In a social process, behavior is easier to understand if you can understand your users’ incentives and the causal impact of different actions on user behavior. Causal effects and user incentives are key to understanding what is happening with your product, but are left out of predictive frameworks.
  4. Social systems have rampant problems of incomplete or one-sided information. It’s easy to mistake quantity for quality of data. However, often data on the most important variables in describing user behavior is not easily collected. The primary variable that is left out of analyses but is vital to determining behavior is motivation or user goals. Users of our web product probably have a variety of motivations, and these motivations impact how they access and view the web product. Having an understanding of the individual goals and motivations can help us improve our product.
  5. Social systems consist of millions of potential behaviors. Omitted information like goals and expectations is very important, yet information we may never obtain. Instead, we could have thousands of much less useful variables that we need to sift through to find the right ones to describe the process.
  6. Inferring causation or why something happens is almost impossible. Creating an A/B test setting os very difficult in a product environment

Techniques

  1. A counterfactual is an alternative situation where everything is the same except the variable that influences our outcome. It allows us to determine how this one difference would change the result.

Predictive vs Causal Inference

  1. Causal inference does not necessarily improve with more data. Causal inference relies on having a valid counterfactual or “placebo” test. Conversely, massive amounts of data do improve predictive models.
  2. Prediction also allows for generalization and validation based on external or new data. Causal inference is difficult to generalize and validate outside of an experiment.
  3. Machine learning prediction is not as useful as causal inference in deriving insights or changing user behavior because it does not help us find variables that cause a user to behave in a certain way, such as deciding to purchase.