Metrics Store Summit 2022

Nick Handel, Co-founder and CEO of Transform will share his vision for handling metrics as code and dive deep into elements of a strong metrics framework. He’ll also look at the modern analyst workflow and how metrics layers are changing the way that data teams work and interact with business teams.

Long before the concept of the “metrics layers” hit Twitter feeds and Substack newsletters, technology companies built metrics stores internally to understand their business, conduct experiments, and share metric logic. Hear from data experts at Airbnb, Spotify, and Slack on what they’ve learned along the way. We will end the mini sessions with a Q&A with all of the experts.

The magic behind Airbnb’s Minerva

Airbnb’s Minerva helps the organization manage the entire lifecycle for metrics. Minerva provides a single source of truth for metrics, and powers many data applications in Airbnb. Xiaohui Sun, Software Engineering Manager at Airbnb will cover: Why the team built Minerva, how Minerva’s architecture covers the full lifecycle of metrics, and an overview of applications powered by Minerva including reporting, analytics, experimentation, metadata management, etc.

Experimentation and metrics at Spotify

Five years ago Spotify set out to build a new experimentation platform. Their goals? To increase the quality of metrics and make analysis simpler. Johan Rydberg, Principal Product Manager at Spotify, will share learnings, including how his team is now implementing the second iteration of their internal metrics store, with the goal of standardizing metrics for all of Spotify, not just experimenters.

Metrics-at-scale at Airbnb and Slack

Aaron Keys, Senior Principal Data Scientist at Slack, will compare and contrast two large-scale metrics repository projects that he worked on at Airbnb and Slack. He will discuss the design tradeoffs that were considered by his team, successes, and mistakes made along the way. Aaron will close with views about how these types of internal projects relate to open source projects and the new companies in this space.

Engagement Metrics Framework at Facebook

Siva will talk about how the data engineering team at Facebook built a framework to compute growth and engagement metrics at scale. The talk will focus on why Facebook needed this framework, the secret sauce and the cost of not building it.

For as long as analysts have relied on SQL, those in the data community have been trying to figure out ways to make analysis reusable. On the surface, it seems imminently possible and immensely valuable: Software is eating the world; we’re all becoming tech companies; we all use the same tools and try to solve more or less the same problems. Yet, despite that, data practitioners still have to rebuild many of the same models and metrics—from pipelines that clean up Salesforce data to dashboards of daily active users—from scratch.

Our struggles to solve this problem are explained by two phenomena: It’s neither as easy nor as useful to build analytical “libraries” as it appears. There are no frameworks for sharing analytical scaffolding, and the small distinctions in our businesses get amplified into very meaningful differences in our reporting needs. As a result, little of the analysis we create is repurposed across the community, and collective best practices only advance through the slow propagation of tribal knowledge.

The metrics layer could change all this. It, along with other recent developments in the data stack, could provide the technological foundation that finally makes community reusability possible. In this talk, I’ll share how metrics layers have the potential not just to unify a company’s metrics governance, but how they could also shape the giant’s shoulders we’ve long been trying to stand on.

Within her creative practice, Stefanie uses data-gathering and data visualization as a design process to communicate warm, subjective and often emotional messages. She will show how in an era of ever-increasing data, we all can – through channeling our inner 'anoraks’ – start to see observing and measuring as valid forms of making and creating as well as an integral part of being human.

After her talk, Stefanie will take you through a fast-paced workshop where you will learn data visualization basics through visualizing your personal data points and learn more about yourself in the process!

Materials needed: Plain letter paper and a thick dark pen.

In a previous time, BI was the only interface to data and the data warehouse. Today, that’s no longer true. It’s allowing data teams to be more impactful to organizations than ever before. In this talk, we’ll talk about the five workflows data teams are tackling and the new technology that is enabling them, as well as what this means for “self-serve analytics” in a new metric-first world.

You can’t talk about metrics stores without talking about how they fit within the modern data landscape. We’ve invited experts from across the modern data stack—from experimentation and observability to business intelligence—to discuss how the metrics layer will impact analyst workflows, both in the short term and the future.

There has been a lot of buzz about running data teams like product teams. But once you get to this point, how do you manage the team? Borrowing management strategies from engineering organizations enables analytics teams to scale and better measure their impact on decision making. Belinda Bennett of Transform and Cynthia Beal of Lithic explore how to leverage product and engineering practices to run, measure, and grow your analytics organization.

The metrics layer is a relatively new concept in the data space and naturally, different vendors are approaching this concept in new and exciting ways. Hear from vendors about their plans for the metrics layer and their vision for the future. You’ll also have the chance to ask questions about their products in our Q&A.