1.
Conviction around a purpose rather than searching for meaning in numbers
The first step to being data-informed is understanding what data can’t do: give you a purpose.
Data does not substitute for a mission or a strategy. It cannot uncover a set of values. Metrics are merely proxies for what matters. “Increasing Metric X” is not a purpose; a true purpose must relate in some way to creating value for other humans. If a data-informed team feels at any point like they are optimizing metrics in a way that compromises their mission, they scrap that work.
Before you can collect data to help you track what matters, you need to define what actually matters. Information itself is not an evaluation criteria.
2.
Setting verifiable goals rather than vague aspirations
How will you know if you are fulfilling your mission? You imagine outcomes that build towards the future you envision. Then you set goals to help you verify whether you are achieving those outcomes.
Vague goals that are difficult to verify aren’t useful and create confusion within a team. We want to make our users happier. We want to make the world safer. We want people to be more productive. These don’t give you an objective baseline to evaluate different strategies or tactics. And you won’t know whether you’ve achieved them.
Data-informed teams push for quantitative goals to the greatest extent possible because they’re the best way for a team to focus on creating impact. They make progress transparent, force accountability, and rally your team around a shared outcome.
Goal-setting is more art than science. All metrics are proxies and every set of verifiable goals will have shortcomings in what they fail to capture. As you learn those shortcomings, you will iteratively refine your methods of measurement and your targets. Setting goals is a skill — you have to practice it to get better.
Data-informed teams adopt verifiable goals to drive impact while acknowledging these imperfections. They recognize that goals serve the team in their march toward the mission, never the other way around.
3.
Company-wide familiarity with metrics rather than outsourcing to “data people”
Everyone has to know the numbers. You cannot outsource a data-informed culture.
You can have the best team of analysts — but if you don’t know the numbers, then your decision-making will suffer.
Data-informed teams regularly meet to review key metrics. Why? Decisions are never made in these meetings, so why have them at all?
These meetings signal something important about the culture — that simply knowing and talking about the key metrics matters. Doing so builds a shared foundation among a team of what is going well or not well, and how they might best prioritize efforts.
Good data-informed teams develop leaders who can weave data into clear narratives. These teams cultivate a skilled understanding of which data patterns are important and which ones are not. They point out conflicts in data and which interpretation is more likely. Most importantly, they are upfront about what the data can and cannot say.
4.
Active testing of beliefs seeking support for intuition
We all have hypotheses about what our customers care about, what products will win, and what decisions are best. The more experienced we become, the more we trust — and even pride ourselves on — our intuition.
A well-honed intuition about which path to take is supremely valuable because we don’t have infinite time and resources to try every path.
The danger comes when your pride in your intuition leads you to close yourself off to evidence that you might be wrong. Instead of testing your intuition with data, you seek out data that confirms your intuition.
Openness to being proven wrong is insufficient. Data-informed teams actively seek out information that might disconfirm their assumptions. They value a high velocity of experimentation and setting hold-outs. Like scientists, they are constantly looking to test their hypotheses and validate their beliefs. They don’t view intuition versus data as a forced choice — they use data to refine their intuition over time.
Members of data-informed teams regularly ask each other: “What evidence would convince you that your intuition is wrong?” If the honest answer is “No piece of data would convince me,” then you have strayed off of the data-informed path.
5.
Accepting probabilities rather than thinking in absolutes
Data will never give you certainty. Interpreting data means taking on many assumptions that are reasonable but not bulletproof. Even trusting a single piece of analysis means having faith that the events were logged correctly, that the metrics have been calculated without error, that everyone understands exactly what was logged and calculated, and that the interpretation is sound. You want to be 100% sure? Good luck.
People who are not truth-seekers constantly take advantage of this. The easiest way to discredit data is to demand perfection from it.
When data and intuition collide, some people always pick their intuition. They would rather be wrong betting on their intuition than wrong betting on data (this is rampant in the world of sports). That is a defeatist mindset. Your commitment to your mission requires that you make good decisions fast. Sometimes you will trust the data and you will be wrong. But if using data increases the likelihood of making the right call by 5–10%, those benefits will quickly compound.
This is the promise of data — to help us gain an edge in our quest to build wonderful things for other humans.