Data-driven design

One skill I wanted to learn at my new gig was to use data to inform design decisions. This was the first time I have had the opportunity to work on a product that is serving users on the scale of hundreds of thousands to millions, which means there was tons of data to poke into and play with. We have a full data science team working running experiments and analyzing the results to inform where the product should go in the future. Coming from an agency that just built the MVP version of a product, shipped the V1, and never touched it again, the idea of staying with one product to look at how the numbers evolve over time and tweaking the product accordingly seemed like a very exciting opportunity to me.

Designers don’t seem to fully understand or appreciate how data can help them become better designers. There’s different types of ways to design with data. There’s being data-driven, where a statistically significant change in a key metric such as retention or time-in-app will literally dictate the features that need to be worked on and will in turn require design to make the right changes that will move the metrics to what we need them to be. There’s being data-informed, where designers and product managers can use specific insights from the data as a starting point to discuss changing a feature in a certain way to ensure that we don’t break what’s already working well and improve what’s not. Finally, there’s being data-aware, where we can look at the data and comparatively weigh it against our qualitative insights from user tests and user interviews to determine the next step forward. Being data-aware doesn’t mean you’re ignoring the data, it just means that you’re giving both your quantitative and qualitative learnings equal weight when making a decision.

Most designers are afraid of designing with data because they immediately assume that they will be doing “data-driven design”, the world where all design must be done to move specific metrics to meet certain numbers. Designers especially hate this because it’s nearly impossible to properly measure the value of emotional design. These are the moments of delight in a product like a neat little pull-to-refresh microinteraction or a fun loading spinner or custom illustrations in the onboarding experience. These are all so subjective that they don’t impact any metrics directly but have a massive impact on the user’s perception and appreciation of the product. Even in qualitative user tests, it’s hard to tell whether people are “enjoying” your product (unless you have an extremely expressive participant). Most users simply use it with a blank expression throughout the entire test and later remark that they like specific aspects of it when questioned about their experience. Designers realize the importance of these little details, which is why they try to use any and every opportunity they get to try and get these into the product.

Most startups like to operate “data-driven”, where they want to measure everything and know how the work being done is moving some key metrics. It makes perfect business sense to do this, as the organization can track the productivity and efficiency of the work being done to great accuracy. I’ll argue that design doesn’t necessarily need to be data-driven. It’s the product decisions that need to be data-driven. Design only needs to be data-aware. As long designers aren’t actively ruining things that are already working well, they won’t harm the metrics. Design should, however, be involved in the product conversations where being data-informed helps a lot. This is how design will get a chance to actually improve the metrics. Being involved in the beginning of a feature discussion provides the designer with so much more context about what is being done and why and informs them what data their design is trying to change or improve.

As an example, I’m about to start a major redesign of an app at my current role. We already have an existing version of it, but it’s very barebones and is in dire need of an overhaul. Before beginning to do any snazzy UI sketches, I reached out to the data science team to give me any data that they’ve currently got about what buttons our users are tapping the most in the app, how users navigate within the app, where users tend to make mistakes, what types of things users are trying to find but can’t, and how frequently users perform certain actions. I don’t plan on relying solely on this data to do the redesign. I already have a wealth of data about how actual users use the product through user tests from the past few months. I’ll only be using the data to validate some of those qualitative insights and will be looking to see if there are any major contradictions to them. After all, we only ran user tests with a disproportionately smaller sample than our entire userbase. I certainly expect to find that most of our users actually use features that many users during our qualitative tests said they never do. It’s simply a way of cross-checking your sources to validate the information you have. This is a perfect example of being data-aware when designing.

When you approach data in this intentional manner, it starts to look a lot less intimidating. Instead of looking at data as this foreign thing that’s going to mess up your design process, you start to look at it as a tool you can use to better your design. Better yet, you can objectively claim that design contributed to making the product better by simply measuring the same metrics after the redesign and comparing the before-and-after. If there’s still issues, you can always keep working on them. Data actually helps clarify that what you’re doing matters. It upsells the value of design to everyone at the organization and proves that it moves the needle on key business metrics.

Sure, it may not be feasible to measure the user “delight” from a beautiful splash screen animation, but these are things that you as a designer will know aren’t measurable and yet matter a lot. It’s your job to fight for getting these in because while they might not impact any key metrics, they will greatly affect the overall user experience. Getting the user in a calmer and relaxed emotional state at the start of the user flow definitely changes how they feel about the rest of the registration experience. It’s your job to explain this to your peers and confidently claim that this is actually more important than some other measurable change that won’t matter as much. The more you do it, the easier it gets. People might be obsessed with tracking revenue, retention, time-in-app, and daily/monthly active users, but we can’t yet track our users’ emotional states and feelings very well. The happier users are and the easier it is for them to get tasks done, the more likely it is that they’ll continue to use the product. This is why it’s important to have the right balance between quantitative and qualitative data while making sure to not rely on one too much over the other. In my experience so far, being data-aware has been the best way to maintain this balance. It will certainly be different for designers in different organizations, but if the thought of designing with data scares you, starting off with being data-aware is a great way to get a feel for the power of data.