Officers in Fort Worth, among cities that have worked with the Behavioral Insights Team to test messages encouraging police force applications (Photo credit: Fort Worth Star-Telegram)

Why Null Isn’t for Naught

Lessons from the Behavioral Insights Team

By Siddharth Mandava and Sasha Tregebov

“It had no impact.”

“The new approach didn’t work.”

“Our test found no difference.”

Conclusions like these may sound pretty discouraging, but they don’t have to be.

As a partner in the What Works Cities initiative, we at the Behavioral Insights Team (BIT) work with cities to conduct randomized controlled trials (RCTs) that test a newly proposed solution against a city’s business-as-usual practices by randomly selecting some people to receive the new solution and others to receive the status quo. This approach allows us to understand the effect of the change we made while controlling for any differences in the broader environment. Through this process, we hope to find that our new approach was more impactful than the status quo. However, we occasionally find that’s not the case.

The City of San Jose worked with BIT to test postcard messaging that would encourage more residents to use the City’s large item collection services, helping to reduce illegal dumping.

While a null result means that we cannot say with statistical confidence that the new program or process had a greater impact than business-as-usual practices, they do not indicate that the trial has failed. In fact, they often provide valuable lessons that can inform future efforts. Consider the case of Albert Michelson, who failed to find evidence that light moved through ether, and yet won a Nobel Prize in large part due to this experiment. Why? Because his “failure” disproved a widespread belief and helped Albert Einstein develop his theory of special relativity.

In an example that’s a bit closer to home, we worked with the Chattanooga Police Department to send postcards to residents to encourage them to apply to the department. These postcards were designed based on research from the behavioral sciences. Of the four different postcards we tested, two significantly increased the number of applications, an impactful and promising result. However, we were equally interested in the fact that the other two postcards, which highlighted public service and impact, did not have an effect on application rates. Because these two postcards did not work, we learned that the postcards’ message mattered and that some messages work better than others. In particular, we learned that the Chattanooga Police Department was already reaching individuals who are motivated by public service through their previous recruitment efforts, and that they should instead highlight different, less conventional motivators to attract additional applicants.

Postcards with behavioral-science-informed messaging sent to Chattanooga residents to encourage them to apply to the Police Department

As with the Chattanooga police recruitment trial, we believe that null results are important for cities as they seek to improve their evaluation efforts and promote evidence-based decision-making. Based on our experience, there are three steps that we recommend cities take after concluding an RCT that shows a null result.

1. Celebrate!

You’ve just completed a randomized controlled trial, one of the most rigorous evaluation methods. This demonstrates real leadership in the use of scientific evaluation methods in local government. Developing the tools and experience to run RCTs within your city is a valuable process in itself. Testing allows you to generate evidence to determine the effectiveness of various projects, and can be implemented for numerous city processes, not just the application of behavioral insights. So take a moment to congratulate yourself and your team; grounding your decision-making in empirical evidence helps ensure that you continue to serve your communities as effectively as possible.

2. Ask Why.

There are a few reasons why an RCT may have resulted in a null result, and it’s worth taking the time to analyze whether any or all of them may have impacted your evaluation.

• Did your new intervention reach the intended recipients? No matter how effective a solution you design, if it doesn’t reach people it will not make an impact. For example, if your city emailed residents about a new program, it’s possible that the emails were accidentally marked as spam. For a meaningful evaluation, you must ensure that residents actually access the relevant city services or communications.
• Was your sample large enough to meet the statistical requirements of your RCT? If not, think about whether you could rerun the trial with a larger sample, perhaps by extending the timeframe or expanding the program’s reach to a few more neighborhoods.
• Did you correctly identify the problems that residents face when making a decision or taking an action? Understanding these barriers is paramount when designing a new solution, and our immediate assumptions may not actually hold true. So while we recommend conducting background research and fieldwork before introducing a new solution, they can also help after the fact either to confirm or update your understanding of the relevant barriers.
• If you correctly identified the barriers, was your proposed intervention strong enough to overcome them? If not, look into best practices from other cities, talk with your community leaders to understand their needs, and draw on the ingenuity of your fellow city staff. Most importantly, continue testing your ideas!

3. Share.

It may not be as exciting as publicizing a significant result, but sharing null results with partners, stakeholders, and other cities is just as important. Not only does it foster transparency, but it also helps others understand the context surrounding your evaluation and why it’s important to test. And of course, be clear on what exactly you learned; you don’t want other cities thinking that all postcards are ineffective when what you really tested was the message. For cities to avoid repeating unproductive projects and develop their understanding of what works, they first have to be aware of what doesn’t work. As Thomas Edison said, “Negative results are as valuable to me as positive results. I can never find the thing that does the job best until I find the ones that don’t.” It’s one of the reasons that the scientific community is also embracing this mindset and promoting initiatives like the Journal of Pharmaceutical Negative Results or the American Journal of Sociology’s call for negative results.

So the next time you get a null result, don’t be discouraged! Sharing all your evaluation efforts, whether significant or not, helps cities determine what works and what doesn’t.

Siddharth Mandava is an Associate Advisor focusing on local government and social policy with BIT’s North America office.

Sasha Tregebov, a Senior Advisor with BIT’s North America office, manages BIT’s work with U.S. cities through Bloomberg Philanthropies’ What Works Cities initiative.

Continue reading: Get BIT’s eight tips on what your city can do today to generate evidence and outcomes.