There were high hopes last year when Penn researchers and Philadelphia policymakers launched the Philly Vax Sweepstakes, a project designed to increase COVID-19 vaccinations by randomly awarding cash prizes to some residents who got the shot.
But the carefully designed, six-week program yielded mixed results. While the overall sweepstakes may have generated extra vaccinations in Philadelphia, its test of targeting specific zip codes with higher odds of a win definitely didn’t work.
Katy Milkman, a Wharton professor and co-director of the Behavior Change For Good Initiative (BCFG) at Penn, is the lead author of the study published this month in the journal Nature Human Behavior. A team of scientists at BCFG partnered with city officials to design and fund the sweepstakes, which was unique from other vaccine incentive programs that were rolling out across the country in 2021. Instead of winning free food, scholarships, gift cards, and other items, Philadelphia residents were treated to a “regret lottery” in which they were automatically entered if they lived in the city (whether or not they’d been vaccinated). They were then contacted if their name was selected and could win up to $50,000, but only if they could prove that they’d taken the shot before the drawing.
The university and the city worked to spread the news about the Philly Vax Sweepstakes through local media and social networks, with a clear message that residents of three particular randomly selected ZIP codes (one picked every two weeks) would have a 50 to 100 times greater chance than other Philadelphians of winning cash. The researchers randomly chose these three ZIP codes from among 20 with the lowest vaccination rates in the city, hoping the boost would incentivize more residents in those ZIP codes to get vaccinated. But the bigger grab at cash did nothing.
“We got an incredibly clear answer: no. It was totally useless to multiply the chances by 50 to 100 of residents in certain ZIP codes winning. We do not see a benefit of that,” Milkman told Wharton Business Radio on SiriusXM.
“Our best guess is that we generated about one extra vaccine per $15 spent on this regret lottery, so not a bad return on investment at all.”— Katy Milkman
The controlled part of the experiment didn’t produce even one extra vaccine for every $5,000 spent. The researchers confidently ruled out any benefit. Milkman said the result is still useful because it shows that geo-targeting, which seems like a sound idea, should not be considered as a way to motivate people.
The professor said there may still be overall value that can be extracted from vaccine regret lotteries, and she believes more research is warranted.
“My takeaway is, this tactic needs more testing,” Milkman said. “We aren’t sure, but we probably increased vaccinations overall in the city of Philadelphia. Our best guess is that we generated about one extra vaccine per $15 spent on this regret lottery, so not a bad return on investment at all.”
The experiment’s design of a regret lottery was based on prior research indicating that a feeling of regret can be a powerful motivator. Milkman said although the Philly Vax Sweepstakes yielded equivocal results, it was generally more promising than the other types of incentive programs that were unveiled around the country last year with a lot of fanfare but little uptake. Another evaluation she co-authored (led by Penn Medicine professor Harsha Thirumurthy) looked at state vaccine lotteries and other incentives, and the results were much more discouraging.
“Maybe there would be value in more cities trying out regret lotteries,” she said. “It could have been a fluke; our results were equivocal. But it was encouraging enough that its worth more testing at that local, community level using this kind of regret-designed lottery.”
One indisputable finding from the study is the importance of mass communication, Milkman said. Spreading the news about the sweepstakes boosted interest in a very measurable way. Although every adult was eligible and registration wasn’t necessary, residents were told they could go to a website to verify that their names were on the sweepstakes list. In the few first weeks of radio, television, newspaper, and online coverage, more than 1 in 20 Philadelphians visited the site to confirm their inclusion.
“You do need to get the word out,” she said. “To get that kind of visibility and that kind of free marketing probably does require something like a city-wide opportunity.”
