Online games and game-like activities have been cropping up more frequently in business, education, fitness and other areas. Big companies such as Starbucks and Unilever have used games for training and customer loyalty programs. Over 70% of Forbes Global 2000 companies surveyed in 2013 said they planned to use gamification for marketing and customer retention. The U.S. Army created a “virtual army experience” for recruiting at shopping malls and public events. And this month The New Yorker reported that a rehabilitative video game called “Bandit’s Shark Showdown,” designed by Johns Hopkins scientists, offered a new approach to helping stroke patients.

It’s tempting to assume that gamification works, given the enormous long-running popularity of video games and gaming systems ever since Atari’s Pong was released in 1972. Who doesn’t love games? But the world’s experts on gamification will tell you that everything about this fledgling field — even the very definition (loosely, the application of game elements to non-game situations) — is still up for grabs. And what succeeds at one company won’t necessarily succeed at another.

Some of these experts recently gathered at a Wharton conference entitled, “Gameful Approaches to Motivation and Engagement.” In a panel titled, “The State of Research,” they discussed what we know and don’t know about gamification, how it works, and what kind of effect it really has on job performance.

“If you listen to the popular magazine conversation, there was this arc of hype and then the classic disillusionment,” said Wharton legal studies and business ethics professor Kevin Werbach, the conference’s organizer. “But I kept having experiences over the last year or so where I was talking to people in companies — including very senior executives of very large companies that you wouldn’t think would be interested — saying, “We get it; you don’t have to convince me there’s a real opportunity here.”

The conversation would then turn to what the data and research was showing, and that is where Werbach found it difficult to reply. “Well, I’ve got a little bit of this and little bit of that and I know a lot of people, but there’s not any one place where it exists.”

Jonna Koivisto, a researcher at the University of Tampere Game Research Lab in Finland, noted that one of her recent projects has been an attempt to synthesize the literature on gamification. She said that while she has identified over 800 papers on gamification, only about 300 are empirical studies. Furthermore, she noted that for papers with “some actual concrete evidence, like ‘this study has found an effect on this or that,’ the number is very, very few. It’s sad to say, but … we can’t very confidently say much of anything about what works and what doesn’t.” Richard Landers, a professor of industrial and organizational psychology at Old Dominion University, echoed her statement: “What are the research gaps that remain? I would say, basically all of them.”

“If you listen to the popular magazine conversation [about gamification], there was this arc of hype and then the classic disillusionment.”–Kevin Werbach

Affecting Work Attitudes

One researcher trying to expand the available data on gamification is Wharton management professor Ethan Mollick. Mollick described how along with Wharton colleague Nancy Rothbard, an organizational behavior expert, he tested a professionally designed online sales game called “Slam-Dunk.” In this basketball-themed game — administered among several hundred salespeople at a fast-growing e-commerce company — large screens and leaderboards vividly illustrated sales activities such as getting leads, making cold calls and closing deals as layups, jump shots and baskets.

Mollick studied “consent” among the players, which denoted the extent to which employees embraced the game. This was measured by the survey questions, “I followed the game closely,” “I understood the rules of the game,” and “The game was fair.” He found that those who consented had strong increases in positive feelings and a positive attitude toward their company. But for those who did not consent, the game actually caused negative feelings about their job and even some decrease in performance.

In Mollick’s view, the experiment is significant in two ways. “We have a big controlled study that shows gamification can have a large impact, but it also says that [a lack] of consent can actually lower performance attitude.” Without consent — people’s sense of willingness and involvement in the game — a company is left trying to create “mandatory fun.” (“Mandatory Fun” is in fact the title of Mollick and Rothbard’s paper.)

He also noted the difficulty of measuring the actual effects of workplace games on performance (outside of established research suggesting that employees with positive attitudes are more productive at work). Although his study showed a slight drop in performance among the non-consenters, there was no corresponding bump up among the consenters. “Did these people make more sales calls, [for example]? We had trouble finding that out. That might be because these people in these environments are very highly motivated anyway. But a lot of conditional stuff happens in games that makes it tough.”

Mollick discussed a separate ongoing gamification study that is yielding intriguing findings about the role of identity in games. In an entrepreneurship “alternate reality world” that he created for his MBA students, the class was divided into groups that had to use a pseudonym, be anonymous or use their real name. Although the students had predicted “that the real names would be much stronger, because they would be held accountable for the work that they’re doing,” the opposite is turning out to be true, according to Mollick. The group with pseudonyms is strongly outperforming the others.

“What are the research gaps that remain? I would say, basically all of them.”–Richard Landers

Koivisto noted that her team, like Mollick’s, has done research in an e-commerce context. In a long-term two-year study, the University of Tampere team found that patches (reward badges that appear when a task is completed) increased work activity at the firm. “In that case, we have a very clear indication that gamification was successful.”

Experiments with leaderboards are a focus of Landers’s gamification work. He said he has found “strong support for leaderboards if designed appropriately in the learning context.” In one study, employees in e-learning programs used leaderboards that rewarded not learning specifically, but putting in time and effort. He noted that the use of the leaderboard caused people to return to the activity and spend more time with it. Spending more time, in turn, led to better learning performance.

Leaderboards can also help with employee goal-setting, according to Landers. Many people do not respond well to being told by their managers what goals they need to achieve. But Landers found that “by giving someone a leaderboard, they actually strive for the top of it even if you don’t tell them to.” People will naturally try to compete with each other.

In the view of Lennart Nacke, director of the HCI Games Group at the University of Waterloo, leaderboards are one of the more promising study areas within gamification. The best uses of the leaderboard, he observed, are in scenarios where “we create the value of the leaderboard ourselves,” which he said Landers’s study reflects.

Capturing the Magic

Mollick referred to a statement made at last year’s conference that games were a “magical, ephemeral thing” and that trying to turn them into quantifiable business tools was essentially a losing battle. He asked the participants, “As we go from, ‘I spend 80 hours playing the game I most love’… to ‘Do we combine leaderboards with points and in what way’… are we moving away from what makes games special or amazing when we try to break it down, study and apply it?”

“Are we moving away from what makes games special or amazing when we try to break it down, study and apply it?”–Ethan Mollick

Sebastian Deterding, a professor in Northeastern University’s Game Design Program, responded that one relevant issue was the inevitable difference in production values between commercial games and gamification. “The question is, can we reach the same point as the entertainment in [a game that is considered] ‘Triple A’ — 300 people working at this for three years for $18 million? … Yes, learnings can be had, but whether we can reach the exact same level, I would be doubtful.”

He also pointed out the issue of choice, which harked back to Mollick’s “Mandatory Fun” study. With a game that is played for leisure and entertainment, he said, “I voluntarily choose to play this game…. It suits my personal predispositions, interests, and my energy level at that very moment. And if I don’t like it any more, I switch it off. In that situation, very likely I quite enjoy myself.” Deterding contrasted this with a workplace game that one is “forcibly put in front of,” saying it would be difficult for it to measure up.

A game designer as well as an academic, Deterding advised those working on gamification to not limit themselves to “a pre-defined set of game elements.” He recounted an example, presented by a consultant at the Bizplay conference in Germany earlier this year, of a German comic book publishing house that wanted to promote its new offerings through gamification. The unexpected end product was a type of controlled audience participation in which readers actually helped determine how various stories ended.

“Some might say that is a game element; some might say isn’t a game element,” said Deterding. “Start with understanding the problem and framing the problem, and that very often leads you down the road where … whatever you produce doesn’t necessarily encompass any of the things currently considered to be a game or gamification, but is a good solution.”