The amount of data that organizations now have has brought massive opportunities to boost innovation and performance. But there are downsides. Beyond the risk of big security breaches that garner so much attention, there also is a highly costly devaluing of qualitative judgments by people, notes Jerry Z. Muller, in this opinion piece. “Becoming aware of the characteristic pitfalls of metric fixation is the first step in helping your organization to recover, and to restore judgment to its proper place.” Muller has adapted this article from his book, The Tyranny of Metrics, just published by Princeton University Press. (Listen to an interview with Muller from the Knowledge at Wharton show on SiriusXM channel 111 using the player at the top of this page.)
Human judgment — based on talent and experience — has become unfashionable. Measurement is in.
Scholars in the field of behavioral psychology delight in demonstrating that biases lead us to misestimate numerical values and probabilities, casting doubt on judgment. Those who belong to groups that may have suffered from social bias identify judgment with prejudice, and prejudice with unwarranted discrimination. To them, “objective” metrics comprised of “hard” numbers may seem like an antidote.
Measured performance is also proposed as a solution to the suspicion that public-sector institutions like schools, police departments or government agencies were being run more for the benefit of their employees than their clients, in other words, as a solution to the problem of time-serving. From yet another direction, management gurus tout the unbeatable effectiveness of algorithms based on big data. Consultants are quick to recommend that the solution to organizational deficiencies demand more data.
Add to that the lure of information technology. The growing opportunities to collect data, and the declining cost of doing so, contribute to the belief that data is the answer, for which organizations have to come up with questions. There is an examined faith that amassing metric data and sharing it widely within an organization will result in improvements of some sort. So who needs judgment based upon experience?
Pitfalls of ‘Managerialism’
The focus on measured “performance outcomes” is related to a broad way of thinking about organizational life that emerged from business schools and business gurus. It substitutes managerialism for management. Managerialism is the belief that all organizations are fundamentally the same, and can be managed by the same tools. For devotees of managerialism, management requires not experience or judgment but technique.
Advocates of managerialism see no problem in top executives moving from a corporation in one field to a corporation in another field; or the head of a corporation becoming the president of a university or the head of a department of the federal government. Everywhere, the theory goes, what you really need is mastery of the same metric tools. (And if your new organization doesn’t yet have them, outside consultants will deliver them.) That managerial mentality has grown in influence in recent decades – in the federal and state governments, under both Democratic and Republican administrations, in business corporations, and in colleges, universities and even philanthropies
All of this has contributed to a cultural pattern that I call “metric fixation.” It is embraced by the management of more and more companies, government agencies and philanthropic organizations.
Key components of metric fixation are the belief that it is possible and desirable to replace judgment, acquired by personal experience and talent, with numerical indicators of comparative performance based upon standardized data (metrics); and that the best way to motivate people within these organizations is by attaching rewards and penalties to their measured performance. Those rewards can be monetary, in the form of pay for performance. Or they can be reputational, in the form of college rankings, hospital ratings, surgical report cards, etc.
“For devotees of managerialism, management requires not experience or judgment but technique.”
Reward and Punish
Data based upon standardized measurement can indeed be valuable. But it depends on how that data is used. There is a big difference between, on the one hand, having practitioners use data to try to improve their performance; and on the other hand, having non-practitioners use data to reward and punish.
In the first case – say a police department using crime reports to decide where to deploy squad cars – measurement serves the purpose of diagnosis. But when measurement is used by management to reward and punish, the stakes are very different. What ensues is gaming: maximizing the metrics in ways that are at odds with the larger purpose of the organization. If the rate of major crimes in a district becomes the metric according to which officers are promoted, some officers will respond by simply not recording crimes or downgrading them from major offences to misdemeanors.
Or take the case of surgeons. By comparing their metrics of success and failure, they can learn from one another. But when metrics of success and failure are made public – and hence affect their reputation and income – some respond by refusing to operate on patients with more complex problems, whose surgical outcomes are more likely to be negative. They thus improve their metric scores. Who suffers? The patients who don’t get operated upon. When reward is tied to measured performance, metric fixation invites this sort of gaming.
In addition to gaming the metrics, there are other unintended negative consequences of tying rewards to metric targets. What gets measured and rewarded is more likely to get done – but at the expense of other essential functions that get overlooked. Thus, teachers “teach to the test” – shortchanging elements of English and math that are not tested, not to speak of other subjects, from music to civics. Doctors “treat to the test,” focusing for signs of the diseases they are incentivized to examine, while overlooking symptoms of diseases not on the list. CEOs trying to maximize stock prices boost quarterly profits by cutting back spending on R&D and on the development of their work force.
“What gets measured and rewarded is more likely to get done – but at the expense of other essential functions that get overlooked.”
In most organizations, there are essential qualities that are invaluable yet not subject to standardized measurement – qualities such as mentoring and co-operating with fellow employees. They can be evaluated but that requires judgment, not measurement. In general a focus on measured targets discourages innovation – if the goals are specified in advance, there is little room for initiative, not to speak of risk-taking.
Last but not least, gathering the data takes time and energy away from the activities that are supposed to get measured – just ask doctors and nurses, or the chairs of academic departments. For all these reasons and more, metric fixation often leads to demoralization.
All of that is not intended to claim that measurement is useless or intrinsically pernicious. But the effective use of metrics requires judgment, and judgment is a matter of discrimination: of knowing which cases are harder, more challenging or more important than can be captured by a standardized measurement. And that discrimination is only possible when the judgment is linked to a sense of the overall purposes and peculiarities of one’s office, department, clinic or organization.
Measurement is not an alternative to judgment: measurement demands judgment. Judgment about whether to measure, what to measure, how to evaluate the significance of what’s been measured, whether rewards and penalties will be attached to the results; and to whom to make the measurements available.
Becoming aware of the characteristic pitfalls of metric fixation is the first step in helping your organization to recover, and to restore judgment to its proper place.
