2
The Adoption, Use and Impacts of Performance Measures in Medium-Size Cities: Progress Toward Performance Management
David H. Folz, Ph.D.
Professor
The University of Tennessee
Department of Political Science
1001 McClung Tower
Knoxville, TN 37996-0410
Ms. Reem Abdelrazek, MPA
Research Associate
Tennessee Advisory Commission
on Intergovernmental Relations (TACIR)
Yeonsoo Chung, Ph.D.
Managing Director
North American Operations
Knowledge Source, Inc.
2
Adoption, Use and Impacts of Performance Measures in Medium-Size Cities
Abstract
Based on a national mail survey of chief executives in mid-sized US cities (populations between 25,000 and 250,000), this study examines the patterns of adoption, use and impacts of performance measures for the purpose of advancing understanding of the challenges involved in moving from performance measurement to performance management. This study identifies the factors that distinguish cities that adopted and used performance measures and the results that chief executives expected to derive from the use of performance measures. What chief executives thought about the helpfulness of performance measures in making various types of decisions and why they thought their use of performance measures met, fell short, or exceeded their expectations are examined. The study finds that while most chief executives thought that performance measures met or exceeded their expectations, several factors helped to explain why the use performance measures fell short of leaders’ expectations. The single most important factor that helped to explain the gap between expectations and actual experience was the extent of “buy-in” of performance measurement by line managers and administrators. The level of workforce unionization and the extent of municipal experience with performance measurement also helped to explain whether or not performance management was perceived to be successful.
Keywords: performance measurement, municipal government, performance management
Performance measurement in the public sector has garnered a great deal of interest among elected and appointed city officials as well as public administration scholars since at least the early 1990s (Bouckaert 1992; Wechsler and Clary 2000). Scholars have described the usefulness of tracking performance (Hatry et al 1990; Wholey 1999) and highlighted the many obstacles and unintended consequences of implementing various kinds of performance measures (Ammons 1992; Smith 1995). Several studies have examined the extent to which local officials adopt various performance measures and use them for different decision applications (de Lancer Julnes and Holzer 2001; Rivenbark and Kelly 2003; Poister and Streib, 1999, 2005; Melkers and Willoughby 2005; Yang and Holzer 2006).
While the performance measurement literature is replete with descriptions of its potential benefits, questions remain about whether the use of performance measures makes a difference in local governance and policy making, particularly in budgeting and resource allocation decisions (de Lancer Julnes and Holzer 2001; Hatry 2002; Ho 2005; Melkers and Willoughby 2005). As Ammons and Rivenbark (2008, 304) observe, “local governments’ progress in using performance measures to influence program decisions and service delivery has lagged behind their pace in collecting and reporting basic measures.” In their survey of 277 city and county administrators, Melkers and Willoughby (2005) found that almost half of the respondents in a mixed sample of governments reported wide use of performance measures. However, respondents were “much less enthusiastic about the effectiveness of using performance measures to influence budgeting processes and outcomes in particular” (Melkers and Willoughby 2005, 188). Likewise, de Lancer Julnes and Holzer (2001) found that only a subset of local governments that collect performance measures actually use them to improve program and service decisions.
While feedback about municipal service performance has been found to be helpful in informing the citizenry, Melkers and Willoughby (2005) found that few local officials expressed strong views about the value of citizen involvement in the performance measurement process. In his study of Midwestern mayors, Ho (2005) found that these officials considered performance measurement to be an important tool for helping to enhance public accountability, but only 17 percent actually involved their citizens in the process of measuring service performance. Ho (2005, 234) suggested that several political and organizational environment variables are helpful for understanding how chief executives perceive the usefulness, sustainability and success of performance measurement but concluded that “how performance measurement is integrated into decision making remains a black box” and merits further study. Ammons and Rivenbark (2008) examined fifteen North Carolina cities and concluded that the likelihood performance data will influence operations is enhanced by the adoption of efficiency measures, the willingness of officials to engage in performance benchmarking, and the incorporation of performance measures into key management systems.
This study examines why city officials adopted performance measures, how they report using them, what impacts and results municipal chief executives realized after adoption and whether these impacts fell short or met/exceeded their expectations. We explore what city leaders perceived to be the helpfulness of particular types of measures for specific types of decisions and what these chief executives thought about the overall impact performance information had on the quality of local decision making. We are particularly interested in identifying the most salient reasons for why CEOs may perceive a gap to exist between their expectations for and actual experiences with performance measures. In other words, an inquiry that investigates why CEOs consider performance management to be successful or not offers the prospect for identifying some of the factors within local control that may affect the extent to which local officials can realize the benefits expected from performance measurement as a key component of a performance management system.
Data and Methods
The data for this research were collected from a mail survey and from US census data. A mail survey was sent to 670 chief executives in US municipalities with populations 25,000 to 250,000. The mayors or city managers of these mid-sized cities comprise the survey target population. There are a total of 1,339 municipalities with populations in the 25,000 to 250,000 range. A stratified random sample of 670 cities (about 50% of the target population) and contact data for their chief executives was obtained from the International City/County Management Association (ICMA) in 2004. In addition to the names and addresses of chief executives, the ICMA data file included information on population, region, metropolitan status, and form of government for each city. The mail survey questions referenced in this study are included in the appendix.
Cities with populations between the 25,000 and 250,000 were chosen because of the availability of socioeconomic data for these cities, the desire to compare findings with previous research on this population stratum (Streib and Poister 2002, 1998; Poister and Streib 1999), and survey budget resource constraints. Cities with smaller populations were excluded because the level of adoption and use of performance measures in these smaller cities is low (Rivenbark and Kelly 2003).
The survey instrument was initially mailed in June 2004 followed by a subsequent second mailing to non-respondents approximately two weeks later in early July 2004. A total of 280 completed surveys were returned for a response rate of about 42 percent. Table 1 shows that the distribution of responses obtained are comparable to the distribution of cities in the sample. With respect to population class, the survey response percentages are within a few percentage points of target population. In terms of geographic region, municipalities from the northeast are somewhat under represented (-6.7%). With respect to form of government, cities with the mayor-council form of government are somewhat under-represented (-6.6%) while cities with a council-manager form of government are somewhat over-represented by 7.4%. In most respects, the profile of the cities that responded to the survey is comparable to that of all medium-sized US cities, allowing generalization to this population.
Table 1 goes here
Adoption and Use of Performance Measures
Poister and Streib (1999) reported that larger cities were much more likely to adopt performance measures. They found that only 30% of cities with populations between 25,000 and 50,000 use performance measures compared to more than half of all cities with populations between 100,000 and 250,000 (Poister and Streib 1999). Our survey results also indicate a gap in the adoption level among large and small cities but we also find that the adoption and use of performance measures has grown in popularity among cities in all population ranges since the Poister and Streib (1999) survey. As Table 2 indicates, more than two-thirds (68.0%) of cities have adopted and use performance measures. Among those cities with populations between 25,000 and 50,000, 59% use performance measures while 83.7% of cities with populations between 100,000 and 250,000 use them.
Table 2 goes here
That more than two-thirds of all medium-sized cities use performance measures supports the conclusion reached by Melkers and Willoughby (2005, 188) that these metrics are now a “fairly pervasive” feature of local governments. The 185 cities that reported both adoption and current use of various performance measures are the main focus of this paper.
Poister and Streib (1999) found that those cities with a council-manager form of government used performance measures more frequently than cities with a mayor-council form. Among the jurisdictions in our survey, council-manager cities also report using performance measures more frequently than mayor-council cities by a significant margin (72.6% v. 56.2%). When using the conceptual definitions advanced by Frederickson, Johnson, and Wood (2004) to classify municipal structures as either “political” (the traditional mayor-council form), “administrative” (the traditional council-manager form) or “adaptive” (a combination of features from the other two types), an interesting pattern emerges.
Table 3 shows the use performance measures across these three structural types. This distribution indicates that cities with either an adapted or an administrative form are much more likely than political cities to use performance measures. In fact, about 70% of those cities
Table 3 goes here
served by a professional chief executive or administrative officer use performance measures compared to just 50% of those led by an elected CEO. This difference is not only statistically significant but substantively important because it suggests that cities led by professionally trained managers and administrators are much more likely to employ performance measures, regardless of whether the city has a mayor-council form or not. Following Keene, O’Neil, Portillo and Svara (2007), this finding underscores another way that professional managers add value to the communities they serve.
How prevalent are particular types of performance measures among the 185 mid-sized cities that report their adoption and use? Previous research by Poister and Streib (1999) found that efficiency measures were the least frequently used type of measure and that workload or output measures were the most frequently used. Ammons and Rivenbark (2008) also report that many local governments measure performance but only with less sophisticated workload or output measures that provide little in the way of diagnostic feedback compared to higher order efficiency, effectiveness and quality of service measures. The findings in Table 4 confirm that the largest proportion of cities report using workload or output measures, but a considerably larger proportion of cities (about half) report using more sophisticated performance measures that include indicators of citizen satisfaction with services, service quality, outcome effectiveness, and efficiency. This pattern of use suggests an increasing level of sophistication in the type of performance information being collected by mid-sized cities.
Table 4 goes here
Chief executives were asked to identify various features related to the organizational culture of their cities. We examined several of these factors to determine whether any might be associated with the use of particular types of performance measures, especially in light of the previous findings by de Lancer Julnes and Holzer (2001) that variables such as management attitudes and risk taking tolerance positively influenced the actual use of performance information. Table 5 indicates that in those cities where the chief executive agreed that the managers in their jurisdiction viewed performance measurement as an important basis for making decisions, the information from outcome, efficiency, and service quality measures was much more likely to be used. The strongest single bivariate association (as measured by gamma, a frequently calculated PRE-based measure of association for ordinal data) occurred between the use of outcome measures and having an organizational culture in which managers viewed performance measurement as an important basis for making decisions. The use of workload measures was strongly correlated with a greater perceived receptivity among non-management employees to organizational change.
While the correlations are in the positive direction predicted by de Lancer Julnes and Holzer (2001), we found no statistically significant connection between the use of the various types of performance information and management’s willingness to implement organizational change, the extent of support by elected officials or the presence of a system that encourages risk-taking. These findings suggest that the actual use of more types of performance measures occurs when managers understand the value of performance data for making decisions and when non-management employees are receptive rather than fearful about possible organizational change in the wake of the use of workload data in making management decisions.
Table 5 goes here
Reasons for Adoption and Expected Results
What reasons do chief executives identify for adopting performance measures? In other words, what motivated local officials to invest the resources to measure and track the various aspects of service performance? Figure 1 shows that a fairly strong consensus exists among chief executives. By a large majorities, chief executives thought that adoption of performance measures would improve management decisions (81.9%), support budget recommendations and decisions (71.9%) and respond to citizen demands for greater accountability (68.6%). These findings are consistent with previous research insofar as the desire to make better management decisions also was the principal motivator reported by Poister and Streib (1999). Significant proportions of chief executives in that study also reported that performance measures were adopted in response to citizen demands for greater accountability and pressure from council members.
As Figure 1 shows, the largest proportion of chief executives in our survey believed that performance measures were adopted to help improve management decisions and support budget recommendations and decisions. This suggests that local officials recognized the potential for integrating performance measures in decisions about management and budgeting decisions to an extent not reported previously.