Independent Evaluation: Insights from Public Accounting
Abigail Brown, Ph.D.
Postdoctoral Lab Fellow
Harvard University, Edmond J. Safra Center for Ethics
Jacob Alex Klerman
Principal Associate
Abt Associates
Abstract
Maintaining the independence of contract evaluation of government programs presents significant contracting challenges. Sometimes the organization wants the evaluation to demonstrate its effectiveness; sometimes the evaluation is forced on the organization from outside (a foundation, OMB, Congress). In either case, the ideal outcome for the organization is both the "impression" of an independent evaluation and a glowing report (with the truth sometimes of secondary importance). In this, independent evaluation is like financial statement audits; firm management both wants a public accounting firm to attest to the fairness of its financial accounts and to be allowed to account for transactions as it sees fit. We review the contracting strategies developed to maintain independence in auditing (particularly those implemented in the recent Sarbanes Oxley Act of 2002) and explore insights for how evaluators might maintain and improve their independence.
I. Introduction
Contracting for the independent evaluation of government programs is awkward (Klerman, 2010).[1] Sometimes the agency wants the evaluation; sometimes the evaluation is forced on the agency from outside (a foundation, OMB, Congress). In either case, for the agency, the ideal outcome is both the impression of an independent evaluation and a glowing report (with the truth sometimes of secondary interest). In this, an independent evaluation is like a financial statement audit; corporate management both wants a credible public accounting firm to attest to the correctness of its financial accounts and to be allowed to proceed as it sees fit (e.g., smoothing earnings or even outright fraud; Fudenberg and Tirole, 1995; Healy and Palepu, 2001; Brown, 2007, 2011a). Due to the similar incentives of the purchasers of audits and the purchasers of evaluations, those tasked with conducting the audit/evaluation face similar pressures from their direct clients that can conflict with their mandate to provide an independent assessment of a financial statement/program.
While this tension between serving the direct client and the public interest exists in both auditing and evaluation, it has received far more public, scholarly, policy and legislative attention in auditing than it has in the evaluation field. Considerable theoretical work in accounting has explored strategies for contracting for auditing (e.g., Jensen and Meckling, 1976; DeAngelo, 1981; Dye, 1993; Coffee, 2006; Brown, 2011b) and significant public policy efforts, most recently the 2002 Sarbanes-Oxley Act, have attempted to facilitate more effective audits. This paper reviews the literature on and the well-documented experiences with the challenge of contracting for independent financial statement auditing. We then suggest ways to alleviate some of the challenges associated with contracting for independent evaluation.
The core argument of this paper is that there is a disjunction between those relying on the evaluation/audit and those overseeing it, and that several complementary strategies are available to address this disjunction. The issue is not simply that the two groups—those relying on the evaluation/audit and those overseeing it—are different; instead, the issue is that the two groups have fundamentally divergent goals for the evaluation/audit. The outsiders (i.e., those relying on the evaluation/audit) want an objective, rigorous characterization of the firm’s financial condition/the program’s impact. In contrast, the insiders (i.e., corporate management or program management) want the report that best advances its own interests and compensation.
In both auditing and evaluation, strategies for addressing this disjunction take a similar form: some combination of giving the auditor/evaluator sufficient incentive to ignore attempts by management to manipulate the process and increasing the oversight provided by representatives of those relying the results of the audit/evaluation. For auditing, the Sarbanes-Oxley Act of 2002 (SOX) required more of both of these strategies. We argue that similar reforms are promising for evaluation.
The paper makes its argument in six sections. The second section presents a review of the issues in contracting for independent evaluation (primarily in government). The third section provides a brief overview of the financial statement process and a motivation for its existence. The fourth section presents a review of the auditing experience, the core of the paper. The last two sections return to the issues related to contracting for evaluations, which are the motivating issues for the readers of Evaluation Review. In the fifth section, we make the analogy from auditing to evaluation. Then, rethinking Klerman’s (2010) catalog of possible strategies to maintain independence, we make recommend several changes. The paper concludes with a brief review of the argument and suggestions for both further research and implementation of reforms to contracting for evaluation.
Finally, some brief notes about language relating to our discussion of financial reporting. We use the term “audit” to refer to the mandatory external review of the annual statements of the financial condition of a publicly traded corporation. We use the term “auditors” to describe the people and firms who provide audits, and “accountants” to describe the people within the audited company who help the executives prepare the financial statements, though people who have earned the professional designation of a Certified Public Accountant (CPA) fill both roles.
We use the term “management” to refer to the people who run a for-profit company on a day-to-day basis. Thus in our language, an accounting firm audits a publicly held company. Management would usually include the Chief Executive Officer (CEO), the Chief Financial Officer (CFO), and other senior officers. We use the terms “owner” or “ownership” to refer to those who own stock in the company.
II. Contracting for Independent Evaluation
A cornerstone of management—in the private sector and in the public sector—is determining “what works.”[2] Programs that do not work should be modified or terminated; programs that work should be expanded. Programs should continuously explore their processes to identify changes that would improve effectiveness and efficiency. Once identified, such changes should be adopted (Besharov, 2009; Klerman, 2011). Indeed the U.S. Office of Management and Budget has emphasized the importance of “rigorous, independent program evaluation [in] help[ing] the Administration to determine how to spend taxpayer dollars efficiently.” (http://www.whitehouse.gov/omb/assets/memoranda_2010/m10-01.pdf).
Implementing this nearly unassailable management prescription requires knowing which programs actually work. Establishing which programs work is the task of the modern (rigorous impact) evaluation industry—in academia, in the non-profit sector, and in the public sector. On appropriate methods, there is a large literature (e.g., Orr, 1999; Shadish, Cook, and Campbell, 2001; Rossi, Lipsey, and Freeman, 2003). For our purposes, the main take away from that literature is that the type of rigorous impact evaluation required to establish whether a program works and which process changes would yield improvements is a major undertaking, requiring considerable expertise and judgment in both the methods being applied and the nature of the program being evaluated. In addition to the reality that most agencies lack the capacity to conduct their own evaluations, there are clear advantages to having evaluations done by an external entity. The organization running or overseeing the program is too invested in its creation, operation, and outcomes to provide an independent evaluation. These two factors—capacity and independence—usually result in government agencies purchasing the evaluation of their programs from external contractors.
When a separate firm conducts the evaluation, however, there arise two sets of contracting challenges. First, there are the standard contracting issues. A bidding process must be established leading to the selection of the “best” proposal. Then, that contract needs to be overseen to assure delivery of the rigorous evaluation specified in the best proposal.
Beyond these conventional contracting challenges, contracting for an independent evaluation has another dimension, and this dimension is the focus of the paper. Why would either party to the contract be interested in enforcing the requirement for it to be “independent”? Standard practice is that the evaluation is overseen by a government agency quite closely related to the program being evaluated—usually the research shop in the same agency. Often the agency would prefer a “positive report.” Such a positive report brings prestige to the agency and its leadership and continued (or additional) funding—preserving the jobs of those in the agency. The agency has ways to induce the evaluator to deliver the report that the agency wants: public release of the report, payment (base and bonus) for this contract, future evaluation business, further non-evaluation business can all be made contingent on the evaluation results. Since using these levers to exert undue influence is often disguised as legitimate contract oversight activity, implementing an evaluation process that strikes the right balance between oversight and independence is challenging.
The prevalence and nature of any manipulation or suppression of evaluations is unclear. Contractual language and the desire for future business make contractors reluctant to talk about attempts at such manipulation (and their success). The resulting difficulty in determining the full extent of the problem complicates the balancing act required, as the trade-offs between independence and oversight are hard to measure.
Though the extent of the problem is not fully known, from the instances of manipulation that have been made public, the existence of a problem is clear. Metcalf (2008) publicly describes two glaring examples. In the first example, made public via a FOIA request, USDA staff requested that the contractor delete one set of analyses—the analyses that that the contractor believed were more appropriate, but that reflected poorly on the program under evaluation (the National School Lunch Program). The contractor refused. In response and without the evaluator’s permission, USDA staff simply deleted the analysis in question and released the report under the authorship of the contractor staff. Then, in reaction to contractor staff noting this change in a public meeting, USDA completed official Past Performance Reports for the contractor such that the contractor would be much less likely to get future evaluation business.
In the second Metcalf (2008) example, the contractor delivered unfavorable results on the long-term impacts of a high profile DOL program (Job Corps). DOL suppressed the results for more than two years. Suppressing results is the ultimate form of manipulation. In this case, the results were only publicly released after the DOL staff mistakenly posted them on a DOL web site. Metcalf (2008), Reingold (2008), and GAO (2010) provide other examples. Given that such incidents are regular subjects of private (off the record) discussion within evaluation firms and in the evaluation community more broadly, it seems clear that such incidents are far from uncommon.
The balance of this paper argues that fundamentally this problem with contracting for independent evaluation arises from the disjunction between the true consumers of the resulting evaluations—the Executive, Congress, the public—and the bureaucracy charged with managing the contract for the evaluation—often an agency closely aligned with the bureaucracy running the program being evaluated. In this, even though they are discussed using different terms, contracting for independent evaluation is like contracting for a financial statement audit.
III. The Motivation for Financial Statement Audits
To understand the similar difficulties that face the audit industry, it is first important to understand the challenges facing corporations, shareholders and management. When the majority owner of a business is also its manager (e.g., a small business), the owner faces primarily a supervision problem: how to keep employees working productively in the interests of the company, rather than shirking, making self-serving decisions, or stealing from the owner.
Once ownership and management are separated—as they are in the modern corporation, including companies traded on the major stock exchanges—the issues are more complicated. Now owners, i.e. shareholders, need to worry that managers, e.g. the CEO, will manage the entire firm for their own private benefit, not for the benefit of owners. This is a central challenge of the modern corporate structure for which auditing is an intuitive and attractive solution.
Separation of ownership and control has enormous economic benefits. With the separation of ownership and control, people with good ideas or management skill, but not capital, can start and run a company. Conversely, people with capital have profitable and diversified ways to invest their capital. However, the incentive design and monitoring problem inherent in the separation of ownership and control (and the coincident separation of information) has been a persistent challenge.
The source of the challenge is the unsurprising temptation that comes from having control over other people’s resources combined with owners so distant from the operation that they cannot even verify management’s claims of performance. The primary avenue by which corporations generally deal with the problems such control could create is to structure executive compensation in such a way as to reduce the incentives for executives to engage in unproductive activities. However, managements’ compensation is, of necessity, tightly tied to success—as reported by management! A board of directors’ decision to renew the executive’s contract, the movement of stock prices, and the achievement of stated criteria for performance bonuses are all functions of corporate data; corporate data that are prepared by management. Certainly, there are other sources of information (news reports, independent analysts’ reports, direct observation of products and traffic at stores, etc.) but most of the information about most corporations comes from insiders whose current compensation and career prospects are directly tied to the content of that information.
That so much of a manager’s prospects is reliant on information that the manager is responsible for providing creates a strong temptation for the manager, if the truth is not flattering, to manipulate the information that is released. Even managers who do not wish to mislead the public can make self-serving judgments and estimates (accounting is a far less concrete and definite art than most outsiders recognize) without fully recognizing their own biases (Bazerman and Tenbrunsel, 2011; Moore et al., 2006; Bazerman et al., 1997; Kunda, 1990).
To reduce the ability of managers to report incomplete, misleading, or even fictional accounts of their company’s performance, modern corporate governance systems use several mechanisms. The primary explicit mechanism is the financial audit, the focus of this paper. Before the company’s financial reports are issued to shareholders and the market, an independent, external accounting firm is hired to audit the financial statements prepared by the issuer. The task of the auditors is to assure that items are accounted for properly (i.e., in accordance with Generally Accepted Accounting Principals (GAAP) and in a way that “fairly” reflects the underlying economics) and to do varying levels of testing to ensure that the items reported are real and complete. For example, they will confirm with banks the values in cash accounts, physically count a sample of inventory, test internal controls to ensure lower ranked employees have not been diverting assets, and perform other checks to provide reasonable assurance that no material fraud is occurring.[3]