Quick Search

Training

Featured Text

 

Performance Management Systems Offer High-value Audit Opportunities

A version of this article originally appeared in GAP News (1st Quarter, 2008, March),The Institute of Internal Auditors’ e-newsletter for government auditors (http://www.theiia.org/gap). Reprinted with permission.

Auditors can add more value to government performance by finding ways to improve the systems government organizations use to measure, manage, and improve results.

PAUL EPSTEIN, Principal, EPSTEIN & FASS ASSOCIATES

Download the complete GAP News article (PDF)

Managing for results has been an accepted idea in government since the 1990s. According to the Governmental Accounting Standards Board, managing for results is a comprehen­sive, systematic process for public organizations to use to accomplish their missions, goals, and objectives.

To transform managing for results from an idea into reality, organizations must establish an effective performance management system that feeds back measured results into decisions. Government organizations across North America have attempted to implement many different forms of performance management systems, and their level of implementation has often been uneven. This variability provides an enormous opportunity for auditors to add value to government performance by finding ways to improve these systems. These audits can provide an effective way to use auditors' professional skills because, rather than produce findings on one program or operation, they offer ways to improve underlying management systems that a government organization can apply to improve decision making, performance, and accountability of all its programs and services.

 

Auditing PM Systems Is One of Five High-Value Auditor Roles

Auditing performance management systems is a major practice (“practice 1b”) in the first of five
high-value auditor roles in the Framework of Auditor Roles and Practices in Performance Measurement. This is the second in a series of articles on auditors using roles and practices of the framework to add value to government performance.

 

Performance Management Involves Using Performance Information

Performance management can mean many things and can be implemented at many levels. Broadly defined, it means using performance information to help make better decisions. This can range from operational decisions of a single program or service, to business or strategy decisions of a govern­ment agency, to budget, policy, or long-term planning decisions of an entire government. Some organizations aim for a line of sight of consistent performance logic from top-level policy made by elected officials to evaluation and compensation of managers and employees as part of agency hu­man resource management. Performance management becomes a system when it is used on a regu­lar basis as part of a continually repeated cycle of performance monitoring, analysis, and improve­ment, in which measured results are fed back into decision making to improve future performance.

 

A Broad Range of Useful Audit Practices

Because performance management is such a broad concept, there is a broad range of approaches auditors can use to find opportunities for improvement. The following paragraphs offer a sample of the possibilities and some real-life examples.

Auditing Performance Management as a Part of Performance Audits
A cost-effective approach is simply to include the use of performance information as part of thescope of performance audits of programs and services. As long as auditors are going to interview staff and collect and analyze data to evaluate program performance, it takes relatively little additional time to ask how performance information is used, review available documentation concerning its use, and assess whether performance information can be used more effectively. These low-cost extra steps can provide great public value if they lead to more careful or more systematic use of performance information to get better results. This has been a regular practice of the Portland City Auditor in Oregon, the Austin City Auditor in Texas, and the Florida Office of Program Policy Analysis and Government Accountability (OPPAGA).

Audits Focusing on Performance Management Systems of a Single Program
The auditor for the metro regional government in the Portland, Ore., region has used this approach. The audit of metro government's Transit Oriented Development [PDF] (TOD) program demonstrated the value of auditing the performance management of a relatively new program still in the process of developing its performance management approach, especially a complex program for which defining performance can be problematic. The audit helped TOD focus on a simpler perform­ance measurement process and vital few measures, and encouraged the regular practice of collecting baseline and later comparative data related to expected benefits for new development projects.

Audits of Underlying Systems that Support Performance Management
When a government follows a regular cycle of performance management, essential information flows through several different systems, from, for example, agency systems that compile and analyze cost and performance data, to central systems for budget management and internal performance review, to public performance reporting systems and websites.  Auditors can play an important role assuring the integrity of these systems to increase decision makers’ confidence in using the data to improve performance.  The Texas State Auditor has conducted high-level reviews of information technology of state entities using complex automated systems to process performance measurement data.  Another example is in the 2006 performance measures review by the Virginia Auditor of Public Accounts, who recommended improving user access controls to ensure the integrity of the data reported in the state’s new performance management system and public performance reporting website.

Agency-by-Agency Performance Management Audits
This type of audit can be particularly useful when new government-wide performance management systems are being implemented. Auditors often find that agencies are simply submitting data to meet system requirements.  Audit recommendations can show agencies how to mindfully define performance in meaningful ways and use the data to make better decisions. The new systems will then add more public value in the form of better program and service results, and will do so years earlier than if these audits were not conducted. For a large government, the roll-out of these systems can take several years and the learning period for agencies even longer, giving an audit office the opportunity to keep providing high value through such audits for quite some time. That was OPPAGA's experience from 1995–2002, following passage of Florida's Government Performance and Accountability Act. OPPAGA's performance accountability ratings and other performance management system reviews led to better agency implementation of the state's Performance Based Program Budgeting System (PB2), and better statewide use of the system. From 1997–2001, OPPAGA regularly issued summary progress reports on PB2 to the state legislature with guidance on using and improving the system.

Government-wide Studies of Performance Management
For the last several years, the Virginia Auditor of Public Accounts has been reviewing aspects of statewide performance management systems and practices as part of annual reviews of performance measures to encourage better implementation of changes in statewide performance management systems. The Austin City Auditor conducted four entity-wide reviews of performance management systems from 1994 to 2002, spurring the city council to encourage performance management improvement, and city management to act on system changes. In 2002, the Portland City Auditor conducted an entity-wide study of performance management from the mayor and city council down. The audit report included a proposal to implement an integrated managing for results process in Portland, which led to a city council ordinance calling for an implementation plan for such a system.

 

Using Performance Models and Best Practices as Audit Criteria

Performance management systems can take a variety of forms, and such systems have been evolving in different ways in governments across North America. That creates a challenge for auditors to find a basis for assessing existing performance management. To meet this challenge, auditors have reviewed best practices used in other organizations and developed performance models or frameworks suitable for the entities they audit, which have varied from graphic models, to decision flowcharts, to lists of relevant best practices. These models then become the audit criteria that existing systems and practices are compared against.

The professional context section of this website provides two graphic managing for results and organizational learning models auditors may find useful, including a government accountability system model used by the Austin City Auditor. Several other forms of performance management audit criteria are documented at this website. For example:

  • In 2001-02, OPPAGA and the Florida Auditor General developed Best Financial Management Practices for Florida School Districts, which were adopted by the state's commissioner of education as standards for "Best Financial Management Practices Reviews" conducted jointly by OPPAGA and the auditor general. Of ten categories covered by the standards, three — cost control systems, administrative and instructional technology, and especially performance accountability systems —include best practices directly related to performance management.
  • The Michigan auditor general has used a continuous quality improvement framework to help determine whether an agency has the measures and systems in place to adequately evaluate its own program effectiveness. The framework is documented as a flowchart with steps for auditors to follow, and key questions that determine the direction of an auditor's study (PDF).
  • In Alberta, Canada, the office of the auditor general (OAG) developed criteria for its managing for results audits, focusing on business planning, performance measurement, and human resource systems. You can download the OAG's detailed audit criteria for all three types of systems from the Auditor Toolkit of this website and adapt these tools for systems audits in your entity.


Performance Mangement Audits in Relation to Other Audit Practices

The value of auditing performance management systems can often be increased when this practice is used in relation to other roles and practices from the Framework of Auditor Roles and Practices With Respect to Performance Measurement. As previously noted, some auditors find it valuable to include limited scope reviews of performance management as part of program or agency performance audits (practice 1a of the framework). Also, it is common to include some degree of testing the relevance or reliability of performance information (practice 2a) in broader audits of performance management systems, as the Austin city auditor has done. Conversely, as the Texas state auditor and Virginia auditor of public accounts have done, it can be valuable and cost-effective to couple reviews of selected aspects of performance management with periodic efforts to test relevance and reliability of agency performance measures.

As performance management systems audits can lead to recommendations for deep systemic changes in the workings of government, an audit executive should be prepared to interact constructively with management and elected officials, as reflected in Roles 4 and 5 of the framework. For example, they may have to advocate to elected officials (practice 5a) to mandate or encourage management to make such changes, as the Portland city auditor did in 2002, and they may need to assist management (practice 4b) by providing guidance or advising in design and implementation of these systemic changes, as numerous auditors across North America have done.

 

Relevant Training and Online Resources Available

The Auditor Roles in Performance Measurement Project will have a CPE training course dedicated to auditing performance management systems available in the fall of 2008. Contact the project team to schedule this course for your organization, association, or chapter. Existing courses relevant to this practice include the New Service Model, which presents auditing performance management systems in the context of all auditor roles in the framework, and Assessing the Reliability and Relevance of Performance Information, which covers in depth a practice often included in performance management audits.  See the times and locations of scheduled courses, or contact the project team to schedule an on-site course for your group.

In addition, this website offers numerous resources to support all auditor roles in performance measurement and the full set of practices in the framework. These resources include eight examples of Practice 1b. Audit performance management systems from different local, state, and provincial audit organizations, most with links to these auditors' reports or tools.