Quick Search
    

Training

Featured Text

 

ROLE 2: Assess Performance Information:

Practice 2a.Test relevance or reliability: Test or certify performance measurement relevance, reliability, or both.

Oregon Internal Auditors

In 2008 the State of Oregon conducted the first statewide series of performance measure “Integrity Audits,” an initiative designed to assure the reliability of “Key Performance Measures” (KPMs). The first year, integrity audits were launched in 16 state agencies in response to a state administrative rule that agency internal auditors review data integrity of performance measures when completing their risk assessments. The integrity audits are intended to verify the data behind the KPMs, which are used in the budgeting process and reported to policy makers and citizens. This is an important step that helps ensure that decision makers have confidence in the data used in performance management.

Oregon has had a mandated performance measurement system for over 15 years. Even years before the system was legislated, Oregon was a leader in capturing and reporting societal outcome data as “Oregon Benchmarks.” State performance measurement guidelines note that KPMs are intended to report progress toward achieving agency goals, mission, and high-level outcomes. The KPMs represent important results that, for many agencies, contribute to achieving the Oregon Benchmarks.  They also provide decision makers high-level results data for agencies that are important to the functioning of state government but do not have a major role in contributing to societal benchmarks, or whose role may be indirect.

The integrity audit objectives were, in summary: determine the accuracy and reliability of the data reported by the agencies in their Annual Performance Progress Report for the year stated, and to help ensure that the data reported were documented, repeatable, and consistent.  The audits were intended to examine 10 to 15 percent of the KPMs of each agency reviewed.

Agency internal auditors voluntarily collaborated with agency performance measure coordinators, and Legislative Fiscal Office staff in a multi-agency, multi-branch initiative to conduct the initial round of integrity audits. A committee with a mix of such staff established a “template” for an audit program (see “Tool” below), which staff reviewing each agency could voluntarily follow or modify.  Highlights of the template’s suggested  fieldwork steps include:

  • Obtain and review documentation of key performance measures to determine completeness.
  • Interview the Performance Measure Coordinator, data owners, and other pertinent personnel as necessary to gain an understanding of performance measurement processes.
  • Determine which key performance measures to test. (Criteria included risk assessment results, measures of major business lines, measures of significant customer interest.)
  • Determine if adequate controls are in place for each measure tested.
  • Determine if the performance data can be recreated.
  • Test the source documentation for accuracy.
  • Review methodology behind measure calculations.

Fourteen agency KPM integrity audits were completed in time for a January 2009 compilation report by the Chief Audit Executive (Internal Audit Division) of the Department of Administrative Services. The compilation of agency findings noted that of 269 KPMs in these agencies, 55 (20 percent) were reviewed.  Five categories were developed for classifying findings on each measure:

  • Verified (36 measures were found to be in this category)
  • Verified with qualifications (9 measures)
  • Factors prevented verification (9 measures)
  • Inaccurate (1 measure)
  • Not applicable, for example, a new measure with no data (0 measures)

Audits with findings of measures in the “qualified,” “prevented,” or “inaccurate” categories included specific recommendations to each agency to improve performance data.  The factors that led to these findings were summarized in the compilation report into four main groupings:

  • Inadequate documentation
  • Inadequate controls in the performance measurement reporting process
  • Lack of validation processes for third-party data
  • Inappropriate data elements and assumptions used in calculations.

The compilation report also stated that in this round of integrity audits, a review of the “adequacy of performance measures” to determine if agencies were measuring the right things was not an audit objective. The report noted that internal auditors, legislative fiscal staff, and budget and management staff could consider reviewing the adequacy—or relevance—of performance measures in the future.

Report: 

Key Performance Measure Data Integrity Review: Compilation of Agency Findings: Jan. 2009 (PDF)

 

Tool:

Performance Data Integrity Audit Template (DOC)