ROLE 2: Assess Performance Information:
Practice 2a.Test relevance or reliability: Test or certify performance measurement relevance, reliability, or both.
Maricopa County, Arizona, Internal Audit [Profile]
In Fiscal Year 2002, the Maricopa County Auditor initiated a performance measurement certification process that builds upon the one used by the State Auditor of Texas to test and certify the reliability of data. The first year of the process, Internal Audit also formally tested and reported on the relevance of performance measures. In a step at the front-end of the process, an auditor reviewed relevance by assessing the alignment of a department’s performance measures with the county's strategic plan. Internal Audit found relevance testing more subjective and time consuming than desired for its formal audit work. So, since Fiscal 2002, Internal Audit may informally suggest to county leadership that the relevance of their measures needs to be improved, but currently, they only report on the accuracy of the measures. In 2003, Internal Audit developed a self-assessment checklist (see “tools” below) to encourage departments to perform a self-review of strategic relevance before Internal Audit certifies reliability, by assessing alignment among department goals, key progress measures, and individual performance plans.
Measures are chosen for certification testing based upon a risk assessment that considers factors including organizational changes, size and volatility of department budgets, unexpected reported performance levels, concerns relayed by county leadership, previous problems, and date of the last review. Audit staff review the calculation of reported measures, evaluate data collection procedures, and test a sample of source documents and reported results. They use a certification checklist (see “tools” below) similar to that of the Texas State Auditor to serve as a basis for reporting results. Upon completing the evaluations, they assign a certification rating and report their conclusions.
One senior auditor was dedicated to certification during Fiscal 2002. The Fiscal 2002 certification audit report covered 34 measures from 7 departments. In Fiscal 2003, Internal Audit decided not to have one person solely devoted to certifying measures. Now, whenever Internal Audit conducts a performance audit, one auditor is assigned to certify the measures of the program being audited. It takes the Internal Audit staff 25-30 hours to certify a measure. They spend about 1,000 staff-hours annually on this practice.
The certification reviews are conducted in accordance with the County's Managing for Results policy passed in Fiscal 2001 by the County Board of Supervisors, and an audit plan approved by the County Board. The policy states that Internal Audit will review and report on strategic plans and performance measures. Performance measurement certification has the full support of both the County Board of Supervisors (to whom the County Auditor and Internal Audit report) and County management. The County Administrative Officer, who spearheaded the County’s managing for results initiative, recognized from the outset the importance of accurate performance measures and has championed Internal Audit's role in making the program credible.
Tools:
The Performance Measurement Certification (PMC) Program Information Package (PDF) on Internal Audit’s website contains several tools, including:
- A flowchart of the PMC Process
- Certification categories and their definitions
- The self-assessment checklist for departments to review strategic alignment (relevance) of measures
- The PMC fieldwork checklist used by Internal Audit staff for testing and certifying reliability of measures.
Reports:
Performance Measurement Certification (PMC) July 2006 (PDF). This report, as well as previous years’ PMC reports dating back to 2002 are available at the Performance Measurement Auditing page of Internal Audit’s website. |
