Quick Search

Training

Featured Text

 

Performance Measurement Helps a New Audit Office
Find a High Leverage Way to Start its Work

By Michael Eglinski, City Auditor, City of Lawrence, Kansas
with Paul Epstein, Principal, Epstein & Fass Associates
Download a PDF file of this Article

“How do you get a new audit function off to a good start?” 
I was faced with that question when I was hired as the first City Auditor in Lawrence, Kansas, in February 2008.   It is a tough question to answer.  Part of my answer has been to use performance measurement in both selecting initial audit topics and thinking about how those audits can result in improvements.

 

MAIN CRITERIA FOR SELECTING INITIAL AUDITS

I report to Lawrence’s elected City Commission, equivalent to a city council.  The City Manager also reports to the commission. In my memo on “Potential Performance Audit Projects” to the commission, I noted general principles I used to identify and help select initial audit topics in Lawrence. They included:

  • Find topics important to residents
  • Find topics important to the organization
  • Look for audits that can provide information to help develop audit plans for future years
  • Pick audit projects with a well defined scope, so early audit work will be most manageable.


HOW PERFORMANCE MEASUREMENT AND MANAGEMENT BECAME A FACTOR

Performance measurement or management can be a factor in all those criteria. They were not the only factors, but they were important factors in Lawrence.  Here’s how:
 

  • Topics important to residents:
        • The City Commission and city management conduct period citizen surveys in Lawrence.  The survey results provide a convenient set of performance measures to help the auditor identify services or issues that are a high priority to residents, and where there are opportunities to improve citizen or customer satisfaction.
  • Topics important to the organization:
        • Lawrence has a pavement measurement system it uses to allocate millions of dollars of services. So, the reliability of this system is critically important if the City is to avoid risking a major miss-allocation of resources.
        • In Lawrence, I saw two opportunities to introduce new kinds of measurement and analysis that could provide valuable information to the city organization: Ethical climate survey questions and analysis of financial indicators.  The financial indicators are not typical performance measures, they are financial measures. But finding useful financial indicators for periodic analysis can become a valuable tool for future performance management systems.
  • Provide information for future years’ audit plans:
        • The pavement measures reliability audit, the ethical climate survey questions, and the financial indicators analysis are expected to provide information that will help identify audit topics in the future.
  • Audit projects with a well defined scope:
        • Developing ethical climate survey questions and conducting financial indicators analysis can readily be framed as projects with clear, limited scope.  The same can be said for a pavement measurement reliability audit, as audits of data reliability can usually be completed in less time needed that for a broader scoped performance audit


MEASUREMENT-RELATED QUESTIONS ANY AUDITOR CAN ASK IN SELECTING AUDITS

The initial audit selection experience in Lawrence suggests questions related to performance measurement that any auditor can ask in selecting audit topics for an audit plan, and not just for the first year of an audit shop.  For example, any government auditor can ask:

  • Are there performance data available on resident priorities and satisfaction to help identify topics important to residents?
  • Are there existing performance measurement systems that present a high risk if they are not relevant or reliable?
  • Are there new forms of measurement or measure analysis the auditor can introduce that can provide valuable information to the organization?
  • How can performance measurement-related audits help the auditor learn about future audit opportunities?


SELECTED PROJECTS IN THE FIRST YEAR PLAN OF THE CITY AUDITOR OF LAWRENCE, KANSAS

In developing my proposal for the first year of audits in Lawrence, I reviewed available documentation and consulted with management and elected officials to obtain information on likely audit topics.  After applying the above criteria, and also looking for audit projects that would help me build relationships with city departments, I proposed five first-year audit projects to the City Commission.  The commission accepted my proposed plan, which includes the three audit projects noted above, and explained further here.  The City Commission also added a 6th project – to look at payments for street lights – which was given a high priority in part because the adequacy of street lights was rated relatively low in the most recent citizen survey.

Auditing the Reliability of Pavement Condition Measures
In Lawrence’s resident survey, residents identified the condition of the city’s streets and infrastructure as the issue the city should most emphasize.  Also, the city government uses a system of pavement condition measures to allocate street maintenance resources, budgeted at $4.8 million in 2008, and also to monitor the results of maintenance.  So, this is a measurement system that can add value to a priority service if it is reliable, but can cause the city government to make expensive, sub-optimum resource allocation decisions if it is not reliable. I have no preconceived notion that the street condition measures are unreliable, but I may find control weaknesses that can compromise reliability and thus the validity of decisions about the use of millions of dollars in resources. And, even if I do not find control weaknesses, by talking to department staff about how they use these measures, and reviewing relevant information, I am likely to learn other things about the street system that are good candidates for future audit topics.

Ethical Climate Employee Survey Questions
Other government audit offices, such as the Austin, Texas, City Auditor’s Office, have found an ethical climate survey to be valuable in determining hidden risks, sometimes called “soft risk.” I have not proposed to do my own ethical climate survey. Instead, I proposed to add selected questions to an employee survey that management wants to conduct in late 2008. This would be consistent with the Austin experience, where the auditor’s office provided questions that were included in a survey by the Human Resources Department of the city government.  Austin found, for example, that departments with lower employee scores on ethical climate questions tended to be departments with the highest risk of losses due to, for example, employee accidents.  The survey will help the City Commission and management promote a strong ethical environment.  Also, if these questions are included in the 2008 employee survey, the results will provide a baseline for comparison in future years.

Financial Indicators Analysis
This audit project is intended to identify a small set of financial indicators based on  the City of Lawrence’s Comprehensive Annual Financial Report (CAFR). These would be measures that could help describe the city’s financial condition and identify warning trends. While these are financial indicators, not performance indicators, they can be an important influence on future performance management.   In particular, financial indicators can highlight the need for more and better measures of service levels and infrastructure condition.  An auditor’s independence in selecting financial condition indicators can be very useful for the city commission and citizens because they typically get financial information from a single source.

 

LOOKING AHEAD EVOLVING AUDITOR ROLES IN PERFORMANCE MEASUREMENT IN LAWRENCE

With the first year’s audit plan established and being implemented, where will the Lawrence, Kansas, City Auditor’s Office likely go in the future?

As an initial series of audits is completed, I will have new sources of formal findings and informal learnings to help determine good candidates for future audits. But I will continue to use Lawrence’s resident survey to help obtain audit topics, so residents’ main concerns and priorities will remain a key factor in developing future audit plans.  I also want to become more systemically involved in encouraging performance management.  Lawrence’s elected city commissioners expect good performance information from staff, creating an environment conducive to improving data.  As auditor, I can help improve the quality of performance measurement and effectiveness of performance management in a variety of ways.

Balance Reliability Audits with Encouraging More Data-driven Decision-making
One way to improve measurement quality is to conduct audits of the reliability of performance data used to make decisions, as in the pavement measures audit.  But, I need to balance audits of reliability with audits that look at areas where performance isn’t being measured.  I don’t want to send the wrong signal by “picking on” departments that measure performance, which could discourage departments from implementing measures.  So in addition to conducting reliability audits for measurement systems used for decisions involving large risks, I will also look for city decisions that can and should be driven by performance data but are not. This will identify opportunities to develop and use critical new performance measures and systems in Lawrence.

Encourage Developing Practical, Reliable Measures and Systems
In the future, I can encourage managers to develop, implement, and use reliable, practical measures.  As auditor, I can make recommendations about performance measurement, emphasize controls on data reliability, and encourage training and experimentation with measures.  I am especially interested in encouraging management to collect and report measures that are easy to collect and maintain while still being relevant to the goals and objectives of the organization and its programs. As an independent auditor, I cannot tell managers what to measure and I cannot define measures for them. Any decisions on what to measure must be up to management. But, through training, technical assistance, or audit recommendations, I can help them achieve a useful balance, to get information they can use for performance management, even if it is not the “ideal” performance information.

 

PERFORMANCE MEASUREMENT PRACTICES: A GOOD WAY TO START, A GOOD WAY TO CONTINUE

Overall, performance measurement and management provide good opportunities for government auditors to do value-added work for their entities.  I found such opportunities from the start of creating a new audit function in Lawrence, and I foresee more opportunities to help improve performance measurement and management in the future.