A One-Day Performance Audit One-Day Performance Audit
3/20/2014
A One‐Day Performance Audit Michael Eglinski City Auditor Lawrence, KS 1
One‐Day Performance Audit • • • •
Is it possible? Can it meet Government Auditing Standards? Can it survive peer review? What are the lessons we can learn to improve audit productivity in general?
2
1
3/20/2014
My perspective • City Auditor for Lawrence, KS; one‐person shop; function created 2008 • Lawrence: 90,000 population (30,000 are students at KU or Haskell); $175 million budget; almost 800 employees • 1992‐2008 Kansas City, MO, a larger shop • Peer reviews in 8 other shops
3
4:30 on a Monday afternoon…
4
2
3/20/2014
On the last couple of pages of a 19 page phone bill – what is this?
5
“mytelebill llc”
6
3
3/20/2014
“Cramming”
7
By 5:00 p.m. Monday…. • Identified a potential problem • Identified criteria from FCC.gov
On the way home from work… • I could do an audit on cramming • And I could do it quickly
8
4
3/20/2014
8:00 Tuesday morning • Review and sign independence statement • Write objectives: – Has the city been billed for charges consistent with cramming? If so, what steps can be taken to strengthen controls to prevent cramming?
• Write work steps
9
The first hurdles • P&P and City Code require City Commission approval of performance audits • P&P and City Code require coordination with the City Manager on selecting audits
Key decision point • “It can’t be done” versus “what would it take to make this work?” 10
5
3/20/2014
Overcoming the hurdles: • 8:42 a.m. Contact City Commissioner who was acting as “liaison” to the audit function • 9:22 a.m. Get the go ahead from the liaison • Talk with the City Manager about the plan and idea – make sure it is ok. • Entrance conference with IT Department • 11:27 a.m. follow‐up with City Manager including scope and objectives • 1:28 p.m. inform all of the City Commissioners of the project 11
Fieldwork • Review criteria from FTC and FCC • Post to ALGA listserv • Review “judgmental sample” of phone bills to identify possible cramming charges • Follow‐up with staff who have crammed phone numbers – Aware of the service? – Aware of the charge? – Did you order the service? 12
6
3/20/2014
What I found • The city appears to have been charged for some services that weren’t ordered, agreed to or used • Staff…had no knowledge of the charges associated with the phone numbers identified as possible cramming charges.
13
Report writing • 2:08 p.m. started writing the report • Short – 2 pages including scope, objectives and method = ~750 words • Positive tone by focusing on the recommendation element of the finding (“The City should strengthen controls to prevent ‘cramming’”) • 3:03 p.m. shared draft with IT management 14
7
3/20/2014
Quality assurance • Referencing the entire report • Completing an audit standards checklist (based on the ALGA peer review engagement guide)
Conclusion of QA process • Followed Government Auditing Standards and I can add the relevant text to the report 15
Final hurdle • City Code requires management response and requires response “within 5 working days” • Report was done but rather than dump it on management at 5 p.m., provided the report for response the next morning • After 4 days, email response from IT management, summarized it and City Manager accepted it as his written response on day 5 16
8
3/20/2014
Is it possible? Can it meet Government Auditing Standards? • Almost…if only management would respond quicker! • Yes
But would a peer review team agree?
17
ALGA peer review a year later Review team noted two concerns: • Improve documentation of assessment of internal controls • Better report on scope of deficiencies in internal controls: “…are not clearly addressed in report. They are inferred in the audit recommendations. Note: an improvement in this area would really strengthen the audit discussion and recommendation.”
18
9
3/20/2014
A note I wrote at the end of the day • …I must have missed something in the standards. But, I completed the checklist and it seems like I’ve done everything ok. There might be an argument that I haven’t adequately described the scope of work on internal controls and deficiencies found...
19
Can it survive a peer review? • Yes, but room for improvement
20
10
3/20/2014
Can we learn lessons to improve audit productivity in general?
21
Flexibility and standard processes • Relatively flexible performance audit policies and procedures • Consider “pruning” your approach to focus on key processes for a given engagement • Balanced with a standard QA process • ALGA standards checklist
22
11
3/20/2014
Flexibility and standard process • Don’t be afraid to scope narrowly and stick to the scope – Petting zoos – Parade safety
• Think about the outcome and use professional judgment to determine the level of evidence
23
Evidence and criteria • The actual phone bills – solid documentary evidence. Approach wouldn’t work if it relied on complicated analysis or lots of interview evidence. • Simple, sound criteria from FTC and FCC (both provided consistent information). Approach wouldn’t work if criteria were inconsistent or not authoritative. 24
12
3/20/2014
Commit – make a prediction and share progress • • • • •
9:39 a.m. “…trying to do a one day audit” 9:50 a.m. “Here’s the scope statement…” 1:54 p.m. “Nearly finished with fieldwork…” 2:46 p.m. “First draft…” 5:00 p.m. “Whew. Done.”
25
Use feedback to motivate • 9:53 a.m. “Sweet. Let me know how it goes. Even two days would be pretty good :)” • 9:54 a.m. “Looks doable. Have fun.” • 1:56 p.m. “Keep going…keep me posted.” • 4:13 p.m. “Very nice! And quick!”
26
13
3/20/2014
Learn from peer review • Pay attention to other approaches • ALGA Peer review is a great way to learn about what works and how to think about audit standards
27
Make praiseworthy failures “An experiment conducted to expand knowledge and investigate a possibility leads to an undesired result.”
Amy Edmondson, “Strategies for Learning from Failure,” Harvard Business Review (April 2012)
28
14
3/20/2014
Make it easier to experiment People worry about image
Make it safer to experiment
…employee may choose to • Make experiments “play it safe” and avoid “rock‐ legitimate the‐boat” innovative behaviors • Make jobs require in order to look socially experiments and innovation appropriate and to prevent negative social evaluations.
Feirong Yuan, “Creativity and Innovation in Organizations,” presentation to University of Kansas Staff Leadership Summit 2011 29
Can you do a one‐day performance audit?
Reality
What you think and say “That won’t work because…”
“What would it take to make that happen?”
Doesn’t work
Right: but sent a signal to “not rock the boat”
Praiseworthy failure: learned and sent a signal to experiment
Does work
Wrong: a missed opportunity
Praiseworthy success: learned and signal to experiment 30
15
3/20/2014
• • • •
www.lawernceks.org/auditor Michael Eglinski [email protected] 785‐832‐3410
31
16
A One‐Day Performance Audit Michael Eglinski City Auditor Lawrence, KS 1
One‐Day Performance Audit • • • •
Is it possible? Can it meet Government Auditing Standards? Can it survive peer review? What are the lessons we can learn to improve audit productivity in general?
2
1
3/20/2014
My perspective • City Auditor for Lawrence, KS; one‐person shop; function created 2008 • Lawrence: 90,000 population (30,000 are students at KU or Haskell); $175 million budget; almost 800 employees • 1992‐2008 Kansas City, MO, a larger shop • Peer reviews in 8 other shops
3
4:30 on a Monday afternoon…
4
2
3/20/2014
On the last couple of pages of a 19 page phone bill – what is this?
5
“mytelebill llc”
6
3
3/20/2014
“Cramming”
7
By 5:00 p.m. Monday…. • Identified a potential problem • Identified criteria from FCC.gov
On the way home from work… • I could do an audit on cramming • And I could do it quickly
8
4
3/20/2014
8:00 Tuesday morning • Review and sign independence statement • Write objectives: – Has the city been billed for charges consistent with cramming? If so, what steps can be taken to strengthen controls to prevent cramming?
• Write work steps
9
The first hurdles • P&P and City Code require City Commission approval of performance audits • P&P and City Code require coordination with the City Manager on selecting audits
Key decision point • “It can’t be done” versus “what would it take to make this work?” 10
5
3/20/2014
Overcoming the hurdles: • 8:42 a.m. Contact City Commissioner who was acting as “liaison” to the audit function • 9:22 a.m. Get the go ahead from the liaison • Talk with the City Manager about the plan and idea – make sure it is ok. • Entrance conference with IT Department • 11:27 a.m. follow‐up with City Manager including scope and objectives • 1:28 p.m. inform all of the City Commissioners of the project 11
Fieldwork • Review criteria from FTC and FCC • Post to ALGA listserv • Review “judgmental sample” of phone bills to identify possible cramming charges • Follow‐up with staff who have crammed phone numbers – Aware of the service? – Aware of the charge? – Did you order the service? 12
6
3/20/2014
What I found • The city appears to have been charged for some services that weren’t ordered, agreed to or used • Staff…had no knowledge of the charges associated with the phone numbers identified as possible cramming charges.
13
Report writing • 2:08 p.m. started writing the report • Short – 2 pages including scope, objectives and method = ~750 words • Positive tone by focusing on the recommendation element of the finding (“The City should strengthen controls to prevent ‘cramming’”) • 3:03 p.m. shared draft with IT management 14
7
3/20/2014
Quality assurance • Referencing the entire report • Completing an audit standards checklist (based on the ALGA peer review engagement guide)
Conclusion of QA process • Followed Government Auditing Standards and I can add the relevant text to the report 15
Final hurdle • City Code requires management response and requires response “within 5 working days” • Report was done but rather than dump it on management at 5 p.m., provided the report for response the next morning • After 4 days, email response from IT management, summarized it and City Manager accepted it as his written response on day 5 16
8
3/20/2014
Is it possible? Can it meet Government Auditing Standards? • Almost…if only management would respond quicker! • Yes
But would a peer review team agree?
17
ALGA peer review a year later Review team noted two concerns: • Improve documentation of assessment of internal controls • Better report on scope of deficiencies in internal controls: “…are not clearly addressed in report. They are inferred in the audit recommendations. Note: an improvement in this area would really strengthen the audit discussion and recommendation.”
18
9
3/20/2014
A note I wrote at the end of the day • …I must have missed something in the standards. But, I completed the checklist and it seems like I’ve done everything ok. There might be an argument that I haven’t adequately described the scope of work on internal controls and deficiencies found...
19
Can it survive a peer review? • Yes, but room for improvement
20
10
3/20/2014
Can we learn lessons to improve audit productivity in general?
21
Flexibility and standard processes • Relatively flexible performance audit policies and procedures • Consider “pruning” your approach to focus on key processes for a given engagement • Balanced with a standard QA process • ALGA standards checklist
22
11
3/20/2014
Flexibility and standard process • Don’t be afraid to scope narrowly and stick to the scope – Petting zoos – Parade safety
• Think about the outcome and use professional judgment to determine the level of evidence
23
Evidence and criteria • The actual phone bills – solid documentary evidence. Approach wouldn’t work if it relied on complicated analysis or lots of interview evidence. • Simple, sound criteria from FTC and FCC (both provided consistent information). Approach wouldn’t work if criteria were inconsistent or not authoritative. 24
12
3/20/2014
Commit – make a prediction and share progress • • • • •
9:39 a.m. “…trying to do a one day audit” 9:50 a.m. “Here’s the scope statement…” 1:54 p.m. “Nearly finished with fieldwork…” 2:46 p.m. “First draft…” 5:00 p.m. “Whew. Done.”
25
Use feedback to motivate • 9:53 a.m. “Sweet. Let me know how it goes. Even two days would be pretty good :)” • 9:54 a.m. “Looks doable. Have fun.” • 1:56 p.m. “Keep going…keep me posted.” • 4:13 p.m. “Very nice! And quick!”
26
13
3/20/2014
Learn from peer review • Pay attention to other approaches • ALGA Peer review is a great way to learn about what works and how to think about audit standards
27
Make praiseworthy failures “An experiment conducted to expand knowledge and investigate a possibility leads to an undesired result.”
Amy Edmondson, “Strategies for Learning from Failure,” Harvard Business Review (April 2012)
28
14
3/20/2014
Make it easier to experiment People worry about image
Make it safer to experiment
…employee may choose to • Make experiments “play it safe” and avoid “rock‐ legitimate the‐boat” innovative behaviors • Make jobs require in order to look socially experiments and innovation appropriate and to prevent negative social evaluations.
Feirong Yuan, “Creativity and Innovation in Organizations,” presentation to University of Kansas Staff Leadership Summit 2011 29
Can you do a one‐day performance audit?
Reality
What you think and say “That won’t work because…”
“What would it take to make that happen?”
Doesn’t work
Right: but sent a signal to “not rock the boat”
Praiseworthy failure: learned and sent a signal to experiment
Does work
Wrong: a missed opportunity
Praiseworthy success: learned and signal to experiment 30
15
3/20/2014
• • • •
www.lawernceks.org/auditor Michael Eglinski [email protected] 785‐832‐3410
31
16
