Booklet: Management Section: IT Risk Management Process Subsection: Measure and Monitor

Previous Subsection Table of Contents Next Subsection

Action Summary

Financial institutions should continuously measure and monitor the risk profile of their IT functions. Metrics, as part of the monitoring process, will aid management in its ability to assess the overall program. The specific metrics reported, and the frequency, will depend upon the IT environment of the institution. Some common examples are:

PLAN-TO-ACTUAL OUTCOME MEASURES (OUTCOME-BASED MEASUREMENT)
Financial institutions should periodically review their IT function and determine if their plan, goals, and expectations are on target. Given the cost of, and business reliance upon, IT functions, failure to perform such measurements could put the institution at risk. Management should measure outsourced relationships by the penalties and incentive clauses in the service contracts.

PERFORMANCE BENCHMARKS
Financial institutions should establish performance benchmarks or standards for IT functions and monitor them on a regular basis. Such monitoring can identify potential problem areas and provide assurance that IT functions are meeting the objectives. Areas to consider include mainframe and network availability, data center availability, system reruns, out of balance conditions, response time, error rates, data entry volumes, special requests, and problem reports.

SERVICE LEVELS
Financial institutions should establish formal service level agreements with their IT provider, for both in-house and outsourced functions. Service level agreements (SLAs) establish mutual expectations and provide a baseline to measure IT performance. Management can also tie SLAs to incentive and penalty actions. SLAs should broadly cover the IT environment to provide the institution the greatest level of assurance. Performance benchmarks and outcome-based measurements (see above) are examples of SLA issues.

QUALITY ASSURANCE/QUALITY CONTROL
Management should establish quality assurance procedures and update future planning with the quality assurance results. These procedures may include internal performance measures, focus groups, and customer surveys. Management should conduct quality assurance reviews for all significant activities both internally and with another organization.

The traditional goal of Quality Assurance (QA) activities is to ensure the product conforms to specifications, and is fit to use. QA asks three fundamental questions: Does it work? Does it do what it is designed to do? Is it fit for use? The purpose of quality Control (QC) activities is to identify weaknesses in work products and to avoid the resource drain and expense of redoing a task. While financial institutions will benefit from that perspective, they also have additional incentives to incorporate QA functions into their IT environment. QA functions can be effective in preventing internal fraud. For example, management can conduct quality assurance testing on a new system before implementation. The testing should be independent of any programming function (if developed in-house) and incorporate user acceptance testing programs (if off-the-shelf). The thorough testing of a new system can identify malicious code or poor functionality. QA reports are a valuable tool for management and help document the control process for the production environment.

POLICY COMPLIANCE
Financial institutions should develop, implement, and monitor a process to measure IT compliance with their established policies, standards, and practices. In addition to the traditional reliance upon internal and third party audit functions, financial institutions should perform self-assessments on a periodic basis. The scope and frequency of self- assessments will depend upon the scale and historical performance of the IT function. Self-assessment activities broaden management’s perspective by involving a varied audience and by requiring acknowledgement of the results by those involved. The self-assessment process can help identify the need for policy changes and updates.

Previous Subsection Table of Contents Next Subsection           Home IT Booklets Glossary Resources Presentations