Booklet: Development and Acquisition
Section: Appendix A: Examination Procedures
Subsection:
 

 

 

 

 

 

EXAMINATION OBJECTIVES: The objectives of Development and Acquisition assessments are to identify weaknesses or risks that could negatively impact an organization, to identify entities whose condition or performance requires special supervisory attention, and to subsequently effect corrective action.

Examiners should not expect organizations to employ formal project management techniques in all situations. Reviews should be risk focused and center on ensuring project management standards, controls, and procedures are present and commensurate with the characteristics and risks of the projects under review.

Examiners are not required to include lengthy responses to each Objective or bulleted item. Often, examiners should be able to simply note an item is adequate or inadequate with yes or no responses. However, examiners must adequately document material findings. Documentation must be sufficient to support the assignment of the Development and Acquisition component rating of the Federal Financial Institutions Examination Council’s Uniform Rating System for Information Technology.

OBJECTIVES AND PROCEDURES

Objective 1: Determine the scope of the Development and Acquisition review.

1.

Identify strengths and weaknesses relating to development, acquisition, and maintenance activities, through a review of:
  Bullet Prior reports of examination;
  Bullet Internal and external audits;
  Bullet Regulatory, audit, and security reports from key service providers;
  Bullet Organizational charts;
  Bullet Network topology maps; and
  Bullet Résumés of technology managers.

2.

Review management’s response to report and audit findings to determine:
  Bullet The adequacy and timing of corrective actions;
  Bullet The resolution of root causes rather than just specific issues; and
  Bullet The existence of outstanding issues.

3.

Review applicable documentation and interview technology managers to identify:

Bullet

The type and frequency of development, acquisition, and maintenance projects;
 
Bullet
 The formality and characteristics of project management techniques;
 
Bullet
 The material changes that impact development, acquisition, and maintenance activities, such as:

 

Proposed or enacted changes in hardware, software, or vendors;

 

Proposed or enacted changes in business objectives or organizational structures; and

 

Proposed or enacted changes in key personnel positions.
 

Objective 2: Assess the level of oversight and support provided by the board and management relating to development, acquisition, and maintenance activities.

1.

Assess the level of oversight and support by evaluating:

Bullet

The alignment of business and technology objectives;

Bullet

The frequency and quality of technology-related board reporting;

Bullet

The commitment of the board and senior management to promote new products;

Bullet

The level and quality of board-approved project standards and procedures;

Bullet

The qualifications of technology managers; and

Bullet

The sufficiency of technology budgets.
 

Objective 3: Assess the organizational structure in relation to the appropriateness of assigned responsibilities concerning technology systems and initiatives.

1.

Evaluate organizational responsibilities to ensure the board and management:

Bullet

Clearly define and appropriately assign responsibilities;

Bullet

Appropriately assign security, audit, and quality assurance personnel to technology-related projects;

Bullet

Establish appropriate segregation-of-duty or compensating controls; and

Bullet

Establish appropriate project, technology committee, and board reporting requirements.
 

Objective 4: Assess the level and characteristics of risks associated with development, acquisition, and maintenance activities that could materially impact the organization.

1.

Assess the risks identified in other objectives and evaluate the adequacy of risk management programs regarding:

Bullet

Risk identification and assessment procedures;

Bullet

Risk reporting and monitoring procedures; and

Bullet

Risk acceptance, mitigation, and transfer strategies.
 

Objective 5: Assess the adequacy of development project management standards, methodologies, and practices.

1.

Evaluate the adequacy of development activities by assessing:

Bullet

The adequacy of, and adherence to, development standards and controls;

Bullet

The applicability and effectiveness of project management methodologies;

Bullet

The experience of project managers;

Bullet

The adequacy of project plans, particularly with regard to the inclusion of clearly defined:

 

Phase expectations;

 

Phase acceptance criteria;

 

Security and control requirements;

 

Testing requirements; and

 

Documentation requirements;

Bullet

The formality and effectiveness of quality assurance programs;

Bullet

The effectiveness of risk management programs;

Bullet

The adequacy of project request and approval procedures;

Bullet

The adequacy of feasibility studies;

Bullet

The adequacy of, and adherence to, standards and procedures relating to the:

 

Design phase;

 

Development phase;

 

Testing phase; and

 

Implementation phase;

Bullet

The adequacy of project change controls;

Bullet

The appropriate inclusion of organizational personnel throughout the project’s life cycle;

Bullet

The effectiveness of project communication and reporting procedures; and

Bullet

The accuracy, effectiveness, and control of project management tools.
 

Objective 6: Assess the adequacy of acquisition project management standards, methodologies, and practices.

1.

Assess the adequacy of acquisition activities by evaluating:

Bullet

The adequacy of, and adherence to, acquisition standards and controls;

Bullet

The applicability and effectiveness of project management methodologies;

Bullet

The experience of project managers;

Bullet

The adequacy of project plans, particularly with regard to the inclusion of clearly defined:

 

Phase expectations;

 

Phase acceptance criteria;

 

Security and control requirements; and

 

Testing, training, and implementation requirements;

Bullet

The formality and effectiveness of quality assurance programs;

Bullet

The effectiveness of risk management programs;

Bullet

The adequacy of project request and approval procedures;

Bullet

The adequacy of feasibility studies;

Bullet

The adequacy of, and adherence to, standards that require request-for-proposals and invitations-to-tender to include:

 

Well-detailed security, reliability, and functionality specifications;

 

Well-defined performance and compatibility specifications; and

 

Well-defined design and development documentation requirements;

Bullet

The adequacy of, and adherence to, standards that require:

 

Thorough reviews of vendors’ financial condition and commitment to service; and

 

Thorough reviews of contracts and licensing agreements prior to signing;

Bullet

The adequacy of contract and licensing provisions that address:

 

Performance assurances;

 

Software and data security provisions; and

 

Source-code accessibility/escrow assertions;

Bullet

The adequacy of project change controls;

Bullet

The appropriate inclusion of organizational personnel throughout the project’s life cycle;

Bullet

The effectiveness of project communication and reporting procedures; and

Bullet

The accuracy, effectiveness, and control of project management tools.
 

Objective 7: Assess the adequacy of maintenance project management standards, methodologies, and practices.

1.

Evaluate the sufficiency of, and adherence to, maintenance standards and controls relating to:

Bullet

Change request and approval procedures;

Bullet

Change testing procedures;

Bullet

Change implementation procedures;

Bullet

Change review procedures;

Bullet

Change documentation procedures;

Bullet

Change notification procedures

Bullet

Library controls; and

Bullet

Utility program controls.
 

Objective 8: Assess the effectiveness of conversion projects.

1.

Evaluate the effectiveness of conversion projects by:

Bullet

Comparing initial budgets and projected time lines against actual results;

Bullet

Reviewing project management and technology committee reports;

Bullet

Reviewing testing documentation and after-action reports;

Bullet

Reviewing conversion after-action reports;

Bullet

Interviewing technology and user personnel; and

Bullet

Reviewing suspense accounts for outstanding items.
 

Objective 9: Assess the adequacy of quality assurance programs.

1.

Assess the adequacy of quality assurance programs by evaluating:

Bullet

The board’s willingness to provide appropriate resources to quality assurance programs;

Bullet

The completeness of quality assurance procedures (Are the deliverables of each project, and project phase, including the validation of initial project assumptions and approvals, appropriately assured?);

Bullet

The scalability of quality assurance procedures (Are the procedures appropriately tailored to match the characteristics of the project?);

Bullet

The measurability of quality assurance standards (Are deliverables assessed against predefined standards and expectations?);

Bullet

The adherence to problem-tracking standards that require:

 

Appropriate problem recordation;

 

Appropriate problem reporting;

 

Appropriate problem monitoring; and

 

Appropriate problem correction;

Bullet

The sufficiency of, and adherence to, testing standards that require:

 

The use of predefined, comprehensive test plans;

 

The involvement of end users;

 

The documentation of test results;

 

The prohibition against testing in production environments; and

 

The prohibition against testing with live data;

Bullet

The sufficiency and effectiveness of testing programs regarding:

 

The accuracy of programmed code;

 

The inclusion of expected functionality; and

 

The interoperability of applications and network components; and

Bullet

The independence of quality assurance personnel.
 

Objective 10: Assess the adequacy of program change controls.

1.

Evaluate the sufficiency of, and adherence to:

Bullet

Routine and emergency program-change standards that require appropriate:

 

Request and approval procedures;

 

Testing procedures;

 

Implementation procedures;

 

Backup and backout procedures;

 

Documentation procedures; and

 

Notification procedures;

Bullet

Controls that restrict the unauthorized movement of programs or program modules/objects between development, testing, and production environments;

Bullet

Controls that restrict the unauthorized use of utility programs, such as:

 

Policy prohibitions;

 

Monitoring of use; and

 

Logical access controls;

Bullet

Library controls that restrict unauthorized access to programs outside an individual’s assigned responsibilities such as:

 

Logical access controls on all libraries or objects within libraries; and

 

Automated library controls that restrict library access and produce reports that identify who accessed a library, what was accessed, and what changes were made; and

Bullet

Version controls that facilitate the appropriate retention of programs, and program modules/objects, revisions, and documentation.
 

Objective 11: Assess the adequacy of patch-management standards and controls.

1.

Evaluate the sufficiency of, and adherence to, patch-management standards and controls that require:

Bullet

Detailed hardware and software inventories;

Bullet

Patch identification procedures;

Bullet

Patch evaluation procedures;

Bullet

Patch request and approval procedures;

Bullet

Patch testing procedures;

Bullet

Backup and backout procedures;

Bullet

Patch implementation procedures; and

Bullet

Patch documentation.

Objective 12: Assess the quality of application, system, and project documentation, and the adequacy of documentation controls.

1.

Assess the adequacy of documentation controls by evaluating the sufficiency of, and adherence to, documentation standards that require:

Bullet

The assignment of documentation-custodian responsibilities;

Bullet

The assignment of document authoring and approval responsibilities;

Bullet

The establishment of standardized document formats; and

Bullet

The establishment of appropriate documentation library and version controls.

2.

Assess the quality of application documentation by evaluating the adequacy of internal and external assessments of:

Bullet

Application design and coding standards;

Bullet

Application descriptions;

Bullet

Application design documents;

Bullet

Application source-code listings (or in the case of object-oriented programming: object listings);

Bullet

Application routine naming conventions (or in the case of object-oriented programming: object naming conventions); and

Bullet

Application operator instructions and user manuals.

3.

Assess the quality of open source-code system documentation by evaluating the adequacy of internal and external assessments of:

Bullet

System design and coding standards;

Bullet

System descriptions;

Bullet

System design documents;

Bullet

Source-code listings (or in the case of object-oriented programming: object listings);

Bullet

Source-code routine naming conventions (or in the case of object-oriented programming: object naming conventions); and

Bullet

System operation instructions.

4.

Assess the quality of project documentation by evaluating the adequacy of documentation relating to the:

Bullet

Project request;

Bullet

Feasibility study;

Bullet

Initiation phase;

Bullet

Planning phase;

Bullet

Design phase;

Bullet

Development phase;

Bullet

Testing phase;

Bullet

Implementation phase; and

Bullet

Post-implementation reviews.

Note: If examiners employ sampling techniques, they should include planning and testing phase documentation in the sample.

 

Objective 13: Assess the security and integrity of system and application software.

1.

Assess the quality of open source-code system documentation by evaluating the adequacy of internal and external assessments of:

Bullet

Evaluate the security and integrity of system and application software by reviewing:

Bullet

The adequacy of quality assurance and testing programs;

Bullet

The adequacy of security and internal-control design standards;

Bullet

The adequacy of program change controls;

Bullet

The adequacy of involvement by audit and security personnel in software development and acquisition projects; and

Bullet

The adequacy of internal and external security and control audits.
 

Objective 14: Assess the ability of information technology solutions to meet the needs of the end users.

1.

Interview end users to determine their assessment of technology solutions.

 

Objective 15: Assess the extent of end-user involvement in the system development and acquisition process.

1.

Interview end users and review development and acquisition project documentation to determine the extent of end-user involvement.
 

CONCLUSIONS
 

Objective 16: Document and discuss findings and recommend corrective actions.

1.

Document findings and recommendations regarding the quality and effectiveness of the organization’s Development and Acquisition standards and procedures.

2.

Discuss preliminary findings with the examiner-in-charge regarding:

Bullet

Violations of laws, rulings, or regulations; and

Bullet

Issues warranting inclusion in the report of examination.

3.

Discuss your findings with management and obtain commitments for corrective actions and deadlines for remedying significant deficiencies.

4.

Discuss findings with the examiner-in-charge regarding:

Bullet

Recommendations regarding the Development and Acquisition rating; and

Bullet

Recommendations regarding the impact of your conclusions on the composite rating(s).

5.

Document your conclusions in a memo to the examiner-in-charge that provides report-ready comments for all relevant sections of the report of examination.

6.

Organize your work papers to ensure clear support for significant findings and recommendations.