| Booklet:
Development
and Acquisition
Section: Appendix
A: Examination
Procedures
Subsection:
|
| |
| |
EXAMINATION
OBJECTIVES: The objectives of Development and Acquisition assessments
are to identify weaknesses or risks that could negatively impact an organization,
to identify entities whose condition or performance requires special supervisory
attention, and to subsequently effect corrective action.
Examiners should not expect organizations to employ formal project management
techniques in all situations. Reviews should be risk focused and center
on ensuring project management standards, controls, and procedures are
present and commensurate with the characteristics and risks of the projects
under review.
Examiners are not required to include lengthy responses to each Objective
or bulleted item. Often, examiners should be able to simply note an item
is adequate or inadequate with yes or no responses. However, examiners
must adequately document material findings. Documentation must be sufficient
to support the assignment of the Development and Acquisition component
rating of the Federal Financial Institutions Examination Council’s
Uniform Rating System for Information Technology.
OBJECTIVES
AND PROCEDURES
2. |
Review
management’s response to report and audit findings to determine: |
| |
 |
The
adequacy and timing of corrective actions; |
| |
 |
The
resolution of root causes rather than just specific issues; and |
| |
 |
The
existence of outstanding issues. |
3. |
Review
applicable documentation and interview technology managers to identify: |
|

|
The
type and frequency of development, acquisition, and maintenance
projects; |
| |
|
The
formality and characteristics of project management techniques; |
| |
|
The
material changes that impact development, acquisition, and maintenance
activities, such as: |
|
|

|
Proposed
or enacted changes in hardware, software, or vendors; |
|
|

|
Proposed
or enacted changes in business objectives or organizational structures;
and |
|
|

|
Proposed
or enacted changes in key personnel positions. |
| |
Objective
2: Assess the level of oversight and support provided by the board
and management relating to development, acquisition, and maintenance
activities. |
1. |
Assess
the level of oversight and support by evaluating: |
|

|
The
alignment of business and technology objectives; |
|

|
The
frequency and quality of technology-related board reporting; |
|

|
The
commitment of the board and senior management to promote new products; |
|

|
The
level and quality of board-approved project standards and procedures; |
|

|
The
qualifications of technology managers; and |
|

|
The
sufficiency of technology budgets. |
| |
Objective
3: Assess the organizational structure in relation to the appropriateness
of assigned responsibilities concerning technology systems and initiatives. |
1. |
Evaluate
organizational responsibilities to ensure the board and management: |
|

|
Clearly
define and appropriately assign responsibilities; |
|

|
Appropriately
assign security, audit, and quality assurance personnel to technology-related
projects; |
|

|
Establish
appropriate segregation-of-duty or compensating controls; and |
|

|
Establish
appropriate project, technology committee, and board reporting requirements. |
| |
Objective
4: Assess the level and characteristics of risks associated with
development, acquisition, and maintenance activities that could
materially impact the organization. |
1. |
Assess
the risks identified in other objectives and evaluate the adequacy
of risk management programs regarding: |
|

|
Risk
identification and assessment procedures; |
|

|
Risk
reporting and monitoring procedures; and |
|

|
Risk
acceptance, mitigation, and transfer strategies. |
| |
Objective
5: Assess the adequacy of development project management standards,
methodologies, and practices. |
1. |
Evaluate
the adequacy of development activities by assessing: |
|

|
The
adequacy of, and adherence to, development standards and controls; |
|

|
The
applicability and effectiveness of project management methodologies; |
|

|
The
experience of project managers; |
|

|
The
adequacy of project plans, particularly with regard to the inclusion
of clearly defined: |
|
|

|
Phase
expectations; |
|
|

|
Phase
acceptance criteria; |
|
|

|
Security
and control requirements; |
|
|

|
Testing
requirements; and |
|
|

|
Documentation
requirements; |
|

|
The
formality and effectiveness of quality assurance programs; |
|

|
The
effectiveness of risk management programs; |
|

|
The
adequacy of project request and approval procedures; |
|

|
The
adequacy of feasibility studies; |
|

|
The
adequacy of, and adherence to, standards and procedures relating
to the: |
|
|

|
Design
phase; |
|
|

|
Development
phase; |
|
|

|
Testing
phase; and |
|
|

|
Implementation
phase; |
|

|
The
adequacy of project change controls; |
|

|
The
appropriate inclusion of organizational personnel throughout the
project’s life cycle; |
|

|
The
effectiveness of project communication and reporting procedures;
and |
|

|
The
accuracy, effectiveness, and control of project management tools. |
| |
Objective
6: Assess the adequacy of acquisition project management standards,
methodologies, and practices. |
1. |
Assess
the adequacy of acquisition activities by evaluating: |
|

|
The
adequacy of, and adherence to, acquisition standards and controls; |
|

|
The
applicability and effectiveness of project management methodologies; |
|

|
The
experience of project managers; |
|

|
The
adequacy of project plans, particularly with regard to the inclusion
of clearly defined: |
|
|

|
Phase
expectations; |
|
|

|
Phase
acceptance criteria; |
|
|

|
Security
and control requirements; and |
|
|

|
Testing,
training, and implementation requirements; |
|

|
The
formality and effectiveness of quality assurance programs; |
|

|
The
effectiveness of risk management programs; |
|

|
The
adequacy of project request and approval procedures; |
|

|
The
adequacy of feasibility studies; |
|

|
The
adequacy of, and adherence to, standards that require request-for-proposals
and invitations-to-tender to include: |
|
|

|
Well-detailed
security, reliability, and functionality specifications; |
|
|

|
Well-defined
performance and compatibility specifications; and |
|
|

|
Well-defined
design and development documentation requirements; |
|

|
The
adequacy of, and adherence to, standards that require: |
|
|

|
Thorough
reviews of vendors’ financial condition and commitment to service;
and |
|
|

|
Thorough
reviews of contracts and licensing agreements prior to signing; |
|

|
The
adequacy of contract and licensing provisions that address: |
|
|

|
Performance
assurances; |
|
|

|
Software
and data security provisions; and |
|
|

|
Source-code
accessibility/escrow assertions; |
|

|
The
adequacy of project change controls; |
|

|
The
appropriate inclusion of organizational personnel throughout the
project’s life cycle; |
|

|
The
effectiveness of project communication and reporting procedures;
and |
|

|
The
accuracy, effectiveness, and control of project management tools. |
Objective
7: Assess the adequacy of maintenance project management standards,
methodologies, and practices. |
1. |
Evaluate
the sufficiency of, and adherence to, maintenance standards and
controls relating to: |
|

|
Change
request and approval procedures; |
|

|
Change
testing procedures; |
|

|
Change
implementation procedures; |
|

|
Change
review procedures; |
|

|
Change
documentation procedures; |
|

|
Change
notification procedures |
|

|
Library
controls; and |
|

|
Utility
program controls. |
| |
Objective
8: Assess the effectiveness of conversion projects. |
1. |
Evaluate
the effectiveness of conversion projects by: |
|

|
Comparing
initial budgets and projected time lines against actual results; |
|

|
Reviewing
project management and technology committee reports; |
|

|
Reviewing
testing documentation and after-action reports; |
|

|
Reviewing
conversion after-action reports; |
|

|
Interviewing
technology and user personnel; and |
|

|
Reviewing
suspense accounts for outstanding items. |
| |
Objective
9: Assess the adequacy of quality assurance programs. |
1. |
Assess
the adequacy of quality assurance programs by evaluating: |
|

|
The
board’s willingness to provide appropriate resources to quality
assurance programs; |
|

|
The
completeness of quality assurance procedures (Are the deliverables
of each project, and project phase, including the validation of
initial project assumptions and approvals, appropriately assured?); |
|

|
The
scalability of quality assurance procedures (Are the procedures
appropriately tailored to match the characteristics of the project?); |
|

|
The
measurability of quality assurance standards (Are deliverables assessed
against predefined standards and expectations?); |
|

|
The
adherence to problem-tracking standards that require: |
|
|

|
Appropriate
problem recordation; |
|
|

|
Appropriate
problem reporting; |
|
|

|
Appropriate
problem monitoring; and |
|
|

|
Appropriate
problem correction; |
|

|
The
sufficiency of, and adherence to, testing standards that require: |
|
|

|
The
use of predefined, comprehensive test plans; |
|
|

|
The
involvement of end users; |
|
|

|
The
documentation of test results; |
|
|

|
The
prohibition against testing in production environments; and |
|
|

|
The
prohibition against testing with live data; |
|

|
The
sufficiency and effectiveness of testing programs regarding: |
|
|

|
The
accuracy of programmed code; |
|
|

|
The
inclusion of expected functionality; and |
|
|

|
The
interoperability of applications and network components; and |
|

|
The
independence of quality assurance personnel. |
| |
Objective
10: Assess the adequacy of program change controls. |
1. |
Evaluate
the sufficiency of, and adherence to: |
|

|
Routine
and emergency program-change standards that require appropriate: |
|
|

|
Request
and approval procedures; |
|
|

|
Testing
procedures; |
|
|

|
Implementation
procedures; |
|
|

|
Backup
and backout procedures; |
|
|

|
Documentation
procedures; and |
|
|

|
Notification
procedures; |
|

|
Controls
that restrict the unauthorized movement of programs or program modules/objects
between development, testing, and production environments; |
|

|
Controls
that restrict the unauthorized use of utility programs, such as: |
|
|

|
Policy
prohibitions; |
|
|

|
Monitoring
of use; and |
|
|

|
Logical
access controls; |
|

|
Library
controls that restrict unauthorized access to programs outside an
individual’s assigned responsibilities such as: |
|
|

|
Logical
access controls on all libraries or objects within libraries; and |
|
|

|
Automated
library controls that restrict library access and produce reports
that identify who accessed a library, what was accessed, and what
changes were made; and |
|

|
Version
controls that facilitate the appropriate retention of programs,
and program modules/objects, revisions, and documentation. |
| |
Objective
11: Assess the adequacy of patch-management standards and controls.
|
1. |
Evaluate
the sufficiency of, and adherence to, patch-management standards
and controls that require: |
|

|
Detailed
hardware and software inventories; |
|

|
Patch
identification procedures; |
|

|
Patch
evaluation procedures; |
|

|
Patch
request and approval procedures; |
|

|
Patch
testing procedures; |
|

|
Backup
and backout procedures; |
|

|
Patch
implementation procedures; and |
|

|
Patch
documentation. |
|
Objective
12: Assess the quality of application, system, and project documentation,
and the adequacy of documentation controls. |
1. |
Assess
the adequacy of documentation controls by evaluating the sufficiency
of, and adherence to, documentation standards that require: |
|

|
The
assignment of documentation-custodian responsibilities; |
|

|
The
assignment of document authoring and approval responsibilities; |
|

|
The
establishment of standardized document formats; and |
|

|
The
establishment of appropriate documentation library and version controls. |
2. |
Assess
the quality of application documentation by evaluating the adequacy
of internal and external assessments of: |
|

|
Application
design and coding standards; |
|

|
Application
descriptions; |
|

|
Application
design documents; |
|

|
Application
source-code listings (or in the case of object-oriented programming:
object listings); |
|

|
Application
routine naming conventions (or in the case of object-oriented programming:
object naming conventions); and |
|

|
Application
operator instructions and user manuals. |
3. |
Assess
the quality of open source-code system documentation by evaluating
the adequacy of internal and external assessments of: |
|

|
System
design and coding standards; |
|

|
System
descriptions; |
|

|
System
design documents; |
|

|
Source-code
listings (or in the case of object-oriented programming: object
listings); |
|

|
Source-code
routine naming conventions (or in the case of object-oriented programming:
object naming conventions); and |
|

|
System
operation instructions. |
4. |
Assess
the quality of project documentation by evaluating the adequacy
of documentation relating to the: |
|

|
Project
request; |
|

|
Feasibility
study; |
|

|
Initiation
phase; |
|

|
Planning
phase; |
|

|
Design
phase; |
|

|
Development
phase; |
|

|
Testing
phase; |
|

|
Implementation
phase; and |
|

|
Post-implementation
reviews. |
Note:
If examiners employ sampling techniques, they should include planning
and testing phase documentation in the sample. |
| |
Objective
13: Assess the security and integrity of system and application
software. |
1. |
Assess
the quality of open source-code system documentation by evaluating
the adequacy of internal and external assessments of: |
|

|
Evaluate
the security and integrity of system and application software by
reviewing: |
|

|
The
adequacy of quality assurance and testing programs; |
|

|
The
adequacy of security and internal-control design standards; |
|

|
The
adequacy of program change controls; |
< |