Why IV&V matters : a white paper

Why IV&V Matters
A White Paper by
GEORGIA TECHNOLOGY AUTHORITY ENTERPRISE PROGRAM MANAGEMENT OFFICE
(EPMO)

Enterprise Program Management Office

Why IV&V Matters

PURPOSE OF IV&V
Industry practice shows that a disciplined approach to project, program and portfolio management increases benefits realized by businesses from their investment in information technology assets and resources. The State of Georgia portfolio of information technology projects was estimated at $367 million (as of September 2007), which had an effective return1 of 55%. This return which is low even by industry standards reflects a loss of workforce productivity, financial resources and benefits to constituencies.
Independent Verification and Validation (IV&V) provides assurance of project success in an enterprise. This is accomplished in two major ways; first, by communicating and educating the project management team on industry best practices for specific undertakings, and secondly, by providing an escalation path for issues and inhibitors of project success. While most of the focus and attention occurs with the first item (issues), the underlying value occurs by reducing the second item (inhibitors).
The primary objective of an IV&V engagement is to provide an objective assessment of products and processes throughout the project lifecycle. In addition, IV&V will facilitate early detection and correction of errors, enhance management insight into risks and ensure compliance with project performance, schedule, and budget requirements.
Industry Basis for IV&V
Verification and Validation (V&V) is a systems engineering discipline which helps a development organization build quality into the software during the software life cycle. Validation is concerned with checking that the software meets the user's needs, and Verification is concerned with checking that the system is well engineered.
Independent Verification and Validation (IV&V) is a set of Verification and Validation activities performed by an agency that is not under the control of the organization that is developing the software. IV&V services must be provided, managed and financed by organizations that are technically, managerially and financially independent of the development project. Technical independence requires that the IV&V does not use personnel who are involved in the development effort. Managerial independence requires that the IV&V effort be vested in an organization separate from the development and program management organizations. The IV&V must be able to submit to State and Federal management, the IV&V results and findings without any restrictions (e.g. without any prior review or approval from the development group). Financial independence requires that control of the IV&V budget be vested in an organization independent of the development organization.

1 Effective Return is calculated by looking at the success rate of projects and the total expenditures to achieve the objectives and returns. Success rates are divided into three (3) categories, Failed, Challenged and Successful, as described in the Standish Chaos report. It has also been shown that Challenged projects deliver at a higher cost than planned which decreases the benefits achieved. Effective Rate = Delivered Project $ Value / (Cancelled Projects $ Cost + Completed Projects Total $ Cost).

White Paper

Page 2

Enterprise Program Management Office

Why IV&V Matters

Industry Standard on IV&V

The definition of activities included under IV&V is quite broad, including both technical and management activities. The most authoritative source for IV&V can be found in the Institute of Electrical and Electronic Engineers Standard for Software Verification and Validation (IEEE Std. 1012-1998). The IEEE Standard describes software IV&V processes as generally determining if development products of a given activity conform to the requirements of that activity, and if the software satisfies the intended use and user needs. As defined in the IEEE standards, IV&V processes include activities such as assessment, analysis, evaluation, review, inspection, and testing of software products and processes. These IV&V processes further include assessing software in the context of the system, including the operational environment, hardware, interfacing software, operators and users. The IEEE standard seeks to assure that software IV&V is performed in parallel with software development, not at the conclusion of the software development.

Federal Perspective on IV&V
The Federal approach to IV&V differs considerably from standard IV&V, such as that described in the IEEE Std. 1012-1998. Federal requirements for IV&V on State automation projects are limited in their scope from the industry standard IEEE definition for IV&V in two key regards:
1. Federal IV&V does not require a continuous on-site presence. Instead, it requires periodic site visits to get a "snapshot" of a project's management and technical processes at predetermined intervals. Further, in some respects, the IV&V Service Provider can be viewed as performing a "Technology Audit."

2. The Federal requirements for IV&V are, in fact, a subset of the full IV&V standard as defined by the IEEE Standard 1012-1098, specifically excluding the activity of extensive testing.

GTA Perspective on IV&V

The Georgia Technology Authority's overall mission was established by the Georgia Legislature.

The legislative authority is provided by the following sections of the Official Code of Georgia Annotated (O.C.G.A.):



O.C.G.A. Section 45-12-70 et seq.



O.C.G.A. Section 50-5-51 (1), (2) and (11).



O.C.G.A. Section 50-25-1(b)(14).



O.C.G.A. Section 50-25-1(c).



O.C.G.A. Sections 50-25-4(a) (10).



O.C.G.A. Section 50-25-5.1(b)(3).

The authority for IV&V is provided from House Resolution 1263 and Senate Resolution 754.

White Paper

Page 3

Enterprise Program Management Office

Why IV&V Matters

A key difference in how GTA performs the IV&V Services is in its ability to qualify and procure outside vendor services for the performance of IV&V. GTA also established a process to ensure there is recognized value in the detailed, structured reports of findings of deficiencies and recommendations to the project sponsor and to GTA. This reporting process, in accordance with GTA regulatory requirements, includes not only final report issuance, but all draft report submissions as well. Again, the intent of the GTA and State in acquiring an IV&V Service Provider, unlike that which might be defined under the IEEE 10121098 standards for IV&V, is not to continually work with various project components to actively participate in the remediation of deficiencies and risks. Rather, the requirement for the IV&V Service Provider is to provide periodic, independent analyses of the areas of responsibility as presented within the scope of services of the project in order to identify, inform and educate project management as well as the cognizant state Office of any areas of weakness and risk to the project, as well to provide proposed and recommended solutions for their remediation and/or mitigation.
Outcomes of Case Studies
During 2008, IV&V has made the following tangible, positive impacts worth an estimated $29.6 million on an investment of approximately $2.1 million:

TRS/DIS - $2.6m at risk and saved; recovery plan and recommendations saved expenditures that would have been wasted.
DCH/HITT - $8.2m at risk and saved; early escalation and recommendations saved expenditures that would have been wasted.
DCH/MEMS - $1.5m at risk and saved; early adoption of recommendations saved delivery schedule and expenditures.
DOAS/TGM - $10.9m at risk and savings of $2.5m; early adoption of recommendations saved delivery schedule and wasted expenditures.
DCH/MMIS - $34.9m at risk with savings of $3.5m; early adoption of recommendations saved procurement and contracting, and efforts on requirements and risk management.
DDS/DLS/EDIS Program - $20.0m at risk with savings of $4.5m; recommendations and changes averted potentially fatal problems during procurement and execution.
DHR/SHINES - $16.0m at risk with savings of $3.8m; recommendations in final phases of delivery and transition averted costly testing and roll-out problems.
DCH/Data Broker - $5.0m at risk with savings of $2.4m; recommendations created project recovery and averted significant issues and risks.
DOR/IT/DW Program - $63.3m at risk creating savings of $0.6m; recommendations and changes in early assessment discussions improved overall performance/success.

For more information on the specific processes or how to conduct IV&V, see the following link:
http://gta.georgia.gov/00/channel_modifieddate/0,2096,1070969_144323748,00.html

White Paper

Page 4

Enterprise Program Management Office

Why IV&V Matters

Summary
IV&V is considered within the technology industry a "best practice", which has been validated by independent research. Its primary value is in identifying high-risk areas early in the project effort which allows the business to either mitigate or prepare contingencies. It also provides business leaders an objective analysis in order to deal with system development issues and it provides IT management with improved visibility into the progress and quality of the development effort.
Ultimately, it provides visibility, accountability and fact-based decision making for technology initiatives which is rewarded by technology systems that provide value and support to the business of the State of Georgia.

White Paper

Page 5