25.12.2014 Views

BOS IV&V Quality Assurance Plan Final - Silver State Health ...

BOS IV&V Quality Assurance Plan Final - Silver State Health ...

BOS IV&V Quality Assurance Plan Final - Silver State Health ...

SHOW MORE
SHOW LESS

You also want an ePaper? Increase the reach of your titles

YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.

<strong>State</strong> of Nevada<br />

Business Operations Solutions (<strong>BOS</strong>)<br />

Independent Verification and Validation<br />

Services (IV&V) of the <strong>Silver</strong> <strong>State</strong> <strong>Health</strong><br />

Insurance Exchange (Exchange)<br />

<strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

Deliverable 5.4.3.6 - Activity 5.4.2.7<br />

August 30, 2013


Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Insurance Exchange Project<br />

IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

Version History<br />

VERSION DATE COMMENTS<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> -<br />

Draft v1<br />

5.4.3.6 - <strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong><br />

<strong>Plan</strong> <strong>Final</strong>.doc<br />

08/30/2013 Initial Draft, K. Marshall<br />

9/06/2013 <strong>BOS</strong> revisions, P. Palmer<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page i


Table of Contents<br />

Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Insurance Exchange Project<br />

IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

1 PURPOSE ................................................................................................................................... 1<br />

1.1 Scope and Objectives .................................................................................................................. 1<br />

2 REFERENCE DOCUMENTS ........................................................................................................... 2<br />

3 STANDARDS AND GUIDELINES .................................................................................................... 2<br />

4 GLOSSARY .................................................................................................................................. 3<br />

5 MANAGEMENT .......................................................................................................................... 4<br />

5.1 Organization ................................................................................................................................ 4<br />

5.2 Tasks ............................................................................................................................................ 6<br />

5.3 Roles and Responsibilities ........................................................................................................... 6<br />

6 QUALITY ASSURANCE APPROACH ............................................................................................... 9<br />

6.1 Process <strong>Quality</strong> .......................................................................................................................... 10<br />

6.2 Product <strong>Quality</strong> .......................................................................................................................... 11<br />

6.3 <strong>Quality</strong> <strong>Assurance</strong> Processes ..................................................................................................... 12<br />

6.3.1 Deliverable Review Process .......................................................................................... 12<br />

6.3.2 Requirements Tracking ................................................................................................. 13<br />

6.4 Testing Strategy ......................................................................................................................... 13<br />

6.5 Continuous Process and Product <strong>Quality</strong> Improvement ........................................................... 13<br />

7 QAP CHANGE PROCEDURE ....................................................................................................... 14<br />

8 APPENDIX A – SAMPLE PRODUCT QUALITY REVIEW CHECKLIST ................................................. 15<br />

Table of Figures<br />

Figure 1 – <strong>Quality</strong> <strong>Assurance</strong> Organization and Flow ................................................................................... 5<br />

Figure 2 – <strong>Quality</strong> <strong>Assurance</strong> Process ......................................................................................................... 10<br />

Figure 3 – Product <strong>Quality</strong> Review Process for Project Deliverables .......................................................... 12<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page ii


Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

1 Purpose<br />

The purpose of the <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> (QAP) is to describe Public<br />

Consulting Group’s (PCG’s) approach to ensuring quality on the project. It<br />

includes assessing the methodology for maintaining quality of the code and<br />

workmanship, managing project schedules, and related subcontractor activities<br />

where appropriate. This plan describes the methods that will be used both to<br />

assess the quality of the work products produced and the quality of the<br />

processes used to produce those products and identifies the activities,<br />

processes, and procedures that will be used to assess and monitor quality. The<br />

QAP also identifies the roles and responsibilities specific to the <strong>Quality</strong><br />

<strong>Assurance</strong> (QA) function needed to successfully implement best practices.<br />

1.1 Scope and Objectives<br />

The Business Operations Solution (<strong>BOS</strong>) Project requires QA services to be<br />

performed by PCG’s Independent Verification & Validation (IV&V) team<br />

throughout the entire project life cycle, including support of required federal<br />

reviews. The objective is to provide ongoing, interactive project management<br />

review and monitoring support, which will ensure that the <strong>BOS</strong> receives quality<br />

deliverables from the Design, Development, and Implementation (DD&I)<br />

vendor while achieving all critical project milestones.<br />

The QAP scope incorporates <strong>BOS</strong>’ Requirements Traceability Matrix (RTM) as a<br />

measure to validate Exchange requirements against the delivered system to<br />

help ensure that the solution complies with the required functionality. This<br />

includes reviewing, monitoring, and providing feedback on all applicable<br />

project deliverables and activities per the <strong>BOS</strong> accepted project plan.<br />

The QAP scope includes the review and evaluation of the testing aspects of the<br />

<strong>BOS</strong> project, which is detailed in the IV&V Test Management <strong>Plan</strong>.<br />

Also within the scope of the QAP is the process of identifying and reporting<br />

project risks, which is detailed in the IV&V Risk Management <strong>Plan</strong>.<br />

This document defines QA roles and responsibilities, standards, methods,<br />

metrics, and reporting requirements that shall be used on the project. The QA<br />

methodology described in this plan shall apply to all processes and deliverables<br />

for the full project lifecycle. Methods identified in the QAP includes applying<br />

quality review processes to the following activities:<br />

1. Project Management <strong>Plan</strong>ning<br />

2. Business Rules Development<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 1


Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

3. Requirements Validation<br />

4. Requirements Specifications<br />

5. Requirements Traceability Matrix<br />

6. Detailed System Design<br />

7. Software Development<br />

8. Test Management<br />

9. Training <strong>Plan</strong>ning, Delivery, and Materials<br />

10. Release Management<br />

11. Defect Management<br />

12. Configuration Management<br />

13. Change Management<br />

14. Implementation<br />

15. Maintenance and Operations<br />

The primary objectives of the QAP are:<br />

Discuss the QA organizational structure<br />

Define the QA roles and responsibilities<br />

Document the QA processes<br />

Identify QA performance measures and reporting standards<br />

2 Reference Documents<br />

This section provides a list of documents referenced in developing the QAP:<br />

<strong>BOS</strong> IV&V Communication <strong>Plan</strong><br />

<strong>BOS</strong> Verification and Validation <strong>Plan</strong><br />

Independent Verification and Validation RFP 1956<br />

<strong>BOS</strong> 5.4.3.4 <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

3 Standards and Guidelines<br />

This section provides a list of the standards and guidelines used in the<br />

development of this QAP, which will guide IV&V quality assurance activities:<br />

ANSI/IEEE Standard 730-2002 - Standard for Software <strong>Quality</strong> <strong>Plan</strong>s<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 2


Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

4 Glossary<br />

IEEE 828-2012 Standard for Configuration Management in Systems and<br />

Software Engineering<br />

IEEE 829-2008 Standard for Software and System Test Documentation<br />

IEEE 1012-2012 Standard for S/W Verification and Validation<br />

IEEE 1028-2008 Standard for Software Reviews and Audits<br />

IEEE 1061-1998 Standard for <strong>Quality</strong> Metrics Methodology<br />

IEEE 1490-2003 IEEE Guide Adoption of PMI Standard a Guide to the<br />

Project Management Body of Knowledge<br />

Project Management Institute’s Project Management Body of Knowledge<br />

(PMBOK®) Guide, 5th Edition.<br />

<strong>State</strong> Policy 4.100000 Section 4.7 Software Development and<br />

Maintenance<br />

Request for Proposal 1956 For Independent Verification And Validation<br />

Services (IV&V) Of The <strong>Health</strong> Care Reform Eligibility Engine Project<br />

CMS Enhanced Funding Requirements: Seven Conditions and Standards<br />

(MITS-11-01-v1.0)<br />

The following acronyms appear within the text of this QAP:<br />

Table 4-1: QA Acronyms and Definitions<br />

Acronym<br />

Definition<br />

<strong>BOS</strong><br />

CMS<br />

DED<br />

DD&I<br />

IEEE<br />

HCR-EE<br />

IV&V<br />

Business Operations Solution<br />

Centers for Medicare & Medicaid Services<br />

Deliverable Expectation Document<br />

Design, Develop, and Implement<br />

Institute of Electrical and Electronics Engineers<br />

<strong>Health</strong> Care Reform – Eligibility Engine Project<br />

Independent Validation and Verification<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 3


Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

Acronym<br />

PMBOK ®<br />

QA<br />

QAP<br />

RFP<br />

SME<br />

Definition<br />

Project Management Body of Knowledge<br />

<strong>Quality</strong> <strong>Assurance</strong><br />

<strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

Request for Proposal<br />

Subject Matter Expert<br />

5 Management<br />

This section describes the QA view of the project organization structure, its<br />

tasks, and its roles and responsibilities. It depicts the project management<br />

organizational structure that influences, monitors, and controls the quality of<br />

the solution. This includes a description of each major element of the<br />

organization, together with the roles and delegated responsibilities, the key<br />

interfaces within the management structure, and the QA responsibilities.<br />

5.1 Organization<br />

The following diagram depicts the organizational structure that controls the<br />

integrated quality control across the project.<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 4


Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

Project Sponsor<br />

and Steering<br />

Committee –<br />

Reviews quality<br />

plan and accepts<br />

quality reports<br />

DD&I Project<br />

Manager –<br />

Forwards/Fixes<br />

<strong>BOS</strong> Project<br />

Manager –<br />

Reviews and<br />

Approves<br />

IV&V Project<br />

Manager –<br />

Reviews,<br />

Comments, and<br />

Assists in<br />

Testing<br />

DD&I<br />

Implementation<br />

Manager –<br />

Reviews, Fixes,<br />

and Forwards<br />

Requirements<br />

Functional<br />

Specs<br />

Test <strong>Plan</strong>s and<br />

Test Results<br />

Testing Status<br />

and Defect<br />

Reports<br />

DD&I Technical<br />

Manager –<br />

Reviews and<br />

Forwards<br />

Technical Specs<br />

and Code<br />

<strong>BOS</strong> UAT Team<br />

– Performs<br />

acceptance<br />

Test, reports<br />

bugs<br />

Workflow<br />

DD&I Functional<br />

Team creates<br />

requirements<br />

and functional<br />

designs<br />

DD&I Testing<br />

Team<br />

DD&I Tecnnical<br />

Team – creates<br />

tech specs,<br />

codes, and fixes<br />

bugs<br />

QA Flow<br />

Figure 1 – <strong>Quality</strong> <strong>Assurance</strong> Organization and Flow<br />

In general, work products are produced by the DD&I project team. The <strong>BOS</strong><br />

Project Manager forwards the deliverable to the appropriate SMEs for review<br />

and comment. The <strong>BOS</strong> Project Manager consolidates the comments, which are<br />

distributed to the DD&I Vendor who then prepares a response that is<br />

distributed to the project team. The DD&I Vendor conducts a<br />

review/comments session where the team reviews the comments and<br />

responses. During the session, the <strong>State</strong> either rejects or approves the DD&I<br />

Vendor response and the DD&I Vendor prepares an updated version of the<br />

deliverable for <strong>BOS</strong>’ approval. The IV&V Vendor reviews each work product and<br />

comments on quality improvements. <strong>BOS</strong> management reviews the comments<br />

from the IV&V Vendor, accepts or rejects each comment, adds their own<br />

comments and returns to the DD&I project manager for response and product<br />

improvements.<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 5


5.2 Tasks<br />

Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

QAP activities provide both verbal and written feedback regarding all aspects of<br />

the project. This includes the following project planning tasks:<br />

1. Reviewing and providing written feedback on the DD&I Vendor’s Project<br />

Deliverables for the DD&I Vendor planning and project documents and<br />

artifacts relating to the QA process;<br />

2. Review of the DD&I Vendor’s project metrics used to monitor the project;<br />

and<br />

3. Review, monitor, and provide feedback on the <strong>BOS</strong> DD&I project<br />

deliverables relating to <strong>Quality</strong> <strong>Assurance</strong> prior to <strong>BOS</strong>’ approval of the<br />

DD&I deliverables<br />

5.3 Roles and Responsibilities<br />

Below are the specific roles that have essential responsibilities as part of the<br />

project’s QA effort.<br />

Table 5.3-1: QA Roles and Responsibilities<br />

Role<br />

<strong>BOS</strong> Project<br />

Manager<br />

DD&I Project<br />

Manager<br />

Responsibilities<br />

The <strong>BOS</strong> Project Manager is directly responsible for the day-to-day activities<br />

of the project. The Project Manager has the following primary responsibilities<br />

for overall quality assurance of the project, including:<br />

<br />

<br />

<br />

<br />

<br />

Develop and track project metrics.<br />

Oversee Vendor activities.<br />

Communicate regularly with the DD&I Project Manager and IV&V Vendor<br />

to ensure quality processes are being followed as per the approved QAP.<br />

Work with DD&I Project Manager to resolve quality issues in a timely<br />

fashion.<br />

Ensure that all QA processes are being followed by all team members.<br />

The DD&I Project Manager has overall responsibility for insuring that the<br />

DD&I project team upholds the Vendor’s internal quality standards and<br />

processes and, by their appropriate use, develops the project’s systems to<br />

meet all functional requirements. The DD&I Project Manager is responsible<br />

for ensuring the development team is familiar with applicable project quality<br />

standards and performs all work in compliance with those standards as<br />

delineated in the project’s approved QAP. It is anticipated that the DD&I<br />

Vendor will work cooperatively with the <strong>BOS</strong> Project Manager on quality<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 6


Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

Role<br />

Responsibilities<br />

related issues during all phases of the contract. This will include, but is not<br />

limited to, joint efforts of data gathering, metric analysis, risk management,<br />

and issue management including:<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

Develop and track project metrics.<br />

Oversee DD&I Vendor activities.<br />

Review DD&I Vendor deliverables.<br />

Oversee DD&I Vendor testing activities.<br />

Communicate regularly with the <strong>BOS</strong> Project Manager and IV&V Vendor<br />

to ensure quality processes are being followed as per the approved QAP.<br />

Work with <strong>BOS</strong> Project Manager to resolve quality issues in a timely<br />

fashion.<br />

Ensure that all QA processes are being followed by all team members.<br />

Ensures that all requirements are fully traced though all sections for the<br />

RTM and communicate in a timely manner any anticipated issues to the<br />

<strong>BOS</strong> Project Manager.<br />

DD&I <strong>Quality</strong> Lead The DD&I <strong>Quality</strong> Lead is responsible for managing the day-to-day quality<br />

assurance activities, including providing oversight to project processes and<br />

procedures, monitoring of various metrics, and the application of best<br />

practices and approved project standards. The <strong>Quality</strong> Manager is responsible<br />

for:<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

Identifying and escalating any critical project issues to the Project<br />

Manager.<br />

Work with the <strong>BOS</strong> Project team and the IV&V Vendor Team to identify<br />

and define project standards and metrics and baseline all quality<br />

measures, metrics, and acceptance criteria.<br />

Providing QA inputs for developing project work products.<br />

Providing oversight of project processes and procedures and providing<br />

evaluation reports related to standards compliance, process variances,<br />

and identifying process improvement opportunities.<br />

Auditing adherence to project standards on a periodic basis.<br />

Coordinating QA findings and mitigation strategies with the IV&V Vendor.<br />

Collecting and analyzing project metrics.<br />

Supporting issue and action item tracking and resolution.<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 7


Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

Role<br />

IV&V Project<br />

Manager<br />

Responsibilities<br />

<br />

<br />

<br />

<br />

<br />

Review DD&I Vendor deliverables and provide comments and<br />

recommendations.<br />

Provide written reports related to standards compliance, identification of<br />

process improvement opportunities, correctness, completeness,<br />

anomalies, and recommendations.<br />

Establish reporting standards that provide findings from quality<br />

measurements identifying areas where business, technical, and/or<br />

management quality objectives are or are not being met, or where trends<br />

in quality are moving in or out of control limits.<br />

Support requirements traceability planning.<br />

Provide oversight of testing activities by the Vendor and project testing.<br />

The IV&V Project Manager and team work under the direction of the <strong>BOS</strong><br />

Project Manager to provide Independent Validation and Verification (IV&V)<br />

services on the project. The IV&V will provide independent, technical review<br />

and verification of project deliverables, as well as independent testing and<br />

auditing of project deliverables against requirements, with a special emphasis<br />

placed on deliverable quality through the following activities:<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

Develop, maintain, and manage to the QAP<br />

Report performance measures and work with the <strong>BOS</strong> Project Manager<br />

and DD&I QA Lead to resolve QA issues in a timely manner<br />

Communicate as needed with DD&I Project Manager to ensure all<br />

deliverables meet expected quality standards (e.g. IEEE, PMBOK®)<br />

Ensure that all QA Processes are being followed by IV&V team members<br />

Provide documented findings and risks related to standards compliance,<br />

identification of process improvement opportunities, correctness,<br />

completeness, anomalies, and recommendations related to project and<br />

DD&I Vendor deliverables.<br />

Review the project reporting standards, which provide insight into the<br />

project’s quality measurements and assist in the identification of areas<br />

where business, technical, and/or management quality objectives are or<br />

are not being met, or where trends in quality are moving in or out of<br />

control limits.<br />

Coordinate with the DD&I QA Lead to help ensure quality objectives for<br />

the new system are satisfied, and pass quality reviews.<br />

Provide oversight and support (including independent verification as<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 8


Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

Role<br />

IV&V <strong>Quality</strong> Lead<br />

Responsibilities<br />

appropriate) of testing activities by the DD&I Vendor and project testing.<br />

The IV&V <strong>Quality</strong> lead is responsible for performing and managing the review<br />

efforts for deliverable reviews and assists as an internal SME for the IV&V<br />

team with regards to quality assurance measures and practices. The <strong>Quality</strong><br />

Lead is responsible for:<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

Provides weekly status to the IV&V team and attend meetings as<br />

appropriate.<br />

Attends project and status meetings as required.<br />

Verifies work products created from the PCG IV&V team, the <strong>State</strong>, and<br />

project vendors by reviewing deliverables for quality assurance<br />

throughout the Project lifecycle and report findings as per the approved<br />

Software Verification & Validation <strong>Plan</strong><br />

Helps to assure that the content and structure of the Testing<br />

documents/artifacts is produced and maintained.<br />

Helps to assure that standards and guidelines as defined by the<br />

organization are documented, implemented, monitored, and enforced<br />

regarding testing efforts.<br />

Escalates any issues concerning project requirements (software,<br />

hardware, resources, and processes) to the IV&V Project Manager.<br />

Provides communication with stakeholders and the project management<br />

team as needed and necessary.<br />

Provides document support by reviewing and commenting on the<br />

various plans and reports prepared by the project team.<br />

Prepare QA reports as required.<br />

Provides support, as needed, to the PCG IV&V Project Manager and<br />

team.<br />

6 <strong>Quality</strong> <strong>Assurance</strong> Approach<br />

Project <strong>Quality</strong> <strong>Assurance</strong> (QA) consists of two components, process quality,<br />

and product quality. Process and product quality objectives will be achieved by<br />

an integrated quality program consisting of Define <strong>Quality</strong>, Measure <strong>Quality</strong>,<br />

and Improve <strong>Quality</strong>. Figure 2 – <strong>Quality</strong> <strong>Assurance</strong> Process shows the<br />

relationship between quality assurance activities:<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 9


Process<br />

Product<br />

Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

<strong>Quality</strong> <strong>Assurance</strong><br />

<strong>Quality</strong> Management<br />

Define <strong>Quality</strong><br />

Improve <strong>Quality</strong><br />

Measure <strong>Quality</strong><br />

Figure 2 – <strong>Quality</strong> <strong>Assurance</strong> Process<br />

PCG defines quality by identifying and documenting quality criteria. The quality<br />

criteria consist of standards and metrics based on compliance with IEEE<br />

standards, <strong>State</strong> specific quality requirements, federal requirements, and best<br />

practices of the IV&V team. The establishment of quality criteria will set quality<br />

expectations for project processes and products. These criteria take the form of<br />

checklists that the IV&V team assesses against each process and artifact in the<br />

project (see Appendix A for a sample checklist).<br />

6.1 Process <strong>Quality</strong><br />

Process quality focuses on the processes used to create the project deliverables.<br />

In this project, process quality also includes the project planning deliverables.<br />

Process quality ensures the project’s policies and procedures are being adhered<br />

to by project participants. The goal of process quality is to enhance the<br />

predictability of process outcomes, both in terms of quality product and<br />

resource utilization required to complete the process.<br />

These measures will be both qualitative and quantitative. The qualitative<br />

measures compare current project processes against best practices for software<br />

development. Implementing review checklists allows the IV&V team to give an<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 10


Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

assessment of current project processes against best practices along with<br />

suggestions to better align project processes to best practices.<br />

Qualitative measures forms the metric portion of the assessment. As in the<br />

qualitative assessment, the quantitative assessment helps the IV&V Vendor<br />

assess the probability of an on-time completion of a project process. These are<br />

then used with a critical path analysis to assess the risk of late deliveries and the<br />

potential impact on the overall project schedule, budget and quality.<br />

6.2 Product <strong>Quality</strong><br />

Product <strong>Quality</strong> focuses on the project deliverables. Product <strong>Quality</strong> ensures the<br />

deliverables are of acceptable quality and that they are complete and correct in<br />

terms of: Conformance to requirements, technology effectiveness and<br />

customer satisfaction.<br />

Like Process <strong>Quality</strong>, Product <strong>Quality</strong> has both qualitative and quantitative<br />

portions. Some of the qualitative requirements relate to assessments of the<br />

product adherence to standards, satisfaction of requirements (just because the<br />

traceability matrix says a particular module satisfies a requirement or piece of<br />

code doesn’t necessarily make it so), and customer satisfaction.<br />

Quantitative assessments describe the characteristics of the product such as<br />

size, complexity, design features, performance, and quality level as well as<br />

defect inventorying, mean time to defect correction, system test cycle<br />

completion rates and mean time to complete cycles, etc.<br />

As in Process <strong>Quality</strong>, the IV&V Vendor will use Product <strong>Quality</strong> checklists based<br />

on standards. These will include the following:<br />

Documentation standards – Internally developed documents will conform<br />

to the <strong>State</strong>’s document management procedures<br />

Design and coding standards – Design and coding standards are the<br />

responsibility of the DD&I Vendor. Coding standards should include<br />

guidelines for commenting of software code and programs. The Vendor<br />

may be required to follow the coding standards specified by project to<br />

minimize the level of training when the system is transitioned back to<br />

project; and<br />

Testing standards and practices – Testing of all system components is<br />

primarily the responsibility of the DD&I Vendor with User Acceptance<br />

testing performed by <strong>State</strong> personnel and verified by the IV&V Vendor.<br />

All testing teams are expected to conduct testing based on testing<br />

standards as described in IEEE Standard 829 for each phase of testing.<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 11


Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

Test plans, test cases, procedures, scenarios, and results are expected to<br />

be developed and presented in a consistent manner and in accordance<br />

with the standard.<br />

6.3 <strong>Quality</strong> <strong>Assurance</strong> Processes<br />

The set of processes that will be used to validate both project Process <strong>Quality</strong><br />

and Product <strong>Quality</strong> are as follows.<br />

6.3.1 Deliverable Review Process<br />

A Deliverable Expectation Documents (DED) is required for all major<br />

deliverables. The purpose of the DED is to provide an outline, and a sample as<br />

necessary, of the deliverable prior to delivery to ensure that expectations are<br />

being met. Each DED is approved by the <strong>BOS</strong> Project Manager before delivery<br />

of the subject deliverable.<br />

Each deliverable is distributed to and reviewed by the <strong>BOS</strong> Project Manager,<br />

Project Team, and the IV&V Vendor, as appropriate. The IV&V Vendor review<br />

will follow the Product <strong>Quality</strong> Review process depicted in Figure 3:<br />

Inputs Process Output<br />

Project<br />

Artifact<br />

Project <strong>Quality</strong><br />

Review<br />

Product<br />

<strong>Quality</strong><br />

Observations<br />

DED<br />

QA Interviews<br />

Project<br />

Checklists<br />

RFP & SOW<br />

Resources<br />

<strong>State</strong> Guidelines<br />

Federal Guidelins<br />

IEEE Stanards<br />

Best Practices<br />

Figure 3 – Product <strong>Quality</strong> Review Process for Project Deliverables<br />

The <strong>BOS</strong> Project Manager returns to the DD&I Project Manager a list of<br />

quality defects in the deliverable through redline changes to the submitted<br />

draft deliverable (standard approach). The DD&I Project Manager responds to<br />

each comment and makes the <strong>State</strong> approved revisions to the deliverable and<br />

resubmits to the <strong>BOS</strong> Project Manager for approval. The <strong>BOS</strong> Project Manager<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 12


Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

approves the deliverable once the DD&I Vendor has provided a final draft<br />

deliverable that satisfies all the accepted and agreed upon requests for<br />

modification and additions. Deliverables which are periodically updated will<br />

follow the same process though in most cases reviews comments will be<br />

limited to changes/net new additions only.<br />

6.3.2 Requirements Tracking<br />

Part of ensuring the quality of the solution is demonstrating that it meets<br />

established requirements (functional, technical, etc.). The IV&V team will<br />

review and assess the processes used to identify, document, track and<br />

maintain project requirements based on the established guidelines and<br />

practices adopted by the <strong>BOS</strong> project that are documented in the project’s<br />

Requirements Traceability <strong>Plan</strong>. The <strong>BOS</strong> DD&I Vendor will track all the<br />

requirements provided in the RFP as well as document all modified and new<br />

requirements discovered during the course of the requirements confirmation<br />

and analysis. Requirements will be tracked in the RTM as required by the<br />

contract. The DD&I team will be responsible for keeping the information in<br />

this system updated for all functional, technical, and other requirements and<br />

providing reports to the <strong>BOS</strong> Project Manager and Project Team for each<br />

phase of the project.<br />

6.4 Testing Strategy<br />

To ensure the quality of the <strong>BOS</strong> solution and validate that requirements are<br />

met, several testing phases will be observed by the IV&V team that are defined<br />

in the Project Work <strong>Plan</strong> as well as the Test Management <strong>Plan</strong>. The IV&V team<br />

will evaluate and review testing strategies, processes, execution and project<br />

results throughout each testing phase and will report findings and risks to the<br />

project through the established risk management processes.<br />

Testing metrics will be managed by the project team and will include tracking<br />

details such as defects by severity, mean time to defect fix and retest, number<br />

of cycles executed and the outcome of those cycles.<br />

6.5 Continuous Process and Product <strong>Quality</strong> Improvement<br />

The project has established quality improvement strategies based on the value<br />

of each improvement with respect to the project’s objectives. Improvement<br />

strategies are determined on such measures as time to recover the cost of the<br />

improvement, improvement in project performance, and the project’s ability to<br />

respond to the changes. IV&V will assist in the development and<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 13


Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

implementation of selected quality improvements through the project’s<br />

established change control process.<br />

7 QAP Change Procedure<br />

This document will be reviewed and updated as needed, as the project<br />

proceeds through each phase of the system development life cycle. Lessons<br />

learned as a result of continuing staff management efforts will be captured at<br />

the end of each project phase and used to improve the project.<br />

This document contains a revision history log. When changes occur, the<br />

version number will be updated to the next increment and the date, owner<br />

making the change, and change description will be recorded in the revision<br />

history log of the document.<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 14


Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

8 Appendix A – Sample Product <strong>Quality</strong> Review Checklist<br />

Detailed Design Checklist<br />

Overall General Assessment/Suggestions (If Appropriate)<br />

1.<br />

2.<br />

ID <strong>Quality</strong> Checklist Item Standard Measurement<br />

Met<br />

Yes<br />

No<br />

Partial<br />

Comments/Recommendations<br />

1 The document should address<br />

how business requirements are<br />

addressed<br />

2 The document should address<br />

how the technical requirements<br />

will be addressed<br />

3 The document should address<br />

what the workflow is for<br />

escalating issues<br />

4 The document should include the<br />

protocols used by the application<br />

5 All data formats should be<br />

outlined in the document<br />

6 The document should include a<br />

full architectural schema<br />

The Vendor's Detailed Design<br />

document should either list out all<br />

business requirements or should<br />

reference an external document where<br />

the business requirements are<br />

addressed.<br />

Additionally, the Detailed Design<br />

document should clearly outline how<br />

the solution addresses specific business<br />

requirements.<br />

The Detailed Design document should<br />

either list out all technical requirements<br />

or should reference an external<br />

document where the technical<br />

requirements are addressed.<br />

Additionally, this document should<br />

clearly outline how the solution<br />

addresses specific technical<br />

requirements.<br />

If there is an outage or issue with the<br />

application, the document should<br />

address the workflow for resolving<br />

issues<br />

The document should include all<br />

protocols used by the application<br />

Data formats between different,<br />

interfaced systems, should be outlined<br />

within the document<br />

The full architectural schema should be<br />

included in the document<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 15


Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />

<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />

August 30, 2013<br />

ID <strong>Quality</strong> Checklist Item Standard Measurement<br />

Met<br />

Yes<br />

No<br />

Partial<br />

Comments/Recommendations<br />

7 The document should address<br />

how the business rules will be<br />

addressed<br />

8 The document should have a data<br />

flow schema<br />

9 User roles should be included in<br />

the document<br />

10 All Project phases should be<br />

documented<br />

11 The document should include the<br />

deployment scenario<br />

The Detailed Design document should<br />

either list out all business rules or<br />

should reference an external document<br />

where the business rules are<br />

addressed.<br />

Additionally, the Detailed Design<br />

document should clearly outline how<br />

the solution addresses specific business<br />

rules.<br />

How data is pulled into the application<br />

via external interfaces and how that<br />

data is then presented to end users<br />

should be included in the document as<br />

a schema<br />

The document should outline all user<br />

roles that will be implemented<br />

Project phases and sub-phases should<br />

be documented including the goals,<br />

resource and time estimates for each<br />

phase<br />

Solution deployment during go-live<br />

specific to cutover activities should be<br />

defined in the document<br />

5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 16

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!