BOS IV&V Quality Assurance Plan Final - Silver State Health ...
BOS IV&V Quality Assurance Plan Final - Silver State Health ...
BOS IV&V Quality Assurance Plan Final - Silver State Health ...
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
<strong>State</strong> of Nevada<br />
Business Operations Solutions (<strong>BOS</strong>)<br />
Independent Verification and Validation<br />
Services (IV&V) of the <strong>Silver</strong> <strong>State</strong> <strong>Health</strong><br />
Insurance Exchange (Exchange)<br />
<strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
Deliverable 5.4.3.6 - Activity 5.4.2.7<br />
August 30, 2013
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Insurance Exchange Project<br />
IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
Version History<br />
VERSION DATE COMMENTS<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> -<br />
Draft v1<br />
5.4.3.6 - <strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong><br />
<strong>Plan</strong> <strong>Final</strong>.doc<br />
08/30/2013 Initial Draft, K. Marshall<br />
9/06/2013 <strong>BOS</strong> revisions, P. Palmer<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page i
Table of Contents<br />
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Insurance Exchange Project<br />
IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
1 PURPOSE ................................................................................................................................... 1<br />
1.1 Scope and Objectives .................................................................................................................. 1<br />
2 REFERENCE DOCUMENTS ........................................................................................................... 2<br />
3 STANDARDS AND GUIDELINES .................................................................................................... 2<br />
4 GLOSSARY .................................................................................................................................. 3<br />
5 MANAGEMENT .......................................................................................................................... 4<br />
5.1 Organization ................................................................................................................................ 4<br />
5.2 Tasks ............................................................................................................................................ 6<br />
5.3 Roles and Responsibilities ........................................................................................................... 6<br />
6 QUALITY ASSURANCE APPROACH ............................................................................................... 9<br />
6.1 Process <strong>Quality</strong> .......................................................................................................................... 10<br />
6.2 Product <strong>Quality</strong> .......................................................................................................................... 11<br />
6.3 <strong>Quality</strong> <strong>Assurance</strong> Processes ..................................................................................................... 12<br />
6.3.1 Deliverable Review Process .......................................................................................... 12<br />
6.3.2 Requirements Tracking ................................................................................................. 13<br />
6.4 Testing Strategy ......................................................................................................................... 13<br />
6.5 Continuous Process and Product <strong>Quality</strong> Improvement ........................................................... 13<br />
7 QAP CHANGE PROCEDURE ....................................................................................................... 14<br />
8 APPENDIX A – SAMPLE PRODUCT QUALITY REVIEW CHECKLIST ................................................. 15<br />
Table of Figures<br />
Figure 1 – <strong>Quality</strong> <strong>Assurance</strong> Organization and Flow ................................................................................... 5<br />
Figure 2 – <strong>Quality</strong> <strong>Assurance</strong> Process ......................................................................................................... 10<br />
Figure 3 – Product <strong>Quality</strong> Review Process for Project Deliverables .......................................................... 12<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page ii
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
1 Purpose<br />
The purpose of the <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> (QAP) is to describe Public<br />
Consulting Group’s (PCG’s) approach to ensuring quality on the project. It<br />
includes assessing the methodology for maintaining quality of the code and<br />
workmanship, managing project schedules, and related subcontractor activities<br />
where appropriate. This plan describes the methods that will be used both to<br />
assess the quality of the work products produced and the quality of the<br />
processes used to produce those products and identifies the activities,<br />
processes, and procedures that will be used to assess and monitor quality. The<br />
QAP also identifies the roles and responsibilities specific to the <strong>Quality</strong><br />
<strong>Assurance</strong> (QA) function needed to successfully implement best practices.<br />
1.1 Scope and Objectives<br />
The Business Operations Solution (<strong>BOS</strong>) Project requires QA services to be<br />
performed by PCG’s Independent Verification & Validation (IV&V) team<br />
throughout the entire project life cycle, including support of required federal<br />
reviews. The objective is to provide ongoing, interactive project management<br />
review and monitoring support, which will ensure that the <strong>BOS</strong> receives quality<br />
deliverables from the Design, Development, and Implementation (DD&I)<br />
vendor while achieving all critical project milestones.<br />
The QAP scope incorporates <strong>BOS</strong>’ Requirements Traceability Matrix (RTM) as a<br />
measure to validate Exchange requirements against the delivered system to<br />
help ensure that the solution complies with the required functionality. This<br />
includes reviewing, monitoring, and providing feedback on all applicable<br />
project deliverables and activities per the <strong>BOS</strong> accepted project plan.<br />
The QAP scope includes the review and evaluation of the testing aspects of the<br />
<strong>BOS</strong> project, which is detailed in the IV&V Test Management <strong>Plan</strong>.<br />
Also within the scope of the QAP is the process of identifying and reporting<br />
project risks, which is detailed in the IV&V Risk Management <strong>Plan</strong>.<br />
This document defines QA roles and responsibilities, standards, methods,<br />
metrics, and reporting requirements that shall be used on the project. The QA<br />
methodology described in this plan shall apply to all processes and deliverables<br />
for the full project lifecycle. Methods identified in the QAP includes applying<br />
quality review processes to the following activities:<br />
1. Project Management <strong>Plan</strong>ning<br />
2. Business Rules Development<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 1
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
3. Requirements Validation<br />
4. Requirements Specifications<br />
5. Requirements Traceability Matrix<br />
6. Detailed System Design<br />
7. Software Development<br />
8. Test Management<br />
9. Training <strong>Plan</strong>ning, Delivery, and Materials<br />
10. Release Management<br />
11. Defect Management<br />
12. Configuration Management<br />
13. Change Management<br />
14. Implementation<br />
15. Maintenance and Operations<br />
The primary objectives of the QAP are:<br />
Discuss the QA organizational structure<br />
Define the QA roles and responsibilities<br />
Document the QA processes<br />
Identify QA performance measures and reporting standards<br />
2 Reference Documents<br />
This section provides a list of documents referenced in developing the QAP:<br />
<strong>BOS</strong> IV&V Communication <strong>Plan</strong><br />
<strong>BOS</strong> Verification and Validation <strong>Plan</strong><br />
Independent Verification and Validation RFP 1956<br />
<strong>BOS</strong> 5.4.3.4 <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
3 Standards and Guidelines<br />
This section provides a list of the standards and guidelines used in the<br />
development of this QAP, which will guide IV&V quality assurance activities:<br />
ANSI/IEEE Standard 730-2002 - Standard for Software <strong>Quality</strong> <strong>Plan</strong>s<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 2
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
4 Glossary<br />
IEEE 828-2012 Standard for Configuration Management in Systems and<br />
Software Engineering<br />
IEEE 829-2008 Standard for Software and System Test Documentation<br />
IEEE 1012-2012 Standard for S/W Verification and Validation<br />
IEEE 1028-2008 Standard for Software Reviews and Audits<br />
IEEE 1061-1998 Standard for <strong>Quality</strong> Metrics Methodology<br />
IEEE 1490-2003 IEEE Guide Adoption of PMI Standard a Guide to the<br />
Project Management Body of Knowledge<br />
Project Management Institute’s Project Management Body of Knowledge<br />
(PMBOK®) Guide, 5th Edition.<br />
<strong>State</strong> Policy 4.100000 Section 4.7 Software Development and<br />
Maintenance<br />
Request for Proposal 1956 For Independent Verification And Validation<br />
Services (IV&V) Of The <strong>Health</strong> Care Reform Eligibility Engine Project<br />
CMS Enhanced Funding Requirements: Seven Conditions and Standards<br />
(MITS-11-01-v1.0)<br />
The following acronyms appear within the text of this QAP:<br />
Table 4-1: QA Acronyms and Definitions<br />
Acronym<br />
Definition<br />
<strong>BOS</strong><br />
CMS<br />
DED<br />
DD&I<br />
IEEE<br />
HCR-EE<br />
IV&V<br />
Business Operations Solution<br />
Centers for Medicare & Medicaid Services<br />
Deliverable Expectation Document<br />
Design, Develop, and Implement<br />
Institute of Electrical and Electronics Engineers<br />
<strong>Health</strong> Care Reform – Eligibility Engine Project<br />
Independent Validation and Verification<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 3
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
Acronym<br />
PMBOK ®<br />
QA<br />
QAP<br />
RFP<br />
SME<br />
Definition<br />
Project Management Body of Knowledge<br />
<strong>Quality</strong> <strong>Assurance</strong><br />
<strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
Request for Proposal<br />
Subject Matter Expert<br />
5 Management<br />
This section describes the QA view of the project organization structure, its<br />
tasks, and its roles and responsibilities. It depicts the project management<br />
organizational structure that influences, monitors, and controls the quality of<br />
the solution. This includes a description of each major element of the<br />
organization, together with the roles and delegated responsibilities, the key<br />
interfaces within the management structure, and the QA responsibilities.<br />
5.1 Organization<br />
The following diagram depicts the organizational structure that controls the<br />
integrated quality control across the project.<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 4
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
Project Sponsor<br />
and Steering<br />
Committee –<br />
Reviews quality<br />
plan and accepts<br />
quality reports<br />
DD&I Project<br />
Manager –<br />
Forwards/Fixes<br />
<strong>BOS</strong> Project<br />
Manager –<br />
Reviews and<br />
Approves<br />
IV&V Project<br />
Manager –<br />
Reviews,<br />
Comments, and<br />
Assists in<br />
Testing<br />
DD&I<br />
Implementation<br />
Manager –<br />
Reviews, Fixes,<br />
and Forwards<br />
Requirements<br />
Functional<br />
Specs<br />
Test <strong>Plan</strong>s and<br />
Test Results<br />
Testing Status<br />
and Defect<br />
Reports<br />
DD&I Technical<br />
Manager –<br />
Reviews and<br />
Forwards<br />
Technical Specs<br />
and Code<br />
<strong>BOS</strong> UAT Team<br />
– Performs<br />
acceptance<br />
Test, reports<br />
bugs<br />
Workflow<br />
DD&I Functional<br />
Team creates<br />
requirements<br />
and functional<br />
designs<br />
DD&I Testing<br />
Team<br />
DD&I Tecnnical<br />
Team – creates<br />
tech specs,<br />
codes, and fixes<br />
bugs<br />
QA Flow<br />
Figure 1 – <strong>Quality</strong> <strong>Assurance</strong> Organization and Flow<br />
In general, work products are produced by the DD&I project team. The <strong>BOS</strong><br />
Project Manager forwards the deliverable to the appropriate SMEs for review<br />
and comment. The <strong>BOS</strong> Project Manager consolidates the comments, which are<br />
distributed to the DD&I Vendor who then prepares a response that is<br />
distributed to the project team. The DD&I Vendor conducts a<br />
review/comments session where the team reviews the comments and<br />
responses. During the session, the <strong>State</strong> either rejects or approves the DD&I<br />
Vendor response and the DD&I Vendor prepares an updated version of the<br />
deliverable for <strong>BOS</strong>’ approval. The IV&V Vendor reviews each work product and<br />
comments on quality improvements. <strong>BOS</strong> management reviews the comments<br />
from the IV&V Vendor, accepts or rejects each comment, adds their own<br />
comments and returns to the DD&I project manager for response and product<br />
improvements.<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 5
5.2 Tasks<br />
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
QAP activities provide both verbal and written feedback regarding all aspects of<br />
the project. This includes the following project planning tasks:<br />
1. Reviewing and providing written feedback on the DD&I Vendor’s Project<br />
Deliverables for the DD&I Vendor planning and project documents and<br />
artifacts relating to the QA process;<br />
2. Review of the DD&I Vendor’s project metrics used to monitor the project;<br />
and<br />
3. Review, monitor, and provide feedback on the <strong>BOS</strong> DD&I project<br />
deliverables relating to <strong>Quality</strong> <strong>Assurance</strong> prior to <strong>BOS</strong>’ approval of the<br />
DD&I deliverables<br />
5.3 Roles and Responsibilities<br />
Below are the specific roles that have essential responsibilities as part of the<br />
project’s QA effort.<br />
Table 5.3-1: QA Roles and Responsibilities<br />
Role<br />
<strong>BOS</strong> Project<br />
Manager<br />
DD&I Project<br />
Manager<br />
Responsibilities<br />
The <strong>BOS</strong> Project Manager is directly responsible for the day-to-day activities<br />
of the project. The Project Manager has the following primary responsibilities<br />
for overall quality assurance of the project, including:<br />
<br />
<br />
<br />
<br />
<br />
Develop and track project metrics.<br />
Oversee Vendor activities.<br />
Communicate regularly with the DD&I Project Manager and IV&V Vendor<br />
to ensure quality processes are being followed as per the approved QAP.<br />
Work with DD&I Project Manager to resolve quality issues in a timely<br />
fashion.<br />
Ensure that all QA processes are being followed by all team members.<br />
The DD&I Project Manager has overall responsibility for insuring that the<br />
DD&I project team upholds the Vendor’s internal quality standards and<br />
processes and, by their appropriate use, develops the project’s systems to<br />
meet all functional requirements. The DD&I Project Manager is responsible<br />
for ensuring the development team is familiar with applicable project quality<br />
standards and performs all work in compliance with those standards as<br />
delineated in the project’s approved QAP. It is anticipated that the DD&I<br />
Vendor will work cooperatively with the <strong>BOS</strong> Project Manager on quality<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 6
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
Role<br />
Responsibilities<br />
related issues during all phases of the contract. This will include, but is not<br />
limited to, joint efforts of data gathering, metric analysis, risk management,<br />
and issue management including:<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Develop and track project metrics.<br />
Oversee DD&I Vendor activities.<br />
Review DD&I Vendor deliverables.<br />
Oversee DD&I Vendor testing activities.<br />
Communicate regularly with the <strong>BOS</strong> Project Manager and IV&V Vendor<br />
to ensure quality processes are being followed as per the approved QAP.<br />
Work with <strong>BOS</strong> Project Manager to resolve quality issues in a timely<br />
fashion.<br />
Ensure that all QA processes are being followed by all team members.<br />
Ensures that all requirements are fully traced though all sections for the<br />
RTM and communicate in a timely manner any anticipated issues to the<br />
<strong>BOS</strong> Project Manager.<br />
DD&I <strong>Quality</strong> Lead The DD&I <strong>Quality</strong> Lead is responsible for managing the day-to-day quality<br />
assurance activities, including providing oversight to project processes and<br />
procedures, monitoring of various metrics, and the application of best<br />
practices and approved project standards. The <strong>Quality</strong> Manager is responsible<br />
for:<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Identifying and escalating any critical project issues to the Project<br />
Manager.<br />
Work with the <strong>BOS</strong> Project team and the IV&V Vendor Team to identify<br />
and define project standards and metrics and baseline all quality<br />
measures, metrics, and acceptance criteria.<br />
Providing QA inputs for developing project work products.<br />
Providing oversight of project processes and procedures and providing<br />
evaluation reports related to standards compliance, process variances,<br />
and identifying process improvement opportunities.<br />
Auditing adherence to project standards on a periodic basis.<br />
Coordinating QA findings and mitigation strategies with the IV&V Vendor.<br />
Collecting and analyzing project metrics.<br />
Supporting issue and action item tracking and resolution.<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 7
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
Role<br />
IV&V Project<br />
Manager<br />
Responsibilities<br />
<br />
<br />
<br />
<br />
<br />
Review DD&I Vendor deliverables and provide comments and<br />
recommendations.<br />
Provide written reports related to standards compliance, identification of<br />
process improvement opportunities, correctness, completeness,<br />
anomalies, and recommendations.<br />
Establish reporting standards that provide findings from quality<br />
measurements identifying areas where business, technical, and/or<br />
management quality objectives are or are not being met, or where trends<br />
in quality are moving in or out of control limits.<br />
Support requirements traceability planning.<br />
Provide oversight of testing activities by the Vendor and project testing.<br />
The IV&V Project Manager and team work under the direction of the <strong>BOS</strong><br />
Project Manager to provide Independent Validation and Verification (IV&V)<br />
services on the project. The IV&V will provide independent, technical review<br />
and verification of project deliverables, as well as independent testing and<br />
auditing of project deliverables against requirements, with a special emphasis<br />
placed on deliverable quality through the following activities:<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Develop, maintain, and manage to the QAP<br />
Report performance measures and work with the <strong>BOS</strong> Project Manager<br />
and DD&I QA Lead to resolve QA issues in a timely manner<br />
Communicate as needed with DD&I Project Manager to ensure all<br />
deliverables meet expected quality standards (e.g. IEEE, PMBOK®)<br />
Ensure that all QA Processes are being followed by IV&V team members<br />
Provide documented findings and risks related to standards compliance,<br />
identification of process improvement opportunities, correctness,<br />
completeness, anomalies, and recommendations related to project and<br />
DD&I Vendor deliverables.<br />
Review the project reporting standards, which provide insight into the<br />
project’s quality measurements and assist in the identification of areas<br />
where business, technical, and/or management quality objectives are or<br />
are not being met, or where trends in quality are moving in or out of<br />
control limits.<br />
Coordinate with the DD&I QA Lead to help ensure quality objectives for<br />
the new system are satisfied, and pass quality reviews.<br />
Provide oversight and support (including independent verification as<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 8
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
Role<br />
IV&V <strong>Quality</strong> Lead<br />
Responsibilities<br />
appropriate) of testing activities by the DD&I Vendor and project testing.<br />
The IV&V <strong>Quality</strong> lead is responsible for performing and managing the review<br />
efforts for deliverable reviews and assists as an internal SME for the IV&V<br />
team with regards to quality assurance measures and practices. The <strong>Quality</strong><br />
Lead is responsible for:<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Provides weekly status to the IV&V team and attend meetings as<br />
appropriate.<br />
Attends project and status meetings as required.<br />
Verifies work products created from the PCG IV&V team, the <strong>State</strong>, and<br />
project vendors by reviewing deliverables for quality assurance<br />
throughout the Project lifecycle and report findings as per the approved<br />
Software Verification & Validation <strong>Plan</strong><br />
Helps to assure that the content and structure of the Testing<br />
documents/artifacts is produced and maintained.<br />
Helps to assure that standards and guidelines as defined by the<br />
organization are documented, implemented, monitored, and enforced<br />
regarding testing efforts.<br />
Escalates any issues concerning project requirements (software,<br />
hardware, resources, and processes) to the IV&V Project Manager.<br />
Provides communication with stakeholders and the project management<br />
team as needed and necessary.<br />
Provides document support by reviewing and commenting on the<br />
various plans and reports prepared by the project team.<br />
Prepare QA reports as required.<br />
Provides support, as needed, to the PCG IV&V Project Manager and<br />
team.<br />
6 <strong>Quality</strong> <strong>Assurance</strong> Approach<br />
Project <strong>Quality</strong> <strong>Assurance</strong> (QA) consists of two components, process quality,<br />
and product quality. Process and product quality objectives will be achieved by<br />
an integrated quality program consisting of Define <strong>Quality</strong>, Measure <strong>Quality</strong>,<br />
and Improve <strong>Quality</strong>. Figure 2 – <strong>Quality</strong> <strong>Assurance</strong> Process shows the<br />
relationship between quality assurance activities:<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 9
Process<br />
Product<br />
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
<strong>Quality</strong> <strong>Assurance</strong><br />
<strong>Quality</strong> Management<br />
Define <strong>Quality</strong><br />
Improve <strong>Quality</strong><br />
Measure <strong>Quality</strong><br />
Figure 2 – <strong>Quality</strong> <strong>Assurance</strong> Process<br />
PCG defines quality by identifying and documenting quality criteria. The quality<br />
criteria consist of standards and metrics based on compliance with IEEE<br />
standards, <strong>State</strong> specific quality requirements, federal requirements, and best<br />
practices of the IV&V team. The establishment of quality criteria will set quality<br />
expectations for project processes and products. These criteria take the form of<br />
checklists that the IV&V team assesses against each process and artifact in the<br />
project (see Appendix A for a sample checklist).<br />
6.1 Process <strong>Quality</strong><br />
Process quality focuses on the processes used to create the project deliverables.<br />
In this project, process quality also includes the project planning deliverables.<br />
Process quality ensures the project’s policies and procedures are being adhered<br />
to by project participants. The goal of process quality is to enhance the<br />
predictability of process outcomes, both in terms of quality product and<br />
resource utilization required to complete the process.<br />
These measures will be both qualitative and quantitative. The qualitative<br />
measures compare current project processes against best practices for software<br />
development. Implementing review checklists allows the IV&V team to give an<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 10
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
assessment of current project processes against best practices along with<br />
suggestions to better align project processes to best practices.<br />
Qualitative measures forms the metric portion of the assessment. As in the<br />
qualitative assessment, the quantitative assessment helps the IV&V Vendor<br />
assess the probability of an on-time completion of a project process. These are<br />
then used with a critical path analysis to assess the risk of late deliveries and the<br />
potential impact on the overall project schedule, budget and quality.<br />
6.2 Product <strong>Quality</strong><br />
Product <strong>Quality</strong> focuses on the project deliverables. Product <strong>Quality</strong> ensures the<br />
deliverables are of acceptable quality and that they are complete and correct in<br />
terms of: Conformance to requirements, technology effectiveness and<br />
customer satisfaction.<br />
Like Process <strong>Quality</strong>, Product <strong>Quality</strong> has both qualitative and quantitative<br />
portions. Some of the qualitative requirements relate to assessments of the<br />
product adherence to standards, satisfaction of requirements (just because the<br />
traceability matrix says a particular module satisfies a requirement or piece of<br />
code doesn’t necessarily make it so), and customer satisfaction.<br />
Quantitative assessments describe the characteristics of the product such as<br />
size, complexity, design features, performance, and quality level as well as<br />
defect inventorying, mean time to defect correction, system test cycle<br />
completion rates and mean time to complete cycles, etc.<br />
As in Process <strong>Quality</strong>, the IV&V Vendor will use Product <strong>Quality</strong> checklists based<br />
on standards. These will include the following:<br />
Documentation standards – Internally developed documents will conform<br />
to the <strong>State</strong>’s document management procedures<br />
Design and coding standards – Design and coding standards are the<br />
responsibility of the DD&I Vendor. Coding standards should include<br />
guidelines for commenting of software code and programs. The Vendor<br />
may be required to follow the coding standards specified by project to<br />
minimize the level of training when the system is transitioned back to<br />
project; and<br />
Testing standards and practices – Testing of all system components is<br />
primarily the responsibility of the DD&I Vendor with User Acceptance<br />
testing performed by <strong>State</strong> personnel and verified by the IV&V Vendor.<br />
All testing teams are expected to conduct testing based on testing<br />
standards as described in IEEE Standard 829 for each phase of testing.<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 11
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
Test plans, test cases, procedures, scenarios, and results are expected to<br />
be developed and presented in a consistent manner and in accordance<br />
with the standard.<br />
6.3 <strong>Quality</strong> <strong>Assurance</strong> Processes<br />
The set of processes that will be used to validate both project Process <strong>Quality</strong><br />
and Product <strong>Quality</strong> are as follows.<br />
6.3.1 Deliverable Review Process<br />
A Deliverable Expectation Documents (DED) is required for all major<br />
deliverables. The purpose of the DED is to provide an outline, and a sample as<br />
necessary, of the deliverable prior to delivery to ensure that expectations are<br />
being met. Each DED is approved by the <strong>BOS</strong> Project Manager before delivery<br />
of the subject deliverable.<br />
Each deliverable is distributed to and reviewed by the <strong>BOS</strong> Project Manager,<br />
Project Team, and the IV&V Vendor, as appropriate. The IV&V Vendor review<br />
will follow the Product <strong>Quality</strong> Review process depicted in Figure 3:<br />
Inputs Process Output<br />
Project<br />
Artifact<br />
Project <strong>Quality</strong><br />
Review<br />
Product<br />
<strong>Quality</strong><br />
Observations<br />
DED<br />
QA Interviews<br />
Project<br />
Checklists<br />
RFP & SOW<br />
Resources<br />
<strong>State</strong> Guidelines<br />
Federal Guidelins<br />
IEEE Stanards<br />
Best Practices<br />
Figure 3 – Product <strong>Quality</strong> Review Process for Project Deliverables<br />
The <strong>BOS</strong> Project Manager returns to the DD&I Project Manager a list of<br />
quality defects in the deliverable through redline changes to the submitted<br />
draft deliverable (standard approach). The DD&I Project Manager responds to<br />
each comment and makes the <strong>State</strong> approved revisions to the deliverable and<br />
resubmits to the <strong>BOS</strong> Project Manager for approval. The <strong>BOS</strong> Project Manager<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 12
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
approves the deliverable once the DD&I Vendor has provided a final draft<br />
deliverable that satisfies all the accepted and agreed upon requests for<br />
modification and additions. Deliverables which are periodically updated will<br />
follow the same process though in most cases reviews comments will be<br />
limited to changes/net new additions only.<br />
6.3.2 Requirements Tracking<br />
Part of ensuring the quality of the solution is demonstrating that it meets<br />
established requirements (functional, technical, etc.). The IV&V team will<br />
review and assess the processes used to identify, document, track and<br />
maintain project requirements based on the established guidelines and<br />
practices adopted by the <strong>BOS</strong> project that are documented in the project’s<br />
Requirements Traceability <strong>Plan</strong>. The <strong>BOS</strong> DD&I Vendor will track all the<br />
requirements provided in the RFP as well as document all modified and new<br />
requirements discovered during the course of the requirements confirmation<br />
and analysis. Requirements will be tracked in the RTM as required by the<br />
contract. The DD&I team will be responsible for keeping the information in<br />
this system updated for all functional, technical, and other requirements and<br />
providing reports to the <strong>BOS</strong> Project Manager and Project Team for each<br />
phase of the project.<br />
6.4 Testing Strategy<br />
To ensure the quality of the <strong>BOS</strong> solution and validate that requirements are<br />
met, several testing phases will be observed by the IV&V team that are defined<br />
in the Project Work <strong>Plan</strong> as well as the Test Management <strong>Plan</strong>. The IV&V team<br />
will evaluate and review testing strategies, processes, execution and project<br />
results throughout each testing phase and will report findings and risks to the<br />
project through the established risk management processes.<br />
Testing metrics will be managed by the project team and will include tracking<br />
details such as defects by severity, mean time to defect fix and retest, number<br />
of cycles executed and the outcome of those cycles.<br />
6.5 Continuous Process and Product <strong>Quality</strong> Improvement<br />
The project has established quality improvement strategies based on the value<br />
of each improvement with respect to the project’s objectives. Improvement<br />
strategies are determined on such measures as time to recover the cost of the<br />
improvement, improvement in project performance, and the project’s ability to<br />
respond to the changes. IV&V will assist in the development and<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 13
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
implementation of selected quality improvements through the project’s<br />
established change control process.<br />
7 QAP Change Procedure<br />
This document will be reviewed and updated as needed, as the project<br />
proceeds through each phase of the system development life cycle. Lessons<br />
learned as a result of continuing staff management efforts will be captured at<br />
the end of each project phase and used to improve the project.<br />
This document contains a revision history log. When changes occur, the<br />
version number will be updated to the next increment and the date, owner<br />
making the change, and change description will be recorded in the revision<br />
history log of the document.<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 14
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
8 Appendix A – Sample Product <strong>Quality</strong> Review Checklist<br />
Detailed Design Checklist<br />
Overall General Assessment/Suggestions (If Appropriate)<br />
1.<br />
2.<br />
ID <strong>Quality</strong> Checklist Item Standard Measurement<br />
Met<br />
Yes<br />
No<br />
Partial<br />
Comments/Recommendations<br />
1 The document should address<br />
how business requirements are<br />
addressed<br />
2 The document should address<br />
how the technical requirements<br />
will be addressed<br />
3 The document should address<br />
what the workflow is for<br />
escalating issues<br />
4 The document should include the<br />
protocols used by the application<br />
5 All data formats should be<br />
outlined in the document<br />
6 The document should include a<br />
full architectural schema<br />
The Vendor's Detailed Design<br />
document should either list out all<br />
business requirements or should<br />
reference an external document where<br />
the business requirements are<br />
addressed.<br />
Additionally, the Detailed Design<br />
document should clearly outline how<br />
the solution addresses specific business<br />
requirements.<br />
The Detailed Design document should<br />
either list out all technical requirements<br />
or should reference an external<br />
document where the technical<br />
requirements are addressed.<br />
Additionally, this document should<br />
clearly outline how the solution<br />
addresses specific technical<br />
requirements.<br />
If there is an outage or issue with the<br />
application, the document should<br />
address the workflow for resolving<br />
issues<br />
The document should include all<br />
protocols used by the application<br />
Data formats between different,<br />
interfaced systems, should be outlined<br />
within the document<br />
The full architectural schema should be<br />
included in the document<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 15
Nevada <strong>Silver</strong> <strong>State</strong> <strong>Health</strong> Exchange Project<br />
<strong>BOS</strong> IV&V <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong><br />
August 30, 2013<br />
ID <strong>Quality</strong> Checklist Item Standard Measurement<br />
Met<br />
Yes<br />
No<br />
Partial<br />
Comments/Recommendations<br />
7 The document should address<br />
how the business rules will be<br />
addressed<br />
8 The document should have a data<br />
flow schema<br />
9 User roles should be included in<br />
the document<br />
10 All Project phases should be<br />
documented<br />
11 The document should include the<br />
deployment scenario<br />
The Detailed Design document should<br />
either list out all business rules or<br />
should reference an external document<br />
where the business rules are<br />
addressed.<br />
Additionally, the Detailed Design<br />
document should clearly outline how<br />
the solution addresses specific business<br />
rules.<br />
How data is pulled into the application<br />
via external interfaces and how that<br />
data is then presented to end users<br />
should be included in the document as<br />
a schema<br />
The document should outline all user<br />
roles that will be implemented<br />
Project phases and sub-phases should<br />
be documented including the goals,<br />
resource and time estimates for each<br />
phase<br />
Solution deployment during go-live<br />
specific to cutover activities should be<br />
defined in the document<br />
5 4 3 6 - <strong>BOS</strong> IVV <strong>Quality</strong> <strong>Assurance</strong> <strong>Plan</strong> Page 16