Test - TTCN-3
Test - TTCN-3
Test - TTCN-3
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
Security Level: Confidential<br />
Ideal <strong>Test</strong> Systems & Target<br />
Development <strong>Test</strong> Process<br />
Tony Chang<br />
June 9, 2010<br />
www.huawei.com<br />
HUAWEI TECHNOLOGIES CO., LTD.<br />
Huawei Confidential
Contents<br />
“<strong>Test</strong>” Fundamental Responsibilities & Activities<br />
Product Delivery <strong>Test</strong> Quality Indicators<br />
Product <strong>Test</strong> Coverage<br />
Ideal <strong>Test</strong> Systems<br />
Target Development <strong>Test</strong> Process<br />
‣ Pre-<strong>Test</strong>ing Preparation<br />
‣ Development <strong>Test</strong>ing<br />
‣ Continuous Integration<br />
‣ DTTP – Design, <strong>Test</strong>, Tools Partnership<br />
‣ Continuous Regression<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 2
“<strong>Test</strong>” Fundamental Responsibilities & Activities<br />
Build up the <strong>Test</strong> case Baseline & Coverage management<br />
system<br />
Develop new Feature <strong>Test</strong> Design & Plan<br />
Develop the Solution Integration <strong>Test</strong> Strategy<br />
Develop & Execute new <strong>Test</strong> cases – manual & automated<br />
Promote <strong>Test</strong> Automation technology<br />
Fault Identification, Trouble Report & Retest<br />
<strong>Test</strong> Environment Management<br />
“Product Quality Assurance” is our #1 mission<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 3
Product Delivery <strong>Test</strong> Quality Indicators<br />
‣ All test cases have been executed & regressed with 95%+ Pass rate<br />
‣ Non-Critical/Gating remaining issues<br />
‣ Manageable number of Minor un-resolved issues<br />
‣ Provide remaining issues Patch Plan<br />
BUT:<br />
‣ Number of <strong>Test</strong> cases cannot direct reflect the Product’s Quality<br />
‣ Number of bugs found during the Product Verification cycle cannot<br />
reflect the Product’s Quality as well<br />
The key factor to evaluate the Product <strong>Test</strong> Quality is the<br />
“Product <strong>Test</strong> Coverage”<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 4
Product <strong>Test</strong> Coverage<br />
Requirements-based <strong>Test</strong> Coverage<br />
<strong>Test</strong> the product/Software behavior against the every claim made in the Requirements &<br />
Design specifications<br />
Functional <strong>Test</strong> Coverage<br />
<strong>Test</strong> the product/Software behavior against the Requirements, Design specifications,<br />
Customer Scenarios and examines what the program accomplishes from various areas:<br />
Level 1 new feature basic function,PQRS <strong>Test</strong> Coverage<br />
Level 2 new features’ interaction <strong>Test</strong> Coverage<br />
Level 3 new & old functional interaction <strong>Test</strong> Coverage (N-1)<br />
Regression <strong>Test</strong> Coverage<br />
Product <strong>Test</strong> case Baseline<br />
Code Coverage<br />
Is a measure used in software testing to describe the degree to which the source code of a<br />
program has been tested and via the analysis process we’ll also be able to:<br />
Find areas of a program not exercised by a set of test cases<br />
Create additional test cases to increase coverage<br />
Determine a quantitative measure of code coverage, which is an indirect measure of Quality<br />
May indentify redundant test cases that do not increase coverage<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 5
Security Level: Confidential<br />
Ideal <strong>Test</strong> Systems<br />
www.huawei.com<br />
HUAWEI TECHNOLOGIES CO., LTD.<br />
Huawei Confidential
Ideal <strong>Test</strong> Systems<br />
Fully Automated R&D <strong>Test</strong> Automation System:Code, Module,Story,Feature,<br />
System, Customer Acceptance <strong>Test</strong><br />
Fully Integrated R&D Management System:Requirements,Documentations,<strong>Test</strong>,<br />
Project,Production,Engineering,Beta/CAT<br />
Measureable Product <strong>Test</strong> Cases Coverage<br />
Measureable & Visible Products Quality Process: Requirements,Development,<br />
<strong>Test</strong>,Project,Production,Engineering,Beta/CAT<br />
Customized <strong>Test</strong> Case Execution Strategy<br />
Commercialized <strong>Test</strong> Environment & Tools:Product Solution Integration ,<br />
Networks Solution Integration,Customer Solution Integration Center<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 7
Fully Automated <strong>Test</strong> Automation Environment<br />
Traffic <strong>Test</strong><br />
Environment<br />
E2E <strong>Test</strong><br />
Environment<br />
Nodal Feature <strong>Test</strong><br />
Environment<br />
Designer Desktop <strong>Test</strong><br />
Environment<br />
Execution<br />
Config<br />
AutoB<br />
Billing<br />
Log/Event<br />
iGUI<br />
STEP<br />
Traffic<br />
Frame<br />
Work<br />
E2E <strong>Test</strong> Head<br />
RFSim TH<br />
AirDriver<br />
Simulation <strong>Test</strong> Heads<br />
PSTN <strong>Test</strong> Heads<br />
Um<br />
RF<br />
BTS<br />
BSC/RNC/LTE/IP<br />
Wireless <strong>Test</strong><br />
Heads<br />
IP <strong>Test</strong> Head<br />
Access <strong>Test</strong> Head<br />
DMS-<br />
MSC<br />
IP<br />
Networks<br />
* Single Framework for Nodal, E2E and Traffic * <strong>Test</strong> Case Sharing<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 8<br />
* Global Lab Sharing * Designer Desktop <strong>Test</strong>ing
Fully Integrated R&D Management System<br />
Marketing, Design<br />
Design Doc. Environment<br />
(API to access Database<br />
for Feature list, FRS/FTS url)<br />
link to<br />
Feature list<br />
feature<br />
and FRS url<br />
details<br />
1 2<br />
Requirements<br />
Management<br />
System<br />
<strong>Test</strong>cases/<strong>Test</strong> Results;<br />
Other customizable data such as<br />
CRs & enhancement<br />
3<br />
Project<br />
Management<br />
System<br />
Design, <strong>Test</strong>, Support<br />
<strong>Test</strong> case<br />
Management<br />
System<br />
Defects & enhancement<br />
Defects<br />
Management<br />
System<br />
Marketing, Design<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 9
<strong>Test</strong> Case Coverage/Management Objectives & Benefits<br />
Objectives<br />
• Ensure Product <strong>Test</strong> case Baseline Coverage<br />
• Enforce <strong>Test</strong> case and <strong>Test</strong> Coverage ownership<br />
• Improve <strong>Test</strong> Case Management (Quality, Efficiency, Coverage, Maintenances …)<br />
Why do we need to implement TCM ?<br />
• Enforce the corresponding primes to improve <strong>Test</strong> case quality and integrity<br />
• Vehicle to ensure & improve Product <strong>Test</strong> Coverage & Code Coverage<br />
• Vehicle to proper upgrade the Regression <strong>Test</strong> cases to the next Release<br />
• Vehicle to ensure the effectiveness of “Agile”, “Iteration”, “RBT” & “DFT”<br />
• Vehicle to ensure the System Integrity & implementation of “VBS”<br />
• Efficient <strong>Test</strong> cases Execution & Maintenance<br />
• Efficient Quality management (Eg. CR/Patch Analysis)<br />
• Vehicle to build up Product <strong>Test</strong> Knowledge base<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 10
TCM and <strong>Test</strong> Coverage Improvement<br />
<strong>Test</strong> Case Management -- by Subsystems<br />
System Upgrade<br />
Robustness &<br />
Capacity<br />
Bearer Service<br />
Teleservices<br />
Call Completion Call Offering Call Related Call Restriction<br />
Charging Supplementary Services GSM Compliance Intelligent Networks<br />
PSTN Signalling Mobility Management Regulatory Translations<br />
Location Based Services<br />
GSM Railways<br />
GSM/UMTS<br />
Functionality<br />
<strong>Test</strong> Case Clean up Process<br />
Manageable Subsystems:<br />
• <strong>Test</strong> cases were grouped into manageable Subsystems<br />
• <strong>Test</strong> cases executed on TICC per subsystem to understand the<br />
failures<br />
• Failures analyzed and test cases corrected<br />
• <strong>Test</strong> cases executed on TICC as frequently as possible, results<br />
analyzed and test cases cleaned up after every run to improve<br />
results<br />
Coding Standards:<br />
• <strong>Test</strong> case coding standards published and used to write and<br />
correct the test cases<br />
• TEP coding standards published and adopted to improve test<br />
case execution results on TICC<br />
<strong>Test</strong> Case Ownership -- Subsystems<br />
<strong>Test</strong> Coverage Improvement<br />
• Organize Product related <strong>Test</strong> cases into manageable<br />
pieces called Subsystems<br />
• Subsystems represent functional areas. Assigning Owners<br />
for each Subsystem<br />
• Subsystem Owners/Primes responsible for documentation,<br />
execution, analysis, debug, creating and maintaining TP/TC<br />
• Each subsystem is further broken down and composed by<br />
multiple Funcnode. Each Funcnode is composed by<br />
multiple Funcleaf, which is the smallest functionality unit in<br />
regression.<br />
• Subsystem Owner assigned, Design Manager identified<br />
• The Subsystem Owners of the identified functionality will<br />
research the coverage improvement and write test cases<br />
to plug the holes in their subsystem coverage<br />
• TCM test plan captures the coverage and functionality<br />
improvement<br />
• Identify the design subsystems and modules for that test<br />
subsystem<br />
• Update test strategy document with test coverage/<br />
Software module information and update document with<br />
code coverage results<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 11
Security Level: Confidential<br />
Target Development <strong>Test</strong><br />
Process<br />
www.huawei.com<br />
HUAWEI TECHNOLOGIES CO., LTD.<br />
Huawei Confidential
Target DT Process – Pre <strong>Test</strong>ing<br />
Preparation<br />
Market Requirement<br />
• Precise Feature<br />
& Tools<br />
Requirements<br />
New<br />
Feature<br />
Analysi<br />
s<br />
Design<br />
Evaluati<br />
on<br />
<strong>Test</strong><br />
Plan/<br />
Strategy<br />
• High<br />
Quality of<br />
Feature &<br />
Solution<br />
<strong>Test</strong> Plan &<br />
Strategy<br />
新 特 性 Pre 开 发 <strong>Test</strong>ing 测 试 Preparations<br />
1. Precise Customer<br />
Requirements<br />
2. Precise Tools Requirements<br />
新 特 性<br />
开 发 / 修<br />
正<br />
调 测 及<br />
自 动 验<br />
证<br />
测 试 建<br />
模 及 用<br />
例 开 发<br />
3. Clearly defined “Design”,<br />
“Story” and “Module”<br />
4. High Quality 每 天 Feature <strong>Test</strong><br />
Plan<br />
5. High Quality Solution <strong>Test</strong><br />
Strategy<br />
• 测 试 建 模 ,<br />
自 动 生 成<br />
“ 模 块 级 ”<br />
测 试 用 例<br />
• 自 动 生 成 与 执 行<br />
“ 白 盒 代 码 覆 盖<br />
用 例 ”<br />
• 自 动 执 行 结 果 验<br />
证 , 测 试 报 告<br />
产 品 需 求<br />
分 析<br />
缺 陷 修<br />
复<br />
版 本 维 护 / 更 新<br />
每 天<br />
缺 新 陷 特 性 验 和<br />
Story 证 调 测<br />
• 一 键 式 特 性 验 证<br />
• 用 例 自 动 筛 选<br />
• 自 动 用 例 执 行<br />
• 结 果 自 动 缺 报 陷 告 报 告<br />
• 自 动 提 交 代 码<br />
缺 陷 分 析<br />
及 定 位<br />
缺 陷 修 复<br />
代 码 归<br />
档<br />
每 周<br />
产 品 持<br />
续 构 建<br />
• 星 期 五 下 午<br />
4:00 pm 自 动<br />
建 新 版 本<br />
• 自 动 冒 烟 测<br />
试<br />
版 本 交 付<br />
新 需 求 / 新<br />
特 性<br />
• 告 警 / 日 志 分<br />
析 辅 助 定 位<br />
新 版 本 开 发<br />
特 性 设 计 及<br />
Story<br />
缺<br />
分<br />
陷<br />
解<br />
分<br />
析 定 位<br />
特 性 测 试 方 案<br />
及 验 证 计 划<br />
新 特 性 和<br />
Story 开 发<br />
特 性 测 试 用<br />
例 自 动 化<br />
特 性 和 Story 自<br />
动 验 证<br />
• 全 面 自 动 回 归 验 证<br />
• 星 期 一 “The<br />
Load is Ready for<br />
New Activities”<br />
代 码 归<br />
持<br />
档<br />
续 回 归<br />
自 动 验 证<br />
持 续 构<br />
建<br />
持 续<br />
构 建<br />
版 本<br />
交 付<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 13
The Importance of Feature <strong>Test</strong> Plan<br />
<strong>Test</strong>er’s most critical deliverable during the test cycle<br />
Can evaluate the tester’s feature knowledge<br />
Can evaluate the new feature’s test coverage<br />
Pre-define the test progress & evaluation point<br />
Ensure the Requirements Traceability’s coverage<br />
Provide the guideline for DTTP (Design, <strong>Test</strong>, Tools Partnership)<br />
define the “Capability <strong>Test</strong> cases” & schedule with designer<br />
Decide which test cases will be used for Regression<br />
Identify which test cases’ results can be shared with Data testing<br />
Lift up the testers test capability & product knowledge<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 14
Agile & Iteration Story-Based <strong>Test</strong> Process<br />
CHAR CDCP PDCP ADCP<br />
Product Req. Feature Strategy, Plan,<br />
Feature <strong>Test</strong><br />
Agile/Iteration Dev/<strong>Test</strong> Stage<br />
SDV/SIT SVT/Beta<br />
Analysis Analysis Design<br />
Complete<br />
TR1<br />
TR2<br />
TR3<br />
UT/IT/ST/BBIT TR4 TR4A TR5<br />
TR6<br />
GA<br />
F1-S1 F1-S2 F1-S3 F1-S4<br />
F2-S1 F2-S2 F2-S3 F2-S4<br />
F3-S1 F3-S2 F3-S3 F3-S4<br />
… … … …<br />
Fn-S1 Fn-S2 … Fn-Sm<br />
Feature 1 Story 1 Story 2 Story 3 Story 4 Remaining Total<br />
White box TC 10 5 10 15 10 50<br />
Black box TCs 10 15 20 25 30 100<br />
Total 20 20 30 40 40 150<br />
• Both the designers & testers are owning the features together – “DTTP, Design <strong>Test</strong> Tools Partnership”<br />
• Clear metric to measure each Story’s progress & Status<br />
• Story 1—4 needs to be completed Coding & <strong>Test</strong>ing prior to TR4 (completed the Agile designer <strong>Test</strong>ing/LLT)<br />
• Remaining <strong>Test</strong> cases will need to be completed at TR4 (completed the Feature testing)<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 15
Target DT Process – Development <strong>Test</strong>ing<br />
市 场 需 求<br />
New Feature Devel. <strong>Test</strong>ing<br />
1. Integrate “White box” test tool<br />
2. “Grey<br />
• 清 晰 明<br />
box<br />
确 的<br />
TCs” generation &<br />
特 性 需 求 和<br />
Execution 新 性<br />
验 证 方<br />
工 具 需 求 based on <strong>Test</strong><br />
分 析 / 修<br />
案 及 计<br />
Modeling & Record Playback<br />
正<br />
划<br />
3. Auto-Results & Log Analysis<br />
4. DTTP—Design, <strong>Test</strong>, Tools<br />
Partnership, early Feature <strong>Test</strong><br />
Automation 设 计 检<br />
5. Designer Desktop 视 <strong>Test</strong>ing for<br />
Capacity<br />
• 高 质 量 的<br />
解 决 方 案<br />
策 略<br />
Feature Development <strong>Test</strong>ing<br />
New<br />
Feature<br />
design &<br />
<strong>Test</strong><br />
Daily<br />
<strong>Test</strong> &<br />
Automatio<br />
n<br />
Verification<br />
<strong>Test</strong><br />
Design &<br />
TC<br />
developm<br />
ent<br />
• Auto-TC<br />
Generation for<br />
“Module/Grey<br />
box” testing<br />
• MBT<br />
• Fully Automated<br />
<strong>Test</strong> Automation<br />
Environment<br />
产 品 需 求<br />
分 析<br />
缺 陷 修<br />
复<br />
版 本 维 护 更 新<br />
每 天<br />
缺 新 陷 特 性 验 和<br />
Story 证 调 测<br />
• 一 键 式 特 性 验 证<br />
• 用 例 自 动 筛 选<br />
• 自 动 用 例 执 行<br />
• 结 果 自 动 缺 报 陷 告 报 告<br />
• 自 动 提 交 代 码<br />
缺 陷 分 析<br />
及 定 位<br />
缺 陷 修 复<br />
代 码 归<br />
档<br />
每 周<br />
产 品 持<br />
续 构 建<br />
• 星 期 五 下 午<br />
4:00 pm 自 动<br />
建 新 版 本<br />
• 自 动 冒 烟 测<br />
试<br />
版 本 交 付<br />
新 需 求 / 新<br />
特 性<br />
新 版 本 开 发<br />
特 性 设 计 及<br />
Story<br />
缺<br />
分<br />
陷<br />
解<br />
分<br />
• 告 警 / 日 析 志 定 分 位<br />
析 辅 助 定 位<br />
特 性 测 试 方 案<br />
及 验 证 计 划<br />
新 特 性 和<br />
Story 开 发<br />
特 性 测 试 用<br />
例 自 动 化<br />
特 性 和 Story 自<br />
动 验 证<br />
代 码 归 持 续 构<br />
持<br />
档<br />
续 回 归<br />
建<br />
自 动 验 证<br />
• 全 面 自 动 回 归 验 证<br />
• 星 期 一 “The<br />
Load is Ready for<br />
New Activities”<br />
持 续<br />
构 建<br />
版 本<br />
交 付<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 16
Fully Automated Designer <strong>Test</strong> Automation Environment<br />
Designer Desktop <strong>Test</strong>ing<br />
7*24 <strong>Test</strong> Automation Factory<br />
(CI)<br />
7*24 <strong>Test</strong> Automation Factory<br />
(SDV)<br />
Coding<br />
MST TCs<br />
DART<br />
DataBase<br />
Via Interface Record &<br />
Playback capability to<br />
test existing function<br />
Compile、<br />
Static Insp.<br />
DART<br />
MST<br />
<strong>Test</strong><br />
Compile、<br />
Statis Insp.<br />
DART<br />
MST<br />
<strong>Test</strong><br />
Module<br />
Capacity<br />
Feature<br />
SDV<br />
Regress<br />
Capacity<br />
Desktop Compile<br />
Module Level Compile<br />
Release <strong>Test</strong>ing<br />
本<br />
地<br />
构<br />
建<br />
check-in<br />
5 分 钟 完 成 70% 代 码 覆 盖<br />
Press<br />
编 译 静 态 检 查 DART<br />
代 码 覆 盖 分 析 函 数 时 长 分 析<br />
SVN<br />
Server<br />
Compile、<br />
Static Insp.<br />
VBS TC<br />
Selection<br />
Sanity<br />
Daily Compile<br />
VBS<br />
<strong>Test</strong><br />
VBS<br />
DataBase<br />
TC & Code<br />
correlation<br />
mapping<br />
New Load<br />
build<br />
Release<br />
10 分 钟 完 成 所 有 模 块 验 证<br />
CI <strong>Test</strong>ing<br />
10 小 时 5 万 用 例 全 回 归<br />
次<br />
级<br />
构<br />
建<br />
Daily Compile<br />
Sanity<br />
New<br />
Load<br />
VBS<br />
版<br />
本<br />
测<br />
试<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 17
DTTP (Design, <strong>Test</strong> Tools Partnership)<br />
Objectives:<br />
Pull forward the <strong>Test</strong> & Tools teams engagements at the early stage of the<br />
Requirements & <strong>Test</strong> Analysis cycle<br />
Ensure the Feature Quality during the TR3 & TR4 interval<br />
Identify the Quality issues at the early stage of the release cycle<br />
Shorten the Issues Resolution time during the development & test cycle<br />
Shorten the overall development & test cycle due to the parallel testing<br />
Enforce the Communication & Partnership among Design, <strong>Test</strong> & Tools<br />
members<br />
Enforce the Early <strong>Test</strong> Automation from the beginning of the Release Cycle<br />
Accommodate the implementations of Iteration, Agile & Early <strong>Test</strong><br />
Lift up the Capability of the <strong>Test</strong> & Tools teams<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 18
The March to UT/IT/ST:<br />
DTTP Feature <strong>Test</strong> Coverage<br />
100%<br />
60%<br />
40%<br />
20%<br />
Capability<br />
1 (first)<br />
Capability<br />
2<br />
Capability<br />
1<br />
<br />
Capability<br />
3<br />
Capability<br />
2<br />
<br />
Capability<br />
1<br />
<br />
Capability<br />
5 (last)<br />
Capability<br />
4<br />
Capability<br />
3<br />
<br />
Capability<br />
2<br />
<br />
Capability<br />
1<br />
<br />
Weekly <strong>Test</strong> Window<br />
TS<br />
Week #<br />
DTTP <strong>Test</strong>ing<br />
UT/IT/ST<br />
= UT/IT/ST ready<br />
Timely contributions from each group (Design, PV, Tools)<br />
required to make capability completion a success!<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 19
DTTP Benefits<br />
Ensure Product Early Quality、Shorten <strong>Test</strong> Cycle & TTM<br />
Pull forward the <strong>Test</strong> & Tools development activities<br />
Early <strong>Test</strong><br />
Enable & Support the designer’s testing<br />
Support the Agile <strong>Test</strong> Strategy – early <strong>Test</strong> Automation<br />
Lift up the Product Early Quality – design & test in-sync<br />
Code Ready<br />
Tools Ready<br />
<strong>Test</strong> cases/Scripts Ready<br />
Doing the <strong>Test</strong> Automation from the beginning<br />
Shorten the Development <strong>Test</strong> Cycle & TTM<br />
Lift up the capabilities for <strong>Test</strong>ers & Tools Developers<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 20
Business<br />
R&D TR<br />
Product Integration Verification and Validation<br />
Solution Verification Process Reference – Clear Milestone Expectations<br />
Opportunity<br />
Strategic<br />
Readiness<br />
Definition<br />
Market<br />
Readiness<br />
Business<br />
Readiness<br />
Implementation<br />
Customer<br />
Readiness<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 21<br />
CRA/TR1 TR2 TR3 TR4 TR4.a TR5 TR6<br />
GA<br />
Deployment<br />
Channel<br />
Readiness<br />
CRA/TR1 TR2 TR3 TR4<br />
TR4a TR5 TR6 GA<br />
Product Integration Verification and Validation<br />
Solution 1<br />
Feature Verification FVP<br />
FV Strategy FIT/FV <strong>Test</strong> Plan<br />
UT<br />
Activity 1<br />
Activity 2<br />
Activity N<br />
SW Integration.<br />
System Verification Process –<br />
Product Scope<br />
System Verification Process –<br />
Product Scope<br />
SD<br />
SD<br />
Feature Verification FVP - product<br />
FV Strategy FIT/FV <strong>Test</strong> Plan<br />
TP<br />
TP<br />
FIT<br />
UT<br />
Activity 1<br />
Activity 2<br />
Activity N<br />
SW Integration.<br />
Integration and Integration<br />
<strong>Test</strong> Process<br />
FIT<br />
TS<br />
TS<br />
Product<br />
Integration and Integration<br />
<strong>Test</strong> Process<br />
Product<br />
Integ.<br />
Feature Verification FVP for Solutions<br />
FV Strategy FIT/FV <strong>Test</strong> Plan Solution Feature Integration <strong>Test</strong><br />
System Verification Process – Solutions<br />
SD<br />
TP<br />
IT TFP<br />
TD SVE<br />
Product Feature Verification<br />
TFP<br />
TD<br />
Product Regression<br />
TS<br />
TFP TD<br />
Product System Verification<br />
TS<br />
TFP TD<br />
CAP <strong>Test</strong>ing<br />
Product<br />
Integ.<br />
TFP<br />
TD SVE<br />
Regression<br />
TS TFP TD SVE<br />
Product System Verif.<br />
TS TFP TD SVE<br />
CAP <strong>Test</strong>ing<br />
IT TFP<br />
TD SVE<br />
Product Feature Verification<br />
SVE<br />
SVE<br />
TS TFP<br />
Solution Feature Verification<br />
TD SVE<br />
TS<br />
TFP<br />
Solution Regression<br />
TD SVE<br />
TS<br />
TFP TD<br />
Solution System Verification<br />
SVE<br />
TS<br />
CAP <strong>Test</strong>ing<br />
TFP TD<br />
Customer<br />
Validation
Target DT Process – Development <strong>Test</strong>ing<br />
市 场 需 求<br />
产 品 需 求<br />
• 清 晰 明<br />
分<br />
确<br />
析<br />
的<br />
特 性 需 求 和<br />
工 具 需 求<br />
新 需 求 / 新<br />
特 性<br />
新 版 本 开 发<br />
新 特 性<br />
分 析 / 修<br />
正<br />
版 本 维 护 更 新<br />
特 性 设 计 及<br />
Story 分 解<br />
设 计 检<br />
视<br />
特 性 测 试 方 案<br />
及 验 证 计 划<br />
验 证 方<br />
新 特 性 和<br />
案 及 计<br />
Story 调 测<br />
划<br />
新 特 性 和<br />
Story 开 发<br />
特 性 测 试 用<br />
例 自 动 化<br />
• 高 质 量 的<br />
缺 解 陷 决 报 方 告 案<br />
策 略<br />
缺 陷 分 析<br />
及 定 位<br />
特 性 和 Story 自<br />
动 验 证<br />
缺 陷 修 复 新 特 性<br />
开 发 / 修<br />
正<br />
每 天<br />
代 码 归<br />
档<br />
调 测 及<br />
自 动 验<br />
证<br />
测 试 建<br />
模 及 用<br />
例 开 发<br />
持 续 构<br />
建<br />
• 测 试 建 模 ,<br />
自 动 生 成<br />
“ 模 块 级 ”<br />
测 试 用 例<br />
持 续 版 本<br />
构 • 建 自 动 生 成 与 交 执 付 行<br />
“ 白 盒 代 码 覆 盖<br />
用 例 ”<br />
• 自 动 执 行 结 果 验<br />
证 , 测 试 报 告<br />
Code Modification <strong>Test</strong>ing<br />
• Alarm/Log<br />
Analysis<br />
Bug<br />
Fix<br />
Daily<br />
Defect<br />
Analys<br />
is<br />
• 1-Key Verification<br />
• TC Auto-Selection<br />
• TC Auto-Execution<br />
• Auto Report<br />
Generation<br />
Bug<br />
<strong>Test</strong><br />
• Auto Code<br />
Submission<br />
• 星 期 五 下 午<br />
4:00 pm 自 动<br />
建 新 版 本<br />
• 自 动 冒 烟 测<br />
Code 代 码 归 Modification 产 品 <strong>Test</strong>ing 持 试<br />
1.<br />
档<br />
For every update, designers<br />
续 构 建<br />
can Auto-<br />
Target select 每 周 & Execute the TCs<br />
2. Automatic Results Analysis + related<br />
Failure data (Call Trace)<br />
3. Automatic 持 续 回 Code 归 Submission after the<br />
Passing 自 动 of 验 Target 证 <strong>Test</strong>ing<br />
• 全 面 自 动 回 归 验 证<br />
• 星 期 一 “The<br />
*CI -- Precise、Accurate,<br />
Load is Ready for<br />
New Activities”<br />
Fast*<br />
版 本 交 付<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 22
Validation Before Submission<br />
•Software Updates/Increments<br />
•(New Features or CRs Updates )<br />
1<br />
Feature Updates<br />
•Design Prime compiles and creates modules/patches<br />
• Submit the Request to VBS process<br />
CRs Updates<br />
2<br />
• CRs designers compile and create modules/patches<br />
• Submit the VBS process<br />
2<br />
TMS<br />
<strong>Test</strong> Cases Selection<br />
based on<br />
TCs & S/W Mapping<br />
TCs Optimization<br />
Compile & Pkg<br />
<strong>Test</strong> cases<br />
3<br />
Retrieve<br />
<strong>Test</strong><br />
cases<br />
4<br />
<strong>Test</strong><br />
Scripts<br />
<strong>Test</strong> Execution<br />
<strong>Test</strong><br />
Reports<br />
Execute<br />
<strong>Test</strong> cases<br />
Monitor<br />
Status<br />
AutoBrain<br />
5<br />
Validate<br />
Billing/Log<br />
Billing/Log<br />
Files<br />
Collect<br />
Billing/Log<br />
Result<br />
Analysis<br />
Expected<br />
Result<br />
Generate<br />
Results<br />
Match<br />
<strong>Test</strong><br />
case<br />
Result<br />
Files<br />
•Correct issues<br />
•Build it into new<br />
modules/patches<br />
2<br />
No<br />
6<br />
Module<br />
Integrity Suite<br />
pass at 100%?<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 23<br />
Yes<br />
•Submit Module to<br />
Weekly S/W Load build<br />
•Put <strong>Test</strong> Case ID’s in the<br />
update text<br />
7
VBS Applications – Continuous Integration & Patch Validation<br />
Release Baseline CI Application Patch Application<br />
ICP S/W Unified Platform<br />
Baseline Insertions<br />
Insertion<br />
Compile<br />
TC & Code Correlation<br />
TC<br />
Execution<br />
Compile,<br />
Static Insp.<br />
Daily Compile<br />
VBS<br />
Modification<br />
Identification<br />
Sanity<br />
Issue<br />
Analysis<br />
Patch Validation<br />
VBS<br />
TC<br />
Feature Target<br />
Interaction Regression<br />
Regression<br />
TC<br />
Patch Validation TC<br />
Code Coverage Analysis<br />
VBS<br />
DataBase<br />
Improve<br />
<strong>Test</strong> Coverage<br />
Retrieve Impacted<br />
TCs<br />
TMSS<br />
DataBase<br />
Select<br />
Impacted TC<br />
VBS<br />
TC<br />
CI TC<br />
Sanity<br />
TC<br />
【Fast TC Selection】Complete the TC Selection Within 10 minutes<br />
【CI】Precise, Accurate & Fast for Code Modification <strong>Test</strong><br />
【Patch Validation】Precise & Complete <strong>Test</strong> Coverage for Patches<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 24
Target DT Process – Development <strong>Test</strong>ing<br />
市 场 需 求<br />
缺 陷 分 析<br />
及 定 位<br />
• 清 晰 明 确 的<br />
特 性 需 求 和<br />
工 具 需 求<br />
产 品 需 求<br />
分 析<br />
新 需 求 / 新<br />
特 性<br />
新 版 本 开 发<br />
新 特 性<br />
分 析 / 修<br />
正<br />
版 本 维 护 更 新<br />
特 性 设 计 及<br />
Story 分 解<br />
设 计 检<br />
视<br />
特 性 测 试 方 案<br />
及 验 证 计 划<br />
验 证 方<br />
案 新 及 特 计 性 和<br />
Story 划 调 测<br />
新 特 性 和<br />
Story 开 发<br />
特 性 测 试 用<br />
例 自 动 化<br />
• 缺 高 陷 质 报 量 告 的<br />
解 决 方 案<br />
策 略<br />
特 性 和 Story 自<br />
动 验 证<br />
缺 陷 修 复<br />
新 特 性<br />
开 发 / 修<br />
正<br />
每 天<br />
代 码 归<br />
调 档 测 及<br />
自 动 验<br />
证<br />
测 试 建<br />
模 及 用<br />
例 开 发<br />
持 续 构<br />
建<br />
• 测 试 建 模 ,<br />
自 动 生 成<br />
“ 模 块 级 ”<br />
测 试 用 例<br />
• 自 动 生 成 与 执 行<br />
持 续 版 本<br />
“ 白 盒 代 码 覆 盖<br />
构 建 交 付<br />
用 例 ”<br />
• 自 动 执 行 结 果 验<br />
证 , 测 试 报 告<br />
• 告 警 / 日 志 分<br />
析 辅 助 定 位<br />
缺 陷 分<br />
析 定 位<br />
• 一 键 式 特 性 验 证<br />
• 用 例 自 动 筛 选<br />
• 自 动 用 例 执 行<br />
• 结 果 自 动 报 告<br />
• 自 动 提 交 代 码<br />
Load Build & Auto-Regression<br />
1. Friday 4:00 pm Auto-Load Build<br />
缺 陷 修<br />
2. Auto-Sanity 复 <strong>Test</strong>ing<br />
缺 陷 验<br />
3. Auto Bugs-Retest,Auto-Regression<br />
每 天 证<br />
4. Monday 8:00am “The Load is Ready<br />
for New Activities”<br />
*Continuous Regression:<br />
Broad、Deep、Thorough*<br />
Weekly Compile & Regression<br />
Code<br />
Subm<br />
issio<br />
n<br />
• Fully Automated<br />
Regression<br />
• At Monday --“The<br />
Load is Ready for<br />
New Activities”<br />
Weekly<br />
Continu<br />
ous<br />
Regres<br />
sion<br />
Continuous<br />
Compile<br />
• Friday 4:00 pm<br />
Auto-Load build<br />
• Auto-Sanity <strong>Test</strong><br />
Load Release<br />
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 25
Security Level: Confidential<br />
Thank You<br />
www.huawei.com<br />
www.huawei.com<br />
HUAWEI TECHNOLOGIES CO., LTD.<br />
Huawei Confidential
Change language