Architecture Evaluation - Department of Computer Information Systems

Architecture Evaluation 

Qualitative and Quantitative

1. Overview and 

Introduction 

A. Target Program 

Summary 

B. Program 

Description 

B. Background 

and History 

C. Solicitation 

Schedule 

D. Opportunity 

Rating 

Summary 

E. Capture 

Program Status 

2. Marketing 

Activities 

A. Customer 

Organization 

and 

Environment 

E. Customer 

Source 

Selection 

Process 

C. Customer 

Hot Buttons 

and Selection 

Criteria 

D. Competitor 

Analysis 

F. Customer 

Analysis and 

Bidder 

Comparison 

F. Capture 

Program 

Status 

3. Program 

Requirements 

4. Planned 

Program 

5. Business 

Issues 

A. Scope of Work A. Win Strategy A. Desirability 

of Project/ 

B. Overview of 

Program 

Requirements 

C. Integrated 

Customer 

Solution 

Worksheet 

D. Customer 

Solution “Actions 

Required” Plan 

B. Teaming 

Approach 

C. Technical 

Approach 

D. Management 

and 

Organizational 

Overview 

E. Personnel and 

Staffing 

Approach 

F. Related 

Experience 

and Past 

Performance 

Approach 

G. Cost/Price 

Approach 

Task 

B. NDA/ 

Teaming 

Agreements 

6. Proposal 

Requirements 

A. Proposal 

Manager and 

Key Team 

Members 

C. Cost to Bid C. Proposal 

Requirements 

D. Risk 

Analysis 

and 

Mitigation 

Plan 

7. Cost and 

Pricing 

A. Executive 

Cost Summary 

B. Proposal Plan B. Contingencies 

and Summary 

Volume Outline 

D. Special 

Requirements 

(i.e., Orals, E- 

mail, etc.) 

C. Material 

and/or 

Subcontractor 

Costs 

D. Fee and 

Operating 

Profit 

Summary 

E. Unusual 

Costing 

Factors/ 

Concerns 

F. BAFO 

Requirements 

(Discounting, 

etc.) 

Term 

Percent Complete 

Cost Performance Index 

or Performance Factor 

To Complete 

Performance Index 

or Verification Factor 

Schedule Performance Index 

Schedule Correlation 

Independent Estimate 

At Completion 

Average Performance 

Average Expected 

Performance To Finish 

Symbol 

% Done 

CPI or PF 

TCPI or VF 

SPI 

SC or S/C 

IEAC 

P CUM 

P TO GO 

1) 

2) 

ACWP + 

Formula 

BCWP 

BAC 

BCWP 

ACWP 

BAC - BCWP 

EAC - ACWP 

BCWP 

BCWS 

PCUM 

SV 

BAC 

PF 

BAC - BCWP 

.8CPI + .2SPI 

BCWPcum 

Duration (wks or mos) 

Since ACWP Began 

BCWPcum 

Duration (wks or mos) 

From Time Now to 

Manager's Stated 

Completion Date 

Checklist Actions 

Ratio of work accomplished in terms of the total amount of work 

Ratio of work accomplished against money spent (an efficiency ra 

Done for Resources Expended) 

Ratio of work remaining against money remaining (Efficiency whic 

achieved to complete the remaining work with the expected remain 

Ratio of work accomplished against what should have been done (E 

Rating: Work done as compared to what should have been done) 

to do. 

ting: Work 

h must be 

ing money) 

fficiency 

Ratio of Schedule Variance (SV) in terms of average amount of wo 

rk 

accomplished (in weeks or months). It indicates a correlation t 

o program true 

schedule condition 

Calculation of a projected Estimate At Completion to compare wit 

h the CAM's 

Estimate At Completion: 

1) Ration of total work to be done against experienced cost eff iciency 

2) Sunk costs added to a ratio of remaining work against weight ed cost and 

schedule efficiencies 

Average rate at which work has been accomplished since work bega 

Average rate at which work must be accomplished in the future to 

date the CAM has forecasted for completion of the work. 

n 

finish on the 

Architecture Management & Planning 

Skills 

Inventory 

Corporate 

Fixed Asset 

Inventory 

Vendor & 

New Technology Research 

Departmental 

Fixed Asset 

Inventory 

IT Hardware 

Inventory 

$ 

$ 

$ 

$ 

$ 

$ 

IT Software / Applications 

Inventory 

$ $ 

$ 

Architecture 

Management 

& Planning 

Today… 

$ 

$ $ $ 

Network Map 

$ 

$ 

$ 

Connectivity 

Diagrams 

Topology Map 

Business Unit & 

Departmental 

Interviews 

Organizational 

Charts 

Project 

Inventories 

Data Element Term Acronym 

Framework or 

Reference Model 

Enterprise Architectures 

Version 

Control 

Scheduled Work Budgeted Cost for Work Scheduled BCWS 

Earned Value Budgeted Cost for Work Performed BCWP 

Actuals Actual Cost of Work Performed ACWP 

Authorized Work Budget At Completion BAC 

Data Element Term Acronym 

Forecasted Cost Estimate At Completion EAC 

Work Variance Schedule Variance SV 

Cost Variance Cost Variance CV 

Completion Var Variance At Completion VAC 

Life Cycle 

Management 

Budgets 

. 

Buy 

List 

Standards 

Professor 

Truex 

Configuration 

Management

Two Techniques 

Architecture Tradeoff Analysis Method (ATAM) 

Qualitative approach to prioritizing requirements 

Cost Benefit Analysis Method (CBAM) 

Takes the output of the ATAM and adds economic 

analysis in the form of cost benefit tradeoffs 



Truex

ATAM 

Multiple stakeholders and participants 

Evaluation team 

Project managers 

Architecture stakeholders 

What characteristics do you want in the team? 

Roles and attributes (c.f. pg 273) 



Truex

ATM outputs/deliverables 

 

Rank ordered priorities in the following forms: 

1. Concise architecture model 

2. Clear business goals of the architecture (system) 

3. Quality requirement scenarios (QRSs) 

4. Mapping of architectural decisions to QRSs 

5. Sensitivity analysis and tradeoff points 

– How important is it and at what tradeoff? 

E.g., back up database important to reliability, problem for 

security 

6. Risk analysis and risk theme clustering 



Truex

Example tabular ATAM output 

Quality Attribute Attribute refinement Scenarios 

Performance 

Transaction Response 

time 

Throughput 

Generating reports 

User updates patient acct. 

in less than .75 second 

Patient acct under peak 

load in response to chg of 

add notification < 4 secs. 

Batch by midday 

Usability Proficiency training Experienced new hires up 

to speed in < 2wks 

Configurability 

Maintainability 


Normal operation 

Set payment plan real 

time with patient w/out 

delays 

No source code changes 

to change fee structures; 

CBAM -- Cost Benefit 

If each architectural decision has costs and tradeoffs 

(risks) 

How do we evaluate economic value and necessity? 



Truex

CBAM (Benefits) vs. ATAM (tradeoffs) 

ATAM identifies sets of key architectural decisions 

BAM quantifies them as to cost 

Scenarios 

Utility curves 

Best case and worst case scenarios compared to current and 

desired states 

Prioritizing scenarios via voting 



Truex

A model of Security Design 

Threats 

Control 

s 

Targets 


What are each of these elements? 


Truex

A model of Security Design 

Threats 

Control 

s 

Targets 

Destruction 

Modification 

Disclosure 

Avoidance 

Tolerance 

Mitigation 

Physical (Hardware, people…) 

Data 

Data Communications 



Truex

Intranet Security: instance 

example 

Intranet security is vital especially if connected to the 

Internet 

Security can be 

threatened (someone tries to break in) 

compromised (someone knows how to break in) 

breached (actually breaks in or infiltrates) 

Security threats can 

come from inside and outside 

be deliberate or accidental 



Truex

Types of Threats 

Threats to hardware 

Theft of equipment 

Tampering by disgruntled employees 

Destruction by natural accidents (fire, flood etc.) 

Ordinary wear and tear 

Threats to software 

Deletion - accidental or deliberate 

Theft by user 

Corruption by virus or hardware malfunction 

Threats to information 

Corruption, theft or deletion of files 



Truex

Planning Intranet Security 

Defining security goals 

Protect what? (hardware? network? data?) 

Protect from whom? (users? outsiders?) 

Protect from what? (fire? natural disasters?) 

Cost effectiveness of measures 

Typical security goals include 

Preventing malicious damage to files and system 

Preventing accidental damage 

Protecting data integrity and confidentiality 

Preventing unauthorized access 

Providing appropriate disaster recovery 



Truex

Intranet Security: Access 

Control 

Isolating the server 

physically 

by specific protocol 

by specific IP address 

Password access 

Passwords should 

be mixture of upper and lowercase; be of sizable length; not 

be words found in dictionary 

be changed regularly 

be changed from vendor supplied defaults 

Other password issues: 

passwords of ex-users should be removed 

no more than two invalid attempts should be allowed 



Truex

Intranet Security Techniques 

Cryptography (or encryption): converting a message (plaintext) 

into a secret code (cyphertext) and the reverse process 

Can be public or private 

Firewall: a device that sits between the internal network and the 

outside Internet 

Can be packet filtering, proxy server or combination (dualhomed) 

Authentication: proving the identity of both clients and servers 

Non-repudiation: proving that a document was originated by the 

sender 



Truex

Use the model to assess and 

plan... 

Threats 

Control 

s 

Targets 

Destruction 

Modification 

Disclosure 

Avoidance 

Tolerance 

Mitigation 

Physical (Hardware, people…) 

Data 

Data Communications 



Truex

Security Planning and Design 

Grid 

Physical Data Data Comm. 

Destruction 

Disclosure 

Modification 

Intentional 

Accidental 

Intentional 

Accidental 

Intentional 

Accidental 



Truex

Risk (Cost) benefit analysis 

E C = P I * C I 

E v = B i - E C 

Overall utility of scenarios 

Where B i = ∑ j (b i,j X W j ) 

Where bi,j is the benefit assigned to a strategy I given its effect on 

scenario j and where Wj is the weighting given to scenario j 

What is an inherent weakness in this formulation? 

Are traditional investment decision metrics adequate? 



Truex

Design benefits, costs and 

Return on Investment 

Question; How good is good enough? 

Once decided and costs are assigned then we compute 

the expected return on investment. That metric is, in 

turn, compared to organizational standards. 

ROI ==> R i = B i / C i 



Truex

CBAM steps 

1. Gather and group (Collate) scenarios 

2. Refine scenarios 

3. Prioritize scenarios 

4. Assign a utility to each 

5. Develop architectural strategies for each and assess 

expected quality attribute levels 

6. Determine utility value for each 

7. Derive expected benefit 

8. Choose strategies based on ROI 

9. Check choices with your intuition (common sense) 



Truex

Collected scenarios 

Response goals 

Refined Scenarios 

Assign utility 

c.f., pgs. 318-323 



Truex

Financial Analysis of Projects 

Financial considerations are often an important 

consideration in selecting projects 

Three primary methods for determining the 

projected financial value of projects: 

Net present value (NPV) analysis 

Return on investment (ROI) 

Payback analysis 



Truex

Net Present Value Analysis 

Net present value (NPV) analysis is a method of 

calculating the expected net monetary gain or loss from 

a project by discounting all expected future cash inflows 

and outflows to the present point in time 

Projects with a positive NPV should be considered if 

financial value is a key criterion 

The higher the NPV, the better 



Truex

Net Present Value Example 

Excel file 



Truex

Return on Investment 

Return on investment (ROI) 

or income divided by investment 

ROI = (total discounted benefits - total discounted costs) / 

discounted costs 

The higher the ROI, the better 

Many organizations have a required rate of return or 

minimum acceptable rate of return on investment for 

projects 



Truex

Payback Analysis 

Another important financial consideration is 

payback analysis 

The payback period is the amount of time it will 

take to recoup, in the form of net cash inflows, 

the net dollars invested in a project 

Payback occurs when the cumulative 

discounted benefits and costs are greater than 

zero 

Many organizations want IT projects to have a 

fairly short payback period 



Truex

NPV, ROI, and Payback 

Analysis for Project 1 

Excel file 



Truex

NPV, ROI, and Payback 

Analysis for Project 2 

Excel file 



Truex

Weighted Scoring Model 

A weighted scoring model is a tool that provides a 

systematic process for selecting projects based on many 

criteria 

First identify criteria important to the project selection process 

Then assign weights (percentages) to each criterion so they add up 

to 100% 

Then assign scores to each criterion for each project 

Multiply the scores by the weights and get the total weighted scores 

The higher the weighted score, the better 



Truex

Sample Weighted Scoring Model for 

Project Selection 

Excel file 



Truex

Architecture Evaluation - Department of Computer Information Systems

Create successful ePaper yourself

Delete template?

Save as template?