A Review of the Event Analysis of Systemic Teamwork Methodology

Analysing C4i Activity: A Review 

of the Event Analysis of Systemic 

Teamwork (EAST) Methodology 

The work described in this document has been undertaken by the Human Factors 

Integration Defence Technology Centre, part funded by the Human Capability Domain of 

the U.K. Ministry of Defence Scientific Research Programme. 

© Human Factors Integration Defence Technology Centre 2005. The authors of this 

report have asserted their moral rights under the Copyright, Designs and Patents act, 

1988, to be identified as the authors of this work. 

Reference ............................................HFIDTC/WP1.1.3/10 

Version.................................................................................2 

Date.............................................................31 October 2005 

©Human Factors Integration Defence Technology Centre 2005

HFIDTC/WP1.1.3/10 

Version 2/ 31 October 2005 

Authors 

Paul Salmon 

Neville Stanton 

Dr Guy Walker 

Dan Jenkins 

Brunel University 




ii



Contents 

1 Executive Summary ................................................................................... 1 

2 Introduction ................................................................................................ 2 

3 Methods Review......................................................................................... 5 

3.1 EAST – Event Analysis of Systemic Teamwork.................................................................. 5 

3.2 Observation....................................................................................................................... 26 

3.3 HTA – Hierarchical Task Analysis..................................................................................... 34 

3.4 CDA – Co-Ordination Demands Analysis ......................................................................... 39 

3.5 CUD – Comms Usage Diagram........................................................................................ 47 

3.6 Social Network Analysis.................................................................................................... 52 

3.7 Operation Sequence Diagrams......................................................................................... 60 

3.8 Critical Decision Method ................................................................................................... 68 

3.9 Propositional Networks ..................................................................................................... 75 

4 Feedback from Questionnaire .................................................................. 87 

4.1 Aim .................................................................................................................................... 87 

4.2 Participants........................................................................................................................ 87 

4.3 Equipment ......................................................................................................................... 87 

4.4 Procedure.......................................................................................................................... 87 

4.5 Results .............................................................................................................................. 88 

4.6 Discussion of Results........................................................................................................ 95 

5 Summary.................................................................................................. 96 

6 Bibliography.............................................................................................. 99 

7 Appendix ................................................................................................ 102 

iii



1 Executive Summary 

This report describes a review of the Event Analysis of Systemic Teamwork (EAST) 

methodology (Baber and Stanton 2004) that was used to analyse C4i activity in the rail, 

air traffic control, energy distribution, military, police, fire service and naval domains. 

The EAST methodology and its component methods were analysed using a set of human 

factors (HF) methods and criteria designed to analyse the performance of the methods. 

The analysis was based upon the EAST applications undertaken as part of the Human 

Factors Integration Defence Technology Centre (HFI DTC) work programme (see 

application domains above). 

The methods review indicated that the EAST methodology is an exhaustive technique 

that is particularly suited to the analysis of C4i, team based and collaborative activity. 

The methods review also indicated that the outputs from the EAST methodology are 

extremely useful, offering a number of different analyses of, and perspectives on the 

scenario in question. Further, it was concluded that the EAST methodology is relatively 

simple to learn and apply. In terms of flaws in the EAST methodology, the review 

indicated that the method can be overly time consuming (although this reflects the 

complexity of C4i domains) in some instances (i.e. for larger, more complex tasks) and 

also that there may be issues with the reliability of the method in some instances. 

1



2 Introduction 

The event analysis of systemic teamwork (EAST) methodology (Baber and Stanton 2004) 

was developed as part of the HFI DTC work programme, and uses a combination of HF 

methods to form a framework for analysing command, control, communication, 

computers and intelligence (C4i) activity. With a view to the development of a generic 

model of C4i, EAST has been applied to a number of diverse C4i scenarios, across a 

number of different domains. The domains and scenarios in which EAST has been used 

to analyse C4i activity are presented in Table 1. 

Table 1 – EAST Analysis 

Domain 

Air Traffic Control 

National Air Traffic Services 

Energy Distribution 

National Grid Transco 

Fire Service 

Military Aviation 

E3D 

Navy 

HMS Driad 

Police 

Rail 

(Signalling) 

Scenario 

Holding 

Overflight 

Departure 

Approach 

Shift handover 

Barking switching operations 

Feckenham switching operations 

Tottenham return to service operations 

Alarm handling operations 

Chemical incident at remote farmhouse 

Road traffic accident involving chemical tanker 

Factory fire exercise #1 

Factory fire exercise #2 

General operation 

Air threat 

Surface threat 

Sub-surface threat 

Car break in caught on CCTV 

Suspected car break in 

Mobile phone robbery 

Detachment scenario 

Emergency Possession scenario 

Handback a Possession scenario 

Possession scenario 

EAST review 

In order to analyse the performance of the EAST methodology and its component 

methods, a review of the method was conducted based upon the applications described 

above. The review was based upon the same criteria that were used in the HF methods 

review (Salmon et al 2004a) that was conducted for the development of the EAST 

methodology. The criteria are outlined below: 

1. Name and acronym – the name of the technique and its associated acronym. 

2



2. Author(s), affiliations(s) and address(es) – the names, affiliations and addresses of 

the authors are provided to assist with citation and requesting any further help in 

using the technique. 

3. Background and applications – This section introduces the method, its origins and 

development, the domain of application of the method and also application areas 

that it has been used in. 

4. Domain of application – describes the domain that the technique was originally 

developed for and applied in. 

5. Procedure and advice – This section describes the procedure for applying the 

method as well as general points of expert advice. 

6. Flowchart – A flowchart is provided, depicting the methods procedure. 

7. Advantages – Lists the advantages associated with using the method in the design 

of C4 systems. 

8. Disadvantages - Lists the disadvantages associated with using the method in the 

design of C4 systems. 

9. Example – An example, or examples, of the application of the method are 

provided to show the methods output. 

10. Related methods – Any closely related methods are listed, including contributory 

and similar methods. 

11. Approximate training and application times - Estimates of the training and 

application times are provided to give the reader an idea of the commitment 

required when using the technique. Training and application times are classified 

as low (0 – 2 hours), medium (2 – 6 hours) or high (6 + hours). 

12. Reliability and Validity - Any evidence on the reliability or validity of the method 

are cited. 

13. Tools needed – Describes any additional tools required when using the method. 

14. Bibliography - A bibliography lists recommended further reading on the method 

and the surrounding topic area. 

The criteria were applied to the overall EAST methodology and also the component 

individual methods. A summary of the EAST review is presented in Table 2. The review 

is presented in section 3 of this report. 

3



Table 2 – Summary of EAST methods review 

Method 

Event Analysis of 

Systemic Teamwork 

(EAST) 

(Baber and Stanton 

2004) 

Observation 

Hierarchical Task 

Analysis 

(Annett 2004) 

Co-ordination 

demands analysis 

(CDA) 

(Burke 2004) 

Type of 

method 

Team 

analysis 

method 

Data 

collection 

Task 

analysis 

Team 

analysis 

Domain SME’s Related 

methods 

Generic Yes Obs 

C4i 

HTA 

CDA 

OSD 

CUD 

SNA 

CDM 

Prop Nets 

Training 

time 

Application 

time 

Tools 

needed 

High High Video / 

Audio 

recording 

equipment 

MS Word 

MS Excel 

MS Visio 

AGNA 

Generic No HTA Low High Video / 

Audio 

recording 

equipment 

MS Word 

Generic No Obs Med High Pen and 

paper 

MS Notepad 

Generic Yes Obs 

HTA 

Low Med Pen and 

paper 

MS Excel 

Reliability Validity Advantages Disadvantages 

Med High 1. EAST offers an extremely 

exhaustive analysis of the C4i 

domain in question. 

2. EAST is relatively easy to train 

and apply. The provision of the 

WESTT and AGNA software 

packages also reduces application 

time considerably. 

3. The EAST output is extremely 

useful, offering a number of 

different analyses and 

perspectives on the C4i activity in 

question. 

High High 1. Acts as the primary input the 

EAST methodology. 

2. Easy to conduct provided the 

appropriate planning has been 

made. 

3. Allows the analysts to gain a 

deeper understanding of the 

domain and scenario under 

analysis. 

Med High 1. Easy to learn and apply. 

2. Allows the analysts to gain a 

deeper understanding of the 

domain and scenario under 

analysis. 

3. Describes the task under 

analysis in terms of component 

task steps and operations. 

Low High 1. Offers an overall rating of coordination 

between team members 

for each teamwork based task step 

in the scenario under analysis. 

2. Also offers a rating for each 

teamwork behaviour in the CDA 

teamwork taxonomy. 

3. The technique can be used to 

identify task-work (individual) and 

team-work task steps involved in 

the scenario in question. 

1. Due to its exhaustive nature, EAST is 

time consuming to apply. 

2. Reliability may be questionable in some 

areas. Reliability of the method is still 

being established. 

3. A large portion of the output is 

descriptive. Great onus is placed upon the 

analyst to interpret the results accordingly. 

1. Observations are typically time 

consuming to conduct (including the 

lengthy data analysis procedure). 

2. It is often difficult to gain the required 

access to the establishment under 

analysis. 

3. There is no guarantee that the required 

data will be obtained. 

1. Can be difficult and time consuming to 

conduct for large, complex scenarios. 

2. Reliability is questionable. Different 

analysts may produce different HTA 

outputs for the same scenario. 

3. Provides mainly descriptive rather than 

analytical information. Also does not cater 

for the cognitive components of task 

performance. 

1. The CDA procedure can be time 

consuming and laborious. For each 

individual task step, seven teamwork 

behaviours are rated. 

2. To ensure validity, SME’s are required. 

3. Intra and inter-analyst reliability may be 

questionable. 

4



3 Methods Review 

3.1 EAST – Event Analysis of Systemic Teamwork 

Background and applications 

The EAST methodology (Baber and Stanton 2004) was developed for the analysis of C4i 

(command, control, communications, computers and intelligence) activity. EAST uses a 

combination of HF methods to form a framework for analysing C4i activity. A brief 

description of each component method is provided below: 

• Hierarchical Task Analysis (HTA) – involves describing the scenario under 

analysis using a hierarchy of goals, sub-goals and operations. 

• Observation – is used during an EAST analysis to gather data surrounding 

the scenario under analysis. The observational data obtained is used as the 

primary input to an EAST analysis. 

• Co-ordination demands analysis (CDA) – involves defining the task-work 

and team-work tasks involved during the scenario, and then rating each 

teamwork task step against the CDA taxonomy of; communication; 

situational awareness; decision making; mission analysis; leadership; 

adaptability; and assertiveness. 

• Comms Usage Diagram (CUD) - is used to describe collaborative activity 

between distributed agents. The output of CUD describes how and why 

communications between a team occur, which technology is involved in the 

communication, and the advantages and disadvantages of the technology 

medium used. 

• Social Network Analysis – involves defining and analysing the 

relationships between agents within a scenario network. A matrix of 

association and a social network diagram are constructed, and agent 

centrality, sociometric status and network density figures are calculated. 

• Operation Sequence Diagram (OSD) – is used to represents the tasks, the 

actors, the communications, the social organisation, the sequence and time 

in which the scenario took place. An OSD captures the flow of information 

among distributed actors and shows how this is mediated through 

technology and team working. Additionally, operational loafing figures are 

calculated for the following OSD operators; Operation, Receive, Delay, 

Decision, Transport, Combined Operations. 

• Critical Decision Method (CDM) – involves the use of interview probes to 

elicit information regarding agent decision making strategies adopted 

during the scenario under analysis. 

5



• Propositional Networks – The CDM output is used to construct 

propositional networks for each incident phase. Propositional networks are 

comprised of nodes that represent sources of information, agents, and 

objects that are linked through specific causal paths. The propositional 

network thus represents the ‘ideal’ collection of knowledge for an incident. 

The EAST methodology has been applied in a number of domains, including the fire 

service (Baber et al 2004a), police service (Baber et al 2004b) naval warfare (Stewart et 

al 2004), military aviation, energy distribution (Salmon et al 2004b), air traffic control 

and rail domains (Walker et al 2004), in order to analyse C4i activity. EAST is a 

comprehensive technique offering a multi-faceted assessment of the C4i network in 

question. EAST provides an assessment of agent roles within the network, a description 

of the activity including the flow of information, the component tasks, communication 

between agents and the operational loading of each agent. Co-ordination between agents 

is also rated and the knowledge required throughout the task under analysis is defined. 

Domain of application 

Generic. The EAST methodology can be used in any domain that utilises a C4i 

infrastructure during task performance. 

Procedure and advice 

Step 1: Define scenario(s) 

The first step in an EAST analysis involves defining the scenario(s) that are to be the 

subject of the analysis. It is recommended that this is done in collaboration with a SME 

from the domain in question. The chosen scenario(s) should be representative of all C4i 

activity in the domain in question. For example, in an analysis of C4i activity in the civil 

energy distribution (Salmon et al 2004), a switching out of circuit’s scenario and a return 

to service of circuits scenario were used. 

Step 2: Conduct HTA for the scenario(s) under analysis 

Once they are clearly defined, the analyst(s) should conduct a HTA for each scenario. 

Again, it is recommended that this is done in collaboration with a relevant SME. 

Step 3: Plan observation 

The next step of the EAST analysis involves planning the observation of the scenario(s) 

under analysis. Typically the scenarios involved in C4i activity are dispersed over a 

number of different locations involving a number of agents, and so who to observe and 

where to observe it requires careful thought. Again, input from an appropriate SME is 

useful here. It is also useful at this stage to clearly define what data is to be collected. The 

use of recording equipment (e.g. type, location, focus etc) should also be clarified. If time 

permits, it may also be pertinent to conduct a trial run of the planned observation. 

Step 4: Observe scenario(s) 

6



The observation step is the most important part of the EAST procedure. Typically, a 

number of analyst(s) are used in scenario observation. All activity involved in the 

scenario under analysis should be recorded along an incident timeline, including a 

description of the activity, the agents involved, any communications made and the 

technology involved. Additional notes should be made where required, including the 

purpose of the activity, any errors made and also any information that the agent involved 

feels is relevant. 

Step 5: Critical decision method 

Once the scenario under analysis is complete, each ‘key’ (e.g. scenario commander, 

agents performing the component tasks etc) involved should be subjected to a CDM 

interview. This involves dividing the scenario into key incident phases and then 

interviewing the agent involved using pre-defined probes. A CDM interview should be 

conducted for each incident phase. 

Step 6: Transcribe scenario data 

Once all of the scenario data is collected, it should be transcribed in order to make it 

compatible to the EAST analysis phase. An event transcript should be constructed. The 

transcript should describe the scenario over a timeline, including descriptions of activity, 

the agents involved, any communications made and the technology used. In order to 

ensure the validity of the data, the scenario transcript should be reviewed by one of the 

SME’s involved. 

Step 7: Re-iterate HTA 

The data transcription process allows the analyst(s) to gain a deeper and more accurate 

understanding of the scenario under analysis. It also allows any discrepancies between 

the initial HTA scenario description and the actual activity observed to be resolved. 

Typically, C4i activity does not run entirely according to protocol, and certain tasks may 

have been performed during the scenario that were not described in the initial HTA 

description. The analyst should compare the scenario transcript to the initial HTA, and 

add any changes as required. 

Step 8: Co-ordination demands analysis 

The CDA involves extracting teamwork tasks from the HTA and rating them against the 

associated CDA taxonomy. Each teamwork task is rated against each CDA behaviour on 

a scale of 1 (low) to 3 (high). Total co-ordination for each teamwork step can be derived 

by calculating the mean across the CDA behaviours. The mean total co-ordination figure 

for the scenario under analysis should also be calculated. 

Step 9: Construct comms usage diagram 

The CUD is used to represent communication between the agents involved in the scenario 

and also to analyse the comms technology utilised. The scenario transcript is used as the 

input to the CUD. 

Step 10: Conduct social network analysis 

7



The SNA is used to analyse the relationships between the agents involved in the scenario 

under analysis. Using the scenario transcript, the analyst should firstly construct an agent 

association matrix, which presents the links between the agents and also the frequency of 

communications between the agents. An example association matrix is presented in Table 

6. From the association matrix, a social network diagram is constructed and agent 

centrality, sociometric status, and network density are calculated. It is recommended that 

the Agna SNA software package is used for the SNA phase of the EAST methodology. 

Step 11: Construct operation sequence diagram 

The OSD represents the activity observed during the scenario under analysis. The analyst 

should construct the OSD using the scenario transcript and the associated HTA as inputs. 

Once the initial OSD is completed, the analyst should then add the results of the CDA to 

each teamwork task step. 

Step 12: Construct Propositional networks 

The final step of the EAST analysis involves constructing propositional networks for 

each scenario phase identified during the CDM interviews. In order to construct the 

propositional networks for each phase, a basic contents analysis should be conducted 

using the associated CDM outputs. Knowledge objects are defined as knowledge, 

information, artefacts and actions. Each knowledge object should have a corresponding 

node in the propositional network. Next, the links between the nodes should be specified. 

To do this, the analyst should specify any links between the knowledge objects within the 

propositional networks, using the following links taxonomy; 

Example 

• Has 

• Is 

• Causes 

• Knows 

• Requires 

• Prevents 

The following example is taken from an analysis of a switching scenario drawn from the 

civil energy distribution domain (Salmon et al 2004b). The scenario took place at Barking 

275Kv, 132Kv and 33Kv substations and the National Grid Transco (NGT) Network 

Operations Centre (NOC) in Warwick. The scenario under analysis involved the 

switching out of three circuits (SGT5 and SGT1A and 1B) at Barking 275Kv, 132Kv and 

33Kv Substations. Circuit SGT5 was being switched out for the installation of a new 

transformer for the nearby channel tunnel rail link and SGT1A and 1B were being 

switched out for substation maintenance. The observation focussed upon four main 

8



agents involved in the scenario. A description of the agents involved is presented in Table 

3. 

Table 3 – Agents involved in Switching Scenario 

Agent Title 

NOC 

SAP 

WOK 

REC 

Details 

Control room operator based at the NOC 

Senior Authorised Person (SAP) at Barking substations. 

Also working with authorised person (AP) 

Control room operator based at Wokingham control room 

Control room operator based at the regional electricity 

company control room 

Activity was observed by two observers. The first observer was situated at the Network 

operations centre (NOC) at National Grid Transco (NGT) house in Warwick, and 

observed the activity of the NOC operator. The second observer was situated at Barking 

275Kv, 132Kv and 33Kv Substations, and observed the activity of the SAP and AP. The 

results of the CDA are presented in Table 4. An extract of the CDA is presented in Table 

5. 

Table 4 – Switching scenario CDA results 

Category 

Result 

Total task steps 314 

Total taskwork 114 (36%) 

Total teamwork 200 (64%) 

Mean Total Co-ordination 1.57 

Modal Total Co-ordination 1.00 

Minimum Co-ordination 1.00 

Maximum Co-ordination 2.14 

The agent association matrix is presented in Table 6. From the association matrix, a 

social network diagram was constructed (Figure 3-1). Directional arrows represent the 

communications between agents, and the frequency of communications between agents is 

also presented. Centrality, and sociometric status were also calculated for each agent 

involved in the scenario. The SNA results are presented in Table 7. 

9



NOC 

Operator 

8 

2 

3 

1 

3 

SAP at 

Barking 

substation 

3 

WOK 

Operator 

2 

1 

2 

1 

1 

REC 

operator 

Figure 3-1 – Social Network Diagram 

10



Table 5 – Extract of CDA analysis 

Task 

Step 

Agent Step No. Task 

Work 

Team 

Work 

Comm SA DM MA Lead Ad Ass TOT 

CO- 

ORD 

Mode 

TOT 

CO- 

ORD 

Mean 

1.1.1 WOK control room operator Use phone to contact NOC 1 

1.1.2 WOK control room operator Exchange identities 1 3 3 1 1 1 1 1 1.00 1.57 

NOC control room operator 

1.1.3 WOK control room operator Agree SSC documentation 1 3 3 3 1 1 1 1 1.00 1.86 


1.1.4.1 NOC control room operator Agree SSC with WOK 1 3 3 3 1 1 1 1 1.00 1.86 

1.1.4.2 NOC control room operator Agree time with WOK 1 3 3 3 1 1 1 1 1.00 1.86 

1.1.5.1 NOC control room operator Record details onto log sheet 1 

1.1.5.2 NOC control room operator Enter details into WorkSafe 1 

1.2.1 NOC control room operator Ask for isolators to be opened 

1 3 3 1 2 2 1 1 1.00 1.86 

remotely 

1.2.2 WOK control room operator Perform remote isolation 1 

1.2.3 NOC control room operator Check barking s/s screen 1 

1.2.4 WOK control room operator End communications 1 3 1 1 1 1 1 1 1.00 1.29 


1.3.1 NOC control room operator Use phone to contact SAP at 1 

Barking 

1.3.2 NOC control room operator Exchange identities 1 3 3 1 1 1 1 1 1.00 1.57 

SAP at Barking 

11



Table 6 – Agent Association matrix 

A B C D 

A 0 3 2 1 

B 8 0 2 2 

C 3 3 0 0 

D 1 1 1 0 

Table 7 – SNA results 

Agent 

B-L 

Centrality 

Sociometric 

Status 

SAP at Barking 2.16 6.33 

NOC operator 2.16 6.00 

REC operator 1.85 2.00 

WOK control operator 1.85 3.66 

A CUD was constructed in order to analyse the communications made and the comms 

technology used. An extract of the CUD for the switching scenario is presented in Figure 

3-2. 

NGC 16 th June 2004 Scenario: Switching Operations 

Time 

NOC Control Room 

Operator 

SAP/AP at Barking 

Substation 

WOK Control 

Room Operator 

REC Control Room 

Operator 

Comms 

Effects of comms 

medium used 

Recommended comms 

11:55 

Contact NOC to 

agree SSC and 

transfer control to 

NOC 

+ Sound is clear and stable 

+ Comms is two way 

+ Can look at site diagram/desktop 

screen at the same time 

+ Confidentiality maintained 

- Data is not recorded. Information 

may be lost 

11:56 

Contact REC/EDF to 

confirm busbar 

reactor B operations 

complete 

+ Ensures high Mobility 

+ Sound is stable and clear 


+ Operator has hands free for 

other tasks 

- Battery may run out 

- Comms may be affected by poor 

signal 

- Data is not recorded, information 

may be lost 

11:57 Contact REC to give 

instructions regarding 

substation circuit and 

POI’s 







may be lost 

12:17 

Contact SAP at 

Barking to give 

switching instructions 







may be lost 

Figure 3-2 – CUD extract 

12



An OSD was constructed in order to represent the flow of activity during the switching 

scenario. The OSD glossary is presented in Figure 3-3. An extract of the OSD for the 

switching scenario is presented in Figure 3-4. From the OSD, operational loading figures 

were calculated in order to determine agent loading during the scenario. The operational 

loading figures are presented in Table 8. 

Table 8 – Operational loading results 

Agent Operation Receive Transport Decision Delay Total 

NOC 98 40 138 

SAP 223 21 19 1 264 

WOK 40 10 50 

REC 15 14 29 

The operational loading figures indicate that the NOC operator was loaded the highest for 

receive (i.e. receipt of information) operations, and the SAP at Barking was loaded the 

highest for operation, transport and delay operations. 

Figure 3-3 – OSD Glossary 

13



Figure 3-4 – OSD Extract 

The CDM analysis for the switching scenario involved interviewing the SAP at Barking 

and the NOC control room operator. In both cases the probes defined by O’Hare et al. 

(2000) were used to evaluate the decision making during the scenario. The CDM probes 

used are presented in Table 9. For the purposes of the CDM analysis, the scenario was 

divided into four phases; First issue of instructions, deal with switching requests, perform 

isolation and report back to NOC. The CDM analysis is presented in Table 10 to Table 

13. 

14



Table 9 – CDM probes (O’Hare et al 2000) 

Goal Specification 

Cue Identification 

Expectancy 

Conceptual 

Influence of uncertainty 

Information integration 

Situation Awareness 

Situation Assessment 

Options 

Decision blocking - stress 

Basis of choice 

Analogy/ 

generalisation 

What were your specific goals at the various decision 

points 

What features were you looking for when you formulated 

your decision 

How did you that you needed to make the decision 

How did you know when to make the decision 

Were you expecting to make this sort of decision during the 

course of the event 

Describe how this affected your decision making process. 

Are there any situations in which your decision would have 

turned out differently 

Describe the nature of these situations and the 

characteristics that would have changed the outcome of 

your decision. 

At any stage, were you uncertain about either the reliability 

of the relevance of the information that you had available 

At any stage, were you uncertain about the appropriateness 

of the decision 

What was the most important piece of information that you 

used to formulate the decision 

What information did you have available to you at the time of 

the decision 

Did you use all of the information available to you when 

formulating the decision 

Was there any additional information that you might have 

used to assist in the formulation of the decision 

Were there any other alternatives available to you other 

than the decision you made 

Was their any stage during the decision making process in 

which you found it difficult to process and integrate the 

information available 

Describe precisely the nature of the situation 

Do you think that you could develop a rule, based on your 

experience, which could assist another person to make the 

same decision successfully 

Why/Why not 

Were you at any time, reminded of previous experiences in 

which a similar decision was made 

Were you at any time, reminded of previous experiences in 

which a different decision was made 

15



Table 10 – CDM Phase 1: First issue of instructions 


Cue identification 

Expectancy 

Conceptual Model 

Uncertainty 

Information 


Situation 

Assessment 

Options 

Stress 

Choice 

Analogy 

Establish what isolation the SAP at Barking is looking for. Depends on 

gear 

Don’t Believe It (DBI) alarm is unusual – faulty contact (not open or 

closed) questionable data from site checking rating of earth switches 

(may be not fully rated for circuit current – so additional earths may be 

required. 

Check that SAP is happy with instructions as not normal. 

Decision expected by DBI is not common. 

Recognised instruction but not stated in WE1000 – as there are not 

too many front and rear shutters metal clad switch gear. 

Confirm from field about planned instruction – make sure that SAP is 

happy with the instruction. 

Reference to front and rear busbars. 

WE1000 procedure 

Metal clad switchgear 

Barking SGT1A/1B substation screen 


Ask colleagues if needed to 

No alternatives 

N/A 

WE1000 – need to remove what does not apply 

Could add front and rear busbar procedures 

Best practice guide for metal clad EMS switching 

16



Table 11 - CDM Phase 2: Deal with switching requests 



Expectancy 


Uncertainty 

Information 

Situation 

Awareness 

Situation 

Assessment 

Options 

Stress 

Choice 

Analogy 

Obtain confirmation from NOC that planned isolation is still required. 

Approaching time for planned isolation. 

Switching phone rings throughout building. 

Airblast circuit breakers (accompanied by sirens) can be heard to 

operate remotely (more so in Barking 275 than Barking C 132). 

Yes – routine planned work according to fixed procedures. 

Wokingham have performed remote isolations already. 

Circuit configured ready for local isolation. 

Physical verification of apparatus always required (DBI – don’t 

believe it). 

Proceduralised information from NOC – circuit, location, time, actions 

required etc. 

Switching log. 


Physical status of apparatus. 

Planning documentation. 

Visual or verbal information from substation personnel. 

Planning documentation used only occasionally 

Refusal of switching request. 

Additional conditions to switching request. 

Some time pressure. 

Yes – highly proceduralised anyway 

Yes – routine activity 

17



Table 12 – CDM Phase 3: Perform isolation 



Expectancy 


Uncertainty 

Information 


Situation 

Assessment 

Options 

Stress 

Choice 

Analogy 

Ensure it is safe to perform local isolation. 

Confirm circuits/equipment to be operated. 

Telecontrol displays/circuit loadings. 

Equipment labels. 

Equipment displays. 

Other temporary notices. 

Equipment configured according to planned circuit switching. 

Equipment will function correctly. 

Layout/type/characteristics of circuit. 

Circuit loadings/balance. 

Function of equipment. 

Will equipment physically work as expected (will something jam etc.). 

Other work being carried out by other parties (e.g. EDF). 


Visual and verbal information from those undertaking the work. 

Physical information from apparatus and telecontrol displays. 

All information used 

Inform NOC that isolation cannot be performed/other aspects of 

switching instructions cannot be carried out. 


Possibly some difficulties in operating or physically handling the 

equipment. 

Yes – proceduralised within equipment types. Occasional non-routine 

activities required to cope with unusual/unfamiliar equipment, or 

equipment not owned by NGT. 

Yes – often. Except in cases with unfamiliar equipment. 

18



Table 13 – Phase 4: Report back to NOC 



Expectancy 


Uncertainty 

Information 


Situation 

Assessment 

Options 

Stress 

Choice 

Analogy 

Inform NOC of isolation status. 

Switching telephone. 

NOC operator answers. 

NOC accepts. 

Manner in which circuit is now isolated. 

Form of procedures. 

No – possibly further instructions, possibly mismatches local situation 

and remote displays in NOC. 


Verbal information from NOC. 


Yes – all information used. 

No (raise or add on further requests etc. to the same call) 

No 

Yes – highly proceduralised 

Yes – frequently performed activity 

From the CDM outputs, propositional networks were constructed for each incident phase. 

The propositional networks are presented in Figure 3-5 to Figure 3-9. Propositional 

networks consist of a set of nodes that represent sources of information, agents, and 

objects etc. that are linked through specific causal paths. From this network, it is possible 

to identify required information and possible options relevant to this incident. The 

concept behind using a propositional network in this manner is that it represents the 

‘ideal’ collection of knowledge for the scenario. As the incident unfolds, so participants 

will have access to more of this knowledge (either through communication with other 

agents or through recognising changes in the incident status). Consequently, within this 

propositional network, Situation Awareness can be represented as the change in 

weighting of links. Propositional networks were developed for the overall scenario and 

also the incident phases identified during the CDM analysis. The propositional networks 

indicate which of the knowledge objects are active (i.e. agents are using them) during 

each incident phase. The light blue nodes in the propositional networks represent 

unactivated knowledge objects (i.e. knowledge is available but is not required nor is it 

being used). The red nodes represent active (or currently being used) knowledge objects. 

19



Gas Insulated 

Circuit 

Breakers 

Airblast 

Notices/Locks 

NOC 

Check Open 

Lock & Caution 

Location 

SystemState 

Certificates 

Isolation 

Control 

Engineer 

Open Lock & 

Caution 

Accept 

Instructions 

Switching 

Wokingham 

Shutters 

Time 

Dressed 

Log Sheet 

Procedures 

Rear 

Refuse 

Switching Log 

Displays 

Front 

Earth Switches 

Isolators 

WE1000 

Points of 

Isolation 

Open 

Busbar 

Circuits 

Electrical 

Contacts 

Closed 

Phone 

Current 

Voltage 

Faulty 

DBI Indication 

Switching 

Phone 

Equipment 

Lables 

Identity 

SAP 

Substations 

Figure 3-5 – Propositional Network for Objects referred to in CDM tables 

Gas Insulated 

Circuit 

Breakers 

Airblast 

Notices/Locks 

NOC 

Check Open 


Location 

SystemState 

Certificates 

Isolation 

Control 

Engineer 

Open Lock & 

Caution 

Accept 

Instructions 

Switching 

Wokingham 

Shutters 

Time 

Dressed 

Log Sheet 

Procedures 

Rear 

Refuse 

Switching Log 

Displays 

Front 


Isolators 

WE1000 


Isolation 

Open 

Busbar 

Circuits 

Electrical 

Contacts 

Closed 

Phone 

Current 

Voltage 

Faulty 


Switching 

Phone 

Equipment 

Lables 

Identity 

SAP 

Substations 

Figure 3-6 – Propositional Network for CDM Phase 1 

20



Gas Insulated 

Circuit 

Breakers 

Airblast 

Notices/Locks 

NOC 

Check Open 

Lock &Caution 

Location 

SystemState 

Certificates 

Isolation 

Control 

Engineer 

Open Lock & 

Caution 

Accept 

Instructions 

Switching 

Wokingham 

Shutters 

Time 

Dressed 

Log Sheet 

Procedures 

Rear 

Refuse 

Switching Log 

Displays 

Front 


Isolators 

WE1000 


Isolation 

Open 

Busbar 

Circuits 

Electrical 

Contacts 

Closed 

Phone 

Current 

Voltage 

Faulty 


Switching 

Phone 

Equipment 

Lables 

Identity 

SAP 

Substations 

Figure 3-7 – Propositional Network for CDM phase two 

Gas Insulated 

Circuit 

Breakers 

Airblast 

Notices/Locks 

NOC 

Check Open 

Lock &Caution 

Location 

SystemState 

Certificates 

Isolation 

Control 

Engineer 

Open Lock & 

Caution 

Accept 

Instructions 

Switching 

Wokingham 

Shutters 

Time 

Dressed 

Log Sheet 

Procedures 

Rear 

Refuse 

Switching Log 

Displays 

Front 


Isolators 

WE1000 


Isolation 

Open 

Busbar 

Circuits 

Electrical 

Contacts 

Closed 

Phone 

Current 

Voltage 

Faulty 


Switching 

Phone 

Equipment 

Lables 

Identity 

SAP 

Substations 

Figure 3-8 – Propositional Network for CDM phase three 

21



Gas Insulated 

Circuit 

Breakers 

Airblast 

Notices/Locks 

NOC 

Check Open 


Location 

SystemState 

Certificates 

Isolation 

Control 

Engineer 

Open Lock & 

Caution 

Accept 

Instructions 

Switching 

Wokingham 

Shutters 

Time 

Dressed 

Log Sheet 

Procedures 

Rear 

Refuse 

Switching Log 

Displays 

Front 


Isolators 

WE1000 


Isolation 

Open 

Busbar 

Circuits 

Electrical 

Contacts 

Closed 

Phone 

Current 

Voltage 

Faulty 


Switching 

Phone 

Equipment 

Lables 

Identity 

SAP 

Substations 

Figure 3-9 – Propositional Network for CDM phase four 

The results from the EAST analysis indicate that the key nodes in the Barking scenario 

network were the NOC operator and the SAP at Barking substation. The majority of the 

communications observed occurred between the NOC and the SAP, and the operational 

loading figures indicate that the SAP at Barking conducted the majority of the work 

required, on instruction from the NOC operator. Over half of all the tasks involved were 

identified as teamwork tasks, and a medium level of co-ordination was calculated 

between the agents involved. The EAST analysis indicated that NOC operator assumes 

the role of commander during the scenario, and all work is undertaken only on his 

instruction. 

Advantages 

• The EAST approach offers an exhaustive analysis of C4i activity. Activity 

is analysed using a number of different approaches, ensuring that a 

comprehensive analysis of the scenario under analysis is achieved. 

• The EAST output is extremely useful, offering a number of different 

analyses and perspectives on the C4i activity in question. Each component 

method offers a different representation or analysis of the C4i activity 

observed. For example, the SNA provides an indication of the key agents 

within the scenario and identifies communication links, the CDA identifies 

teamwork tasks and provides a co-ordination rating for each teamwork task 

step, and the propositional network component identifies the ideal 

knowledge (i.e. SA) required for effective task performance. 

22



Disadvantages 

• The methods within the EAST framework are conceptually simple, ensuring 

that the methodology requires minimal training and is relatively easy to 

apply. 

• The provision of WESTT and Agna software packages reduces the EAST 

application time considerably. 

• The EAST methodology is generic, allowing it to be applied in any domain 

where C4i activity takes place. To date, the EAST methodology has been 

used to analyse C4i activity in a number of diverse scenarios, including fire 

service training scenario’s (Baber et al 2004), military aviation scenarios 

(Stewart et al 2004a), railway signalling tasks (Walker et al 2004a), air 

traffic control scenarios (Walker et al 2004b), energy distribution scenarios 

(Salmon et al 2004) and naval warfare scenarios (Stewart et al 2004b). 

• The methodologies that make up EAST share a common theoretical 

background. This makes integration viable (and embodies a form of 

congruent validity). 

• There are possibilities for further recombination of methods to explore 

issues outside of the EAST framework (Walker et al 2005a). 

• All the methods have a validation history and prior context of use. 

• Due to the techniques exhaustiveness it is time consuming to apply. The 

initial observation, construction of the HTA and OSD are particularly time 

consuming elements of the methodology. 

• The data obtained during the CDM analysis a function of the analysts skill 

and the quality of the SME used. 

• Reliability of the technique in some areas is questionable. Much of the 

analysis is based upon the subjective judgement of the analyst involved, and 

so intra-analyst and inter analyst reliability may suffer. 

• For the technique to work properly, access to the system and personnel 

under analysis is required. It is often difficult to gain sufficient access to the 

system under analysis. Further, additional access to SME’s for data 

validation purposes is also difficult to obtain. 

• The methodology is still evolving and as a consequence, some EAST 

outputs may differ in format and presentation. 

• Without the provision of the appropriate software, an EAST analysis may 

become laborious to perform for larger, complex scenarios. 

• Some prior knowledge of HF methods is required. 

23



Related methods 

The EAST methodology is comprised of a number of HF methods, including HTA, 

observation, CDA, CUD, OSD’s, SNA, CDM and propositional networks. The authors 

have also developed the WESTT methodology, which can be used to automate a large 

portion of the EAST analysis procedure. 

Approximate training and application times 

The EAST methodology is conceptually simple, however due to the number of 

component methods involved, requires a lengthy training session. Some prior experience 

in the application of HF methods is required. For the recent applications of EAST 

described in this report, a training session lasting approximately 8 hours was given to the 

analysts involved. Due to the exhaustive nature of the EAST methodology, and the 

complex nature of C4i activity, the typical application time is high. The construction of 

the HTA and the OSD for C4i scenarios is a particularly lengthy process. Conducting an 

EAST analysis manually can take up to two weeks for large, complex scenarios. The 

application time is significantly reduced by the provision of the WESTT software 

package, which automates a large portion of the EAST analysis process. 

Reliability and validity 

Intra and inter reliability may be questionable in some instances. For example, different 

analysts may make different interpretations of the OSD symbols in the OSD glossary. 

The use of multiple methods within the EAST framework heightens the potential for 

unreliability. It is difficult to assess the validity of the data obtained during an EAST 

analysis. Typically, SMEs are used during data collection, and so a high level of validity 

is assumed. 

Congruent validity is favourable due to effective methods integration. Construct validity 

also good, as component methods relate well to the core descriptive constructs of C4i 

(Walker et al 2005b). Construct and face validity are also good because many of the 

features of interest are also observable and manifest in the environment of interest. The 

CDM complements this approach by tapping into the ‘implicit’ parts of the decision 

making process. 

Tools needed 

Due to the various component analyses involved in an EAST analysis, a number of 

different tools and software packages are required. For the observation and CDM phases 

of an EAST analysis, video and audio recording equipment is required. A word 

processing software package such as Microsoft Word is required to transcribe the 

observation and CDM data. The CDA is typically conducted using Microsoft Excel. A 

large part (e.g. OSD and SNA) of the EAST procedure can be automated by using the 

WESTT software package. Alternatively, the OSD CUD, and propositional networks are 

typically drawn using Microsoft Visio. The SNA is conducted using the Agna SNA 

software package. 

24



Flowchart 

START 

Define scenarios 

Conduct HTA for 

scenario under 

analysis 

Plan observation 

Conduct observation 

and CDM analysis 

Create scenario 

transcript 

Re-iterate HTA 

Conduct CDA 

Construct CUD 

Conduct SNA 

Construct OSD 

Construct Prop 

networks 

STOP 

25



3.2 Observation 


Observational techniques are used to gather data regarding the physical and verbal 

aspects of a particular task or scenario. Observational techniques are used to collect data 

regarding various aspects of system and task performance, such as data regarding the 

tasks catered for by the system, the individuals performing the tasks, the tasks themselves 

(task steps and sequence), errors made, communications between individuals, the 

technology used by the system in conducting the tasks (controls, displays, communication 

technology etc), the system environment and the organisational environment. 

Observation has been extensively used in the HF community, and typically forms the 

beginning of an analysis effort. 


Generic. 


Step 1: Define the objective of the analysis 

The first step in observational analysis involves defining the aims and objectives of the 

observation. This should include determining which product or system is under analysis, 

which environment the observation will take place, which user groups will be observed, 

what type of scenario’s will be observed and what data is required. Each point should be 

clearly defined and stated before the process continues. 

Step 2: Define the scenario(s) 

Once the aims and objectives of the analysis are clearly defined, the scenario(s) to be 

observed should be defined and described further. For example, when conducting an 

observational analysis of control room operation, which type of scenario is required 

should be clearly defined. Is normal operation under scrutiny or is the analysis focussed 

upon operator interaction and performance under emergency situations. The exact nature 

of the required scenario(s) should be clearly defined by the observation team. It is 

recommended that a HTA is conducted for the task or scenario under analysis. 

Step 3: Observation plan 

Once the aim of the analysis is defined and also the type of scenario to be observed is 

determined, the analysis team should proceed to plan the observation. The team should 

consider what they are hoping to observe, what they are observing, and how they are 

going to observe it. Depending upon the nature of the observation, access to the system 

in question should be gained first. This may involve holding meetings with the 

establishment in question, and is typically a lengthy process. Any recording tools should 

be defined and also the length of observations should be determined. Placement of video 

and audio recording equipment should also be considered. To make things easier, a 

walkthrough of the system/environment/scenario under analysis is recommended. This 

26



allows the analyst(s) to become familiar with the task in terms of time taken, location and 

also the system under analysis. 

Step 4: Pilot observation 

In any observational study a pilot or practice observation is crucial. This allows the 

analysis team to assess any problems with the data collection, such as noise interference 

or problems with the recording equipment. The quality of data collected can also be 

tested and also any effects of the observation upon task performance can be assessed. If 

major problems are encountered, the observation may have to be re-designed. Steps 1 to 4 

should be repeated until the analysis team are happy that the quality of the data collected 

will be sufficient for their study requirements. 

Step 5: Conduct Observation 

Once the observation has been designed, the team should proceed with the observation(s). 

Observation length and timing are dependent upon the scope and requirements of the 

analysis. Once the required data is collected, the observation should stop and step 6 

should be undertaken. 

Step 6: Create observation transcript 

Once the observation is complete, the analyst(s) should construct an observation 

transcript. An extract of an observation transcript from an observation of an energy 

distribution switching scenario (Salmon et al 2004b) is presented in Table 14. The 

observational transcript should contain a description of all activity observed during the 

analysis, including the agents involved, the time, the technology used and also any 

additional notes (e.g. purpose of activity). 

Step 7: Data analysis 

Once the observation is complete, and the observation transcripts have been constructed, 

the data analysis procedure can begin. Depending upon the analysis requirements, the 

team should analyse the data in the format that is required, such as frequency of tasks, 

verbal interactions, sequence of tasks etc. When analysing visual data, typically user 

behaviours are coded into specific groups. The software package Observer is used to 

aid the analyst in this process. 

27



Table 14 – Observation transcript extract (Source Salmon et al 2004b) 

Time Process Comms Location Notes 

09:54 GA & DW engage in general pre-amble about the 

forthcoming switching ops. Considering asking the 

Operations Centre in Wokingham to switch out SGT1A1B 

early so that SGT5 can be done at the same time 

10:15 Wokingham call Barking ask is they still want isolation – 

GA confirms yes. 

Person 

person 

Telephone 

to 

Barking 

275Kv 

switchhouse 

10:19 Switching phone rings throughout building. NOC Wants 

132Kv busbar opened [GA]. 

Green 

Telephone 

10:40 SGT1A1B waiting to be handed over to NOC. Delay. 

10:40 Wokingham report to Barking 275 [GA] complication with 

EDF. EDF want to reselect circuits at the last minute at 

another substation due to planned shutdown of SGT1A 

10:53 GA and DW discuss EDF problem. Can DW reconfigure 

circuits in Barking West (33Kv) 

Telephone Wokingham contact EDF to confirm switching (as they did with 

Barking 275 at 10:15). EDF report a problem. Wokingham pass 

this onto Barking 275 

Person to 

person 

10:53 GA contact Wokingham. Confusion as to what circuits need 

reconfiguring and who can do it. GA talks to DW at the 

same time. Decided that DW can reconfigure circuits. 

Wokingham give GA name and phone number of EDF 

contact 

10:58 GA and DW discuss plans for Barking West 33. Discuss 

who owns what, safety rules etc. Also discuss and decide 

order of subsequent site visits (might have to go to Barking 

West 33 twice) 

11:04 GA & DW Waiting to travel to Barking West 

GA/DW 

Person to 

Person, 

Wokingham 

Telephone 

This is an unplanned measure – now need to go to Barking West 

33Kv to reconfigure local electricity supply circuits 

28



Step 8: Further analysis 

Once the initial process of transcribing and coding the observational data is complete, 

further analysis of the data begins. Depending upon the nature of the analysis, 

observation data is used to inform a number of different HF analyses, such as task 

analysis, error analysis and communications analysis. Typically, observational data is 

used to develop a task analysis (e.g. HTA) of the task or scenario under analysis. 

Step 9: Participant feedback 

Once the data has been analysed and conclusions have been drawn, the participants 

involved should be provided with feedback of some sort. This could be in the form of a 

feedback session or a letter to each participant. The type of feedback used is determined 

by the analysis team. 

Example 

An observational analysis of a fire service training scenario was conducted as part of an 

analysis of existing C4i activity in civil domains (Baber et al 2004). Three observers 

observed the fire training service-training scenario, “Hazardous chemical spillage at 

remote farmhouse”. The three observers sat in on the exercise and recorded the 

discussion of the participants. The notes from the discussion were then collated into a 

combined timeline of the incident. This timeline, and the notes taken during the exercise, 

then formed the basis for the following HF analyses. 

Hierarchical task analysis 

Operator sequence diagram 

Social network analysis 

Co-ordination demand analysis 

Comms usage diagram 

Critical decision method 

The exercise involved a combination of focus group discussion with paired activity in 

order to define appropriate courses of action to deal with the specified incident. The class 

facilitator provided the initial description of an incident, i.e. a report of possible 

hazardous materials on a remote farm, and then added additional information as the 

incident unfolded, e.g. reports of casualties, problems with labelling on hazardous 

materials etc. The exercise was designed to encourage experienced fire-fighters to 

consider risks arising from hazardous materials and the appropriate courses of action they 

would need to take, e.g. in terms of protective equipment, incident management, 

information seeking activities etc. From the data obtained during the observation, an 

event flowchart was constructed, which acted as the primary input to the analysis 

techniques used. The event flowchart is presented in Figure 3-10. 

29



Police called to break-in 

at remote farmhouse 

Control seeks description 

and informs police 

officer 

Police officer proceeds 

to farmhouse 

Child admitted to 

hospital with respiratory 

problems 

Control inform Police 

officer hazardous 

materials on farm 

Hospital inform police of 

hazardous materials at 

farm 

Police control call fire 

brigade 

Police officer sets up 

outer cordon at farm 

Fire squadron proceed to 

farmhouse 

Fire brigade set up inner 

cordon at farm 

Hospital contact fire 

control requesting urgent 

diagnosis of chemical 

substance 

Fire brigade discuss 

protection options 

Fire brigade conduct 

search activity 

Locate chemical drums 

Chemical diagnosis 

Chemical diagnosis questioned as 

substance found is powder and not 

liquid 

Check UN number on 

chem.database 

Check UN number on 

Chemsafe database 

Contact supplier 

Identify substance 

Control inform hospital 

of diagnosis 

Begin decontamination 

process 

Figure 3-10 - Hazardous chemical spillage event flowchart 

30



Flowchart 

START 

Define study requirements 

Define scenario(s) to be observed 

Prepare/design observation 

Conduct pilot observation session 

Are there any 

problems 

Y 

Conduct observation of scenario(s) 

N 

For data analysis, choose from the following based on study/data 

requirements: 

• Transcribe scenario 

• Record task sequences 

• Record task times 

• Record any errors observed 

• Record frequency of tasks 

STOP 

31



Advantages 

Disadvantages 

• Observation technique data provides a ‘real life’ insight into man-machine, 

and team interaction. 

• Various data can be elicited from an observational study, including task 

sequences, task analysis, error data, task times, verbal interaction and task 

performance. 

• Observation has been used extensively in a wide range of domains. 

• Observation provides objective information. 

• Detailed physical task performance data is recorded, including social 

interactions and any environmental task influences (Kirwan & Ainsworth 

1992). 

• Observation is excellent for the initial stages of the task analysis procedure. 

• Observation analysis can be used to highlight problems with existing 

operational systems. It can be used in this way to inform the design of new 

systems or devices. 

• Specific Scenarios are observed in their ‘real world’ setting. 

• Observational data is used as the core input into numerous HF analyses 

techniques, such as human error identification techniques (SHERPA), task 

analysis (HTA), communications analysis (Comms Usage Diagrams), and 

charting techniques (operator sequence diagrams). 

• Observational techniques are intrusive to task performance. Knowing that 

they are being watched tends to elicit new and different behaviours in 

participants. For example, when observing control room operators, they 

may perform exactly as their procedures say they should. However, when 

not being observed, the same control room operator’s may perform 

completely differently, using short-cuts and behaviours that are not stated in 

their procedures. This may be due to the fact that the operator’s do not 

wish to be caught bending the rules in any way i.e. bypassing a certain 

procedure. This effect on behaviour can bias the observational data that is 

obtained. 

• Observational techniques are time consuming in their application, 

particularly the data analysis procedure. Kirwan & Ainsworth (1992) 

suggest that when conducting the transcription process, 1 hour of recorded 

audio data takes on analyst approximately 8 hours to transcribe. 

32




• Cognitive aspects of the task under analysis are not elicited using 

observational techniques. Verbal protocol analysis is more suited for 

collecting data on the cognitive aspects of task performance. 

• An observational study can be both difficult and expensive to set up and 

conduct. Gaining access to the required establishment is often extremely 

difficult and very time consuming. Observational techniques are also costly, 

as they require the use of expensive recording equipment (digital video 

camera, audio recording devices). 

• Causality is a problem. Errors can be observed and recorded during an 

observation but why the errors occur may not always be clear. 

• The analyst has a very low level of experimental control. 

• In most cases, a team of analysts is required to perform an observation 

study. It is often difficult to acquire a suitable team with sufficient 

experience in conducting observational studies. 

There are a number of different observational techniques, including indirect observation, 

participant observation and remote observation. Other related techniques include verbal 

protocol analysis, critical decision method, applied cognitive task analysis, walkthroughs 

and cognitive walkthroughs. All of these techniques require some sort of task 

observation. Observational data is also used as an input to numerous HF techniques, such 

as task analysis, human error identification, and charting techniques. 


Whilst the training time for an observational analysis is low (Stanton & Young 1999), the 

application time is normally high. The data analysis stage can be particularly time 

consuming. Kirwan & Ainsworth (1992) suggest that in the transcription process, 1 hour 

of audio recorded data would take approximately 8 hours to transcribe. 


Observational analysis is beset by a number of problems that can potentially affect the 

reliability and validity of the technique. According to Baber & Stanton (1996) problems 

with causality, bias (in a number of forms), construct validity, external validity and 

internal validity can all arise unless the correct precautions are taken. Whilst 

observational techniques possess a high level of face validity (Drury 1990) and ecological 

validity (Baber & Stanton 1996), analyst or participant bias can adversely affect the 

reliability and validity of the techniques. 

Tools needed 

For a thorough observational analysis, the appropriate visual and audio recording 

equipment is necessary. Observational studies can be conducted using pen and paper, 

33



however this is not recommended, as crucial parts of data are often not recorded. For the 

data analysis process, a PC with the Observer software is required. 

3.3 HTA – Hierarchical Task Analysis 

Background and Applications 

Hierarchical task analysis (HTA) (Annett 2004) is the most popular task analysis 

technique and has become perhaps the most widely used of all HF techniques. HTA 

involves describing the task under analysis through the break down of the task into a 

hierarchy of goals, sub goals, operations and plans. Goals are the unobservable task goals 

associated with the task in question, operations are the observable behaviours required to 

accomplish the task goals and plans are the unobservable decisions and planning made on 

behalf of the operator. The end result is an exhaustive description of task activity. HTA 

has been applied across a wide spectrum of domains, including the process control and 

power generation industries (Annett 2004), emergency services (Baber et al 2004) 

military applications (Kirwan & Ainsworth, 1992; Ainsworth & Marshall, 1998/2000), 

civil aviation (Marshall et al 2003), driving, public technology (Stanton and Stevenage 

1998) and even retail (shepherd 2001). One of the main reasons behind the enduring 

popularity of HTA is its flexibility and the scope for further analysis of the sub-goal 

hierarchy that it offers to the HF practitioner. The majority of HF analysis methods either 

require an initial HTA of the task(s) under analysis as their input, or at least are made 

significantly easier through the provision of a HTA. HTA acts as an input into numerous 

HF analyses techniques, such human error identification (HEI), allocation of function, 

workload assessment, interface design and evaluation and many more. In a review of 

ergonomics texts, Stanton (2004) highlights at least twelve additional applications to 

which HTA has been put, including interface design and evaluation, training, allocation 

of functions, job description, work organisation, manual design, job aid design, error 

prediction and analysis, team task analysis, workload assessment and procedure design. 


HTA was originally developed for the chemical processing and power generation 

industries (Annett 2004). However the technique is generic and can be applied in any 

domain. 


Step 1: Define task under analysis 

The first step in conducting a HTA is to clearly define the task(s) under analysis. As well 

as identifying the task under analysis, the purpose of the task analysis effort should also 

be defined. For example, Marshall et al (2003) conducted a HTA of a civil aircraft 

landing task in order to predict design induced error for the flight task in question. 

Step 2: Data collection process 

Once the task under analysis is clearly defined, specific data regarding the task should be 

collected. The data collected during this process is used to inform the development of the 

34



HTA. Data regarding the task steps involved, the technology used, interaction between 

man and machine and team members, decision making and task constraints should be 

collected. There are a number of ways to collect this data, including observations, 

interviews, and questionnaires. The technique used is dependent upon the analysis effort 

and the various constraints imposed, such as time and access constraints. Once sufficient 

data regarding the task under analysis is collected, the development of the HTA should 

begin. 

Step 3: Determine the overall goal of the task 

The overall task goal of the task under analysis should first be specified at the top of the 

hierarchy i.e. ‘Land aircraft X at New Orleans Airport using the Auto-land system’ 

(Marshall et al 2003), ‘Boil kettle’, or ‘Listen to in-car entertainment’ (Stanton and 

Young 1999). 

Step 4: Determine task sub-goals 

Once the overall task goal has been specified, the next step is to break this overall goal 

down into meaningful sub-goals (usually four or five but this is not rigid), which together 

form the tasks required to achieve the overall goal. In the task, ‘Land aircraft X at New 

Orleans Airport using the Auto-land system’ (Marshall et al 2003), the overall goal of 

landing the aircraft was broken down into the sub-goals, ‘Set up for approach’, ‘Line up 

aircraft for runway’ and ‘Prepare aircraft for landing’. In a HTA of a Ford in-car radio 

(Stanton & Young 1999) the overall task goal, “Listen to in car entertainment”, was 

broken down into the following sub-goals, ‘Check unit status’, ‘Press on/off button’, 

‘Listen to the radio’, ‘Listen to cassette’, and ‘Adjust audio preferences’. 

Step 5: Sub-goal decomposition 

Next, the analyst should break down the sub-goals identified in step four into further subgoals 

and operations, according to the task step in question. This process should go on 

until an appropriate operation is reached. The bottom level of any branch in a HTA will 

always be an operation. Whilst everything above an operation specifies goals, operations 

actually say what needs to be done. Thus operations are actions to be made by the 

operator in order to achieve the associated goal. For example, in the HTA of the flight 

task ‘Land aircraft X at New Orleans Airport using the Auto-land system’ (Marshall et al 

2003), the sub-goal ‘Reduce airspeed to 210 Knots’ is broken down into the following 

operations, ‘Check current airspeed’ and ‘Dial the Speed/MACH selector knob to enter 

210 on the IAS/MACH display’. 

Step 6: Plans analysis 

Once all of the sub-goals and operations have been fully described, the plans need to be 

added. Plans dictate how the goals are achieved. A simple plan would say Do 1, then 2, 

and then 3. Once the plan is completed, the operator returns to the super-ordinate level. 

Plans do not have to be linear and can come in any form such as Do 1, Or 2 and 3. The 

different types of plans used are presented in Table 15. 

35



Table 15 - Example HTA plans 

Plan 

Example 

Linear Do 1 then 2 then 3 

Non-linear 

Do 1, 2 and 3 in any order 

Simultaneous 

Do 1, then 2 and 3 at the same time 

Branching 

Do 1, if X present then do 2 then 3, if X is not 

present then EXIT 

Cyclical 

Selection Do 1 then 2 or 3 

Do 1 then 2 then 3 and repeat until X 

Advantages 

Disadvantages 

• HTA is a technique that is both easy to learn and easy to implement. 

• The output of a HTA is extremely useful and forms the input for numerous 

HF analyses, such as error analysis, interface design and evaluation and 

allocation of function analysis. 

• Quick to use in most instances. 

• Comprehensive technique covers all sub-tasks of the task in question. 

• HTA has been used extensively in a wide range of contexts. 

• Conducting an HTA gives the user a great insight into the task under 

analysis. 

• HTA is an excellent technique to use when requiring a task description for 

further analysis. If performed correctly, the HTA should depict everything 

that needs to be done in order to complete the task in question. 

• The technique is generic and can be applied to any task in any domain. 

• Tasks can be analysed to any required level of detail, depending on the 

purpose. 

• Provides mainly descriptive information rather than analytical information. 

• HTA contains little that can be used directly to provide design solutions. 

• HTA does not cater for the cognitive components of the task under analysis. 

36



Related Methods 

• The technique may become laborious and time consuming to conduct for 

more complex and larger tasks. 

• The initial data collection phase is time consuming and requires the analyst 

to be competent in a variety of HF techniques, such as interviews, 

observations and questionnaires. 

• The reliability of the technique may be questionable in some instances. For 

example, for the same task, different analysts may produce very different 

sub-goal hierarchies. 

• HTA has often been labelled as an art rather than a science. 

• An adequate software version of HTA has yet to emerge. 

HTA is widely used in HF and often forms the first step in a number of analyses, such as 

HEI, HRA and mental workload assessment. In a review of ergonomics texts, Stanton 

(2004) highlights at least twelve additional applications to which HTA has been put, 

including interface design and evaluation, training, allocation of functions, job 

description, work organisation, manual design, job aid design, error prediction and 

analysis, team task analysis, workload assessment and procedure design. As a result 

HTA is perhaps the most commonly used HF method, and is typically used as the start 

point or basis of any HF analysis. 

Approximate Training and Application Times 

The training time for the HTA method is largely dependent upon on the skill of the 

analysts involved. It is often said that using HTA is more of an art than a science, and the 

training time required reflects this. Some analysts may pick up the method immediately, 

whilst others may use the technique repeatedly without becoming skilled in its use. It is 

therefore estimated that the training time for HTA is high. A survey by Ainsworth & 

Marshall (1998/2000) found that the more experienced practitioners produced more 

complete and acceptable analyses. Stanton & Young (1999) report that the training and 

application time for HTA is substantial. The application time associated with HTA is 

dependent upon the size and complexity of the task under analysis. For large, complex 

tasks, the application time for HTA would be high. 

Reliability and Validity 

According to Annett (2004), the reliability and validity of HTA is not easily assessed. 

From a comparison of twelve HF techniques, Stanton & Young (1999) reported that, the 

technique achieved an acceptable level of validity but a poor level of reliability. The 

reliability of the technique is certainly questionable. It seems that different analysts, with 

different experience may produce entirely different analyses for the same task (intraanalyst 

reliability). Similarly, the same analyst may produce different analyses on 

different occasions for the same task (inter-analyst reliability). 

37



Tools needed. 

HTA can be carried out using only pencil and paper. The HTA output can be developed 

and presented in a number of software applications, such as Microsoft Visio, Microsoft 

Word and Microsoft Excel. A number of HTA software tools also exist, such as the HFIT 

DTC’s HTA tool. 

Flowchart 

START 

State overall goal 

State subordinate operations 

Select next operation 

State plan 

Check the adequacy of 

rediscription 

Revise 

rediscription 

Is redescription 

ok 

Consider the first/next 

suboperation 

Is further 

redescription 

required 

Terminate the redescription of this 

operation 


more 

operations 

STOP 

38



3.4 CDA – Co-Ordination Demands Analysis 

Background and application 

Co-ordination demands analysis (CDA) is used to rate the co-ordination between agents 

involved in teamwork activity. CDA is used to identify the extent to which team 

members have to work with each other in order to accomplish the task(s) under analysis. 

The CDA procedure allows analysts to identify the operational skills required within 

team tasks, but also the teamwork skills needed for smooth coordination among team 

members. In its present usage teamwork skills are extracted from a HTA (Annett 2004) 

and rated on a scale of 1 (low) to 3 (high) against each behaviour in the co-ordination 

behaviour taxonomy presented in Table 16 (Source: Burke 2004). From these individual 

ratings a ‘total coordination’ figure for each teamwork task step is derived. 

Table 16 - CDA teamwork taxonomy (Source: Burke 2004) 

Coordination 

Dimension 

Communication 

Situational 

Awareness (SA) 

Decision Making 

(DM) 

Mission analysis 

(MA) 

Leadership 

Adaptability 

Assertiveness 

Total Coordination 

Definition 

Includes sending, receiving, and acknowledging information among 

crewmembers. 

Refers to identifying the source and nature of problems, maintaining an 

accurate perception of the aircraft’s location relative to the external 

environment, and detecting situations that require action 

Includes identifying possible solutions to problems, evaluating the 

consequences of each alternative, selecting the best alternative, and 

gathering information needed prior to arriving at a decision 

Includes monitoring, allocating, and coordinating the resources of the crew 

and aircraft; prioritizing tasks; setting goals and developing plans to 

accomplish the goals; creating contingency plans. 

Refers to directing activities of others, monitoring and assessing the 

performance of crew members, motivating members, and communicating 

mission requirements 

Refers to the ability to alter one’s course of action as necessary, maintain 

constructive behaviour under pressure, and adapt to internal or external 

changes. 

Refers to the willingness to make decisions, demonstrating initiative, and 

maintaining one’s position until convinced otherwise by facts. 

Refers to the overall need for interaction and coordination among crew 

members 


The CDA technique is generic and can be applied to any task that involves teamwork or 

collaboration. 

39




Step 1: Define task(s) under analysis 

The first step in a CDA is to define the task or scenario that will be analysed. This is 

dependent upon the focus of the analysis. It is recommended that if team coordination in 

a particular type of system (e.g. command and control) is under investigation, then a set 

of scenarios representative of team performance in the system should be used. If time 

and financial constraints do not allow this, then a task that is as representative as possible 

of team performance in the system under analysis should be used. 

Step 2: Select appropriate teamwork taxonomy 

Once the task(s) under analysis are defined, an appropriate teamwork taxonomy should 

be selected. Again, this may depend upon the purpose of the analysis. However, it is 

recommended that the taxonomy used covers all aspects of teamwork in the task under 

analysis. 

Step 3: Data collection phase 

Typically, data regarding the task(s) under analysis is collected using observation and/or 

interviews. Specific data regarding the task under analysis should be collected during 

this process, including information regarding each task step, each team member’s roles, 

and communications made. It is recommended that video and audio recording equipment 

are used to record any observations or interviews conducted during this process. 

Step 4: Conduct a HTA for the task under analysis 

Once sufficient data regarding the task under analysis has been collected, a HTA should 

be conducted. The HTA should represent a breakdown of the task under analysis in 

terms of goals, sub-goals, operations and plans. 

Step 5: Construct CDA rating sheet 

Once a HTA for the task under analysis is completed, a CDA rating sheet should be 

created. The rating sheet should include a column containing each bottom level task step 

as identified by the HTA. The teamwork behaviours from the taxonomy should run 

across the top if the table. An extract of a CDA rating sheet is presented in Table 17. 

Step 6: Taskwork/Teamwork classification 

Once the CDA rating sheet is complete, the analyst(s) should then determine which of the 

bottom level task steps from the HTA involve taskwork and which involve teamwork. 

Only those task steps defined as teamwork task steps should be rated according to the 

teamwork taxonomy. Those task steps that are conducted individually involving no 

collaboration are classified as taskwork, whilst those task steps that are conducted 

collaboratively, involving more than one agent are classified as teamwork. 

Step 7: SME rating phase 

40



Appropriate SME’s should then rate the extent to which each teamwork behaviour is 

required during the completion of each teamwork task step. This involves presenting the 

task step in question and discussing the role of each taxonomy behaviour in the 

completion of the task step. An appropriate rating scale should be used e.g. low (1), 

medium (2) and high (3). 

Step 8: Calculate summary statistics 

Once all of the teamwork task steps have been rated according to the teamwork 

taxonomy, the final step is to calculate appropriate summary statistics. In its present 

usage, a total co-ordination value and mean co-ordination value for each teamwork task 

step are calculated. The mean co-ordination is simply an average of the ratings for the 

teamwork behaviours for the task step in question. A mean overall co-ordination value 

for the entire scenario is also calculated. 

Example 

The following example is an extract of a CDA analysis of an energy distribution task 

(Salmon et al 2004b). The task involved the switching out of three circuits (SGT5 and 

SGT1A and 1B) at Barking 275Kv, 132Kv and 33Kv Substations. Circuit SGT5 was 

being switched out for the installation of a new transformer for the nearby channel tunnel 

rail link and SGT1A and 1B were being switched out for substation maintenance. 

Observational data from Barking substation and the NOC control room was used to 

conduct a HTA of the switching scenario. Each bottom level task in the HTA was then 

defined by the analyst(s) as either taskwork or teamwork. Each teamwork task was then 

rated using the CDA taxonomy on a scale of 1 (low) to 3 (high). An extract of the HTA 

for the task is presented in Figure 3-11. An extract of the CDA is presented in Table 17. 

The overall CDA results are presented in Table 18. 

NGC Switching operations HTA 

0. Co-ordinate and carry out switching operations on circuits SGT5. SGT1A and 1B at 

Bark s/s (Plan 0. Do 1 then 2 then 3, EXIT) 

1. Prepare for switching operations (Plan 1. Do 1.1, then 1.2, then 1.3, then 1.4, then 1.5, 

then 1.6, then 1.7, then 1.8, then 1.9,then 1.10 EXIT) 

1.1. Agree SSC (Plan 1.1. Do 1.1.1, then 1.1.2, then 1.1.3, then 1.1.4, then 1.1.5, EXIT) 

1.1.1. (WOK) Use phone to Contact NOC 

1.1.2. (WOK + NOC) Exchange identities 

1.1.3. (WOK + NOC) Agree SSC documentation 

1.1.4. (WOK+NOC) Agree SSC and time (Plan 1.1.4. Do 1.1.4.1, then 1.1.4.2, 

EXIT) 

1.1.4.1. (NOC) Agree SSC with WOK 

41



1.1.4.2. (NOC) Agree time with WOK 

1.1.5. (NOC) Record and enter details (Plan 1.1.5. Do 1.1.5.1, then 1.1.5.2, EXIT) 

1.1.5.1. Record details on log sheet 

1.1.5.2. Enter details into worksafe 

1.2. (NOC) Request remote isolation (Plan 1.2. Do 1.2.1, then 1.2.2, then 1.2.3, then 

1.2.4, EXIT) 

1.2.1. (NOC) Ask WOK for isolators to be opened remotely 

1.2.2. (WOK) Perform remote isolation 

1.2.3. (NOC) Check Barking s/s screen 

1.2.4. (WOK + NOC) End communications 

1.3. Gather information on outage at transformer 5 at Bark s/s 

(Plan 1.3. Do 1.3.1, then 1.3.2, then 1.3.3, then 1.3.4, EXIT) 

1.3.1. (NOC) Use phone to contact SAP at Bark 

1.3.2. (NOC + SAP) Exchange identities 

Figure 3-11 - Extract of HTA for NGT Switching scenario 

42



Table 17 - CDA extract (Source: Salmon et al 2004) 

Task Step Agent Step No. Task 

Work 

Team 

Work 

Comm SA DM MA Lead Ad Ass TOT 

CO-ORD 

Mode 

TOT 

CO-ORD 

Mean 

1.1.1 WOK control room Use phone to contact NOC 1 

operator 

1.1.2 WOK control room Exchange identities 1 3 3 1 1 1 1 1 1.00 1.57 

operator 


1.1.3 WOK control room Agree SSC documentation 1 3 3 3 1 1 1 1 1.00 1.86 

operator 


1.1.4.1 NOC control room Agree SSC with WOK 1 3 3 3 1 1 1 1 1.00 1.86 

operator 

1.1.4.2 NOC control room Agree time with WOK 1 3 3 3 1 1 1 1 1.00 1.86 

operator 

1.1.5.1 NOC control room Record details onto log sheet 1 

operator 

1.1.5.2 NOC control room Enter details into WorkSafe 1 

operator 

1.2.1 NOC control room Ask for isolators to be opened remotely 1 3 3 1 2 2 1 1 1.00 1.86 

operator 

1.2.2 WOK control room Perform remote isolation 1 

operator 

1.2.3 NOC control room Check barking s/s screen 1 

operator 

1.2.4 WOK control room End communications 1 3 1 1 1 1 1 1 1.00 1.29 

operator 


1.3.1 NOC control room Use phone to contact SAP at Barking 1 

operator 

1.3.2 NOC control room Exchange identities 1 3 3 1 1 1 1 1 1.00 1.57 

operator 


43



Table 18 - CDA results (Source: Salmon et al 2004) 

Category 

Result 

Total task steps 314 

Total taskwork 114 (36%) 

Total teamwork 200 (64%) 

Mean Total Co-ordination 1.57 

Modal Total Co-ordination 1.00 

Minimum Co-ordination 1.00 

Maximum Co-ordination 2.14 

Advantages 

Disadvantages 

• Offers a rating of co-ordination between team members for each teamwork 

based task step. 

• The output of a CDA is very useful, offering an insight into the use of 

teamwork behaviours and also a rating of team coordination and its 

components. 

• CDA is particularly useful for the analysis of C4i activity. 

• The teamwork taxonomy presented by Burke (2004) covers all aspects of 

team performance and coordination. The taxonomy is also generic, allowing 

the technique to be used in any domain without modification. 

• CDA provides a breakdown of team performance in terms of task steps and 

the level of co-ordination required. 

• The technique is generic and can be applied to teamwork scenarios in any 

domain. 

• The CDA rating procedure is time consuming and laborious. The initial 

data collection phase and the creation of a HTA for the task under analysis 

also add further time to the analysis. 

44




• For the technique to be used properly, the appropriate SME’s are required. 

It may be difficult to gain access to SME’s for the required period of time. 

• Intra analyst and inter analyst reliability is questionable. Different SME’s 

may offer different teamwork ratings for the same task (intra analyst 

reliability), whilst SME’s may provide different ratings on different 

occasions. 

In conducting a CDA analysis, a number of other HF techniques are used. Data regarding 

the task under analysis is typically collected using observations and interviews. A HTA 

for the task under analysis is normally conducted, the output of which feeds into the CDA 

rating sheet. A likert style rating scale is also normally used during the team behaviour 

rating procedure. Burke (2004) also suggests that a CDA should be conducted as part of 

an overall team task analysis procedure. The CDA technique has also recently been 

integrated with a number of other techniques (HTA, observation, comms usage diagram, 

social network analysis, operator sequence diagrams and propositional networks) to form 

the event analysis of systemic teamwork (EAST) methodology, which has been used to 

analyse C4i activity in a number of domains. 


The training time for the CDA technique is minimal, requiring only that the SME’s used 

understand each of the behaviours specified in the teamwork taxonomy and also the 

rating procedure. The application time is medium to high, depending upon the size and 

complexity of the scenario under analysis. In the CDA provided in the analysis, the 

ratings procedure alone took approximately four hours. This represents a medium 

application time. 


There are no data regarding the reliability and validity of the technique available in the 

literature. Certainly both the intra analyst and inter analyst reliability of the technique 

may be questionable, and this may be dependent upon the type of rating scale used e.g. it 

is estimated that the reliability may be low when using a scale of 1-10, whilst it may be 

improved using a scale of one to three (low to high). 

Tools needed 

During the data collection phase, video (e.g. camcorder) and audio (e.g. recordable minidisc 

player) recording equipment are required in order to make a recording of the task or 

scenario under analysis. Once the data collection phase is complete, the CDA technique 

can be conducted using pen and paper. 

Flowchart 

45



START 

Define the task(s) 

under analysis 

Data collection phase 

Conduct a HTA for the 

task under analysis 

Create CDA rating 

sheet 

Take the first/next step 

on the rating sheet 

Rate following behaviours on 

scale of 1 (low) to 3 (high): 

• Communication 

• Situation 

awareness 

• Decision making 

• Mission analysis 

• Leadership 

• Adaptability 

• Assertiveness 

Calculate the total 

coordination involved 

in the task step 

Are 

there any 

more 

STOP 

46



3.5 CUD – Comms Usage Diagram 


Comms Usage Diagram (CUD) (Watts & Monk 2000) is used to describe collaborative 

activity between teams of agents dispersed across different geographical locations. A 

CUD output describes the order of communications during the scenario under analysis, 

how and why communications between agents occur, which technology is involved in the 

communication, and the advantages and disadvantages of the technology used. The CUD 

technique was originally developed and applied in the telecommunications domain, 

whereby the technique was used to analyse ‘telemedical consultation’ involving a 

medical practitioner offering advice regarding a medical ailment from a different location 

to the advice seeker (Watts & Monk 2000). The technique has more recently been used 

by the authors in the analysis of C4i activity in a number of domains, including energy 

distribution, naval warfare, fire services, air traffic control, military, rail and aviation 

domains (see example section). A CUD analysis is typically based upon observational 

data of the task or scenario under analysis, although talk through analysis and interviews 

are also used (Watts and Monk 2000). 


Although the technique was originally developed for use in the medical domain, it is 

generic and can be applied in any domain that involves distributed collaboration. 


Step 1: Define the task or scenario under analysis 

The first step in a CUD analysis is to clearly define the task or scenario under analysis. It 

may be useful to conduct a HTA of the task under analysis for this purpose. A clear 

definition of the task under analysis allows the analyst(s) to prepare for the data 

collection phase. 

Step 2: Data collection 

Next, the analyst(s) should collect specific data regarding the task or scenario under 

analysis. Watts & Monk (2000) recommend that interviews, observations and task talkthrough 

should be used to collect the data. Specific data regarding the personnel 

involved, activity, task steps, communication between personnel, technology used and 

geographical location should be collected. 

Step 3: Complete Initial Comms report 

Following the data collection phase, the raw data obtained should be put into a report 

form. According to Watts & Monk (2000) the report should include the location of the 

technology used, the purpose of the technology, the advantages and disadvantages of 

using such technology, and graphical account of a typical consultation session. The 

report should then be made available to all personnel involved for evaluation and 

reiteration purposes. 

47



Step 4: Construct CUD output table 

The graphical account developed in step 3 forms the basis for the CUD output. First of 

all, the analyst should construct an appropriate CUD template, containing a column for 

each agent involved in the scenario, and columns for the technology used, the advantages 

and disadvantages associated with the technology used and the recommended technology. 

An example CUD template is presented in Figure 3-12. The CUD output contains a 

description of the task activity at each geographical location and the collaboration 

between personnel at each location. Directional arrows (i.e. from/to) should then be used 

to represent the communications between personnel at different locations. The comms 

column specifies the technology used in the communication and the effects column lists 

any advantages and disadvantages associated with the technology in question. This is 

based upon the initial observation of the scenario and also analyst subjective judgement. 

Finally, the recommended comms column is used to provide a recommendation of the 

most suitable technology available for the comms in question. 

NGT Feckenham Switching Operations 12 th July 2004 

Com ms Usage Diagram 

TIME 

NOC control room 

operator 

SAP/AP at 

Tottenham 

Substation 

SAP/AP at 

W altham C ross 

Substation 

SAP/AP at 

Brimsdown 

substation 

Overhead Line 

Party contact 

W OK control 

room operator 

Comms 

Effects of comms 

medium used 

Recommended 

comms 

14:48 

15:00 

15:03 

15:32 

Step 5: Calculate comms technology figures 

Figure 3-12 - CUD template 

In order to summarise the CUD analysis, the frequency of communications made by each 

agent and the comms technology used is calculated. An example CUD summary table is 

presented in Table 19. 

Advantages 

• The CUD output is particularly useful, offering a description of the task 

under analysis, and also a description of collaborative activity involved, 

including the order of activity, the communications between agents, the 

personnel involved, the technology used and its associated advantages and 

disadvantages, and recommendations regarding the technology used. 

48



Disadvantages 

Example 

• The output of a CUD analysis is particularly useful for highlighting 

communication flaws in a particular network of agents. 

• The CUD technique is particularly suited to the analysis of teamwork and 

C4i activity. 

• The CUD technique is also flexible, and could potentially be modified to 

make it more comprehensive. Factors such as time, error and workload 

could be incorporated into a CUD analysis, ensuring a much more 

exhaustive analysis. 

• Although the CUD technique was developed and originally used in the 

medical domain, it is a generic technique and could potentially be applied in 

any domain involving distributed collaboration. 

• The technique is easy to use and quick to learn. 

• For large, complex tasks involving many agents, conducting a CUD 

analysis may become a lengthy and laborious process. 

• In its present usage, the CUD analysis only defines the communications 

technology used at the source of the communication in question (i.e. the 

comms resource at the other end of the communication in question is not 

defined). 

• The initial data collection phase of the CUD technique is time consuming 

and labour intensive. 

• No validity or reliability data are available for the technique. 

• Application of the CUD technique appears to be limited. 

• No guidelines are offered to the analyst in analysing the technology used in 

the communications under analysis. Typically this is based upon the 

analyst’s subjective judgement. This may affect the reliability of the 

technique. 

The following example is a CUD analysis of an energy distribution task. The task 

involved the return from isolation of the Brimsdown -Tottenham-Waltham Cross Circuit 

(Salmon et al 2004c). The data collection phase involved two observers. The first 

observer was situated at the National Grid Transco (NGT) National Operations Centre 

(NOC) in Warwick and observed the activity of the NOC control room operator. The 

second observer was situated at the Tottenham substation and observed the activity of the 

senior authorised person (SAP) and authorised person (AP) who completed work required 

to return the circuit from isolation. The CUD analysis used observational data and a HTA 

49



of the task as its primary inputs. An extract of the CUD analysis for the energy 

distribution task is presented in Figure 3-13. In order to summarise the CUD analysis, 

comms frequency figures were calculated. The CUD summary results are presented in 

Table 19. 

NGT Feckenham Switching Operations 12 th July 2004 

TIME 

NOC control room 

operator 

SAP/AP at 

Tottenham 

Substation 

SAP/AP at 

W altham Cross 

S ubstation 

SAP/AP at 

Brimsdown 

substation 

Overhead Line 

Party contact 

WOK control 

room operator 

Comms 

Comms Usage Diagram 

Effects of comms medium 

used 

R ecomm ended 

comms 

08:50 

C ontact N OC 

control room 

operator to ask 

w hen deearthing 

ops can 

begin 



+ Can look at site diagram / 

desktop screen at the sam e 

time 


- D ata is not recorded. 

Information may be lost 

09:04 

Contact SAP to 

issue first 

instructions 





time 




09:05 

Contact SAP to 

discuss 

forthcoming 

operations 





time 




09:45 

Contact NOC to 

report las t 

instructions 

c omplete and to 

ask for next 

instructions 





time 




10:00 

C ontact OLP to 

discuss 

forthcoming 

operations 





time 




Figure 3-13 - Comms Usage Diagram for an energy distribution task (Salmon et al 

2004) 

Table 19 - CUD summary table 

Agent 

Telephon 

e comms 

Mobile phone 

comms 

PC comms 

Total 

NOC operator 11 11 

SAP/AP 

Tottenham 

SAP/AP 

Waltham Cross 

SAP/AP 

Brimsdown 

at 

at 

at 

8 1 9 

5 5 

7 7 

Overhead 

party contact 

line 

1 1 

WOK operator 1 1 

Total 32 2 34 

50




During the data collection phase, a number of different techniques may be used, such as 

observational analysis, interviews and talk-through type analysis. It is also useful to 

conduct a HTA of the task under analysis prior to constructing the CUD. The CUD 

technique has also recently been integrated with a number of other techniques (HTA, 

observation, co-ordination demands analysis, social network analysis, operator sequence 

diagrams and propositional networks) to form the event analysis of systemic teamwork 

(EAST) methodology, which has been used to analyse C4i activity. 

Flowchart 

START 

Define the task or scenario 


Use observation and 

interviews to collect 

specific task data 

Transcribe the raw data 

into report form 

Construct comms usage 

diagram 

Calculate comms and 

technology frequency 

figures 

STOP 


The training time associated with CUD is very low, assuming that the practitioner was 

already proficient in data collection techniques such as interviews and observational 

analysis. In a recent training programme for the EAST methodology, the CUD training 

session lasted under 1 hour. The application time associated with the CUD technique is 

also typically low to medium, depending upon task size and complexity. In the analyse of 

51



energy distribution scenarios (Salmon et al 2004a, b, c) the application time for CUD 

ranged between 1 and 4 hours, depending upon the size and complexity of the scenario 

under analysis. 


No data regarding the reliability and validity of the technique are available. 

Tools needed 

A CUD analysis can be conducted using pen and paper only. However, it is 

recommended that video and audio recording equipment are used during the data 

collection phase. The CUD can be constructed using Microsoft Visio. 

3.6 Social Network Analysis 


Social Network Analysis (SNA) is used to analyse and represent the relationships 

between teams or agents within a social network. A social network is defined as a set or 

team of agents that possess relationships with one another (Driskell & Mullen 2004). 

SNA is based upon the notion that the relationship between agents within a social 

network has a significant effect upon the actions performed and also the performance 

achieved by the network. SNA uses both graphical and mathematical procedures to 

represent social networks. Typically, centrality measures are calculated for each agent 

(e.g. degree, betweenness and closeness) and the overall network density is calculated. 

This allows the identification of the key agents within the network and also the 

classification of the network structure. The technique has previously been used for the 

analysis of networks in a number of areas, including military C4ISR architectures 

(Dekker 2002), C4i activity in the fire service (Baber et al 2004), energy distribution 

(Salmon et al 2004), air traffic control, rail (Walker et al 2004), naval warfare (Stewart et 

al 2004) and military aviation domains, counselling and computer-mediated 

communications (Dekker 2002). 


Generic. 


Step 1: Define network or group 

The first step in a SNA involves defining the network or group of networks that are to be 

analysed. The type of network to be analysed should be specified first. For example, for 

work package 1.1, the authors specified C4i networks as the focus of their analysis. The 

purpose of this was the evaluation of C4i activity with a view to developing a generic 

model of C4i. Once the overall network type is specified, further social networks within 

the network type specified should be defined. For the analysis of C4i networks, the 

authors identified a number of different C4i networks that could be analysed. These were 

52



C4i networks within the emergency services, rail, military, aviation, nuclear reprocessing, 

air traffic control, civil energy distribution, and naval domains. 

Step 2: Define scenarios 

Typically, networks are analysed over a number of different scenarios. Once the type of 

network under analysis has been defined, the scenario(s) within which they will be 

analysed should be defined. For a thorough analysis, a number of different scenarios 

should be analysed. The scenarios used should be representative of all C4i activity within 

the network under analysis. In the analysis of naval warfare C4i (Stewart et al 2004), the 

following scenarios were analysed: 

• Air threat scenario 

• Surface threat scenario 

• Subsurface threat scenario 


Once the network and scenario(s) under analysis are defined clearly, the data collection 

phase can begin. The data collection phase involves the collection of specific data on the 

relationship (e.g. communications and activity) between the agents during the scenario. 

Typical HF data collection techniques should be used in this process, such as 

observational analysis, interviews and questionnaires. Typically agent activity and the 

frequency, direction and content of any communications between agents in the network 

are recorded. 

Step 4: Construct agent association matrix 

Once sufficient data regarding the scenario under analysis is collected, an agent 

association matrix should be constructed. The matrix represents the frequency of 

associations between each agent in the network. 

Step 5: Construct social network diagram 

Once the matrix of association is completed, the social network diagram should be 

created. The social network depicts each agent in the network and the communications 

between them. The communications are represented by directional arrows, and the 

frequency of communications is also presented. An example social network diagram is 

presented in Figure 3-14. 

Step 6: Calculate agent centrality 

Agent centrality is calculated in order to determine the central or key agent(s) within the 

network. There are a number of different centrality calculations that can be made. For 

example, agent centrality can be calculated using Bavelas-Leavitt’s index. The mean 

centrality + standard deviation can then be used to define key agents within the network. 

Those agents who posses a centrality figure that exceeds the mean + standard deviation 

figure are defined as the key agents within the network. 

53



Step 7: Calculate sociometric status 

The sociometric status of each agent refers to the number of communications received 

and emitted, relative to the number of nodes in the network. The mean sociometric status 

+ standard deviation can also be used to define key agents within the network. Those 

agents who posses a sociometric status figure that exceeds the mean + standard deviation 

figure can be defined as the key agents within the network. 

Step 8: Calculate network density 

Network density is equal to the total number of links between the agents in the network 

divided by the total number of possible links. Low network density figures are indicative 

of a well distributed network of agents. High density figures indicate a network that is not 

well distributed. 

Advantages 

• SNA can be used to determine the importance of different agents within a 

social network and also to classify the network type. 

• The SNA offers a comprehensive analysis of the network in question. The 

key agents within the network are specified, as are the frequency and 

direction of communications within the network. Further classifications 

include network type and network density. There are also additional 

analyses that can be calculated, such as betweenness, closeness and distance 

calculations. 

• Networks can be classified according to their structure. This is particularly 

useful when analysing networks across different domains. 

• SNA is particularly suited to the analysis of C4i scenarios. 

• The technique has been used extensively in the past for the analysis of 

various social networks. 

• The technique is simple to learn and easy to use. 

• SNA is a very quick technique to apply. The provision of the Agna SNA 

software package also reduces application time further. 

• SNA is a generic technique that could potentially be applied in any domain. 

Disadvantages 

54



Example 

• For large, complex networks, it may be difficult to conduct a SNA. 

Application time is a function of network size, and large networks may 

incur lengthy application times. 

• The data collection phase involved in a SNA is also resource intensive. 

• Some knowledge of mathematical techniques is required. 

• It is difficult to collect comprehensive data for a SNA. For example, a 

dispersed network of ten agents would require at least 10 observers in order 

to accurately capture the communications made between all agents. 

The following example is taken from an EAST analysis of a civil energy distribution 

scenario (Salmon et al 2004). The scenario involved the return to service of the 

Brimsdown-Tottenham-Waltham Cross Circuit and took place at Brimsdown, Tottenham, 

Waltham Cross substations and the National Grid Transco (NGT) Network Operations 

Centre (NOC) in Warwick. The agents involved in the scenario are presented in Table 20. 

Table 20 - Agents involved in the return to service scenario 

Role of agent A NOC control room operator 

Role of agent B SAP/AP at Tottenham substation 

Role of agent C SAP/AP at Waltham Cross 

substation 

Role of agent D SAP/AP at Brimsdown substation 

Role of agent E Overhead line party contact 

Role of agent F WOK control room operator 

From the list of agents identified for the scenario, a matrix of association can be 

constructed. This matrix shows whether or not an agent within the system can be 

associated with any other agent, specifically through frequency of communications. The 

association matrix for the switching scenario is presented in Table 21. 

55



Table 21 - Agent association matrix 

A B C D E F 

A 0 2 2 2 1 4 

B 8 0 0 0 0 1 

C 4 0 0 0 0 0 

D 8 0 0 0 0 0 

E 1 0 0 0 0 0 

F 0 1 0 0 0 0 

Finally, a social network diagram is created. The social network diagram illustrates the 

proposed association between agents involved in the scenario. The numbers associated 

with the links between the agents in the system indicate the strength of association. The 

strength of association is defined by the number of occasions on which agents exchanged 

information. The direction of association is represented by directional arrows. The social 

network diagram is presented in Figure 3-14. 

Figure 3-14 - Return to service social network diagram. 

56



There are a number of ways to analyse social networks. In this case, agent centrality, 

sociometric status and network density were calculated. Agent centrality was calculated 

using Bavelas-Leavitt’s index. Table 22 shows the Centrality for the agents in this 

incident. The mean centrality was calculated as 3.13. A notion of ‘key’ agents can be 

defined using the mean + 1 standard deviation (i.e., 3.13 + 0.74 = 3.87). Using this rule, 

the B-L centrality calculation indicates that the NOC operator and the SAP/AP at 

Tottenham substation are the key agents in the network. 

Table 22 - Agent centrality (B-L Centrality) 

Agent 

B-L Centrality 

NOC operator 4.72 

SAP/AP at Tottenham 3.25 

SAP/AP at Waltham Cross 2.73 

SAP/AP at Brimsdown 2.73 

Overhead line party 2.73 

WOK operator 2.6 

Table 23 shows the Sociometric Status for each agent involved in the scenario. From the 

calculation, a mean status of 2.26 (±3.82) was found. The value of mean + one standard 

deviation, i.e. 2.26 + 3.82 = 6.08, is used to define ‘key’ agents in this network. Again, 

the sociometric status analysis indicates that the NOC operator is the key agent within the 

network. 

Table 23 - Agent sociometric status 

Agent 

Sociometric status 

NOC operator 6.4 

SAP/AP at Tottenham 2.4 

SAP/AP at Waltham Cross 1.2 

SAP/AP at Brimsdown 2.0 

Overhead line party 0.4 

WOK operator 1.2 

An overall measure of network density was also be derived by dividing the links actually 

present in the scenario, by all of the available links. For the Tottenham scenario, the 

overall network density is calculated as 0.2 (6 links present divided by 30 possible links). 

This figure is suggestive of a well distributed, (and therefore less dense) network of 

agents. 

57




In terms of rating relationships in networks the SNA technique appears to be unique. 

SNA also uses mathematical techniques for the analysis of the networks involved. During 

the data collection phase, techniques such as observational study, interviews and 

questionnaires are typically used. 


The associated training time for the SNA methodology is low. Although some knowledge 

of mathematical analysis is required, the basic SNA procedure is a simple one. It is 

estimated that the technique could be trained to HF practitioners within 1 hour. The 

application time for the SNA technique is also minimal. The provision of Agna SNA 

software package reduces the application time even further. In a SNA of an energy 

distribution switching scenario (involving a network of four agents) the application time 

was approximately 2 hours. However, the data collection phase involved adds to the 

overall application time, and this is typically lengthy. Also, the application time is 

dependent upon the size and complexity of the network under analysis, larger more 

complex networks may incur greater application times. 


No data regarding the reliability and validity of the SNA technique are available. 

Tools needed 

SNA can be conducted using pen and paper, once the data collection phase is complete. 

However, there are various SNA software packages that can be used that automate the 

SNA procedure. For example, based upon an association matrix as input, the AGNA 

software package constructs the social network and also performs various statistical 

analyses of the network in question. Further software packages are available, such as 

UCINET (Borgatti, Everett, and Freeman, 2002) and STRUCTURE (Burt, 1991). The 

tools required during the data collection phase for a SNA would be dependent upon the 

type of data collection techniques used. Observational analysis, interviews and 

questionnaires would normally require visual and audio recording equipment (video 

cameras, minidisc recorder, PC). 

58



Flowchart 

START 

Define network to be 

analysed 

Define scenarios required 

Collect data for the 

scenario(s) under analysis 

Construct association 

matrix 

Calculate the following as 

appropriate: 

• Centrality 

• Sociometric status 

• Network density 

• Betweenness 

• Distance 

Classify network 

STOP 

59



3.7 Operation Sequence Diagrams 


Operation Sequence Diagrams (OSD) are used to graphically describe the activity and 

interaction between teams of agents within a network. According to Kirwan and 

Ainsworth (1992), the original purpose of OSD analysis was to represent complex, multiperson 

tasks. The output of an OSD graphically depicts the task process, including the 

tasks performed and the interaction between operators over time, using standardised 

symbols. There are various forms of OSD’s, ranging from a simple flow diagram 

representing task order, to more complex OSD which account for team interaction and 

communication. OSD’s have been applied in a number of domains, including the fire 

service (Baber et al 2004), naval warfare (Stewart et al 2004), aviation (Stewart et al 

2004), energy distribution (Salmon et al 2004), air traffic control (Walker et al 2004) and 

rail (Walker et al 2004) domains. 


The technique was originally used in the nuclear power and chemical process industries. 

However, the technique is generic and can be applied in any domain. 


Step 1: Define the task(s) under analysis 

The first step in an OSD analysis is to define the task(s) or scenario(s) under analysis. 

The task(s) or scenario(s) should be defined clearly, including the activity and agents 

involved. 


In order to construct an OSD, the analyst(s) must obtain sufficient data regarding the task 

or scenario under analysis. It is recommended that the analyst(s) use various forms of 

data collection in this phase. Observational study should be used to observe the task (or 

similar types of task) under analysis. Interviews with personnel involved in the task (or 

similar tasks) should also be conducted. The type and amount of data collected in step 2 

is dependent upon the analysis requirements. The more exhaustive the analysis is 

intended to be, the more data collection techniques should be used. 

Step 3: Describe the task or scenario using HTA 

Once the data collection phase is completed, a detailed task analysis should be conducted 

for the scenario under analysis. The type of task analysis is determined by the analyst(s), 

and in some cases, a task list will suffice. However, it is recommended that a HTA is 

conducted. 

Step 4: Construct the OSD diagram 

60



Once the task has been described adequately, the construction of the OSD can begin. The 

OSD template should include the title of the task step, a timeline, and a row for each 

agent involved in the task. An OSD template is presented in Figure 3-15. In order to 

construct the OSD, it is recommended that the analyst walks through the HTA of the task 

under analysis, constructing the OSD in conjunction. An OSD symbol glossary is 

presented in Figure 3-16. The symbols involved in a particular task step should be linked 

by directional arrows, in order to represent the flow of activity during the task. Each 

symbol in the OSD should contain the corresponding task step number from the HTA of 

the task under analysis. The artefacts used during the communications should also be 

annotated onto the OSD. 

REC control 

room operator 

WOK Control 

Room 

Operator 

SAP/AP at 

Barking 

Substation 

NOC Control 

Room 

Operator 

Step 5: Overlay additional analyses results 

Figure 3-15 - Example OSD template 

One of the endearing features of the OSD technique is that additional analysis results can 

easily be added to the OSD. According to the analysis requirements, additional task 

features can also be annotated onto the OSD. For example, Baber et al (2004) added total 

co-ordination values for teamwork task steps (from an initial CDA analysis) onto the 

OSD for a fire service task. 

Step 6: Calculate operation loading figures 

From the OSD, operational loading figures are calculated for each agent involved in the 

scenario under analysis. Operational loading figures are calculated for each OSD operator 

or symbol used e.g. operation, receive, delay, decision, transport, and combined 

operations. The operational loading figures refer to the frequency in which each agent 

was involved in the operator in question during the scenario. 

Advantages 

61



Disadvantages 

• The OSD provides an exhaustive analysis of the task in question. The flow 

of the task is represented in terms of activity and information, the type of 

activity and the agents involved is specified, a timeline of the activity, the 

communications between agents involved in the task, the technology used 

and also a rating of total co-ordination for each teamwork activity is also 

provided. The techniques flexibility also permits the analyst(s) to add 

further analysis outputs onto the OSD, adding to its exhaustiveness. 

• An OSD is particularly useful for analysing and representing distributed 

teamwork or collaborated activity. 

• OSD’s are useful for demonstrating the relationship between tasks, 

technology and team members. 

• High face validity (Kirwan & Ainsworth 1992). 

• OSD’s have been used extensively in the past and have been applied in a 

variety of domains involving team based activity, including the emergency 

services (Baber et al 2004), energy distribution (Salmon et al 2004), rail 

(Walker et al 2004), air traffic control (ATC) (Walker et al 2004) and naval 

operations (Stewart et al 2004). 

• A number of different analyses can be overlaid onto an OSD of a particular 

task. For example, Baber et al (2004) add the corresponding HTA task step 

numbers and co-ordination demands analysis results to OSD’s of C4i 

activity. 

• The OSD technique is very flexible and can be modified to suit the analysis 

needs. 

• The WESTT software package can be used to automate a large portion of 

the OSD procedure. 

• The application time for an OSD analysis is lengthy. Constructing an OSD 

for large, complex tasks can be extremely time consuming and the initial 

data collection adds further time to the analysis 

• The construction of large, complex OSD’s is also quite difficult. 

• OSD’s can become cluttered and confusing (Kirwan & Ainsworth 1992). 

• The output of OSD’s can become large and unwieldy. 

• The present OSD symbols are limited for certain applications (e.g. C4i 

scenarios). 

62



• The reliability of the technique is questionable. Different analysts may 

interpret the OSD symbols differently. 


Various types of OSD exist, including temporal operational sequence diagrams, 

partitioned operational sequence diagrams and spatial operational sequence diagrams 

(Kirwan & Ainsworth 1992). During the OSD data collection phase, techniques such as 

observational study and interviews are typically used. Task analysis techniques such as 

HTA are also used during the construction of the OSD. Timeline analysis may also be 

used in order to construct an appropriate timeline for the task or scenario under analysis. 

Additional analyses results can also be annotated onto an OSD, such as CDA and comms 

usage diagram. The OSD technique has also recently been integrated with a number of 

other techniques (HTA, observation, co-ordination demands analysis, comms usage 

diagram, social network analysis and propositional networks) to form the event analysis 

of systemic teamwork (EAST) methodology (Baber et al 2004), which has been used to 

analyse C4i activity in a number of domains. 


The training time associated with the OSD methodology is low. However, the typical 

application time for the technique is high. The construction of an OSD can be a very time 

consuming process, involving many re-iterations. A typical OSD normally can take 12 + 

hours to construct. 


According to Kirwan & Ainsworth (1992), OSD techniques possess a high degree of face 

validity. The intra analyst reliability of the technique may be suspect, as different 

analysts may interpret the OSD symbols differently. 

Tools needed 

When conducting an OSD analysis, pen and paper could be sufficient. However, to 

ensure that data collection is comprehensive, it is recommended that video or audio 

recording devices are used in conjunction with the pen and paper. For the construction of 

the OSD, it is recommended that a suitable drawing package, such as Microsoft Visio 

is used. The WESTT software package can also be used to automate a large portion of the 

OSD procedure. WESTT constructs the OSD based upon an input of the HTA for the 

scenario under analysis. 

Example 

The following example is an extract of an OSD of an energy distribution scenario 

(Salmon et al 2004b). The task involved the switching out of three circuits (SGT5 and 

SGT1A and 1B) at Barking 275Kv, 132Kv and 33Kv Substations. Circuit SGT5 was 

being switched out for the installation of a new transformer for the nearby channel tunnel 

rail link and SGT1A and 1B were being switched out for substation maintenance. 

63



Observational data from Barking substation and the NOC control room was used to 

conduct a HTA of the switching scenario. A HTA was then created, which acted as the 

primary input into the OSD diagram. Total co-ordination values for each teamwork task 

step are also annotated onto the OSD. The glossary for the OSD is presented in Figure 

3-16. An extract of the HTA for the corresponding energy distribution task is presented 

after Figure 3-16. The corresponding extract of the OSD is presented in Figure 3-17. The 

operational loading figures are presented in Table 24. 

Table 24 - Operational loading results 

Agent Operatio 

n 

Receive Transport Decision Delay Total 

NOC 98 40 138 

SAP 223 21 19 1 264 

WOK 40 10 50 

REC 15 14 29 

The operational loading analysis indicates that the senior authorised person (SAP) at 

Barking substation has the highest loading in terms of operations, transport, and delay 

whilst the network operations centre (NOC) operator has the highest loading in terms of 

receipt of information. This provides an indication of the nature of the roles involved in 

the scenario. The NOC operators role is one of information distribution (giving and 

receiving) indicated by the high receive operator loading, whilst the majority of the work 

is conducted by the SAP at Barking, indicated by the high operation and transport loading 

figures. 

64



Figure 3-16 - OSD Glossary (Source: Salmon et al 2004) 

0. Co-ordinate and carry out switching operations on circuits SGT5. SGT1A and 1B at 

Bark s/s (Plan 0. Do 1 then 2 then 3, EXIT) 

1. Prepare for switching operations (Plan 1. Do 1.1, then 1.2, then 1.3, then 1.4, then 1.5, 

then 1.6, then 1.7, then 1.8, then 1.9,then 1.10 EXIT) 

1.1. Agree SSC (Plan 1.1. Do 1.1.1, then 1.1.2, then 1.1.3, then 1.1.4, then 1.1.5, EXIT) 

1.1.1. (WOK) Use phone to Contact NOC 

1.1.2. (WOK + NOC) Exchange identities 

1.1.3. (WOK + NOC) Agree SSC documentation 

1.1.4. (WOK+NOC) Agree SSC and time (Plan 1.1.4. Do 1.1.4.1, then 1.1.4.2, 

EXIT) 

1.1.4.1. (NOC) Agree SSC with WOK 

1.1.4.2. (NOC) Agree time with WOK 

1.1.5. (NOC) Record and enter details (Plan 1.1.5. Do 1.1.5.1, then 1.1.5.2, EXIT) 

1.1.5.1. Record details on log sheet 

65



1.1.5.2. Enter details into worksafe 

1.2. (NOC) Request remote isolation (Plan 1.2. Do 1.2.1, then 1.2.2, then 1.2.3, then 

1.2.4, EXIT) 

1.2.1. (NOC) Ask WOK for isolators to be opened remotely 

1.2.2. (WOK) Perform remote isolation 

1.2.3. (NOC) Check Barking s/s screen 

1.2.4. (WOK + NOC) End communications 

1.3. Gather information on outage at transformer 5 at Bark s/s 

(Plan 1.3. Do 1.3.1, then 1.3.2, then 1.3.3, then 1.3.4, EXIT) 

1.3.1. (NOC) Use phone to contact SAP at Bark 

Figure 3-17 - Extract for NGT Switching scenario (Source: Salmon et al 2004) 

66



Flowchart 

START 

Define task(s) under 

analysis 

Collect specific data 

regarding the task 


Conduct a HTA for the 

task under analysis 

Construct Operation 

sequence diagram 

Calculate operational 

loading figures for each 

agent involved in the 

task 

Add additional 

analysis results e.g. 

CDA, Comms usage 

diagram 

STOP 

67



3.8 Critical Decision Method 


The Critical Decision Method is a semi-structured interview technique that uses cognitive 

probes in order to elicit information regarding expert decision-making. According to the 

authors, the technique can serve to provide knowledge engineering for expert system 

development, identify training requirements, generate training materials and evaluate the 

task performance impact of expert systems (Klein, Calderwood & MacGregor 1989). 

The technique is an extension of the Critical Incident Technique (Flanagan 1954) and 

was developed in order to study the naturalistic decision-making strategies of 

experienced personnel. The CDM is perhaps the most commonly used cognitive task 

analysis technique and has been applied in a number of domains, including the fire 

service (Baber et al 2004), military and paramedics (Klein, Calderwood & MacGregor 

1989), air traffic control (Walker et al 2004), civil energy distribution (Salmon et al 

2004), naval warfare (Stewart et al 2004), rail (Walker et al 2004) and even white water 

rafting (O’Hare et al 2000). 


Generic. 


Step 1: Select the Incident to be analysed 

The first part of a CDM analysis is to select the incident that is to be analysed. 

Depending upon the purpose of the analysis, the type of incident may already be selected. 

CDM normally focuses on non-routine incidents, such as emergency incidents, or highly 

challenging incidents. If the scenario under analysis is not already specified, the 

analyst(s) may identify an appropriate incident via interview with an appropriate SME, 

by asking them to describe a recent highly challenging (i.e. high workload) or non routine 

incident in which they were involved. The interviewee involved in the CDM analysis 

should be the primary decision maker in the chosen incident. 

Step 2: Select CDM probes 

The CDM technique works by probing a SME using specific probes designed to elicit 

pertinent information regarding the decision making process during key points in the 

incident under analysis. In order to ensure that the output is compliant with the original 

aims of the analysis, an appropriate set of CDM probes should be defined prior to the 

analysis. The probes used are dependent upon the aims of the analysis and the domain in 

which the incident is embedded. Alternatively, if there are no adequate probes available, 

the analyst(s) can develop novel probes based upon the analysis needs. A set of CDM 

probes used in work package 1 to analyse C4i activity are presented in table 9 of this 

report. Step 3: Select appropriate participant 

68



Once the scenario under analysis and the probes to be used are defined, the analyst(s) 

should proceed to identify an appropriate SME or set of SME’s. Typically, operators of 

the system under analysis are used. 

Step 4: Gather and record account of the incident 

The CDM procedure can be applied to an incident observed by the analyst or to a 

retrospective incident described by the participant. If the CDM is based upon an observed 

incident, then this step involves firstly observing the incident and then recording an 

account of the incident. Otherwise, the incident can be described retrospectively form the 

participants memory. The analyst should ask the SME for a description of the incident in 

question, from its starting point to its end point. 

Step 5: Construct Incident Timeline 

The next step in the CDM analysis is to construct a timeline of the incident described in 

step 4. The aim of this is to give the analyst(s) a clear picture of the incident and its 

associated events, including when each event occurred and what the duration of each 

event was. According to Klein, Calderwood & MacGregor (1989) the events included in 

the timeline should encompass any physical events, such as alarms sounding, and also 

‘mental’ events, such as the thoughts and perceptions of the interviewee during the 

incident. 

Step 6: Define scenario phases 

Once the analyst has a clear understanding of the incident under analysis, the incident 

should be divided into key phases or decision points. This should be done in conjunction 

with the SME. Normally, the incident is divided into four or five key phases. 

Step 7: Use CDM probes to query participant decision making 

For each incident phase, the analyst should probe the SME using the CDM probes 

selected during step 2 of the procedure. The probes are used in an unstructured interview 

format in order to gather pertinent information regarding the SME’s decision making 

during each incident phase. The interview should be recorded using an audio recording 

device such as a mini-disc recorder. 

Step 8: Transcribe interview data 

Once the interview is complete, the data should be transcribed accordingly. 

Step 9: Construct CDM tables 

Finally, a CDM output table for each scenario phase should be constructed. This involves 

simply presenting the CDM probes and the associated SME answers in an output table 

(see Table 25 to Table 28). 

69



Advantages 

Disadvantages 

• The CDM analysis can be used to elicit specific information regarding the 

decision making strategies used in complex systems. 

• The CDM output can be used to construct propositional networks which 

describe the knowledge or SA objects required during the scenario under 

analysis. 

• The technique is normally quick in application. 

• Once familiar with the technique, CDM is relatively easy to apply. 

• The CDM is a popular procedure and has been applied in a number of 

domains. 

• The reliability of such a technique is questionable. Klein & Armstrong 

(2004) suggests that methods that analyse retrospective incidents are 

associated with concerns of data reliability, due to evidence of memory 

degradation. 

• The data is obtained is highly dependent upon the skill of the analyst 

conducting the CDM interview and also the quality of the participant used. 

• A high level of expertise and training is required in order to use the CDM to 

its maximum effect (Klein & Armstrong 2004). 

• The CDM relies upon interviewee verbal reports in order to reconstruct 

incidents. How far a verbal report accurately represents the cognitive 

processes of the decision maker is questionable. Facts could be easily 

misrepresented by the participants involved. 

• It is often difficult to gain sufficient access to appropriate SME’s in order to 

conduct a CDM analysis. 

70



Flowchart 

START 

Define scenario(s) 


Select appropriate 

CDM probes 

Select appropriate 

SME(s) 

Ask SME to give a 

description of incident 

Divide the incident 

into key phases 

Take the first/next 

incident phase 

Use probes to gather 

information for each 

incident phase 


more incident 

phases 

Transcribe data 

Construct CDM 

output tables 

STOP 

71



Example 

The following example is taken from a CDM analysis that was conducted in order to 

analyse C4i activity in the civil energy distribution domain (Salmon et al 2004b). The 

scenario under analysis involved the switching out of three circuits at Barking 275Kv, 

132Kv and 33Kv Substations. Circuit SGT5 was being switched out for the installation 

of a new transformer for the nearby channel tunnel rail link and SGT1A and 1B were 

being switched out for substation maintenance. For the CDM analysis, the control room 

operator co-ordinating the activity and the senior authorised person at Barking substation 

who conducted the activity were interviewed. The set of CDM probes used are presented 

in Table 9. The scenario was divided into four key phases: 

First issue of instructions 

• Deal with switching requests 

• Perform isolation 

• Report back to network operations centre 

The CDM output is presented in Table 25 to Table 28. 

Table 25 - Phase 1: First issue of instructions 

Goal 

Specification 

Cue 

identification 

Expectancy 

Conceptual 

Model 

Uncertainty 

Information 

Situation 

Awareness 

Situation 

Assessment 

Options 

Stress 

Choice 

Analogy 

Establish what isolation the SAP at Barking is looking for. Depends on gear 

Don’t Believe It (DBI) alarm is unusual – faulty contact (not open or closed) 

questionable data from site checking rating of earth switches (may be not fully rated 

for circuit current – so additional earths may be required. 



Recognised instruction but not stated in WE1000 – as there are not too many front 

and rear shutters metal clad switch gear. 

Confirm from field about planned instruction – make sure that SAP is happy with the 

instruction. 








N/A 




72



Table 26 - Phase 2: Deal with switching requests 

Goal 

Specification 

Cue 


Expectancy 

Conceptual 

Model 

Uncertainty 

Information 

Situation 

Awareness 

Situation 

Assessment 

Options 

Stress 

Choice 

Analogy 




Airblast circuit breakers (accompanied by sirens) can be heard to operate remotely 

(more so in Barking 275 than Barking C 132). 




Physical verification of apparatus always required (DBI – don’t believe it). 

Proceduralised information from NOC – circuit, location, time, actions required etc. 












Table 27 - Phase 3: Perform Isolation 

Goal 

Specification 

Cue 


Expectancy 

Conceptual 

Model 

Uncertainty 

Information 

Situation 

Awareness 

Situation 

Assessment 

Options 

Stress 

Choice 

Analogy 

Inform NOC of isolation status 


NOC operator answers 

NOC accepts. 



No – possibly further instructions, possibly mismatches local situation and remote 

displays in NOC. 






No 



73



Table 28 - Phase 4: Report back to NOC 

Goal 

Specification 

Cue 


Expectancy 

Conceptual 

Model 

Uncertainty 

Information 

Situation 

Awareness 

Situation 

Assessment 

Options 

Stress 

Choice 

Analogy 


















Inform NOC that isolation cannot be performed/other aspects of switching 

instructions cannot be carried out. 


Possibly some difficulties in operating or physically handling the equipment. 

Yes – proceduralised within equipment types. Occasional non-routine activities 

required to cope with unusual/unfamiliar equipment, or equipment not owned by 

NGT. 


Related Methods 

The CDM is an extension of the critical incident technique (Flanagan 1954). The CDM 

is also closely related to other cognitive task analysis (CTA) techniques, in that it uses 

probes to elicit data regarding task performance from participants. Other similar CTA 

techniques include ACTA (Militello and Hutton 2000) and cognitive walkthrough 

analysis (Polson et al 1992). 


Klein & Armstrong (2004) report that the training time associated with the CDM would 

be high. Experience in interviews with SME’s is required, and also a grasp of cognitive 

psychology. The application time for the CDM is medium. The CDM interview takes 

between 1 - 2 hours, and the transcription process takes approximately 1 – 2 hours. 


Both intra and inter analyst reliability of the CDM approach is questionable. It is 

apparent that such an approach may elicit different data from similar incidents when 

applied by different analysts on separate participants. Klein & Armstrong (2004) 

suggests that there are also concerns associated with the reliability of the CDM due to 

evidence of memory degradation. 

74



Tools needed 

When conducting a CDM analysis, pen and paper could be sufficient. However, to 

ensure that data collection is comprehensive, it is recommended that video or audio 

recording equipment is used. A set of CDM probes, such as those presented in Table 9 

are also required. The type of probes used is dependent upon the focus of the analysis. 

3.9 Propositional Networks 


Propositional networks are used identify the knowledge objects related to a particular task 

or scenario, and also the links between each of the knowledge objects identified. 

According to Baber and Stanton (2004), the concept of representing ‘knowledge’ in the 

form of a network has been subjected to major discussion within cognitive psychology 

since the 1970s. Propositional networks consist of a set of nodes that represent 

knowledge, sources of information, agents, and artefacts that are linked through specific 

causal paths. Thus the propositional network offers a way of presenting the ‘ideal’ 

collection of knowledge required during the scenario in question. Networks are 

constructed from an initial critical decision method analysis of the scenario in question. A 

simple content analysis is used to identify the knowledge objects for each scenario phase 

as identified by the CDM analysis. A propositional network is then constructed for each 

phase identified by the CDM analysis, comprised of the knowledge objects and the links 

between them. Propositional networks have been used to represent knowledge and shared 

situation awareness as part of the EAST methodology (Baber et al 2004), which has been 

used to analyse C4i scenarios in the civil energy distribution domain (Salmon et al 2004b 

+ c), the rail domain (Walker et al 2004), the air traffic control domain (Walker et al 

2004), the naval warfare domain (Stewart et al 2004) and the emergency services domain 

(Baber et al 2004). 


Generic. 

Procedure and Advice 

Step 1: Define scenario 

The first step in a propositional network analysis is to define the scenario under analysis. 

The scenario in question should be defined clearly. This allows the analyst(s) to 

determine the data collection procedure that follows and also the appropriate SME’s 

required for the CDM phase of the analysis. 

Step 2: Conduct a HTA for the scenario 

Once the scenario has been clearly defined, the next step involves describing the scenario 

using HTA. A number of data collection techniques may be used in order to gather the 

75



information required for the HTA, such as interviews with SME’s and observations of the 

task under analysis. 

Step 3: Conduct a CDM analysis 

The propositional networks are based upon a CDM analysis of the scenario in question. 

The CDM analysis should be conducted using appropriate SME’s (see section 3.8 for a 

full description of the CDM procedure). The CDM involves dividing the scenario under 

analysis into a number of key phases and then probing the SME using pre-defined 

‘cognitive’ probes, designed to determine pertinent features associated with decision 

making during each scenario phase. 

Step 4: Conduct content analysis 

Once the CDM data is collected, a simple content analysis should be conducted for each 

phase identified during the CDM analysis. In order to convert the CDM tables into 

propositions, a content analysis is performed. In the first stage, this simply means 

separating all content words from any function words. For example, the entry in table one 

“Respiratory problems caused by unknown, airborne material” would be reduced to the 

following propositions ‘respiratory problems’, ‘airborne’ and ‘material’. Working 

through the table leads to a set of propositions. These are checked to ensure that 

duplication is minimised and then used to construct the propositional network. 

In order to specify the knowledge objects for each phase, the analyst simply takes the 

CDM output for each phase and using a simple content analysis, identifies the required 

knowledge objects. Knowledge objects include any knowledge, information, agents and 

artefacts identified by the CDM analysis. A simple list of knowledge objects should be 

made for each scenario phase. 

Step 5: Define links between knowledge objects 

Once the knowledge objects for each scenario phase have been identified, the next step 

involves defining the links between the knowledge objects in each phase. The following 

knowledge objects links taxonomy is used: 

• Has 

• Is 

• Causes 

• Knows 

• Requires 

• Prevents 

76



For those knowledge objects that are linked during the scenario, the type of link should 

be defined using the links taxonomy above. 

Step 6: Construct propositional networks 

The final step is to construct the propositional network diagrams for each scenario phase. 

A propositional network diagram should be constructed for the overall scenario (i.e. 

including all knowledge objects) and then separate propositional network diagrams 

should be constructed for each phase, with the knowledge objects required highlighted in 

red. Further coding of the knowledge objects may also be used e.g. shared knowledge 

objects can be striped in colour, and inactive knowledge objects that have been used in 

previous scenario phases are typically shaded. 

Advantages 

Disadvantages 

• The output represents the ideal collection of knowledge required for 

performance during the scenario under analysis. 

• The knowledge objects are defined for each phase of the scenario under 

analysis, and the links between the knowledge objects are also specified. 

• The technique is easy to learn and use. 

• The technique is also quick in its application. 

• Propositional networks are ideal for analysing teamwork and representing 

shared situation awareness during a particular scenario. 

• The initial HTA and CDM analysis add considerable time to the associated 

application time. 

• Inter and intra analyst reliability of the technique is questionable. 

• A propositional network analysis is reliant upon acceptable CDM data. 

• It may be difficult to gather appropriate SME’s for the CDM part of the 

analysis. 

77



Flowchart 

START 

Define scenario 


Conduct a HTA for 

the scenario under 

analysis 

Conduct a CDM 

analysis with SME(s) 

Take first/next CDM 

phase 

Identify knowledge 

objects using content 

analysis 

Define knowledge object 

links using the associated 

links taxonomy 

Construct propositional 

network 


more CDM 

phases 

STOP 

78



Example 

The following example is taken from an analysis of a switching scenario drawn from the 

civil energy distribution domain (Salmon et al 2004b). The propositional networks 

presented in Figure 3-18 to Figure 3-22 present the knowledge objects (shaded in red) 

identified from the corresponding CDM output for that phase. The CDM outputs are 

presented in Table 29 to Table 32. The propositional network consists of a set of nodes 

that represent sources of information, agents, and objects etc. that are linked through 

specific causal paths. From this network, it is possible to identify required information 

and possible options relevant to this incident. The concept behind using a propositional 

network in this manner is that it represents the ‘ideal’ collection of knowledge for the 

scenario. As the incident unfolds, so participants will have access to more of this 

knowledge (either through communication with other agents or through recognising 

changes in the incident status). Consequently, within this propositional network, Situation 

Awareness can be represented as the change in weighting of links. Propositional 

networks were developed for the overall scenario and also the incident phases identified 

during the CDM analysis. The propositional networks indicate which of the knowledge 

objects are active (i.e. agents are using them) during each incident phase. The light blue 

nodes in the propositional networks represent unactivated knowledge objects (i.e. 

knowledge is available but is not required nor is it being used). The red nodes represent 

active (or currently being used) knowledge objects. 

Table 29 - CDM Phase 1: First issue of instructions 



Establish what isolation the SAP at Barking is looking for. Depends on gear 

Don’t Believe It (DBI) alarm is unusual – faulty contact (not open or closed) 

questionable data from site checking rating of earth switches (may be not fully 

rated for circuit current – so additional earths may be required. 


Expectancy 


Uncertainty 

Information 

Situation 

Awareness 


Recognised instruction but not stated in WE1000 – as there are not too many front 

and rear shutters metal clad switch gear. 

Confirm from field about planned instruction – make sure that SAP is happy with 

the instruction. 






Situation 

Assessment 


79



Options 

Stress 

Choice 


N/A 



Analogy 


Table 30 - CDM Phase 2: Deal with switching requests 



Expectancy 


Uncertainty 

Information 

Situation 

Awareness 

Situation 

Assessment 

Options 

Stress 

Choice 

Analogy 




Airblast circuit breakers (accompanied by sirens) can be heard to operate remotely 

(more so in Barking 275 than Barking C 132). 




Physical verification of apparatus always required (DBI – don’t believe it). 

Proceduralised information from NOC – circuit, location, time, actions required etc. 












Table 31 – CDM Phase 3: Perform isolation 

80





Expectancy 


Uncertainty 

Information 


Situation Assessment 

Options 

Stress 

Choice 

Analogy 


















Inform NOC that isolation cannot be performed/other aspects of switching instructions 

cannot be carried out. 


Possibly some difficulties in operating or physically handling the equipment. 

Yes – proceduralised within equipment types. Occasional non-routine activities required 

to cope with unusual/unfamiliar equipment, or equipment not owned by NGT. 


81



Table 32 - Phase 4: Report back to NOC 



Expectancy 


Uncertainty 

Information 

Situation 

Awareness 

Situation 

Assessment 

Options 

Stress 

Choice 

Analogy 

Inform NOC of isolation status. 


NOC operator answers. 

NOC accepts. 



No – possibly further instructions, possibly mismatches local situation and remote 

displays in NOC. 






No 



82



Gas Insulated 

Circuit 

Breakers 

Airblast 

Notices/Locks 

NOC 

Check Open 


Location 

SystemState 

Certificates 

Isolation 

Control 

Engineer 

Open Lock & 

Caution 

Accept 

Instructions 

Switching 

Wokingham 

Shutters 

Time 

Dressed 

Log Sheet 

Procedures 

Rear 

Refuse 

Switching Log 

Displays 

Front 


Isolators 

WE1000 


Isolation 

Open 

Busbar 

Circuits 

Electrical 

Contacts 

Closed 

Phone 

Current 

Voltage 

Faulty 


Switching 

Phone 

Equipment 

Lables 

Identity 

SAP 

Substations 

Figure 3-18 - Propositional Network for Objects referred to in CDM tables. 

Gas Insulated 

Circuit 

Breakers 

Airblast 

Notices/Locks 

NOC 

Check Open 


Location 

SystemState 

Certificates 

Isolation 

Control 

Engineer 

Open Lock & 

Caution 

Accept 

Instructions 

Switching 

Wokingham 

Shutters 

Time 

Dressed 

Log Sheet 

Procedures 

Rear 

Refuse 

Switching Log 

Displays 

Front 


Isolators 

WE1000 


Isolation 

Open 

Busbar 

Circuits 

Electrical 

Contacts 

Closed 

Phone 

Current 

Voltage 

Faulty 


Switching 

Phone 

Equipment 

Lables 

Identity 

SAP 

Substations 

Figure 3-19 - Propositional Network for CDM phase one. 

83



Gas Insulated 

Circuit 

Breakers 

Airblast 

Notices/Locks 

NOC 

Check Open 

Lock &Caution 

Location 

SystemState 

Certificates 

Isolation 

Control 

Engineer 

Open Lock & 

Caution 

Accept 

Instructions 

Switching 

Wokingham 

Shutters 

Time 

Dressed 

Log Sheet 

Procedures 

Rear 

Refuse 

Switching Log 

Displays 

Front 


Isolators 

WE1000 


Isolation 

Open 

Busbar 

Circuits 

Electrical 

Contacts 

Closed 

Phone 

Current 

Voltage 

Faulty 


Switching 

Phone 

Equipment 

Lables 

Identity 

SAP 

Substations 

Figure 3-20 - Propositional Network for CDM phase two. 

84



Gas Insulated 

Circuit 

Breakers 

Airblast 

Notices/Locks 

NOC 

Check Open 

Lock &Caution 

Location 

SystemState 

Certificates 

Isolation 

Control 

Engineer 

Open Lock & 

Caution 

Accept 

Instructions 

Switching 

Wokingham 

Shutters 

Time 

Dressed 

Log Sheet 

Procedures 

Rear 

Refuse 

Switching Log 

Displays 

Front 


Isolators 

WE1000 


Isolation 

Open 

Busbar 

Circuits 

Electrical 

Contacts 

Closed 

Phone 

Current 

Voltage 

Faulty 


Switching 

Phone 

Equipment 

Lables 

Identity 

SAP 

Substations 

Figure 3-21 - Propositional Network for CDM phase three. 

Gas Insulated 

Circuit 

Breakers 

Airblast 

Notices/Locks 

NOC 

Check Open 


Location 

SystemState 

Certificates 

Isolation 

Control 

Engineer 

Open Lock & 

Caution 

Accept 

Instructions 

Switching 

Wokingham 

Shutters 

Time 

Dressed 

Log Sheet 

Procedures 

Rear 

Refuse 

Switching Log 

Displays 

Front 


Isolators 

WE1000 


Isolation 

Open 

Busbar 

Circuits 

Electrical 

Contacts 

Closed 

Phone 

Current 

Voltage 

Faulty 


Switching 

Phone 

Equipment 

Lables 

Identity 

SAP 

Substations 

Figure 3-22 - Propositional Network for CDM phase four. 

85




Propositional networks require an initial CDM analysis as an input. A HTA is also 

typically conducted prior to the propositional network analysis. The technique has also 

been used in conjunction with a number of other techniques (HTA, observation, coordination 

demands analysis, comms usage diagram, social network analysis) in the form 

of the event analysis of systemic teamwork (EAST) methodology (Baber et al 2004), 

which has been used to analyse C4i activity in a number of domains. 


The propositional network methodology requires only minimal training. In a recent HF 

methods training session, the training time for the propositional network technique was 

approximately one hour. However, the analyst should be competent in the HTA and 

CDM procedure in order to conduct the analysis properly. The application time for 

propositional networks alone is high, as it involves a content analysis (on CDM outputs) 

and also the construction of the propositional networks. 


No data regarding the reliability and validity of the technique are available. From 

previous experience, it is evident that the reliability of the technique may be questionable. 

Certainly, different analysts may identify different knowledge objects for the same 

scenario (Intra-analyst reliability). Also, the same analyst may identify different 

knowledge objects for the same scenario on different occasions (Inter-analyst reliability). 

Tools needed 

A propositional network analysis can be conducted using pen and paper. However, it is 

recommended that during the CDM procedure, an audio recording device is used. When 

constructing the propositional network diagrams it is recommended that Microsoft Visio 

is used. 

86



4.1 Aim 

4 Feedback from Questionnaire 

The aim of this investigation was to understand the user acceptance of the EAST 

methodology by its users. 

4.2 Participants 

Nine participants responded to the questionnaire, 8 Male and 1 Female. All had a 

minimum of a Masters degree with 5 of the group having a PhD. The sample was split 

over four institutions including: Brunel University; Birmingham University; Cranfield 

University; and Lockheed Martin. The method had been applied over a wide range of 

situations including. Air force (E3D), Naval (HMS Dryad), Army (a theoretical ‘Scud 

Hunt’ paradigm), Air Traffic Control, Rail, Energy Distribution (NGC), and the 

Emergency Services (police / fire). 

4.3 Equipment 

Questionnaires 

4.4 Procedure 

The questionnaire was sent out via email to a number of users within the HFI-DTC 

consortium who had used the EAST Method. The questionnaire was completed 

electronically and returned via email. 

The questionnaire captured demographic data in the first section. In the second section 

the questionnaire captured participant opinion of each of the methodologies in turn that 

make up EAST: 

Observation 

HTA – Hierarchical Task Analysis 

CDA – Coordination Demand Analysis 

CUD – Comms. Usage Diagram 

SNA – Social Network Analysis 

OSD – Operation Sequence Diagrams 

CDM – Critical Decision Method 

PN – Propositional Networks 

87



Participants were asked to rate each of the methods against the following criteria: 

• Ease of learning 

• Time to learn 

• Ease of application 

• Time to Apply 

• Usefulness of output 

• Predictability of Output 

• Validity of output 

The participants were asked to respond on a Likert scale that was rated so that 5 was a 

positive reaction and 1 a negative reaction. In the third and final section of the 

questionnaire participants were asked to evaluate the methodology as a whole. (Questions 

posed can be seen in the results section) 

4.5 Results 

All of the respondents stated that they would use the East methodology again. When 

questioned in more detail about which of the individual tools they would use again; there 

was not 100% acceptance for all of the methods. The results of this can be seen in Figure 

4-1 the methods that did not receive 100% acceptance were the CDA, CUD, and OSD 

methods. 

88



Percentage who would use method again 

100% 

100% 100% 

100% 

100% 100% 

90% 

89% 

89% 

80% 

78% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 

Observation HTA CDA CUD SNA OSD CDM PN 

Figure 4-1 - Percentage who would use method again 

The results collected are plotted on radar plots. A red line has been added to the 

document so that anything outside the line can be considered positive and inside the line 

negative. The original questionnaire along with further plots can be found in the 

Appendix of this document. 

Observation 

Observation scores above average on each of the criteria with the exception of the time to 

apply. This method is rated as the easiest to learn and scores very highly on validity and 

usefulness. 

89



Observation 

Ease of learning 

Validity of output 

Time to learn 

Predictability of 

Output 

Ease of 

application 

Usefulness of output 

Time to Apply 

Figure 4-2 - Radar plot of medians for observation 

HTA – Hierarchical Task Analysis 

HTA scores very highly on usefulness and validity of output. However when looking at 

the means the method performs negatively for the remaining attributes which 

predominantly relate to the ease and time to learn and apply. The predictability of the 

output is also brought into question with the below average rating. 

HTA 


5 


4 

3 

Time to learn 

2 

1 

Predictability of Output 

Ease of application 


Time to Apply 

Figure 4-3 - Radar plot of medians for HTA 

90



CDA – Coordination Demand Analysis 

The majority of the ratings of the attributes of CDA fall on or near the neutral line when 

looking at the means. The method scores higher on the ease and time to learn and also 

the ease of application. 

CDA 


5 


4 

3 

Time to learn 

2 

1 




Time to Apply 

CUD – Comms Usage Diagram 

Figure 4-4 - Radar plot of medians for CDA 

The CUD shows similar results to the CDA method. This method scores poorly for both 

predictability and usefulness. The method scores higher on the ease and time to learn and 

apply. 

91



CUD 


5 


4 

3 

Time to learn 

2 

1 




Time to Apply 

SNA – Social Network Analysis 

Figure 4-5 - Radar plot of medians for CUD 

The majority of the ratings of the attributes of SNA fall on or near the neutral line when 

looking at the means. The method scores higher on the validity, predictability and 

usefulness of the output. 

SNA 


5 


4 

3 

Time to learn 

2 

1 




Time to Apply 

Figure 4-6 - Radar plot of medians for SNA 

92



OSD – Operation Sequence Diagrams 

As with the SNA and CDM method this method shows fairy indifferent responses to the 

ease and time to apply and learn. It scores highly on validity of output. The method 

received negative feedback on the predictability of the output. 

OSD 


5 


4 

3 

Time to learn 

2 

1 




Time to Apply 

CDM – Critical Decision Method 

Figure 4-7 - Radar plot of medians for OSD 

This method shows similar results to the SNA method scoring higher on the validity, 

predictability and usefulness of the output. 

93



CDM 


5 


4 

3 

Time to learn 

2 

1 




Time to Apply 

PN – Propositional Networks 

Figure 4-8 - Radar plot of medians for CDM 

This method scores very poorly (the worst of all the methods examined) on the ease and 

time to apply and learn. The method scores significantly higher on the validity, 

predictability and usefulness of the output. 

PN 


5 


4 

3 

Time to learn 

2 

1 




Time to Apply 

Figure 4-9 - Radar plot of medians for PN 

94



4.6 Discussion of Results 

The overall acceptance of the methodology can be understood from the last section of the 

questionnaire. the results are shown as medians in the radar below in Figure 4-10. It can 

be seen that the only negative response was the time taken to complete the method. This 

is clearly something that all of the respondents felt strongly about with six of them rating 

it as 1 and a further three rating it as 2. This view is supported by the rest of the review 

of the method in this document. 

Medians 

Your overall acceptance of the methodology. 

5 

Please rate the overall ease of use of the methodology. 

4 

How open you perceive the methodology is to scrutiny 

3 

How w ell you think you carried out the EAST methodology 

2 

1 

The breadth of coverage and its ability to describe a range 

of behaviour. 

How useful do you find the EAST methodology 

How likely do you think the methodology is to produce 

consistent results 

Please rate the amount of resources (time & effort) required 

to conduct an evaluation. 

The extent to w hich the methodology has theoretical 

foundations. 

Figure 4-10 - Overall rating of the Methodology 

95



5 Summary 

This report presents a review of the EAST methodology and its component methods. The 

following conclusions can be taken from the EAST methods review. 

1. The EAST methodology is an exhaustive technique. A number of different 

analyses are conducted and various perspectives on the scenario(s) under analysis 

are offered. In its present form, the EAST methodology offers the following 

analyses of a particular C4i scenario; 

• A step-by-step (goals, sub-goals, operations and plans) description of the 

activity in question. 

• A definition of roles within the scenario. 

• An analysis of the agent network structure involved (e.g. network type and 

density). 

• A rating of co-ordination between agents for each team-based task step and 

an overall co-ordination rating. 

• An analysis of the current technology used during communications between 

agents and also recommendations for novel comms technology. 

• A description of the task in terms of the flow of information, 

communications between agents, the activity conducted by each agent 

involved and a timeline of activity. 

• An analysis of agent centrality, sociometric status and betweenness within 

the network involved in the scenario. 

• A definition of the key agents involved in the scenario. 

• A cognitive task analysis of operator decision making during the scenario. 

• A definition of the knowledge objects (information, artefacts etc) required 

and the knowledge objects used during the scenario. 

• A definition of shared knowledge or shared situation awareness during the 

scenario. 

2. The EAST methodology is particularly suited to the analysis of team-based or 

collaborative activity, such as that seen in C4i environments. The method was 

originally developed for this purpose and applications so far have proved 

extremely successful, highlighting its suitability for such applications. 

3. The EAST methodology is relatively simple to use. The method requires an initial 

understanding of HF and experience in the application of HF methods. However, 

96



from analyst reports it can be concluded that the EAST methodology is relatively 

simple to apply. The methods ease of use is heightened when compared to the 

exhaustive output that is generated from an EAST analysis. 

4. The EAST procedure can be made sufficiently quicker and easier with the 

provision of software assistance. The WESTT software package developed during 

WP 1 can be used to automate a large portion of the EAST procedure (i.e. 

construction of OSD’s, SNA). The Agna software package can also be used for 

the SNA part of EAST. 

5. The EAST methodology is generic and can be applied in any domain in which 

collaborative activity takes place. Despite the number of component methods 

involved in the EAST procedure, the method is generic and can be applied in any 

domain that uses collaborative activity. 

Despite the positive conclusions presented above, the methods review also highlighted 

some flaws within the EAST methodology. These are summarised below. 

1. Due to its exhaustive nature, the EAST methodology is time consuming to apply. 

A typical EAST analysis of a moderately complex scenario takes around 5 days to 

complete. For complex scenarios the analysis may also become laborious at times 

and also unwieldy in places. 

2. A large portion of the methods output is descriptive. A large onus is placed upon 

the analyst to interpret the analysis for more informative conclusions. 

3. In order to work properly and produce valid outputs, a large amount of access to 

the system and agents involved in the scenario under analysis. This is often 

difficult to obtain in domains where C4i activity takes place, due to sensitivity 

issues (i.e. the military, nuclear power and police domains). 

4. The reliability of the technique could be questionable at times. Much of the 

analysis is based upon the subjective judgement of the analyst involved, and so 

intra-analyst and inter analyst reliability may suffer. However, this does not seem 

to have been a problem in EAST applications so far. 

As an overall conclusion, it is felt by the authors that the EAST methodology has been 

successful in its applications thus far, and despite the flaws outlined above, the 

methodology is perfectly suited to the analysis of C4i activity. Each EAST application 

has produced valid and useful results which are currently forming the basis for the 

development of a generic model of C4i. Despite its success, the EAST methodology is 

still evolving, and subsequently it is felt that the technique can only become more useful, 

usable and valid. On a final note, the following recommendations regarding the 

development of the EAST methodology were made as a result of the methods review. 

1. EAST software package. Despite the provision of the WESTT, AGNA and 

Microsoft Visio software packages which automate large portions of the EAST 

procedure, it is felt by the authors that a comprehensive EAST software package 

requires development. To support the EAST procedure, a software package that 

could generate CDA, CUD, SNA, OSD, CDM and propositional network data 

97



from an initial HTA input is required. This solution is not that far away (WESTT 

already produces SNA, OSD and Prop network data) and it is felt that it would 

remove the problem of high application times and also concerns regarding the 

reliability of the technique. 

2. EAST instructions booklet. There appears to be limited guidance for the 

application of a full EAST analysis. As a solution, it is felt that an EAST 

guidelines or instruction booklet should be constructed, containing a step-by-step 

procedure and worked examples. 

3. EAST validation studies. Although a number of studies using the EAST 

methodology have already been conducted, it is felt that the methodology requires 

further validation. 

4. OSD symbols. It is felt that the current set of OSD symbols are lacking when used 

to describe C4i activity. A set of C4i specific OSD symbols should be developed, 

based upon previous EAST applications. 

5. CUD. The technology used column in the CUD output should describe the comms 

technology at either end (e.g. from = telephone, to = mobile phone). 

98



6 Bibliography 

Ainsworth, L. & Marshall, E. (1998). Issues of quality and practicality in task analysis: 

preliminary results from two surveys. Ergonomics 41(11), 1604-1617. Reprinted in J. 

Annett & N.A. Stanton (2000) op.cit. Pp. 79-89. 

Annett, J. 2004, Hierarchical Task Analysis. In N. A. Stanton, A. Hedge, K. Brookhuis, 

E. Salas, & H. Hendrick (eds.) Handbook of Human Factors methods. Taylor and 

Francis, London. 

Baber, C. & Stanton, N. A. (1996). Human error identification techniques applied to 

public technology: predictions compared with observed use. Applied Ergonomics, 27(2), 

119-131. 

Baber, C. and Stanton, N. A. 2004, Methodology for DTC-HFI WP1 field trials. Defence 

Technology Centre for Human Factors Integration, Report 2.1 

Baber, C., Walker, G., Stanton, N. A., & Salmon, P. (2004a). Report on Initial Trials of 

WP1.1 Methodology conducted at fire service training college. HFI-DTC Technical 

Report, WP1.1.1/01, 29 th January 2004. 

Baber, C., Houghton, R. J., McMaster, R., Salmon, P., Stanton, N. A., Stewart, R. J., & 

Walker, G. (2004). Field studies in the emergency services. HFI-DTC Technical 

Report/WP 1.1.1/1-1, November 2004. 

Burke, C. S. (2004). Team Task Analysis. In N. A. Stanton, A. Hedge, K. Brookhuis, E. 

Salas, & H. Hendrick. (Eds.), Handbook of Human Factors methods. Boca Raton, USA, 

CRC Press. 

Dekker, A. H. (2002). C4ISR Architectures, Social Network Analysis and the FINC 

Methodology: An Experiment in Military Organisational Structure. DSTO Electronics 

and Surveillance Research Laboratory. DSTO-GD-0313 

Driskell, J. E. & Mullen, B. (2004) Social Network Analysis. In N. A. Stanton, A. 

Hedge, K. Brookhuis, E. Salas, & H. Hendrick. (2004) (Eds.) Handbook of Human 

Factors methods. UK, Taylor and Francis. 

Drury, C. G. (1990). Methods for direct observation of performance, In Wilson, J. R. And 

Corlett, E. N (Eds.) ‘Evaluation of Human Work – A practical ergonomics methodology’ 

Taylor and Francis, London. 

Flanagan, J. C. (1954). The Critical Incident Technique. Psychological Bulletin, 51, 

327-358. 

Kirwan, B., & Ainsworth, L. K. (1992). A guide to Task Analysis, Taylor and Francis, 

London, UK. 

99



Klein, G. & Armstrong, A. A. (2004). Critical Decision Method. In N. A. Stanton, A. 

Hedge, K. Brookhuis, E. Salas, & H. Hendrick. (Eds.) Handbook of Human Factors 

methods. UK, Taylor and Francis. 

Klein, G. A., Calderwood, R., & MacGregor, D. (1989). Critical Decision Method for 

Eliciting Knowledge. IEEE Transactions on Systems, Man and Cybernetics, 19(3), pp 

462-472. 

Marshall, A., Stanton, N., Young, M., Salmon, P., Harris, D., Demagalski, J., Waldmann, 

T., Dekker, S. (2003). Development of the Human Error Template – A new methodology 

for assessing design induced errors on aircraft flight decks. Project Report. 

Militello, L. G. & Hutton, J. B. (2000). Applied Cognitive Task Analysis (ACTA): A 

practitioner’s toolkit for understanding cognitive task demands. In J. Annett & N. S 

Stanton (Eds.), Task Analysis, pp 90-113. UK, Taylor & Francis. 

O’Hare, D., Wiggins, M., Williams, A., & Wong, W. (2000). Cognitive task analyses for 

decision centred design and training. In J. Annett & N. Stanton (Eds.) Task Analysis, pp 

170-190. UK, Taylor and Francis. 

Polson, P. G., Lewis, C., Rieman, J. & Wharton, C. (1992). Cognitive walkthroughs: a 

method for theory based evaluation of user interfaces. International Journal of Man- 

Machine Studies, 36 pp741-773. 

Salmon, P. M., Stanton, N. A., Walker, G., & Green, D. (2004a) Human Factors Design 

and Evaluation Methods Review. Defence Technology Centre for Human Factors 

Integration, Report No. HFIDTC/WP1.3.2/1. 

Salmon, P. M., Stanton, N. A. and Walker, G. 2004b, National Grid Transco: Switching 

operations report. Defence Technology Centre for Human Factors Integration Report. 

Salmon, P. M., Stanton, N. A. and Walker, G. 2004c, National Grid Transco: Return to 

service report. Defence Technology Centre for Human Factors Integration Report. 

Shepherd, A. (2001). Hierarchical Task Analysis. Taylor and Francis, London. 

Stanton, N. A. (2004). Hierarchical Task Analysis: Developments, Applications and 

Extensions. HFI-DTC Technical Report. 

Stanton, N. A. & Stevenage, S. V. (1998). Learning to predict human error: issues of 

acceptability, reliability and validity. Ergonomics, 41(11), 1737-1756. 

Stanton, N. A, and Young, M. S. (1999). A guide to methodology in ergonomics: 

Designing for human use, London, Taylor and Francis. 

Stewart, R. J., Harris, D. Stanton, N. A., Salmon, P. Linsell, M. & Dymott, R. (2004). 

HMS Driad: Air, Surface and Subsurface Scenario Report. Defence Technology Centre 

for Human Factors Integration Report. 

100



Walker, G., Stanton, N. A & Young, M. (2002). Hierarchical Task Analysis of Driving 

(HTAoD) 

Walker, G. H., Stanton, N. A., Gibson, H., Baber, C., Young, M., & Green. D. (2005a). 

Analysing the role of communications technology in C4i scenarios: A distributed 

cognition approach. Accepted for publication in Special Issue of Intelligent Systems. 

Walker, G. H., Gibson, H., Stanton, N. A., Baber, C., Salmon, P., & Green, D. (2005b). 

Event analysis of systemic teamwork (EAST): A novel integration of ergonomics 

methods to analyse C4i activity. Accepted for publication in a Special Issue of 

Ergonomics on C4i. 

Watts, L. A., & Monk, A. F. (2000). Reasoning about tasks, activities and technology to 

support collaboration. In J. Annett & N. Stanton (Eds.) Task Analysis. UK, Taylor and 

Francis. 

101



7 Appendix 

The further the point 

from the centre the 

better. 

Anything outside the 

red ring is possitive 

inside is negative 

Medians 

5 

4 

3 

5 

5 5 5 

5 5 

4 4 

4 4 

4 4 4 

4 4 4 

4 4 4 

4 4 4 

3 3 3 

3 3 3 3 

3 3 3 3 

3 3 3 

3 3 3 3 3 

3 

3 3 3 3 3 

Observation 

HTA 

CDA 

CUD 

2 

2 2 2 2 2 

2 

2 

2 

2 

SNA 

OSD 

Percentage use again 

Observation 100% 

HTA 100% 

CDA 89% 

CUD 78% 

SNA 100% 

OSD 89% 

CDM 100% 

PN 100% 

1 

0 

Ease of learning Time to learn Ease of application Time to Apply Usefulness of output Predictability of 

Output 


CDM 

PN 

Observation 

HTA 


5 


5 


4 

Time to learn 


4 

Time to learn 

3 

3 

2 

2 

1 

1 






Time to Apply 


Time to Apply 

CDA 

CUD 


5 


5 


4 

Time to learn 


4 

Time to learn 

3 

3 

2 

2 

1 

1 






Time to Apply 


Time to Apply 

SNA 

OSD 


5 


5 


4 

Time to learn 


4 

Time to learn 

3 

3 

2 

2 

1 

1 






Time to Apply 


Time to Apply 

CDM 

PN 


5 


5 


4 

Time to learn 


4 

Time to learn 

3 

3 

2 

2 

1 

1 






Time to Apply 


Time to Apply 

Figure 7-1 - Review of EAST Methods – Medians 

102



Means 

The further the point 

from the centre the 

better. 

Anything outside the 

red ring is possitive 

inside is negative 



HTA 100% 

CDA 89% 

CUD 78% 

SNA 100% 

OSD 89% 

CDM 100% 

PN 100% 

5 

4 

3 

2 

1 

0 

4.4 4.4 

4.3 

4.2 

4.1 4.1 

4.0 

4.0 

3.9 

3.7 

3.7 

3.7 

3.6 

3.6 3.6 

3.6 

3.6 3.6 

3.3 

3.2 

3.2 3.2 

3.2 3.2 

3.1 

3.1 

3.1 

3.0 3.0 

2.9 2.9 

2.9 

2.8 

2.8 

2.8 

2.7 

2.7 

2.6 

2.4 2.4 

2.4 

2.4 

2.2 

2.2 2.2 

2.0 

1.9 

1.8 


Output 

4.2 4.2 

4.1 

3.9 

3.7 

3.6 

3.1 

3.0 


Observation 

HTA 

CDA 

CUD 

SNA 

OSD 

CDM 

PN 

Observation 

HTA 


5 


5 


4 

Time to learn 


4 

Time to learn 

3 

3 

2 

2 

1 

1 






Time to Apply 


Time to Apply 

CDA 

CUD 


5 


5 


4 

Time to learn 


4 

Time to learn 

3 

3 

2 

2 

1 

1 






Time to Apply 


Time to Apply 

SNA 

OSD 


5 


5 


4 

Time to learn 


4 

Time to learn 

3 

3 

2 

2 

1 

1 






Time to Apply 


Time to Apply 

CDM 

PN 


5 


5 


4 

Time to learn 


4 

Time to learn 

3 

3 

2 

2 

1 

1 






Time to Apply 


Time to Apply 

Figure 7-2 - Review of EAST Methods – Means 

103



Standard Deviations 

2 

1.7 

The further the point from the centre the better 



HTA 100% 

CDA 89% 

CUD 78% 

SNA 100% 

OSD 89% 

CDM 100% 

PN 100% 

1.5 

1 

0.5 

1.4 

1.3 

1.3 

1.2 

1.1 

1.1 

1.1 

1.1 1.1 1.1 1.1 

1.0 

0.9 

0.8 

0.8 

0.6 

1.2 

1.0 

1.0 

0.9 

0.7 

0.7 

0.4 

1.3 

1.3 

1.1 

1.0 1.0 

0.8 

0.7 

0.7 

1.3 

1.2 

1.2 

1.1 

1.1 

1.1 

1.0 

0.9 

0.9 0.9 0.9 0.9 

0.7 0.7 0.7 

0.7 

1.1 1.1 

1.0 

0.9 

0.9 

0.8 0.8 

0.7 

Observation 

HTA 

CDA 

CUD 

SNA 

OSD 

CDM 

PN 

0 


Output 


Observation 

HTA 


2 


2 


1.5 

Time to learn 


1.5 

Time to learn 

1 

1 

0.5 

0.5 

0 

0 






Time to Apply 


Time to Apply 

CDA 

CUD 


2 


2 


1.5 

Time to learn 


1.5 

Time to learn 

1 

1 

0.5 

0.5 

0 

0 






Time to Apply 


Time to Apply 

SNA 

OSD 


2 


2 


1.5 

Time to learn 


1.5 

Time to learn 

1 

1 

0.5 

0.5 

0 

0 






Time to Apply 


Time to Apply 

CDM 

PN 


2 


2 


1.5 

Time to learn 


1.5 

Time to learn 

1 

1 

0.5 

0.5 

0 

0 






Time to Apply 


Time to Apply 

Figure 7-3 - Review of EAST Methods – Standard Deviation 

104



Medians 


5 


4 


3 

How well you think you carried out the EAST methodology 

2 

1 

The breadth of coverage and its ability to describe a range of 

behaviour. 


How likely do you think the methodology is to produce consistent 

results 

Please rate the amount of resources (time & effort) required to 

conduct an evaluation. 

The extent to which the methodology has theoretical foundations. 

Means 


5 


4 


3 


2 

1 


behaviour. 



results 




Standard Deviations 


2 


1.5 


1 


0.5 

0 


behaviour. 



results 




Figure 7-4 - Holistic Review of EAST 

105



Figure 7-5 - EAST User Questionnaire page 1 of 4 

106




107




108




109



- End of Document - 

110

A Review of the Event Analysis of Systemic Teamwork Methodology

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?