polarization in research methods application: examining the examiner

Proceedings
of the
11 th European Conference
on Research Methods
University of Bolton, UK
28-29 June 2012
Edited by
Rachel McClean
University of Bolton, UK

Copyright The Authors, 2012. All Rights Reserved.
No reproduction, copy or transmission may be made without written permission from the individual
authors.
Papers have been double-blind peer reviewed before final submission to the conference. Initially, paper
abstracts were read and selected by the conference panel for submission as possible papers for
the conference.
Many thanks to the reviewers who helped ensure the quality of the full papers.
These Conference Proceedings have been submitted to Thomson ISI for indexing.
Further copies of this book and previous year’s proceedings can be purchased from http://academicbookshop.com
CD version ISBN: 978-1-908272-46-1
CD version ISSN: 2049-0984
Book version ISBN: 978-1-908272-45-4
Book Version ISSN: 2049-0968
Published by Academic Publishing International Limited
Reading
UK
44-118-972-4148
www.academic-publishing.org

Contents
Paper Title Author(s) Page
No.
Preface vi
Conference committee vii
Biographies ix
Polarization in Research Methods Application:
Examining the Examiner
Faculty’s Attitudes Towards Research Supervision
and Postgraduate Students’ Programme
Completion at the University of Ibadan, Nigeria
The Roots and Components of "Workflow' Using
Visual Methodologies
Evaluating Doctoral Education: A Conceptual
Paper
Silent Voices in Organisations: Conscientization as
a Reflexive Research Methodology
The Role of Reflection by the Professional
Information Systems Practitioner in South Africa
A Technical Guide to Designing and Implementing
Effective web Surveys
Utilising a Sociological Analytical Scheme and
Intellectual Device to Critically Appreciate Software
Project Management Methodologies
Validation of Grounded Theory Based Data by
Means of Analytical Mapping Techniques
Ethnographic Research Methods in Business and
Information Systems Research
Quality Criteria for Interpretive Research in
Information Systems: A Reflexion About Klein and
Myers’ set of Principles
Substantiate the Reflexivity: The Insider-Outsider
Role of an Ethnographic Researcher
Students so Close, yet so far away: A Case Study
and Best Practices for Teaching Research Methods
Online
Systematic Review of Empirical and Psychometric
Studies on Organizational Commitment Conducted
in Turkey
Online Formative Assessment: Does it add up to
Better Performance in Quantitative Modules?
Alleviating Design Silence in Design Science
Research: a Proposal of a Design Method
i
Edwin Asiamah Acheampong, Marcia
Mkansi, Kondal Reddy Kondadi and
Baomi Qi
Emmanuel Olufemi Adeniyi and
Maruff Akinwale Oladejo
Saleh Alhalalat
Valerie Anderson and Sarah Gilmore
Andrew Armitage
Udo Richard Averweg
Greg Baatard
Gary Bell
Martin Bergaus, Bernd Stottok and
Andrea Gorra
Ann Brown and Jessica Iacono
Ana Cardoso and Isabel Ramos
Ke Cui
Leslie Dinauer
Serkan Dolma, Ozlu Azakli, Yagizhan
Yazar and Talha Demirbas
Elena Fitkov-Norris and Becky Lees
Francis Gacenga, Aileen Cater-Steel,
Mark Toleman and Wui-Gee Tan
1
7
16
26
34
41
48
55
70
79
87
95
102
110
115
122

Paper Title Author(s) Page
No.
Measuring the Students’ Satisfaction With the
Romanian Higher Education Environment: The
Case of Blended Learning System at the University
of Oradea
The Educational Scorecard: A More Appropriate
Evaluation?
Appreciative Inquiry as Alternative Method to
Assess Service Quality in a Special Library
ii
Adriana Giurgiu, Carmen-Alina Popa
and Adrian Negrea
Vivienne Griggs, Michelle Blackburn
and Joanna Smith
Elrita Grimsley
Generating new Interview Method Yayoi Hirose, Kiyoshi Itao and
Tomohiro Umeda
The new Role of Marketing Research Toward
Sustainability: A Value-Driven Perspective
Student Satisfaction Formation: Linkage Between
Information Satisfaction and College Choice
Satisfaction
Weaving the Threads of Reflexivity: Coming to
Terms With Grounded Theory Research
The use of the Time Diary Method to Explore
Academic Time Management: Insights From an
Australian University
Why Should I Care?” Facilitating Learning in Higher
Education
Establishing and Maintaining Focus in Your
Research: Promoting the use and Effective
Implementation of Focus Group Methodology in a
‘Real-Life’ Research Study
Issues and Challenges of Evaluating Digital Divide
Projects
Thing’ Qualities - Grounded Theory Method,
Discourse Analysis and Semiotics Approaches
Are Employment Tribunals a Barrier to Justice? A
Mixed Methods Approach
Towards Bridging the Quantitative–Qualitative
Divide in Business and Management Research
A Model Development Strategy to Determine
Factors That Influence Knowledge Retention in
Organisations
Somdee Hongphisanvivat
Nurlida Ismail, Faridah Hj Hassan
and Nooraini Mohamad Sheriff
MacDonald Kanyangale and Noel
Pearse
Branka Krivokapic-Skoko, Roderick
Duncan and Kerry Tilbrook
Maria Kyriacou and Panayiotis
Constanti
David Lamb
Keri Logan and Barbara Crump
Arminda Lopes
Jonathan Lord
Hendrik Marais
Searching for a Third way: Self-Justification John Mendy
Research Philosophical Debates and
Classifications: Students’ Dilemma
The Question of Mixed Methods Suitability to RBV
Research: A Literature Review
Ellen Caroline Martins and Nico
Martins
Marcia Mkansi, Edwin Asiamah
Acheampong, Baomi Qi and Kondal
Reddy Kondadi
Marcia Mkansi, Baomin Qi and
Gillian Green
132
144
152
161
171
181
190
199
207
215
222
229
239
248
258
268
277
285

Paper Title Author(s) Page
No.
Solving Problems: A Living Theory Approach to
Research Development
A Journey into Analysis in the Development of a
World of Meta-Science
Researching Resilience as Psychological Resource
Capacity as Possible Predictor of Students'
Academic Experience at a South African University
Cross-Cultural Marketing Research:
Neuromarketing and In-Depth Interview
Utilising Assessment Tasks to Support the
Research of MBA Students
Using Technology to Support and Monitor
Independent Learners
A Methodological Approach to Investigating the
Transferral of Human Resource Management
Practice From a Multinational to the Host Country
Beyond the Glass Ceiling: A Gendered and Cultural
Hospitality Management Discourse on The
Advancement of Women Based on Integrated
Research Paradigms
An Amalgamation of Grounded Theory Method and
Object – Orientation Concept: A Practical Approach
for Theory Development
iii
Graham Trevor Myers
Graham Myers and Verna Yearwood
Rita Niemann and Tina Kotze
Maia Ozdemir
Noel Pearse
Martin Rich
Kate Rowlands
Faith Samkange and Sihle Dingani
Transmissia Semiawan
True Science and Project Management Miles Shepherd and Roger Atkinson 359
Students’ Entrepreneurial Transition in Enterprise
Education: A UK/China Comparison
Parameter Estimation Using Asymptotic Analogy Anthony Stacey
Jiwei Jenny Shi, Naomi Woodier-
Harris and Quan Jin
Research Ethics for Sustainable Development Anthony Stacey and Julie Stacey 391
Eating our own Cooking: Toward a Design Science
of Research Methods
Constructing Homogeneous Likert-Type
Summative Rating Scales According To Classical
Measurement Theory
Interpretative Research Paradigms: Points of
Difference
Application of Cluster Analysis and Discriminant
Analysis in Market Segmentation and Prediction
John Venable and Richard
Baskerville
Marianne Viljoen
292
297
304
313
319
326
333
341
349
368
380
399
408
Nevan Wright and Erwin Losekoot 416
Ruth Yeung and Wallace Yee 423
PHD Papers 431
Application of Delphi Expert Panel in Joint Venture
Projects in Malaysian Construction Industry
Action Learning and the Environmental Factors in
Developing Engineering Expertise
Hamimah Adnan, Azizan Supardi,
Zul Zakiyuddin Ahmad Rashid, and
Norazian Mohamad Yusuwan
Matthew Bell
433
446

Paper Title Author(s) Page
No.
An Insight Into the Promise and Problems of
Combining Life History and Grounded Theory
Research
The Doctoral Level Practitioner: The Development
of new Knowledge Through an ‘All Channels Open
Project’ for the Teaching of Biosciences in National
and International Nurse Education
Can Post Modernism Contribute to Saving the
World?
Modeling Organisational Trust: Addressing
Challenges of Measurement Invariance
Health and Safety of a Project Design: Professional
Designers and Malaysian Legal Perspective
Promoting Innovation in Data Poor Public and
Private Business Areas Through Systems Analysis
Appreciative Leadership: Dreaming the Underlandto-Wonderland
Dream for Managing a Higher
Education Institution
Developing Methodology for Subcontractors’
Security of Payment Under Malaysian Construction
Industry
Researching Implementation of Quality Assurance
Policy in Malaysian Private Higher Education
Methodology for Investigating Rationale for
Unsuccessful EOT Claim in Malaysian Construction
Industry
The Study of Strategy-Making Process in UK
Universities: An Interpretative Approach
iv
Bronwyn Betts
John Lorimer Campbell, Robert
Campbell and Gill Green
Christine Gilligan
Carvell McLeary and Paula Cruise
Zul Zakiyuddin Ahmad Rashid,
Hamimah Adnan, Azizan Supardi,
and Norazian Mohamad Yusuwan
Chiang Ren, Gillian Green and
Robert Wood
Emmie Smit and Rita Niemann
Azizan Supardi, Hamimah Adnan,
Zul Zakiyuddin Ahmad Rashid, and
Norazian Mohamad Yusuwan
Siew Fun Tang and Sufean Hussin
Norazian Mohamad Yusuwan,
Hamimah Adnan, Azizan Supardi
and Zul Zakiyuddin Ahmad Rashid
Lilia Zerguit
Work In Progress Papers 537
Focus Groups as Evaluation: Exploring Issues
Connected With “Insider” Research
The Drivers Behind the Establishment of a
Dissertation Supervision Process in the French
Grande Ecole System
Deborah Anderson
Marie Ashwin and Alan Hirst
Community Participants Wellbeing Karen George, Petia Sice, Robert
Young, Safwat Mansi, and Jeremy
Ellman
Enhancing Students Academic Skills: The Case of
the Bucharest Academy of Economic Studies
Irina Purcarea
Abstracts only n/a
Embedding Research Data Management
Behavioural Change Through a Policy, Systems
and Human Support Infrastructure
Lindsay Wood, Niall O'Loughlin and
Janet Wheeler
446
453
461
470
480
489
499
508
516
523
527
539
542
548
552
n/a

Paper Title Author(s) Page
No.
Towards an Outcome-Based Approach to
Literature Review for Business and Management
Research Students
v
Kambidima Wotela
n/a

Preface
The 10th European Conference on Research Methodology for Business and Management Studies is
hosted this year by the University of Bolton. The Conference Chair is Gill Green I am pleased to be
Programme Chair.
ECRM has become a recognised event on the European research conferences calendar and provides
the opportunity for individuals, working in the area of business and management research to meet
and discuss their experiences of using the varied and expanding range of research methods available
to them.
We are pleased to welcome three keynote speakers this year who will address diverse topics. Bob
Wood from University of Manchester, UK will look at Mixed, Multi and Muddled: Methods, Paradigms
and Research Integrity. Professor Peter Kawalek, also from the University of Manchester will talk
about The Struggle to be Engaged and Relevant and Dr Tony Hirst from the Open University, UK will
open up the proceedings on the second day.
In addition to the main themes of the conference there are mini tracks on Researching the Student
Experience in Higher Education, Teaching Research Methods, Mixed Research Methods, Project
Management Research Methods and Ethnographic Research Methods. For the third year ECRM is
incorporating a PhD Colloquium and we have been impressed with the range of research and calibre
of work that has been submitted for this.
With an initial submission of 139 abstracts, after the double blind, peer review process there are 53
academic papers, 13 PhD papers and 4 work in progress papers published in these Conference Proceedings.
These papers come from some 22 different countries including Australia, Austria, Bosnia
and Herzegovina, China, Cyprus, Czech Republic, France, Indonesia, Iran, Japan, Malaysia, New
Zealand, Nigeria, Portugal, Romania, Saudi Arabia, South Africa, Switzerland, Thailand, Turkey,
United Kingdom, and the USA.
A selection of the best papers – those agreed by a panel of reviewers and the editor will be published
in a conference edition of the EJBRM (Electronic Journal of Business Research Methods
www.ejbrm.com). These will be chosen for their quality of writing and relevance to the Journal’s objective
of publishing papers that offer new insights or practical help into the application of research methods
in business research.
I wish you a most interesting conference.
Rachel McLean
Programme Chair
June 2012
vi

Conference Executive
Dr Marie Ashwin, Normandy Business School, France
Dr Ann Brown, CASS Business School, London, UK
Dr Gill Green, University of Bolton, UK
Dr Rachel McLean, University of Bolton, UK
Mini track Chairs
Dr Andrew Armitage, Anglia Ruskin University, UK
Dr Marie Ashwin, Normandy Business School, France
Dr Roger Atkinson, Bournemouth University, UK
Dr Gary Bell, London South Bank University, UK
Dr Angela M. Benson, University of Brighton, UK
Dr Ann Brown, City University London, UK
Dr Renalde Huysamen, University of the Free State, South Africa
Jonathan Lord, University of Salford, UK
Professor Julie McLeod, Northumbria University, UK
Professor Jim Stewart, Leeds Metropolitan University, UK
Dr Kevin Voges, University of Canterbury, New Zealand
Professor Jon Warwick, London South Bank University, UK
Conference Committee
The conference programme committee consists of key people in the entrepreneurship and innovation
community, both from the UK and overseas. The following people have confirmed their participation:
Khodayar Abili (The University of Tehran, Iran); Bulent Acma (Anadolu University, Eskisehir, Turkey);
Hamimah Adnan (Universiti Teknologi MARA, Malaysia); Gisela Schulte Agyeman (Learning Services,
Hull City Council, UK); Maizam Alias (Universiti Tun Hussein Onn Malaysia, Malaysia); George
Allan (University of Portsmouth, UK); Helena Alves (University of Beira Interior, Covilhã, Portugal);
Andrew Armitage (Anglia Ruskin University, UK); Marie Ashwin (Ecole de Management de Normandie,
France); Gabriela Avram (University of Limerick, Ireland); Joseph Azzopardi (University of
Malta, Malta); Sue Balint (Westminster Business School, London, UK); Joan Ballantine (University of
Ulster, UK); Heather Banham (Okanagan College, Kelowna, Canada); Frank Bannister (Trinity College,
Ireland); Edward Barratt (University of Newcastle Upon Tyne, UK); Andrew Basden (University
of Salford, UK); David Bednall (Deakin University, Australia); Peter Bednar (Department of ISCA,
Portsmouth University, UK); Lau Bee Theng (Swinburne University of Technology Sarawak Campus,
Malaysia); Gary Bell (London South Bank University, UK); Angela Benson (University of Brighton,
UK); Egon Berghout (University of Groningen, Netherlands); Frank Bezzina (University of Malta,
Malta); Milena Bobeva (Bournemouth University, UK); Stephan Boehm (RheinMain University of Applied
Sciences, Germany); Laurent Bourdeau (Faculty of Business Administration at Université Laval,
Canada); Ann Brown (CASS Business School, UK); Cathal Brugha (University College Dublin, Ireland);
Roslyn Cameron (Central Queensland University, Australia); Marian Carcary (University of Limerick,
Ireland); Sven Carlsson (School of Economics and Management, Lund University, Sweden);
Felice Corona (University of Salerno, Italy); Prasenjit Chatterjee (MCKV Institute of Engineering, India);
Vikas Choudhary (National Institute of Technology, Kurukshetra, India); Jyoti Choudrie (University
of Hertfordshire, UK); Murray Clark (Sheffield Business School, UK); Aileen Corley (Liverpool
John Moores University, UK); Reet Cronk (Harding Universtity, USA); Barbara Crump (Massey University,
New Zealand); Geoffrey Darnton (Bournemouth University, UK); Linda Dawson (Monash University,
Australia); David De Vaus (University of Queensland, Brisbane, Australia); Verena Dorner
(Chair of Business Computing II, University of Passau, Germany); David Douglas (Staffordshire University,
UK); Prokopios Drogkaris (University of Aegean, Greece); Ken D'Silva (London South Bank
University, UK); Dev Dutta (University of New Hampshire, Durham, UK); Rohit Dwivedi (Rajiv Gandhi
Indian Institute of Management Shillong, India, www.iimshillong.in); Hossein Ebrahimpour (The University
of Kashan, Iran,); Jose Esteves (Instituto de Empresa, Madrid, Spain); Mahmoud Fakhra (College
of Business Studies, Kuwait); Jason Ferdinand (University of Liverpool, UK); Paula Odete Fernandes
(Polytechnic Institute of Bragança, Portugal); Adriana Giurgiu (University of Oradea, Faculty
of Economic Sciences, Romania); Gerald Goh (Multimedia University, Melaka, Malaysia); Karuna
Gomanee (Regents College, UK); Andrea Gorra (Leeds Metropolitan University, UK); Gillian Green
vii

(School of Informatics, University of Northumbria, UK); Ray Hackney (Brunel Business School, UK);
Yunke He (Okanagan College, Kelowna, Canada); Alex Hiller (Nottingham Trent University, United
Kingdom); Alan Hirst (London South Bank University, England); Clive Holtham (Cass Business
School, City of London University, UK); Bob Hughes (University of Brighton, UK); Gordon Hunter
(University of Lethbridge, Canada); Rahinah Ibrahim (Universiti Putra Malaysia, Malaysia); Fahmi
Ibrahim (Glasgow Caledonian University, UK); Mohammad Intezar (King Saud University, Saudi Arabia,);
Wyn Jenkins (Staffordshire University, Stoke-on-Trent, UK); Kristina Jesperson (Aarhus University,
Denmark); Bev Jones (Management Research Centre, Wolverhampton Business School, UK);
Patricia Joubert (University of Swaziland, Swaziland); Helena Karjalainen ( Normandy Business
School, France); Diane Keeble-Allen (Anglia Ruskin University, UK); Habibul Khondker (Zayed University,
Abu Dhabi, UAE); Cyril Kirwan (Cyril Kirwin Associates, Dublin, Ireland); Rembrandt Klopper
(University of Kwa Zulu Natal, South Africa); Jenny Knight (University of Brighton Business School,
UK); Deborah Knowles (University of Westminister, London, UK); Karl Knox (Nottingham Trent University,
UK); Mortaza Kokabi (Shaheed Chamran, Iran); Branka Krivokapic-Skoko (Charles Sturt University,
Australia); João Leitão (Technical University of Lisbon, Portugal); Walter Leite (University of
Florida, USA,); Arminda Lopes (Instituto Politecnico, Portugal); Eurico Lopes (Polytechnic Institute of
Castelo Branco, Portugal); Jonathan Lord (University of Salford, UK); Sam Lubbe (University of South
Africa, South Africa); Sandy MacDonald (University College Northampton, UK); Bob Macklin (Charles
Sturt University, Australia); Dalgobind Mahto (Green Hills Engineering College, India); Garance Marechal
(University of Liverpool, UK); Maria do Rosário Martins (Universidade Cape Verde, Portugal);
Lisa McNeill (University of Otago, Dunedin, New Zealand); Ian Michael (Zayed University, Dubai,
United Arab Emirates); Elisabeth Michielsens (University of Westminister, London, UK); Avinash
Mulky (Indian Institute of Management, Bangalore, India); Kersti Nogeste (RMIT University, Melbourne,
Australia); Chetsada Noknoi (Thaksin University, Thailand,); Miguel Nunes (Sheffield University,
UK); Madeleine Ogilvie (Edith Cowan University, Perth, Australia); Steve Page (University College
Chester, UK); Noel Pearse (Rhodes University, Grahamstown, South Africa); Ganesh Prabhu
(Indian Institute of Management at Bangalore, India); Diana Rajendran (Swinburne University of
Technology, Melbourne, Australia); Thurasamy Ramayah (Universiti Sains Malaysia, Malaysia); Karsten
Boye Rasmussen (University of Southern Denmark, Odense, Denmark); Theo Renkema (Rabobank
Nederland, Netherlands); Martin Rich (CASS Business School, UK); Bikramjit Rishi (University
of Liechtenstein, Liechtenstein,); Kristina Risom (The Aarhus School of Business, Denmark); Bob
Ritchie (Lancashire Business School, UK); Isabelle Royer (University Lyon 3, France); Maria Ryan
(Edith Cowen University, Perth, Australia); Faith Samkange (SHMS Hotel Belvedere, Switzerland);
Joseph Santora (Thomas Edison State College, Trenton, USA); Risto Säntti (University of Vaasa,
Finland); Mark Saunders (University of Surrey, UK); Junaid Shaikh (Curtin University, Malaysia); Anshuman
Sharma (College of Applied Sciences, Ministry of Higher Education, Sultanate of Oman,);
Marie Sheahan (Charles Sturt University, Australia); Chandranshu Sinha (Amity Business School,
Amity University, India); Ali Simsek (Anadolu University, Turkey); Ibrahim Sirkeci (Regent's College,
London, UK); Peter Smith (University of Sunderland, UK); Rajeev Srivastava (Motilal Nehru National
Institute of Technology, India); Jim Stewart (Leeds Business School, UK); Kerstin Thomson (Stockholm
University, Sweden); Claudine Toffolon (Université du Mans - IUT de Laval, France); Shiv Tripathi
(Mzumbe University, Tanzania,); Jim Underwood (University of Technology, Sydney, Australia);
Arvind Upadhyay (University of Bergamo, Italy); Jocene Vallack (CQ University, Rockhampton, Australia);
Ana Vasconcelos (University of Sheffield, UK); John Venable (Curtin University of Technology,
Perth, Australia); Kevin Voges (University of Canterbury, Christchurch, New Zealand, New Zealand);
David Wainwright (University of Northumbria, UK); Jon Warwick (London South Bank University, UK);
Teresa Waring (Northumbria University, UK); Christine Welch (University of Portsmouth, UK); Roy
Williams (University of Portsmouth, UK); Michael Wood (University of Portsmouth, UK); Les Worrall
(University of Coventry, UK); Mingming Zhou (Nanyang Technological University, Singapore); Lilia
Zerguit (Sheffield Business School, Sheffield Hallam University, UK).
viii

Biograaphies
Confereence
Chair
Programmme
Chair
Dr Rachhel
Mclean jooined
the Unniversity
of BBolton
as a Principal
Lecturer
(Creativve
Technologies
and Teaching
& LLearning)
in DDecember
2008.
Prior too
this I was aat
Manchester
Metropoolitan
Universsity
Businesss
School (Se enior Lectureer
in Businesss
Informattion
Technoloogy
and morre
recently AActing
Chair of o Doctoral PProgrammess).
Before joining
MMUU
she was a full time PhhD
student at a Salford UUniversity.
Heer
current rresearch
inteerests
lie in thhe
area of soociology
of in nformation coommunicatioon
technoloogies
and neew
media or social technnologies
in a range of coontexts
includd
ing home,
work and society. Thiss
research drraws
upon co ommunicatioon
studies, mmedia
studies s, cultural
studies, literary theoory
and socioology.
She hhas
an intere est in research
methodss
and seeks to make
both meethodological
as well as ppractical
and theoretical contributions
c
to her reseaarch.
Keynotee
Speakers
Prrofessor
Peeter
Kawaleek
is Profess sor of Informmation
Systeems
and Str rategy at
Manchester
BBusiness
Schhool.
He is also a a Visitinng
Professor
at Instituto o de Em-
prresa
in Madrrid
and Letteerkenny
Instit tute of Technology
in Coounty
Doneg gal. He is
ann
external exxaminer
at TTrinity
Colleg ge, Dublin. PPeter's
PhD is in Comp puter Sci-
ennce.
He has experience to board leve el in media aand
rail industries,
and works w extensively
with government,
including Office O an Taoiseach,
Depaartment
of Communi- C
ties and Local Goverrnment,
the NNHS,
Leeds City Council l, Tameside Council and Salford City Council.
Dr Tony
Hirst is a lecturer in ICCT
at the Oppen
Universit ty in the UK. . With a backk
ground aand
publication
history inn
the areas oof
evolutiona ary computattion
and artiffi
cial intelligence,
he has co-authhored
severaal
Open University
coursses
on topiccs
ranging from roboticcs
to informaation
literacy. . A founder member m of the
Open Unni
versity RRobotics
Outtreach
Groupp,
Tony is a firm believer
in wideningg
participatioon
and pubblic
engagemment
in sciencce
and technnology.
He re egularly runss
partial attenn-
tion hands-on
workshops
about ssocial
softwaare
and web search.
Professsor
Bob Woood
is Profeessor
Emeritus
of Inform mation Systems
in the MManchester
Business B
School aat
the University
of Mancchester
followwing
several
years as thhe
Head of SSchool
of Info ormatics.
He is thhe
co-author of a text onn
the Develoopment
of Web W Information
Systems
and has published p
widely inn
the area oof
the social and organizational
aspe ects of informmation
systemms
developm ment and
use. Hiss
current reseearch
interessts
focus on iissues
to do with the straategic
deployyment
of new w collaborative
teechnologies
in areas succh
as Knowlledge
Manag gement, Bussiness
Intelligence
and Business B
Analyticss.
Mini Traack
Chairs
GGill
Green iss
Reader in Business Te echnologies in School business
and creative
teechnologies
at Universityy
of Bolton. Co-ordinator
of doctoral school, wor rked pre-
vviously
for Thhe
Universityy
of Durham m, University of Sunderlaand
and Nor rthumbria
UUniversity
witth
domain foocus
on Infor rmation systeems
and ressearch
metho ods. Cur-
reent
interestss
are in area of open sou urce softwaree
initiatives, SSMEs,
organ nisational
leearning
and music and ccreative
indu ustries. Treassurer
of UKAAIS
( UK Aca ademy of
Innformation
SSystems)
Andrew
Armitaage,
Fellow of the Highe er Education Academy, aand
lectures in Manageement
Develoopment
and Research Methods M at the
Ashcroft Innternational
Business B
Schhool
on postggraduate
andd
undergrad duate programmes.
He hhas
been a tutor t with
the Open Univeersity
for eighhteen
years and is curreently
tutoringg
in the OU Business
B
Schhool
on the MMBA
B852 Reesearch
Methods
modulee.
ix

Dr Mariee
Ashwin haas
almost twwo
decades oof
experience e in higher eeducation
in tthe
UK and the Far Easst.
Marie joinned
the Normmandy
Busin ness School two years aago
where sshe
works with
colleaguees
from around
the world d on teachingg
and research
projects to enhance the standingg
of RM amoongst
staff an nd students, and add value
to the teeaching
of methodology.
A qualitativee
researcher r by nature, sshe
is currenntly
inveestigating
Soocial
Networkking
behavio our with an innternational
rresearch
team m.
Dr Gary Bell is a Researrch
Fellow at a London SSouth
Bank UUniversity
and
a co-
founder
of the SSocial,
Finanncial
& Socia al Systems (SFSR) centtre.
He is de eveloping
andd
applying thhe
Holon Framework
for
Software PProject
Manaagement
and
Higher
Eduucation
Manaagement.
Dr. Annn
Brown Sennior
Lecturerr
in Information
Managem ment, Facultyy
of Manageement,
Cass Buusiness
Schhool.
Doctoraate
from Citty
University y (2005), baased
on worrk
into
problemms
and potenntial
of Informmation
Systeems
applicati ions to creatte
Business Value
for orgaanisations.
Editor of EElectronic
Joournal
of Business B Reesearch
Meethods
(EJBRMM).
Researchh
interests innclude
IS implementation
n; IS exploitaation
by organisa
tions; reesearch
methhods
for bussiness
and eexploitation
of o electronic tools for teaaching
and learrning.
Jonathan
Lord is Lecturerr,
HR Manag gement and Employmennt
Law, Univ versity of
Salfford.
Was HR
Director, MManager
and d Consultantt,
working accross
all sect tors, spe-
cifically
transporrt
and construuction
indust tries. HR proojects
within public/volun ntarysec- tor. Undertook rresearch
involving
analysis
of Recruuitment
and Selection tec chniques
within
Private, PPublic
and Voluntary
sec ctors, analysiing
reasons why there are a differ-
encees
in techniqques.
Charteered
Fellow, Chartered Institute
of PPersonnel
an ndDevel- opmment.
Currenntly
researchhing
purpose e of employmment
tribunals
and their effective- e
ness usiing
mixed meethod
approaach
Biograaphies
of Presentinng
Authorrs
Edwin AAsiamah
Accheampong
is a Senior Public Serva ant from Ghaana
and a first-year
PhD D student
at the University
of BBolton,
UK. HHis
academicc
research is s focused on Knowledge Managemen nt (KM) in
public seector
organizzations
(PSOOs).
His brooader
acade emic interest includes addaptation
of KM K prac-
tice frammework
in thee
advanced wworld
for sysstematic
infus sion into PSOOs
in developping
countrie es.
Saleh AAlhalalat
recceived
a PhDD
from Lougghborough
University,
UKK
in 2005. HHe
has more e than 12
years off
manageriall
and techniccal
experiennce.
He is ce ertified in Baalanced
Scorrecards
(BSC C), Busi-
ness Proocess
Managgement
(BPMM),
and Six SSigma.
Three
years ago, , Dr. Alhalalaat
earned the e prize of
Excellennce
Researchh
in FFF Proogram
in Jorddan.
Deborahh
Andersonn
is a principaal
lecturer in marketing at a Kingston BBusiness
Schhool,
part of Kingston
Universiity
in the UKK.
Research iinterests
incllude
marketing
communiications
but aalso
focus on
educa-
tional research
with a particular eemphasis
onn
learning and d teaching inn
higher educcation.
Dr. Valeerie
Andersson
is respoonsible
for ddoctoral
deve elopment processes
within
the Univ versity of
Portsmoouth
Graduatte
School. Heer
research iinterests
are e in learning aand
development
in HE and a work
organisaations;
in parrticular
the eevaluation
of f learning and d the use of f measures aand
metrics. She is a
committeed
‘practitionner-researchher’
seeking to foster bo oth research--led
practice and practic ce-led re-
search aagendas.
Udo Ricchard
Averwweg
is an Infformation
Teechnology
(IT T) Project Maanager
at eTThekwini
Mun nicipality,
Durban, South Africaa.
He entereed
the IT induustry
in 197 79 and holds a Masters TTechnology
degree d in
Informattion
Technology
(cum laaude),
a secoond
Masters s degree in SScience
fromm
University of Natal
and a thhird
Masters degree in Coommerce
froom
University y of KwaZuluu-Natal
(UKZZN),
South Africa. A He
has been
appointed as an Honorrary
Researcch
Fellow at UKZN.
x

Ozlu Azakli works as a research assistant at Istanbul University at the Department of Organizational
Behavior. She has studied management at Bilkent University in Ankara. Then, she has studied human
resources management at Yildiz Technical University Graduate School. Now, she is a Ph.D. student
at Marmara University at the Department of Organizational Behavior.
Dr Greg Baatard teaches computer science at Edith Cowan University in Western Australia, specialising
in web development, programming and databases. His research interests include web-based
applications, e-learning, Computer Supported Collaborative Work (CSCW) and computer-supported
research methodologies.
Matthew Bell is an Integrated Project Team (IPT) Leader with experience in developing and deploying
aircraft maintenance, training and capability insertion programmes through concept, design to inservice
deployment. He has worked for British Aerospace, BAE SYSTEMS and completed a PhD by
research in maintenance optimizing programmes for the Typhoon, Tornado, Hawk, Harrier and Nimrod
aircraft.
Martin Bergaus is a PhD candidate at Leeds Metropolitan University, UK. Beside 17 years of professional
experiences in the ICT, he holds 3 master degrees. He teaches as an associate lecturer at the
University of Applied Science in Vienna. His current research interests focus on a qualitative research
using Grounded Theory in the field of Ubiquitous Computing.
Doctor Bronwyn Betts is a Senior Lecturer at Anglia Ruskin University. She has lectured extensively
on Human Resource Management programmes over a period of 26 years. She is an external
examiner for CIPD programmes and has served on validation panels as an external advisor. Research
interests include workplace learning and international HRM.
John Lorimer Campbell is a Senior Lecturer in nursing studies at the University of Cumbria, teaching
bioscience and other clinical topics at pre and post registration levels. In addition to nursing and educational
training he also holds academic qualifications in biology and health science. John produces
educational materials for international distance learning.
Dr Aileen Cater-Steel is Associate Dean (Research) in the Faculty of Business and Law, and Associate
Professor of Information Systems at USQ, Toowoomba, Australia. She leads two industry
funded research projects in IT Service Management. Aileen’s work has been published in many international
IS journals and conference proceedings and she has compiled three edited books.
Panayiotis Constanti teaches in the fields of human resources and organizational behaviour with his
main research interests focusing on people at work and learning and teaching. He has published in a
number of academic journals including the Service Industries Journal; Education and Training; the
International Journal of Education Management and the International Journal of Contemporary Hospitality
Management.
Barbara Crump’s research involves evaluation of digital divide projects and research projects investigating
the culture of the computing tertiary and work environments and collaborated with research
colleagues from Japan, Malaysia and the UK. She is a Senior Lecturer in the information systems
group in the School of Management, Massey University, Wellington, New Zealand.
Ke Cui is a PhD student specializing in sociology and social policy. She employed life-history and
narrative research methods to explore the relationship between parental expectation and their children’s
status attainment. She got a bachelor’s degree of economics and management in China Agricultural
University in 2009.
Leslie Dinauer is Professor and Program Director for Research Methods in the Doctor of Management
Program at University of Maryland University College. She has been developing and teaching
research methods online for over eight years. Her secondary research program investigates the cognitive
processes of attitude formation and change, particularly within the context of consumer behavior.
Elena Fitkov-Norris is Principal Lecturer in Information Management at the Department of Informatics
and Operations Management, Kingston Business School, London, UK. She received a BSc in
xi

Mathematics and Management from King’s College London and an MSc and a PhD from University
College London. Dr. Fitkov-Norris’s research interests are the relationship between study habits and
learning.
Karen George Researches Information and Communication Management in Community Participation,
aiming to investigate complex characteristics of communication and information management
amongst stakeholders in community participation, and to explore applicability of information communication
technologies for community participation to support collaborative practices and multi-criteria
decision making.
Christine Gilligan is senior lecturer at Sheffield Business School, Sheffield Hallam University, teaching
and researching in the area of Sustainable Development, complexity, strategy and change. She
has a particular interest in voluntary sector/social enterprise governance and is studying for a DBA
entitled 'Sustainable Development in the Voluntary Sector: a complex problem'.
Ass. Prof. Dr. Adriana Giurgiu is a post-doctoral researcher in the project: “economic scientific research,
reliance of human welfare and development in the European context”, financed by the European
social fund and the Romanian government through the Sophrd/Posdru 2007-2013posdru/89/1.5/s/62988;
beneficiary: Ince-Romanian Academy; Jean Monnet module leader "eu sustainable
development and competitiveness"; specialist in european economics and international business.
Vivienne Griggs is Course Leader for the Post-Graduate Diploma at Leeds Business School. She
teaches on a range of courses specializing in learning and development and business skills. Prior to
moving to academia ten years ago Vivienne was an HR Manager and maintains a strong business
focus through research and as an Employment Tribunal panel member.
Elrita Grimsley is a Librarian in the special library, Information Service on Higher Education. She
holds a Masters Degree in both Library Science as well as Higher Education Studies. She holds the
post of Librarian of this service for the past 18 years.
Yayoi Hirose is a Research Manager in Japan Science and Technology Agency, which is an independent
public body of the Ministry of Education, Culture, Sports, Science and Technology (MEXT).
She manages commercialization programs of academic technology, sponsored by Japanese government,
and has been collaborating with technology researchers to generate new application of advanced
research technology for over 10 years.
Somdee Hongphisanvivat is an Associate Professor (Ph.D) at the Faculty of Commerce & Accountancy,
Thammasat University, Thailand. She is specialized and experiencing in marketing research,
marketing strategy and planning. Her research focus is currently on sustainable development, value
creation, new concept/direction of marketing management and research.
Nurlida Ismail has been in the teaching profession with more than 20 years of experience. Her research
has been published in international conference proceedings and journals. Her research interests
are in the areas of marketing specifically consumer behavior and higher education marketing.
She is a senior lecturer and is currently pursuing her PhD in Marketing.
MacDonald Kanyangale is a Senior Lecturer at Rhodes Business School at Rhodes University,
South Africa.He lectures strategic management, Operations Management and Entrepreneurship. His
research interests embrace strategy as practice, strategic leadership in small and medium enterprise
development and performance.
Dr Branka Krivokapic-Skoko is a Senior Lecturer in Management at the School of Management and
Marketing, Charles Sturt University, Australia. She published in the area of mixed methodology, and
the Boolean based comparative method. She is currently a Chief Investigator on two large projects
funded by Australia Research Council and Rural Industry & Rural Development Corporation.
David Lamb is a Senior Lecturer in Sport, Recreation and Events Management at Edith Cowan University
in Perth, Western Australia and lectures on a range of sport/event/recreation management
xii

units and supervises postgraduate student research in the same areas. He is a keen advocate of using
qualitative methods in his research, such as focus groups.
.
Rebecca Lees is the Director of Undergraduate Studies for Kingston Business School and a lecturer
within Informatics and Operations Management. She holds a BA Business Studies from Kingston
University and MSc Computer Science from Birkbeck College. Her teaching focuses on Quantitative
Methods and Management Science, with research interests in formative assessment and teaching
and learning.
Arminda Guerra Lopes, Degree in Languages and Modern Literature, Universidade Clássica, Lisboa.
Msc in Speech and Language Processing, Essex University, Uk. PhD in Human Computer Interaction,
Leeds Metropolitan University, UK. Professor at Instituto Politécnico of Castelo Branco, Portugal.
Research Interests: Interaction Design, Interdisciplinary Collaboration, Research Methodologies,
Social Informatics.
Hendrik (Bok) Marais holds a Ph D in experimental social psychology and specialises in research
methodology and management as well as innovation policy. He is currently Professor in the Graduate
School of Technology Management at the University of Pretoria
Ellen Martins is a co-director of Organisational Diagnostics. She holds a doctorate degree in Information
Science focusing on knowledge retention. Her Masters degree (cum laude) focused on the impact
of organisational culture on creativity and innovation. She has extensive experience in numerous behavioural
assessments and qualitative research methods such as interviews and focus groups
Carvell N. McLeary is a doctoral candidate in Organisational Behaviour in the Faculty of Social Sciences,
University of the West Indies, Jamaica. He is the Senior Director of Human Resources Management
at the Norman Manley International Airport. He is a member of the Academy of Management
and the European Group for Organization Studies.
Dr John P. Mendy is at The University of Lincoln. He is currently conducting a post-doctoral study on
people’s experiences of organisational change, culture and ethical issues. Other research areas that
fascinate him are employee-management-societal interactions and the processes of becoming an
independent person during challenging periods of personal, organisational and social development.
Marcia Mkansi is currently pursuing a PhD at the University of Bolton with special focus in ebusiness.
My research interest is in e-business application (ERPs, CRM etc) in supply chain management.
The current research focuses on issues relating to management of supply and distribution
by four major UK grocery e-retailers from a resource-based view perspective.
Graham Trevor Myers Senior Lecturer, Durban University of Technology where he teaches Taxation,
and Research Methodology and Financial Management. Degrees from The University of Natal,
Technikon Natal, and University of Wales, studies have covered Soils Science, Educational Psychology,
Management and Business Studies. Having established numerous campuses for polytechnic in
early 1990’s, as well as numerous courses over same period, interest is really in teaching.
Ass. Asistent Adrian Negrea, ph.d.s., is doctoral researcher at the national institute of economic
research “costin c. kiritescu” (ince), romanian academy, in the field of economics and international
business, supervised by the academician aurel iancu. since 2010, he is also associate member of the
department of international business, faculty of economic sciences, University of Oradea, Romania.
Prof Rita Niemann holds a Ph.D. in Education Management and a MBA, specialising in Higher Education
management and postgraduate teaching. She is Director for Research and Postgraduate Studies
in the Faculty of Education. She lectures and provides supervision to Master's and Ph.D.-students
in Higher Education Studies and does the research training for the university's Business School.
Dr. Maruff Akinwale Oladejo Faculty in Department of Educational Foundations, Federal College of
Education (Special), Oyo State, Nigeria. PhD in Educational Management from University of Ibadan,
Nigeria. Experience includes Educational Planning, Policy and Efficiency of Open-Distance Learning.
Member of Editorial Advisory Board, Journalsbank Publishing Inc, United Kingdom. Participated at
ECEL2010, held at University of Brighton, Brighton, UK.
xiii

Niall O’Loughlin is the ‘Policy and Information Officer’ for Research & Enterprise Services at Newcastle
University. His role involves the maintenance, development and co-ordination of the systems /
guidance which support the institutions funded research and commercial activities. He has a distinct
responsibility for the ethical policies and guidelines.
Maia Ozdemir PhD candidate at Tomas Bata University in Zlin, Czech Republic. Currently working
over her Dissertation in the field of International marketing. Research has been published in Journal
"Problems of Management in the 21st Century", her articles have appeared in a few conference proceedings
including WSEAS / EURO-SIAM / EUROPMENT International Conferences Montreux Switzerland
2011.
Noel Pearse is an Associate Professor in the Rhodes Business School of Rhodes University, where
he is the Academic Co-ordinator. He teaches in the areas of Research, Organisational Behaviour and
Leadership. His current research interests are leadership development, change management, teaching
of research methods, and the use of grounded theory.
Ass.Prof.Dr. Carmen-Alina Popa is ph.d. in education sciences and member of the council of the
department of education sciences of the faculty of social-humanistic sciences, university of oradea;
since 2010, she is also responsible for reduced frequency education bachelor programs within the
university of oradea, department for distance learning and reduced frequency education (didifr).
Irina Purcarea graduated from the Bucharest Academy of Economic Studies, Business Administration
specialization (English language). She is assistant lecturer at the UNESCO Department for Business
Administration, Faculty of Business Administration in foreign languages. Irina Purcarea’s research
interests lie in the field of innovation management, innovation in SMEs and quality management
in higher education.
Isabel Ramos is an assistant professor at the Department of Information Systems, Universidade do
Minho, Portugal, where she also coordinates the research group Knowledge Management and Organizational
Learning. She has served as the secretary of IFIP TC8, and has also been awarded the IFIP
Outstanding Service Award.For more information, consult her website at
http://www.dsi.uminho.pt/~iramos.
Martin Rich initially worked in the information systems sector as a consultant and project manager,
before joining Cass Business School as a lecturer. During his career at Cass he has taken responsibility
for a series of innovative applications of technology to management learning, all of them underpinned
by a thorough research foundation
Kate Rowlands Lecturer in Human Resource Management at Salford Business School, University of
Salford for five years and working towards PhD in field of International Human Resource Management
at University of Manchester. Recently won Vice Chancellor Scholarship for Early Career Research
Excellence (September 2011). Prior to academia held position of Senior Personnel Manager, Tesco
Stores PLC (10years) which involved variety of roles at Senior Management level.
Faith Samkange PhD. is a lecturer and Research Consultant at the Swiss Hotel Management School
University Centre, Switzerland in Tourism and Hospitality Management and Research. She is interested
in Development Research. Her international university teaching and research experience spans
over 20 years in Africa, Europe and America.
Transmissia Semiawan is a senior lecturer at Politeknik Negeri Bandung, Bandung Indonesia. She
received BSc, MIT, PhD from University of Kentucky, USA, Queensland University of Technology,
Australia and Leeds Metropolitan University UK consecutively. Having a strong milieu in information
systems and information management, she has research interests that ranged across the area of
qualitative method.
Miles Shepherd is an independent consultant specialising in programme and project management in
the business and public sectors. He is Chair of the ISO Committees on project management and
works extensively in the Higher Education area in UK and overseas. He is Past President of IPMA
and Past Chairman of APM.
xiv

Jenny Shi is a lecturer in Enterprise and Employability at UCLan, she worked across many private
and public sectors, both in the UK and China for the last 19 years. She has been actively engaged in
several Prime Minister’s Initiative projects, helping to promote British entrepreneurship education and
also to foster partnerships between the two countries.
Emmie Smit delivered five articles and two conference papers on research addressing authentic institutional
identity formation within Appreciative Leadership approach. This forms part of a PhD in Higher
Education Studies. Previous studies include under graduate studies in History of Art and Visual Culture,
a PGC (Arts and Culture Management) and MA in performing arts project management. As
study leader for master’s students specialises in Appreciative Inquiry as method.
Prof Anthony Stacey Associate Professor in decision sciences and research methodology at Wits
Business School and Assistant Dean (Postgraduate Affairs) in Faculty of Commerce, Law and Management,
University of the Witwatersrand, Johannesburg. Having trained as an Electrical Engineer,
his research interests now include techniques for analysis of ordinal level survey data, multivariate
analytical methods, group decision making, group dynamics, and numerical modelling.
Julie Stacey has a background in earth and life-sciences, and has worked operationally and at global
policy level in the mining industry for the past two decades. She is now an independent consultant on
sustainable development, and lectures at the Centre for Sustainability in Mining and Industry (CSMI)
at the University of the Witwatersrand in South Africa.
Bernd O. Stottok is a PhD candidate at Leeds Metropolitan University, UK. He has got almost 20
years of experience as Electrical Engineer in industry. In addition he holds an MSc in Applied Science
and Research from Danube University, Austria. His current research interests are in the railway
telematics area and applications of Grounded Theory.
‘Azizan Supardi is currently a lecturer at the Department of Quantity Surveying, Faculty of Architecture
Planning and Surveying, Universiti Teknologi MARA, Shah Alam, Malaysia. Completed his 2007
MSc in Construction Contract Management, Universiti Teknologi Malaysia, currently he is a candidate
of PhD on Security of Payment. His main interest is Construction Law and Contract Management.
Siew Fun Tang, currently a senior lecturer at Taylor’s University, Malaysia and is pursuing her PhD in
Education Management. She has been in the education industry for 15 years, both in teaching and
administration. She specialises in quality assurance management and policy implementation.
Associate Professor John Venable is Director, Research in the School of Information Systems and
Co-Director of the Centre for Organisational Analysis at Curtin University, Perth, Western Australia.
He has held academic positions at Binghamton University (PhD 1994), Central Connecticut State University,
Aalborg University (Denmark), University of Waikato (New Zealand), and at Murdoch University
(Australia).
Marianne Cecile Viljoen (D. Phil) University of Free State, Bloemfontein, 9301, South Africa. Senior
lecturer researcher at School of Higher Education Studies, University of Free State, South Africa.
Specialises in quantitative research design and teaches and supervises postgraduate students. She
currently serves on panel of Acta Academica, journal of University of Free State. Co-authored several
articles in field of Higher Education.
Kambidima Wotela. Read demography and population studies for all his three degrees and is a demographer
by profession. Occupationally he convenes “policy management, monitoring and evaluation”
and teaches “economic development and population trends”. Supervises research students pursuing
masters and doctoral degrees in public and development management. Skills include ethnography,
quantitative and qualitative research approaches and techniques.
Dr. Ruth Yeung obtained her PhD in Marketing from Cranfield University and is presently working as
an Assistant Professor at Institute for Tourism Studies, Macau. Her research interest lies in the fields
of perception of risk and trust, consumer behaviour, marketing strategies, and management in food
and tourism industry.
xv

Lilia Zerguit Senior Lecturer at Sheffield Business School specializing in Strategic Management. Currently
in year 2 Part-time PhD looking at how strategy develops and forms within UK HEIs. Areas of
interest include strategy-making processes, strategy-making within universities, international students
experience and engagement.
xvi

Polarization in Research Methods Application: Examining
the Examiner
Edwin Asiamah Acheampong, Marcia Mkansi, Kondal Reddy Kondadi and
Baomi Qi
University of Bolton, UK
eea1lis@bolton.ac.uk
mm1lis@bolton.ac.uk
k.r.kandadi@bolton.ac.uk
bqi1@bolton.ac.uk
Abstract: This paper emerges as a reaction to the assertion of polarization or dichotomies in research method.
The aim is to find out whether this assertion by some writers of research methods application, ‘hold water’ within
academia. To that extent, the study designed a survey questionnaire to elicit the opinions of PhD supervisors and
examiners from three North West region of England: University of Bolton, Salford, and Manchester. The study
addresses the issue of polarization in research methods by investigating whether some researchers become
dogmatic in the application of research method through supervision and examination of prospective researchers,
PhD students? If so, what does that posture represents? Can a supervisor-student relationship be marred by
these preferences? This research paper attempts to answer these questions from supervisors’/examiners’
perspective. A purposive and referral sampling methods were adopted to reach supervisors, who otherwise,
would have declined to partake in such an incisive study. The findings of the study confirm some of the
relationships between research philosophies and research approaches, and most importantly reveal the
sentiments of the surveyed population on the apparent rivalry between research philosophies and approaches.
Furthermore, the paper presents the candid observations and opinions of the writers on the
qualitative/quantitative debate. The major limitation of the study is the abysmal response from the surveyed
universities. This will in no doubt detract from the overall impact of the findings of the study.
Keywords: examiner(s), supervisor(s), polarization, qualitative research, quantitative research, research
philosophy
1. Introduction
Research methods, essentially, are to give the researcher the needed mechanism to carry out
effective investigation in his/her field of study. A research method largely defines the design of the
problem under investigation (Saunders et al., 2009; Khotari, 2006; Kumar, 2011). Whatever the
preferred research method, there are three distinct main approaches to addressing any research
problem: Qualitative, Quantitative, and mixed methods (Bryman, 2006). This means that based on the
approach, a research study can be either qualitative, quantitative or a mixture in nature. These
research approaches and their application have their roots in one research philosophy or the other. In
other words, the research philosophy which defines the assumptions constructed about the
phenomena of interest and so predefines the ontological, epistemological and methodological scope
of the study (Guba and Lincoln, 1994; Ritchie and Lewis, 2003). Thus, according to Bryman (1984),
much of the research literature considers, to some extent, the research philosophy to determine, by
and large, which approach the researcher should adopt. It is therefore fairly straight forward to
determine the likely approach of a study from the underlying philosophy.
On the basis of research philosophies, researchers identify themselves as interpretivist, positivist,
critical realist, pragmatist, etc.; with their differences, at the lowest ebb, separating or grouping them
into those who espouse the qualitative methods (Caelli et al., 2003 ; Seale, 1999); those who
predominantly approach research problems quantitatively (Sharf, 1995; Polit and Beck, 2008); and
those who consider a blend of the two approaches as the best option (Johnson and Onwuegbuzie,
2004; Howe, 1988) .
Speaking on the dichotomy between quantitative and qualitative methods in his book, Doing
Qualitative Research, Silverman (2010), notes, “In the context of this book, I view any such
dichotomies or polarities in social sciences as highly dangerous. At best, they are pedagogic devices
for students to obtain first grasp on a difficult field; they help us to learn the jargons. At worst, they are
excuses for not thinking, which assemble groups of researchers into “armed camps”, unwilling to learn
from one another’. The scholar continues, ‘the fact that simple quantitative measures are a feature in
some good qualitative research shows that the whole “quantitative/qualitative” dichotomy is open to
question’.
1

Edwin Asiamah Acheampong et al.
Saunders et al. (2009), point out that there is nothing like pure quantitative study or pure qualitative
study. Again, Sekaran and Bougie (2010), explain that the choice of a quantitative or qualitative
method should be informed by the type of investigation under consideration. Debates have been
ongoing, tackling which method is better than the other. The reason why this remains unresolved until
now is that each has its own strengths and weaknesses which actually vary depending upon the topic
the researcher wants to discuss. (Experiment-resources.com, 2009)
Miles and Huberman (1994) agree that quantitative and qualitative research methods need each other
more often than not. In their book Qualitative Data Analysis, quantitative researcher Fred Kerlinger is
quoted as saying “There’s no such thing as qualitative data. Everything is either 1 or 0”. As if
responding directly to Fred Kerlinger’s assertion, researcher D. T. Campbell, states “all research
ultimately has a qualitative grounding”. Miles and Huberman (1994), note that the
quantitative/qualitative debate is to all intents and purposes fruitless.
There are fundamental differences between quantitative and qualitative methods. Typically, qualitative
data involves words and quantitative data involves numbers. Qualitative research is inductive and
quantitative research is deductive (Saunders et al., 2009). Again, in qualitative research, a hypothesis
is not needed to begin research. However, all quantitative research requires a hypothesis before
research can begin. Does this make one more scientific than the other?
This study surveys the opinions of PhD supervisors and examiners in the Greater Manchester locality
of North West England on the subject and their views on the impact such an inclination could have on
supervisor-student relationships. More specifically, the study will answer the following questions:
Q1. Do supervisors/examiners’ philosophical inclinations make them prefer one research approach to
the other?
Q2. How do supervisors/examiners view the qualitative/quantitative debate?
Q3. How do supervisors/examiners feel about an inclination towards a particular research method?
It is expected that the findings of this incisive study will add to the debate while presenting the opinion
of an important constituency on the issue.
2. Methodology
2.1 Sampling
Three universities, Bolton, Salford and Manchester, were targeted for the study but only supervisors
and examiners from the universities of Bolton and Salford participated. The sampling frame was the
databases of all PhD supervisors and examiners within the three universities. With the exception of a
few survey questionnaires that were administered face-to-face, the bulk of the questionnaires were
mailed to the supervisors and examiners via their university email addresses. In all, over ninety
questionnaires were administered to the three universities. No response came from the University of
Manchester and the remaining two universities produced only ten responses. The response rate was
thus slightly above 11%. A good comprehension of survey practice which guided this study can be
found in (Kelly et al., 2003; Krosnick, 1999; Sanders et al., 2009).
2.2 Participants
All the ten respondents were PhD Supervisors, with three doubling as examiners as well. Two of the
supervisor-cum-examiners were professors from the social sciences and engineering faculties and
the third, a doctor, also from the social science faculty. Thus, many more supervisors participated in
the study than examiners. Majority of respondents have worked as supervisors between one and five
years. Six of the respondents were from Bolton and four from Salford.
2.3 Data collection
A carefully worded survey questionnaire was designed to elicit the views of supervisors and
examiners for the study. There were a couple of demographic variables. Respondents were asked to
indicate their title, gender, number of years as a supervisor/examiner and their faculty.
2

Edwin Asiamah Acheampong et al.
The questions were generally centred on research philosophy and its implication for PhD students’
research projects. Most of the questions were closed-ended. Written answers were to be provided for
the few open-ended questions. The questionnaire took about ten minutes to complete but this was no
incentive to attract satisfactory participation.
2.4 Procedure
Confidentiality of responses and anonymity of respondents were clearly communicated and so
respondents completed the questionnaires based on informed consent. Respondents returned the
completed via emails soon after completion. Three questionnaires were administered in person. No
incentive was given for participation.
2.5 Results
All responses were coded and analyzed with SPSS. Being as study to survey perception and opinion,
only simple frequencies and cross tabulations were run on the data collected.
Frequencies
As noted earlier the response rate was abysmal, with only ten supervisors and examiners choosing to
participate against an expectation of between fifty and seventy respondents. So the results provided
represent the views of a minority in the studied universities. This downside notwithstanding, the
results are worth communicating given the incisive character of the study.
The ten respondents came from the social sciences and engineering faculties. Other faculties
targeted were the Humanities, Health Sciences and the Sciences. Three respondents identified
themselves as primarily interpretivist, positivist or critical realist. The remaining seven saw themselves
as espousing a combination of the research philosophies. All respondents claimed that their choice of
research philosophy (ies) is based wholly on personal reasons. Probing this claim to know when
personal reasons inform a supervisor’s research philosophy would have been interesting as a parallel
study showed that almost all PhD students’ acclaimed philosophical stance is informed by their
research approach. Only a few attribute their philosophical stance to their supervisors’ leadings.
Twenty percent of respondents indicated that they were qualitative researchers. Ten percent, on the
other hand, were inclined to both approaches with each treated as an independent approach based
on the research problem at hand. The rest of the respondents did not have any inclination towards
either of the approaches. Their views corroborate advocates of either of the approaches, whether
seale’s (1999) qualitative, or Polit and Beck’s (2008) quantitative. They would go along with a blend of
the two, as echoed by Johnson and Onwuegbuzie’s (2004). Eighty percent of respondents agree that
an inclination towards a particular research philosophy could lead to polarization of research methods,
where a supervisor/examiner may have strong preference for a particular method to the neglect of
others. In the same vein, majority of respondents believe that a supervisor-student relationship may
be affected negatively if there is an inclination towards a particular research method. Such a
tendency, if negatively pursued, could limit the student’s research experience and is likely to breed
discontent and disagreements.
With respect to students’ research projects, the respondents indicated that supervisors and examiners
attraction to particular research methods could lead to negative consequences if not managed well.
These may include:
Wrong outcomes
Wastage of resources
De-motivation of students
Respondents were unanimous in their call for institutional intervention if supervisors’ and examiners’
inclination towards particular research methods leads to strong disagreements.
Respondents indicated categorically that supervisors and examiners should not at any point impose
their research method preferences on their students. This reflects the awareness of half of
respondents who have witnessed supervisor-student relationships that have turned sour because of
disagreements on the choice of research methods. The weird incidence of a supervisor/examiner
3

Edwin Asiamah Acheampong et al.
insisting on a particular research method(s) without regard to the subject under investigation or the
student’s research objectives has been witnessed by only 10% of respondents.
The assertion by Silverman (2009) that a researchers’ inclination to particular research methods is an
excuse for refusing to learn from one another was put to the test. Eighty percent of the respondents
agreed with the assertion. The remaining two did not respond to the question. Proffering their opinion
on how supervisors and examiners can be convinced to be more tolerant of other research methods
they may not find appealing, the responses included open mindedness of supervisors and examiners,
personal learning and supervisors and examiners becoming active reviewers of academic journals.
This comes alongside the opinion of others who believe convincing supervisors and examiners who
might have such stance would be a difficult task.
The qualitative/quantitative debate is indeed not likely to end in any time in the foreseeable future.
Fifty percent of respondents believe there are pure qualitative or quantitative methods that do not
consider the other in any conceivable way. This opinion comes in sharp contrast to that of the other
fifty percent of respondents who are of the opinion that there is not such study as pure qualitative or
pure quantitative.
A summary of the comments made by respondents with regard to study includes:
Open and continuous discussion on the subject is necessary
The war between philosophies and paradigms in unnecessary
The qualitative/quantitative debate is overplayed
A timely study
2.6 Cross tabulations
With cross tabulations various views are combined to obtain insights that may not be apparent from
the raw frequencies. For instance, all respondents who identified themselves as interpretivists were
from the social science faculty. This finding unconsciously captures the ‘novel gospel’ preached by
Interpretivist advocates including Babbie (2007). Whereas the respondent that claims to be positivist
is from the engineering faculty. This relation endorses the assertion of some researchers that
positivists are inclined to the sciences (Alvesson and Skoldberg, 2009). Four respondents from the
social sciences and one from the engineering faculties indicated that they espoused more than one
research philosophy. It was also insightful that the two respondents who opined that they are inclined
towards qualitative research approach were from the social sciences faculty as this has been the
claim widely (Babbie, 2007). There was no direct relationship between the research philosophies and
the research approaches. All respondent who identified themselves as interpretivists had no special
preference for the qualitative approach. In fact, as note earlier, majority of respondents had no
inclination towards the qualitative or quantitative approach or a blend of the two.
It is notable that the two professors who participated in the survey did not identify themselves with any
particular research philosophy though they belonged to different faculties.
3. Discussions
By and large, the literature on research methods associates the interpretivist philosophy with the
qualitative approach (Babbie, 2007; Seale, 1999). In the same vein, the social sciences including ITbased
programmed are said to be largely interpretive and hence inclined towards the qualitative
approach. Given the major setback of the study, however, this trend is perceptible from the findings of
the study. Table 1 gives a cross-tabulation outlook of the respondents’ faculty and adopted research
philosophy.
Table1: Respondents’ faculty and espoused philosophy
Research Philosophy
Interpretivist Positivist Critical Realist Other Total
Faculty
Social Sciences 3 0 0 4 7
Engineering 0 1 1 1 3
Total 3 1 1 5 10
4

Edwin Asiamah Acheampong et al.
The scientists are generally considered positivists (Alvesson and Skoldberg, 2009), and there may be
a confirmation of this fact by the respondent from engineering faculty who identified himself as such.
Another critical observation is the split-decision on whether there are research methods that are ‘pure’
qualitative or quantitative in nature. The fifty percent of respondents who did not believe anything like
‘pure’ qualitative or quantitative studies included all the respondents who identified themselves as
interpretivist, positivist or critical realist. Only respondents who espouse a blend of philosophies have
come across such ‘pure studies’. None of the literature surveyed for this study underscored such strict
application of approaches in a research study. It would therefore be insightful if further studies will
delve into this research methods and studies that are wholly or strictly qualitative or quantitative in
nature.
From the findings, there is no gainsaying that a supervisor’s/examiner’s inclination towards a
particular research method could have a positive or negative consequence on a student’s research
work. According to views expressed in the study, institutional intervention should be swift if strong
disagreements ensue as a result of such inclinations. But how many students would like to incur the
displeasure of their supervisors or examiners by resisting their ‘determined intrusion’, as far as, their
research methods are concerned? This remains a research question.
The abysmal response rate cannot be glossed over. The writers of this paper would want to believe
the subject area is a slippery turf for some supervisors and examiners. In fact, some supervisors who
did not respond to the survey questionnaire were of the opinion that the questions were ‘strong’.
Others misconstrued the intentions of the writers to mean ‘an attack’ on supervisors and examiners.
However one interprets this hitch in the study, most of the non-respondents cannot be absolved from
a ‘calculated attempt to frustrate the study’. It is against this backdrop that the writers agree with
suggestions from respondents to have an open and continuous discussion on the subject matter.
4. Conclusion
This short study has presented the opinions of supervisors and examiners on a sensitive debate. A
revelation from the three universities studied is that most supervisors and examiners are apathetic to
such a discourse. Perhaps, organisers of conferences such as the European Conference of Research
Methods should facilitate discussions on research methods in universities so that supervisors and
examiners can engage with students on such a ‘potentially distractive’ subject matter.
This study, in no doubt, will contribute its quota to the quantitative/qualitative debate, which stem from
the established philosophies and paradigms. The association between the interpretivist and positivist
philosophies and qualitative/quantitative dichotomy was observable. Again, the relationship between
science and non-science faculties and their inclined research philosophies was revealed by the study.
While some of the respondents referred to the qualitative/quantitative debated (and by extension, their
underlying philosophies) unnecessary and unfruitful, the fact still remains that students would have to
justify their research methods at one time of their studies or the other. Thus, continuous engagement
on the issue cannot be trivialised.
As trumpeted throughout the study, the major limitation of the study is the abysmal response rate;
only ten out of over ninety survey questionnaires administered received feedback from supervisors
and examiners in the studied universities. A more representative response would have projected the
true position of supervisors and examiners on the subject under investigation. The writers were
unable to adopt any corrective measures due to the brevity of time. Nonetheless, the apparent
sensitive nature of the study makes the findings from the study worth disseminating and discussing.
It is recommended that in future studies, measures should be taken to secure the best of responses
from supervisors and examiners so that finding would be largely representative of the universities
studied.
References
Alvesson, M. and Skoldberg, K (2009) Reflexive Methodology: New Vistas for Qualitative Research. Sage.
Babbie, E. (2007) The practice of social research.Thomson Wadsworth.
Bryman, A. (2006) ‘Integrating quantitative and qualitative research: how it is done?’. Qualitative Research, Vol.
6, No. 1, pp. 97-113.
5

Edwin Asiamah Acheampong et al.
Bryman, A. (1984) ‘The debate about Quantitative and Qualitative Research: A question of Method or
Epistemology? The British Journal of Sociology, Vol. 35, No. 1, pp.75-92.
Caelli, K.; Ray, L.; Mill, J. (2003) ‘Clear as Mud: Toward Greater Clarity in Generic Qualitative Research’.
International Journal of Qualitative Methods, Vol. 2, No, 2.
Guba, E. G. and Lincoln, Y. S. (1994) Competing Paradigms in Qualitative Research, Handbook of qualitative
research. Sage.
Howe, K. R. (1988) ‘Against the Quantitative-Qualitative Incompatibility Thesis or Dogmas Die Hard’. Educational
Researcher, Vol. 17, No.8, November, pp. 10-16.
Johnson, B. R. and Onwuegbuzie, A. J. (2004) ‘A Research Paradigm Whose Time Has Come’. Educational
Research, Vol. 33, No. 7, pp. 14-26.
Kelly, K.; Clark, B.; Brown, V.; Sitzia, J. (2003) ‘Good practice in the conduct and reporting of survey research’.
International Journal for Quality in Health Care, Vol. 15, No. 3, pp. 261-266. Oxford University Press.
Khotari, C. R. (2006) Research Methodology: methods and techniques. New Age International (P) Ltd., 2 nd Ed.
Krosnick, Jon, A. (1999) ‘Survey Research’. Annual Reviews Psychology, Vol. 50, pp. 537-67. Annual Reviews.
Kumar, R. (2011) Research Methodology: A step-by-step Guide for Beginners 3 rd Ed. Sage Publications, Ltd.
Martyn Shuttleworth (2008). What is Research?. Retrieved 26 Jan. 2012 from Experiment Resources:
http://www.experiment-resources.com/what-is-research.html [11 January 2012]
Miles and Haberman (1994) Qualitative Data Analysis: An Expanded Sourcebook 2 nd Ed. Thousand Oaks, Sage.
Polit, D. and Beck, C., T. (2008) Nursing Research: Generating and Assessing Evidence for Nursing Practice.
Lippincott Williams and Wilkins.
Ritchie, J. and Lewis, J. (2003) Qualitative research practice: a guide for social science students and
researchers. Sage.
Saunders, M.; Lewis, P.; Thornhill, A. (2009) Research Methods for Business Students. Pearson Education
Limited, 5th Ed.
Scharff, R. C. (1995) Comte After Positivism. Cambridge University Press.
Seale, C. (1999) ‘Quality in Qualitative Research’. Qualitative Inquiry, Vo. 5, No. 4, pp. 465-478 (1999). Sage
Sekaran, U. and Bougie, R. (2010) Research Methods for Business: A skill building approach. Wiley.
Silverman, D. (2010) Doing qualitative research, 3 rd edition. Sage, London.
6

Faculty’s Attitudes Towards Research Supervision and
Postgraduate Students’ Programme Completion at the
University of Ibadan, Nigeria
Emmanuel Olufemi Adeniyi and Maruff Akinwale Oladejo
Federal College of Education (Special) Oyo, Nigeria
maruvoladejo@rocketmail.com
Abstract: The supervision of research students is undoubtedly an integral part of the knowledge management
activities in Universities. It is a complex process that demands positive attitudinal disposition especially on the
part of the supervisor. This is contingent upon the fact that faculty’s positive attitudes in supervisory process and
practice holds the key to the completion rate of the research work because research supervision requires
extensive attention and discussion. This study therefore investigated faculty’s attitudes towards postgraduate
research supervision and programme completion at the University of Ibadan, Nigeria. It adopted the descriptive
survey research, which is ‘ex-post facto’ in nature. This University was purposively selected being the Nigeria’s
premier University, recognized for running quality postgraduate programmes. Two hundred academics, selected
through simple random sampling technique participated in the study. Data were collected through a self-designed
validated instrument (r= .85). Three hypotheses were formulated and tested at 0.05 level of significance. Pearson
product moment correlation coefficient and t-test statistical tools were used to analyze data. The study
established a significant and positive relationship between faculties’ attitudes towards research supervision and
postgraduate students’ programme completion (r=.48). There were also significant differences in faculties’
attitudes toward research supervision based on gender (t=2.36, df= 198, P

Emmanuel Olufemi Adeniyi and Maruff Akinwale Oladejo
Latona & Browne, 2001; Pearson & Brew, 2002; Rodwell & Neumann, 2007; Seagram, Gould, &
Pyke, 1998) include attendance status (part- or full-time), availability of research funding, age, prior
completion of an honours degree, discipline (sciences or humanities), gender, research topic
suitability, the intellectual environment of the department, and access to appropriate equipment and
computers.
The present researchers however, are inclined to include faculty’s attitude towards research
supervision as another important factor to be considered. This is contingent upon the fact that
attitudes play important roles in human endeavour. Thus, the development of positive attitudes
especially on the parts of the faculties is sine qua non to successful programme completion. Shannon
(in Olaoye, 2005), described attitude as a mental state that exerts influence on a person's response to
people, objects and situations. Oladejo (2010) also posited that attitude toward teaching and learning
activity is an important factor in eventual academic success. According to (Okwilagwe, 2002), the
possession of positive attitude is so crucial in a student’s life that various educationists over the years
have been addressing the importance of its development. In fact, the development of right and
positive attitude to academic work is more important than the attainment of high grades (Okwilagwe).
According to (Dinham & Scott, 1999: 45) "the student-supervisor relationship has the potential to be
wonderfully enriching and productive, but it can also be extremely difficult and personally
devastating". We are of the views that negative attitudes of the supervisor towards research
supervision can make such relationship difficult and discouraging, hence, the need for positive
attitudes on the parts of the supervisors.
1.1 The concept of research supervision
Research supervision is a process of fostering and enhancing learning, research and communication
at the highest level (Laske & Zuber-Skerritt, 1996). According to (Johnston, 2000), although, there
have been calls for the conceptualization of research supervision as a teaching/learning process,
there is still a tendency to equate it with research training and with the research responsibilities of the
academic role. Thus, developing research candidates as capable researchers is the primary goal of
research supervision (Down, Martin & Bricknell, 2000).
1.1.1 History of postgraduate studies: Nigerian perspective
Established as a College in 1948 as an affiliate of the University of London, the University of Ibadan,
Nigeria remains the nation’s premier University. The first postgraduate degree of this University was
awarded in 1952 when it was still a University College under the scheme of the special relationship
with the University of London. By 1962, when the University College transformed into the independent
University of Ibadan, there were 64 students registered for higher degrees, some of whom decided to
transfer their registration to the University of Ibadan.
In order to administer the higher degree programme, the Senate of the University of Ibadan created
the Higher Degrees Committee on 28 January 1963, and charged it with the responsibility of
considering recommendations from the Faculty Boards on matters relating to the pursuit of higher
degrees at the university. As time went and the number of higher degrees students increased, it was
found that the cumbersome procedure involved in processing postgraduate matters from Faculty
Boards of Studies through Faculty Boards to the Higher Degrees Committee and, finally, to Senate
was delaying recommendations, which often took as long as a year to get to Senate. As a result of
recommendations from the Faculty Boards and the Higher Degrees Committee, Senate in 1970
established the Board of the Postgraduate Studies to replace the Higher Degrees Committee. This
new Board was given the power to exercise its own initiative in the development of postgraduate work
in the university.
In the 1975/76 session, the Federal Government through the National Universities Commission
designated University of Ibadan a centre of excellence for postgraduate enrolment to cater for the
academic staff requirements of the newly established universities and other national and international
obligations. On June 28, 1976, Senate approved the reorganization of the Board of Postgraduate
Studies, and upgraded it in status to the School of Postgraduate Studies.
The Postgraduate School, as it is presently structured, was approved by Senate (Senate Paper No.
2618 of 29th May 1978). Postgraduate teachers are based in the University departments and their
inputs into the decision-making processes of the Postgraduate School are made through their
8

Emmanuel Olufemi Adeniyi and Maruff Akinwale Oladejo
respective Deans and Sub-Deans (Postgraduate) of Faculties Representatives, and the Heads of
Departments who are the Chief Examiners of departmental postgraduate examinations.
1.2 Faculty’s attitudes towards research supervision
Although, much research has been done to explore the perceptions and attitudes towards research,
(Sterner, 1999; Tang & Chamberlain, 1997) observed that less emphasis is seen on the specific
nature of the problem. Studies in attitudes towards research have been carried out especially in the
Western Universities particularly in the United States (Murrey, Joe, & Robberts, 1994; Sterner, 1999;
Tang & Chamberlain, 1997). For instance, MacKinnon (2003)’s study focused on the perceptions and
attitudes toward teaching and faculty development in research. Murrey et al. (1994) explored the risk
and insurance faculty attitudes towards research and publishing in 36 Universities in the States. One
hundred and thirty two respondents were involved in the study where 48% were Professors, 21%
Assistant Professors, 21% Associate Professors, 6% Instructors and 4% making up others. The study
found out that majority of the respondents was actively involved in research supervising and
publishing.
Jenkins (1995) argued that there should be a reappraisal of the impact of Research Assessment
Exercise (RAE), on UK higher education as it prioritizes research at the expense of teaching. Rowley
(1996) on the other hand, explores strategies, which can be adopted to support the development of a
research supervision ethos in a teaching institution.
It appears this area of academic activity has not been researched extensively in Nigeria. The need to
extend the frontier of knowledge in this very important aspect of faculty’s responsibilities in a
developing nation like Nigeria motivated the present researchers to carry out this study.
1.3 Problem statement
Postgraduate research students often experience difficulties and problems which delay or prevent
them from completing their dissertations or theses as and when due. These problems may be
attributed to many factors among which faculties’ attitudes towards research supervision appears to
be important factor for consideration. This study therefore investigated the relationship between
faculties’ attitudes towards research supervision and postgraduate students’ programme completion
at the University of Ibadan, Nigeria. To this end, the Researchers formulated and tested the under
listed hypotheses to guide the study:
There is no significant relationship between faculty’s attitudes towards research supervision and
post-graduate students’ programme completion at the University of Ibadan, Nigeria.
There is no significant gender difference in faculty’s attitudes towards research supervision at the
University of Ibadan, Nigeria.
There is no significant difference in faculty’s attitudes towards research supervision at the
University of Ibadan, Nigeria based on rank/status.
2. Methodology
This section discusses the adopted methodology vis-à-vis design, sample and sampling techniques,
instrumentation and methods of data analysis. Each of these sub-sections is described as highlighted
below:
2.1 Design
The study adopted the descriptive survey design where the variables are examined “ex-post facto”.
2.2 Sample and sampling techniques
The University of Ibadan, Nigeria’s premier University, noted for over fifty years of running
postgraduate programmes was purposively selected for the study. A total number of 200 lecturers,
who were simple randomly selected, participated in the study. The participants were those who had
been supervising postgraduate students for at least ten years.
9

2.3 Instrumentation
Emmanuel Olufemi Adeniyi and Maruff Akinwale Oladejo
A self-designed instrument captioned ‘Attitudes Towards Postgraduate Research Supervision
Questionnaire (ATPRSQ) was used for data collection by the Researchers. It consisted of two
sections. Section A contained items on respondents’ socio-demographic background such as gender,
years of PG research supervision, rank/status, and marital. Section B consisted of 15 items on
attitudes towards research supervision. The items were drawn on a modified four-point Likert scale of
Most Like Me (MLM), Like Me (LM), Least Like Me (LLM), and Not Like Me (NLM), and carried the
weights of 4,3,2,1 respectively.
2.4 Validation of the instrument
All the items in the instrument were drawn after extensive search in the literature. The instrument was
later given to experts in measurement and evaluation for contents, construct and face validity. This
was to determine the proper structuring, adequacy and contents validity of each of the items in each
of the instruments, and to ensure that the instrument measured what it was actually expected to
measure. Initially, the instrument consisted of twenty items. This was reduced to 10, based on
experts’ comments. The Face and Content Validities were established by careful scrutiny and
modifications of the instrument by experts in this field, thus ensuring that the questions were relevant,
clear, and unambiguous, and adequately covered the prescribed domain for the variables under
study. Thus, the instrument used for this study was validated by experts. Furthermore, a reliability
study was again carried out using 50 academics of the University of Ilorin, Nigeria. A colleague to one
of the Researchers, who is currently on postgraduate study at this University assisted in this regards.
These academics were found appropriate for the purpose of this study because they share similar
characteristics with the subjects of the study. They were not however included in the real study.
Cronbach’s coefficient was computed based on their responses. The alpha value obtained was 0.85,
which makes the instrument reliable.
2.5 Procedure for administration
The researchers employed the services of some postgraduate students who used to be the College
students to one of the Researchers. These Research Assistants had earlier been contacted for the
purpose. They were briefed properly on how they would go about the administration of the
questionnaires. The questionnaires were administered on University academics. The administration of
the questionnaires took us two weeks. Three hundred copies of the questionnaires were sent out to
all the faculties at the University. Two hundred and eighty-seven copies of the questionnaire were
returned. This gave 82.5% rate of return However, 87 out of these questionnaires were not completely
filled. The remaining 200 copies used for the scholarship were found to be appropriately and
completely filled.
2.6 Data analysis techniques
Data were analyzed through the Pearson Product-Moment Correlation Coefficient (PMCC) and t-test
statistical tools respectively. The PMCC was used to test hypothses1, while t-test was used to test
hypotheses 2 and 3 respectively.
3. Results
The three formulated hypotheses were tested in this section. Thus, this section presents results of the
analysed data. It also discusses findings there from.
Hypothesis 1: There is no significant relationship between faculty’s attitudes towards research
supervision and post-graduate students’ programme completion at the University of Ibadan, Nigeria.
Finding
Table 1 shows a significant relationship between faculty’s attitudes towards research supervision and
post-graduate students’ programme completion at the University of Ibadan, Nigeria (r =0.481,df =198,
P

Emmanuel Olufemi Adeniyi and Maruff Akinwale Oladejo
Table 1: PMCC summary table showing significant relationship between faculty’s attitudes towards
research supervision and post-graduate students’ programme completion at the University
of Ibadan, Nigeria
Variables N Mean SD r df Sig
Attitudes Towards Research Supervision 200 36.19 8.20
Programme Completion 200 34.72 8.06 0.481 198 0.00
P

Emmanuel Olufemi Adeniyi and Maruff Akinwale Oladejo
performance/completion is so strong to the extent that the two have reciprocal effect on each other.
Neale (Olaleye, 2003: 165) pointed out that “… attitude and task performance/completion have a
reciprocal effect in their relationship in that attitude affects performance/completion and
performance/completion affects attitude”.
Furthermore, finding from the hypothesis two showed that there was significant difference in the
faculty’s attitudes towards research supervision between male and female academics with male
having better attitudes towards research supervision than their female counterparts. This finding is in
line with the previous findings like those of Cullen, et al. (1994) that established significant difference
in faculty’s attitudes towards research supervision on the basis of gender. Finding from the present
study however disagreed to Abdelhafez’s (2007) study that reported insignificant difference in
faculty’s attitudes towards research supervision on gender basis. According to Abdelhafez,
supervisor’s attitudes were not found to be predicted by gender
The reason for the significant difference in faculty’s attitudes towards research supervision on gender
basis that eventually favoured male might not be unconnected with the fact that male academics are
less occupied, especially after the close of work. Therefore, it is possible that they usually have more
time to study and go over their students’ works than their female counterparts. The male academics
that participated in the study might be more accommodating than the female ones.
Furthermore, finding from hypothesis three which established significant difference in faculty’s
attitudes towards research supervision at the University of Ibadan, Nigeria on the basis of rank (t
=2.43, df =198, P

4.1 Recommendations
Emmanuel Olufemi Adeniyi and Maruff Akinwale Oladejo
Based on the findings and conclusion drew there from, the Researchers hereby recommend that:
Supervisors should ensure a friendly atmosphere necessary for successful research supervision,
and good working relationship with their students to pave the way for in-depth discussions on the
research work.
Female academics should endeavour to exhibit better and positive attitudes towards research
supervision like their male counterparts. They should act like mothers that they are. They should
create conducive and friendly atmosphere between them and their research students.
In addition, senior and experienced supervisors should try to inculcate into the younger ones, the
spirit of friendship, and also educate them on the need for them to display friendly and conducive
supervisory climate.
Also, University administrators should foster conducive knowledge environment and culture, and
also improve upon exiting incentive reward systems that will motivate faculties to have positive
attitudes towards students’ research works.
4.2 Contributions to knowledge
This study has succeeded in contributing knowledge about quality supervision by providing researchbased
facts as regards what supervisors could do to improve the quality and impact of their research
training on their students. It underscores the need for faculty to exhibit positive attitudinal dispositions
towards postgraduate research supervision. Their positive attitudes might inspire their research
students to do better within the stipulated period of time.
Appendix 1
SCHOOL OF EDUCATION
FEDERAL COLLEGE OF EDUCATION (SPECIAL), OYO
FACULTY’S ATTITUDES TOWARDS POST-GRADUATE SUPERVISION QUESTIONNAIRES
Dear Respondent,
All the attached questionnaires are basically meant to elicit information on faculty’s attitudes towards
post-graduate supervision. They are primarily for research purpose. All your responses will therefore,
be treated with utmost confidentiality.
Your maximum cooperation is hereby solicited.
Thanks.
SECTION A FACULTY’S SOCIO-DEMOGRAPHIC BACKGROUND
Instruction: Kindly indicate your response by putting a tick (√) appropriately:
Rank: Gender:
Senior (SL above) Male
Junior (Below SL) Female
Years of PG Supervision:
05-10
11-20
21-30
30+
13

Emmanuel Olufemi Adeniyi and Maruff Akinwale Oladejo
S/N Statements MLM LM LLM NLM
1 I do not think that post-graduate research supervision is tedious, but I
always try to do it well
2 I think about my research student only when they are around.
3 I do not usually absent myself from my research student.
4 I always look at my research student’s work on the advice of my
colleagues.
5 I would like to always supervise post-graduate research works.
6 Post-graduate research supervision is time consuming.
7 I like supervising post-graduate research works to keep myself busy
stay at home.
8 I supervise post-graduate research works to enable me use students
for information gathering.
9
10
I like supervising post-graduate research works so as to improve my
academic experience.
Post-graduate research supervision involves too much reading.
References
THANKS FOR TAKING YOUR TIME TO COMPLETE THIS QUESTIONNAIRE
Abdelhafez, A. M. M. (2007). Postgraduate Students’ Knowledge and Attitudes Towards Good Supervisory
Practice at Exeter University. Exeter, UK: Exeter University.
Acker, S., Hill, T., & Black, E. (1994). Thesis Supervision in the Social Sciences: Managed or Negotiated? Higher
Education, 28(4), 483-98.
Bourke, S., Holbrook, A., Lovat, T., & Farley, P. (2004). ‘Attrition, Completion and Completion Times of PhD
Candidates’. In Proceedings of the Australian Association for Research in Education Annual Conference,
28. Adelaide, South Australia.
Connell, R. (1985). How to Supervise a PhD. Vestes, 2, 38-41
Conrad, L. (1999). Contextualizing Postgraduate Supervision to Promote Quality. In Wisker, G. & Sutcliffe, N.,
Good Practice in Postgraduate Supervision, London: SEDA, 13-24
Cullen, D., Pearson, M., Saha, L. J., & Spear, R. H. (1994). Establishing Effective PhD Supervision. Canberra:
AGPS.
David,D. (nd). A Review of the Literature in Effective Phd Supervision. Centre for Academic Practice and Student
Learning
Delamont, S., Atkinson, P. and Parry, O. (1997). Supervising the PhD: A Guide to Success. Buckingham: The
Society of Research into Higher Education and Open University Press
Down, C. M., Martin, E. and Bricknell, L. (2000) Student Focused Postgraduate Supervision: A Mentoring
Approach to Supervising Postgraduate Students (Version 1). Melbourne: RMIT University
Evans, T. and Pearson, M. (2000) Off-campus Doctoral Research and Study in Australia: Emerging Issues and
Practice. In A. Holbrook & S. Johnston (Eds.), Supervision of Postgraduate Research in Education (pp. 185-
206). Victoria: AARE.
Fawole, I., Egbokhare, F. O., Itiola, O. A. Odejide, A. I. and Olayinka, A. I. (2006). Definition, Spectrum and
Types of Research in A.I. Olayinka, Taiwo, V.O., A. Raji-Oyelade, and I.P. Farai (Eds). Methodology of
Basic and Applied Research, 2 nd Edition. Ibadan. University of Ibadan Postgraduate School.
Fraser, R and Mathews, A (1999). An Evaluation of the Desirable Characteristics of a Supervisor. Australian
Universities' Review, 42(1):5±7.
Gasson, S., and Reyes, L. (2004). Ph.D. ‘Completion within Four Years’. In Proceedings of the Australian
Association for Research in Education Annual Conference, 28. Adelaide, South Australia.
Green, B., and Lee, A. (1995). Theorising Postgraduate Pedagogy. Australian Universities' Review, 38(2), 40-45.
History of the Postgraduate School, University of Ibadan,Nigeria. www.pgschoolui.edu.ng.com
Izah M. T and Nor M .A (2009) An Evaluation of Lecturers Perceptions Towards Research. The Social Sciences.
Vol 4 Issue 5. Pp.416-423
Jenkins, A., (1995). The Research Assessment Exercise, Funding and Teaching Quality. Qual.
AssuranceEduc.,3:4-12.
Johnston, S. (2000). Research Supervision – Setting the Scene. In A. Holbrook & S. Johnston (Eds.),
Supervision of Postgraduate Research in Education (pp. 17-31). Victoria: AARE
Laske, S. and Zuber-Skerritt, O. (1996) Frameworks for Postgraduate Research and Supervision. Lismore:
Southern Cross University Press
Latona, K., and Browne, M. (2001). Factors Associated with Completion of Research Higher Degrees. Canberra,
Australia: Government report in Higher Education Series, 37.Department of Education, Training and Youth
Affairs - Higher Education Division.
MacKinnon, G.E., 2003. Administrator and Deans’ Perceptions toward Faculty Development in Academic P. Am.
J. Pharm. Educ., 67: 1-14.
Murrey, J., Joe H.T. and Robert, L. 1994. Risk and Insurance Faculty Attitudes Toward Research and Publishing.
CPCU J., 47: 162-162.
14

Emmanuel Olufemi Adeniyi and Maruff Akinwale Oladejo
Oladejo, M.A. (2010). A Path-Analytic Study of Socio-Psychological Variables and Distance Learners’ Academic
Performance in Nigerian Universities. Unpublished Ph.D Thesis, University of Ibadan, Ibadan.
Olaleye, O.O. (2003). Some Psychological Determinants of Secondary School Female Students’ Achievement in
Mathematics in Osun and Oyo States, Nigeria. Unpublished Ph.D Thesis, University of Ibadan, Ibadan.
Olaoye, J. A. (2005). Teacher Characteristics and Students’ Attitudes as Determinants of Students’ Performance
in Economics at the Secondary School Level in Oyo State, Nigeria. Unpublished Ph.D. Thesis, ICCE,
Institute of Education, University of Ibadan, Ibadan.
Okwilagwe, E. A. (2002). “Patterns of Undergraduates’ Attitude to Academic Work”. Ibadan Journal of
Educational Studies. 2(2), Pp. 551-562.
Park, C. (2005). New Variant PhD: The Changing Nature of the Doctorate in the UK. Journal of Higher Education
Policy and Management, 27(2), 189-207.
Pearson, M., and Brew, A. (2002). Research Training and Supervision Development. Studies in Higher
Education, 27(2), 135-150.
Rademeyer, G (1994). Thesis supervision: getting the genie out of the lamp. South African Journal of Higher
Education 8(2):92±95.
Rodwell, J., and Neumann, R. (2007). Predictors of Timely Doctoral Student Completions by Type of Attendance:
The Utility Pragmatic Approach. Sydney, Australia: Macquarie Graduate School of Management.
Rowley, J., (1996). Developing constructive tension between teaching and research. Int. J. Educ. Manage., 10: 6-
10.
Seagram, B. C., Gould, J., and Pyke, S. W. (1998). An Investigation of Gender and Other Variables on Time to
Completion of Doctoral Degrees. Research in Higher Education, 39(3), 319-35.
Simpson, R. (1983). How the PhD Came to Britain. A Century of Struggle for Postgraduate Education. Guilford:
Society for Research in Higher Education.
Sterner, A., (1999). Faculty attitudes toward involvement in grant-related activities at a predominantly
undergraduate institution (PUI). SRA J., 31: 5-21.
Tang, T.L.P. and M. Chamberlain, (1997). Attitudes towards research and teaching: Differences between
administrators and faculty members. J. Higher Educ., 68: 212-227.
Van Schalkwyk, O. J (1994). Navorsingsontwerp. Progressio 16(2):35±39.
Zhao, F. (2001). Post-graduate Research Supervision: A Process of Knowledge Management. E-journal:
UltiBASE [Online] Available HTTP: http://ultibase.rmit.edu.au/Articles/may01/zhao1.htm back
15

The Roots and Components of "Workflow” Using Visual
Methodologies
Saleh Alhalalat
King Saud University, Riyadh, Kingdom of Saudi Arabia
salhalalat@ksu.edu.sa
Abstract: This paper studies the term "workflow", together with its roots and components. The experiences of the
author are based on his full-time job as an executive consultant at King Abdullah Institute for Research and
Consulting Studies (KAI) and these experiences have generated best practices and implementations in the field
of Organisational Development (OD). The researcher argues that the term "workflow" should be considered as a
main factor in the business operations and success of OD projects. The main research focus is: "What are the
components of workflow? Improving each component of the workflow umbrella means improving the workflow
itself; equally, improving the workflow itself leads to the improvement of all components. Thus, performance will
be more effective and efficient". The contribution of this paper is that it demonstrates how to understand workflow
components using visual methodologies as these increase opportunities for improving performance which, in
turn, enhance the possibility of a business succeeding in its own market, not just locally but abroad as well. Also,
performance improvement leads to entrepreneurship. The main outcome of this paper is the production of a
comprehensive model of workflow which has been reviewed by several universal experts. This paper is divided
into three parts to offer a clear interpretation of the argument. After an introduction, a workflow portfolio is
presented; this is followed by a visualisation of workflow components. Finally, conclusions and recommendations
are provided.
Keywords: performance, process, project management, visualisation, workflow
1. Introduction
There was no business case for the projects. The term "work" is complex; it has the following meaning
according to the Oxford Dictionary (2012): "Activity involving mental or physical effort done in order to
achieve a result”. Workflow is central to the life of any organisation and so each organisation must
manage its work efficiently or it will face problems with both internal customers (employees) and
external ones (i.e. the company’s customers). Any change in the organisation should be controlled to
achieve a balance of satisfaction for both sets of customer (i.e. internal and external); this is
challenging.
Throughout history, markets have always re-created themselves, shifting the economic fortunes of
those present at a market’s creation (Fingar et al., 2002). To be competitive, companies need to
collaborate seamlessly, both internally and with other companies. An opportunity to be competitive
can be found by improving workflow and increasing its efficiency, so a useful decision in this regard
may be to adopt a workflow system. In general, there are certain key benefits to improving workflow:
improving efficiency by automating many business processes; achieving better process control;
improving customer service; obtaining flexible software control over processes in order to enable them
to be re-designed in line with changing business needs; and improving business processes (eworkflow,
2002).
The research problem here deals mainly with not having a comprehensive business workflow model
and, as a result of this, there are problems in workflow projects. So, improving an understanding of
the roots and components of workflow using different visualisation methodologies will support
workflow projects. This study assumes that the main method for being competitive lies in the
workflow, particularly in the distribution of the right work to the right person at the right time.
Consequently, this study seeks to develop a convenient visualised model for achieving efficient
workflow in the business domain. To find a solution that can be adopted in order to improve this style
of working and also develop the infrastructure of a business is a researchable issue. So, the main
question in this research is: What workflow components should be considered in workflow projects?
An inductive research approach was used in this study to move from observations to theories in
sequential phases: from observation, pattern-identification, tentative hypothesis and theory
respectively. As such, it makes a contribution to both the scientific and behavioural schools.
After reviewing a significant number of articles related to workflow, the overall observations were as
follows:
16

Saleh Alhalalat
Knowledge about workflow definitions was lacking in the business rather than the technical side.
There was a failure to provide comprehensive solutions. These tended to focus on partial
solutions rather than on comprehensive ones.
Business owners could not define the future state of their workflow.
There were varied problems in the implementation of workflow solutions. Business domains were
not effectively understood.
Furthermore, several visualisation tools were used including: Spider Diagrams, Swimlane Modelling,
Data Flow Diagrams (DFD), and Flow Charts. These were used to show a proposed workflow model.
The study derives its importance from two significant points: firstly, it has theoretical significance,
deriving from the workflow field itself; it is a conceptual-based paper. Secondly, its practical
significance relates to the provision of a visualised model of workflow. This dual significance is
highlighted in the conclusions and recommendations presented at the end of this study. To achieve
the above aims, the remaining research is divided as follows: workflow portfolio, visualisation of
workflow components, and conclusions and recommendations.
2. Workflow portfolio
This part presents the historical roots, components, workflow management systems (WFMS), and
standards for workflow.
2.1 Historical roots
Frederick Tayler, who is often called the “father of scientific management”, introduced the principle of
“Scientific Management” in the 1920s. He viewed the work of an organisation as a collection of tasks,
each of which could be described by a precise procedure, and whose steps were particular motions
and activities of the employees within the organisation. The job of management is to identify and
optimise the tasks and to supervise the employees performing the tasks (Fischer, 1995, 60).
Conversely, the “Behavioural Management” school seeks to provide employees with moral and social
benefits. In the 1940s, Herbert Simone was the principal actor in this school. These two views still
prevail and have resulted in two arguments regarding the use of workflow in organisations. The first
group argues that workflow assists the automation of tasks (supporting scientific management),
whereas the second argues against automating the tasks (supporting behavioural management). As
behaviourists focus on workflow projects failing, a question must be posed: Will workflow automation
take us back to the scientific management paradigm of the 1920s? However, the historical roots of the
field of workflow management are considered a controversial issue; some researchers indicate that
this field emerged in the 1970s (Aalst and Hee, 2002, 27) while others say that organisational
workflow was developed as early as 1961 (Muehlen and Zhao, 2002).
2.2 Workflow components
The definition of the term “workflow” must be offered before analysing its components. Some sources
have defined workflow as follows:
The Workflow Management Coalition (WfMC) defined workflow as “the computerized facilitation
or automation of a business process, in whole or part” (Hollingsworth, 1995).
Workflow was defined by the Workflow Management Coalition as “The automation of a business
process, in whole or part, during which documents, information or tasks are passed from one
participant to another for action according to a set of procedural rules” (Lawrence, 1997).
Kamath and Ramaritham (2002) defined workflow as “a technique to integrate and automate the
execution of steps that comprise a complex process”.
The Workflow Management Coalition defined workflow management as a “system that defines,
manages, and executes workflows through the execution of software whose order of execution is
driven by a computer representation of the workflow logic'' (Bea, 2000; Hollingsworth, 1995).
It can be said that the term “workflow” is concerned with information, data, processes, tasks,
procedures, and document management. The movement of information and material is a
consequence of work and supports workflow; however, it is not the work itself (Fischer, 1995, 61).
17

Saleh Alhalalat
From the practical point of view of the researcher, the term “workflow” consists of four main
components. These are as follows:
Component 1 (Task): This can be defined as the “core process activities of a workflow; tasks evolve
through various task events as the workflow progresses (created, executed, activated, done)” (Bea,
2000). It is the organisational goal(s), department goal(s), and the core of the job description for each
employee. "A workflow schema is defined using a set of tasks that are coordinated using
dependencies; workflows generated from the same schema may differ with respect to the tasks
executed" (Parvathi, 2011).
Component 2 (Process): Goleman et al. suggested that process is a “structured and managed set of
work activities designed to produce a particular output” (Goleman et al., 2002, 1314). While
organisations are built on the basis of vertically functioning groups, the flow of most processes is
horizontal (Maddison, Darnton, 1996, 139). Why is process necessary? It is easy to measure and is
more measurable than other components; furthermore, the increase in customer-oriented
developments supports the impact of internal and external processes. The voice of process can be
measured using the Sig Sigma method which is used to eliminate defects to the level of 3.4 per
million. Jae-Yoon Yang, Hoontae Kim, and Suk-Ho Kang (2006) developed a prototype workflow
system that electronically aids the progress of B2B processes. This means that workflow systems
support the running of processes.
Component 3 (Information): Information is at the core of the knowledge hierarchy, which consists
accumulatively of signals, data, information, knowledge, wisdom and facts. Information is the
knowledge core. Data are defined by Reitz (2004) as: “Facts, figures, or instructions presented in a
form that can be comprehended, interpreted, and communicated by a human being or processed by a
computer”. Also, data are the raw information collected without processing, whereas information is the
data processed to be meaningful (BPP Publisher Ltd, 1988, 124). Data lead to information, which
leads to knowledge, which leads to wisdom that is worth the truth itself (Jashpara, 2004, 14-17].
Regarding information, Bocij et al. (2005, 5) stated that information is data that have been processed.
Also, Orna (2004, 7) described information as the “food of knowledge”. Knowledge is the
accumulative understanding of signals, data and information and greater understanding of such
knowledge sources means the creation of a more skilled and experienced generation, which leads to
wisdom and the emergence of fact. It can be said that information improvement leads to work
improvement; this, in turn, leads to organisational learning that reaches the level of entrepreneurship.
If the knowledge hierarchy is managed effectively, the organisation will achieve a high level of
learning. Both Alhalalat (2005), and Kruger and Johnson (2010) confirmed that information
management is an enabler of mature knowledge management. In fact, information is a core
component. Furthermore, Stohr and Zhao (2002), and WfMC argue about using information instead of
workflow. As stated before, information management is part of work management. Nowadays,
transparency and accountability in modern organisations have resulted in a high demand for workflow
which comprises the flow of tasks, information, documents and processes. According to Bocij et al.
(2003, 244), the operational definition of workflow is based on information rather than documents and
tasks. Both documents and tasks are efficient while information is already efficient. Moreover, Zhen et
al. (2009) confirmed the importance of information content with regard to a valuable workflow.
Component 4 (Document): Documents are the space where information is stored while document
management is the technique used to control the increasingly growing collection of electronic or nonstructured
information; specialised Information Technology (IT) tools are available to help in this.
Associated with workflow is document management. Modern electronic document management
systems (EDMS) store all documents, classified according to set criteria, in a repository (Ballard,
2001). However, documents in general are of two types: electronic and paper.
It is important to construct knowledge of work (tasks) in any organisation because this knowledge
contributes significantly to decreasing cost and maximising revenue. “Between 10 and 15 per cent of
a company’s revenue is spent on finding and managing documentary information, so having the right
knowledge at the right time clearly has an impact on operational efficiency” (Emerald, Canon
Business Solutions, 2002). It is really important to implement a new technology in order to manage
the processes. Workflow ensures tasks are performed swiftly and that the steps of a process are clear
to everyone, ensuring the fulfillment of the process in the time specified while keeping everyone
18

Saleh Alhalalat
informed of the sequences, providing statistical data, and finalising the task by removing it from the
task list when it is done (Workflowing, 2003). When a work item is generated, the workflow will not
continue until the workflow item has been processed (Workflowing, 2001).
2.3 Workflow management systems (WMS)
Workflow management systems are a new generation of technology designed to automate business
processes by coordinating and controlling the flow of work and information between participants
(Stohr and Zhao, 2001). Workflow systems fall into four categories according to their aims. These are:
transaction or production workflow, ad hoc workflow, administrative workflow, and collaborative
workflow (Khoshafian and Buckiewicz, 1995, 226-233). Such systems enhance many aspects in the
structure of modern administration through communication, collaboration, coordination, knowledge
and productivity (Sarmento and Machado, 2000).
Automation does not mean the work doing itself; it means instead to manage and control the flow of
work as planned before. Workflow automation is a major application in industry because of the urgent
need to reduce process costs, speed up cycle times, and provide flexible services. However, the
automation of business processes is a technical, informational and managerial issue. Given the
emerging importance of this set of technologies, it is important for more information systems
researchers to become involved in the study of workflow automation (Stohr and Zhao, 2002).
Workflow software products automate the management of processes (DCE, 2003). They are also
used, not only for coordinating office tasks, but also for managing inter-organisational information
flows. This is a new style of business process engineering for both intra- and inter-organisational
processes (Stohr and Zhao, 2001). Furthermore, Hajo, Reijers and Wil M.P. van der Aalst (2005) said
WFMSs are widely used and are reputable in improving organisational performance. Today, workflow
systems have an overlapping functionality that leads to a taxonomy. This helps in providing end users
with mechanisms to choose a suitable workflow system for their application (Ewa Deelman, 2009). To
sum up, a WMS provides different types of benefit for both internal and external stakeholders of any
organisation that uses it. Internal stakeholders include those at a management level (employees)
while external stakeholders include customers, shareholders, agents and suppliers.
2.4 Workflow standards
The efficiency of workflow means that the flows of work (its components) should be standard and
timely; should satisfy goals; use appropriate channels, trusted recipients and trusted sources; and be
available. Who has the responsibility for providing workflow standardisations? “The Workflow
Management Coalition is now globally recognised for the advancement of workflow management
technology and its use in industry” (Jon Pyke, WfMC chair, WfMC, 2002). Organisations that invest in
workflow software want to be sure that their investment is going to be protected. With standards,
users can make sure that essential criteria will be met, hence reducing the risk involved. This
obviously becomes of great importance when workflow systems are required to interoperate with
those of other organisations whenever business processes are implemented across organisational
boundaries (WFMC, 2003).
3. Visualisation of workflow components
This section presents a revision of the research methods, conceptual models of workflow
components, and workflow projects.
3.1 Revision to the research methods
What is the role of research methods here? Using suitable research methods provide a clear
understanding of the business domain. That is to say:
The term "research methodologies" plays a “vital role in knowledge development of international
business" (Zhilin Yang, Xuehua Wang and Chenting Su, 2006).
The research approach consists of two types: deductive and inductive.
In terms of research strategies, scientific strategies are more suitable for workflow studies than
interpretivist strategies. To be more direct, scientific strategies include laboratory experiments,
field experiments, surveys, case studies, forecasting (future research), grounded theories,
simulations, and game/ role playing. "Research conducted in lab settings is often associated with
an experimental approach while research conducted in the field is associated with a correlational
19

Saleh Alhalalat
approach" (Allen, Eby, O’Brien and Lentz, 2008) whereas interpretivist strategies include:
subjective and argumentative strategies (phenomenology, hermeneutics), and action research.
There are two types of time horizon: cross-sectional and longitudinal studies.
Two types of sampling exist: probability sampling and non-probability sampling. Lee (2010) noted:
“The theory behind sampling is that: if a sample selected is as close as possible to being
representative of a population, then any observations made regarding that sample should also
hold true for the population. With probability sampling, the chance that the sample will truly
represent the population is high. Hence, the researchers aim to be as wide and as random as
possible in coverage".
Data are actually collected from two main sources: primary sources (reports, theses, emails,
conference reports, company reports, some government publications, and unpublished
manuscript sources), and secondary sources (documentary sources, and surveys).
To have a clear understanding of the business domain, Soft System Methodology (SSM), devised
by Peter Checkland, can be used to gain greater insight into the business perspective rather than
the technical perspective of workflow. Hajo, Reijers and Wil M.P. van der Aalst (2005) reported
results from a longitudinal, multi-case study into the effectiveness of workflow management
technology. The results of the study showed significant improvement in the following process
parameters: lead time, service time, waste time, and resource utilisation.
Visualisation is used to show the interaction between different variables. For example, Moreno,
Bischof and Hoover (2012) presented an overview of an interactive visualisation of dependencies.
They confirmed that, in workflow modeling, some decisions may require the approval of one or
more executives in a certain order, followed by approval from an oversight committee. This
means that the visualisation of a scenario is an important phase in understanding, improving and
managing the processes.
3.2 Workflow conceptual models
According to the literature on "System Theory", "Efficient Workflow" includes inputs, processing,
outputs, and feedback, as follows:
Figure 1: Efficient workflow model
To manage any workflow domain, each sub-architecture needs to be understood. They are four types,
as follows:
Tasks Architecture: This is actually divided into three layers: organisational architecture (main
goals), department level (department goals), and individual level (tasks on the job description
form).
Information Architecture: This depends on the tasks’ architecture.
Documents Architecture: This depends on the information architecture.
Processes Architecture: This consists of three levels: top processes, core processes, and support
processes.
To be clearer, a conceptual model of workflow components is offered below as follows:
20

Saleh Alhalalat
Figure 2: Conceptual model of workflow components
In a mathematical, model-driven approach, the generic model can be presented as follows:
y = a + b1x1 + b2x2.... + bnxn
Workflow studies deal with phenomena which are multi-causational: that is, there is more than one
factor (Xs) affecting one dependent factor (Y). Thus, multiple regression analysis can be used instead
of simple regression. The equation above refers to the impact of Xs (task, information, process and
document) on the efficiency of the workflow (Y). To have valuable and efficient workflow, Xs should be
managed and defined as follows:
X1: Task and its sub-components (organisational, departmental and individual tasks).
X2: Process and its sub-components (top management, core and support processes).
X3: Information and its sub-components (signal, data, information, knowledge, wisdom, and fact).
X4: Document and its sub-components (paper-based and electronic documents).
Workflows in an organisation can be modelled like the flow of blood in a biological neuron in the
human body; they seem to flow from a cell body to the intended location through axons. In fact, this is
a scenario or protocol between all the workflow stakeholders. Studying workflow is not sufficient; it
needs more visualisation to increase understanding of that scenario. According to Sharp and
McDermott (2001, 144), the scenario of work comprises the relationship between actors and their
roles, like the relationship between nouns and verbs. Yeo (2002) confirmed that listing "weak
business processes" is surely a critical failure factor for information system projects. However, the
following is an example of a workflow scenario:
Sara is looking for a suitable coat (medium-size, red, 10% cashmere, etc.). She has tried out
many online shops and the one that has most interested her is called www.PetraSalon.com
(for example). She navigates into its portal and chooses the most suitable coat for her. She
pays for it using her MasterCard, which is verified by a third party (Bank) which then sends the
money to the Financial Department of the PetraSalon company. A message is sent to her
email box saying "Your coat will be delivered to your address by the Shipment Department
within three days and you can follow up your order by clicking on the following URL:
www.PetraSalon.com/followup”. However, she receives it before the three days are up, so it is
within the target, and she signs the receipt form. According to the Customer Care survey, she
was happy after trying it on.
Figure 3: Example of a workflow scenario
21

Saleh Alhalalat
It is clear that optimising the connection of tasks with stakeholders (i.e. verbs with nouns) in Swimlane
Modelling, increases the value of information and improves business performance. The above
scenario contains verbs (in blue) and nouns (in green). A Spider Diagram can be used to visualise
these, as follows:
Spider Diagram of Nouns Spider Diagram of Verbs
Figure 4: Spider diagram of nouns and verbs
To visualize them together; nouns (stakeholders or actors) are putting on the Y axis. Whereas, verbs
are put on the X axis, because they are based on the time line as follows.
Figure 5: Swimlane diagram of workflow
The figure above shows a Swimlane diagram of workflow scenarios.
3.3 Workflow project management
This section presents a discussion of three stages: pre-stage, workflow project stage, and post stage.
Pre-Stage: In adopting a workflow program, many needs should be satisfied. These include: poor
document control, poor tracking of progress, loss of an unacceptable number of tasks, disputes over
responsibilities, and uncertainty and inconsistency in the processing of similar tasks (dcs, 2003). This
means the different problems that are found in the organisation. The following table illustrates them:
Nowadays, the process-driven approach is the most commonly known approach but it is only a partial
solution, not a comprehensive one. It is the lowest in cost but it does not offer a complete solution.
The workflow-driven approach is costly but it is more comprehensive. It takes a long time to
22

Saleh Alhalalat
implement this approach. It can be summarised that the best approach to use is the workflow-driven
approach.
Table 1: Matrix of workflow components and solutions
Problem Change Solution (Approach) Success Level Examples of Projects
Tasks related Tasks change Tasks-driven App. Partial solution Job Analysis Project
Information related
Information
change
Information-driven
Approach
Partial solution
Knowledge
Management Project
Documents related
Documents Documents-driven
Partial solution
Content Management
Processes related
Problems of tasks,
information,
processes, and
documents
change
Processes
change
Workflow
change
Approach
Processes-driven
Approach
Workflow-driven
Approach
Partial solution
Comprehensive
solution
Project
Processes
Reengineering Project
Workflow Project
Project Stage: According to Project Management Institute (PMI) standards, projects have the following
phases: Initiation, Planning, Executing, Monitoring and Controlling, and Closing. A workflow
management system must provide its support in three broad areas. These are: workflow definition and
execution, workflow monitoring, and data administration (Bea, 2000). Also, Berdebes (2001)
described workflow projects in the following steps:
Analyse business needs and identify the right solution or solutions for achieving the goals that
have been set.
Define the objectives and deliverables of the project.
Plan the organisation’s implementation model and the enterprise software to fit the business
processes.
Integrate the Enterprise Software (ES) into existing processes and systems. Look at the total
picture.
Deploy the solution across the organisation.
Train users and teams to achieve user acceptance.
Support the software to ensure continued success.
Measure the success of the initiative at specific intervals.
Post Stage: This is the level of deliverables gained. Generally, the expected deliverables of workflow
projects are as follows:
The right work,
At the right time,
From the right source (worker),
To the right source (recipient),
Through the right channel.
The success of a workflow system stands or falls on the quality of the workflows put into it (Aalst and
Hee, 2002, 31). There are a great many criteria regarding the critical efficiency of workflow and some
of these, which are used to evaluate the workflow project, are: efficiency of workflow, monitoring and
controlling the workflow system, and the benefits of the implemented workflow system.
Unfortunately, regarding a cost-benefit analysis of the workflow project, many workflow projects are
unsuccessful. In a related area, Grove et al. (1995) estimated that 50% of business process
reengineering projects fail. The most important reasons for failure include poor change management,
resistance from rigid bureaucracy, and lack of sustained top management support (Stohr and Zhao,
2001).
Lauras, Guilaume and Gourc (2010) confirmed the difficulty of controlling complex projects; this
means more risks are taken to complete them. However, they provided a multi-dimensional project
performance measurement system which comprised effectiveness, efficiency, and the relevance of
23

Saleh Alhalalat
four Key Performance Indicators (KPIs): delay, cost, quality and risks. The starting point for reaching
these KPIs is to improve the project work itself (i.e. the tasks, information, processes and documents).
In fact, workflow technology has its limitations as well as its benefits (Zhao, 2002). On the other hand,
successful WfMS deployment results in significant reductions in the time cycle of processes, in cost
reductions, improved accuracy, greater control, and greater worker satisfaction (Stohr and Zhao,
2001).
To sum up: using the proposed workflow model (of tasks, processes, information and documents), as
discussed in depth in this study, surely decreases the failure percentage of workflow projects which is
an important issue for project management.
4. Conclusions and recommendations
From reviewing previous literature related to the roots and components of workflow, efforts here were
directed to technical aspects of workflow rather than business aspects. In fact, the technical side
focuses on hard solutions (e.g. software applications) whereas the business side focuses on soft
solutions (e.g. workflow KPIs, the success of workflow projects). Consequently, there is a real need to
focus on business aspects in the future. One possible application of this study might be to develop a
new reference model for business workflow applications. As a result of the study’s analysis, the main
components of workflow were found to be: tasks, processes, information and documents and, in order
to achieve a successful workflow project, these must be considered or efficient workflow will not result
and the project itself may not deliver certain aspects. Some visualised models of workflow were
reviewed by certain universal experts in the Business Process and Information Management fields.
The study recommends considering business workflow as defined in this study: that is, as tasks,
processes, information and documents. It also recommends paying attention to the business
perspective (soft solutions) rather than a technical one (hard solutions). Regarding limitations, the
main limitation of this study lies in its theoretical scope; more experimental and data collection
methods are needed to develop the reference model. On the subject of a future perspective, the
model proposed in this study should be developed into a "Business Reference Model" of workflow.
Acknowledgments
The author is most grateful to the three reviewers whose remarks were most valuable; their
constructive and useful comments helped to improve the proposed models, making them more
applicable for business workflow use. Thanks to them for their contribution. Also, special thanks must
go to Dr. Basheer Eisawy for his valuable help with editing.
References
Aalst W., Hee K. (2002), Workflow Management: Models, Methods, and Systems, Massachusetts Institute of
technology, USA.
Alhalalat S. (2005), Information Flow in Virtual Organizations, Unpublished PhD Thesis, Loughborough
University, UK, 2005.
Ballard C. (2001), Improving Workflow Efficiency, < http://www.itweb.co.za/office/didata/0111070821.htm >,
[accessed 16.5.2003].
Bea, (2000), Weblogic Process Integrator Release 1.1, ,
[accessed 6.2.2003].
Berdebes G. (2001), Enterprise Software Implementation:
The process for successful deployment, , [accessed 17.5.2003]
Bocij P., D. Chaffey, A. Greasley, S. Hickie (2005), Business Information Systems, 2 nd edn., Prentice Hall,
England, 2003.
BPP Published Ltd (1988), Nature of Management, 3 rd ed., BPP Published Ltd, London.
Brian L. (2010), "Exploring a New Research Method in Diversity Research", International Conference on Learner
Diversity, Procedia-Social and Behavioral Sciences, Vol. 7, pp 494-503.
Camilo Arango Moreno, Walter F. Bischof, and H. James Hoover (2012), "Interactive visualization of
dependencies", Computers & Education,, Vol. 58, pp 1296-1307.
DCE, (2003), Workflow Management,
, [accessed 22.4.2003].
Emerald, ( 2002), "Smoothing the workflow process", Work Study, Vol.51, No., p1.
Ewa Deelman, Dennis Gannon, Matthew Shields, and Ian Taylor (2009), "Workflow and e-Science: An overview
of workflow system features and capabilities", Future Generation Computer systems, Vol. 25, pp 528-540.
e-workflow, (2002), e-workflow- the workflow portal, < http://www.e-workflow.org/ >, [accessed 25.10.2002].
Fingar, at el (2002), "21 st Century Markets: From Places to Spaces", First Monday Journal [online],
,[ 25.10.02].
24

Saleh Alhalalat
Fischer L. (1995), The Workflow Paradigm, 2 nd ed., Future Strategies Inc., USA.
Goleman, et al 2002, Business: the Ultimate Resource TM , 1 st ed., Bloomsbury Publishing Plc., UK .
Grove V, Jeong SR, Kettinger WJ, Teng TC (1995), "The implementation of business process engineering",
Journal of MIS 1995;12(1):109–144.
Hajo A. Reijers and Wil M.P. van der Aalst (2005), "The effectiveness of workflow management systems:
Predictions and lesson learned", International Journal of Information Management, Vol. 25, Issue 5, pp 458-
472.
Hollingsworth D. (1995), Workflow Management Coalition: The Workflow Reference Model, < www.wfmc.org >,
[accessed 12.3.2003].
Jae-Yoon Jung, Hoontae Kim, and Suk-Ho Kang, (2006), "Standards-based approaches to B2B workflow
integration", Computers & Industrial Engineering, Vol. 51, Issue 2, pp 321-334.
Jan Pyke, WFMC Chair (2002), Workflow Management Coalition, .
Jashpara A. (2004), Knolwege Management: An Integrated Approach, Pearson Education, UK, 2004.
Kamath A., Ramamritham K. (2002), "Correctness issues in workflow management", Distributed Systems
Engineering, Vol. 3, No. 4, PP. 213--221.
Khoshafian S. and Buckiewicz M. (1995), Introduction to Groupware, Workflow, and Workgroup Computing, 1 st
ed., John Wiley& Sons, Inc, Canada
Kruger C.J. and Johnson R. (2010), "Information Management as an Enabler of Knowledge Management
Maturity: A South African Perspective", International Journal of Information Management, 30, pp 57-67,
2010.
Lauras M., Guilaume M., and Gourc D. (2010), "Towards a multi-dimensional project Performance Measurement
System", Decision Support Systems, 48, pp 342–353.
Lawrence P. (1997), Workflow Handbook 1997, John Wiley& Sons Ltd, UK.
Maddison R., Darnton G. (1996), Information Systems in Organisations: Improving Business Processes, 1 st ed.,
Chapman& Hall, UK.
Muehlen M., Zhao J. (2002), "Tutorial Workflow and Process Automation in the Age of E-Business", Proceeding
of the Thirty-First Annual Hawaii International Conference on System Sciences, USA.
Orna E. (2004), Information Strategy in Practice, Gower Publishing Limited, England, 2004.
Oxford Dictionary (2012), Work Definition, , [accessed
16.12.2011],
Parvathi Chundi, Tai Xin, and Indrakshi Ray, (2011), "When does a workflow complete?", Computer Standards &
Industries, Vol. 33, Issue 3, pp 292-300.
Reitz J. (2004), Online Dictionary for Library and information Science, [e-book], Published by Libraries Unlimited,
, [accessed 16.1.2005], 2004.
Sarmento A., MachadoA. (2000), "Impact evaluation of organisational changes enabled by workflow systems", 6 th
International Workshop on Groupware (CRIWG00), Portugal.
Sharp A., McDermott P. (2001), Workflow Modeling: Tools for Improvement and Application Development, 1 st
ed., Artech House Publishers.
Stohr E., Zhao J. (2002), "Workflow and process automation in the age of E-business– technical, Organizational
and Educational", HICSS-35 Tutorials, Advanced Seminars, and Workshops.
Tammy D. Allen, Lillian T. Eby, Kimberly E.O's Brien, Elizabeth Lents (2008), "The state of mentoring research: A
qualitative review of current research methods and future research implications", Journal of Vocational
Behavior, Volume 73, Issue 3, December 2008, Pages 343-357.
WFMC (2003), Workflow Management Coalition Standards, .
Workflowing, 2001, FAQ Workflow Techniques, , [accessed 17.5.2003].
Workflowing (2003), < www.workflowing.com/id19.htm >, [accessed 17.5.2003].
Yeo K.T. (2002), "Critical Failure Factors in Information System Projects", International Journal of Project
Management, Vol. 20, pp 241-246.
Zhen L., G. Huange, and Z. Jiang, (2009), "Recommender system based on workflow", Decision Support
Systems, 48, pp 237–245, 2009.
Zhilin Y., Xuehua W. and Chenting S. (2006), "A Review of Research Methodologies in International Business",
International Business Review, Vol. 15, No. 6, pp. 601-617.
25

Evaluating Doctoral Education: A Conceptual Paper
Valerie Anderson and Sarah Gilmore
University of Portsmouth, Richmond Building, Portsmouth, UK
Valerie.anderson@port.ac.uk
Sarah.gilmore@port.ac.uk
Abstract: This paper focuses on the evaluation of doctoral education in UK Higher Education Institutions (HEIs).
It critically reviews approaches to the evaluation of doctoral education and proposes a revised framework.
Pressures of budgetary constraints and ‘marketisation’ in Higher Education (HE) have promoted managerialist
approaches to evaluation that prioritise the needs of students as consumers and government funding agencies
as ‘clients’. Such approaches are epitomised by the Research Councils UK (RCUK) ‘Researcher Development
Framework’ which articulates doctoral development through a set of lists of behaviours, standards and
descriptors which overlook intangible features of intellectual development and research expertise. We critique the
rational / objectivist paradigm of the current ‘Impact Framework’ (IF) approach to evaluating doctoral education
highlighting its linear, operational-level, and summative purpose. We argue that the environment for research and
knowledge exchange in HE requires a constructivist, pluricentric understanding of the processes of interaction
between students and other stakeholders which collectively influence doctoral education outcomes. We propose
an inclusive, multidimensional framework derived from the literature about different forms of evaluation which
provides a balanced assessment of the perspectives of: learners, educators; supervisors; principal investigators,
deans, senior managers, research sponsors and funders. Our proposed framework utilises longitudinal,
aggregated quantitative data and qualitative information. First, it includes effectiveness and efficiency measures
to evaluate the student experience and achievement of learning outcomes. Second, it encourages participants in
doctoral learning processes, such as students, supervisors and leaders of faculties; schools or research groups
to articulate their expectations of doctoral education processes and the extent to which these expectations have
been met. Finally, it assesses achievements resulting from doctoral education against HEI strategic aspiration
measures.
Keywords: doctoral education; evaluation; higher education; measurement
1. Introduction
This paper focuses on the evaluation of doctoral education in UK HEIs. Estimates suggest that
research activities account for one-quarter of HEI revenue in UK (UUK, 2005) and the movement of
trained researchers into the labour market is important for the development of innovative responses to
global competitive challenges (RCUK, 2006). Most HE policy attention addresses the student
experience of those undertaking taught courses although the requirement for effective researcher
development processes has been increasingly articulated by the Research Councils, employer
representatives and the UK Quality Assurance Agency for Higher Education (RCUK, 2008; QAA,
2011).
From 2001 to 2009 doctoral education and development processes focused on the achievement of a
RCUK endorsed Joint Skills Statement (Vitae, 2001). This was replaced in 2010 by a more explicit
and detailed Researcher Development Framework (RDF) (Vitae, 2011) comprising extensive lists of
behaviours and standards that research students are expected to achieve. Alongside the
development of the RDF an ’Impact Framework’ (IF) for the evaluation of doctoral development has
been developed (Rugby Team, 2008). Derived from the work of Kirkpatrick and Kirkpatrick (2006) and
Kearns (2005a) this approach categorises five hierarchical points of impact: infrastructure investment;
student reaction; learning results; learner behaviour; and final outcomes.
The argument of this paper is that the current IF approach represents the dominance of summatively
focused, externally orientated and hierarchical approaches to learning, teaching and education which
are inappropriate to the circumstances of research and knowledge exchange in a context of rapid
change in Higher Education.
The aims of the paper are: first, to critically review current approaches to monitoring and evaluation of
doctoral education and second, to propose a revised framework.
2. Measurement and evaluation in doctoral education
The terms ‘standards’, ‘measures’ and ‘evaluation’ are closely connected and, following Scriven
(1991), we define measurement as a process of obtaining information as a result of comparison with a
given standard. Evaluation we define as the process of making judgments and determining value
26

Valerie Anderson and Sarah Gilmore
based on the information provided by measurement. Measurement and evaluation of doctoral
education can provide insights about past performance but can also inform future direction, therefore
contributing to researcher and institutional performance. Processes involved in measurement and
evaluation are differently enacted as a result of the socially situated, complex and dynamic
environmental contexts in which they are undertaken (Russ-Eft and Preskill, 2005). In the UK
changes to funding processes, pressures of accountability, budgetary constraints and ’marketisation’
as well as the increasing prevalence of the ideologies and values of ‘new managerialism’ within HE
(Deem and Brehony, 2005) form the context for the current approach. This reflects the needs of
‘students as consumers’ and government and other funding agencies as ‘clients’ (Harvey, 2002), and
takes the form of performance indicators, league tables, and student satisfaction targets. Critics argue
that this overlooks intangible and developmental features of student learning resulting in a
commoditisation of education which prioritises educational ‘productivity’ and ‘student satisfaction and
throughput’ and focuses attention on students’ achievement against check-lists of behaviours,
standards and indicators (Schuck et al, 2008; Arthur, 2009; Blackmore, 2009).
The doctoral education Researcher Development Framework (RDF) illustrates the dominance of
standards, descriptors and check-lists in contemporary higher education in UK. Supported by QAA
and RCUIK, the framework was launched in September 2010 and articulates extensive lists of
knowledge, behaviours and attributes associated with research. The framework is structured into four
domains: knowledge and intellectual abilities; personal effectiveness; research governance and
organisation; and engagement, influence and impact. Each of these domains is subdivided into three
further sub-domains and the sixty-three descriptors that result are distributed across twelve subdomains
which each contain further indicators of expectation spread over three to five ‘phases’, which
represent distinct stages of development or levels of performance.
Further evidence of the dominance of the explicit and measureable over the tacit and
intangible features of learning is provided by the framework for evaluation developed to
support doctoral education processes. Originally known as the ‘Rugby Framework’ an
‘Impact Framework’ (IF) was launched in 2008 by Vitae, the organisation supported by
RCUK to champion the development of doctoral researchers in UK HEIs and research
institutes. Derived from a framework of evaluation proposed with business organisations
in mind it categorises five hierarchical points of impact. The first (foundational) level
relates to investment in the infrastructure for doctoral education. The second level
(referred to as impact level one) is the reaction of participants to training activities. Impact
level two concerns the extent to which knowledge, skills or attitudes have been
developed as a result of attending doctoral education programmes. Impact level three
refers to the extent to which behaviour change and applications have occurred and
impact level four assesses the final results indicated by research outcomes, research
quality and research capability.
The RDF and the IF approach were developed following a process of extensive consultation within
the HE sector and with non-Higher Education employers and are extensively publicised in association
with RCUK and individual Research Funding Councils (see, for example, Vitae, 2012). We argue that
the conceptual basis of these frameworks is under-developed. In the next section we offer a
conceptual critique of the IF and show the limiting implications of this for doctoral education.
3. Critique and discussion
Measurement and evaluation processes are undertaken to fulfil three interrelated functions
(Edelenbos and van Buuren, 2005). The distinction between formative and summative functions is
well-known within the education environment as is the second distinction between continuous
improvement and ‘quality assurance’; the former being an internal process of assessing and
enhancing ‘the student experience’ and the latter being led by external or regulatory evaluators. The
third distinction, which is fundamental to this paper, is between a rational / objectivist function
involving judgement and assessment of achievement against linear and hierarchical targets and a
constructivist function to enhance learning opportunities through valuing and fostering intangible and
unexpected outcomes of learning that arise through processes of interaction between multiple
interested parties or stakeholders (Guba and Lincoln, 1989).
Originally postulated in 1959, the Kirkpatrick taxonomy of evaluation from which the IF is derived, has
been an enduring feature of training and development practice (Russ-Eft and Preskill, 2005). It
represents a summatively focused, externally orientated and hierarchical approach to learning. In the
27

Valerie Anderson and Sarah Gilmore
critique that follows we draw on insights from organisational Human Resource Development (HRD)
and academic development and pedagogy literatures to highlight six areas of conceptual and practical
difficulty.
One area of difficulty is time orientation. The focus of the IF, which is aligned with the emphasis on
performance indicators and student satisfaction targets is on ’proving’ outcomes (Russ-Eft and
Preskill, 2005) of programmes previously undertaken. Opportunities for forward-looking assessments
to ’improve’ learning and adapt provision to developing challenges in the research environment are
overlooked. Schuck et al (2008), for example, highlight the importance of autonomy and expertise for
academic professionalism and Bottery and Wright (1997) argue that the value of students’ learning
beyond ‘current performance’ is increasingly overlooked as ‘time past’ and ‘time present’
measurements result in tutor and student compliance rather than professional, reflective, academic
practice.
A second problem is the IF focuses on formal taught programmes. The increasingly diverse profile of
the research student population in UK as elsewhere in the developed world (Hodsdon and Buckley,
2011; CFGE, 2010) requires increasing recognition of the role of flexible, ‘informal’ and nonclassroom
based learning and development processes for doctoral students grounded in regular
supervision processes and supplemented through activities such as mentoring, work-placements and
informal learning opportunities (Boud and Lee, 2005; Malfroy, 2011).
Third, critics of the Kirkpatrick approach to the evaluation of learning (see, for example, Guerci and
Vinante, 2011; Nickols, 2005) highlight how measurement at the ’higher levels’ of the framework is
problematic as the distinction between ‘outputs’ (quantities) and ‘outcomes’ (consequences) is
unclear, something which is acknowledged by the ‘Rugby Team’ responsible for the Impact
Framework. Even in business environments, for which the Kirkpatrick framework was originally
devised, survey evidence from UK and USA (CIPD 2006; Sugrue and Kim, 2004) suggests that only
one-third of UK work organisations manage to capture the ‘ultimate effect’ of learning and 80 percent
of learning and development specialists in work organisations believe that it is not possible to
measure the organisational benefits of learning.
The fourth issue is the problematic nature of the data required and the direct attribution of cause and
effect which is assumed. At an institutional level research activities and outcomes result from the
collective effect of a range of interrelated factors including: the research environment, quality of
supervision, ability, and motivational factors. Detailed attempts to directly attribute cause and effect
relationships to any one variable (or training programme) are conceptually inappropriate; a focus on
the aggregate contribution made by a more dispersed range of learning processes is required
(Sugrue et al, 2005; Deem and Brehony, 2005).
Fifth, the ’separation’ between ’evaluator’ and the ’subject’ of the evaluation, which undergirds the
linear and objectivist approach of the IF fosters a dysfunctional distance between those involved in
measurement and evaluation and other stakeholders involved in doctoral learning (Jayanti, 2011;
Edelenbos and van Buuren, 2005; Guerci and Vinante, 2011). The IF pays scant attention to the
different perceptions and expectations of stakeholders (such as supervisors, principal investigators,
deans and research directors) without whose engagement doctoral student development is
diminished. Nickols (2005) argues for a ‘stakeholder-return-on-expectation’ approach to evaluation
incorporating both formative and summative functions to incorporate multiple perspectives about the
value of development processes, taking into account both past achievement and expected
contribution.
Finally, although the systematic nature of the ‘levels’ approach to the IF makes it attractive to policy
makers its comprehensive approach is unwieldy and time-consuming for evaluators. Surveys of
training practitioners suggest that ‘traditional’ evaluation processes are not systematically enacted
because of the high levels of resourcing and time required (Anderson, 2007). In addition, whilst a
comprehensive approach provides a range of different general descriptions of training effects, this
‘one-size-fits-all-approach may fail to highlight factors underpinning the success (or failure) of aspects
of doctoral education provision. Brinkerhoff (2005) argues instead for a focus on the experiences of
individual students or members of research teams who have been most (or least) successful to
enable a more telling identification of the actual nature of the outcomes and to understand and learn
from how they have occurred.
28

Valerie Anderson and Sarah Gilmore
In summary, the IF approach is grounded in a framework which is characterised by conceptual and
practical limitations. It is aligned with the marketised and managerialist approach that dominates the
HE environment and the focus is linear, objectivist and grounded in a simplified view of training
effectiveness. There are assumptions of causal linkages among educational outcomes at different
levels that are not empirically verified and it lacks a multi-actor perspective. In a context where a step
change in research and knowledge exchange process are required within both HEIs and the national
and international labour market we argue that this is inappropriate and the next section outlines our
proposed alternative framework.
4. Proposal for a new framework
Derived from the literature about different forms of evaluation (see, for example, Kearns, 2005b;
Nickols, 2005; Guerci and Vinante, 2011; Anderson, 2007; Kaplan and Norton, 2001) the framework
we propose is inclusive, multi-dimensional and grounded in a stakeholder approach (Donaldson and
Preston, 1995; Freeman, 1984; Nickols, 2005). We use the term ‘pluralistic’, a term used to refer to
languages (such as English) or phenomena that have more than one centre (see, for example, Guba
and Lincoln, 1989; Edelenbos and van Buuren, 2005; Van der Meer and Edelenbos, 2006) to connote
the recognition that learning and evaluation are both outcomes of interaction and argumentation
processes between different interested parties or stakeholders. This means that, in addition to data
about the experiences of learners and training and education ‘deliverers’ there is an equivalent focus
on the expectations or the concerns of stakeholders including supervisors, principal investigators,
deans, research directors, senior managers, project sponsors and funders.
There are three elements to our framework. First, although we argue for a constructivist approach we
accept the institutional imperative of efficient and effective provision of doctoral education providing
an appropriate student experience and the achievement of learning outcomes. The second element
recognises the socially constructed nature of learning and we include an examination of data from
those involved in doctoral learning processes about their expectations of and experiences associated
with doctoral development. To achieve this we suggest engaging students, supervisors and leaders of
faculties; schools or research groups to construct benchmark standards of expectation for doctoral
development and education processes. Third, building on our awareness of the contextually situated
nature of doctoral education we include an evaluation of achievement against HEI strategic aspiration
measures. For each element of the framework we highlight measurement options and suggest
potential data sources.
Four key principles undergird our approach. First, a recognition that student learning, at doctoral as at
any other level, is a messy, unpredictable process that may lead to unintended, often tacit, outcomes
which may be at least as valuable as explicit expectations of behaviours, skills or knowledge. Second,
to illuminate the effects of this more dispersed range of learning processes and outcomes we propose
a focus on longitudinal, aggregated institutional data in place of the ‘event by event’ traditional
evaluation approach of the IF. Third, the utilisation of existing institutional data sources is advocated
where possible to avoid diverting resources from learning, teaching and the management of doctoral
education into the unwieldy processes assumed within the IF. Fourth, whilst we accept the necessity
of appropriate quantitative data measures we argue that the constructed nature of learning requires
qualitative information to be integrated within the evaluation process recognising the important
involvement of a range of stakeholders to both doctoral education and its continuous enhancement.
4.1 Efficiency and effectiveness
Whilst many stakeholders may view measures of operational efficiency and effectiveness within HEIs
with some suspicion we recognise that this is an increasing concern for senior managers and those
responsible for the performance of Graduate schools in a context of increasing funding restriction. We
include these measures within our framework although we aim for a balance between ‘throughput’
indicators and measures of the outcomes of doctoral education processes. This element of our
framework includes two ‘categories’: first, the operational efficiency of education processes and
second, the extent to which learning outcomes are applied to doctoral research projects.
In most HEIs indicators of operational efficiency are already available within institutional information
systems and relevant measures are: annual development days per research student; course-based
training days (FTE) provided per year; the proportion of research students engaging with doctoral
education processes; the proportion of research students with a personal development plan; and
29

Valerie Anderson and Sarah Gilmore
doctoral progression data. In most cases annual or biannual data would form the basis for this
element.
Whilst operational efficiency measures are a useful baseline, information about the outcomes of
doctoral education processes and the extent to which these are applied to benefit research projects is
of more importance. The contribution of supervisors and principal investigators in doctoral education
processes is often underestimated (Lee, 2008; Halse and Malfoy, 2010) and we argue that the
supervisor’s perspective is important for any assessment of the extent to which students are able to
apply their learning to their doctoral research projects. Therefore, in addition to the use of learning
event feedback processes provided by learners (workshop; course; mentoring programme feedback
etc.) we advocate the inclusion of feedback from supervisors gathered on a regular basis as well as
qualitative information from doctoral progression events to enable a formative assessment of the
extent to which learning has occurred and has been applied to the benefit of both the individual
student and the research project which they are undertaking.
4.2 Stakeholder engagement
The second element of our framework seeks to overcome the separation between ’evaluators’ and
‘the evaluated’ and to foster a shared ownership of both doctoral education and its evaluation by
developing the remit of the concept of ‘engagement’ with learning. Biggs and Tang (2007) argue that
engagement is reflected in students’ perceptions of personal involvement with learning leading to
expectations and confidence in future ‘deeper’ enquiry and achievement. We extend this beyond the
focus on learners to incorporate other important stakeholders in doctoral development processes
such as supervisors, tutors, members of research groups, faculties and schools. This element also
reflects the constructivist features of learning recognising that all those involved in formal and informal
learning events consciously and unconsciously make use of new information and experiences to reexamine
their existing understanding of both themselves and their subject.
Research into the evaluation of learning in work organisations highlights the importance of measures
of performance against standards of ‘good practice’ or ‘excellence’ (Anderson, 2007) which may be
undertaken as a ‘one-off’ but are more useful when treated as a continuous process of enhancement.
This element of the framework encourages consultation processes with Deans, Heads of Schools and
other key stakeholders within the research environment of the HEI leading to the construction of a
range of benchmarks which reflect different expectations of doctoral education. This element has both
formative and summative intent, taking into account both past achievement and expected contribution
and may well involve an assessment of research environment indicators; doctoral completion rates;
publications per research student. External benchmarking is also possible through participation in the
bi–annual UK Higher Education Academy (HEA) Postgraduate Research Experience Survey (PRES)
which provides an opportunity for HEIs to benchmark their results against those for the sector
aggregate and within benchmarking groups providing a trigger for enhancement and the ability to
share effective practice relating to areas such as: research supervision; the doctoral education
infrastructure and students’ perception of provision for the development of their professional research
skills and career development capabilities (Hodsdon and Buckley, 2011).
4.3 Strategic aspiration measures
Different approaches to strategic aspirations are evident in the strategic management literature
(Shinkle, 2012) incorporating concepts associated with forward looking strategic choice, goal seeking
and analysis as well as behaviours associated with assumptions about bounded rationality and
adaptive learning. Within the HE environment the generic nature of the IF makes it difficult to link with
institutional strategic reference points and the third element of our framework comprises such an
assessment of the alignment of doctoral learning and education against the strategic aspirations of
the HEI or research institute in which it occurs.
These measures are the most difficult to articulate and assess as there is often very little ‘base data’
against which appropriate comparisons and judgements can be made. Given the different strategic
aspirations of HEIs relating to research appropriate indicators will vary but measures might include an
assessment of the research culture of the HEI; the development of supervisory capability throughout
the organisation and the contribution that doctoral education processes make to University and
Faculty research strategies as articulated in internal and externally ‘facing’ documents. Whatever
indicators are identified it is important that they meet four criteria. First, they should be accepted as
30

Valerie Anderson and Sarah Gilmore
relevant by users. Second, they should be ‘aligned’; compatible with other organisational and external
measures recognised as important by key stakeholders. Third, they should have the capacity of
longitudinal tracking and fourth, they should reflect activities that are within the control of those
responsible for the research environment in the HEI.
Such assessments are likely to incorporate qualitative as well as quantitative data relating to: final
examination and doctoral ‘success’ rates; research student progress; comments by external
examiners and the ‘return on expectations’ of doctoral education articulated by key stakeholders such
as Directors of Research, the Professoriate, and Faculty and University Research Committees. This
element of our framework assumes the existence of strategic aspirations for research at strategic
level although it may be that such aspirations are implicit and tacit rather than expressed explicitly
within institutional or faculty strategic documents. In addition, it requires an acceptance that
institutional strategy is rarely static; strategic aspirations will change as a result of environmental,
systemic and opportunistic factors, requiring adaptation and responsiveness of those responsible for
doctoral education processes.
5. Conclusion
Our paper describes the main features of the IF model of evaluation of doctoral education which, we
argue, represents the dominance of summatively focused and linear, hierarchical approaches to
learning, teaching and education. However, new models of funding and financial accountability in the
Higher Education sector provide a context of rapid change in the supply of students into research
degrees and an increase in the number of non-traditional students who are older, engage in work
activities alongside their research and bring with them experiential learning from previous career
experience. Such systemic changes challenge traditional classroom dominated training approaches to
doctoral education. New models of doctoral education are likely to focus instead on the development
of qualities of self-directed professionalism and career preparation for life outside of the academy
requiring educative approaches that are flexible, often ‘informal’ and situated alongside or outside of
classroom based training processes. Such a context is inappropriate for the IF model of evaluation.
Our proposed framework adopts a pluricentric approach, accepting that learning and value are
attributed as a result of interactions between, and the engagement of, different stakeholders (Guba
and Lincoln, 1989).
Adoption of such an approach requires those involved in doctoral education to develop new frames of
reference. First, it requires development of metrics and indicators relating doctoral education to
institutional strategic aspirations for research. Second, it requires doctoral educators to work closely
with other institutional departments and functions, for example from academic registry, finance or
marketing, to identify where qualitative and quantitative data relevant to doctoral education and its
contribution to faculty and institution-wide objectives may already exist. Our framework encourages
an integration of the work of those involved in doctoral education with a wide range of stakeholders
outside of the Graduate School.
We acknowledge limitations and questions raised by our proposed approach. A key question is: ‘who
owns evaluation?’ The framework we have proposed attempts to incorporate a range of ‘owners’,
from senior management to doctoral candidates and their supervisors; our argument blurs the
distinction between development processes and evaluation. Our framework is both normative, being
grounded in an acceptance of the priority of HEI research performance and constructivist, seeking to
recognise the emergent and potentially disruptive and unanticipated consequences of learning and
education at doctoral level. We accept the likely ambiguity and tension between these different
positions which arise from our pluricentric intent. Further empirical work through pilot studies is
needed to establish the extent to which our proposed approach is achievable in practice.
In conclusion, our paper critiques the ‘one-size-fits-all’ linear and hierarchical approach of the IF. Our
proposed framework provides the basis for those involved in HEIs to identify relevant measures to
evaluate the efficiency and effectiveness of doctoral education processes; to foster stakeholder
engagement with doctoral development and assess the contribution that doctoral education makes to
the achievement of strategic research aspirations. Russ-Eft and Preskill (2005:71) comment, with
regard to processes outside of HEIs, that “evaluation ...seems to be stuck in an intellectual
quagmire...because of an over-reliance on a conceptually simple approach to evaluation, namely the
four-level taxonomy introduced and promoted by Kirkpatrick”. Drawing on approaches developed in
31

Valerie Anderson and Sarah Gilmore
organisational and policy contexts we assert that the IF in UK doctoral extends this quagmire to HEIs
and we propose an alternative framework.
References
Arthur, L. (2009) “From Performativity to Professionalism: Lecturers’ Responses to Student Feedback”, Teaching
in Higher Education, Vol 14, No. 4, pp 441-454.
Anderson, V. (2007) The Value of Learning: from Return on Investment to Return on Expectation, London,
Chartered Institute of Personnel and Development.
Biggs, J. B., & Tang, C. (2007) Teaching for quality learning at university, Buckingham: SRHE and Open
University Press.
Blackmore, J. (2009) “Academic Pedagogies, Quality Logics and Performative
Universities: Evaluating Teaching and What Students Want, Studies in Higher Education, Vol 34, No. 8, pp857-
872.
Bottery, M. and Wright, N. (1997) “Impoverishing a Sense of Professionalism”, Educational Management and
Administration, Vol. 25, No. 1, pp 7-24.
Boud, D. and Lee, X. (2005) “‘Peer Learning’ as Pedagogic Discourse for Research Education”, Studies in Higher
Education, Vol. 30, No. 5, pp 501-516.
Brinkerhoff, R.O. (2005) “The Success Case Method: A Strategic Evaluation Approach to Increasing the Value
and Effect of Training”, Advances in Developing Human Resources, Vol. 7, No, 1, pp. 86 – 101
Cassell, C. and Johnson, P. (2006) “Action Research: Explaining the Diversity, Human Relations, Vol 59, No. 6,
pp 783–814.
Commission on the Future of Graduate Education (2010), The Future of Graduate Education in the United
States. [online] http://www/fgereport.org/rsc/pdr/ExecSum.pdf.
Deem, R. and Brehony, K.J. (2005) “Management as Ideology: the Case of ‘New Managerialism’ in Higher
Education, Oxford Review of Education, Vol. 31, No. 2, pp 217-235.
Donaldson, T. and Preston, L. (1995). “The Stakeholder Theory of the Corporation: Concepts, Evidence, and
Implications”. Academy of Management Review, Vol. 20, No. 1, pp 65-91.
Edelenbos, J. and van Buuren, R. (2005) “The Learning Evaluation: a Theoretical and Empirical Exploration”,
Evaluation Review, Vol. 29, pp 591-612.
Freeman, R. (1984). Strategic Management: A Stakeholder Approach. Boston: Ballinger.
Guerci, M. and Vinante, M. (2011) “Training Evaluation: an Analysis of the Stakeholders’ Evaluation Needs”,
Journal of European Industrial Training, Vol. 35, No. 4, pp 385-410.
Guba, E.G. and Lincoln, Y.S. (1989) Fourth Generation Evaluation, Newbury Park, CA, Sage.
Halse, C. and Malfoy, J. (2010), “Retheorizing Doctoral Supervision as Professional Work”, Studies in Higher
Education, Vol. 35, No. 1, pp. 79-92
Harvey , L. (2002) “Evaluation for What?”, Teaching in Higher Education, Vol. 7, No. 3, pp 245-263.
Hodsdon, A. and Buckley L. (2011), Postgraduate Research Experience Survey: 2011 Results. [online]
http://www.heacademy.ac.uk/assets/documents/postgraduate/PRES_report_2011.pdf
Jayanti, E.B. (2011) “Through a Different Lens: a Survey of Linear Epistemological Assumptions Underlying HRD
Models”, Human Resource Development Review, Vol. 10, No. 1, pp 101-114.
Kaplan R and Norton D. (2001). The Strategy-Focused Organization: How Balanced Scorecard Companies
Thrive in the New Business Environment, Boston, MA, Harvard Business School Press.
Kearns, P. (2005a) Training Evaluation and ROI: How to Develop Value-Based Training, London, Chartered
Institute of Personnel and Development.
Kearns, P. (2005b) From Return on Investment to Added Value Evaluation: the Foundation for Organizational
Learning, Advances in Developing Human Resources, Vol. 7, No. 1, pp 135-145.
Kirkpatrick, D. L.and Kirkpatrick, J. D. (2006) Evaluating Training Programmes, San Fransisco, Berrett-Koehler.
Lee, A. (2008) “How are Doctoral Students Supervised? Concepts of Doctoral Research Supervision”, Studies in
Higher Education, Vol. 33, No. 3, pp. 267-281.
Malfroy, J. (2011). “The Impact of University–Industry Research on Doctoral Programs and Practices”, Studies in
Higher Education, Vol. 36, No. 5, pp. 571-584.
Nickols, F.W. (2005) “Why a Stakeholder Approach to Evaluating Training”, Advances in Developing Human
Resources, Vol. 7, No. 1, pp 121- 134.
Quality Assurance Agency for Higher Education (QAA) (2011) “Doctoral Degree Characteristics”, [online]
http://www.qaa.ac.uk/Publications/InformationAndGuidance/Documents/Doctoral_Characteristics.pdf
Research Councils UK, (2006) “Adding Value: How the Research Councils Benefit the Economy”, [online]
http://www.rcuk.ac.uk/documents/publications/addingvalue.pdf
Rugby Team (2008) “The Research Councils UK, (2008) “The Concordat to Support the Career Development of
Researchers”, [online] http://www.researchconcordat.ac.uk/
Rugby Team (2008), “Rugby Team Impact Framework”,[online] Careers Research and Advisory Centre (CRAC),
http://www.vitae.ac.uk/CMS/files/1.Rugby%20Impact%20Framework_33.pdf
Rugby Team (2009), “Rubgy Team Impact Framework: One Year on”, [online]
http://www.vitae.ac.uk/CMS/files/upload/RTIF_update_Sept09.pdf
Russ-Eft, D. And Preskill, H. (2005) In Search of the Holy Grail: Return on Investment Evaluation in Human
Resource Development, Advances in Developing Human Resource , Vol. 7, No, 1, pp 71-85.
Scriven, M. (1991) Evaluation Thesaurus, Thousand Oaks, C, Sage.
32

Valerie Anderson and Sarah Gilmore
Shinkle, G.A. (2012) Organizational Aspirations, Reference Points and Goals: Building on the Past and Aiming
for the Future, Journal of Management, Vol. 38, No. 1, pp 415-455.
Schuck, S., Gordon, S. and Buchanan, J. (2008) What are we Missing Here? Problematising Wisdoms on
Teaching Quality and Professionalism in Higher Education, Teaching in Higher Education, vol. 13, No. 5, pp
537-547.
Sugrue, B., O’Driscoll, T., and Vona, M.K. (2006) C Level Perceptions of the Strategic Value of Learning,
Research Report, Alexandria, VA, American Society for Training and Development and IBM.
Universities UK (2005) “The Economic Impact of UK Higher Education”, University of Strathclyde, [online]
http://www.universitiesuk.ac.uk/Publications/Documents/economicimpact3.pdf.
Van der Meer, F, and Edelenbos, J. (2006) Evaluation in multi-actor policy processes: accountability, learning
and cooperation, Evaluation, vol. 12, No. 2, pp 201-218.
Vitae (2001) “Joint Statement of the Research Councils’ Skills Training Requirements
for Research Students”, [online] http://www.vitae.ac.uk/CMS/files/upload/RCUK-Joint-Skills-Statement-2001.pdf
Vitae (2012) “Introducing the researcher development framework”, [online]
http://www.vitae.ac.uk/researchers/429351/Introducing-the-Researcher-Development-Framework.html
33

Silent Voices in Organisations: Conscientization as a
Reflexive Research Methodology
Andrew Armitage
Anglia Ruskin University, UK
andrew.armitage@anglia.ac.uk
Abstract: Whilst there has been a steady and growing interest in collaborative, and emancipatory organisational
research practices that espouse inclusive organisational discourses and challenge, and reject the power relations
between the researcher and the researched, few have considered the dynamics of the dialogical process that
ensues within these contexts. Furthermore, the concept of what Paulo Freire calls conscientization is also largely
ignored in the research literature, this being central to the ownership of research problems, processes, eventual
solutions, and subsequently the liberation of the researched as they emerge from cultures of silence and
oppression. This paper advocates that changes in research practice and the way those who conduct research
engage with those they research are needed if modernist discourses and dominating ideologies that deny
individual voice are to be challenged. It is argued this can only be achieved by means of the dialogical process. It
is argued that dialogical studies offer democratic and emancipatory practices and the creation of shared
meanings, and understandings for those who partake in the research act. The ‘four moments of dialogic
research’: Engagement; Self awareness; the aesthetic; and Transformation are presented to illustrate how
dialogue can be used to uncover the silence of organisation life.
Keywords: silent voices, dignity, reflexivity, dialogue, respect, emancipation
1. Introduction
Morrison and Rothman (2008) have noted ‘Power differences between employees and their
organisations are at the heart of understanding the distortion in information and knowledge transfer
that are likely to occur as employees are silent about potential problems or concerns they have’. This,
it can be argued is central to the research act, especially if the researcher and the institution are to
gain from the experience of undertaking organisational research then it should be a collaborative and
participative endeavour that enables the outcomes of studies to be of benefit to both professional
practice and institutional policymaking (see, for example, Heron, 1996). It has previously been noted
that the actualities of undertaking “real world” research does not play out in this manner (see, for
example, Alvesson and Sköldberg, 2009) and was anticipated by Nadler (1980) and Lawler, et al
(1980) who identified the following widely held assumptions that researchers hold prior to entering the
research arena:
Assumption 1: The researcher has most or all the information and knowledge needed to carry
out a well-designed research project.
Assumption 2: Any instrument the researcher designs or selects will be accepted by the
organisation.
Assumption 3: The institution and its stakeholders do not need to know the researcher’s
orientation or the purposes of the research.
Assumption 4: Researchers do not need to know the institutions orientation or “implicit”
organisational theories.
Assumption 5: Institutional commitment to participating in research can be obtained in the
interest of ‘science’.
Assumption 6: Adoption and implementation will follow assessment, diagnosis, and solution
identification.
The forgoing suggests what might be termed a “Banking Method” of undertaking research that is
couched firmly within positivism and the scientific method, where the researcher acts as “expert”, and
assumes that institutional stakeholders will be ready and willing to cooperate with the research
agenda defined by the researcher. It also assumes that the researcher can remain oblivious to the
organisational context into which they intend do their study. However, those wedded to qualitative
research approaches have challenged what they consider as the objectification of the research act as
exemplified in perspectives that fall broadly under the quantitative and scientific paradigms. In their
challenge to quantitative research paradigms, qualitative researchers have emphasised the need for,
and recognition of human intervention at all levels and stages in the research process as necessary to
34

Andrew Armitage
Investigate and report reality “as it really is”. However, some have rejected the qualitative “battle cry”
of subject centred enquiry as traditionally presented in the research literature and text books, and
have called for the research act, and its associated methodologies to be driven not by paradigmatic
ideologies, methods and procedures, but rather by a reflexive methodology that is informed by
ontological and epistemological considerations that focus upon the researchers disposition to their
research environment rather than any specific methodological likings, methods and procedures he or
she prefers (see, for example, Alvesson and Sköldberg, 2009). What will be advocated here is that a
reflexive methodology is central to any research act, whether this is of a quantitative or qualitative
perspective, and a radical ontological and epistemological re-evaluation of traditional research
practices are needed if we are to truly humanise the research act. As such, it will argue that Paulo
Freire’s dialogical process, and his concept of conscientization is a necessary condition for a reflexive
methodology. It will then present the four moments of dialogical research, these being: Engagement;
Self awareness; The Aesthetic; and Transformation as a means to illustrate how dialogical inquiry can
be used to uncover the silent voices of organisation reality.
2. The culture of silence: Conscientization, reflexivity and dialogue
It can be argued that organisations are places where two cultures exist. On the one hand there is the
official cultural norm’s propagated by organisational customs, working practices, and behaviours. On
the other there is a culture of silence that operates at a subterranean level of organisational life (see,
for example, Greenberg and Edwards, 2009). However, research into the silent culture of
organisations still remains underdeveloped, and what has to date been published tends to adopt a
natural scientific attitude to their research designs and reporting mechanisms, with the notable
exceptions of Detert and Edmondson, 2006; Edwards and Gardner, 2007; and Milliken et al, 2003,
and Premeaux and Bedian, 2003. However, all of these studies whilst using qualitative research
designs do not offer those taking part in the study any hope of empowerment and escape from the
oppressive official organisational culture and rhetoric, and lack any notions of the concept of
conscientization, the process where an individual develops their sense of being a subject, and of
apprehending their ability to intervene in external reality (McCowon, 2006).
Intervening in realty to transform the world was central Paolo Freire’s concept of conscientization, and
dialogic practice. For Freire, the silent culture and his concerns for the oppressed was not only part of
his pedagogical approach of human freedom, but was also at the heart of his methodological rigour as
a researcher in the exploration of self and other, being predicated on the Hegelian concept of
becoming, and the Marxist theory of praxis (action and reflection) as a means to gain authentic
knowledge and reality of the world. Central to Freire’s approach was the concept of conscientization
and liberation, whereby individuals become critically aware of their true situation, intervene in its realty
and take charge of their destiny, but as he acknowledged that for the oppressed change and taking
control of their destiny was particularly difficult to put into practice as they belonged to “the culture of
silence”. As Freire noted there is no indivisible solidarity between humans and their world, and no
dichotomy can be made between the two, and that ‘Authentic reflection considers neither abstract
man nor the world without men, but men in their relations to the world’ (Freire, 1972:54), and that ‘we
are not only in the world but also with the world’ that is ‘essentially related to it’ (Freire, 1972:51). Thus
conscientization for Freire, together with the dialogical process was the means to unlock and
challenge the culture of silence.
Gajardo (1991:40) notes that conscientization also introduces notions of reflexivity, and that a
conscientized person is the ‘subject of the processes of change, and actor in the management and
development of the critical and reflexive process, where an individual is capable of understanding his
or her reality in order to transform it’. There are according to Bourdieu and Wacquant (1992) different
varieties of reflexivity, which include postmodern sociology, ethno-methodological ethnography as
text, critical phenomenology, social scientific studies of the natural sciences, plus Gouldner and
Giddens double hermeneutics. Bourdieu’s interpretation of reflexivity sees the researcher as part a
social field, with specific relationships of power conditions that generate a specific “habitus”, that is, a
pattern of action dispositions among participants – also belongs here (Alvesson and Sköldberg,
2009). Other discussions on reflexivity have focused on the sociology of knowledge (see, for example,
Ashmore, 1989; Lynch, 2000; Woolgar, 1988) or the politics of doing and publishing research
(Alvesson and Sveningsson, 2008). Thus the literature identifies different uses of reflexivity that draw
attention to the complex relationship between the process of knowledge production and the various
contexts of such processes, as well as the involvement of the knowledge production (Alvesson and
Sköldberg, 2009). Calas and Smircich (1992:240) speak of reflexivity that constantly assesses the
35

Andrew Armitage
relationship between “knowledge” and the ways of “doing knowledge”. This implies that serious
attention is paid to the way different types of logistic, social, political, and theoretical elements are
woven together in the process of knowledge development, during which empirical material is
constructed, interpreted, and written. As Alvesson and Sköldberg, 2009:9) note:
‘Empirical research in a reflexive mode starts from a sceptical approach to what appear
at a superficial glance as unproblematic replicas of the way reality functions, while at the
same time maintaining the belief that the study of suitable (well thought out) excepts from
the reality can provide an important basis for a generation of knowledge that opens up
rather than closes, and furnishes opportunities for understanding rather than establishes
“truths”’.
According to Roebuck (2007) reflexive and reflective practice together can be described as a process
of inquiry which facilities appreciation and understanding of contextualised views, a deeper learning
experience, the development of ideas, and conditions for actual change. Furthermore, Cunliffe (2004)
notes that reflexivity is a process where ‘Critically reflexive practice embraces subjective
understandings of reality as a basis for thinking more critically about the assumptions, values, and
actions on others’. According to Cunliffe (2004:407) reflexive practice is therefore important because
‘it helps us understand how we constitute our realities and identifies in relational ways, and where we
can develop more collaborative and responsive ways of managing organizations’. Cunliffe (1999:8)
suggests critical theory draws on social constructionist suppositions that we construct social realties
between us in our interactions (Prasad and Caproni, 1997) and that ‘organisational realities and
identities are interwoven in a continuous process of mutual construction; we co-construct our realities
in our conversations. Accordingly our knowledge of the world is also constructed through interaction
because we make sense of what is happening around us as we talk. As Glass (2001:21) notes human
beings inhabit, and are inhabited by, the structures, institutions, social relations, and selfunderstandings
that comprise a people’s culture, and that:
‘The practice of freedom, as critical reflexive practice, must grasp the outward direction,
meaning, and consequences of action, and also its inward meaning as a realization and
articulation of the self. Therefore, education as a practice of freedom must include a kind
of historic-cultural, political psychoanalysis that reveals the formation of the self and its
situation all their dynamic and dialectical relations’.
Freire contends that people must first (critically) recognize how their reality comes into being so that
their ‘transforming action can create new realities, which makes possible a fuller humanity’ (Freire,
1972:29). Critical reflection is central to understanding reality, and an individual’s relationship with the
world and ‘Consciousness is constituted in the dialectic of man’s objectification of and action upon the
world. However, consciousness is never a mere reflection of, but a reflection upon material reality’
(Freire, 1972:53). The process of conscientization has two central features: dialogue and
problematization, and Freire (1972) describes the process of conscientization as having three stages,
whereby individuals move from magical, to naive, and finally to critical consciousness. For Freire
conscientization is a dialectic of reflection and action, and the gaining of critical consciousness will not
of itself transform the world (McCowon, 2006). As Freire (1972:47) notes ‘this discovery cannot be
purely intellectual but must involve action; nor can it be limited to mere activism, but involve serious
reflection’. Furthermore, conscientization is not purely one of individual development, as is must be
located within the context of the collective, in mutually supportive horizontal relationships (McCowon).
According to Bolton (2001) effective reflective practice is a dynamic and challenging process, and
requires those who partake in its process learn to question reflexively, through dialogue, their
personal and professional practices, and the impact these will have on the wider society and
individuals they interact with, and Steier (1995:163) notes, reflexivity is when ‘we contextually
recognise the various mutual relationships in which our knowing activities are embedded’.
Dialogical research focuses upon the power-knowledge nexus of human encounters, and the
fragmentation and disunity of discourse. As such, studies that take a dialogical perspective focus
upon the realities of individuals in their social, political and cultural contexts and ‘includes focusing on
the constructed nature of people and reality, emphasising language as a system of distinctions which
are central to the construction process, arguing against grand narratives and large scale theories
[and] systems‘ (Alvesson and Deetz: 2000:36). As such, dialogical studies reject functional
conceptions of the world which are regarded as theories of domination and oppression, and eschew
the power/knowledge nexus and the role that powerful elites play in the domination of discourse. They
are therefore a response to research practices that are grounded within modernistic processes,
36

Andrew Armitage
practices, and ideologies that deny individuals a voice in the research act. Their focus is instead being
directed upon the “silent voices” of those who are “oppressed” by organisational domination in order
to help them problematize their social, political, and cultural situationality, this being central to for
example, participatory research studies, action research, co-operative research, and critical
ethnographies. Therefore, dialogical studies give attention to the creation of a space of recovery
whereby individuals and groups who are marginalised can reclaim their identity, transform, and give
meaning to their lives in the shadow of those who reside within powerful (dominating) elites (Freire,
1970). Dialogue espouses a value and ethical approach that engenders dignity, honesty and trust
built upon equitable personal relationships and offers creative approaches to establish common
meanings and understandings. As Bohm et al (1991) state:
‘It [dialogue] enables inquiry into, and understanding of, the sorts of processes that
fragment and interfere with real communication between individuals, nations and even
different parts of the same organization. In our modern culture men and women are able
to interact with one another in many ways: they can sing dance or play together with little
difficulty but their ability to talk together about subjects that matter deeply to them seems
invariable to lead to dispute, division and often to violence. In our view, this condition
points to a deep and pervasive defect in the process of human thought.’’
For Bohm dialogue has the potential for profound change, stating that ‘I’m suggesting that there is the
possibility for the transformation of the nature of consciousness, both individually and collectively, and
that whether this can be achieved culturally and socially depends on dialogue’ (Bohm, 1996:54). In
Dialogue, a group of people can explore the individual and collective presuppositions, ideas, beliefs,
and feelings that subtly control their interactions and it provides an opportunity for individuals to
participate in a process that is safe for them to present their ideas without the fear of ridicule. It can
reveal the patterns of incoherence that lead the group to address contradictory issues or, on the other
hand, to suspend judgement of opposing views. Dialogue is a way of observing, collectively, how
hidden values and intentions can control our behaviour, and how “unnoticed” cultural differences
clash and is an arena in which collective learning takes place and out of which a sense of increased
harmony, fellowship and creativity can arise (see, for example, Gadamer, 1979; Habermas, 1984).
Freire (1970 and 1972) argues that dialogue and critical reflection are central to what he called a
problem posing mode of inquiry, and uses the concept of conscientization as a means whereby
individuals gain critical awareness to overcome the oppression of their situation, and to achieve their
socio-cultural realty which shapes their lives, and to collectively transform that realty. Transformative
pedagogy and research according Freire (1972) is central to humanistic and emancipatory practices,
whereby individuals “‘exist in and with the world”. Freire (1972:51) notes that conscientization is
where ‘Only men, as “open” beings are able to achieve the complex operation of simultaneously
transforming the world by their action and grasping and expressing the world’s reality in their creative
language’.
3. Researching the culture of silence: Personal reflections from the field
What follows are personal insights from the field of researching silent voices, which are presented as
the ‘four moments of dialogical research’ being: Engagement (Setting the scene); Self awareness
(Letting steams of consciousness flow); Aesthetic (Moving beyond self and the other); Transformation
(Dialogue, hope and freedom). These moments are taken from personal insights, and direct
experience of using dialogue as a method of inquiry to elicit authentic accounts of reality (see, for
example, Armiatge, 2010; Armitage, 2011; Armiatge and Thornton, 2009; Rees and Armitage, 2010)
and represent the different movements and moods of conscientization within dialogic inquiry, whether
these are of the self or of self and other to investigate concrete and abstract realities of organisational
life.
4. First moment of conscientization: Engagement (setting the scene)
For dialogue to take place those involved must be prepared to adopt an open attitude and suspend
their judgements and preconceptions they bring to the research act. In their research concerning
silence in organisations, Armiatge and Thornton (2009) found that it was useful for those who wish to
be engaged in the dialogic process to meet informally in order to converse as to whether they can
work in this mode of open engagement and trust. It is imperative that discussing their personal
perspectives are aired for collaborative exchanges to take place (see, for example, Bohm, 1996b).
Notions of the research-researched relationship as framed by traditional “domination lead” research
approaches must be re-framed not as an object-subject relationship, but as a subject-subject
relationship. This requires positions of power must be discarded, whether real or perceived. This is a
37

Andrew Armitage
requirement if mutually agreed to exchange their personal perspectives and initial ideas and issues
are to emerge in an open and transparent environment of mutual respect and dignity (see, for
example, Rees and Armitage, 2010). This allows a “drama” to unfold as dialogue progresses from
concrete to abstract realities, being facilitated by an emergent methodology of dialogue as a creative
process (see, for example, Bohm et al, 1991 and Bohm 1996). As such, dialogue allows a display of
thought and meaning that makes possible a kind of collective “proprioception” or immediate mirroring
back of both the content of thought and the less apparent, dynamic structures that govern it (Bohm,
1996). In dialogue, this can be experienced both individually and collectively. Each listener is able to
reflect back to each speaker and to the rest of the group, a view of some of the assumptions and
unspoken implications of what is being expressed, together with what is “being avoided” in the
conversation. It creates the opportunity for each participant to examine the preconceptions, prejudices
and the characteristic patterns that lie behind his or her thoughts, opinions, beliefs and feelings; along
with the roles he or she tends habitually to play. Dialogue introduces by its very act an ethical means
of engagement.
5. Second moment of conscientization: Self awareness (letting streams of
consciousness flow)
Collecting data from the field is a human act, and one that helps those who take part in personal
exchanges to focus on the ways in which they co-create and interpret their personal and social
realities as a means to seek truth, and meaning in the world. For example, Armitage (2010) in his
study with Human Resource professionals explored how individuals created and came to understand
their stories as social products that are produced in the context of their specific social, historical and
cultural locations, being interpretive a lens through which they can represent their words, which can
be communicated and re-told to others. This often results in a stream of “spontaneous” responses
allowing exchanges to follow their own “course of revelation”. However, this can only happen if
exchanges are conducted within a safe environment of trust, honesty and integrity, and where mutual
respect allows on the one hand candid exchanges and on the other the exploration of esoteric subject
matter as exchanges become more intense. Here, voice is an essential element of personal
exchanges, for example, within the context of doing interviews and focus groups. The exchanges
allow a poetical space where different views and ideas are weaved into a discourse and to
communicate something important for those who participate in personal exchanges. What is clear is
that there are moments in which those engaged in dialogue can “enter” each other’s minds, but this
can only be realised but listening to and valuing each other’s concerns and views.
6. Third moment of conscientization: The aesthetic (moving beyond self and
the other)
The dialogical exchanges can reveal through their autographical and autobiographical (see Bullough
and Pinnear, 2001) and at times surreal discourses hidden truths that come to the surface at “seminal
moments” in personal exchanges, being an expression of freedom, ideas, and feelings, producing
authentic dialogue that verges on the aesthetic. For example, Rees and Armitage (2010) through their
dialogical exchanges revealed hidden truths about themselves regarding individual identity, and their
roles and place in organisation life, and in doing so created an “aesthetic space” where beauty
became manifest in the style and content of their personal exchanges, and personal relationship as
they engaged in the dialogical process. Exchanges if allowed to flow and take their “natural course”
become creative, and where excitement and pleasure are experienced as the dialogue twists and
turns. This sense of freedom is a significant (aesthetic) aspect to allow a creative space to open, and
where new understandings emerge as a stream of consciousness.
7. Fourth moment of conscientization: Transformation (dialogue, hope and
freedom)
Exchanges can be liberating in at least two ways. First, they allow free expression, and second they
allow perspectives that transcended the “normal boundaries” of research discourse, and where minds
meet in the recognition of individual freedom and mutual respect. This allows individuals not to simply
accept the world as-it-is, but rather it is a space where they can freely challenge their pre-determined
assumptions in a safe environment. This allows them to transform their inner monologues into
dialogue. The decision to transform personal situations is achieved through spoken and written
exchanges in order to project concerns and issues onto the world. Narratives can emerge which are
transformative as individual learn of each other’s perspectives, and personal situations, which they
bring together in a plot line. As such dialogic narratives have a temporal dimension, disparate
38

Andrew Armitage
elements of personal exchanges come together to create personal realities. Here a critical turn arises
if individuals are to establish the essence of what confronts them in their organisational contexts, and
to embrace their ‘self-transformation’. As Armiatge (2010) found in his study of HR professionals,
individuals became emancipated as they moved beyond the aesthetic dimension to question how they
interpreted their living realities to give meaning to their social and political contexts they work in. This
he found awoke individuals to the domination of their organisational reality, whereby they described
through rich pictures and dialogue groups how they perceived, and were coping in modernistic
organisational value systems that diminishes individual worth, and that they had the power to create
their own “spaces of recovery” through shared meaning and language where they could reclaim their
identity. As Myers (2009:173) notes ‘All language can be treated as a social interaction (there is
always a speaker/listener and writer/reader), but discourse analysis focuses mostly on language in
use – the use of naturally occurring language in speech and/or written texts’. The decision to
transform personal situations must ultimately be that of the participants involved in the exchanges to
recognise and understand the power-knowledge nexus that they are subjected to, and this can only
be achieved if they wish to project their concerns and issues onto the world. Whilst individuals cannot
necessarily change the super-structure or the grand narrative of the organisation, individual freedom
and liberation is won in an individual’s personal space of existence in that they can better negotiate
their way around the local hidden power structures of the organisation more effectively.
8. Conclusions
As Gustavsen (2006:19) notes ‘All participants have the same status in the dialogic arena’ However,
this can only be achieved if researchers challenge their existing practices and adopt a role that is ‘to
“problematize” and to use the critical faculty’ (Freire, 1998:33). An approach that requires an authentic
research environment is one where the researcher and the researched become “conscientizised” in
the transformational engagement of their socio-historical-political worlds of self and other. This
requires a reflexive research methodology as advocated by Alvesson and Sköldberg (2009) which is
located within the social context of its practices, and the power relations underpinning the context they
are enacted within. This demands that researchers have to submit themselves to an attitude whereby
they acquire new knowledge in the process of becoming and praxis – research is not just the
collection of facts and figures that are to be submitted to analysis using pre-determined methods and
procedures. The following are therefore intended as the first practical steps towards these goals,
being a re-interpretation of the initial assumptions of Lawler, et al (1980):
Assumption 1: The researcher needs the information and knowledge that organizational
members have in order to correctly design the research.
Assumption 2: The kinds of measurement instruments used and the content of the measurement
instruments need to reflect a good knowledge of the local culture and climate of an organization.
Assumption 3: The client needs to know a considerable amount about the researcher’s
orientation to organisations and the reason for doing the particular research study.
Assumption 4: The researcher needs to know the client’s orientation and implicit theories of
organisation.
Assumption 5: Client commitment to participation in research can best be obtained by involving
the client in the design of the research and fully communication to the client the purpose of the
research
Assumption 6: Implementation and adoption are most likely to follow when research subjects
have been actively involved in the research process
References
Alvesson. M. and Deetz, S. (2000) Doing Critical Management Research, London: Sage.
Alvesson, M. and Sveningsson, S. (2008) Changing Organizational Culture, London: Sage.
Alvesson, M. and Sköldberg, K. (2009) Reflexive Methodology, London: Sage.
Armiatge, A. (2010) From sentimentalism towards a critical HRD pedagogy, Journal of European Industrial
Training, Vol.34, No.8/9, pp.735-752.
Armitage, A. and Thornton, A. (2009) The use of personal narratives to improve organisational life,
Interconnections, Issue 4, pp.17-22.
Ashmore, M. (1989) The Reflexive Thesis: Wrighting Sociology of Scientific Knowledge, Chicago IL: University of
Chicago Press.
Bohm, D. (1996) On Dialogue, London: Routledge.
Bohm, D, Factor, D and Garrett, P. (1991) Dialogue: A proposal, http://world.std.com/~lo/bohm/0000.html.
39

Andrew Armitage
Bolton, G. (2001) Reflective Practice, London: Sage.
Bourdieu, P. and Wacquant L. J. D. (1992) An Invitation to Reflexive Sociology, Cambridge MA: Polity Press.
Bullough, R.V. and Pinnear, S. (2001) “Guidelines for Quality in Autobiographical Forms of Self-Study Research”.
Education Researcher, 30 (3), pp.13-31.
Calas, M. and Smircich, L. (1992) Writing gender into organizational theory: directions from feminist perspectives.
In M. Reed and M. Hughes (Eds.) Re-thinking organizations: New directions in organisational theory and
analysis, London: Sage.
Cunliffe (1999:8) Critical Pedagogy: Reflexive Dailgical Practice. In Management Learning, Critical Management
Studies Conference, University of Manchester, 14-16 July.
Cunliffe, A. L. (2004) On Becoming a Critically Reflexive Practitioner, Journal of Management Education August
2004 Vol. 28 No. 4, pp.407-426.
Detert, J.R. and Edmondson, A. 2006 Everyday failures in organisational learning: Examining the high threshold
for speaking up at work. Working paper no.6-024, Harvard Business School, Cambridge, MA.
Edwards, M.S. and Gardner, J. (2007) The role of emotions in employees’ decisions to report or remain silent
about serious wrongdoing . In J Greenberg and M.S. Edwards (chairs), The role of emotion in individual
decision-making. Symposium presented at the annual meeting of the Academy of Management,
Philadelphia, PA.
Freire, P. (1970) Pedagogy of the oppressed. London, Penguin.
Freire, P. (1972) Cultural Action for Freedom, London: Penguin.
Gadamer, H-G. (1979) Truth and Method, London: Sheed and Ward.
Gajardo, M. (1991) La Concientización en América Latina: una Revisión Crítica. Michoacán: CREFAL – OEA.
Glass, R. D. (2001) On Paulo Freire’s Philosophy of Praxis and the Foundations of Liberation Education,
Educational Researcher, Vol. 30, No.2, pp.15-25.
Greenberg, J. and Edwards, M.S. (2009) Voice and Silence in Organisations, San Diego CA: Elsevier
Habermas, J. 1984, The Theory of Communicative Action, Translated by Thomas McCarthy. Cambridge: Polity
Press.
Heron, J. (1996) Co-operative Inquiry, London: Sage.Lawler, E.E., Nadler, D.A. and Cammann, C. (1980)
Organizational Assessment, New York: John Wiley and Sons.
Lynch, M. (2000) Against reflexivity as an academic virtue and source of privileged knowledge. Theory, Culture
and Society, 17, pp.26-54.
Milliken, F.J., Morrison, E.W. and Hewlin, P.F. (2003) An exploratory study of employee silence. Issues that
employees don’t communicate upward and why. Journal of Management Studies, 40, pp.1453-1476.
Myers, M.D. 2009, Qualitative Research in Business and Management, London: Sage.
Nadler, D. A. (1980) Role of Models in Organizational Assessment. In Lawler, E.E., Nadler, D.A. and Cammann,
C. (1980) Organizational Assessment, New York: John Wiley and Sons.
McCowon, T. (2006) Approaching the political in citizenship education: The perspectives of Paulo Freire and
Bernard Crick, Educate, Vol.6, No.1, pp. 57-70.
Morrison, E.W. and Rothman, N. (2008) Silence and the dynamics of power. In J. Greenberg and M.S. Edwards
(2009) Voice and Silence in Organisations, Vol.6, pp.111-134, San Diego CA: Elsevier.
Premeaux, S.F. and Bedian, A.G (2003) Breaking the silence: The moderating effects of self-monitoring in
predicting speaking up in the workplace. Journal of Management Studies, 40, pp.1537-1562.
Prasad and Caproni (1997) Critical Theory in the Management Classroom: Engaging Power, Ideology, and
Praxis, Journal of Management Education, 21(3), pp.284-291.
Rees, B. and Armitage, A. (2010) Talking about organisations, Interconnections, Issue 5, pp.64-68.
Roebuck, J. (2007) Reflexive practice: To enhance student learning, Journal of Learning Design, Vol. 2, No.1.
Siraj-Blatchford, I. & Siraj-Blatchford, J. (1997) Reflexivity, Social Justice and Educational Research. Cambridge
Journal of Education, 27, pp.235-248.
Steier, F. (Ed.) (1995). Research and Reflexivity. London: Sage Publications.
Woolgar, S. (ed.) (1988) Knowledge and Reflexivity: New frontiers in the sociology of knowledge, London: Sage.
40

The Role of Reflection by the Professional Information
Systems Practitioner in South Africa
Udo Richard Averweg
Information Services, eThekwini Municipality and Graduate School of
Business & Leadership, College of Law and Management Studies, University
of KwaZulu-Natal, Durban, South Africa
averwegu@durban.gov.za
Abstract: A professional is a practitioner whose practice is based on a significant body of theory, has appropriate
tertiary qualifications from a recognised body (in South Africa, usually a university or university of technology), is
committed to undergoing continuous professional development, consults best practices before undertaking work
and subscribes to a code of conduct (or code of ethics). The Computer Society South Africa (CSSA) has a Code
of Practice (Professional Conduct) for its registered members. The CSSA is widely recognised as a professional
body for information systems (IS) practitioners. In this paper focus is made on the professional IS practitioner
who conducts practitioner inquiry research in the IS discipline in organisations in South Africa. The goal of this
paper is to explore and discuss the role of reflection by the professional IS practitioner who conducts computing
research in organisations in South Africa. The author argues that the benefit of including reflection in the
professional IS practitioner’s learning is that it allows him to understand why he did something (e.g. a research
inquiry) in a particular way, recognising how he felt about it and thereby identifying his own personal strengths
and weaknesses. Such reflection then allows the IS practitioner to build on his personal strengths and to develop
approaches to minimise his weaknesses. The professional IS practitioner has to recognise ‘his’ IS practice as an
avenue for learning. This can be achieved by using reflection as a reflexivity tool to understand the professional
IS practitioner’s ‘true’ situation in an organisation and create the freedom he needs. Professional skills should
therefore be enhanced through a link between reflection and the IS practitioner’s praxis. The professional IS
practitioner develops praxis since action is informed by reflection. Professional IS practitioners need to explore
and be reflective as it is this habit of mind which is indeed a useful source of professional development. However,
for them to be also reflexive, supports critical reflection. To be reflexive can nourish reflections as introspection
leads to heightened awareness, change, growth and improvement of the self and for the IS discipline.
Keywords: computer society South Africa, critical reflection, professional Information Systems practitioner,
reflection, reflective practice, reflexivity
1. Introduction
A computing professional must take responsibility for acquiring and maintaining competence. He
requires some focus for his professional life in an organisational domain
(CC 2005 Overview Report 2005: 9). The importance of information systems (IS) and
information technology (IT) to organisations and the need for well-educated professionals in the
computing field is a basis for a strong link between formal educational programs (and resulting
qualifications) and the professional community of IS practitioners (Granger, Dick, Luftman, Van Slyke
and Watson, 2007; Bullen, Abraham, Gallagher, Simon, and Zwieg 2009). Sometimes reflection is
used by professional IS practitioners to build up their portfolio to demonstrate professional
development.
A professional is a practitioner whose practice is based on a significant body of theory, has
appropriate tertiary qualifications from a recognised body (in South Africa, usually a university or
university of technology), is committed to undergoing continuous professional development (CPD),
consults best practices before undertaking work and subscribes to a code of conduct (or code of
ethics). The Computer Society South Africa (CSSA) has a Code of Practice (Professional Conduct)
for its registered members. The CSSA is widely recognised as a professional body for IS practitioners.
In South Africa, some IS practitioners are registered as professional members of the CSSA –
designated by PMCSSA.
This position paper focuses on the professional IS practitioner who conducts practitioner inquiry
research in the IS discipline in organisations in South Africa. A position paper presents an arguable
opinion about an issue – in this case, the context of the professional IS practitioner who conducts
computing research in the workplace environment. The goal of this paper is to ‘convince’ the reader
that the author’s opinion (in the form of a short report and discussion of ideas, facts and situations) is
valid and worth listening to. To commence the discussion, the following question is therefore posed:
What is the role of reflection by the professional IS practitioner (PMCSSA) who conducts computing
41

Udo Richard Averweg
research in organisations in South Africa? Exploring this question is the objective of this position
paper.
This paper is organised as follows: Background to the concept of reflection is introduced. The family
of computing disciplines, with specific focus on the IS discipline, is then discussed. IS professionals in
the IS discipline is followed by a discussion of the CSSA. IS practitioner research and practitioner
inquiry research in organisations are then explored. Next follows a discussion of reflection by the
professional IS practitioner. Finally a conclusion is given.
2. Background to reflection
The literature on reflection originates from a range of disciplines including sociology and psychology.
Essentially reflection is the process of creating and clarifying meaning of an experience in terms of
self (for example, the professional IS practitioner in the IS discipline) and the outcome is a refined
conceptual perspective. For example, Dewey (1933: 12) describes the process of reflection as “the
kind of thinking that consists in turning a subject over in the mind and giving it serious thought”.
Moon (1999) argues that differences in reflection are not to different types of reflection but to
differences in the application of the reflective process ie. authors present a range of frameworks for
reflection. Moon suggests that reflection involves the process of learning and the representation of
that learning implies purpose and involves complex mental processing for issues that have no obvious
solution.
Pritchard (2002) indicates that there are different methods, such as note-taking, journal writing and
conversations with work colleagues, which facilitate reflection. These methods allow the professional
IS practitioner engaged in inquiry research to better understand ‘his’ practice. Learning thereby takes
place for the professional IS practitioner through reflective practice. Fox, Martin and Green (2008)
suggest that practitioners engage in learning which ‘taps’ into their experiences and critical reflection
on any difficulties which may have been encountered. The professional IS practitioner needs critical
reflection on ideologies that inform the systems in which he works by taking meaningful action to
advocate positive change and to envisage new possibilities in the organisation. The IS professional
practitioner may also yield to an overall improved reflection on the conditions for creating knowledge
in ‘his’ practice and reflexivity may then occur. Freshwater, Jasper and Gilbert (2003) distinguish
between the following allied terms:
Reflective practice – thinking about how one practises and developing an awareness of how
practice is structured;
Critical reflection – thinking about how one is thinking about practice and deconstructing practice
from a personal perspective; and
Reflexivity – thinking about how one is thinking within the political, ethical, social and historical
context from a global perspective.
For the professional IS practitioner, a focus on the process of learning makes reflection important
since it encourages IS practitioners to examine how, why and what they are thinking and feeling
during an experience rather than focusing solely on the outcome. The professional IS practitioner also
needs to examine the assumptions he brings to ‘his’ IS practice. This is further compounded by the
rapidly evolving IS discipline wherein professional IS practitioners work in changing political and
professional contexts in organisations. Rapid IT growth has placed increased demands on
professional IS practitioners.
Donald Schön is recognised as the leading theorist on reflection as a means to enhance professions
(Gidman 2007: 56). Schön proposed that practitioners draw on theories-of-practice more than they
draw on espoused theories. From Schön’s research and drawing on the work by Argyris, Putman and
Smith (1985), theories-of-practice comprise the implicit beliefs, values, assumptions, knowledge and
emotions that individual professional IS practitioners bring to their practice. They include their
personal theories (the judgments and evaluations they make about themselves and the surrounding
world) and their understanding of general IS theories. Theories-of-practice is the lens through which
professional IS practitioners view their IS practice, guiding the decisions they make about the actions
they will take, the ways in which they make sense of new knowledge and what they view as useful
(new) knowledge. Espoused theories are those theories that are learned by professional IS
practitioners in formal educational settings (such as universities or universities of technology).
42

Udo Richard Averweg
Practitioner research serves as a medium for practitioners to critically inspect practice and to take a
closer look at the foundations thereof (Fox et al. 2008). These researchers suggest that through
reflective activities, practitioner researchers can apply new theory to their work contexts.
Schön (1991) distinguishes between reflection-in-action and reflection-on-action as follows:
Reflection-in-action is situated within the activity itself. It occurs when there is no obvious action
and it is associated with theories-in-practice. A professional IS practitioner conducting practitioner
inquiry research, makes decisions during his practitioner research methodology process and
judges the success of each process step at the same time while he is conducting that step. It is
the ability of professionals to think what they are doing while they are doing it. It seems right to
say that our knowing is in our action (Schön 1991); and
Reflection-on-action occurs after the activity has taken place (ie. retrospectively). The
professional IS practitioner judges how successful he was and whether any changes to what he
did could have resulted in different outcomes.
Fox et al. (2008) highlight Schön’s view that practitioners learn by experience and through reflecting
on problems experienced. According to Reason and Bradbury (2006: 1) it is a participatory process
which involves “action and reflection, theory and practice, in participation with others”. The focus of
professional IS practitioner research is thus on genuine problems faced by individuals in an
organisation, thereby allowing for an insider IS discipline perspective. A look is now taken at the IS
discipline situated in the family of computing disciplines.
3. Family of computing disciplines
Computing means “any goal-oriented activity requiring, benefiting from, or creating computers”
(CC 2005 Overview Report 2005: 9). Computing is not just a single discipline but is a family of
disciplines. According to the CC 2005 Overview Report (2005: 14-15), there are five major computing
disciplines:
Computer Engineering which is concerned with the design and construction of computers and
computer-based systems;
Computer Science spans a wide range, from its theoretical and algorithmic foundation to
developments in robotics, computer vision, intelligent systems, bioinformatics;
Information Systems specialists focus on integrating IT solutions and business processes to meet
the information needs of businesses enabling them to achieve their objectives in an effective,
efficient way. This discipline’s perspective on IT emphasises information and views technology as
an instrument for generating, processing and distributing information;
Information Technology. While IS focus on the information aspects of IT, IT is the complement of
that perspective: its emphasis is on the technology itself more than on the information it conveys;
and
Software engineering is the discipline of developing and maintaining software systems that
behave reliably and efficiently, are affordable to develop and maintain, and satisfy all the
requirements that customers have defined for them.
For the purposes of this paper, focus is made on the IS discipline. In this discipline there is a plethora
of IS research that has been carried out and published in journals, textbooks, trade magazines. In the
case of IS research published in refereed journals, very often empirical research methodology
approaches have been used by the researcher(s). However, it is increasingly accepted in IS that
empirical research is not the only valid scientific methodology that can be used to produce good
research. For example, the ‘father’ of IS research in South Africa, the (late) Professor Dewald Roode
has criticised IS researchers for being too positivistic in their leanings, and for being unaware of an
array of methodological choices (Roode 2003). Practitioner inquiry research may therefore be another
methodology that could introduce a ‘softer’ view and use of information and IT in organisations in
South Africa – this may be more applicable than the ‘harder’ approaches that are typically found in the
natural sciences (such as, for example, Physics). Robson (2002) suggests that practitioners require a
high level of flexibility in their choice of methods and tools to help ensure credibility and dependability
in the complex-bound research situation.
43

Udo Richard Averweg
4. Information systems (IS) professionals in the IS discipline
In the IS 2010: Curriculum Guidelines, the authors note that “[t]he IS discipline contributes
significantly to several domains, including business and government. IS are complex systems
requiring both technical and organizational expertise for design, development, and management”
(Topi, Wright, Kaiser, Nunamaker, Sipior and de Vreede 2010: 365). According to this guidelines
report, in “... conceptualizing the role of information systems in the future ... several elements remain
important and characteristic of the discipline” (ibid: 369-370). These characteristics evolve around four
major areas of the IS profession:
IS professionals exist in a broad variety of domains (e.g. business, government, non-profit
organisations);
IS professionals must have strong analytical and critical thinking skills to thrive in a competitive
global environment;
IS professionals must exhibit strong ethical principles and have good interpersonal
communication and team skills (Overby 2006); and
IS professionals must design and implement IT solutions that enhance organisational
performance.
Professionals in the IS discipline “are primarily concerned with the information that computer systems
can provide to aid an enterprise in defining and achieving its goals, and the processes that an
organisation can implement or improve using information technology” - IS focus on the information
aspects of IT (CC 2005 Overview Report 2005: 14). IS professionals work with IT and must have
sound technical knowledge of computers, communications and software (Topi et al. 2010: 374). Some
IS practitioners in South Africa are registered as professional members of the CSSA (PMCSSA).
5. Computer Society South Africa (CSSA)
The Skills Framework for the Information Age (SFIA) is a common reference model for the
identification of skills needed to develop effective IS making use of IT. SFIA is used in more than 150
countries and uses a common language and logical structure outlining required skills, knowledge and
competence. Organisations seeking International Professional Practice Partnership (IP3)
accreditation define their professional standard requirements in terms of SFIA at Level 5 – see
www.ipthree.org. The IP3 professional standard includes all elements found in ‘traditional’
professions: skills based on theoretical knowledge; demonstration of competence; a defined work
autonomy; adherence to a professional code of conduct; and self-regulation through professional
certification.
The Computer Society South Africa is a member of the IP3 of the International Federation for
Information Processing (IFIP), which has an arm called IP3 of which the CSSA is also a member. The
CSSA therefore has a responsibility to monitor and enforce continuing development and maintenance
of professional competence of its professional members. Article 3.4 of the CSSA’s Articles of
Association states that one of the purposes of the Society is
“To improve the technical and general knowledge and to elevate the professional status
of persons engaged in ICT”.
The Foreword to the CSSA’s Code of Practice (Professional Conduct) states:
“The Code of Practice deals with the ways in which all members of the Society are
expected to exercise their professional competence …” – see
www.cssa.org.za/page.php?p_id=176
6. IS practitioner research
Practitioner researchers are distinct from other academic researchers in how they consider practice
and research due to the reflexivity that originates from the close environment of the practitioner in the
research (Fox et al. 2008). In IS, practitioners’ involvement in research and general inquiry that is
small-scale, local, grounded and carried out by IS professionals who directly deliver IS services, is an
essential ingredient of good practice in the business world and management workplace environment.
The professional IS practitioner must have strong ethical principles, demonstrate relevant, up-to-date
skills and capabilities appropriate to a particular role with practical experience complementing
theoretical knowledge. In order to maintain up-to-date skills, CPD is required. CPD “is a hallmark of a
Professional” (de Roche 2011).
44

Udo Richard Averweg
The professional IS practitioner should be able to demonstrate a combination of practical IS work and
engage in workplace specific research/innovation and be able to critically reflect on his practice – this
is a development of Schön’s notion of the reflective practitioner (Schön 1991). After all, in IS
practitioner research, there must be an inducement for refinement and/or improvement in the IS
practice. The reflective IS practitioner is therefore someone who is simply thoughtful about his
practice and such practice forms an integral part of the family of computing disciplines.
Dadds (2004: 9) suggests that practitioner research demands much of professionals since it calls for
“... an open-mindedness, courage in the face of self-critique and public sharing, emotional fortitude in
dealing with uncertainty and profound change, spiritual energy in sustaining curiosity, compassion
and the eternal search for new, improved practices”. The IS practitioner therefore needs to be
open-minded to constructive criticism regarding ‘his’ practice.
7. Practitioner inquiry research in organisations
Research is to be understood as original investigation undertaken by the professional IS practitioner
in order to gain knowledge and understanding which is of direct relevance to the needs of
organisations and where these lead to new or substantively improved insights. Practitioners usually
need to combine reflection and practice when addressing practice issues in organisational settings
(Hardman and Averweg 2011: 375).
Practitioner-research can be described as “a systematic form of enquiry that is collective,
collaborative, self-reflective, critical and undertaken by the participants of the inquiry” (McCutcheon
and Jung 1990). Practitioner researchers very often apply exemplars and theories to their own
experiences and situations in their jobs in organisational settings while maintaining strong ethical
principles. Practice must usually be accompanied by documentation of the research process, some
form of explanation or textual analysis, and a demonstration of critical reflection by the professional IS
practitioner. Reflection is seen as the practice of periodically stepping back to ponder on the actions
of oneself and in one’s immediate environment (Raelin 2001). Such critical reflection by the
professional IS practitioner in ‘his’ work practices in an organisation may be focused in three areas:
content reflection (how a practical problem in an organisation was solved);
process reflection (examining the procedures and sequences of events); and
Premise reflection (questioning the presuppositions in attending to the identified practical
problem).
Practitioner inquiry research suggests an open view to any appropriate methodology which may
inform the professional IS practitioner in the workplace in an organisation in South Africa. It is evident
that when selecting the appropriate methodology for conducting practitioner inquiry research, the
professional IS practitioner will require skills based on his theoretical IS and IT knowledge. This in turn
should ‘grow’ the professional IS practitioner’s practical IS and IT work skills and competencies to
achieve higher professional recognition and also serve as an avenue for such an IS practitioner to
continue to retain ‘his’ PMCSSA membership status. The focal point for IS practitioner inquiry
research is therefore identifying computing problems as they are seen by professional IS
practitioners. While there is active participation by professional IS practitioners in the computing
research realm in organisations in South Africa, further analysis is required of the actual context within
which computing and IS practices function.
8. Reflection by the professional IS practitioner
When a professional IS practitioner reflects, he considers something deeply which he may otherwise
not have given much thought to. Reflection is concerned with consciously looking at and thinking
about experiences, feelings, actions and responses and then interpreting them in order to learn from
them (Boud, Keogh and Walker 1994). The professional IS practitioner does this by asking himself
questions about what he did, how he did it and what he learnt from doing it. By the IS practitioner
reflecting on his professional practice, his personal beliefs, biases and expectations are made more
evident to him. Sometimes a new heightened awareness that comes with reflection spawns new
understandings and the professional IS practitioner has an entirely new perspective. However,
caution must be exercised so as not to create ‘false’ reflection. Reflection makes the professional IS
practitioner aware of the assumptions that he may make ‘automatically’ or uncritically as a result of his
weltanschauung (worldview).
45

Udo Richard Averweg
The benefit of including reflection in the professional IS practitioner’s learning is that it allows him to
understand why he did something (e.g. a research inquiry) in a particular way, recognising how he felt
about it and thereby identifying his own personal strengths and weaknesses. Such reflection then
allows the IS practitioner to build on his personal strengths and to develop approaches to minimise his
weaknesses. The professional IS practitioner has to recognise ‘his’ IS practice as an avenue for
learning. This can be achieved by using reflection as a reflexivity tool to understand the professional
IS practitioner’s ‘true’ situation in an organisation and create the freedom he needs. Professional skills
should therefore be enhanced through a link between reflection and the IS practitioner’s praxis
(Greenwood and Lewin 2007). The professional IS practitioner develops praxis since action is
informed by reflection.
Schön (1987, 1991) proposed the use of a practicum to allow practitioners to develop in risk-free
environment “... in the varied topography of professional practice, there is a high, hard ground where
practitioners can make effective use if research-based theory and technique, and there is a swampy
lowland where situations are confusing ‘messes’ incapable of technical solution ...”. Professional IS
practitioners need to develop an ability to respond to a range of complex situations in this swampy
lowland of organisations. This will result in reflection-in-action occurring when an action has potential
for unexpected consequences. Reflection addresses the need for (professional IS) practitioners to
learn from practice experience and to use uncertainty to prompt learning (Gidman 2007: 57).
Reflection by the professional IS practitioner has many benefits in terms of promoting personal and
professional development as well as empowering him to become an effective professional IS
practitioner. In order for practitioners to become effective, it is suggested that there is a need to
facilitate the progression of professional IS practitioners from thinking to reflection, critical reflection
and reflexivity. Such progression can be facilitated by universities and universities of technology
through programs underpinned by reflective theories and which draw on theories-of-practice and
espoused theories. Professional IS practitioners will then be enabled in ways of thinking and ways of
doing that they become apprenticed into the practice of practitioner research (Hardman and Averweg
2011).
The role of reflection by the professional IS practitioner in South Africa can now be summarised.
Professional IS practitioners need to explore and be reflective as it is this habit of mind which is
indeed a useful source of professional development. However, for them to be also reflexive, supports
critical reflection. To be reflexive can nourish reflections as introspection leads to heightened
awareness, change, growth and improvement of the self and for the IS discipline.
9. Conclusion
Schön (1991) asserts that the only way to manage the “indeterminate zones of (professional)
practice” is through the ability of the practitioner to think on his feet and apply previous experiences to
new situations. The knowing is in the action. The knowledge derived is from practitioner inquiry
research and also from the professional practitioner’s own experience and reflection. Such reflection
may be in the areas of content, process and premise reflection.
Professional knowledge creation is knowledge which is derived from the systematic accumulation of
evidence in the professional IS practitioner’s workplace environment. It is developed on strong ethical
principles and from systematic forms of practitioner inquiry from the domains of reflective practice,
critical reflection and reflexivity. Engaging in practitioner inquiry research can thus contribute to the IS
practitioner’s professional knowledge, CPD and his understanding of learning theory, although it is
accepted that the links between theory and practice appear to be complex. When one critically
reflects thereon, this complexity can sometimes be mitigated by IS practitioners who are
professionally registered and who conduct computing research in the swampy lowland in
organisations.
References
Argyris, C., Putman, R. and Smith, D. M. (1985) Action Science: Concepts, Methods, and Skills for Research
and Intervention. Jossey-Bass, San Francisco.
Boud, D., Keogh, R. and Walker, D. (1994) Reflection: Turning Experience into Learning. Kogan Page,
London.
46

Udo Richard Averweg
Bullen, C. V., Abraham, T., Gallagher, K. Simon, J. C. and Zwieg, P. (2009) “IT Workforce Trends: Implications
for Curriculum and Hiring”, Communications of the Association for Information Systems, Vol 24, No. 9.
http://aisel.aisnet.org/cais/vol24/iss1/9/ (accessed on 30/9/11).
Computing Curricula 2005 “The Overview Report (CC 2005 Overview Report) 2005”. A volume of the
Computing Curricula Series, ISBN 1-59593-359-X, 30 September.
Dadds, M. (2004) “Perspectives on Practitioner Research”. National College for School Leadership, Cranfield,
Bedfordshire. Available at http://networkedlearning.ncsl.org.uk/collections/network-researchseries/reports/teacher-researcher-reports/nlg-perspectives-on-practitioner-research.pdf
(accessed on
20/1/12).
de Roche, M. (2011) Director, Laragh Skills. “Continuous Professional Development”.
http://www.ipthree.org/blog/professionalism-what-does-it-mean/260-moira-de-roche (accessed 21/9/11).
Dewey, J. (1933) How we Think. Heath and Co, Boston, MA.
Fox, M., Martin, P. and Green, G. (2008) Doing Practitioner Research. Sage, London.
Freshwater, D., Jasper, M. and Gilbert A. (2003) “Reflective Master Class”. In: Qualitative Research in Health
and Social Care Conference, Bournemouth, United Kingdom, 8-11 September.
Gidman, J. (2007) “Reflecting on reflection”. In: J Woodhouse (ed) Strategies for Healthcare Education. How
to Teach in the 21st Century. Radcliffe Publishing Ltd, Abingdon, Oxon.
Granger, M. J., Dick, G., Luftman, J., Van Slyke, C. and Watson, R. T. (2007) “Information Systems Enrolments:
Can They Be Increased?”, Communications of the Association for Information Systems, Vol 20,
No. 41, http://aisel.aisnet.org/cais/vol20/iss1/41/ (accessed on 30/9/11).
Greenwood, D. J., and Levin, M. (2007) Introduction to action research, social research for social change,
Second edition. Sage, Thousand Oaks, CA.
Hardman, S. and Averweg, U. R. (2011) “Practitioner Research from a Critical Systems Perspective”.
Alternation: Journal of the Centre for the Study of Southern African Literature and Languages,
ISSN 1023-1757, Vol 18, No. 1, pp 375-385.
McCutcheon, G. and Jung, B. (1990) “Alternative perspectives on action research”, Theory into Practice, Vol
29, No. 3, pp 144-151.
Moon, J. (1999) Reflection in Learning and Professional Development. Kogan Page, London.
Overby, S. (2006) “Staffing: How to Hook the Talent You Need”, CIO, 40-54, 1 September,
http://www.cio.com/article/24439/Staffing_How_to_Hook_the_Talent_You_Need (accessed on 30/9/11).
Pritchard, I. A. (2002) “Travelers and Trolls: Practitioner research and institutional review boards”, Educational
Researcher, Vol 31, No. 3, pp 3-13.
Raelin, J. A. (2001) “Public Reflection as the Basis of Learning”, Management Learning, Vol 32, No. 1, pp 11-
30.
Reason, P. and Bradbury, H. (2006) Handbook of action research: The concise paperback edition. Sage,
London.
Robson, C. (2002) Real World Research: A Resource for Social Scientists and Practitioner-Researchers.
Second Edition. Blackwell, Oxford.
Roode, D. (2003) “Information Systems research: A matter of choice”, South African Computer Journal,
Vol 30, pp 1-2.
Schön, D. (1987) Educating the reflective Practitioner: towards new design for teaching and learning in
the professions. Avebury Academic Publishing, Aldershot.
Schön, D. (1991) The Reflective Practitioner: how practitioners think in action. Avebury Academic
Publishing, Aldershot.
Topi, H., Valacich, J. S., Wright, R. T., Kaiser, K., Nunamaker, Jr, J. F., Sipior, J. C. and de Vreede, G. J. (2010)
“IS 2010: Curriculum Guidelines for Undergraduate Degree Programs in Information Systems”,
Communications of the Association for Information Systems, Vol 26, No. 1, pp 359-428, April.
47

A Technical Guide to Designing and Implementing
Effective web Surveys
Greg Baatard
Edith Cowan University, Perth, Australia
g.baatard@ecu.edu.au
Abstract: The Internet is becoming an increasingly prominent medium for the administration of surveys. Although
individual findings vary, the majority of the literature agrees that the appropriateness and response rates of web
surveys is expected to rise in the future as we enter a generation of “digital natives” and mail-based
communication becomes increasingly antiquated. Just about every aspect and tenet of traditional survey
methodology has received attention in academic literature, positioning it as one of the most mature data
collection techniques and a mainstay in all areas of research. While much of this accumulated knowledge is
applicable and relevant to web surveys, there are numerous issues that arise specifically when surveys are
delivered online. Such issues concern the overall design, delivery and administration of web surveys and the
structure, presentation and layout of their questions. The decisions made in these areas can influence the
efficacy of a web survey in a number of ways, including the rate, integrity and quality of responses. This paper
discusses such issues, and makes a number of recommendations to assist researchers in manually developing
an effective web survey and in evaluating survey creation products and services.
Keywords: web, online, survey, questionnaire, guide
1. Introduction
Dillman, Smyth and Christian (2009) describe the past two decades as “turbulent times” for the survey
methodology. The telephone-based surveys popular throughout the 1970s and 1980s have become
substantially less popular amongst researchers, marred by numerous factors including a surge in
telemarketing and the distasteful phenomenon of receiving calls during the “dinner hour”. While still
heavily used and indeed the most suitable medium in some demographics, mail-based surveys have
begun to lose ground with advances in technology spurring web surveys to prominence. As Internet
access and eMail become almost synonymous with the ownership of or access to a personal
computer, delivering self-administered surveys via such technologies has become an increasingly
common practice. Doing so offers the same benefits of mail surveys – allowing the researcher to
reach potential respondents all over the world with minimal cost of data collection and processing.
Furthermore, web surveys require less effort by respondents to complete and return than mail surveys
(Couper 2000; Fricker and Schonlau 2002; Lyons, Cude et al. 2005; Deutskens, de Ruyter et al.
2006; Couper and Miller 2008; Malhotra 2008; Dillman, Smyth et al. 2009; Vicente and Reis 2010).
The medium also affords greater control in terms of validation and the delivery of the survey items; for
example, conditional questions can be presented only if required (Oppenheim 1992; Couper 2000;
Fricker and Schonlau 2002; Shropshire, Hawdon et al. 2009; Vicente and Reis 2010).
The already low response rates typically seen in mail surveys are set to become lower still as more
and more communication occurs electronically, making the completion and return of a printed survey
a somewhat archaic process. Dillman, et al. (2009) echo this sentiment, stating that “the shift toward
eMail as the communication mode of choice for significant sectors of the population is somewhat
ironic as it is one of the very factors that make Internet surveys possible, but it is also making surveys
by traditional modes more difficult to complete.” Web surveys normally require no more than eMail
and/or a Web browser to complete and submit – both of which the large majority of today’s computer
users are familiar with. Furthermore, the completion of trivial online quizzes, polls and tests has
emerged as a social pastime for many Internet users, particularly those in teen and young adult
demographics (Fricker and Schonlau 2002). Although done for amusement, these activities serve to
familiarise Internet users with the mechanics of web surveys and potentially increase their willingness
to respond to other surveys.
Several studies conducted in the past decade have found that web surveys can achieve similar
response rates to mail surveys, particularly for younger respondents and those in demographics that
regularly use the Internet. Some of the studies finding response rates to mail surveys to be higher
than those of web surveys have also acknowledged the need to take demographics into account and
the likelihood of response rates differing in the future. These trends suggest that web surveys will
become increasingly prominent and result in higher response rates as the population becomes
increasingly made up of “digital natives” (Jones and Pitt 1999; Cook, Heath et al. 2000; Couper 2000;
48

Greg Baatard
Couper, Traugott et al. 2001; Fricker and Schonlau 2002; Kaplowitz, Hadlock et al. 2004; van Selm
and Jankowski 2006; Lusk, Delclos et al. 2007; Converse, Wolfe et al. 2008; Couper and Miller 2008;
Shih and Fan 2008). Indeed, the emergence of the Internet as a major medium for the delivery of selfadministered
surveys is easily observable and cannot be denied.
The issues of survey design are well established and mature in academic literature, covering topics
such as question wording, question types and survey length. While such issues are entirely applicable
and must be respected in web surveys, this paper focuses upon issues which arise specifically when
surveys are delivered online. The issues discussed can have a significant impact upon the
accessibility and usability of web surveys, potentially influencing both response rates and the quality
of responses themselves – “Before writing a questionnaire for the web, it is important to remember
that a poorly designed survey can discourage people from responding, and it can also give skewed
results” (Gonzalez-Bañales and Adam 2007). Several of the issues and recommendations are of a
technical nature, written with the assumption that interested readers will have a moderate level of
technical knowledge or at least an understanding of the technical aspects of developing and
implementing a web survey.
2. Overall design, delivery and administration issues
This section discusses the primary issues related to the overall design, delivery and administration of
web surveys as a whole. The first issue is that of the delivery and hosting of the survey. The author
recommends against utilising eMail for anything other than contacting potential respondents, despite
possibly appearing as a method of minimising the effort required to respond. While it is possible to
embed or attach a survey to an eMail, this practice is relatively uncommon and is less likely to be
supported in a uniform fashion by eMail client software (Fricker and Schonlau 2002; Lyons, Cude et
al. 2005). Thus, response rates of eMail-based surveys may be hampered by both unfamiliarity and
technical hurdles. Web surveys should be hosted on the Internet as standard Web pages. They
should minimise any reliance on supporting software or technologies such as Flash or JavaScript
(Kaczmirek 2005; Gonzalez-Bañales and Adam 2007). As discussed in the next section, if such
technologies are used they should not be relied upon and the survey should function properly without
their presence. Like all Web pages, all web surveys should be tested thoroughly before deployment to
ensure that they appear and function correctly in all likely browsers, browser versions, screen
resolutions, and so forth (Fricker and Schonlau 2002; Thompson, Surface et al. 2003; Kaczmirek
2005; Lyons, Cude et al. 2005; Gonzalez-Bañales and Adam 2007).
Access to and availability of web surveys must also be considered. In addition to adhering to
appropriate sampling procedures (Simsek and Veiga 2001; Lyons, Cude et al. 2005; van Selm and
Jankowski 2006; Couper and Miller 2008; Vicente and Reis 2010), web surveys should require some
form of unique identifier such as an ID number, code, IP address or eMail address to deter automated
responses or multiple responses by a single person (Couper, Traugott et al. 2001; Thompson,
Surface et al. 2003; Lyons, Cude et al. 2005). Such measures can increase the quality of the data
gathered by deterring behaviour which negatively influences response data. When using an ID
number or access code, Couper, et al. (2001) recommend avoiding those containing potentially
ambiguous characters such as the letter “l” and the number “1” or the letter “o” and the number “0”.
Limiting the availability of a web survey has the potential to strengthen the data set. For example, if a
survey is to be administered only to employees of a single organisation, it may be appropriate to
deploy it in such a way that it can only be accessed via the organisation’s internal network (Simsek
and Veiga 2001). Controlling and limiting the availability of a web survey also serve to prevent abuse.
Though the likelihood of this is minimal for most surveys, this is of particular importance if the survey
concerns topics considered to be controversial, sensitive or high-profile. The potential for abuse has
been illustrated several times by the abuse of online polls – a different but definitely similar scenario.
Time magazine’s 2009 “Time 100” poll was manipulated with enough sophistication to arrange the top
21 results (Schonfeld 2009). A similar hoax in 2010 targeted singer Justin Beiber; an online poll being
abused in order to add North Korea as a destination in his first world tour (Emery 2010).
It is worthwhile at this point to acknowledge that numerous applications and Web-based services
exist, which can be used to create and administer web surveys. Modern products offer high degrees
of customisability and sophistication, allowing a web survey to be created and administered with
minimal technical knowledge. While reviews or recommendations of these are not included in this
paper (interested readers see for example, Thompson, Surface et al. 2003; Wright 2005 and
http://websm.org/), the issues discussed here can serve as criteria by which to assess such
49

Greg Baatard
applications or services. In addition, one must consider factors such as the cost and the
confidentiality, security and ownership of data when determining their suitability (Lyons, Cude et al.
2005; Wright 2005; Greenlaw and Brown-Welty 2009). Despite the availability of these products,
various factors often make a purpose-built web survey a necessity.
A final consideration pertinent to the overall design of web surveys is that of the processing and
storage of responses. The ability to store response data directly in a database or spreadsheet format
is a definite advantage of administering surveys electronically, and this can be maximised with a welldesigned
web survey (Lyons, Cude et al. 2005; Gonzalez-Bañales and Adam 2007; Greenlaw and
Brown-Welty 2009). A web survey is essentially a form, and as such must be processed like any other
Web-based form once submitted. Although it is possible to create a form which simply eMails the
response to the researcher, entering the response into a database is a much more effective method.
This can be facilitated by ensuring that the names of the form elements such as text fields and radio
button groups correspond to the database column names, allowing simple and generic form
processing scripts to be created. Care should be taken to ensure that web surveys are submitted and
stored in a secure manner, utilising appropriate encryption (Lyons, Cude et al. 2005). As well as being
easily exportable to spreadsheet or statistical analysis software formats, survey responses in a
database can be queried using SQL (Structured Query Language), offering sophisticated means of
extracting and visualising meaningful information. To allow effective querying, the values submitted in
a web survey should be numeric wherever appropriate. For example, a five-point Likert scale question
ranging from Strongly Disagree to Strongly Agree and including a Neutral response should be stored
as -2 to 2 with 0 representing Neutral, rather than storing responses textually.
This section has mainly discussed issues that aim to improve the quality, correctness, usability and
reliability of data gathered via web surveys. The recommendations are overarching – benefiting the
researcher directly, rather than via the improvement of the respondent’s experience. The following
sections focus upon issues that improve the respondent’s experience.
3. Structure, presentation and question layout issues
This section discusses issues of structure, presentation and question layout within web surveys that
can influence the respondent’s experience. Regardless of the medium, a survey that is clear,
convenient and usable is of more appeal to potential respondents, resulting in higher response rates
and potentially better quality data. The author feels it is important to make all relevant information
available with the survey itself. While eMail or other methods may be used to introduce the research
and request participation, all pertinent information should also be available with the web survey. While
this may result in some redundancy, it ensures that respondents do not need to search outside of the
survey Web page for any instructions or information they may need. While minor, inconveniences
such as these have the potential to deter a respondent – particularly if they encounter the
inconvenience before beginning the survey, when they have not yet invested any time or effort into its
completion.
The issue of survey length has been addressed in numerous pieces of academic literature, often
finding that longer surveys are not only less likely to be completed, but also that “questions asked
later in the questionnaire bear the risk of producing lower quality data, especially if they are in open
format or in long grids” (Galesic and Bosnjak 2009). Based on such findings (Herzog and Bachman
1981; Gonzalez-Bañales and Adam 2007), two recommendations that are applicable to surveys in
any medium are worth repeating. Lengthy surveys should be avoided if possible, and the ordering of
questions should be considered in longer surveys – acknowledging that the quality of responses to
questions towards the end may be lower than that of those near the beginning (Vicente and Reis
2010). A further consideration regarding the ordering of questions arises from Shropshire, Hawdon
and Witte (2009), which found respondent interest to be a significant factor in early termination of a
questionnaire. Hence, questions likely to be of highest interest to respondents should appear early in
the survey. Two further recommendations can be made for lengthy web surveys in particular. Firstly,
the design and presentation of the survey can be tailored to make a long survey more manageable for
respondents. One method is to divide the survey into sections and display one section at a time
(Figure 1).
50

Greg Baatard
Figure 1: Example of a long survey divided into sections
This serves to avoid overwhelming the respondent with a large number of questions on one page and
the need to scroll, which have been found to influence item non-response and survey non-completion
rates (Couper, Traugott et al. 2001; Schonlau, Fricker et al. 2002; Toepoel, Das et al. 2008; Vicente
and Reis 2010). Research (see for example, Tourangeau, Couper et al. 2004; Dillman, Smyth et al.
2009; Toepoel, Das et al. 2009) has found that respondents see questions on the same page as
being more closely related, having a slight impact upon the intercorrelation of responses. This should
be taken into account when determine if and how to divide a survey into multiple pages. An overall
progress bar and section-by-section validation are advisable if a survey is divided into sections. Yan,
Conrad, Tourangeau and Couper (2010) examine the use of a progress indicator in web surveys,
concluding that it encourages completion if it accurately reflects the expected duration of the survey,
particularly if the survey is short. This further emphasises the need to minimise the length of web
surveys and to provide information such as its expected duration. Other research regarding the use of
progress indicators in surveys divided into sections is generally supportive, however findings vary and
the effects of progress indicators are often minor (Couper, Traugott et al. 2001; Crawford, Couper et
al. 2001; Vicente and Reis 2010).
The second recommendation specific to lengthy web surveys is giving respondents the ability to save
their progress and resume the survey at a later time. This is recommended in Kaczmirek (2005), who
states “Do not introduce problems in your online questionnaire which would not occur in a paper and
pencil questionnaire.” While implementing the ability to save and resume survey progress is likely to
be beyond the technical capabilities of a person otherwise capable of creating a web survey, the
feature is common amongst applications and Web-based services that allow surveys to be created.
Unlike mail surveys, web surveys can adapt to a respondent’s answers to questions on the fly. In a
mail survey, conditional questions are typically implemented via instructions to the respondent – for
example, “If you answered ‘No’ to this question, continue to Question 12”. Using Web-based scripting
languages such as JavaScript, web surveys can implement conditional questions in a more effective
manner, showing or hiding parts of the survey based on the answers to questions (Oppenheim 1992;
Fricker and Schonlau 2002; Gonzalez-Bañales and Adam 2007). In addition to helping to ensure the
correctness of data, this serves to “reduce the length of a survey to the individual minimum”
(Kaczmirek 2005). Though JavaScript can improve the implementation of web surveys, it is
inadvisable to rely upon it heavily. Ideally it should be possible and convenient to complete the survey
without requiring the respondent’s browser to have JavaScript enabled. If the survey cannot be
completed without JavaScript, potential respondents with JavaScript disabled should be informed of
its need when they attempt to access it.
The following recommendations concern the presentation, layout, accessibility and formatting of web
survey questions. While some of the recommendations are relatively minor, they are all worthy of
consideration when creating a web survey or deciding which survey-generation product to use.
Research such as that of Christian, Dillman and Smyth (2007) underpin the potential effects of
seemingly minor issues in web survey presentation, finding that the relative size of text fields resulted
in an eight percent increase in correctly formatted responses and the use of symbols rather than
words as labels resulted in a seven percent increase. In order to avoid excess technical detail, the
recommendations are presented in dot-point form. The term “form elements” refers to text fields, radio
51

Greg Baatard
buttons, checkboxes and other form components used to allow people to respond to questions in web
surveys.
Make use of space to clearly separate questions from one another. If utilising a grid-based layout,
often used to present series of Likert scale questions, use alternating background colours to
clearly align question text with the appropriate form elements.
Use spacing to ensure that radio buttons and checkboxes are clearly associated with their labels,
preventing ambiguity and confusion. Furthermore, use the “label” tag to increase the clickable
region of form elements. These points are illustrated in Figure 2.
Figure 2: Likert scale with ambiguous spacing and no labels (A) and Likert scale with clear spacing
and labels (B). Clickable regions have been highlighted
If there is insufficient space to repeat labels next to each radio button or checkbox, consider using
the “title” attribute in the “” or “” tag. The content of this attribute will appear as a
tooltip when the mouse is over the radio button, allowing respondents to select a response
without needing to refer back to other labels.
The layout of Likert scale responses is explored in Tourangeau, Couper and Conrad (2004), who
find that respondents make inferences about the meaning of survey items based on visual cues
such as the perceived midpoint of responses, their spacing and order. These findings should be
taken into consideration when designing Likert scales in web surveys.
If a question requires participants to select from a number of options, use radio buttons rather
than drop down lists as radio buttons present the options in a more convenient and accessible
manner. Healey (2007) found that drop-down lists result in higher item non-response rates, longer
response times and increased accidental changing of responses. Kaczmirek (2005) recommends
only using drop-down lists “if respondents know the answer without having to look at all entries”,
and to make the default selection is a placeholder such as “select here” so that non-response can
be identified.
Users can navigate form elements via the tab key, with each press bringing the focus to the next
element from top-left to bottom-right. The layout of form elements in a survey may cause the
default tab order to traverse form elements in an incorrect or unintuitive manner. If this is the
case, the ordering can be changed via the “tabindex” in form element tags. The “accesskey”
attribute can also be used to allow keyboard shortcuts to form elements if deemed necessary for
accessibility reasons.
On the spot validation of responses is possible in web surveys, via scripting languages such as
JavaScript. While it is possible to enforce strict validation criteria such as only allowing digits to
appear in a text field for a phone number, it is not always advisable to do so. When faced with
overly strict, inappropriate or frustrating validation requirements, respondents may invent data,
attempt to circumvent the validation, or discontinue the survey entirely (Best and Krueger 2004;
Kaczmirek 2005; Christian, Dillman et al. 2007; Gonzalez-Bañales and Adam 2007; Vicente and
Reis 2010). It is also worthwhile reiterating that reliance upon JavaScript is not recommended.
Failing to take heed of issues such as these can lead to web surveys which are unclear, confusing
and frustrating, often resulting in discontinuation or by respondent, or lower quality data. By taking
these recommendations into consideration, researchers can design, develop and implement an
effective web survey, or select a product which will allow them to create one.
4. Conclusion
There is much academic literature which addresses the design and implementation of effective
surveys. Of that, a relatively small proportion discusses issues specific to the increasingly prominent
medium of Internet-based surveys. Web surveys present unique opportunities and challenges not
applicable to mail-based surveys, and care must be taken to ensure that web surveys are developed
and implemented in an effective manner. This paper has discussed numerous issues, primarily
technical, that should be taken into consideration in regards to web surveys.
52

Greg Baatard
Issues pertaining to the overall design, delivery and administration of web surveys were covered first.
EMail-based surveys are advised against, and the importance of appropriate advertising, unique
identifiers and data format were discussed. The recommendations in this section seek to give
researchers overarching guidance towards creating web surveys that are more likely to produce data
of high quality, correctness, usability and reliability. Next, a number of recommendations were made
regarding the structure, presentation and layout of questions in web surveys. These focus upon
increasing the clarity, convenience and usability of web surveys, promoting higher response rates and
higher quality data.
As the world’s population becomes increasingly made up of “digital natives”, the Internet has emerged
as a key medium for self-administered surveys. The sophistication and availability of modern
applications and Web-based services for the creation of web surveys has also made the medium
more accessible to researchers in all domains, furthering their use. The recommendations made in
this paper assist researchers both in manually developing a web survey and evaluating survey
creation products. A well-developed web survey provides both the researcher and respondents with a
clear, accessible and highly usable mechanism, minimising the confusing and frustrating elements
that may lower response rates or the quality of data.
References
Best, S. J., and Krueger, B. (2004). Internet Data Collection. Thousand Oaks, CA, SAGE Publications.
Christian, L. M., Dillman, D. A., and Smyth, J. D. (2007). “Helping Respondents Get It Right the First Time: The
Influence of Words, Symbols, and Graphics in Web Surveys” Public Opinion Quarterly, Vol 71, No. 1, pp
113-125.
Converse, P. D., Wolfe, E. W., Huang, X., and Oswald, F. L. (2008). “Response Rates for Mixed-Mode Surveys
Using Mail and EMail/Web.” American Journal of Evaluation, Vol 29, No. 1, pp 99-107.
Cook, C., Heath, F., and Thompson, R. L. (2000). “A Meta-Analysis of Response Rates in Web- or Internet-
Based Surveys.” Educational and Psychological Measurement, Vol 60, No. 6, pp 821-836.
Couper, M. P. (2000). “Web surveys: A review of issues and approaches.” Public Opinion Quarterly, Vol 64, No.
4, pp 464-494.
Couper, M. P., and Miller, P. V. (2008). “Web Survey Methods: Introduction.” Public Opinion Quarterly, Vol 72,
No. 5, pp 831-835.
Couper, M. P., Traugott, M. W., and Lamias, M. J. (2001). “Web Survey Design and Administration.” Public
Opinion Quarterly, Vol 65, No. 2, pp 230-253.
Crawford, S. D., Couper, M. P., and Lamias, M. J. (2001). “Web surveys: Perception of burden.” Social Science
Computer Review, Vol 19, No. 2, pp 146-162.
Deutskens, E., de Ruyter, K., and Wetzels, M. (2006). “An Assessment of Equivalence Between Online and Mail
Surveys in Service Research.” Journal of Service Research, Vol 8, No. 4, pp 346-355.
Dillman, D. A., Smyth, J. D., and Christian, L. M. (2009). Internet, Mail, and Mixed-Mode Surveys: The Tailored
Design Method. Hoboken, New Jersey, John Wiley & Sons, Inc.
Emery, D. (2010). “Prank leaves Justin Bieber facing tour of North Korea.” BBC News Retrieved 15/12/11, from
http://www.bbc.co.uk/news/10506482.
Fricker, R. D., and Schonlau, M. (2002). “Advantages and Disadvantages of Internet Research Surveys:
Evidence from the Literature.” Field Methods, Vol 14, No. 4, pp 347-367.
Galesic, M., and Bosnjak, M. (2009). “Effects of Questionnaire Length on Participation and Indicators of
Response Quality in a Web Survey.” Public Opinion Quarterly, Vol 73, No. 2, pp 349-360.
Gonzalez-Bañales, D. L., and Adam, M. R. (2007). Web Survey Design and Implementation: Best Practices for
Empirical Research. European and Mediterranean Conference on Information Systems, Valencia, Spain.
Greenlaw, C., and Brown-Welty, S. (2009). “A Comparison of Web-Based and Paper-Based Survey Methods:
Testing Assumptions of Survey Mode and Response Cost.” Evaluation Review, Vol 33, No. 5, pp 464-480.
Healey, B. (2007). “Drop Downs and Scroll Mice: The Effect of Response Option Format and Input Mechanism
Employed on Data Quality in Web Surveys.” Social Science Computer Review, Vol 25, No. 1, pp 111-128.
Herzog, A. R., and Bachman, J. G. (1981). “Effects of Questionnaire Length on Response Quality.” Public
Opinion Quarterly, Vol 45, No. 4, pp 549-559.
Jones, R., and Pitt, N. (1999). “Health surveys in the workplace: comparison of postal, email and World Wide
Web methods.” Occupational Medicine, Vol 49, No. 8, pp 556-558.
Kaczmirek, L. (2005). Web Surveys. A Brief Guide on Usability and Implementation Issues. Usability
Professionals 2005. M. Hassenzahl and M. Peissner. Stuttgart, Usability Professionals' Association
(German Chapter): 102-105.
Kaplowitz, M. D., Hadlock, T. D., and Levine, R. (2004). “A Comparison of Web and Mail Survey Response
Rates.” Public Opinion Quarterly, Vol 68, No. 1, pp 94-101.
Lusk, C., Delclos, G. L., Burau, K., Drawhorn, D. D., and Aday, L. A. (2007). “Mail versus Internet surveys:
Determinants of method of response preferences among health professionals.” Evaluation & the Health
Professions, Vol 30, No. 2, pp 186-201.
53

Greg Baatard
Lyons, A. C., Cude, B., Lawrence, F. C., and Gutter, M. (2005). “Conducting Research Online: Challenges Facing
Researchers in Family and Consumer Sciences.” Family and Consumer Sciences Research Journal, Vol
33, No. 4, pp 341-356.
Malhotra, N. (2008). “Completion Time and Response Order Effects in Web Surveys.” Public Opinion Quarterly,
Vol 72, No. 5, pp 914-934.
Oppenheim, A. N. (1992). Questionnaire design, interviewing, and attitude measurement. New York, Pinter
Publishers.
Schonfeld, E. (2009). “4Chan Takes Over The Time 100.” The Washington Post Retrieved 15/12/11, from
http://www.washingtonpost.com/wp-dyn/content/article/2009/04/21/AR2009042101864.html.
Schonlau, M., Fricker, R. D., and Elliott, M. N. (2002). Conducting research surveys via eMail and the Web.
Santa Monica, CA, Rand Corporation.
Shih, T.-H., and Fan, X. (2008). “Comparing Response Rates from Web and Mail Surveys: A Meta-Analysis.”
Field Methods, Vol 20, No. 3, pp 249-271.
Shropshire, K. O., Hawdon, J. E., and Witte, J. C. (2009). “Web Survey Design: Balancing Measurement,
Response, and Topical Interest.” Sociological Methods Research, Vol 37, No. 3, pp 344-370.
Simsek, Z., and Veiga, J. F. (2001). “A Primer on Internet Organizational Surveys.” Organizational Research
Methods, Vol 4, No. 3, pp 218-235.
Thompson, L. F., Surface, E. A., Martin, D. L., and Sanders, M. G. (2003). “From Paper to Pixels: Moving
Personnel Surveys to the Web.” Personnel Psychology, Vol 56, No. 1, pp 197–227.
Toepoel, V., Das, M., and van Soest, A. (2008). “Effects of Design in Web Surveys: Comparing Trained and
Fresh Respondents.” Public Opinion Quarterly, Vol 72, No. 5, pp 985-1007.
Toepoel, V., Das, M., and van Soest, A. (2009). “Design of Web Questionnaires: The Effects of the Number of
Items per Screen “ Field Methods, Vol 21, No. 2, pp 200-213.
Tourangeau, R., Couper, M. P., and Conrad, F. G. (2004). “Spacing, Position, and Order. Interpretive Heuristics
for Visual Features of Survey Questions.” Public Opinion Quarterly, Vol 68, No. 3, pp 368-393.
van Selm, M., and Jankowski, N. W. (2006). “Conducting Online Surveys.” Quality & Quantity, Vol 40, No. 3, pp
435-456.
Vicente, P., and Reis, E. (2010). “Using Questionnaire Design to Fight Nonresponse Bias in Web Surveys.”
Social Science Computer Review, Vol 28, No. 2, pp 251-267.
Wright, K. B. (2005). “Researching Internet-Based Populations: Advantages and Disadvantages of Online Survey
Research, Online Questionnaire Authoring Software Packages, and Web Survey Services “ Journal of
Computer-Mediated Communication, Vol 10, No. 3, article 11.
Yan, T., Conrad, F. G., Tourangeau, R., and Couper, M. P. (2010). “Should I Stay or Should I go: The Effects of
Progress Feedback, Promised Task Duration, and Length of Questionnaire on Completing Web Surveys.”
International Journal of Public Opinion Research.
54

Utilising a Sociological Analytical Scheme and Intellectual
Device to Critically Appreciate Software Project
Management Methodologies
Gary Bell
London South Bank University, Faculty of Business, London, UK
bellgaa@sbu.ac.uk
Abstract: This paper offers a sociological analytical scheme and intellectual device as a broad approach to
benchmarking software project management methodologies. The software industry is still bedevilled by problems
(or symptoms) such as cost overruns, schedule slippage and poor product quality. The combination of these
symptoms has been described as the ‘software crisis’. Project management and associative methodologies are
considered to be one of root causes of this crisis. Hence, there is a need to benchmark methodologies connected
with the software project management. Furthermore, it is asserted that the project management discipline should
be associated with Social Sciences. An analytical scheme is derived in order to highlight appropriate social theory
and clarify scientific philosophical assumptions of the identified management methodologies. Key parts of the
intellectual device are described. Additionally, its connectivity with scientific philosophy and facets of the single
and double learning concept are explored. The device is used to guide the critical appreciation process of an
improvement methodology known as the Goal/Question/Metric (GQM) Paradigm. The appreciation aspect
examines the original methodology and important developments which assists in uncovering key characteristics
(e.g. concepts and principles). Furthermore, the research strategy associated with GQM is identified. The critical
aspect investigates the benefit and cost of implementing this methodology. This novel combinational approach
comprehensively benchmarks the GQM Paradigm, and generates its epistemological framework. Moreover, it
can guide informed changes to this traditional improvement methodology. Finally, future research is outlined.
Keywords: intellectual device, methodology, sociological analytical scheme, and software project management
1. Introduction
Bell et al (2002) believe that improving and controlling are the two key activities associated with the
management of software projects. Thus, Software Project Management can be split into two important
study areas, namely Software Process Improvement (SPI) and Software Process Control (SPC)
Research. The former area is underpinned by the view that all issues of quality and productivity
revolve around continuously improving the software product and software development approach.
The latter area is underwritten by the notion that a disciplined approach to software production is
required in order to achieve sensible cost, schedule and quality targets. There is a raft of
methodologies, techniques, and methods linked with SPI research and SPC research. However, the
symptoms of the ‘software crisis’ still persist. Hence, the need to benchmark established management
methodologies. Fuggetta (1999) believes researchers should pursue a deeper understanding of
issues and offered solutions within this discipline.
Bell et al (2009) contend that the Project Management discipline is elevated to the complexity
hierarchical level which is addressed by the Social Sciences. Burrell and Morgan’s (1979) paradigm
map is a way of negotiating through the different types of social theory and organisational analysis.
The map is a useful heuristic device for establishing where you are, where you have been and where
it is possible to go in the future. Moreover, it can be employed to explore a wide variety of disciplines
related to social sciences. This initiates the challenge to explore the connectivity between social
theories and methodologies associated with software project management. This helps with
highlighting the social reality implied and clarify the social/scientific philosophical assumptions of a
management approach. The philosophy of science can be delineated into four assumptions related to
ontology, epistemology and axiology and methodology. An overview of each theoretical assumption is
given. Additionally, the extreme subjective/objective attributes of each assumption are highlighted.
This work overarches the ‘development and the nature of knowledge’ generated by software project
management methodologies and respective research strategies.
The offered intellectual device, which is grounded in scientific philosophy, is considered a broad
approach to studying a methodology which is applied in practice. The origins of the device (Checkland
and Holwell 1998) which had three key parts, namely: Framework of Ideas, Methodology and Area of
Concern are discussed. Three more parts are proposed labelled Organised Learning, Cost Benefit
Appraisal and Knowledge Library and their importance are substantiated. It is argued that the device
is underwritten by the single and double loop learning concepts (Argyris and Schon 1978). The GQM
55

Gary Bell
Paradigm is selected, and was developed and applied at NASA/GSFC over a period of 25 years. The
documented library of published materials by Victor Basili was useful source of data for the purpose of
this research. The intellectual device is used to guide the critical appreciation of the ‘in practice’ GQM
Paradigm. Key findings from this work are highlighted. Additionally, different uses of the analytical
scheme and intellectual device are explored, and their potential contributions to both methodological
pluralism and the Project Management discipline.
2. Social sciences
Burrell and Morgan’s intellectual map has two principal dimensions: Philosophy of Science and
Theory of Society (see table 1.0). When the two principal dimensions are cross linked four distinctive
paradigms emerge. These are labelled Functionalist, Interpretive, Radical Humanist and Radical
Structuralist. Most software project management methodologies are linked to Sociology of Regulation
– thus we are interested in the two former paradigms. This has led to the inception of the sociological
analytical scheme (see figure 1).
Solipsism
Phenomenology
Burrell and
Morgan (1979)
Nominalism
Anti-positivism
Value-full
Ideographic
The Sociology of Regulation
The Interpretive Paradigm The Functionalist Paradigm
Phenomenalogical
Sociology
Hermeneutics
Interactionism and
Social Systems
Social Action
Theory
Theory Integrative Theory
A Scheme for Analysing Assumptions about the nature of Social Science
Ontology
Epistemology
Axiology
Methodology
Figure 1: Sociological analytical scheme
Table 1: The theory of society (regulation-radical change) dimension
GQM Paradigm
(Basili and Weiss, 1984)
Objectivism
Realism
Positivism
Value-free
Nomothetic
The Sociology of Regulation is concerned The Sociology of Radical Change is
with:
concerned with:
(a) The status quo (a) Radical change
(b) Social order (b) Structural conflict
(c) Consensus (c) Modes of domination
(d) Social integration and cohesion (d) Contradiction
(e) Solidarity (e) Emancipation
(f) Need satisfaction (f) Deprivation
(g) Actuality (g) Potentiality
2.1 Sociological ‘regulation’ paradigms
Functionalist and Interpretive paradigms are connected with the sociology of ‘regulation’ dimension.
These paradigms are defined by the theoretical assumptions which underpin the frame of reference,
mode of theorising and modus operandi of social theorists. The Functionalist paradigm (see table 2.0)
is the dominant framework of reference within sociology. It emphasises the essentially objectivist
nature of the social world and is concerned with explanation that stresses ‘regulation’ in social affairs.
It is problem - orientated, and associated with producing solutions to practical problems. Moreover, it
is committed to a philosophy of social engineering as a basis of change and stressing the importance
of understanding order, equilibrium and stability within a social situation. Burrell and Morgan identify
four broad categories of thought, and labelling them: Objectivism, Social System Theory, Integrative
Theory, and Interactionism and Social Action Theory (see table 2).
56

Gary Bell
Table 2: Four broad categories of thought found within functionalist paradigm
Category of
Thought
Objectivism
Social System
Theory
Integrative
theory
Interactionism
and Social
Action Theory
Key
characteristic
s
• Firmly linked with the
traditions of sociological
positivism. C ommitted to
the natural science
research method.
• Objectivists
conceptualise people as
passive responders,
machine - like in their
response to stimuli .
• Represents a direct
development of
sociological positivism in
its most pure form.
•It generally adopts the
biological analogy for
investigating social
situations.
• Integrative theory
combines various aspects
of interactionism and
social systems theory
•A category of thought
which combines aspects
of Sociological Positivism
and German Idealism
Functionalism
Schools of Key characteristics
Thought
Behaviourism • Skinner attempted to develop casual theories of behaviour based upon
analysis of stimulus and response.
• Individuals are regarded little more than machines, responding in a
deterministic way to the external conditions.
• Behaviourism denies that consciousness has any relevance to the
understanding of human behaviour.
• Usually conducts laboratory experiments i.e. to isolate the problem from
the environment.
• The researcher observes the situation.
Abstrac t
Empiricism
Struc tural
Functionalism
•A highly nomothetic methodology is used to test a theory which is based
upon an ontology, an epistemology and a theory of human nature of a
more subjectivist kind.
• Structural functionalism employs the biological analogy in the tradition of
Comte, Spencer and Durkheim.
• Building upon the concepts of holism, interrelationships between parts,
structure and functions, the biological analogy has developed in many
diverse ways to produce an alternative social science perspective and
firmly rooted in the sociological of regulation.
• Two names stand out as particular influential in this endeavour –
Malinowski and Radcliffe-Brown.
• Malinowski establishes the importance of fieldwork when conducting
social research.
• Primitive social systems could be understood in terms of the functions
which they performed. He view of was that society or ‘culture’ should be
regarded as a complex whole and understood in terms of relationships
between various parts and their surroundings.
• Radcliffe-Brown argued that the ongoing life of a society could be
conceived in terms of the functioning its structure – hence the notion of
‘structural functionalism’.
• He saw social institutions as contributing to the ‘integration’, ‘stability’ and
‘maintenance’ of the social system as a whole.
Sy stems Theory • The ‘systems approach’ has assumed increasingly importance in various
disciplines within social science. Parsons, Homans, Katz and Kahn,
Easton and Buckley are key academics.
• Bertalanffy aim can be viewed as archetypical of the positivist
perspective: it is based upon epistemological assumptions dominated by
concerns to search for explaining the underlying regularities and structural
uniformities that characterise the world.
• Firmly rooted against reductionism which underpins and dominates most
social and natural scientific research.
• Bertalanffy believes that General Systems Theory is an alternative to
reductionism. Instead of reducing all phenomena of study to their
elements, he advocates that researchers study them as systems.
• Bertalanffy positivism is of a non-traditional kind and views a system is
an organising concept.
• Open systems exchange with their environment.
Morphogenic
Sy stems Theory
• Buckley introduces systems theory to social science through the use of
cybernetics, information and communication theory and general systems
res earch.
• Buckley‘s morphogenic view of society moves him away from the
majority of social systems theorists, for he sees social structure as
emerging from the process of social interaction.
• The researcher may need to be closer to the situation in order to enable
the identification of relevant information used by the stakeholders.
Interactionis m •Simmel drove a middle way between idealism and positivism, retaining
only those aspects that lent themselves to his own particular needs.
• rejected extremes of both positions in favour of analysis of human
association and interaction.
• Basic patterns can be found beneath social situations and extracted
through formal analysis.
• Mead asserts the conscious person evolves through a social process,
which involves the development of language and hence mind and self.
Social Action
Theory
• Derived largely from the work of Weber who attempted to bridge the gap
between German Idealism and Sociological Positivism.
• Weber believes that the task of sociology was to analyse and explain
social action by investigating subjective meanings by which individuals
direct their action.
• Weber attempted to build an objective science of sociology based upon
the foundations of subjective meaning and individual action. This is
achieved through the method of verstehen, which places the researcher in
the role of the individual to enable understanding and interpretation of
meaningful action.
The Interpretive paradigm is an alternative framework of reference within Sociological of Regulation,
which offers a subjectivist approach to the social world. It encompasses a wide range of sociological
thought that is concerned with understanding the world as it is, and at the level subjective experience.
Moreover, it seeks explanation within the realm of individual consciousness and subjectivity, and
takes the stance of participant as opposed to the observer of action. It views the social world as an
emerging social process created by the relevant individuals. Social reality, with regards to having
existence outside the consciousness of an individual, is considered as being little more than a
network of assumptions and inter-subjectivity shared meanings. Interpretive philosophers and
sociologists attempt to understand the essence and source of social reality. Furthermore, they
research human consciousness and subjectivity in order to gain the fundamental meanings that
underwrite social situations. Burrell and Morgan have identified four broad categories of thought and
labelled them: Hermeneutics, Phenomenological Sociology, Phenomenology and Solipsism (see table
3.0).
57

Gary Bell
Table 3: Four broad categories of thought found within interpretivism
Category of
thought
Hermeneutics
Phenomenological
Sociology
Phenomenology
Interpretivism
Key Characteristics Schools of
thought
• Dilthey developed a methodology for
study ing t he objectif ications of mind. The
approach is to generate objectively valid
knowledge in the Geisteswissenschaften
through the method of verstehen (means
by which researchers comprehend the
meaning of a social situation or cultural
artefact).
• One of the main avenues for verstehen
was through the study of empirical life
assertions – institutions, historical
situations, languages – which reflects the
inner life of their creators. The
investigation of these social creations was
viewed as the main avenue to
understanding of the world of objective
mind.
• Concerned with the fundamentals of
ontology.
• It studies essences and clarifies the
relationship between them; it seeks to
delve into experiences and clarifies the
very ground of knowledge.
• in this endeavour the methods of ‘direct
intuition’ and ‘insight into essential
structures’ are offered as the principal
means of penetrating the depths of
consciousness and transcending the world
of everyday affairs in search of subjectivity.
Solipsism • Solipsism denies that the world has any
distinct independent reality. The world is a
creation of an individual mind.
3. Philosophy of science
Key characteristics
Ethnomethodology • Ethnomethodology is concerned with identifying the
assumptions that characterise any social situation and
the ways in which the members involved, through the use
of everyday practices, make their activities’ rationally
accountable’.
• A social situation is viewed as a process of accountable
action which is sustained by the effort of the participants;
the participants are seen as attempting to order
experience so as to sustain the everyday, common-sense
suppositions which characterise the routine of everyday
life.
Symbolic
Interactionism
Transcendental
Phenomenology
Existential
Phenomenology
• Emphasis upon the emergent properties of interaction,
through which individuals create their social world rather
than merely reacting to it.
•Meaning is attributed to the environment, not derived
from and imposed upon individual actors; action is built
up ins tead of being a response or mere realise
mechanism
• interactionists treats the point-of-view of the ac tor as
only one aspect of the problem of order, seeking to reality
it to a wider context in terms of scientifically validation
description and explanations.
•To study the way social reality reflects a balance of
intersubjectivity shared meanings, which are continually
negogaited, sustained and changed through the everyday
interactions of individuals. Social reality is for them either
reaffirmed or created afresh in every social encounter.
• Husserl was concerned about the foundations of logic
and science. An early observation was that science was
characterised by ‘intentionality’. Despite the fact that the
results of science were imperfect, it was guided by the
intention of absolute objectivity.
• Rather than bother with factual realities or the
formulation of hypothesis, Husserl addresses the central
question of meaning. He put reality aside and sought to
penetrate the level of the phenomenon. Hence, he
sought to practise phenomenology.
• Transcendental philosophy aims to grasp ‘the world as
phenomenon’ – to grasp it not as an object, but as pure
meaning. The purpose is to ‘bring to light’ the essential
intentional contact between consciousness and the world.
• All assumptions of everyday life are brushed aside in
pursuit of pure subject ivity, of transcendental
consciousness, the intentionality of which is the sources
of all meaning.
• A common concern for what is known as ‘life-world’, for
the world of everyday experience as opposed to the
realm of transcendental consciousness. Schutz can be
characterised as a sustained effort to relate the idea of
phenomenology to the problems of sociology.
• Schutz embarks upon a phenomenological analysis of
meaning, searching for origins in the ‘stream of
consciousness’. It is argued that consciousness is an
unbroken stream of lived experiences which have no
meaning in themselves.
• Meaning is dependent upon reflexivity – the process of
turning back on oneself and looking at what has been
going on. Meaning is attached to actions retrospectively;
only the already-experienced is meaningful, not that
which is in the process of being experienced.
• The process of attributing meaning reflexively is
dependent upon the actor’s identifying the goal which the
individual is supposedly seeking.
• This introduces the notion of being able to attribute
meaning, in advanc e, to future experienc es . The concept
of meaningful action thus contains elements of both past
and anticipated future; intrinsically has a temporal
dimension.
Burrell and Morgan (1979), Mingers (2003) and Saunders et al (2009) have examined the philosophy
of science dimension respectively within Social Sciences, Management Science and Business
Research. It is explored through four theoretical assumptions namely: ontology, epistemology,
axiology and methodology (see figure 1.0). An overview of each assumption is given. Moreover, the
extreme subjective/objective attributes of each key assumption are highlighted.
Ontology is concerned with the study of existence itself, and differentiates between ‘real existence’
and ‘appearance’. Burrell and Morgan highlights the nominalist and realist position (see table 4).
Epistemology addresses issues related to the nature and derivation of knowledge, the scope of
knowledge, and the reliability claims of knowledge. Critically, philosophers, e.g. empiricists and
idealists, are divided over how knowledge is generated. Burrell and Morgan explore the assumptions
about the grounds of knowledge – how the world can be understood, and the communication of this
58

Gary Bell
knowledge. This presupposes specific anti-positivist and positivist epistemological stances.
Furthermore, these epistemological assumptions determine extreme points – on the issue whether
knowledge can be acquired or is something that has to be personally experienced.
Axiology 1 is the study of values in the fields of ethics and aesthetics. The process of social enquiry
and the role of values to ensure credible research results are of interest. Critically, Ackoff (1979), a
leading Management Scientist, divides the notion of objectivity into ‘value-free’ and ‘value-full’ which
are linked respectively to traditional and alternative Operational Research. Hence, these extreme
subjective and objective value positions for axiology are offered. Importantly, It is recognise that
method and methodology can have different meanings. Therefore, in this paper, method is defined as
‘a systematic way of doing a thing’, and methodology as ‘a body of methods used in a particular
activity, e.g. improving’. The identified three sets of assumptions have methodological implications.
Each assumption has consequences that impact upon investigations and attaining ‘knowledge’ about
the social phenomenon. Different ontologies, epistemologies and axiologies influence scientists
towards the selection of appropriate methodologies. Burrell and Morgan expand upon the ideographic
and nomothetic methodological dichotomy. These scientific philosophical assumptions are believed to
impact upon the offered intellectual device.
Table 4: The philosophy of science (subjective-objective) dimension
The Subjectivist Approach to Social Theoretical The Objectivist Approach to Social
Science
Assumptions
Science
Nominalism Ontology Realism
- the nominalist position revolves around the
- realism postulates that the social world external to
assumption that the social world external to
individual cognition is a real world made up of hard,
individual cognition is made up of nothing more
tangible and relatively immutable structures.
than names, concepts and labels which are used
- whether or not we label and perceive these structure,
to structure reality.
the realists maintain, they still exist as empirical
- the nomalist does not admit to there being any
entities.
‘real’ structure to the world which these concepts
- the social world exist independently of an individual’s
describe.
appreciation of it. The individual is seen to as being
- the ‘names’ used are regarded as artificial
born into and living within a social world which has a
creations whose utility is based upon their
reality of its own.
convenience as tools for describing, making sense
- It is not something which the individual creates – it
of and negotiating the external world.
exists ‘out there’; ontologically it prior to the existence
and consciousness of any single human being. For the
realist, the social world has an existence which is as
hard and concrete as the natural world.
Anti-positivism Epistemology Positivism
- Firmly against the utility of a research for laws or
- seeks to explain and predict what happens in the
underlying regularities in the social world.
social world by searching for regularities and causal
- For anti-positivists, the social world is essentially
relationships between its constituent elements.
relativistic and can only be understood from the
- positivist epistemology is essentially based upon the
point of view of the individuals who are directly
traditional approaches which dominate natural science.
involved in the activities which are to be studied.
- They maintain that one can only ‘understand’ by
occupying the frame of reference of the
participants in action. A researcher has to
understand from the inside rather than the
outside.
- from this point of view social science is seen as
being essentially a tend to reject the notion that
science can generate objective knowledge.
- positivists may differ in terms of detailed approach.
Value-full Axiology Value-free
- intention to attain the objectivity notion ‘value-full’
- intention to attain the objectivity notion ‘value-free’
-role of the modeller – facilitator.
- role of the modeller – observer.
- role of the stakeholder – active participant.
- role of the stakeholder – passive participant.
- individuals are autonomous and free willed.
- individuals and their activities are completely
determined by the situation or ‘environment’ in which
he is located.
Ideographic Methodology Nomothetic
- the view that understanding the social world by
- emphasises the importance of basing research upon
obtaining first-hand knowledge of the subject
systematic protocol and technique. It is epitomised in
under investigation. It stress getting close to the
the approach and methods employed in natural
subject and exploring its detailed background and
science, which focus upon the process of testing
life history.
hypotheses in accordance with the canons of scientific
- the ideographic approach emphasises the
rigour.
analysis of the subjective accounts which one
- it is preoccupied with the construction of scientific
generates by ‘getting inside’ situations.
tests and the use of quantitative techniques for data
– the detailed analysis of the insights generated
analysis. Survey and questionnaires are standard
by such encounters with the subject and the
research instruments that comprise nomothethetic
insights revealed in accounts found in written
documents.
- the ideographic method stresses the importance
of letting the subject unfold its nature and
characteristics during the process of investigation.
methodology.
1
Burrell and Morgan identified Human Nature as an assumption of scientific philosophy. This has been substituted by axiology
which is viewed as a broader theoretical assumption.
59

4. Intellectual device
Gary Bell
An intellectual device, originally constructed by Checkland and Holwell (1998) (see figure 2 - the
italics are our proposed enhancements), as a broad approach to methodological investigations. The
original device entails three basic parts that are applied to any piece of research. These are:
Framework of ideas (F), Methodology (M) and Area of Concern (A). Three more parts are proposed
and labelled Organised Learning (OL),Cost Benefit Appraisal (CBA) and Knowledge Library (KL)
which closes the research loop and facilitates learning through reflection. Analysis of the parts has
identified some key elements, which provides further definition of the intellectual device.
A set of ideas and concepts (F) are used in a Methodology (M) which is applied to investigate some
Area of concern (A) (or phenomenon). A methodology has a set of principles associated with a group
of methods used to focus on the identified activity, e.g. improving or controlling. Additionally, a
process model is employed to structure or mould the intervention. Every methodology evokes
distinctive roles for the modeller, e.g. facilitator, and stakeholders, e.g. active, in the intervention
process. The methodology may yield abstract models (e.g. for understanding or explaining) of specific
entities of the identified phenomenon. The former three parts of the device are considered as the
appreciation of the identified approach.
Framework of ideas
and concepts (F)
Epistemology
Modification to
(the development of knowledge) F,M
(‘the how’)
Published Findings
Embodied in
Axiology
(the value of knowledge)
(‘the why’)
Methodology (M) Area of Concern (A)
(Principles,
Methods.
Process models,
roles)
Contributes
to
Exploring of
Collated
Research
Knowledge
Library (KL)
Define Research Select Research Approach,
Questions and purpose Strategy, Time horizon.
Data collection
for assessment
and reflection of study
Cost/
Benefit
techniques
(3Es)
Applied
to
Organised Learning (OL)
Cost Benefit Appraisal
(CBA)
problems
problems
issues
Models,
Datasets,
Tools
problems
issues
yields
issues
problems
problems
Ontology
(the nature of knowledge)
(‘the what’)
Scientific Philosophy
Figure 2: Key parts and elements of the intellectual device
The fourth part, OL, induces the selection of an appropriate research strategy (Saunders et al 2009),
e.g. case study, which should be philosophically compatible with the identified methodology (M). OL
encompasses various aspects (or layers) of the research onion. Additionally, OL encourages
assessment and reflection of the completed intervention. Assessment inspects the theory,
methodology and results generated from the practice. The fifth part, CBA explores the cost and
benefits of the methodology through the use of various techniques (e.g. cash flow spreadsheets). This
enriches the overall assessment of the methodology. The combination of OL and CBA can ensure
new knowledge, generated by the identified methodology, is rigorously examined. Moreover,
Reflection (Schon, 1983) facilitates a review of the whole experience of applying a methodology.
Furthermore, reflection initiates meta-thinking which is revealed through the generation of lessons
learnt which could lead to modifications of F(concepts), M (principles), A (phenomenon), CBA
(techniques) and OL (research strategy) or in some instances abandoning the methodology
altogether. Thus, OL is considered to be the critical aspect of investigating the identified methodology.
The sixth part of the device is labelled Knowledge Library (KL). It encourages the management and
collation of the completed studies and respective outputs, e.g. models, or datasets, which might be
used in future studies – and this may reduce future costs.
60

Gary Bell
The single and double learning concepts, associated with Organisational Learning (Argyris and Schon
1978), are thought to underwrite the intellectual device (see figure 3). Single loop learning is the
correction of errors without altering the organisation’s norms or objectives, whilst questioning these
norms and objectives involves double loop learning. It is asserted that a methodology operates within
its identified ideas and principles (linked to norms). When ‘in practice’, problems (or anomalies) might
be experienced and engender modifications to the methodology. Double loop learning emerges when
problems (or concerns with effectiveness) are so serious – a threshold is reached, e.g. a need for an
ontological shift, which leads to major changes to the underpinning methodology and associative
ideas and principles. Critically, this can engender an epistemological framework shift.
Results,
Anomalies,
Consequences
Single (first) Loop Learning
Double (second) Loop Learning
Methodology
‘in action’
Overarching
Ideas, concepts and
principles
Figure 3: Single and double loop learning concept underpinning the intellectual device
It is important to establish the link between the key parts of the intellectual device, scientific
philosophy assumptions, and methodology and models (see table 5).
Table 5: The connectivity between intellectual device, philosophy and methodology
Parts of the Intellectual Philosophy of
Device
Science
Framework of ideas and
concepts (F)
Epistemology
Methodology (M) Epistemology
Methodology and Models
How we model.
The forms of knowledge and
knowledge creation the methodology
uses. This is realised in terms of
where the model comes from (‘the
world’ or someone’s belief) and the
form in which it is represented
(Mingers, 2003).
Area of concern (A) Ontology What we model.
What types of entities are taken to
have existence. What kinds of things
the methodology will build models of
(Mingers, 2003).
Organised Learning (OL) Axiology Why we model.
What is valued or considered good.
This is manifest in what the purpose
or uses of the model are, and who
develops and uses the model
(Mingers, 2003).
Research Value (RV) Axiology
Knowledge library (KL) Axiology,
Epistemology,
Ontology.
The intellectual device is employed to organise the critical appreciation process of the GQM
Paradigm.
5. Critical appreciation of goal/question/metric (GQM) paradigm
The GQM Paradigm (Basili and Weiss 1984) has evolved over years of practice to meet the changing
goals of NASA/Goddard Space Flight Centre (GSFC). This improvement programme encountered
unforeseen issues and problems at different hierarchical levels (i.e., product, process and
61

Gary Bell
organisational) which led to the creation of the GQM Method (Basili and Rombach 1988), Quality
Improvement Programme (QIP) (Basili et al 1994a) and Experience Factory (EF) (Basili et al 1994b).
These developments are respectively connected with a three phase model (see figure 4) (i.e.
Understanding, Assessing and Packaging).
The individual phases are associated with modifications to appropriate aspect of the GQM Paradigm
(linked to single loop learning). The emergence of new requirements lead to the inception and
development of GQM, QIP and EF, and a new phase. Thus, the different phases prompt three
iterations of the intellectual device to respective aspects of the GQM Paradigm in order to generate
the epistemological framework.
Assessing
Packaging
Improvement
Determine the impact of a change, e.g.
• Does object-oriented design produce more reusable code?
• Do inspects results in lower error rates?
Evaluate OO
Evaluate clean room
goal
Compare test techniques
Assess design criteria
Goal-question-metrics model
Evaluate Ada
Domain analysis
Understanding
Quality improvement paradigm
Evaluate cost and resource models
Experience factory model
Build a baseline of process and products, e.g.
• What development techniques are used?
• What is the unit cost of software?
• What types of errors are most common?
• How much reuse occurs?
Initial clean room study
Initial Ada-FORTRAN study
Reuse analysis
IV and V
Error and change profiles
Environment
Approach to data collection Initial OO study
Design measurements
Resource and effort profiles Relationship among development measures
Capture improved techniques as a part of modified process, e.g.
• Incorporate inspections into development standards
• Develop inspections training program
Recommended approaches
Training material
Cleanroom process model
SME Ada users manual
Manager’s handbook
Programmer's handbook
1976 1980 1990 1995
Continual improvement over time
Measurement characterisation
Figure 4: The three phase process model populated with experiments (Basili et al 2002)
5.1 Appreciation aspect of GQM
First Iteration
In the first phase, there was a requirement to understand the software processes and products at
NASA/GSFC. NASA/GSFC is chartered to produce software to support flight missions. The software
ranged from 5KLOC 2 s to 1MLOCs, but was usually around 100 KLOCs. There were normally 5 to 15
projects being developed at the same time employing a 100 to over 300 software developers (see A
in figure 2). Analytical thinking is used to examine individual problems (see F in figure 2).
Measurement is central to quantitative understanding of both processes and products. It was
important to make sense of software measurement and its potential role (i.e. knowing what to
measure and how to measure it). The GQM method employed a template (see table 6) to examine
vendor claims about respective technologies and methodologies. These claims were used to generate
the operational goals, questions and metrics. Moreover, goals are refined into several questions (see
figure 5). Every question is refined into objective and subjective metrics (linked to the principle of
reductionism). Additionally, the same metric can address different questions under the same goal.
However, each goal is treated separately. The modeller assumes the role of the observer (see M in
figure 2.0). Additionally, the managers and developers that participated with the modeller assume the
role of passive stakeholder.
Importantly, NASA/ GSFC was conceived as an experimental laboratory and every evaluation of a
technology or methodology was viewed as an experiment (see OL in figure 2). Basili et al (1986)
produced a structure that enables researchers to define the purpose of the experiment and the object
of study. The positivistic scientific method underwrites GQM research, and numerous experiments
were conducted in the Understanding Phase (see figure 4). Reflection encompasses an appreciation
2 Lines of Code (LOC)
62

Gary Bell
of the quality initiative used and individual experiments that can lead to the manifestation of lessons
learnt and may evoke modifications to F, M, A, OL. Basili et al (2002) identified 7 lessons learnt (see
table 7) which are connected with the Understanding Phase. Additionally, further lessons associated
with this phase were learnt which guided the key principles of GQM method: understand the goals,
make sure the measures apply to the goals, and keep the number of measures to a minimum. He
believes during the Understand Phase an insight of how to metricate was achieved. Moreover, this led
to the identification of key measures that characterise the software project which is known as GSFC’s
baseline datasets.
Table 6: Template for goal definition (Basili and Rombach 1988)
Template
Purpose:
To (characterise, evaluate, forecast, motivate, etc.) the (process, product, model, metric, etc.) in order
to (understand, assess, manage, engineer, learn, improve, etc.) it.
Example: To evaluate the system testing methodology in order to improve it.
Perspective:
Examine the (cost, effectiveness, correctiveness, defects, changes, product metrics, reliability, etc.)
from the point of view of the (developer, manager, customer, corporate perspective, etc.).
Example: Examine the effectiveness from the developer’s point-of-view.
Environment:
The situation consists of the following: process factors, people factors, problem factors, methods,
tools, constraints, etc.
Example: The product is an operating system that must fit on a PC, etc.
Goal 1
Question Question Question Question Question
Metric Metric Metric Metric Metric Metric
Figure 5: Structured GQM Model
Table 7: Lessons learnt from the understanding
phase (Basili et al 2002)
Number Lesson
1 Data collection requires a rigorous process and professional staff.
2 You must compromise on asking only as much information as is feasible to obtain.
3 Staff training in data collection is a never-ending vigil.
4 As important as data collection is, it still takes second place to deadlines.
5 Establishing a baseline of an organisation’s products, process, and goals is critical
to any improvement programme.
6 The accuracy of the measurement data will always be suspect, but you have to
learn to live with it and understand its limitations.
7 There will always be tension between the need to rapidly feed back information to
developers and the need to denote sufficient time to do an analysis of the collected
data.
Second Iteration
In the second phase, there was a requirement to assess the effects of various technologies and
methodologies upon the software processes and products at NASA/GDFS. Problem-solving was
conceptualised as the introduction of a technology or methodology to a specific problem. Changes to
the software process (independent variables) are made in an attempt to improve the product
(dependent variable) is a critical notion of this quality initiative. Baseline data generated in the
Understanding Phase is utilised to construct mathematical relationships between variables (see eqn
63
Goal 2

Gary Bell
1.0 to 4.0) (linked to the cause/effect concept). Hence, the effectiveness of implementing a change
can then be quantitatively evaluated.
Effort (staff – months) = 1.48 (size) 0.98 (1.0)
Duration (months) = 4,6 (effort)
e (co t $)
0.26 (2.0)
Annual Maintenanc s = 0.12 (development cost) (3.0)
Average Staff Size = 0.24 (effort) 0.73 (4.0)
This
led to the development of the QIP methodology which employs a set of methods for the activity
of improving the software process and product (see M in figure 2.0). QIP was constructed as a way of
employing the positivistic scientific method in order to improve upon the stated goal. The QIP process
model (see figure 6.0) has six key steps (see table 8.0). The principles that underwrite QIP have
evolved through experimental practice (see table 9.0). The GQM template is used as a mechanism for
defining and interpreting a set of operational goals using measurement.
Package Characterise
Analyse Set goals
Execute Choose process
Figure 6: The QIP process model (Basili et al 1994b)
Table 8: Description of the QIP steps (Basili et al 1994b)
Step Quality Step Description of Step
1 Characterise Understand the situation using available models, data intuition, etc.
Establish baselines with the existing business processes in the
organisation and characterises their criticality.
2 Set Goals On the basis of the initial characterisation and of the capabilities that has
strategic relevance to the organisation, set quantifiable goals for
successful project and organisation performance and improvement. The
reasonable expectations are defined based upon the baseline fashioned
by the characterisation step.
3 Choose On the basis of the characterisation of the situation and of the goals that
Process
have been set, choose the relevant processes for improvement, and
supporting methods and tools, making sure that they are consistent with
the goals that have been set.
4 Execute Perform the process constructing the products and providing feedback
based on the data on goal achievement that are being collected.
5 Analyse At the end of each specific project, analyse the data and the information
gained to evaluate the current practices, determine problems, record
findings, and make recommendations for future project improvements.
6 Package Consolidate the experience gained in the form of new, or updated and
refined, models and other forms of knowledge gained from this and prior
projects and store it in an experience base so it is available for future
projects.
Table 9:
Some key principles gained through experimental research (Basili et al 1994b
Identified Principles
1 There are factors that create similarities and differences among projects; this means that
one model for software production does not work in all circumstances.
2 There is a direct relation between process and product; this means the selection of the
right process to create the desired production characteristics.
3 Measurement is necessary and should be based on the relevant goals and models;
appropriate measurement provides visibility.
64
)

Gary Bell
Abstraction sheets (Briand et al 1996) (see M in figure 2) have been utilised to elicit knowledge (or a
priori knowledge) from stakeholders (linked with the Characterise Step of the QIP methodology).
Moreover, Abstraction Sheets are used to bridge the gap between goal and questions (associated
with the Set-Goals Step). It seems the GQM modeller assumes the role of the expert or teacher. The
managers and programmers that participate with the modeller seem to assume the role of student, for
they focus on developing software and are not familiar with this approach.
The QIP methodology promotes the view that continuous process and product improvement should
utilise the positivistic scientific method. The experimental component employs this method for the
activity of understanding through hypothesis formulation and testing, and applies the measures
generated by the GQM method. Moreover, collated datasets and development of mathematical
relations between variables enables comparisons with previous analysis in the Understand Phase.
Reflection
encompasses an appreciation of this quality initiative used and
individual experiments
which can lead to the identification of empirical based lessons learnt. Basili et al (2002) identified 3
methodological lessons learnt (see table10.0) that are associated with the Assessing Phase. It is
contended that further lessons associated with this phase were learnt and are key principles of QIP:
understand that software measurement is a means to an end - not an end in itself, focus on applying
results rather than collecting data, understand how to apply measurement, focus locally, start small,
avoid
over-reporting measurement data, use simple definitions of life-cycle phases, make providing
data easy, and use commercial tools.
Table 10: Lessons learnt from the assessing phase (Basili et al 2002
Third Iteration
Number Lesson
8 Having a shared commitment over research and development is vital for success.
9 There is a symbiotic relationship between research and practice in software
engineering, and both activities gain from the interaction.
10 Close proximity of research to development aids both.
In the third phase, there was a requirement to package study results which enables reuse by other
NASA/GSFC software projects. The critical concept is the reuse of knowledge, models, products and
experiments as solutions that contribute to the building of high quality software products at a lower
cost. Organisations attain quality improvement through defining and developing a relevant set of
strategic capabilities (e.g., the company must understand the software process and products) and
supporting core competences (e.g., critical areas are packaged and reusing cluster of experiments).
Software
Organisations
can be divided into a Project Organisation (PO) and EF, which perform
different activities and generating specific products. NASA/GSFC is conceptualised as the PO. GSFC
aims to support reuse of experiments and collective learning through developing, updating, and
providing upon request clusters of competencies to be applied by the project organisation. The
interaction between the EF and PO must be continuous and effective. The process model (see figure
7) shows the synergy between the PO and EF (see M of Figure 2).
The PO aims to develop and maintain software products. It provides environment and project
characteristics, development data, resource usage data, quality records and process data. This gives
feedback to the performance of software models utilised by the project. The EF produces baselines
(see eqn 1.0 to 4.0), tools, lessons learnt, and parameterisation of some form in order to adapt a
project specification. Interestingly, the EF employs the process model (see figure 7) to guide the
activity of improving. The main products of EF are core competencies which are aggregated
packages.
The PO applies core competences which are in experimental
packages. Additionally, some
key principles that underpin EF are identified in table 11.0. Additionally, the modeller and developer
within EF maintain respectively the roles of teacher and student (see M in figure 2).
GSFC conceptualises NASA/GSFC as an industrial based laboratory, and software projects are
individual experiments (see OL of Figure 2.0). Numerous experiments applying QIP were conducted
in the Packaging Phase (see figure 6.0). Collected project data is collated in order to test the
generated experimental hypothesis. Moreover, this data was used to update the baseline data and
appropriate mathematical models enabling comparisons with past baselines and parameters. The
updated baselines and mathematical models can be used to inspect the performance of NASA/GSFC
through contrasting with the organisational goals. Reflection encompasses an appreciation
of the EF,
65
)

Gary Bell
QIP and GQM and individual experiments which can lead to the identification of lessons learnt.
Lessons learnt can facilitate improvements to F, M, A, OL . Basili et al (2002) identified 3 lessons
learnt (see table 12.0) which are connected with the Packaging Phase. Furthermore, the 13 lessons
have been placed into one of four lessons learnt from GSFC’s quality programme (see table 13.0). It
is contended that further lessons associated with this phase were learnt and are key principles of EF:
set expectations, organise the modellers separately from the developers, budget for the cost of the
measurement programme, do not expect to find a database of process measurements, understand
the high-level process characteristics, and do not expect to automate data collection.
Characterise
set goals
choose process
Execute
plans
Execute process
Environment
characterise
Goals, processes
Tools, Products,
Resource models,
Defect models
Data,
Lessons
learned
Project
Analysis
Project
support
Experimental
Base
Analyse
Project Organisation Experiment Factory
Figure 7: Experience factory (Basili et al 1994b)
Table 11: Key principles of the EF (Jeletic et al 1996)
Package
Generalise
Tailor
Formalise
Principle
1 Process improvement is driven by internal goals and local experiments.
2 Each domain is examined in a different way according to specificity.
3 The situation is characterised according to organisational dependent measures.
4 No assumptions are made about the best practices in the software process.
5 The fundamental measure of success is the software product delivered by the organisation.
Table 12: Lessons learnt from the understanding phase
(Basili et al 2002)
Number Lesson
11 Having upper management support is important for continued success.
12 The organisation trying to improve their process has to own the improvement
process.
13 It is difficult to make an engineering organisation aware of the importance of
software engineering to their mission.
The nature of software has facilitated the awareness of the importance of replicating studies. Basili et
al (1999) advocate building empirically based knowledge through families of software experiments. It
is argued that replications and complimentary studies can be encouraged by laboratory packages
that
document an experiment.
Table 13: Lessons Learnt from the GSFC SPI Programme (Basili et al 2002)
Overall Lessons Lesson
Need for Collective project data. Lesson 1, Lesson 2, Lesson 5, Lesson 6.
Need for management buy-in to the Lesson 8, Lesson 11, Lesson 12, Lesson 13.
process.
Need for a focused research agenda. Lesson 9, Lesson 10
Need for continued staff support. Lesson 3, Lesson 4, Lesson 7. )
66

5.2 Critical aspect of GQM
Gary Bell
The GQM Paradigm has underwritten many experiments conducted at NASA/GFSC (see figure 4)
which has led to numerous published articles by Basili. Some of these articles have been collated into
a Software Engineering book (Boehm et al 2005). The benefits (see OL and CBA in figure 2) of
applying this improvement initiative are highlighted in table 14.0. Furthermore, we found the practice
of GQM confirms the conjecture: “quality improvement decreases costs because of less rework, fewer
mistakes and fewer delays which in turn increases productivity” Deming (1982).
Table 14: The Impact of GSFC (Basili et al 2002)
Attribute Change:
Change:
1987-1991
1991-1995
Decreased development defect
rates
75% 37%
Reduced cost 55% 42%
Improved reuse 300% 8%
Basili et al (2002) revealed that the University of Maryland
received a total funding of $5.7M over the
25 years for conducting SPI research within GSFC. GSFC began with a one year grant to University
of Maryland for $50K. Funding for the University share of GSFC activities rarely exceed $300K per
year. To gain contractor co-operation, a 10% overhead cost to projects was allocated for data
collection and processing. However, this was rarely required. The costs were initially 5% and over
time fell to 1% to 2% of software development costs. However, with the analysis activity of GSFC
added in - the total costs remained about 10%. This included experimental analysis, measurement and report generation.
6. The GQM epistemological framework and social theory
The epistemological framework for the GQM Paradigm is established. It is evident that GQM, QIP and
EF have been developed for different phenomenon – product, process and organisation. The
combination of ontological and requirement changes have facilitated 3 iterations of the intellectual
device in order to update the epistemological framework. The fundamental characteristics of this
quality initiative are: the idea of Analytical Thinking, the concepts of teleology and cause/effect, and
the principle of reductionism (see F and M in figure 2.0 and epistemology in table
4.0). Furthermore,
the intention of this SPI approach is to attain the objectivity notion ‘value-free’
(see M in figure 2.0 and
axiology in table 4.0). Additionally, the original role of the modeller (observer) and stakeholder
(passive) are identified (see M in figure 2.0). The laboratory research strategy and positivistic
scientific method are intertwined with the GQM Paradigm (see OL in figure 2.0). Importantly, it is
recognised that this SPI methodology is continually evolving – as highlighted in its shift in modeller
(teacher) and stakeholder (pupil) roles. The GQM Paradigm is linked to the following philosophy of
science assumption attributes: realism (ontology) (linked to ‘what’ we model), Positivism
(epistemology) (linked to ‘how’ we model), value-free (axiology) (linked to ‘why’ we model) and
nomothetic (methodology) (determines overall activity) (see table 4.0). This objectivist approach is
linked to the social theory Objectivism (see figure 1.0 and table 2.0). This improvement methodology
has made important contributions to the body of knowledge within Empirical Software Engineering.
7. Conclusion
It is concluded that the sociological analytical scheme and intellectual device are promising tools for
benchmarking the whole methodology through undertaking a critical appreciation process. However,
the appraisal findings of the benefit and cost of the GQM paradigm feels limited – though Basili’s use
of regression analysis (see equations 1) and their parameters as indictors of affect upon the software
process is an interesting idea. There is a need to develop a tool box of cost benefit approaches to
assist with insightful and rigorous
appraisals. Basili’s reflection of the GQM Paradigm through the
lessons learnt seems
to support the idea of meta-thinking. Moreover, it is felt that the intellectual
device has been verified by GQM library of published materials generated by Victor Basili and coresearchers.
The intellectual device can explicitly guide thinking about the practice and reflection of a methodology.
Moreover, every intervention using an improvement methodology can be considered as research. The
critical appreciation findings underpin the epistemological framework for the GQM Paradigm – and
reveal ‘ the nature of knowledge, the development of knowledge, and the benefit of knowledge and its
cost’. Moreover, the work uncovers its connectivity with the social theory of objectivism and its
67

Gary Bell
extreme scientific philosophical assumptions. Therefore, it is contended that the GQM Paradigm is
strongly connected with Empiricism and Natural Science.
The initial impact of the GQM method upon software products developed by NASA/GSFC was
significant. However, the benefit with respect to cost seems to diminish at software process and
organisational hierarchical levels. This could be because of the nature of the methodology – the
fundamental idea, concept, principle and philosophy underpinning each development of the GQM
Paradigm remain the same. Moreover, the evolving QIP and EF developments within the GQM
paradigm are associated by the single loop concept. It is asserted that as complexity and uncertainty
increases at higher hierarchical levels such as software
and organisational process levels. A
phenomenon
which involves people is normally thought of as highly complex and uncertain and can
be described as problematic situations. Thus, it is contended that this improvement methodology (i.e.
QIP and EF) might implicitly become the constraining factor to improvement at process and
organisational hierarchical levels.
It is believed there is a need to apply the intellectual device to various traditional management
methodologies, e.g. CMM. It is asserted that similar characteristics and phenomenon will be revealed.
Furthermore, this would support the conjecture: ‘orthodox software project management approaches
are not powerful enough to treat the software crisis’. Moreover, orthodox methodologies (underpinned
by empiricism which is the norm) maybe reinforcing single loop learning, i.e. correcting errors to
orthodox methodologies, and preventing double loop learning within software project management.
Therefore, it is argued a ceiling (links
with asymptotic behaviour) has been reached and there is a
need
to develop richer improving and controlling methodologies that perhaps empower stakeholders,
i.e. value-full, which initiates the selection of alternative research strategies, e.g. Action Research.
Additionally, phenomenon such as situations involving people can initiate a shift to the subjectivist
scientific philosophy and perhaps the use alternative characteristics.
This could initiate double loop learning (changing of norms) within Software Project Management.
Moreover, this would encompass an intellectual shift towards: the interdisciplinary notion rather than a
uni-disciplinary one, and theoretical approaches rather than empirical. The discarding of orthodox
project management methodologies is not advocated, but to compliment them with established
methodologies from different disciplines, e.g. Systems Movement, which might generate a new
software project management methodology. This idea is connected with methodological pluralism
which is associated with the social sciences. For example, coupling
aspects of Soft Systems
Methodology
(Checkland 1981) with the GQM Paradigm. Finally, the intellectual device could
underpin this SPI research.
References
Ackoff, R.L. (1979) “The Future of Operational Research is Past”, Journal of Operational Research Society, Vol.
30, No. 2, pp 93-104.
Argyris, C., and Schon, D. (1978) Organisational Learning: A Theory of Action Perspective, Addison Wesley,
Reading.
Basili, V., McGarry, F., Pajerski, R. and Zelkowitz, M. (2002) “Lessons Learned from 25 Years of Process
Improvement: The Rise and Fall of the NASA Software Engineering Laboratory” ,In the Proceedings of the
International Conference on
Software Engineering ,May, Orlando.
Basili, V., Shull, F.,
and Lanubile, F. (1999) “Building Knowledge through Families of Experiments”, IEEE
Transactions on
Software Engineering, Vol.25, No. 4, pp 456-473.
Basili, V., Caldiera,G., and Rombach, H. (1994a) “GQM Paradigm” , In Marciniak (Ed.) (1994).
Basili, V., Caldiera,G., and Rombach, H. (1994b) “Experience Factory” , In Marciniak (Ed.) (1994).
Basili, V.R., and Rombach, H.D. (1988) “ The TAME Project. Towards Improvement- Oriented Software
Environments”
, IEEE Transactions on Software Engineering, Vol.14, No 6, pp 758-773.
Basili, V.R., Selby, R. and Hutchens, D. (1986) “Experimentation in Software Engineering”, IEEE Transactions
on
Software Engineering, Vol.12, No 7, pp 733-743.
Basili, V.R., and Weiss, D. (1984) “A Methodology for Collecting Valid Software
Engineering Data”, IEEE
Transactions on Software Engineering, Vol. 10, No 6, pp.728-738.
Bell, G.A., Warwick, J. And Kennedy, M. (2009) “Rethinking The Software
Project Management Discipline”,
Inside OR, January , No457, pp 20-21.
Bell, G.A., Cooper, M. and Qureshi, S. (2002) “The Holon Framework and Software Process Improvement:
A
Radiotherapy Project Case Study”, International Journal of Software Process: Improvement and Practice.
Vol. 7, No 2, pp 57-70.
Boehm, B.W., Rombach, D., and Zelkowitz, M. (Eds) (2005) Foundations of Empirical Software Engineering: The
Legacy of Victor R Basili, Springer, Berlin.
68

Gary Bell
Briand, L.C.,Christiane, M., Differding, H., and Rombach, D.(1996) “Practical Guidelines for Measurement-Based
Process Improvement” , Fraunhofer Institute for Experimental Software
Engineering ISERN– 96-05,
Kaiserslauten.
Burrell, G., and Morgan, G. (1979) Sociological Paradigms
and Organisational Analysis, Gower, Aldershot.
Checkland, P.B. (1981) Systems Thinking, Systems Practice, John Wiley and Sons, Chichester.
Checkland,
P.B., and Holwell, S. (1998) Information, Systems and Information Systems: Making Sense of the
Field, John Wiley and Sons,
Chichester.
Deming, W.E.(1982) Out of the Crisis, MIT Centre for Advanced Engineering Study, Cambridge.
Flew, A. (1979) A Dictionary of Philosophy, Pan Books, London.
Fuggetta, A. (1999) “Rethinking The Modes of Software Engineering Research”, The Journal of Systems and
Software, Vol. 47, pp 113 -138.
Jeletic, K., Pajerski, R. And Brown, C. (1996) “Software Process Improvement Guidebook: Revision 1”, SEL
Technical Report (NASA-GB-001-95) ,Maryland.
Marciniak, J.J (Ed) (1994) Encyclopaedia of Software Engineering, John Wiley and Sons, New York.
Mingers, J. (2003) “A Classification of the Philosophical Assumptions of Management Science Methods”, Journal
of Operational Research, Vol. 54, No. 6, pp
559-270.
Saunders, M., Lewis, P., and Thornhill, A. (2009) Research Methods for Business Students, (5 dn) Prentice-
th e
Hall, Harlow.
Schon, D.A. (1983) The Reflective Practitioner: How Professionals Think in Action, Basic Books, USA.
69

Validation of Grounded Theory Based Data by Means of
Analytical Mapping Techniques
Martin Bergaus, Bernd Stottok and Andrea Gorra
Leeds Metropolitan University, UK
martin.bergaus@gmx.at
b.stottok@gmx.de
a.gorra@leedsmet.ac.uk
Abstract: Mapping techniques serve as the visualisation of abstract and complex connections. The use of maps
of scientific literature research enables researchers a rapid, structured management of different topics. The goal
is to support analytical process development for the structuring and grouping of data and texts by using mapping
techniques. This article presents an analytical mapping process which validates empirical data analysed by
Grounded Theory Methodology (GTM). The mapping process to be presented is a representation of mind and
concept mapping. With every map generated, the core information of a specific topic is visually shown to the
viewer. In this way the researcher is challenged to find an understandable connection for every term, and also to
be able to create obvious key information for other researchers in a quick and simple way. For a research topic,
one or various structured maps are created, in which text passages with similar contents are summarised in
hierarchical graphs. This method offers the researcher the advantage of being able to systematically extract
fundamental aspects out of extensive literature. Combinations, hierarchies, and causal connections within a given
subject are prepared in a graphical way. Complex topics can be understood quickly and simultaneously
structured, so that they can be taken as indicators for a validation of empirically determined data. The method,
moreover, helps with the validation of large amounts of data by researchers in areas where they are not
necessarily experts. This article describes the steps necessary when an analytical mapping process is used for
structuring of literature based data. The usability of this method is evaluated through a discussion of its
advantages and disadvantages, and the presentation of three studies in which the method has been successfully
applied. The article is therefore relevant for all those interested in the practical use of mapping techniques in the
context of Grounded Theory.
Keywords: mind and concept mapping, grounded theory, data validation, empirical research, qualitative data
1. Introduction
This paper presents a mapping process that can be used for structuring of literature based data in
order to create a structured base for validation of empirical data. It firstly offers a brief introduction to
mapping techniques, their advantages and features, including existing challenges for the researchers.
This is followed by a description of data preparation derived from literature data using mapping
techniques.
The mapping process basically follows traditional mind and concept mapping, consisting of the
following four phases: firstly the literature review phase, in which corresponding data sources on a
subject are collected and recorded; secondly the data collection phase, in which data can
be structured into topics; thirdly the requirement definition phase, in which conditions in terms of
subject areas and individual areas are defined and illustrated; and fourthly the data grouping phase, in
which specific aspects are assigned to group data and visualise them in a map (Bergaus and Stottok
2010: p 113).
During the generation of maps, literature is being analysed. Based on this, a general and detailed
(two step) list of criteria is developed with details of sources, which contains links to each literature
reference. From the lists of criteria, the visual transformation takes place in form of maps. The aspects
of a generated map will be summarised as a broader category. These categories will then serve as a
basis for the validation of empirical data (Bergaus 2010b).
This method was developed in the context of grounded theory and has been successfully used in
three research studies. The achievement of interim results of two on-going PhD theses and
additionally in a completed master's thesis. All three studies were based on empirical studies, which
have been validated by literature, generated from parent categories, as the results of this mapping
process. The main reason why this mapping process was developed and used was the validation of
empirical data with literature-based data (Bergaus 2010a, Bergaus and Stottok 2010, Stottok, 2010).
70

Martin Bergaus, Bernd Stottok and Andrea Gorra
This paper illustrates, step by step, and with the aid of specific examples, how to use the method. It
discusses the advantages and disadvantages of the process. It is especially suited for research
studies in which empirically gathered data is analysed and validated through literature.
2. Overview of mapping techniques
There are different visualisation techniques that support the learning and understanding of certain
topics (Brühlmann 2007). Mapping techniques may be used especially when they offer advantages
over other visualisation techniques, for example they can be used non-specific contexts (contexts
from different topics) and therefore various areas (Buzan and North 1999).
There are two main mapping techniques: mind maps and concept maps. The former can be seen as
general maps of thoughts and ideas, whereas the latter rather shows hierarchies of concepts in the
form of cognitive maps. Mapping techniques are used to visualise abstract and complex relationships.
Mind and concept mapping techniques are thus used for external visualisation. Each map present
ideas to the viewer in a visual manner in order to make them accessible. These techniques lead to a
whole new way of thinking, and encourage creativity, because they appeal to our brain holistically
(Brühlmann 2007, Buzan and North 1999). The main difference between mind and concept maps can
already be seen in their structure. Concept maps are subordinate that entities are hierarchically
structured whereby the lines between the different terms are always named explicitly. In addition, the
generic term is not often in the middle of the map, as is usual in the mind map, but on top of the map
in a hierarchical position of importance.
The primary difference between mind maps and concept maps lies in the different theoretical views of
knowledge representation in the brain. Tony Buzan, the inventor of mind maps, mainly aimed at the
associative structure of memory and therefore attempted to primarily support the process of
association. This means that, in an ideal case, the individual branches of a map are each a single
term. The term can be regarded as an associative key word, and can act like a "door opener" for more
terms and structures for the outgoing strand of the association. If we fail to find appropriate terms,
short sentence-like phrases can also be beneficial (Buzan 2003). In contrast, Novak and Gowin
(1984), the inventors of concept maps, believe that knowledge in the brain is organized hierarchically
and is based on concepts (Brühlmann 2007).
When reading a linear text, only one half of the brain is being exercised, whilst visualisation enables
both halves of the brain. The interaction between the left and right side of the brain when learning with
maps offers the major advantage of being able to comprehend new and complex topics very quickly,
and to restructure them at the same time. During the map-drawing a deep processing of thematic
contents is triggered, and thus a connection between prior knowledge and new information is
established, since the information structure must be reduced to the most important points. Thus, links,
hierarchies and causal relationships within the considered topic have to be worked out (Brühlmann
2007, Buzan and North 1999).
Another advantage of working with learning maps is the metacognition function. With the help of the
creation of a map the learner will be made aware of what he knows about the topic to be learned and,
whether this is really understood, since he must put every word into context [within the map].
Therefore the closure of knowledge gaps can be possible, if the learner is forced to deal with the
matter in more detail. The stimulation of multiple representation capabilities represents a third benefit
of the external visualization. If the content of a text is depicted using a map, it must be put from text
format into the representation of a map. A type of translation of a representations code to another
graphical based code takes place. The fourth advantage of the mapping technique is the inference
function, because in a map is easily understood. The key information of the learning content can in
most cases, be evident and the reader no longer needs to filter the meaning out of the text
(Brühlmann 2007).
An additional advantage of mapping techniques is their attractiveness for researchers who are
visually oriented. Mapping techniques are just as attractive for their creator as for those to whom the
map is presented. Maps are often shown in presentations, with different colours and images, in order
to motivate the audience to listen. Information is also easier and faster to read and thus to
communicate. Furthermore, time can be saved with the mapping technique because maps only
display relevant and short terms or phrases and not complete sentences (Buzan and North 1999).
This can be perceived as time saving in comparison to working with linear notes. Thus, learning with
71

Martin Bergaus, Bernd Stottok and Andrea Gorra
maps effectively saves time. In addition, using maps means that the natural tendency of the brain to
produce countless associations can be taken advantage of. This advantage is especially true for mind
mapping. A mind map gives by its appearance the impression of the feeling that countless other
branches may be added, but it can also be viewed as a whole in itself. Buzan speaks of “radiant
thinking” (Brühlmann 2007: pp 12-13).
3. Use of mapping techniques for this study
The Grounded Theory Methodology (GTM) offers researchers a systematic conglomerate of various
methods and techniques to be able to handle the processing of empirical data, its analysis and
conceptualization. The methods and techniques provided by GTM are not rigid instructions, but
practical research "recipes" (Strauss and Corbin 1996: Preface). Grounded Theory (GT) is classified
as an inductive research method, because a theory is developed based on empirical data. Glaser and
Strauss (1967) wanted to show with their research method, that results also from qualitative studies
are verifiable.
In the last 40 years, since the discovery of the GT, several improvements and simplifications of the
theory and variations have taken place, depending on the research purpose and circumstances of the
research project (Paris and Hurzeler 2008). Today, GTM is amongst the most common approaches to
qualitative social research and it is used, in addition to sociological research, in other disciplines such
as psychology or pedagogy (Paris and Hurzeler 2008).
The researchers of the study presented in this paper were faced with the challenge of conducting an
extensive literature survey, as part of their grounded theory-based study, “Improved Use of
Telematics Systems in Rail Traffic Through the Provision of Innovative Service Delivery Platforms”
(Bergaus and Stottok 2010). At the same time, they were aware that although the reviewing of
literature can be a part a modern approach to Grounded Theory, this process of literature review
should to affect the process of collecting and analysing data (Charmaz 2006).
An important aspect in this context was that the above-mentioned study was a collaborative effort, in
which both researchers connected their own areas of research. They had to work out not only links,
hierarchies and causal relationships in their own research area, but also had to access and
understand them, in the hitherto unknown specialist field of the other researcher. The mapping
techniques used thus allowed us to quickly develop a mutual understanding of a different subject
area. The two researchers conducted an initial literature review prior to the first gathering of empirical
data within their own topics of Railway Telematics and Service Delivery Platforms.
This approach followed the GT approach of Kathy Charmaz. A detailed review of the literature
followed after the data collection phase, which was based on findings in the data analysis phase.
The mapping process for this study began with a literature search of the topics studied. A first rough
structure of the individual subject areas to be used to create the maps was established during the
course of the data collection phase. Following this, terms were grouped, according to topic-specific
definitions. This procedure is detailed in the next section.
4. Data preparation through the use of mapping techniques
Bergaus and Stottok’s study (2010), deals with the analytic perspective for general requirements of
the communication infrastructure, based on literature regarding the topics of railway telematics and
the aspect of Service Delivery Platforms. Additionally, the researchers supported their work with the
following, closely described, mapping process, in which aspects investigated in the literature were
structured into general as well as detailed demands, and were clearly grouped. The mapping process
pursued three essential aims (Bergaus and Stottok 2010: pp 112-113):
The development of a criteria catalogue for requirements and design aspects of a railway
telematics system of the future, which uses a Service Delivery Platform.
The structuring and grouping together of the aspects as support for a quicker understanding.
Validation of the results from the empirical study by means of the compiled criteria catalogue, so
that a model based on the criterion catalogue was able to be developed.
In this research project maps were provided within the scope of the mapping process, in order to look
at most of the different aspects from literature on and about Railway Telematics, Next Generation
Networks, Ubiquitous Services and Service Delivery Platforms, from varying perspectives. Besides
72

Martin Bergaus, Bernd Stottok and Andrea Gorra
this, the various mind maps represent the results of the mapping process in regard to a particular
situation.
For the part of the study in which the literature was analysed and processed the following sequential
expiry phases were carried out:
Figure 1: General representation of the phases for the analytic, literature-based part of the study
(Bergaus and Stottok 2010: p 113)
In the following the general phases represented in figure 1 will be described:
Literature Review Phase: Books, dissertations, and specialist articles about the subject areas
Railway Telematics and Service Delivery Platforms were researched, the corresponding sources
of data were retained and filed. It is important to mention that both researchers follow the starting
point of Charmaz (2006: p 178), and consciously with into consideration the literature along with
empirical data collection and analysis.
Data Collection Phase: Collection of data connected to the subject areas already pursued in the
initial structuring process, in which topics were identified.
Conditions Definition Phase: Conditions in relation to theme fields and single areas were defined
and requirements for the design aspect represented (i.e. target group specific requirements from
the perspective of operators, customers and manufacturers).
Data Grouping Phase: The data grouping phase arranged specific requirement groups and
aspects specified in the frame of the Conditions Definition Phase, so that structured, clear, mind
maps were created.
On the basis of this initial literature research, results from the study were brought together, structured
and visualised, by means of mapping processes.
The mind maps created as a result also had a second fundamental purpose in this work: They were
brought forward at the end of the analysis process to act as as validation of the results and findings
which were obtained by the researchers from the interviews. With analytical mapping processes, at
the beginning, following an initial research, collected literature are available. Thereby, in the frame of
the research, the publications, dissertations and internet sites discovered were sorted out and filed
electronically and according to subject blocks, so that, at a later time, they could be found and
processed.
To be able to summarise this, checks were undertaken in a number of steps, and if necessary they
are collected, grouped and further summarized. In the following, an example, taken from the study will
be shown:
Figure 2 shows an initial bringing together and structuring of relevant, important functions for Service-
Delivery-Platforms, whereby, in this first step, a summary and grouping takes place. Important here
for the purpose of being able to follow, also the retaining of the literature reference, from which the
original document originates, in this case a “Mobile Service Platforms Cluster – White Paper – Mobile
Service Platforms – Architecture Whitepaper” (see the underlined sentence between both brackets in
figure 2). Other references would also be possible, for example a reference to an original document in
the Harvard notation (John and Jones 2010: p 12). The requirement, for example function, which is
represented by the colour blue in figure 2, is further dealt with in a more detailed way in illustration 3.
73

Martin Bergaus, Bernd Stottok and Andrea Gorra
Figure 2: Extract taken from literature showing important functions for Service Delivery Platforms as a
general criteria catalogue (Bergaus 2010b: p 21)
Based on this first step a criteria catalogue was set up, which grouped further, ordered aspects from
different documents. A document reference was also incorporated. The following figure shows this
step visually:
Figure 3: Excerpt taken from a criteria catalogue which represents and more closely specifies the
aspect for Beyond 3 rd Generation (B3G) of system of a Service Delivery Platform (Bergaus
2010b: p 21)
Following this, the criteria catalogue was transferred to a map and the main and detailed objects were
subdivided, because in the representation of more than two hierarchy levels it was no longer possible
to recognize a clear graphic on an A4 format. The following figure 4, as well as illustration 7 both
show, as examples, a further deepening of the (both represented in purple) aspects in figure 3.
Figure 4: Requirements on the Services and Mobile Applications from a User Centred Perspective
(Bergaus and Stottok 2010: p 153)
74

Martin Bergaus, Bernd Stottok and Andrea Gorra
In order to represent the general and detailed aspects of the requirement of the communication
infrastructure for the use of innovative, ubiquitous technologies for railway telematics, the results of
literature research were included in the mapping process. This was to be able to structure and group
railway telematics aspects, and those from Service Delivery Platforms. These aspects worked out by
both authors in the context of the mapping process, represented, fundamentally, the criteria catalogue
for design aspects in railway telematics systems of the future (Bergaus and Stottok 2010. pp 143-
144):
General requirements for the communication infrastructure in railway telematics
Requirements for the communication infrastructure from the point of view of different target
groups in railway telematics
Requirements for the telecommunication infrastructure of the future
Requirements for the official development and official supply
Requirements for services and mobile uses from a user-centred view (see figure 4)
The following figure 5 shows, as an example, also, the map for the above five points which come to
notice:
Figure 5: Requirements for the service development and -provisioning (Bergaus and Stottok 2010: p
152)
5. Results validation of the interview data from the study
In Bergaus and Stottok’s study, Grounded Theory categories from interview data were generated.
Subsequently, both researchers attempted to examine the categories developed and compare them
to the available literature. The validation was, thereby, so carried out that the maps created out of the
literature were, once again examined, and through this, single aspects could be clustered together
into sub categories. Therefore, as an example taken from figure 6, the aspect of “Interoperability”
should be possible on a network and device level, and “own services for special user groups or single
users” as well as the aspect “User centered design in a wide multimedia context” and “User
acceptance - possibility for the user to recognize context, to understand and process ...” from the
figure 4, together with one suitable aspects from other maps, brought together in the table 1, into the
category “Usability”:
75

Martin Bergaus, Bernd Stottok and Andrea Gorra
Figure 6: Clustering of aspects to a higher category
The category comparison from the field study, namely the data extracted from interviews, with the
categories worked out of the literature through the mapping process, has been summarised by the
authors in the following table. This means that in a horizontal as well as a vertical comparison the
aspects from the railway telematics, and those from the Service Delivery Platforms show similar
challenges:
Table 1: Overview of categories originating from the interviews and the literature, regarding the
challenges in railway telematics and service delivery platforms (Bergaus and Stottok 2010:
p 185)
Therefore the authors were, during the analysis process, able to compare and validate categories
discovered in the interviews with those generated from the literature. The validation of the categories
in the study by means of table 1 were achieved, at the same time the results for other readers were
understandable and made sense.
6. Discussion of the results
It can be understood from the table above (table 1) that there are a number of advantages when the
mapping technique is applied within a research project:
It becomes possible to picture aspects present in literature, for different hierarchical levels,
showing them in a well-arranged way regarding all topics, and making them transparent for other
researchers.
It is feasible to structure every analysed document in such a way that all information hierarchies
can be clearly displayed.
76

Martin Bergaus, Bernd Stottok and Andrea Gorra
It becomes easier for different researchers to work together on, and familiarise themselves with
new topics.
It is possible to display complex topics in a clearly structured manner.
It is also possible to split up extensive data and show it in different sub-maps.
This method, therefore, has the advantage that a researcher is able to systematically extract general
aspects, also from voluminous literature. Combinations, hierarchies, and causal relations within a
topic, for example relations, are graphically provided. In addition very complex topics can be
understood quickly and can be structured at the same time. They can then be used to validate
empirically gained data. The method also serves to validate large amounts of data through various
researchers simultaneously as the results in the maps created are provided in a way which is
transparent for everybody. Due to this they assist researchers in familiarising themselves with
different topics for which they do not necessarily have to be experts.
There are also some disadvantages, although in our opinion they do not diminish the general
suitability of the proposed mapping technique, as the advantages clearly outweigh them:
Extensive literature cannot usually be displayed in one map alone.
Normally, it is not possible to use only single terms in the maps, as they do not reflect a
meaningful content. Instead of this parts of a sentence have to be shown.
Experts of the respective subject compile independently, within the scope of this mapping
process, maps from their field of research, and make them available to others. This offers
researchers the possibility in areas they aren't well experienced in to get a quick overview about
these new topics.
Furthermore, a mind map could be also split in two parts, for the reason that several hierarchy
levels could seem blind analogously of the above action.
Challenges and problems occurred when grouping aspects on the first hierarchic level, as here it
was often necessary to provide synonyms or short descriptions for the topics addressed. In the
next lower hierarchic level all the aspects were collected which could be allocated to such a topic.
The same principle was used for all further hierarchical levels.
Furthermore, it became evident that it is necessary to split large portions of data into several submaps,
to be able to display it in a well-arranged way, and make it legible for the reader. Moreover, it
became apparent, due to visualisation reasons, that the option is a single-levelled hierarchical map.
Based on such a map, again single-levelled or two-tiered hierarchies should be chosen for the next
lower level, whereby more details of the superior aspect are displayed. When a larger amount of
single aspects exist, it makes sense to split these aspects into two or more maps in order to maintain
visual lucidity.
7. Conclusion, lessons learned and contribution to knowledge
In our study a number of requirements could be derived from the literature reviewed, which then
formed the framework within which first conclusions could be drawn about how improved
telecommunication systems, as used in railway telematics, can benefit from new innovative
connectivity and mobility platforms.
Both authors were able to acquire additional knowledge and understanding and to gain new
experience during the research project:
The mapping technique supports researchers in understanding different topics, for which they
need not necessarily be experts.
The process, as described in this article, supports the validation of large amounts of data through
different researchers, as the maps that are generated as results are set out in a transparent
manner for all those involved.
It is possible to validate the data derived from the empirical study with those categories generated
via the mapping process.
This mapping process is at the present time being used successfully in two other research studies
(Bergaus 2010a and 2010b, Stottok 2010). From our point of view, it has therefore proven its practical
usability, especially as the following points were able to be achieved:
77

Martin Bergaus, Bernd Stottok and Andrea Gorra
Relevant aspects can be systematically derived from extensive literature. Accordant hierarchies
and correlation are, at the same time, graphically reprocessed.
Generating structured information trees from literature based data, in the form of maps, enables
the possibility of rapid acquisition of until then unknown topics.
Complex topics can be comprised quickly and at the same time they can be structured. Hence
this method helps to validate large amounts of data through different researchers, as the results
of the maps created are facilitated in a transparent way.
When using a Grounded Theory approach, it is feasible as per the concept of theoretical sensitivity
(including pre-gained knowledge about a topic), to use the aspects taken from a map to validate
empirical data, if the latter has already been coded independently and as long as the inductive data
analysis process has not been manipulated by information gained from literature.
References
Bergaus, Martin (2010a) Mind- and concept mapping method for literature based criteria catalogue, Grounded
Theory Research Salon, 18 January 2010, Leeds Metropolitan University.
Bergaus, Martin (2010b) Design aspects on service delivery platforms by analysing perception of ubiquitous
services – Data collection and methodology, Grounded Theory Research Salon, 18. January 2010, Leeds
Metropolitan University.
Bergaus, Martin N. and Stottok, Bernd O. (2010) Verbesserter Nutzen von Telematiksystemen im
Schienenverkehr durch Bereitstellung innovativer Service-Delivery-Plattformen, [online], Master Thesis,
Danube University Krems, stottok.rollsport-nuernberg.de/.
Brühlmann, Marina (2007) Sind Mind Mapping und Concept Mapping effektive Lernstrategien? Department for
Psychology, University Freiburg.
Buzan, T. and North, V. (1999) Business Mind Mapping: visuell organisieren, übersichtlich strukturieren,
Arbeitstechniken optimieren, Wirtschaftsverlag Carl Ueberreuter, Frankfurt and Vienna.
Buzan, T. (2003) How to Mind Map: Make the most of your mind and learn to create, organize and plan,
Thorsons, London.
Charmaz, Kathy (2006) Constructing Grounded Theory: A Practical Guide Through Qualitative Analysis, SAGE
Publications, London.
Glaser, Barney G. and Strauss, Anselm L. (1967) The Discovery of Grounded Theory: Strategies for Qualitative
Research, Aldine de Gruyter, Hawthorne, New York.
Novak, J. D. And Gowin, D. B. (1984) Learning how to learn, Cambridge University Press, New York
Paris, Rubert and Hürzeler, Peter (2008) Was versteht man unter Grounded Theory? Institut für
Technologiemanagement, University St. Gallen, St. Gallen.
Stottok, Bernd O. (2010) Colour Coding – A Grounded Theory Method to Analyse Data, Grounded Theory
Research Salon, 18 January 2010, Leeds Metropolitan University.
Strauss, Anselm L. and Corbin, J. (1996) Grundlagen Qualitativer Sozialforschung, Beltz, Psychologie Verlags
Union, Weinheim.
78

Ethnographic Research Methods in Business and
Information Systems Research
Ann Brown and Jessica Iacono
Cass Business School, City University, UK
a.p.brown@city.ac.uk
J.C.Iacono@city.ac.uk
Abstract: Ethnographic research has its roots in anthropological studies, where researchers would travel to
faraway places to study the customs and practices of less known societies. Ethnographers try to immerse
themselves in a setting and become part of the group being investigated in order to understand the meanings
that actors put upon phenomena or situations. Since organisations can be viewed as societies with their own
peculiar customs and practices, ethnographic research is well suited to the study of management in organizations
and can yield results which are inherently more valid and relevant to the organizational actors. This produces an
extra-ordinary depth of knowledge on the context of the research study and can therefore produce rich insight
into the problem. First-hand knowledge of what happens in the field may contradict commonly held assumptions,
highlight hidden agendas and reveal fundamental misconceptions. Nonetheless, it still forms only a small
subgroup of empirical studies in Information Systems. The method is potentially expensive in terms of researcher
time and presents a number of problems in application. This paper reviews the recent work carried out in
Information Systems studies using this method.
Keywords: ethnographic methods; participant observer, empirical Information Systems research methods,
professionally qualified doctoral student
1. Introduction
Ethnographic research methods can yield extra-ordinarily insightful results but are costly for the
ethnographer in terms of time and are complex in application. Hence for any researcher in IS, the
decision to use these methods has serious consequences. The case for adoption needs to be thought
through carefully. This paper aims to establish the type of Information Systems research question for
which it is a good choice of research method.
Ethnographic research has its roots in anthropological studies, where researchers would travel to
faraway places to study the customs and practices of less well known societies (Sanday, 1979;
Fetterman, 2010). Ethnographers seek to immerse themselves in a social situation in order to
understand how the group and its individual members see themselves and how they ascribe meaning
to the various phenomena that affect them as a group. To achieve the potential benefit of this method
the ethnographer usually needs a lengthy period living with the target social group – hence the high
cost of the method. The method yields large benefits for research questions addressing inexplicable
or puzzling or unwelcome group behaviour. The approach offers the researcher an opportunity to
achieve a greater understanding of the underlying factors that drive people’s actions (Myers, 1999,
2009).
Since organisations can be viewed as societies with their own peculiar customs and practices,
ethnographic research has great potential for the study of management in organizations (Myers,
1999; Evered and Louis, 1981). For a similar reason it has an important place in studying the impact
of new Information systems on organisations. Nonetheless, it still forms only a small subgroup of
empirical studies in Information Systems (Mingers, 2003; Leidner and Kayworth, 2006; Tan and Hall,
2007).
The next section outlines the characteristics of ethnographic research methods. Section three
establishes the potential value of the method for some of the current Information Systems research
questions. Section four outlines the main costs and problems of employing this research method. The
special case of the professionally qualified doctoral student (PQDS) (Klein and Rowe, 2008) is
presented in section five as one way to solve some of the more intractable problems associated with
the technique. Section six assesses the quality and value of recent Information Systems research
applying this method. The paper concludes with the type of Information Systems research question
and situation for which this technique is well suited.
79

Ann Brown and Jessica Iacono
2. Key characteristics of ethnographic methods
This is a methodology based around extensive empirical research. ‘Fieldwork is the heart of the
ethnographic research design’ (Fetterman, 2010). The ethnographer can expect to spend
considerable amounts of time in the field, typically between 6 months to a year (Sanday, 1979; Van
Maanen, 1988; Yin, 1994; Klein and Myers, 1999). He or she immerses themselves in the field so that
they come to see how life and the phenomena under investigation are viewed by the social group
and/or organisation. The researcher seeks to see what people and the organisation actually do as
well as what they say they do. A key aim is to understand how the subject group interacts with each
other and with the specific social and cultural context of the time and place of the research, when
dealing with the phenomenon under investigation. Hence the study results will be critically affected by
the contextual circumstances. When behaviour fails to match expectations, this research method
enables the ethnographer to critically assess the taken for granted assumptions on which such
expectations are based and perhaps to create a better understanding of the drivers for individual and
group behaviour (Myers, 1999). First-hand knowledge of what happens in the field may contradict
commonly held assumptions, highlight hidden agendas and reveal fundamental misconceptions
(Myers, 1999; Schultze, 2000).
Ethnographic methods include a wide range of possible approaches and the researcher has to make
a series of decisions on these, early on in the project. Fetterman (2010) suggests that a key choice
set by the original aim is between basic research (concerned with theory development) or applied
(with direct practical implications). The main schools of thought vary in their view as to what type of
relationship the ethnographer should aim for when immersed in the social setting (Sanday, 1978; Van
Maanen, 1988). The options vary from ‘going native’ to soak in the way of life of the target group to
the semiotic school in which the interpretation of the symbols used by the social group is the
paramount objective. The researcher can choose various ways to represent their work including for
example performance ethnography (McBride, 2008), autoethnography and confessional writing
(Shultze, 2000). Although ethnographic methods are most closely associated with the
phenomenological philosophic paradigm, in fact these methods can be and have been applied within
all three of the dominant philosophical perspectives accepted by Information Systems researchers -
positivist, interpretivist and critical (Orlikowski and Baroudi, 1991).
Data collection, analysis, interpretation and reporting are carried out in parallel. The results of one
activity can and often do alter the direction of the others. As the data accumulates from conventional
sources (documents, interviews, surveys) and from personal observations of events and human
action, the ethnographer’s ongoing analysis is likely to change his or her early ideas (Fetterman,
2010; Myers, 1999; Van Maanen 2010). The enormous volume of relatively unstructured data
produced presents the researcher with a major challenge to create order out of this rich picture. There
are no routine pathways or techniques of analysis. The writing up of notes, theories and reports is at
the core of this process of analysis and interpretation. This is probably the most demanding and
creative part of the project but the stage on which the success of the whole project rests (Van
Maanan, 1988). The ethnographer is making choices and applying personal judgements at all stages
of the project and hence his/her role becomes an important element in the final shape of the project
and the results obtained. The evaluation of the quality of the research and credibility of the final
results remains a challenging activity. The inclusion of participant observation poses problems of
assessment that differ from all other research methods.
Ethnographic research methods pose major challenges for the researcher but for situations involving
human actions and choices it may offer the only way to develop workable theories.
3. Potential Value for information systems research
The long running concern of researchers and practitioners alike as to the disappointing return from
investment in IT (McKinsey Global Institute (MGI) report, 2002; Farbey et al, 1993; Sauer and
Cuthbertson, 2003) has been ascribed to many factors. Nonetheless it is clear that some of these
poor results are due in some part to our lack of understanding of how organisations and individuals
actually interact with new IT systems. Hence many of the research questions of interest to Information
Systems researchers and practitioners deal with managerial and organisational issues raised by
contemporary information systems developments where the context interacts with innovation
(Benbasat et al, 1987).
80

Ann Brown and Jessica Iacono
It is problems of this nature that have triggered the recurring issue of the relevance of published
Management Information Systems (MIS) research. The call for rigorous research into MIS, which is of
practical relevance to Information Systems professionals, has been raised repeatedly in the MIS
literature over the last decade (Benbasat and Zmud, 1999). In an interesting paper on scholarship and
Practice, Harvey and Myers (1995) agree that there is a gap between the way practitioners and
researchers generate knowledge about IS. They suggest that ‘ethnography provides the principles to
support both groups, hence bridging this gap’ (Harvey and Myers, p13, 1995).
For research questions relating to the effective organisational exploitation of IS, case study is the
most commonly used qualitative research method (Myers, 2004) and has been regarded as a
particularly effective research method (Benbasat et al, 1987; Darke et al, 1998; Walsham, 1993;
Myers 2004). Nonetheless case methods have their limitations. Data collection is from existing
sources of: documents, surveys and interviews with participants. If the participants do not understand
the situation or the IS, then interviews will not necessarily help to understand the phenomena. The
immersion of the ethnographer in an environment can enable the collection of a level of detail not
available through conventional data collection methods used in case studies. The ethnographer will
use all the data collection methods used for case studies but will deepen his or her knowledge
through being present to see and observe events and human action (Fetterman, 2011; Myers, 1999,
2009). This produces an extra-ordinary depth of knowledge on the context of the research study and
can therefore produce rich insight into the problem.
Traditional quantitative research methods can produce significant results for a narrow group of
problems in Information Systems research. Many of the urgent (relevant) questions of Information
Systems development and implementation pose intractable problems for the researcher. For these
problems ethnographic research methods may offer a more promising approach than traditional
research methods.
4. Problems in application
This section outlines the main challenges in adopting ethnographic methods – the practical issues,
the research process itself and evaluating the quality and value of the work.
4.1 Practical problems
The main practical problems associated with ethnography include time, access to the case site(s), the
potentially ambiguous role and/or isolation of the participant observer (the ethnographer) and ethical
issues (Myers, 1999; Schultze, 2000; Fetterman, 2011; Harvey and Myers, 1995). The researchers
research aims would normally determine such decisions as the scope of the investigation in terms of
the amount and type of data to be collected, the number of sites and/or cases to include and time
period to cover. However there will be practical limitations dictated by for example the time that the
ethnographer can afford to spend immersed in the organisation and the contribution of the
collaborating organisation. The requirement for extended access to or membership of the social group
under scrutiny, brings up the question as to what the organisation can or will offer in terms of access
to key staff, documentation on key decisions and the role that the ethnographer can take within the
work group. The researcher will need to fit the research question to a design that can be
implemented. The role of the ethnographer in the initial stages can be awkward. He or she is new to
the organisation and initially has no clear job. Finding a niche where he or she can contribute and reassuring
the natural suspicions of the workers with whom they wish to interact can take time. The
report may raise ethical issues in the choice of what to include.
4.2 The research process
The period during which the empirical work is carried out constitutes a major (often the largest) part of
the project in terms of elapsed time and project work. All the steps in this process are intimately
interconnected with decisions made at any one stage likely to have an effect on both previous
decisions as well as the ones yet to come. The direction of the research (even the research question
and initial boundaries) may undergo a number of changes as the project progresses. The challenge
for the researcher is to carry out the research in such a way as to be able to produce a convincing
report.
The use of theory has been a point of contention for many writers. While the ethnographer is enjoined
to work in a non judgemental way, rigorously excising bias, theory is important for setting the
81

Ann Brown and Jessica Iacono
boundaries and the aims of the project and in also directing the data collection stage (Fetterman,
2010). Criticisms have been voiced about some ethnology studies ‘just providing common sense
descriptions, rather than theoretically grounded conceptual analyses’ (Hammersley,1992 cited in Tan
and Hall, 2007).
Participant observation can be a very powerful technique for data collection, but presents the
researcher with unique challenges (Evered and Louis, 1981). These include: finding a role acceptable
to the social group or organization, accurately assessing the effect that the presence of the
investigator has on the informants, and managing the analysis and reporting of the findings, so as to
gain the insight without losing sight of the need for some degree of rigour and objectivity.
Management research presents challenges of its own. Managers are busy individuals, and are
typically reluctant to allow access unless they can see some benefit to the organisation. Hence,
access for fieldwork may be difficult to obtain, and, if granted, it may be subject to various conditions
about confidentiality. In the most conventional sense the researcher/observer is an outsider. While in
the field, the researcher participates in the daily routines of the group being investigated, but remains
an outsider and never completely blends in. Hence, participant observation carries with it the concern
that the presence of the investigator may influence the way informants behave. Informants may be
suspicious of the researcher and reluctant to participate or be eager to please etc. Personal
relationship between researcher and informants may also influence the interaction (e.g. the
researcher may empathise with his/her informants and vice versa). The informant may mislead the
researcher in many ways and for many reasons, from self-preservation to peer protection, ignorance,
etc. (Van Maanen, 1979).
The analysis and interpretation stage pose the greatest challenge. The data collected tends to be
predominantly qualitative, relatively unstructured, and, as is often the case with qualitative research,
the sheer volume of data (potentially far more than that produced in case studies) can be
overwhelming and difficult to manage (Miles and Huberman, 1984). There are no clear paths. The
researcher is enjoined to look for the patterns on which the final analysis and report will be based
(Fetterman, 2010). The early writers on ethnography worked in sociology or anthropology and tended
to make suggestions on specific issues that are likely to occur in these studies such as culture.
Analytical techniques included such ideas as organisation charts, identification of key events and
content analysis. These may be less relevant to Information Systems researchers. What will be highly
relevant is knowledge and understanding of the technology artefact under investigation. However
what comes through very clearly is that this stage is the most unstructured one and the one in which
the results depend critically on the skill and creativity and insight of the ethnographer (Van Maanan,
1979a). This is the stage at which the ethnographer routinely redesigns data collection to include
factors that now appear important, which at the start did not. The process can jump backwards and
forwards between analysis and data collection many times until the final picture is deemed to have
become clear enough for the report to be written. Since writing up is a key step in understanding the
data patterns, it is perhaps the most important tool for analysis at the researchers command. The
process of writing is in effect an integral part of the analysis and interpretation of the data. Hence the
emphasis again and again on the importance of writing up experiences, interviews and emerging
ideas literally on a day to day basis throughout the period of immersion in the workplace (Myers,
1999; Van Maanan 1979b, Fetterman, 2010).
Much more has been written on how to write up ethnology research (Myers 1999, Van Maanen, 1988;
Golden-Biddle and Locke, 1993; van der Blonk, 2003; McBride, 2008). Writing a case study is a
process of selecting, ordering, editing, shaping the case material into a narrative which makes sense
and serves a purpose – to describe, explore or explain the phenomenon, depending on the aim of the
research. No two studies are the same, and ethnographers have experimented with different forms of
writing. Each typology has some merit in that it addresses the problem from a different perspective.
Van Maanen’s ‘Tales of the Field’ (1988) is generally regarded as the definitive work on ethnographic
writing. The author identifies three narrative forms, namely, realist tales, confessional tales and
impressionist tales, reflecting the choices the researcher as writer needs to make with regards to how
to present the findings, what to include or omit, and what perspective to adopt, the researcher’s or the
actors’. Thus, realistic tales provide a direct, factual representation of the phenomenon under study;
confessional tales reflect the researcher’s personal experience and perspective, and impressionist
tales are dramatic reconstructions of fieldwork stories. Van Maanen recognises that these are to be
seen as loose types and that in the practice of ethnography the distinction is sometimes blurred.
82

Ann Brown and Jessica Iacono
At every stage in the research process, the role of the ethnographer is a critical one. The social world
is potentially unstable, with many possible realities. It is complex involving many variables both
perceived, hypothesised and unknown. The most significant factors are people and their interaction
with each other as well as with the technology artefact. Research questions tend to raise issues for
which the boundaries may well be unclear initially. The ethnographer’s series of choices throughout
the project will determine many aspects of the research and the value and quality of the final results.
Hence the call for final reports and journal papers to include information about the individual’s prior
experiences, attitudes and beliefs as well as the pathway they took through the research study and
the reasons for the decisions taken.
4.3 Evaluation of the research
Good research practice calls for critical evaluation by the authors (Klein and Myers, 1999). Convincing
your readers that the work reported has met agreed standards of rigour and produced valuable
results, of relevance to them, presents a complex problem for ethnographic research methods.
Traditional research methods (of for example positivist science using quantitative tools) have well
understood criteria by which the rigour of application can be assessed and the limitations of the
degree of generalisability of the research results established. Researcher objectivity is a key aim for
these type of methods. The research results convince through their adherence to accepted standards
and practices that other researchers can follow and replicate. Ethnographers deal with social systems
that contain as many realities as there are members and moreover evolve over time. There is a less
clear cut pathway for the steps of the research process. Researcher objectivity is inappropriate -
indeed it is in the ethnographer’s personal vision and individual perceptions that the strength of the
work is likely to lie. How do we the readers decide that a valid, valuable version of the social world
has been created and in the case of Information Systems research workable guides developed for the
effective exploitation of IT artefacts?
According to Golden-Biddle and Locke (1993), ethnographic writing must convince readers on two
levels – that the researcher has carried out a credible piece of research (the steps of the project have
been carried out well) and that the findings and their interpretation appear credible and relevant to the
reader. In contrast to the traditional scientific method, ethnographers reports depend critically on our,
the reader’s belief in their authority as researchers. To achieve this authority, Golden-Biddle and
Locke (1993) propose three qualities that reports should seek to achieve - authenticity, plausibility and
criticality. These are the criteria by which the reader can also assess the quality and credibility of the
work reported.
Authenticity focuses on the role of the ethnographer. How well does the text establish that the
ethnographer has indeed been in the field for a period of time and at a level of immersion that would
enable them to obtain firsthand experience of the social world under investigation? Does the text
convince the reader that the researcher has truly grasped the key aspects of this social world – is it
authentic to the member’s perceptions?
Plausibility focuses on the quality of story told, the findings and their interpretation. This is where the
reader’s experiences, discipline background and concerns come into play. The story has to make
sense to the reader. It has to be familiar, relevant and distinctive and add to the existing body of
knowledge in the field (Myers, 1999).
Criticality focuses on the impact of the work on the reader. Does the text contain thought-provoking
insights which stimulate the reader to question their assumptions and consider new possibilities?
According to Golden-Biddle and Locke (1993) a text achieves criticality ‘ by disrupting readers’
common sense and enabling them to step back and to reflect on and question the personal
assumptions underlying their work’.
5. Professionally qualified doctoral students (PQDS))
Klein and Rowe (2008) have pointed out the potential value of doctoral students who come from the
professional community of IT professionals with some years of work experience (Professionally
Qualified Doctoral Students - PQDS). They have a high level of professional expertise in Information
Systems and start with a profound understanding of the questions that concern practitioners. Their
knowledge is likely to be undocumented or tacit but will inform their choice of research subject. A
83

Ann Brown and Jessica Iacono
doctoral programme designed for this group would be in a position to leverage their professional
knowledge and contribute research that was relevant to both Information Systems researchers and
Information Systems professionals. Moreover these students are well placed to carry out
ethnographic research within their organisations particularly if they are working on their doctorate part
time, while continuing with a full time or part time job. Most have strong ideas about the research topic
which is almost always related to questions raised by their professional experience. Hence the
process of establishing the research question can be relatively quick.
PQDS are in a position to manage many of the practical problems of ethnographic research outlined
above. They have the contacts to arrange access to appropriate groups and/or sites. They are in a
good position to get agreement from their employers for the level of co-operation needed. They are
part of the working group and have a head start on the empirical work from their inside knowledge of
the culture and routines of the organisation. They have a role and do not need to spend time getting
accepted (Iacono et al, 2009).
Their position also offers unique benefits with respect to data collection. They are in an excellent
position to collect information by informal means. They understand the context at a gut level. They are
well qualified to identify the most appropriate interviewees or through their company network find the
right members of staff. Perhaps the only really tricky aspect is working out how to be effective in
communicating the research and keeping all participants aware of developments, without
compromising confidentiality.
6. Applications in information systems research
Case examples of ethnographic research methods used in Information Systems research are still few.
The way in which they are written up varies widely. Key questions for each case include:
Is the research question one that can benefit from this approach. Is it worth the cost of the
empirical work to address the question posed?
The quality of the description of the research process. What confidence do we the reader have in
the rigour with which the work has been carried out? Does it convince us?
The quality of the final results and clarity of the presentation of these results. Does the work tell us
something unexpected?
Four case examples of Information Systems research based on ethnographic methods are assessed
below - ‘Using Performance ethnography to explore the human aspects of software quality’ by Neil
McBride (2008); ‘A confessional Account of an Ethnography about Knowledge Work’ by Ulrike
Schultze (2000); ‘Hidden Agendas, power and managerial assumptions in information systems
development; An ethnographic study’ by Michael Myers and Leslie Young (1997) and ‘Telemedecine
in the Upper Amazon: Interplay with Local health Care Practices’ by Gianluca Miscione (2007).
Is this method appropriate for the proposed research question/aim? Appropriate questions would be
those that deal with an Information Systems artefact or concept that is ill understood by practitioner
and researchers alike being implemented or installed within an organisation in which many groups
and individuals are affected by its presence in a way that is also complex and opaque. All four case
examples conformed to this and hence ethnography was an appropriate choice for each. Schultze
(2000) investigates Knowledge Work (ill understood) in order ‘to develop a grounded understanding of
what producing information entails and what Knowledge Work is like’. Neil McBride (2008) started
from the unusual view that software quality is as much affected by the social context and the
developer’s personal skills as by rules and standards. The introduction of the social context makes
ethnography a promising approach. Myers and Young (1997) also start from the position of
considering the social context as key in their investigation of the introduction of a new Information
Systems into the New Zealand Health Sector. Miscione (2007) evaluates the new Telemedecine
system implemented in Peru in terms of its effects on a wide number of stakeholders (public health
care system, local health care practice, local population, patients, traditional healers). The large
number of the user groups, the differing roles that each carries out as well as the variety of roles that
Telmedecine can play make ethnography a natural choice.
Is there a comprehensive description of the research process? Do we have confidence in the results?
Only Shultze’s (2000) paper meets this standard. She explains at length the whole process and offers
an in depth evaluation of her own work using the Biddle and Locke (1993) criteria. This inevitably
84

Ann Brown and Jessica Iacono
means that the paper is long. Miscione (2007) gives an excellent description of the research process
and how he reaches his conclusions. The only omission is any attempt at self evaluation. McBride
(2008) applies performance ethnography which naturally focuses on himself and his experience of
software development. The research process is described well as it is interwoven with the final
presentation. As for the previous case there is no attempt at self evaluation. Finally Myers and Young
make little attempt to describe the research process. Most of the paper is taken up with the theory that
they were applying (Habbermas’ model of societal development).
Do the results reveal something new – perhaps unexpected? All four papers revealed interesting and
to some extent unexpected results. McBride (2008) uses the unusual performance techniques to
stunning effect to describe both the research process and present the results. The results are counter
the received view about how to achieve quality software. Miscione (2007) presents his work extremely
effectively leaving us with a deep understanding of how the various stakeholders react to and use the
new Telemedecine system. These reactions are not what had been planned for or expected when the
new system was designed and installed. Schultze is less concise but also leaves us with a deep
understanding of knowledge work and in no doubt of ‘subjectivity and objectivity as a duality in
knowledge work’ (Schultze ,p36, 2000). This is in opposition to the main stream of research with
respect to Knowledge Work, which aims to create a structured approach to knowledge creation and
dissemination – seeking to excise subjectivity. Myers and Young (1997) demonstrate that the
Habbermas model can be applied to a part of the project operation. The conclusions are less than
satisfying but support the significance of social factors for the successful exploitation of information
systems whish at the time was not the main stream view of practitioners or researchers.
7. Conclusion
Ethnographic research methods are powerful but demanding on the researcher. They are costly in
terms of researcher time. Ethnographers face many challenges in taking a research project to final
completion. The quality and value of the final results is critically dependent on their skills, knowledge
and attitude.
However these methods offer the possibility of obtaining a rich understanding of how IT artefacts
interact with organisations and their employees. They are particularly appropriate for questions for
which existing research results are ambiguous or contradictory. For those problems where the
Information Systems phenomenon under investigation is unclear, complex and embedded in a social
system that is poorly understood or even unknown, ethnographic research methods are likely to yield
more valuable results and insights than other methods (Tan and Hall, 2007; Harvey and Myers, 1995;
Myers, 1999), and be worth the extra effort required.
References
Benbasat I, Goldstein D and Mead M (1987) ‘The Case Research Strategy in Studies of Information Systems’
MIS Quarterly 11. pp369-386
Benbasat I and Zmud R (1999) ‘Empirical Research in Information Systems: The practice and Relevance’ MIS
Quarterly vol 23 issue 1 pp3-16
Darke P, Shanks G and Broadbent M (1998) ‘Successfully completing case study research: combining rigour,
relevance and pragmatism’ Information Systems Journal 8, 273-289
Evered, R. and Reis Louis, M. (1981) ‘Alternative Perspectives in the Organizational Sciences: ‘Inquiry from the
Inside’ And ‘Inquiry from the Outside’’ in Academy of Management Review, Vol. 6, No. 3, pp. 385-395.
Farbey B, Land F and Targett D (1993) ‘How to assess your IT investment’ Management Today
Fetterman D (2010) ‘Ethnography step by step’ (3 rd edition) Applied Social Research Methods Series volume 17
Sage Publications
Golden-Biddle K and Locke K (1993) ‘Appealing Work: An Investigation of how Ethnographic texts Convince’
Organization Science vol4 no 4
Harvey L and Myers M (1995) ‘Scholarship and practice the contribution of ethnographic research methods to
bridging the gap’ Information technology & people vol 8 (3) pp13-27
Iacono J, Brown A and Holtham C (2009) ‘Research Methods – a case example of Participant Observation’
Electornic Journal of Business research Methods vol 7 issue 1, December 2009
Klein, H. K. and Myers M (1999) ‘A Set of Principles for Conducting and Evaluating Interpretive Field Studies in
Information’, MIS Quarterly, Special Issue on Intensive Research (23:1), pp. 67-93.
Klein and Rowe (2008), ‘Marshalling the Professional Experience of Doctoral Student: A Contribution to the
Practical Relevance Debate’ MIS Quarterly 32(4) pp675-686 December 2008
Leidner D and Kayworth T (2006) ‘A review of Culture in Information Systems Research: Towards a Theory of
Information Technology Culture Conflict’ MIS Quarterly Vol 30 (2) , pp. 357-399
85

Ann Brown and Jessica Iacono
McBride N (2008) ‘Using Performance ethnography to explore the human aspects of software quality’ Information
Technology & People vol 21 (1) pp91-111
McKinsey Global Institute (2002) ‘ How IT enables productivity Growth’ at
http://www.mckinsey.com/knowledge/mgi/IT/
Mingers J (2003) ‘The Paucity of multimethod research: a review of the information systems literature’
Information Systems Journal vol 13 pp233-249
Miscione, G (2007) ‘Telemedecine in the Upper Amazon: Interplay with Local health Care Practices’ MIS
Quarterly vol 31 (2) pp403-425
Myers M (1999)’ Investigating Information Systems with ethnographic Research’ Communications of The
Association for Information Systems vol 2 article 23
Myers, M. D. (2004) "Qualitative Research in Information Systems," MIS Quarterly (21:2), June 1997, pp. 241-
242. MISQ Discovery, archival version, June 1997, http://www.misq.org/discovery/MISQD_isworld/. MISQ
Discovery, updated version, last modified: February 27, 2004 www.qual.auckland.ac.nz
Myers M (2009)’ Qualitative Research in Business and Management’ Sage
Myers M and Young L (1997) ‘Hidden Agendas, power and managerial assumptions in information systems
development; An ethnographic study’ Information Technology & People vol 10 (3) pp224-240
Orlokowski W J and Baroudi JJ (1991) "Studying Information Technology in Organizations: Research
Approaches and Assumptions", Information Systems Research (2) 1991, pp. 1-28
Sanday PR (1979) ‘The Ethnographic Paradigm(s)’ Administrative Science Quarterly vol 24 pp527-538
Sauer and Cuthbertson (2003) ‘The State of IT Project Management in the UK 2002-2003’ final report from the
Computer Weekly Project/Programme Management Survey funded by Computer weekly, Templeton
College and The French Thornton Partnership (see also Computer Weekly November 4 th 2003 ‘Hitting
targets? The state of UK IT project management’ P22)
Schultze U (2000) ‘A confessional Account of an Ethnography about Knowledge Work’ MIS Quarterly vol 24 (1)
pp3-41
Tan M. and Hall W (2007) ‘Beyond the Theoretical and Methodological Pluralism in Interpretive IS Research: The
example of Symbolic Interactionist Ethnography’ Communications of The Association for Information
Systems vol 19 pp589-610
Van de Blonk H (2003) ‘ Writing case studies in information systems research’ Journal of Information Technology
vol 18 pp 45-52
Van Maanan (1988) ‘Tales of the Field: On Writing Ethnography’ University of Chicago Press
Van Maanan (1979a) ‘The territory is not the map’ Administrative Science Quarterly vol 24 pp 539-550
Van Maanan (1979b) ‘The Fact or Fiction in Organizational Ethnography’ Administrative Science Quarterly vol 24
pp 539-550
Walsham G (1993) ‘Interpreting Information Systems in Organizations’ Wiley Series on IS
Yin R (1994, 2003) ‘Case Study Research Design and Methods’ second and third edition Applied Social
Research Methods Series, Volume 5, Sage Publications.
86

Quality Criteria for Interpretive Research in Information
Systems: A Reflection on Klein and Myers’ set of Principles
Ana Cardoso and Isabel Ramos
University of Minho, Guimarães, Portugal
anachcardoso@gmail.com
iramos@dsi.uminho.pt
Abstract: In the last two decades, interpretive research has become more established and more popular in
information systems (IS) field (Walsham 2006), (Walsham 1995a). Publications with detailed information and
practical guidelines for conducting this type of studies have contributed to grow confidence in interpretive
methods, as for example (Walsham 1995b), (Walsham 2006), (Myers 1997), and (Klein & Myers 1999). The work
of Klein and Myers (1999) consists of a set of principles for conducting and evaluating interpretive research,
which provide fair and appropriate criteria for judging the validity and reliability of such studies. In this article we
focus on understanding how this set of principles has informed research articles published in the highest-ranked
IS journals (according to Association of Information Systems): Management Information Systems Quarterly and
Information Systems Research. We question if these principles have been translated into common practices
when conducting interpretive research in the field of information systems and whether authors incorporate them
explicitly when they communicate the results of their research. Articles published in these two journals are
recognized as finest exemplars of rigorously conducted research and thus represent a relevant sample for this
exploratory study. This work can be extended in the future to include more journals. This research is empirical
and the analysis is done at the level of articles published in those two journals. We browsed the entire collection
of articles published in these journals between 1999 and 2009 and selected research papers reporting field
studies of interpretive nature. Selected articles were read, with particular focus in the methodology section, and
explicit or implicit evidence of quality criteria that informed the research was collected. Direct or indirect reference
to Klein and Myers’ set of principles was highlighted. We summarize and compare our findings in a
comprehensive table, and note that, apparently, the principle of hermeneutic circle is the most explicitly
discussed in this sample. Moreover, Klein and Myers’ set of principles seem to have had a greater influence in
Management Information Systems Quarterly publications, where several articles illustrate the compliance with
such principles with examples. This is particularly visible in the papers published in the period from 2002 to 2006.
This study aims at contributing to the literature of qualitative research methodology in information systems field,
and provides a reflexion about methodological rigor in interpretive research that, to our knowledge, had ever
been done. Thus, the findings here presented may be useful for junior researchers and doctorate level students
to understand how validity and quality criteria are enacted in high-quality research.
Keywords: interpretive research evaluation, quality and rigor criteria, information systems, Klein & Myers’ set of
principles, hermeneutics
1. Introduction
Research in information systems (IS) has had a long positivist tradition. However, qualitative research
methods, in particular interpretive studies, have been gaining their own place within IS research
community, spurred by serious and valuable contributions under various themes that have been
published in past decades. The problem of rigor and relevance of interpretive research is central in
the debate about using appropriate research methods in the IS field because interpretive researchers
often face misjudgements about the rigor of their research, as they are sometimes required to satisfy
criteria that are not adequate to their genre of interpretive research (Sarker 2007), (Klein & Myers
1999).
The paper entitled “A Set of Principles for Conducting and Evaluating Interpretive Field Studies”, by
Klein and Myers (1999) is a significant contribution to the topic of quality and validity criteria for
interpretive research in IS field. Several pieces of interpretive studies have been published since. We
collected a sample of high-quality interpretive research papers published after 1999 and assessed the
methodological rigor and quality criteria concerns that have driven the researchers. In doing so, we
highlight the explicit reference to Klein and Myers’ set of principles, the implicit form of the principles
in the report, and any other quality and validity criteria cited by the authors.
This work aims at contributing to the literature of qualitative research methodology in IS area, in
particular interpretive studies. The interested audience are junior researchers and doctorate level
students that intend to undertake interpretive fieldwork. Our contribution provides an informative
account of how authors of these top-quality journals conceive methodological rigor when
communicating their research and how Klein and Myers’ (K&M) set of principles are enacted. This
87

Ana Cardoso and Isabel Ramos
reflection can help junior level researchers develop a critical view of their readings by having as a
reference this sample of high quality articles.
The structure of this paper is as follows: the next section discusses quality and validity criteria for
conducting and evaluating interpretive research in the context of IS literature. In section 3, we explain
the methodology for this article, that is, how did we get to the results. Finally, we discuss the results in
section 4.
2. Quality and validity criteria for interpretive research
The concerns of validity and quality of a research contribution are not exclusive of interpretive
research. To establish credibility and validity of a research study for an audience is necessary to
demonstrate care in reporting, which is both a matter or argumentation style and accuracy of methods
used (Walsham 1995b). The justification of methodological approaches is often done on the basis of
criteria utilized or developed by very experienced researchers in particular methods, disciplines or
strands of thought. For example, the dimensions of authenticity, plausibility, and criticality (Golden-
Biddle & Locke 1993) that ethnographic texts are required to develop in order to create a convincing
argument.
Another example is the set of principles proposed by Klein and Myers (2009). These principles
explore the role of hermeneutics, anthropology and phenomenology in eliciting knowledge from
interpretive studies. Adherence to the principles does not guarantee interesting results per se but,
instead, they are useful for orienting the conduct and report of interpretive research and “can also be
used for post hoc evaluation” (Klein & Myers 1999, p.71). However, they advert that researchers
should not apply them mechanistically but reflect how and if they are useful to any particular project.
The principles overarch the research process and provide tools for understanding the problem studied
as a whole. Therefore, the resulting report will very likely be plausible and convincing.
The hermeneutic principle is a fundamental meta-principle, and the other six principles are in its
dependency. The principles are as follows (in numerical order) (Klein & Myers 1999): the principle of
hermeneutic circle; the principle of contextualization; the principle of interaction between the
researcher and the subjects; the principle of abstraction and generalization; the principle of dialogical
reasoning; the principle of multiple interpretations; and the principle of suspicion. These principles are
not to be thought of in a bureaucratic way but instead in the spirit of fundamental ideas that embed
insights from relevant strands of interpretivism, such as hermeneutics and phenomenology, and also
from the authors’ experience with anthropological research.
Even though a thorough description of each principle can be found in the cited work, we next explain
briefly each one. The fundamental meta-principle is the hermeneutic circle, which suggests that
human understanding is achieved by iterating between the interdependent meaning of the parts and
the whole they form. The parts and the whole can be at various levels, as for example different levels
of observations (researcher and participants) and their shared understanding, various historical and
social aspects and the whole perspective of the context, and also the partial understanding resulting
from a study and the relation with the whole of the extant literature. The principle of contextualization
thus follows from this idea, in that an explanation of the situation under investigation requires a critical
understanding of the social and historical aspects.
The principle of interaction between the subjects and the researchers suggests that there should be a
critical reflection of how research materials were socially constructed through the interaction between
researchers and participants. On the other hand, the principle of abstraction and generalization
recommends that researchers relate the idiographic details revealed by the data interpretation to
theoretical and more general concepts. Furthermore, the principle of dialogical reasoning cautions
that there should be sensitivity to possible contradictions between the theoretical framework guiding
the research and the actual findings. The sixth principle underscores the sensitivity to possible
differences in interpretations of the events under study among the participants, and the principle of
suspicion follows from this one in that the researcher should also be sensible to possible biases and
distortions in the narratives collected from the participants.
The methodological rigor of interpretive research is a concern that has lingered among qualitative IS
researchers. Hence, the aforementioned set of principles was received with enthusiasm and, in many
cases, included in the mandatory readings for doctoral-level qualitative research courses. Therefore, it
88

Ana Cardoso and Isabel Ramos
is reasonable to expect that future interpretive research would incorporate the principles as a
reference work. This is what we wanted to confirm with our study and we report our findings in the
next sections.
3. Methodology
The work of Klein and Myers (1999) has had a significant impact in IS research community and
accounts for an impressive number of citations in for example Google Scholar: more than 2300
citations in January 2012. Thus, we wanted to learn about its influence in the two most reputed
journals of IS research (according to the ranking of Association of Information Systems): Management
Information Systems Quarterly (MISQ) and Information Systems Research (ISR). In the next section
we explain how we sampled the articles and in Section 3.2. we provide a detailed account of our
study of these articles.
3.1 Bibliographical search
We limited the bibliographical search to a ten-year period (1999 - 2009) and to MISQ and ISR journals
because they are representative of highest-quality research in the IS area. We acknowledge this
limitation and contend that this work, being exploratory, may be extended in the future in order to
include a larger sample of journals. As the set of K&M’ principles addresses only the quality standards
of interpretive field studies, we selected only articles reporting this type of studies, that is ethnographic
or case studies of interpretive nature.
Articles were collected from EBSCO database, where a filtering search was first done with the terms
“interpretive field studies” in full text. This search retrieved 36 articles. We perused all 36 articles and
discarded conceptual or opinion articles. Then, we read thoroughly the methodological section of the
remainder articles and categorized each article according to the authors’ description of the
methodological approach. This categorization was validated and discussed between the two authors
of this paper. Articles categorized as positivist case studies, critical research, and mixed methods
research were then discarded. Thus, our bibliographical search resulted in 8 articles from MISQ and 6
articles from ISR reporting ethnographic or case studies of interpretive nature. Table 1 provides the
details of each article and our comment on the quality and validity criteria evoked or implied from the
report. For the sake of readability, articles’ details are displayed sequentially.
3.2 Examination of articles
Interpretive research articles are typically long and dense. In our case, the articles in the sample have
an average of 23 pages length. We learned from the examples provided in Klein and Myers (2009)
how to identify the principles in the narrative. Some articles are not explicit about any validation or
quality criteria that informed their research and we tried to infer them in an implicit form, as Klein and
Myers (2009) explain. Furthermore, we complemented our analysis with the operationalization of the
principles as explained by O’hEocha et al. (2012), which is based on questions derived from
keywords in Klein and Myers’ article.
Table 1: Details and methodological criteria of sampled articles
Article 1 (Choudhury & Sabherwal 2003)
Research Method Multiple case study
1 diversified services company; 1 financial services firm; 1 software manufacturer; 1
Research Site
Focus
large county in Southeast US; 1 large food product firm
Evolution of portfolios of control over the duration of outsourced information systems
development (ISD) projects
Main Contribution New insights for the use of portfolios of control in the clients’ perspective
Methodological
criteria
The report does not refer explicitly to Klein and Myers’ criteria. The analytical process
is described as interpretive and an explanation of the data analysis process is
provided together with the reflection of how the findings relate with the known
literature, adding parallels, differences and new questions.
Article 2 (Schultze & Orlikowski 2004)
Research Method Ethnographic field study
Research Site A bricks-and-clicks dot-com that uses self-serve technology
Focus
Implications of inserting self-serve technology into embedded relationships at the
inter-firm micro level
Main Contribution
Managing the complementarity between relationships and IT in practice is fraught
with tension at the micro-level with negative consequences for business strategies
89

Methodological
criteria
Ana Cardoso and Isabel Ramos
The intellectual basis of the research is clear and mindful of the findings being
contradictory with some established theories. Moreover, the article examines
alternative views and notes different values and opinions expressed by strategic and
tactical teams. Authors do not refer explicitly Klein and Myers’ set of principles.
Article 3 (Levina 2005)
Research Method Ethnographic field study
Research Site Web-based application development project
Focus
Understanding the actual collaborative practices on multiparty ISD projects, and how
the diversity of professionals involved influences the system being designed
Main Contribution
Theoretical constructs to understand the cycle of multiparty collaboration on ISD
projects
Methodological
criteria
Strong in terms of theory building, generalization and reasoning. Multiple viewpoints
are critically presented. Refers to an inductive, grounded fashion way of developing
theory. Does not refer explicitly to Klein and Myers’ set of principles.
Article 4 (Sarker & Sarker 2009)
Research Method Case study
Research Site Multinational high tech organization
Focus
Understanding how agility is developed and enacted in a globally distributed ISD
team
Main Contribution
Development of an empirically grounded framework of agility in ISD settings and
identification of some actionable tactics for enhancing agility within such settings
Authors refer specifically to Klein and Myers’ set of principles in the methodological
Methodological
criteria
section and illustrate with examples from how they conducted the study. In the same
section, they refer also other methodological considerations and cite other authors,
namely (Bryant & Charmaz 2007), (Charmaz 2000), (Flick 1998), (Myers & Newman
2007), and (Patton 1990).
Article 5 (Vidgen & Wang 2009)
Research Method Multiple case study
Research Site Two software development teams
Focus Study the factors that enable or inhibit agility in agile software development
Main Contribution Framework for the organization of agile software development
Methodological
criteria
There is no explicit reference to Klein and Myers’ set of principles. The study is
accounted as interpretive and we could find implicit evidence of principle of
abstraction and generalization and principle of multiple interpretations.
Article 6 (Vaast & Walsham 2009)
Research Method Case study
Research Site French Public Health Administration
Focus Practice-based learning dynamics that emerge in networks of practice
Main Contribution
Formulates a model of trans-situated learning supported by an information
infrastructure
Authors acknowledge that the study was informed by the set of principles of Klein
Methodological and Myers, and explain that the data analysis was made in consideration of the
criteria
principle of abstraction and generalization, principle of hermeneutic circle and
principle of suspicion. We could also find implicit evidence of the other 4 principles.
Article 7 (Davidson 2002)
Research Method Case study
Research Site Health care insurance company in Eastern United States
Focus
Requirements processes in organizations and the socio-cognitive basis of power in
ISD
Main Contribution
Develops a socio-cognitive process model of how frames and shifts in frame
influence sense-making during requirements identification on ISD projects
The researcher provides an appendix with criteria for assessing the quality and rigor
Methodological of her work, where she directly refers to Klein and Myers’ set of principles and to
criteria
criteria used in ethnographic texts, namely authenticity, plausibility, and criticality
(Golden-Biddle & Locke 1993)
Article 8 (Pawlowski & Robey 2004)
Research Method Case study
Research Site Large manufacturing and distribution company based in United States
Focus
The role of IT professionals in internal knowledge transfer, in particular as knowledge
brokers
Main Contribution
Formulates a conceptual framework that shows the conditions, practices and
consequences of knowledge brokering by IT professionals
90

Methodological
criteria
Ana Cardoso and Isabel Ramos
We found explicit reference of Klein and Myers’ set of principles in the method
section another section entitled “evaluation”, where the authors provide a reflection
about their conformance to established principles for evaluating interpretive research.
Besides demonstrating the strengths of their work in terms of adhering to some
principles, as for example the hermeneutic circle, they also account for how the work
could better reflect the principles of multiple interpretations and the principle of
suspicion.
Article 9 (Backhouse et al. 2006)
Research Method Case study
Research Site British standard in information systems security
Focus
The influence of exogenous contingencies for the creation of a de jure standard and
reflection about the power mechanisms required for a standard to evolve
Main Contribution
Deep understanding of how human, social and economical aspects influence the
development of a standard and theorization about power circuits
Methodological In the research methodology section authors provide a table with their reflection of
criteria
how they validate their interpretations in the light of Klein and Myers set of criteria.
Article 10 (Hanseth et al. 2006)
Research Method Case study
Research Site Norwegian hospital
Focus
The intended and unintended consequences of the implementation of a standard for
an electronic patient record
Demonstrate the socio-technical complexity of IS standardization and the related
Main Contribution disorders created by multiple actors that generates reflexive mechanisms and
undermine the creation of a standard
Methodological
criteria
Authors acknowledge that Klein and Myers’ set of principles informed data analysis
and collection and illustrate with the fundamental principle of hermeneutic circle,
though without naming it. We could find implicit evidence of the other principles.
Article 11 (Puri 2007)
Research Method Case study
Research Site A district situated in the rain-shadow zone of peninsular India
The importance of knowledge alliances between multiple knowledge systems in order
Focus
to support effective development and implementation of a GIS to address the
problem of land degradation in India
Main Contribution
Provides an articulation of the concept of building knowledge alliances across
communities of practice and broadens the concept of boundary objects
Methodological Author cites Klein and Myers’ set of principles but does not elaborate explicitly on any
criteria
particular principle. We could find implicit evidence of the principles in the report.
Article 12 (Davidson & Chismar 2007)
Research Method Case study
Research Site Hospital in US
Focus
Interaction between changes in social structures of an institution and changes in
technology to engender alignment
Uncovers two processes of institutionally-triggered changes and one process of
Main Contribution technology triggered change related with the implementation of a computerized
physician order system
Methodological
criteria
The article elaborates on how the authors applied the principle of hermeneutic circle
and refers to Klein and Myers’ set of principles. The other principles were not
explicitly illustrated in the report, but we could find them in an implicit form.
Article 13 (Levina & Vaast 2008)
Research Method Case study
Research Site
Large multinational, financial services firm headquartered in United States and
Western Europe that outsourced projects to Russia and India
How differences in organizations and countries give rise to boundaries and status
Focus
differences in offshore ISD projects and what practices alleviate these differences to
establish multiparty collaboration
Theory-based practice framework that explains how boundaries and status
Main Contribution differences limit collaboration effectiveness in offshore ISD and the key role of onshore
middle managers in engaging others in shared practices
The article refers to Klein and Myers’ set of principles but does not explicitly
Methodological
criteria
particularize any principle. We could imply from the report that the principle of
interaction between the researchers and the subjects was applied to improve
inductive theory. Authors also refer that the principles of grounded theory informed
data collection and analysis.
Article 14 (Olssom et al. 2008)
Research Method Multiple case study
91

Ana Cardoso and Isabel Ramos
Research Site Irish sites of two large companies based in United States
Investigates and develops a theoretical model of the dual bridge role in a two-stage
Focus
offshoring relationship
Theoretical model that illustrates the off-shoring bridge model, where companies
Main Contribution differ in terms of their approaches to team integration, organizational level
implementation, and site hierarchy.
The principle of dialogical reasoning and the principle of multiple interpretations are
explicitly evoked. Moreover, the principle of abstraction and generalization is clearly
demonstrated in the description of the approach to data collection and analysis. The
Methodological principle of interaction between the researchers and the subjects can be implied from
criteria
the account of how the data was collected, and the principle of suspicion is
demonstrated through quotes. Other criteria, besides Klein and Myers’ set of
principles, are also explicitly justified and demonstrated namely reliability, construct
validity and external validity.
4. Discussion
In Table 2, we summarize our detailed assessment of quality and validity criteria in the articles. This
process had two phases: in the first phase, we examined carefully all articles and registered our
impressions about the methodological criteria reported by the authors for each article (see Table 1).
We also registered, (in Table 2), explicitly evoked principles (X) and any other criteria (yes (Y) or no
(N) when no other criteria was referred). In the second phase, we did a comparison among the
articles in terms of the type of implicit support we could find for each principle: either strong implicit
support (S), or weak implicit support (W) and registered the results of that comparison in Table 2.
Table 2: Summary of results
Principle
Hermeneutic
circle
Contextualization
Art.
1
Art.
2
Art.
3
Art.
4
Art.
5
Art.
6
Art.
7
W S S X W X X X X S S X S S
W S S X W S X X X S S S S W
Interaction W S S X W S X X X S S S S S
Abstraction &
generalization
Dialogical
reasoning
Multiple
interpretation
Suspicion
Other criteria
S S S X S S X X X S S S S S
S S S X W S X X X S S S S X
S S S X S S X X X S S S S X
W S S X W X X X X S S S S S
N N Y Y N N Y N N N N N Y Y
Legend: W – weak implicit support; S – strong implicit support; X – explicitly evoked; Y – yes; N – no.
Our results seem to show that Klein and Myers’ set of principles had a greater influence in the articles
published in MISQ (art. 7 to 14), specially in the period shortly after the publishing of K&M’ article,
where we can find several articles that explicitly state their conformation to that set of principles.
These articles usually provide a highly structured explanation in the form of a comprehensive table or
an annex where they justify their methodological considerations and provide examples for each
principle.
Another finding is that the principle of hermeneutic circle is, individually, the most cited and explicitly
developed by the authors, probably due to its fundamental nature and overarching importance for
interpretive research. On the other hand, the principle of dialogical reasoning, the principle of multiple
interpretations and the principle of suspicion are also individually cited and demonstrated with
examples in some articles, but we could not find any particular justification for this fact other than
random occurrences.
92
Art.
8
Art.
9
Art.
10
Art.
11
Art.
12
Art.
13
Art.
14

Ana Cardoso and Isabel Ramos
In terms of the implicit form of the principles, we found that the principles of abstraction and
generalization and the principle of multiple interpretations can be implied in all articles. Indeed, the
articles examined strive to generate a theoretical contribution that builds on the extant literature and to
convey multiple accounts of the events under study. Conversely, the principle of contextualization is
sometimes disregarded and minimized in the reports as some authors describe the case sites very
broadly and do not provide a critical account of the historical context.
Finally, a curious aspect is that three researchers authored more than one study in this sample of
articles. In this respect, we could find some regularities in their reports. For example, articles number
3 and number 13 do not explicitly refer any K&M’ principle but acknowledge the adherence to
grounded theory principles. Articles 7 and 12 are a different example. In this case, the author explicitly
refers to K&M’ set of principles, but the most recently published article only explains how the
hermeneutic circle was applied and does not delve into the other principles. Finally, articles 6 and 13
also share one author but in this case we could not find a salient pattern in the report style.
Acknowledgements
We acknowledge support of Fundação para a Ciência e Tecnologia via Bolsa de Doutoramento
SFRH/BD/60838/2009. We also acknowledge support of Fundos Feder – COMPETE and Fundação
para a Ciência e Tecnologia via FCOMP-01-0124-FEDER022674.
References
Backhouse, Hsu, C.W. & Silva, L. (2006). “Circuits of power in creating de jure standards: shaping an
international information systems security standard”. MIS Quarterly, 30(Special Issue), pp.413-438.
Bryant, A. & Charmaz, K. (2007). The Sage Handbook of Grounded Theory, London: Sage Publications.
Charmaz, K. (2000). Grounded theory: objectivist and constructivist methods. N.K. Denzin, Y.S. Lincoln, eds. In
Handbook of qualitative research. Thousand Oaks, CA, USA: Sage Publications.
Choudhury, V. & Sabherwal, R. (2003). "Portfolios of Control in Outsourced Software Development Projects".
Information Systems Research, Vol. 14, No. 3, pp.291-314.
Davidson, E.J. (2002). "Technology Frames and Framing: A Socio-Cognitive Investigation of Requirements
Determination". MIS Quarterly, Vol. 26, No. 4, pp.329-358.
Davidson, E.J. & Chismar, W.G. (2007). "The interation of institutionally triggered and technology-triggered social
structure change: an investigation of computerized physician order entry". MIS Quarterly, Vol. 31, No. 4,
pp.739-758.
Flick, U. (1998). An introduction to qualitative research, London: Sage Publications.
Golden-Biddle, K. & Locke, K. (1993). "Appealing work: an investigation of how ethnographic texts convince".
Organization Science, Vol. 4, No. 4, pp.595-616.
Hanseth, O. et al. (2006). "Reflexive standardization: side effects and complexity in standard making". MIS
Quarterly, 30(Special Issue), pp.563-581.
Klein, H.K. & Myers, M.D. (1999). "A Set of Principles for Conducting and Evaluating Interpretive Field Studies in
Information Systems". MIS Quarterly, Vol. 23, No. 1, pp.67–93.
Levina, N. (2005). "Collaborating on Multiparty Information Systems Development Projects: A Collective
Reflection-in-Action View". Information Systems Research, Vol. 16, No. 2, pp.109-130.
Levina, N. & Vaast, E. (2008). "Innovating or doing as told? status differences and overlapping boundaries in
offshore collaboration". MIS Quarterly, Vol. 32, No. 2, pp.307-332.
Myers, M.D. (1997). "Qualitative Research in Information Systems". Management Information Systems Quartely,
Vol. 21, No. 2, pp.241–242.
Myers, M.D. & Newman, M. (2007). "The qualitative interview in OS research: examining the craft". Information
and Organization, Vol. 17, No. 1, pp.2-26.
Olssom, H.H. et al. (2008). "Two-stage offshoring: as investigation of the Irish bridge". MIS Quarterly, Vol. 32,
No. 2, pp.257-279.
O’hEocha, C., Wang, X. & Conboy, K. (2012). "The use of focus groups in complex and pressurized IS studies
and evaluation using Klein & Myers principles for interpretive research". Information Systems Journal,
Forthcoming.
Patton, M.Q. (1990). Qualitative evaluation and research methods, Newbury Park: Sage Publications.
Pawlowski, S.D. & Robey, D. (2004). Bridging user organizations: knowledge brokering and the work of
information technology professionals". MIS Quarterly, Vol. 28, No. 4, pp.645-672.
Puri, S.K. (2007). "Integrating scientific with indigenous knowledge: constructing knowledge alliances for land
management in India". MIS Quarterly, Vol. 31, No. 2, pp.355-379.
Sarker, Suprateek. (2007). Qualitative research genres in the IS literature: emerging issues and potential
limitations". In Proceedings of the 40th Hawaii International Conference on System Sciences. HICSS’07.
IEEE Computer Society.
Sarker, Saonee & Sarker, Suprateek. (2009). "Exploring Agility in Distributed Information Systems Development
Teams: An Interpretive Study in an Offshoring Context". Information Systems Research, Vol. 20, No. 3,
pp.440-461.
93

Ana Cardoso and Isabel Ramos
Schultze, U. & Orlikowski, W.J. (2004). "A Practice Perspective on Technology-Mediated Network Relations:The
Use of Internet-Based Self-Serve Technologies". Information Systems Research, Vol. 15, No. 1, pp.87-106.
Vaast, E. & Walsham, G. (2009). "Trans-situated learning: supporting a network of practice with an information
infrastructure". Information Systems Research, Vol. 20, No. 4, pp.547-564.
Vidgen, R. & Wang, X. (2009). "Coevolving systems and the organization of agile software development".
Information Systems Research, Vol. 20, No. 3, pp.355-376.
Walsham, G. (2006). "Doing interpretive research". European Journal of Information Systems, Vol. 15, No. 3,
pp.320-330.
Walsham, G. (1995b). "Interpretive Case Studies in IS Research: Nature and Method". European Journal of
Information Systems, Vol. 4, No. 2, pp.74-81.
Walsham, G. (1995a). "The Emergence of Interpretivism in IS Research". Information Systems Research, Vol. 6,
No. 4, pp.376-394.
94

Substantiate the Reflexivity: The Insider-Outsider Role of
an Ethnographic Researcher
Ke Cui
Durham University, Durham, UK
ke.cui@durham.ac.uk
Abstract: In order to explore the notion and operational application of reflexivity in qualitative research, this
article examines the insider-outsider identity of ethnographic researcher as experienced in a Chinese study.
Drawing on my doctoral research which focuses on the impact of parental expectations upon children’s social
mobility, this paper compares how I have positioned myself as an insider-outsider before and during the
fieldwork. According to the differences between what I expected myself to be and what actually happened in the
fieldwork setting, this paper argues that positioning an ethnographic researcher within the research process is
problematic. This article explores my empirical experiences from the fieldwork I undertook to examine this issue,
the result of which reveals that unexpected social boundaries can be raised by the research participants. Thus,
we cannot make priori assertions about our position as a researcher before entering in the fieldwork. Moreover,
based on the notion of the insider-outsider continuum, my experience reveals that as the research proceeds, a
researcher moves constantly within this continuum; therefore, it is possible for an ethnographic researcher being
closer to one pole at one moment while being closer to the other pole at other moments. In addition, my
experience affirms the necessity to reflect on pre-existing relationships between the researcher and the research
participants while creating and re-creating research identities during the course of the fieldwork. Pre-existing
relationships inform the various perceptions that informants have of the researcher and this in turn determines
the depth of information they want to reveal. Looking at the complexities that arise in negotiation about the ways
in which insider-outsider role of the researcher is played out in the research process is a key concern of this
article. Additionally, it considers the importance of reflexivity in examining this role within qualitative research. In
doing so, I will contribute to the growing body of scholarship that examines the practical realities and ethical
dilemmas that arise when one is configured as an insider-outsider in ethnographic research.
Keywords: reflexivity, ethnographic researcher, fieldwork role, insider-outsider role, empirical data, research
experience
1. Introduction
Concepts and interpretations facilitating the understanding of reflexivity have been addressed in this
paper firstly. As reflexivity requires the researcher to be self-aware and examine his/her research role
critically, the argument then shifts to explore the insider-outsider role of an ethnographic researcher.
Insider-outsider relationship has been investigated across different cultures (Merriam et al. 2001) and
different disciplines. But it is still underdeveloped in the Chinese literature. Therefore, by involving in
several aspects of the Chinese culture, this article attempts to explore the insider-outsider relationship
through a comparison between what I expected my research status to be before the fieldwork and
what actually happened during the fieldwork.
2. Reflexivity
Reflexivity is one of the most basic concepts and practices differentiate qualitative from quantitative
research (Hsiung 2008). As a key concept in the world of qualitative social research, reflexivity has
been defined in different ways through various perspectives and is taken as a way to guarantee the
validity of qualitative research (Guillemin and Gillam 2004).
From the phenomenological perspective, a researcher cannot avoid bringing his or her individual
experience, personal background and pre-existing understanding of a phenomenon into the research
process (Finlay 2002). Specifically, Reinharz (1997) indicates that an ethnographic researcher brings
the self to the field and those “brought selves” could be a mother, a woman or an academician. Given
this, Finlay (2002) has suggested “Reflexivity” as a tool for evaluating how such inter-subjective
elements bringing in by a researcher impact on the research process, thereby increasing the integrity
and trustworthiness of qualitative research. From this perspective, reflexivity is defined as thoughtful,
conscious self-awareness of the roles that qualitative researchers play (Finlay 2002).
Additionally, Guillemin and Gillam (2004) claim that reflexivity is not a discrete entity; rather, it is an
on-going and active process of critical reflection both on what has been produced (“what do I know”)
and on how this has been generated (“how do I know what I know”). Furthermore, they explained that:
95

Ke Cui
“Reflexivity involves critical reflection of how the researcher constructs knowledge from
the research process—what sorts of factors influence the researcher’s construction of
knowledge and how these influences are revealed in the planning, conduct, and writing
up of the research (Guillemin and Gillam 2004: 275).”
The role that a researcher plays during the course of ethnographic research can impact upon the
construction of knowledge. Our representation of others’ accounts and experiences is related to who
we are (Bolak 1997). That is to say, researchers might interpret the same phenomenon in different
ways due to their personal backgrounds or pre-existing understandings of the phenomenon. This
requires a researcher to be able to reflexively examine his or her role throughout the research
process and be aware of the potential influences resulting from this role on the research results.
Although a qualitative researcher is in the central position for collecting, selecting and interpreting
data, the research is a product of the co-operation between participants, researchers and their
relationships (Finlay 2003). Given this, reflexivity is considered a good way to construct a relationship
between the researcher and the research participants (McKay et al. 2003) and it also provides a
greater insight into this relationship (Burns et al. 2012; Krieger 1985).
3. Distinguishing the research role
The application of reflexivity by an ethnographic researcher, who is considered as the major research
“tool” of ethnographic study by Allen (2004), becomes so important as reflexivity has been suggested
by Richardson (2000) as one of the five criteria (Substantive Contribution; Aesthetic Merit; Reflexivity;
Impact; Express a reality) to evaluate ethnographic studies. As a research student, it is the contention
from Hellawell (2006) that leads me to integrate “being a reflexive researcher” with “examining the
researcher’s insider-outsider role”: “I am arguing that a consideration by students of where their
doctoral research falls on a series of insider-outsider continua is one way of guiding students into
what is often the secret garden of reflexivity (p: 492).”
3.1 The single identity: Insider or outsider
Insider and outsider have been distinguished by Merton (1972) from the structural perspective:
“Insiders are members of specified groups of collectivities or occupants of specified social statuses;
Outsiders are the non-members (p: 21).” More specifically, people are classified as insiders if they
belong to a certain social group such as “race”, gender, or class status while outsiders are those who
belong to the different groups. For example, women are considered as outsiders by men and middle
class workers treat working class people as outsiders. After Merton’s positivistic conceptualization for
insider/outsider dynamic which emphasized social boundaries in defining a researcher’s role, recent
researchers such as Hill-Collins (1990) indicates that the term “Insider” representing researchers who
study a group to whom they belong and then have an advantage in gaining more intimate insights
because they are able to use their existing knowledge of the group. Similarly, researchers who have
prior or existing membership of the group being studied are recognized as insiders (Burns et al.
2012). Dwyer and Buckle (2009) state that if a researcher shares the characteristic, role or
experiences under study with the participants, then he or she is a member of the studied group and
thus is an insider. For example, in this collaborating paper, Dwyer positioned herself as an insider
researcher initially when she decided to investigate White parents of children adopted from Asia
because she herself was a White parent who had adopted an Asian child.
Instead of emphasizing the role of “membership” in defining a researcher’s position, distinguishing a
researcher’s insider/outsider status has shifted to concern his or her familiarity with or knowledge of
the group being studied. Griffith (1998) argues that familiarity with the group being studied positions
the researcher as an insider while a researcher who has no pre-existing knowledge of the group being
researched is an outsider. Additionally, in the paper written by Bonner and Tolhurst (2002), as a
nephrology nurse who participated in the group of other nephrology nurses, Bonner analysed her
research role as an “insider” regarding to her familiarity with the professional knowledge around
nephrology nurse; conversely, Tolhurst considered herself as an “outsider” in the research setting due
to her blank knowledge about the clinical teachers in major teaching hospitals.
3.2 The dual identity: Insider-outsider
With the development of ethnographic studies, the dichotomy between “insider” and “outsider” has
been challenged by the dual “insider-outsider” identity (Dwyer and Buckle 2009), which represents
96

Ke Cui
that instead of being completely included or excluded by informants, the researcher is often
suspended in a “betwixt-and-between” position (Ergun and Erdemir 2010).
Bolak (1997) argues that the “insider” and “outsider” are relative and exist on a continuum. Dwyer and
Buckle (2009) acknowledged the complexities inherent in occupying the role of either insider or
outsider and suggested the possibilities that researchers can only occupy “the space between” insider
and outsider instead of fully occupy one or the other of the two positions. For example, Dwyer initially
identified herself as an insider researcher when she decided to investigate other White parents of
children adopted from Asia. However, as the research went on, she finally realized that: “I sometimes
shared experiences, opinions, and perspectives with my participants, and other times I did not. It is
not that I sometimes saw myself as an outsider instead of an insider. Rather, not all populations are
homogeneous, so differences are to be expected (p: 56).” After this dialectical reflection, she
confirmed her role as an “insider-outsider” that she was both an insider and an outsider at the same
time.
A researcher can be an insider on some dimensions of the research and simultaneously be an
outsider on other dimensions (Hellawell 2006). Furthermore, the author explains this point of view by
two notions called: “elements of insiderness” and “elements of outsiderness”. For example, if you are
a female interviewer who is interviewing other females, there is an element of insiderness on a gender
dimension; on the other hand, if you are a young woman who is interviewing the relatively older
females, there is an element of outsiderness on the age dimension in your research role. Thus, a
young female researcher is an insider-outsider to a group of old female subjects.
4. Practical application
The two words insider and outsider and the relationship between them are still underdeveloped in the
Chinese literature. In consideration of this fact, I am going to draw on my research as having been
conducted in a Chinese context to discuss how I positioned myself as an insider and an outsider
separately, how I shifted to consider myself as playing a dual role of insider-outsider before entering
in the fieldwork setting. I will give a brief description of the research question and the methodologies I
have utilised before embarking on a consideration of my role.
4.1 Research background
My research focuses on the impact of parental educational expectations on the status achievements
of children. Life-history interviews were conducted to collect personal stories concerning the role of
aspirations or expectations in pursuing social mobility. Based on the insights of grounded theory and
narrative analysis, microanalysis is involved in the stage of analysing data. The analytic unit was a
statement of the informant presenting a story, an event, a choice, a saying or any other topics. I
started out analysing the data by coding, whereby I identified several themes. Then I classified these
themes into different categories.
The participants were recruited by means of a “snowball” sampling strategy in China, Henan province.
Sampling criteria were set according to the categorization of generations after 1949. Three cohorts
were included pertaining to the year in which people had their first child, both parents and children
were asked to participate in the research. Thus the oldest interviewee was over 70 years old while the
youngest one was aged 15. I went through a variety of interview experience benefited from the
diversity of the research subjects. Therefore, I will discuss under what conditions I defined my
outsider-insider status by referring to this research project.
4.2 4.2 Where I was: before the fieldwork
4.2.1 I was an insider?
Before going to do the fieldwork, on the one hand, given the assumption that researchers from other
countries may lack information about the studied country and may lose access to communities
traditionally closed to foreigners (Yakushko et al. 2011), I positioned myself as an insider in
consideration of the greater ease of access to China for me and my affinities with the culture in the
Chinese society (e.g., the Confusion philosophy has caused many Chinese parents to take their
children’s education as the only road to get a better social status). On the other hand, as an
Indigenous researcher (Sherif 2001), I felt much closer to the Indigenous participants who were from
the same place as me, Henan province of China (my home province). We spoke the same language
97

Ke Cui
(Henan dialect) and hold the same ethnicity (Han). I thought that I could communicate with the
participants much more easily than a researcher who had a different background. In consideration of
these similarities and familiarities with the individuals being studied, I positioned myself as an insider.
4.2.2 I was an outsider?
I also reflected on my social positions when I was examining my research role before meeting the
interviewees. In terms of my gender position, a female, I was an insider to the women interviewees. In
contrast, I was an outsider in front of the men participants. Willot’s (1998) research explored the
construction of masculine identities of men for which she had to conduct the interview by herself. She
indicated that she was considered or located as an outsider within the male-dominated arenas where
she carried out her fieldwork. In terms of the social status, my family origin assigned me to the middle
class group. Thus, from Merton’s (1972) perspective, I was an insider to the middle class interviewees
but I was an outsider to interviewees from other social classes. Since my participants were recruited
from different age groups, I was an outsider to those who were belonging to different generations from
mine because I was not able, for example, to experience certain historical events. But I was an insider
to those who were born after 1980s since we both had undergone the “college expansion” which gave
us more opportunities to receive higher education and impacted our opinion towards academic
degrees; thus I got more empathy with this group of people
4.2.3 I was an insider-outsider
After moving from the insider pole to the outsider pole instantly, I started to realize that I could not
locate myself in either one of them; rather, I could occupy both of these two positions simultaneously.
This mirrors Reinharz’s (1997) experience. She conducted a participant observation in a kibbutz (A
kibbutz is a collective community in Israel that was traditional based on agriculture). Her brought
selves being an American and a Jew made her an outsider to these kibbutz residents but being a
mother and a wife benefited her a lot for constructing an intimate relationship with some of the
residents; thus her role could be considered as an insider-outsider one. I was involved in a similar
situation as Reinharz’s. For example, I was Chinese so I was an insider but I could not be a true
insider because I was still a woman, a young adult with no motherhood experience and a research
student. These various social positions varied my role in the course of the research. Thus, I could not
just consider one specific aspect of myself at a moment without considering the others at the same
time because they exist simultaneously and have combined effects in defining my research role.
5. Examining my research role during the fieldwork
The empirical data I have collected and the practical experiences I have gained reveal that we cannot
make priori assertions about our position as a researcher before entering in the fieldwork setting.
Rather, it needs to be discovered carefully and negotiated constantly with the research participants as
the research proceeds (Allen 2004; Ergun and Erdemir 2010). Those empirical data have reminded
me that in the initial stage of this research I was overly guided by my own viewpoints upon the
participants without considering how they would judge my status as either “insider” or “outsider”.
Based on the diversified experiences I will turn to discuss how my research identity was created and
re-created during the course of the fieldwork.
5.1 Emerging boundary
Before doing the fieldwork, I examined several factors informing the interaction between me and my
informants, such as gender, age, social class and nationality, marital status. Although the interplay
among these different factors did play a significant role in shaping the insider-outsider relationship,
some other unexpected factors and boundaries also emerged in the field.
5.1.1 Political identity
The same with the situation of Brayboy and Deyhle (2000), my political motive was questioned by one
interviewee. Different from Sherif’s (2001) opinion that an Indigenous researcher is considered as a
member of the studied group, I was ascribed as a “spy” by this interviewee who said that: “I know you
are a research student in the UK and all the information you get from China will be shown to the
government of the UK. You are just a spy authorized by the UK government.”
I was being absolutely isolated by him, who was a man in his middle age, working as a higher official,
living in a community of five minutes’ walk from my home, etc. From his viewpoint, I was singled out
98

Ke Cui
because my identity as a Western researcher who was attempting to “utilize his knowledge for the
sake of other countries.” I came to acknowledge this political boundary only after being in the field and
felt the extent to which I was considered an outsider.
5.1.2 Level of education
As a PhD student in the UK, I was highly respected by some of the interviewees who were studying in
a college of China, where education is valued greatly (Li 2001). I interviewed a young man, who was
a third year undergraduate student. I was taken as his “idol” because of my education level: “Oh, I
really hope I could be like you, studying in one of the world’s best universities and getting a doctoral
degree. You are my idol from now on.” I was reflecting on my role from the perspective of “being an
idol”, the one I could never had anticipated before meeting this interviewee. He expressed his concern
about my outsider status and questioned my capacity to appreciate his experience: “You are living a
life as good as in the paradise and you can never understand how terrible my life is.”
5.1.3 Moving in the insider-outsider continuum
Later in the same interview, I replied to some of his questions, such as how the life was in abroad,
how I got the good academic performance. I attempted to answer his questions as best as I could
because this was necessary to assist the storyteller in making sense of his own story and in finding
deeper meaning in the narrative (Atkinson 1998). Moreover, this was approved to be a good way to
minimize any status differences that exist between the researcher and the informants in an interview
situation (Gough 2003). I did this also in order to establish a proper rapport for gathering information.
In the middle of our conversation, he involved in a short deep thinking before giving the follow
statement: “Your life is actually not as wonderful as I thought. I think you are just one of us....” I reexamined
my role at this point. I thought I was getting more accepted by him. What I could tell from
this interview was my research role changed as the conversation proceeded.
This story persuaded me that I was moving in the insider-outsider continuum that I could be closer to
the insider pole at some moments while being closer to the outsider pole at other moments. The
inclination of my research position on the insider-outsider continuum was not a fixed one; rather it
changed constantly.
5.1.4 The role of pre-existing relationships
Due to the characters of snowball sampling strategy, some of the recruited participants had intimate
relationships with me. For example, some of them were friends of mom. Those pre-existing
relationships impacted upon the researcher –research participants relationships that would be built in
the field, thereby generating certain outcomes for the research. The first story I am going to tell under
this case is about a woman interviewee, who has been my mother’s best friend and I always consider
her as my aunty.
In consideration of the Chinese culture which requests younger people always being polite to their
elders, I did not dare to ask directly about her family income because this was too sensitive a question
that few people would love to answer. So I initially referred to her social status rather than asking her
family income: “Which social class are you in, from your point of view?” She said that: “Well, you
know, family income is the most important indicator of social class in my opinion. I am in the lowermiddle
class pertaining to my salary. But I am not the bread winner in my family because my husband
earned much more than me.” I probed: “Well, so what do you think of the social status of your
husband or your family?” She replied by mentioning her husband’s income directly: “Regard to the
monthly income of my husband, I have not even told this to my other family members… I take you as
my own child since you are my friend’s daughter. I feel responsible to help you with your study, so I
will let you know the amount [laughs]….”
I was involved in a reflection about my research role after this. The interviewee mentioned that I was
given this specific information because she treated me as her daughter; thus she thought she had the
responsibility to support my study. This attitude has been defined as “paternalistic” (Easterday et al.
1977). Atkinson (1998) says that when conducting an interview, the existing relationship between
interviewer and interviewee move to a different phase and this exchange changes everything between
you and the story teller. When came to the research setting, the intimate relationship between us still
existed and it was reflected by her feeling of responsibility as an eldership to help my study. However,
99

Ke Cui
this relationship shifted to a different phase that she was a “responsive informant” and I was an “eager
information collector”. She decided to tell me her husband’s salary both because of her underlying
responsibility as an intimate eldership but also because I was a researcher who had subjected to
ethical review and committed to confidentiality and anonymity as what was informed in the consent
form.
In contrast to the first story, the experience I have had with my friend indicates a different role of the
pre-existing researcher-research participant relationship in positioning a researcher in the course of
fieldwork. We were so relaxed that the interview with her was just like a daily talking until I posited the
question that: “I know you had been doing very well in the secondary school and, as you mentioned to
me before, you mother was very strict with your study; but do you remember what was your mother’s
attitude when you got bad exam record at that time?’” She came into a silence which was out of my
expectation. After a while, she continued to say that: “Everything has passed and I do not want to
recall it, so let us move on to the next question, OK?”
In the first story, my “aunty” revealed the sensitive information to me because of the intimate
relationship between us. But in the second story, my “friend” refused to answer the question which
seemed sensitive to her. Thus, it indicates that my pre-existing relationships with the informants
determined my research position and in turn controlled the depth of information I could get. However,
before going to the field I could hardly assume what kind of new relationship would be built based on
the existing relationship or which phase the existing relationship would move in. This further affirms
that the identity of an ethnographic researcher needs to be negotiated constantly and it is also
necessary to contextualize the pre-existing relationships in re-configuring these identities.
6. Conclusions
In this paper, I attempt to make a contribution to the understanding of reflexivity through examining
the insider-outsider role of a qualitative researcher. I have also made visible the practices through
which the fieldwork role is defined and re-defined based on my research experience. It proves that we
are not free to make our identities before going into the field due to the various unpredicted field
contexts. And we cannot control our identities in the field because different perceptions toward our
identities come out from informants’ accounts and their prior knowledge of us based on pre-existing
relations. Identifying a researcher’s insider-outsider role is a constant task that it needs to be
discovered carefully and negotiated continuously throughout the whole course of a research. My
practical experiences reveal that it is possible for an ethnographic researcher sometimes to be closer
to the insider pole while at other times being closer to the outsider pole on the insider-outsider
continuum. It is necessary for a reflexive researcher to be aware of this situation and examine his/her
role constantly because its shifting inclination on the insider-outsider continuum impacts on the depth
of insights can be gained from the research participants. In addition, we should reflect on our preexisting
relationships with the research participants. Whether we will be included or excluded by the
research participants is partly determined by the existing relationships between us.
References
Allen D. (2004) Ethnomethodological insights into insider–outsider relationships in nursing ethnographies of
healthcare settings. Nursing Inquiry 11: 14-24.
Atkinson R. (1998) The life story interview / R. Atkinson, London: SAGE.
Bolak, C. H. (1997) Studying One’s Own in the Middle East: Negotiating Gender and Self-Other Dynamics in the
Field. In: R. Hertz (eds) Reflexivity and voice. Thousand Oaks, CA: Sage, pp.95-118
Bonner, A and G. Tolhurst. (2002) Insider-outsider perspectives of participant observation. Nurse Researcher 9:
7-19.
Brayboy BM and Deyhle D. (2000) Insider-Outsider: Researchers in American Indian Communities. Theory Into
Practice 39: 163-169.
Burns E, Fenwick J, Schmied V, et al. (2012) Reflexivity in midwifery research: The insider/outsider debate.
Midwifery 28: 52-60.
Corbin Dwyer, S., & Buckle, J. ( 2009) The space between: On being an insider/outsider in qualitative research.
International Journal of Qualitative Methods 8: 54-63.
Easterday L, Papademas D, Schorr L, et al. (1977) The Making of a Female Researcher. Journal of
Contemporary Ethnography 6: 333-348.
Ergun A and Erdemir A. (2010) Negotiating Insider and Outsider Identities in the Field: “Insider” in a Foreign
Land; “Outsider” in One’s Own Land. Field Methods 22: 16-38.
Finlay L. (2002) “Outing” the Researcher: The Provenance, Process, and Practice of Reflexivity. Qualitative
Health Research 12: 531-545.
100

Ke Cui
Finlay, L. (2003) The reflexive journey: mapping multiple routes. In: L. Finlay and B. Gough (eds) Reflexivity: a
practical guide for researchers in health and social sciences. Oxford: Blackwell Science, pp.3-20.
Gough, B. (2003) Shifting researcher positions during a group interview study: a reflexive analysis and re-view.
In: L. Finlay and B. Gough (eds) Reflexivity: a practical guide for researchers in health and social sciences.
Oxford: Blackwell Science, pp.146-159.
Griffith AI. (1998) Insider / Outsider: Epistemological Privilege and Mothering Work. Human Studies 21: 361-376.
Guillemin M and Gillam L. (2004) Ethics, Reflexivity, and “Ethically Important Moments” in Research. Qualitative
Inquiry 10: 261-280.
Hellawell D. (2006) Inside–out: analysis of the insider–outsider concept as a heuristic device to develop reflexivity
in students doing qualitative research. Teaching in Higher Education 11: 483-494.
Hertz, R. (1997) Introduction: Reflexivity and voice. In: R. Hertz (eds) Reflexivity and voice. Thousand Oaks, CA:
Sage, pp.vii-xviii.
Hill-Collins, P. (1990) Learning from the outsider within the sociological significance of black feminist thought. In:
Fonow, M., Cook, J. (eds) Beyond Methodology Feminist Scholarship as Lived Research. Indiana University
Press, Bloomington, pp.35-59.
Hsiung P-C. (2008) Teaching Reflexivity in Qualitative Interviewing. Teaching Sociology 36: 211-226.
Krieger S. (1985) Beyond “subjectivity”: The use of the self in social science. Qualitative Sociology 8: 309-324.
Li J. (2001) Expectations of Chinese Immigrant Parents for Their Children's Education: The Interplay of Chinese
Tradition and the Canadian Context. Canadian Journal of Education / Revue canadienne de l'éducation 26:
477-494.
McKay, E., Ryan, S. and Sumsion, T. (2003) Three journeys toward reflexivity. In: L. Finlay and B. Gough (eds)
Reflexivity: a practical guide for researchers in health and social sciences. Oxford: Blackwell Science,
pp.53-65.
Merriam SB, Johnson-Bailey J, Lee M-Y, et al. (2001) Power and positionality: negotiating insider/outsider status
within and across cultures. International Journal of Lifelong Education 20: 405-416.
Merton RK. (1972) Insiders and Outsiders: A Chapter in the Sociology of Knowledge. American Journal of
Sociology 78: 9-47.
Reinharz, S. (1997) Who Am I? The Need for a Variety of Selves in the Field. In: R. Hertz (eds) Reflexivity and
voice. Thousand Oaks, CA: Sage, pp.3-20.
Sherif B. (2001) The Ambiguity of Boundaries in the Fieldwork Experience: Establishing Rapport and Negotiating
Insider/Outsider Status. Qualitative Inquiry 7: 436-447.
Steedman, R.H. (1991) On the Relations between Seeing, Interpreting and Knowing. In: F. Steier (eds) Research
and reflexivity. London: Sage, pp.53-62.
Strauss AL. (1990) Basics of qualitative research : grounded theory procedures and techniques / A. Strauss, J.
Corbin, Newbury Park : Sage.
Willot, S. (1998) An outsider within: A feminist doing research with men. In: K. Henwood, C. Griffin & A. Phoenix
(eds) Standpoints and differences. London: Sage, pp.174-190
Yakushko O, Badiee M, Mallory A, et al. (2011) Insider Outsider: Reflections on Working with One's Own
Communities. Women & Therapy 34: 279-292.
101

Students so Close, yet so far Away: A Case Study and Best
Practices for Teaching Research Methods Online
Leslie Dinauer
University of Maryland University College, Adelphi, USA
ldinauer@umuc.edu
Abstract: An increasing number of students are participating in online learning. When students must take a
research methods course online, they often have great difficulty and become anxious. Statistics anxiety appears
heightened in the online environment, fueled not only by discomfort with the statistical content of the course, but
also by the increased cognitive load of managing the learning software and the challenge of reading and
understanding statistics textbooks without the benefit of face-to-face feedback and instruction. Thus, the
outcomes for students taking research methods courses online can be much poorer than for students who take
similar classes face-to-face. The purpose of this exploratory study is to explore the online experience for students
in a research methods class, paying particular attention to the significant ways in which an online class differs
from face-to-face. This study first examines the concept of statistics anxiety, then moves to consider the online
experience for students in a research methods class, paying particular attention to the significant ways in which
an online class differs from face-to-face. Using a case study methodology and narrative analysis, the paper
presents the experiences of one class of management graduate students enrolled in a 13-week online research
methods class in a large, American university. The data richly demonstrate how the online classroom enhances
feelings of alienation with respect to students’ self-beliefs about their ability to participate in the class, and
facilitates avoidance behavior that adversely affects performance on homework. Over the course of the semester,
however, most students’ confidence and ability improve to acceptable levels.
Keywords: teaching research methods, online learning, statistics anxiety
1. Introduction
An increasing number of students are participating in online learning, either through a single course
(or a few courses) in an otherwise traditional face-to-face degree program or through the acquisition
of an entire degree online (Allen and Seaman 2007). When students who have a limited experience in
research and statistics must take a research methods course online, they often have great difficulty
(Hsu, Wang and Chiu 2009). Anxiety about statistics appears heightened in the online environment,
fueled not only by discomfort with the statistical content of the course, but also by the increased
cognitive load of managing the learning software – as well as, very often, statistical software (Hsu,
Wang and Chiu 2009) – and the challenge of reading and understanding statistics textbooks (Collins
and Onwuegbuzie 2007) without the benefit of face-to-face feedback and instruction. Thus, the
outcomes for students taking research methods courses online can be much poorer than for students
who take similar classes face-to-face.
The purpose of this exploratory study is to explore the online experience for students in a research
methods class, paying particular attention to the significant ways in which an online class differs from
face-to-face. Answering Mills and Raju’s (2011) call for additional research in this area, the study
hopes to begin to provide additional information about how the online environment uniquely affects
students of research methods and statistics, for the purpose of offering pedagogical advice to
instructors. Using a case study methodology and narrative analysis, the paper examines the
experiences of one class of management graduate students at a large, American university. This
conference paper first reviews literature in statistics anxiety in general, then looks briefly at what we
know about statistics anxiety in the context of teaching statistics online. Next, the methodology is
explained, to include a detailed description of online classroom that comprises the case. Finally,
emergent themes from the case analysis are described and discussed.
2. Literature review
It has been long known that many students of business are frustrated in their attempts to learn
research methods and statistics because of the degree of anxiety that they feel when encountering
the subject matter, a problem that other content areas (e.g., leadership, strategic planning, logistics,
etc.) do not equivalently face. Statistics anxiety has been defined as “feelings of anxiety encountered
when . . . doing statistical analyses; that is gathering, processing and interpreting data” (Cruise, Cash
and Bolton 1985: 92). It is routinely measured by the Statistics Anxiety Rating Scale (STARS; Cruise
and Wilkins 1980), which assesses personal attitudes about the worth of statistics, interpretation
anxiety, test anxiety, computation self-concept (feelings of computational inadequacy), fear of asking
102

Leslie Dinauer
for help, and fear of statistics teachers. Zeidner (1991) has added to this standard definition, stating
that statistics anxiety is accompanied by worry, tension, and physiological symptoms of stress.
Onwuegbuzie (2000) has suggested a relationship between an individual’s cognition and affect that
manifests in statistical anxiety.
A complex phenomenon, statistics anxiety is probably related to math anxiety, but it has been shown
also to include additional factors (see, e.g., a discussion of this in Cruise et al. 1985). In fact, Zerbolio
(1999) argues that statistics is more related to verbal reasoning than to mathematical reasoning, and
suggests that success in statistics is linked more strongly to logical reasoning skills than math skills.
So, there is something unique about the cognitive demands of a statistics course that causes a great
deal of stress in students.
Moreover, student outcomes are affected by student anxiety. Statistics anxiety has been shown to be
negatively related to statistics achievement. Baloglu and Zelhart (2003), for example, found a
negative relationship between statistics anxiety and student achievement, as did Lalonde and
Gardner (1993). Zanakis and Valenza (1997) have also demonstrated that the general underlying
math anxiety that is often co-morbid with statistics anxiety is a detriment to academic success.
As increasing numbers of students take classes and entire courses of study online, the number of
students enrolled in statistics courses online has also increased. Although some instructors have
sought simply to replicate their successful face-to-face statistics classes online component-bycomponent
(Boettcher and Conrad 1999), advancements in understanding the effects of the online
environment on student learning have shown that effective teaching in the online environment
requires new strategies and techniques (Christie and Jurado 2009).
Thus, DeVaney (2010: 12) has called for additional research in the realm of statistics anxiety
specifically in the online environment. Toward that end, he compared level of statistics anxiety and
attitude toward statistics for graduate students in face-to-face and online statistics courses. He
obtained significant results, finding that students taking statistics classes online had more anxiety and
less favorable attitudes than did students taking statistics classes face-to-face (DeVaney 2010: 9).
One potential cause of students’ increased anxiety in the online classroom could be their relative
isolation. Students in the online statistics classroom are on their own – they do not come to class
each week, making it more difficult to form connections with both the instructor and other students –
and they often fail to ask for help when they need it, for fear of ‘looking stupid.’ Often, students who
need help stop logging into the class and disappear for days or weeks at a time. Mann (2005: 43)
explains that students learning online may feel “held back, blocked, inhibited, estranged or isolated
from what it is they are learning, and the study practices and learning processes, both individual and
social, which are part of their particular learning context.”
Another possible cause of student anxiety in the online statistics classroom could be instructors’
online practices; there is little published research to know. It seems logical, however, that instructors
who are more skilled in maneuvering within the online classroom would possess more successful
teaching strategies than instructors who do not. For example, many online research methods faculty
do not have access to (or do not know how to use) technology that allows them to graph or derive
formulas dynamically, which in the face-to-face classroom allows students to gain a sense of process,
and understand a “reason why.” In the absence of such explanations, students are forced to follow
derivations on the printed page of their textbooks, with no opportunity to ask questions at any step
along the way. Many students do not possess the reading ability to adequately process those textbased
explanations (Collins and Onwuegbuzie 2007). The notion that professor-student personal
interactions are critical to online student success is supported by Sebastianelli and Tamimi (2011),
whose findings suggest that online classroom features involving professor-student interaction are the
most useful of the virtual classroom. They also argue that conference discussion forums (i.e., studentto-student
communication) are of limited value in learning highly quantitative content.
Mills and Raju’s (2011) review of the literature about teaching statistics online from 1999 – 2009
highlighted a number of challenges to teaching statistics that are brought about by the nature of the
online (and usually asynchronous) classroom environment. These include: Inability to motivate
students directly/lack of access to students beyond email or classroom posts; Difficulty identifying
students needing help immediately; Technology problems such as slow speeds, computers “hanging
103

Leslie Dinauer
up” and a lack of basic technical skills for some students; Miscommunication/misunderstanding
between instructor and students as a result of lack of nonverbal information (e.g., tone, humor); and
Difficulty providing effective feedback and/or assessment. Overall, the literature examining teaching
statistics online does not yield consistent or universally-accepted results about either student
experiences or instructor best practices. Thus, this study endeavored to learn more about how
students experience the online statistics classroom, and what kinds of tools (teaching, technological,
or otherwise) might improve that experience to produce successful student outcomes.
3. Method
An inductive, exploratory case study was conducted to explore the online experience of students in a
research methods class. Case study facilitates the discovery and understanding of real-life events,
and permits an in-depth exploration of a program, activity, process, or action of one or more
individuals, as was the desire in the current study where the researcher wished to gain deep insight
into students’ feelings, attitudes, and motivations during the course of learning quantitative research
methods (Creswell 2009; Yin 2009). Additionally, case study was the logical method of investigation
as it would allow a grounded theory approach, a rich, naturalistic inquiry in which the researcher was
also the instructor for the class to be studied (Glasser and Strauss 1967; Lincoln and Guba 1985).
The researcher’s engagement in the case was not without precedent, and adhered closely to the
tenets of action research (Greenwood and Levin 2007).
3.1 Description of the case classroom and participants
The case of interest was an introductory graduate-level research methods class at a large, American
university. The class was taught by the researcher. The class was 13 weeks in length and was taught
mostly online, with a two-day (8 hours each day) face-to-face mandatory residency component at the
end of week 6. Two teleconferences, of approximately one hour each, were also held.
The course covered basic research method topics, including:
Introduction to epistemology, and research methods worldviews
Research ethics
Research design
Sampling
Measurement
Reliability and validity
Data collection
Data editing, transformation, and index construction
Charts and graphs
Contingency tables
Descriptive statistics
Hypothesis testing
Statistical tests for examining group differences
Statistical tests for examining association
Linear regression
Choosing the correct statistic
The online classroom used a learning management system (LMS) that is proprietary to the
researcher’s institution, but is similar to a number of popular off-the-shelf systems, like Blackboard.
Students could access the all of course materials (e.g., syllabus, readings, lectures) in the online
classroom, and they were expected to participate in directed, asynchronous conference discussions
weekly.
For each week of the class, there were assigned textbook readings, a written “lecture” to elucidate the
readings and the weekly topic(s) in general, a ten minute audio “lecturette” from the instructor to
further explore the ideas of the weekly topic(s), and two required interactive conference discussions.
104

Leslie Dinauer
The conference discussions required significant effort on the part of the students to engage the
topic(s) at hand. In addition to a participation grade, students completed seven homework
assignments during the semester (due approximately every other week), and took a final exam at the
end of the course.
The semester began with eight students, six of whom finished the term (two withdrew during the
semester, one male and one female). Of the six who finished, four were female and two were male.
The average age of the students was 41.2 years. One of the six students had significant prior
experience working with an understanding statistics. All of the students were employed, mostly midlevel
managers, and were seeking a graduate degree in management for the primary purpose of
career advancement.
3.2 Data collection
The week prior to the commencement of class, students were contacted via email, informed of the
researcher’s general intent, and asked for consent to use their classroom contributions in the study.
Each student was informed that participation was voluntary, and that all study-related communication
would remain anonymous in presentation of the case study. Students were assured there was no
penalty for refusing to participate. Informed consent and permission to use classroom communication
for the study were obtained from all students.
The online classroom served as a data repository, capturing and saving all conference discussion
during the semester. In addition to retaining conference discussions, the classroom contained an
integrated messaging tool for private instructor-student interaction. Thus, with the exception of
conversations during the face-to-face residency and two teleconferences held during the semester, all
classroom communication for the semester was recorded and saved.
Additionally, during the face-to-face residency (end of week six) and the two teleconferences (middle
of weeks three and ten), extensive handwritten field notes of discussion and activities were kept. The
handwritten notes were transcribed into Microsoft Word and reviewed to ensure accuracy. The
process of transcribing provided the researcher with insight into possible themes that would emerge
from the data when it was combined with the text from the classroom.
3.3 Data analysis
The study used narrative analysis to gain an understanding of participants’ lived experiences and
perceptions in order to better comprehend the nature of their feelings, attitudes, and motivations in the
online research methods classroom. This inductive approach “honors people’s stories as data that
can stand on their own as pure description of experience, worthy as narrative documentary of
experience (the core of phenomenology) or analyzed for connections between the psychological,
sociological, cultural, political, and dramatic dimensions of human experience” (Patton 2002: 115-
116). The data were examined for prominent and emergent themes, particularly in light of both the
literature and the researcher’s record of significant conversations that occurred during the semester.
4. Results
Overall, three primary themes emerged from the data, and they were not necessarily surprising. As
the literature shows, statistical anxiety across all media of content delivery (e.g., face-to-face, online,
etc.) is quite common. What bears consideration, however, are the findings that relate to the role that
(1) feelings of alienation, and (2) technical difficulties play in exacerbating that anxiety. Thirdly,
students’ dependence on, and literal interpretations of, written asynchronous communication (e.g.,
textbooks and written lectures) in order to learn statistical concepts often makes even “easy” concepts
difficult to understand. These three emergent themes are discussed in greater detail below.
4.1 Theme 1: Students feel isolated, and this fuels anxiety
Consistent with the literature, at one time or another – and predominantly in the first half of the
semester, prior to residency – every student in the classroom signified his or her feelings of aloneness
during conference discussion, usually in the form of a reaching out to others and an asking for
connection:
105

Leslie Dinauer
I had trouble with this last week on my own and now I am swallowing my pride and
asking for help. Is anybody else stuck?
Am I the only one who still gets sweaty palms when looking at statistics?
My statistical knowledge is pretty much null and void. This class is going to be very
difficult for me. Does anyone want to form a study group?
I'm having a tough time. . . Any suggestions or help you can provide would be great. I’m
out on a limb here.
In a face-to-face classroom, students have the regular opportunity to meet together to support each
other, commiserate, and socialize. Students in this classroom seemed to miss that experience
acutely, and appeared to long for more connection. They seemed to want to band together in their
suffering in order to better conquer the material. When they met during the residency at the end of
week six (about the half-way point of the course), they worked hard to establish relationships with
each other. They spent their breaks, lunch hours, and dinner the first evening sharing their
experiences to day in the class, and offering support to each other. It was only after the face-to-face
residency that the idea of a study group came successfully to fruition.
4.2 Theme 2: Difficulty communicating and understanding statistical processes via
text
There is, of course, no such thing as a dumb question. Many of the questions that were posted in the
classroom, however, were trivial. They seemed created more by syntactic confusion than by
conceptual confusion – a syntactic confusion that might not have occurred if students had the ability
to stop me in the middle of a lecture to ask for clarification. The lack of immediate feedback in the
online classroom was certainly a disadvantage; on a number of occasions small misunderstandings
quickly grew into sources of great stress:
I’m so confused. I thought we wanted alpha to be .75 or higher, not .05 or less.
If I know the average, why do I need to know the mean?
I would like to understand what is the difference between Interquartile range and
Interquartile range of data?
Question - is the researcher creating the dummy variable or do they show up in the data
some how? I am not sure I understand this concept at all.
Why is the regression linear?
Many questions like this arose due simply to either a poor or highly literal reading of the text or lecture
material. In a face-to-face classroom, students are able to ask clarifying questions in real time as the
material is presented, and receive immediate answers (as opposed to waiting an entire day for the
instructor to post a response). The effect of waiting in uncertainty seems to increase students’ anxiety
levels and create a cognitive load that makes the statistics feel that much more difficult. Moreover,
having to deliberately ask questions like these by typing them out appeared to reinforce students’ low
opinions of their own ability to master the content.
4.3 Theme 3: Technological barriers
Students encountered technological difficulties along two fronts: navigating the Learning Management
System (LMS), and using SPSS 18.0 to run their statistical analyses.
4.3.1 Navigating the LMS environment
When students enter an online classroom or an online degree program, they must familiarize
themselves with the LMS platform. Where can they find the syllabus? Where can they find the
homeworks? Where do they post the homework when it’s completed? Although these tasks are
usually fairly easy to do, together they increase students’ cognitive load when interacting with course
content and can increase frustration as well as interfere with learning.
I had trouble with my password, so that is why I am just now getting started with the
course.
I have been looking around in the classroom and did not see any information on the
homework assignments. Can you please tell me where I can find them?
106

Leslie Dinauer
I see that Homework 1 is due this weekend, and I am still unable to find it. Has anyone
found it? Where is it located?
In a face-to-face classroom, there is no corresponding challenge to learning the LMS. Students come
into the classroom and sit down. They are generally well-normed to the expectations and rules of the
classroom.
4.3.2 Using SPSS 18.0
Many students in research methods, in both face-to-face and online courses, are required to use
statistical analysis software. In some situations, students begin the class with no knowledge of the
software and must learn to use it concurrently with learning substantive statistical analysis content.
Similar to how they allow their frustrations to grow in navigating the LMS, students in the online
classroom seemed to interpret the normal learning curve for statistical software as personal failure,
and viewed that learning curve as evidence of their own inability to master the content.
I'm unable to open the third document that starts with "public perceptions". I anyone else
having difficulty? If you've already downloaded it to your computer, can you please send
it to my personal e-mail. I hate this.
Hello XXX, Did you install SPSS in your computer? You should be able to open the file if
you had SPSS.
I tried just to delete that value, but I ended up getting rid of the whole column and I can’t
get it back so I’ll never make it.
With me, it's usually "user error" so I will try again.
Thank you Apple, Java and IBM for another job well done.... NOT!!! I can’t make this
work!!! I will never catch up!!!
In a face-to-face classroom, there can be similar challenges with learning a statistical software
package. Very often, however, students have the opportunity to visit a computer lab with their
instructor and work through at least basic commands together. Or, instructors can use projectors in
the classroom to share screen shots, and show students what each point-and-click looks like. In many
online classrooms, there is not the infrastructure to support a similar learning experience.
4.4 Student outcomes
Despite the struggles that students reported facing and feeling, at the end of the course most students
reflected back on the experience as positive.
Thanks for taking us through an interesting learning period, where at times self- doubt
had been overpowering.
At the beginning, Leslie pointed out that this would be one of the most intensive classes
of the program and I can definitely say that was almost an understatement. I feel that I
have grown tremendously in my knowledge of statistics, but also in how I think about
research. I have had the opportunity to re-review some of the articles that I used in [an
earlier course] and feel that I now look at them in a new light.
While the course was very challenging, I really enjoyed the opportunity to learn together
with all of my cohorts in the class. In looking at the prior posts, there is definitely a
tremendous amount of learning as the weeks progressed and it's quite noticeable how
we became more comfortable in the different topics and worked to question, challenge
and support each other throughout the term.
This is consistent with DeVaney’s (2010: 12) finding that “anxiety and attitudinal levels of online
students at the end of the course were similar to those in traditional, on-campus settings”.
The semester began with eight students, six of whom finished the term (two withdrew during the
semester, one male and one female). Data analysis did not include an attempt to correlate individual
final grades with individual student narrative themes because of the very small sample. Of the six
students who completed the class, two earned As, three earned Bs, and one earned a C (which is a
failing grade in the graduate school and resulted in the student having to repeat the course).
107

5. Discussion
Leslie Dinauer
Overall, the results suggest that some characteristics unique to the online classroom experience
serve to exacerbate the anxiety that students taking courses in research methods and statistics may
feel. Consequently, those characteristics having been identified, the range of strategies available to
instructors to mitigate those characteristics can be explored.
The first theme to emerge from the narratives of the case study classroom was that students feel
isolated, and this fuels anxiety. With respect to those feelings, the researcher observed that they were
greatly dissipated by the face-to-face residency; students were noticeably energized by the residency,
even though they spent 16 hours over two days tackling the most difficult content of the course.
Despite all of the recent technological advances, students still want to talk to their instructors and
classmates. Such a statement may not surprise opponents of online learning, but the reality in higher
education is that online course offerings will continue to increase in the foreseeable future. So,
instructors who wish to create the most effective, successful classrooms can take this lesson to heart;
attempts by instructors to connect with students in ways other than textually – for example by posting
audio or video recordings in the classroom regularly, and by hosting synchronous teleconferences –
will help to “humanize” the classroom. Use of audio messaging discussion boards (e.g. via Horizon’s
Wimba conferencing system), where available, can also meet students’ need to make interpersonal
connections verbally. These kinds of practices are are important because where students are able to
connect, they are able to coach each other through difficult material in cooperative learning.
Additionally, instructors who are aware of this challenge can better empathize with students, opening
discussion to allow students to share their feelings and then working to alleviate those feelings.
The second theme that emerged concerned difficulty communicating and understanding statistical
processes via text. These observed difficulties are consistent with recent research that has suggested
that poor reading ability can seriously hinder the ability of students to learn statistics (Collins and
Onwuegbuzie 2007). This finding suggests that best practices in the statistics classroom should
include a variety of content delivery media, similar to the recommendations made above. Indeed, a
pattern begins to form here about the importance of finding alternatives to text-based communication
in the classroom. Challenges arise, however, when institutions do not provide adequate resources for
such alternatives. Not all universities, or all departments, offer online instructors robust multimedia
classroom tools (e.g., Horizon’s Wimba or Macromedia’s Breeze) that include white boards where the
instructor can write on his or her desktop – to graph, for example, or to derive a formula – and
students can view it on their computer screens in real time. Or, when such tools are available,
instructors are not always properly trained to use them. At a minimum, a bank of short video
presentations (produced by the instructor, or even simply “borrowed” from another source) on the
most common visually-oriented lecture topics can be extremely helpful.
The final theme to emerge from the data dealt with technological barriers; not only did most students
struggle with technology (both the LMS and SPSS), but in their already heightened sense of anxiety
they frequently attributed their struggles to a personal lack of ability to handle the course content
rather than to an ordinary learning curve. Dedicating a classroom conference discussion thread to
“Technical Issues,” and reassuring students that such issues are normal and to be expected, can help
to alleviate any additional stress caused by technology. Challenges arise here when the instructors,
themselves, are not quite ‘tech-savvy’ and unable to answer more than the most basic question. In
this case, instructors are well-served either to allow a technology helper into the classroom (this can
be a student or a T.A.; in fact a peer tutor – student who has already successfully completed the
course – is a good choice) or to start the semester with explicit guidance for students about where
they should go if they have technical questions. When the technology gets in the way of learning, then
we are failing to truly realize the potential of the online classroom.
5.1 Limitations and suggestions for future research
Case study research results often present threats to external validity because the ability to generalize
the findings broadly can be limited (Yin 2009). The validity of, meaningfulness of, and perspectives
gained from the qualitative inquiry arise from the information richness of the individual participant
responses as well as the observations and analysis by the researcher (Patton, 2002). Readers are
the ultimate arbiters of what can be generalized to their specific contexts.
108

Leslie Dinauer
Additionally, this was a small class which, therefore, can present only a limited number of points of
view. (It bears mentioning, however, that the themes which emerged from this class are consistent
with the researcher’s experience in previous semesters.)
In the future, additional studies (both quantitative and qualitative) focused on comparing the attitudes
and affect of students in the online classroom versus face-to-face should be conducted, with an eye
toward looking for causal relationships between those qualities and mastery of content. An additional
need is for studies that compare online classrooms in which different anxiety reduction plans are
used; again, there should be a focus on trying to determine not only if certain practices reduce
anxiety, but also if reduced anxiety is, then, related to improved outcomes.
References
Allen, I. and Seaman, J. (2007). Online National: Five Years of Growth in Online Learning, Needham, MA: Sloan
Consortium Publishing.
Baloglu, M. and Zelhart, P. (2003) “Statistical Anxiety: A Detailed Review”, Psychology and Education, Vol 40,
pp. 27-37.
Boettcher, J. and Conrad, R. (1999) “Distance Learning: A Faculty FAQ”, [online], University of North Carolina,
http://technologysource.org/article/distance_learning_a_faculty_faq (accessed 1/20/2001)
Christie, M. and Jurado, R. (2009) “Barriers to Innovation in Online Pedagogy”, European Journal of English
Education, Vol 34, pp. 273-279.
Collins, K. and Onwuegbuzie, A. (2007) “I Cannot Read My Statistics Textbook: The Relationship Between
Reading Ability and Statistics Anxiety”, The Journal of Negro Education, Vol 76 pp. 118-129.
Creswell, J. (2009) Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, Thousand
Oaks, CA: Sage Publications.
Cruise, R. and Wilkins, E. (1980) STARS: Statistical Anxiety Rating Scale. Unpublished.
Cruise, R., Cash, R. and Bolton, D. (1985) “Development and Validation of an Instrument to Measure Statistics
Anxiety”, Proceedings of the American Statistical Association, pp. 92-96.
DeVaney, T. (2010) “Anxiety and Attitude of Graduate Students in On-Campus vs. Online Statistics
Courses”, Journal of Statistics Education, Vol 18, pp. 1-15.
Dunn, D., Smith, R., and Beins, B. (eds.) (2007) Best Practices for Teaching Statistics and Research Methods in
the Behavioral Sciences, Mahwah, NJ: Lawerence Erlbaum Associates.
Glasser, B. and Strauss, A. (1967) A Discovery of Grounded Theory: Strategies for Qualitative Research, Mill
Valley, CA: Sociology Press.
Greenwood, D. and Levin, M. (2007) Introduction to Action Research: Social Research for Social Change, 2 nd
edition, Thousand Oaks, CA: Sage Publications.
Hsu, M., Wang, S. and Chiu, K. (2009) “Computer Attitude, Statistics Anxiety and Self-Efficacy on Statistical
Software Adoption Behavior: An Empirical Study of Online MBA Learners”, Computers in Human Behavior,
Vol 25 pp. 412-420.
Hulsizer, M. and Woolf, L. (2009) A Guide to Teaching Statistics: Innovations and Best Practices, Malden, MA:
Wiley-Blackwell.
Lalonde, R. and Gardner, R. (1993) “Statistics as a Second Language? A Model for Predicting Performance in
Psychology Students”, Canadian Journal of Behavioral Science, Vol 25, pp. 108-125.
Lincoln, Y. and Guba, E. (1985) Naturalistic Inquiry, Beverly Hills, CA: Sage Publications.
Mann, S. (2005) “Alienation in The Learning Environment: A Failure of Community?”, Studies in Higher
Education, Vol 30, pp. 43-55.
Mills, J. and Raju, D. (2011) “Teaching Statistics Online: A Decade’s Review of the Literature About What
Works”, Journal of Statistics Education, Vol 19, pp. 1-28.
Onwuegbuzie, A. (2000) “Statistics Anxiety and the Role of Self-Perceptions”, Journal of Educational Research,
Vol 93, pp. 323-330.
Patton, M. (2002) Qualitative Research and Evaluation Methods, 3 rd edition, Thousand Oaks, CA: Sage
Publications.
Sebastianelli, R. and Tamimi, N. (2011). “Business Statistics and Management Science Online: Teaching
Strategies and Assessment of Student Learning”, Journal of Education for Business, Vol 86, pp. 317-325.
Yin, R. (2009) Case Study Research: Design and Methods, 4 th edition, Thousand Oaks, CA: Sage Publications.
Zanakis, S. and Valenza, E. (1997) “Student Anxiety and Attitudes in Business Statistics”, Journal of Education
for Business, Vol 73, pp.10-16.
Zeidner, M. (1991) “Statistics and Mathematics Anxiety in Social Science Students: Some Interesting Parallels”,
British Journal of Educational Psychology, Vol 61, pp. 319-328.
Zerbolio, D. (1999) “A Bag of Tricks for Teaching About Sampling Distributions” in Ware, M. and Brewer, C.
(eds.) Handbook for Teaching Statistics and Research Methods, 2 nd edition, Mahwah, NJ: Lawerence
Earlbaum Associates.
109

Systematic Review of Empirical and Psychometric Studies
on Organizational Commitment Conducted in Turkey
Serkan Dolma, Ozlu Azakli, Yagizhan Yazar and Talha Demirbas
Istanbul University, Turkey
serkandolma@gmail.com
ozluazakli@gmail.com
yagizhanyazar@gmail.com
talha@istanbul.edu.tr
Abstract: Although it is far from a consensus, organizational commitment is considered to be a multi-component
construct by many researchers of the subject area. Several different multi-component conceptualizations are
proposed in the literature, but the most widely preferred model (therefore, the most frequently applied “nonunidimensional”
scale) in empirical studies is that of Allen and Meyer (1990). In their model, Allen and Meyer
theorize 3 related but different components of organizational commitment, namely Affective, Continuance and
Normative Commitment components. They stated that they specifically refer to them as components, i.e. not as
types or factors, because an employee’s relationship with an organization might reflect varying degrees of all
three (Meyer & Allen, 1997). In other words, in this model (and in the most of the other multi-component models
as well) the term organizational commitment is not considered to be a construct per se, but just the group name
of these three constructs. Notwithstanding this, a casual perusal of the international journals reveals that this fact
has, albeit exceptionally, eluded a few researchers who either (i) computed a “total organizational commitment
score” besides the three composite scores or (ii) performed a second-order factor analysis of the three
components, both of which are clear signs of a practice that is not in compliance with the model. Contrary to the
international literature, papers with these kinds of unfortunate oversights are quite common in the scientific
journals in Turkey, let alone being exceptional. It prompted us to turn this casual observation into a systematic
review and to conduct an exhaustive survey of the empirical organizational commitment studies carried out in
Turkey, in order to assess the pervasiveness of the aforementioned poor practices. Besides our initial motivation,
we would like to provide a comprehensive systematic review of the organizational commitment research. Within
this framework, we surveyed the empirical studies on organizational commitment conducted in Turkey since 1998
and identified and thoroughly examined (i) 69 articles from twelve social science journals (ii) 75 papers from the
proceedings of the four of the most prominent national conferences on the related areas and (ii) 55 doctoral
dissertations which have organizational commitment as one of the variables. We provided summarized
information regarding the variables of the studies, the measurement instruments used to assess organizational
commitment, and the statistical methods utilized for data analysis, and several other methodological properties of
these studies that are included in this review.
Keywords: organizational commitment, three-component model, systematic review, composite score
1. Background and Introduction
Organizational commitment continues to be one of the most researched topics in organizational
behavior field. Yet, as it is also true for many other topics in social and behavioral sciences, there is
little unanimity among scholars in conceptualizing organizational commitment or in defining its
structural properties. Several scholars, such as Mowday, Porter and Steers (1979) and Jaworski and
Kohli (1993), define organizational commitment as a unidimensional concept while others, like Meyer
and Allen (1990, 1997) and O’Reilly and Chatman (1986) specify it as a multi-component construct.
We will focus on Meyer and Allen’s three-component conceptualization of organizational commitment
in this paper.
In their review of the commitment literature, Meyer and Allen (1991) observed that, although the
definitions of the concept vary, they mainly reflect three broad themes. Commitment has been viewed
as either (i) reflecting an affective orientation toward the organization, (ii) a recognition of costs
associated with leaving the organization, or (iii) a moral obligation to remain with the organization.
Based on this observation, to acknowledge that each of the three sets of definitions represent a
legitimate but distinct conceptualization of the commitment construct, Meyer and Allen proposed a
three-component model of organizational commitment (1990, 1991 and 1997).
According to this classification, ‘affective commitment’ refers to the employee’s emotional attachment
to the organization whereas ‘continuance commitment’ refers to an awareness of the costs associated
with leaving the organization. ‘Normative commitment’, on the other hand, reflects a feeling of
obligation to continue employment (Meyer and Allen, 1991). Meyer and Allen used the term
component, instead of factor, in order to reflect their view that affective, continuance and normative
110

Serkan Dolma et al.
commitments are not sub-factors of a general commitment construct. Rather, they are three different
connotations that the term organizational commitment used for in the literature. The implication of this
interpretation is that organizational commitment is not a psychological construct per se. Instead, it is
just a rubric under which the three components are grouped. Therefore, ‘organizational commitment’
should not be a variable in an empirical study, if it was based on Meyer and Allen’s conceptualization.
It is rather a general label, which may be used as a shorthand way of referring to the three
components as a group.
Although Meyer and Allen or others have not explicitly elucidated that organizational commitment, in
and of itself, should not be treated as a construct/variable in a given empirical study, most researchers
perceive the phenomenon as it is intended. Therefore, majority of studies have not calculated or
employed the “total score” for organizational commitment as one of the variables of the research
model (Some exceptions are; Hunton and Norman, 2010; Jain, Giga, and Cooper, 2009 and Kang,
Stewart, and Kim, 2011). It might be tempting to calculate the composite score, for the sake of
measuring “the overall commitment to organization”, by summing the three scores obtained from the
components. However, one should bear in mind that affective, normative and continuance
commitments are not sub-factors or constituents of commitment. They are merely, in O’Reilly and
Chatman’s terms (1986), bases of commitment. They are the mechanisms by which commitment
forms or develops. That is why, for example, an employee who has a high score on all three of the
components cannot be evaluated as “more committed” than an employee who has a very high score
on affective commitment but not on the others. Having a high score on all of the components may just
mean that, that employee has not only one but three different types of reason to commit to the
organization (i.e. not to quit his or her organization). For instance, she might not be planning to quit in
the near future, not only because he/she is affectively attached to the organization, but also because
there are no better alternatives present. We cannot claim that this person has a higher level of
commitment to or identification with the organization compared to one who stays with the organization
merely because of her emotional attachment to or high level of involvement in the organization. From
one point of view, these components are reasons of individuals to be committed to their organizations
and having more reasons may not mean having higher level of commitment since not all of the
reasons are considered to be pro-organizational.
Another inappropriate practice, which is effectively equivalent to the fallacy explained above, would be
conducting a higher-order factor analysis on data acquired by using the three-component scales.
Hypothesizing a second order (i.e. overarching) latent construct, of which the components are the
effects or reflections, tantamount to calculating the composite score, since, by doing this one still
conceptualizes a construct that would be interpreted as an overall commitment (see Figure 1). A
second-order commitment factor would imply that there exists a common cause of the affective
normative and continuance commitment, which does not make much sense. Fortunately, the number
of studies in which the higher-order factor analysis has been invoked is even less than that of studies
in which the composite score fallacy has been committed (See, Park and Rainey, 2007 and Chi-
Cheng, Meng-Chen, and Meng-Shan, 2011 for exceptions).
d1
Affective
Commitment
AC Item 1 AC Item 2 AC Item 3 AC Item 4
d2
Organizationa
l Commitment
Normative
Commitment
NC Item 1 NC Item 2 NC Item 3 NC Item 4
1
d3
Continuance
Commitment
CC Item 1 CC Item 2 CC Item 3 CC Item 4
Figure 1: Path diagram of a hypothetical higher-order factor model for organizational commitment
111

Serkan Dolma et al.
Contrary to the international literature, we observed that papers, in which the unfortunate oversight of
utilizing the total (composite) commitment score is committed, are quite common in the scientific
journals in Turkey, let alone being exceptional. It prompted us to turn this casual observation into a
systematic review and to conduct an exhaustive survey of the empirical organizational commitment
studies carried out in Turkey, in order to assess the pervasiveness of the aforementioned poor
practice. Since this kind of review requires a thorough examination of the literature, it would be
regrettably uneconomical to only examine studies to check whether they entail the aforesaid fallacy.
Accordingly, we decided to broaden our initial motive and review the organizational commitment
studies also with regard to their other methodological properties. This type of research is called
systematic review, which is defined as a type of study that “aims to comprehensively locate and
synthesize research that bears on a particular question, using organized, transparent, and replicable
procedures at each step in the process” (Littell, Corcoran and Pillai, 2008). A short summary of our
review is presented in the next part.
2. Review of the organization commitment studies conducted in Turkey
For the systematic review part of our study, we took 1998 as the starting year of our national literature
search and surveyed 13 national scientific journals, 4 local conferences and the national doctoral
dissertation database. We identified and retained 193 empirical studies that employed organizational
commitment or component(s) of commitment as one of the variables in their research model. Six
psychometric studies, which focus on the development or validation of an organizational commitment
scale are also included in the review (for only applicable summarizations), while theoretical texts or
studies in which organizational commitment has not been measured are excluded. As a result, 69
journal articles, 75 papers (which are published in conference proceedings) and 55 doctoral
dissertations are examined within this review. The distribution of the included studies per publication
year is showed in Figure 2.
2.1 The dependent and independent variables
The examination of the studies included in our review showed that rather than the possible effects of
commitment on organizational or individual factors, the antecedents or predictors of organizational
commitment were often investigated. Hence, commitment was the independent/predictor variable in
73 percent of the studies whereas it served as the dependent variable in only 13 percent of the
retained papers and dissertations. Commitment was the interaction variable in only a small proportion
of the retained studies, therefore, very rarely the possible mediating effect of commitment was
investigated (7.5%).
Wide range variables were investigated in these empirical studies as a possible predictor of the
commitment, yet few concepts were relatively more frequently employed. Demographic factors (f=38)
such as age and income, leadership (f=21), perceived organizational justice (f=15), job satisfaction
(f=11) and organizational support (f=7) are the most frequently used independent variables. Although
commitment was an independent variable in only few researches, it might be worthwhile to note that
organizational citizenship behavior and job performance are the most frequently used criterion
variables among these studies.
Figure 2: The distribution of the retained studies per publication year
112

2.2 Data analysis methods
Serkan Dolma et al.
Regardless of the role assigned to commitment in the research model (i.e. whether it is the
independent, dependent or mediator variable), correlation/regression analysis was the most
commonly utilized method of analysis (f=141). 29 papers employed analysis of variance while only 15
empirical studies tested models with structural equation modeling technique.
2.3 The organizational commitment scales
The measurement instruments that were used and the reliability and validity analyses that were
conducted in the retained studies were also examined: As a part of validity analysis, 38 percent of the
studies invoked exploratory factor analysis while 11 percent used confirmatory factor analysis (5
studies employed both exploratory and confirmatory factor analyses). This means that, approximately
in half of the studies, the factorial structure of the measurement instrument or its construct validity for
the sample under investigation was not analyzed. Compared to validity analyses, reliability of the
measures is more commonly assessed by the researchers. In majority of the research (81%)
Cronbach’s alpha reliability index was applied and reported.
Table 1: Organizational commitment scales used in the retained studies
Multi Dimensional Scales (Total) 132
Meyer - Allen (1990, 1997, 2004) & Meyer - Allen - Smith (1993) 121
Cook - Wall (1980) 3
O'Reilly - Chatman (1986) 1
Buchanan (1974) 1
Other Multi Component Scales 6
One Dimensional Scales (Total) 48
Mowday - Porter - Steers (1979) 41
Jaworski - Kohli (1993) 5
Hunt - Chonko - Wood (1985) 2
Other One Dimensional Scales 0
Specialized Scales Designed by the Author of the Corresponding Research 19
Total 199
2.4 The composite score fallacy
Our examination of the empirical studies on organizational commitment also revealed that, Meyer and
Allen’s three-component commitment scale is the most preferred measurement instrument by the
Turkish researchers while Mowday, Porter and Steers’ Organizational Commitment Questionnaire
(OCQ) is the second (see Table 1). It is obvious from Table 1 that organizational commitment is
deemed by most of the Turkish scholars as a non-unidimensional concept. It would be fair to say that
there is a tendency among Turkish scholars to view commitment as a concept, which involves multiple
components. Although Meyer and Allen’s model is the most frequently used one, not in all of these
studies that employed Meyer and Allen’s scale, was the organizational commitment of interest as a
whole (i.e. with it’s all components). In 49 studies, only one of the three components is integrated into
the research model as a variable, therefore, only the items measure that component are asked the
participants. 72 studies in which Meyer and Allen’s scale was utilized, investigated all three of the
components’ relationships with other variables. Of these 72 studies, 40 of them calculated the “total
score for organizational commitment”, besides or instead of the individual scores for each construct,
and treat this composite score as a variable. In other words, authors of these studies assumed that
organizational commitment is a construct in and of itself, although they based their measurement
model on Meyer and Allen’s conceptualization. We considered this approach to be a type of fallacy,
rationale of which is described in the first part of the current paper. It should be noted that the ratio of
the papers in which aforementioned fallacy is observed to the total number of studies, which used the
three-component model, is much higher compared to the international literature on commitment. It
may be seen as a sign that many Turkish researchers, unfortunately, do not pay enough attention to
the model posited by Meyer and Allen, although they prefer to use the scale that is developed based
on this model.
113

3. Conclusions
Serkan Dolma et al.
The objective of this study was two-fold. Firstly, we tried to expound why it is illogical to invoke the
composite score for organizational commitment when Meyer and Allen’s commitment scale is used.
Although they are few, papers exist in the commitment literature that fell into composite score fallacy.
Therefore, we believe this sort of explanation–that using total scores for commitment scales, which
are based on the three-component model, is an inappropriate practice–was necessary and to our
knowledge it has not been explicitly posited before.
Our second objective was to provide a summary description of the Turkish literature on organizational
commitment. Examination of the studies on commitment revealed that (i) Meyer and Allen’s
commitment scale is, by far, the most preferred measurement instrument in the field, (ii) researchers
are more inclined to use the total score in their analyses, compared to the international academia and
(iii), unfortunately, analyzing the validity of the scales for the sample under investigation is not a
common practice among Turkish researchers.
References
Allen, N. J., Meyer, J. P. (1990) “The Measurement and Antecedents of Affective, Continuance and Normative
Commitment to the Organization”, Journal of Occupational Psychology, Vol.63, No.1, pp 1-18.
Buchanan, B. (1974) “Building Organizational Commitment: The Socialization of Managers in Work
Organizations”, Administrative Science Quarterly, Vol.19, No.4, pp 533-546.
Chi-Cheng, C., Meng-Chen T., Meng-Shan T. (2011) “The Organizational Citizenship Behaviors and
Organizational Commitments of Organizational Members Influences the Effects of Organizational Learning”,
International Journal of Trade, Economics and Finance, Vol.2, No.1, pp 61-66.
Cook, J., Wall, T. (1980) “New Work Attitude Measures of Trust, Organizational Commitment and Personal Non-
Fulfillment”, Journal of Occupational Psychology, Vol.53, No.1, pp 39-52.
Hunt, S. D., Chonko, L. B., Wood, V. R. (1985) “Organizational Commitment and Marketing”, The Journal of
Marketing, Vol.49, No.1, pp 112-126.
Hunton, J. E., Norman, C. S. (2010) “The Impact of Alternative Telework Arrangements on Organizational
Commitment: Insights from a Longitudinal Field Experiment”, Journal of Information Systems, Vol.24, No.1,
pp. 67–90.
Jain, A. K., Giga, S. I., Cooper,C. L. (2009) “Employee Wellbeing, Control and Organizational Commitment”,
Leadership & Organization Development Journal, Vol.30, No.3, 2009, pp 256-273.
Jaworski, B. J., Kohli, A. K. (1993) “Market Orientation: Antecedents and Consequences”, The Journal of
Marketing, Vol.57, No.3, pp 53-70.
Kang, D., Stewart, J., Kim, H. (2011) “The Effects Of Perceived External Prestige, Ethical Organizational Climate,
And Leader-Member Exchange (LMX) Quality On Employees’ Commitments And Their Subsequent
Attitudes”, Personnel Review, Vol.40 No.6, pp 761-784.
Littell J. H., Corcoran, J., Pillai, V. (2008) Systematic Reviews and Meta-Analysis, Oxford University Press,
Newyork.
Meyer, J. P., Allen, N. J. (1997) Commitment in the Workplace: Theory, Research, and Application (Advanced
Topics in Organizational Behavior), Sage Publications, U.S.A.
Meyer, J. P., Allen, N. J. (1991) “A Three-Component Conceptualization of Organizational Commitment”, Human
Resource Management Review, Vol.1, No.1, pp 61-89.
Meyer, J. P., Allen, N. J. (2004) “TCM Employee Commitment Survey, Academic Users Guide”, [online],
University of Western Ontario, http://audacityblog.info/wordpress/wp- content/uploads/2011/03/Meyer-Allen-
Empl-Commitment-Survey.pdf.
Meyer, J. P., Allen, N. J., Smith, C. A. (1993) “Commitment to Organizations and Occupations: Extension and
Test of a Three-component Conceptualization”, Journal of Applied Psychology, Vol.78, No.4, pp 538-551.
Mowday, R. T., Steers, R. M., Porter, L. W. (1979) “The Measurement of Organizational Commitment”, Journal of
Vocational Behavior, Vol.14, No.2, pp 224-247.
O’Reilly, C., Chatman, J. (1986) “Organizational Commitment and Psychological Attachment: The Effects of
Compliance, Identification, and Internalization on Prosocial Behavior”, Journal of Applied Psychology,
Vol.71, No.3, pp 492-499.
Park, S. M., Rainey, H. G. (2007) “Antecedents, Mediators, and Consequences of Affective, Normative, and
Continuance Commitment Empirical Tests of Commitment Effects in Federal Agencies”, Review of Public
Personnel Administration, Vol.27, No.3, pp 197-226.
114

Online Formative Assessment: Does it add up to Better
Performance in Quantitative Modules?
Elena Fitkov-Norris and Becky Lees
Kingston University, London, UK
e.fitkov-norris@kingston.ac.uk
r.lees@kingston.ac.uk
Abstract: Students who follow a social science programme often find quantitative methods challenging. Poor
numeracy skills impact upon performance of tertiary education students, particularly in the first year of their
programme, and since first year performance has the greatest impact upon retention (Yorke 2005), supporting
students to develop their numeracy skills early on in higher education is vital. Previous studies concentrating on
the use of online formative assessment methods to support learning in quantitative modules report positive
benefits, and that exposure to online formative tests is sufficient to enhance subsequent summative exam
performance (Angus and Watson 2009). Given these benefits a similar assessment framework was implemented
on a first year quantitative methods module taken by approximately 400 business students, many of whom have
not taken any mathematical studies since their GCSEs. The aim of this study was to consider whether student
participation in a series of online formative assessments had an impact upon overall performance on this module,
both within the formative tests and subsequent summative exam. This empirical study analysed the participation
and performance data from a cohort of business students, taking into consideration their level of prior attainment,
extrinsic motivation, gender and age. A retrospective ridge regression model was fitted and used to examine the
nature of the relationship between participation and performance in online formative assessments and overall
module attainment, as preliminary analysis identified high collinearity between formative test participation and
formative test grades. The results suggest that this data set does not support the hypothesis that participation,
rather than performance, in formative assessment is a better predictor of student end-of-year exam performance.
However, there is evidence that although small, the impact of participation in formative assessment on end of
year assessment is positive and significant. Furthermore, the study confirmed that prior attainment in quantitative
modules is a significant predictor of future attainment, particularly for male students and that student motivation
plays a significant part in overall achievement.
Keywords: online formative assessment, performance, evaluation, quantitative methods
1. Introduction
Assessment forms an indelible and sometimes controversial part of education, with claims of overassessment
in schools making appearances in headlines with predictable regularity. Although not
subjected to the same scrutiny, the subject of assessment in higher education has also attracted
considerable attention. The changing nature of higher education in terms of raised expectations,
larger classes and more diverse student populations has lead to more research into the efficacy of
existing assessment and the introduction of more innovative assessment methods (Bartram and
Bailey 2010). This in turn has lead to the foregrounding of the different types of assessment, with the
majority of all discussions focused on the relationships between summative and formative methods of
assessment.
Summative assessment is widely agreed to be a form of evaluation that provides a judgment which
encapsulates all the evidence up to a given point, usually the end of a teaching period (Taras 2005).
It is a powerful tool that has the purpose of providing a student with a grade, thus enabling them to be
differentiated amongst their peers (Wininger 2005). The research on summative assessment methods
exposes several criticisms of the approach. (Yorke 2005) leads a discussion as to whether students’
final achievements are adequately represented by summative assessment methods, in particular as
to the scale of the final awards. The other major criticism of summative assessment is that it focuses
too much on ‘How did I do?’ rather than ‘How am I doing?’ (Kibble et al. 2011), which supports the
now extensive body of research into methods of formative assessment that help students develop as
learners (Yorke 2005) and permit interventions during learning rather than after learning has taken
place (Black and Wiliam 1998).
If an assessment method provides a student with an opportunity to gauge their progress whilst there
is time to rectify any lack of understanding, it starts to take on a formative role rather than a
summative one. Since the seminal review on formative assessment methods by (Black and Wiliam
1998), the definition of formative assessment has been discussed, debated and developed through
several iterations, all of which have the intention to improving student learning at their core.
115

Elena Fitkov-Norris and Becky Lees
Several studies suggest that formative assessment must provide fast feedback in order to be useful,
although this might cause tensions given the fact that there has been a shift towards modularisation
and semesterisation in UK HE, causing reduced timeframes for the provision of feedback (Yorke
2005; Bartram and Bailey 2010). The implications of time and implementation requirements in a
shorter timeframe has helped encourage the development of online assessment methods a ‘low-cost,
reusable, customisable and scalable initiative’ (Armellini and Aiyegbayo 2009) and although this
approach carries a fixed overhead, it is better suited for a mass approach to higher education than
more traditional methods (Yorke 2005) and in general students are receptive to the use of online
assessment methods (Dermo 2009).
Many quantitative studies confirm that formative assessment contributes positively to gains to student
understanding and achievement (Black and Wiliam 1998; Angus and Watson 2009; Dobson 2008;
Kibble et al. 2011) and employability (Yorke 2005). However, some research studies report less
favourably on the benefits of formative assessment and there are examples of contradictory results of
the effect of formative assessment on performance (Dunn and Mulvenon 2009). Therefore, the
research would suggest that formative assessment could be both constructive and inhibitory for
learning achievement (Yorke 2005). Part of the reason for the lack of clarity over the effects of
formative assessment may relate to confusion in the extent to which an assessment event is classed
as either formative or summative (Bennett, 2009; Yorke, 2005; Taras, 2008) and difficulty in its
definition and understanding given the term formative has been open to misinterpretation (Bennett,
2009) and often seen as an add-on to regular teaching and learning methods rather than an integral
part of the overall strategy (Yorke 2005).
The purpose of this paper is to examine the impact of frequent formative assessment on student
performance and examine the validity of this as a support tool for the learning of students on a
general business degree participating in a quantitative methods module. It is recognised that
students who follow a social science based programme find quantitative methods challenging
(Murtonen and Lehtinen 2003) and this awareness has encouraged research into the effectiveness of
frequent formative assessment in supporting student’s learning. Of particular interest is a large-scale
study by Angus and Watson (2009), which concluded that it is exposure to, rather than performance
in, frequent online formative assessment, which has a significant positive impact on end-of-module
examination scores for students on a business mathematics module. This empirical study will test if
data from a large sample confirm Angus and Watson’s (2009) findings by analysing the participation
and performance data from a cohort of business students, controlling for level of prior attainment,
motivation and gender biases. This study aims to further Angus and Watson’s (2009) work by
attempting to assess the relative predictive power of performance and participation in frequent online
formative tests in end of year exam performance.
2. Educational context
The Quantitative Methods module is offered as a core module for all year one students across a
range of programmes ranging from general business management to specialist programmes such as:
Marketing, Human Resource Management or Accounting and Finance. The assessment for this
module consists of unseen end of module exam and individual coursework. The exam comprises of
two sections: a multiple choice questions section which covers the entire module content and an indepth
questions section which allows students to demonstrate more detailed knowledge on topics
covered in the module. The coursework is a case study which assesses students’ critical analysis
skills. The overall module mark also includes a small contribution from four formative assessment
tests which the students complete online. Each test can be taken twice so a student unhappy with
their first grade has an opportunity to improve it. Test questions are drawn at random from a large
question bank so a student taking the test for the second time will not be answering questions
identical to their first attempt.
The sample consists of 462 students who sat the end of year exam and summarises the number of
formative tests they attempted, their average grade from the tests and exam grade, together with their
gender, type of degree (general business, specialist or Accounting and Finance), length of degree (3year
or 4-year depending on their intention to do a placement) and age at the start of the module. The
additional indicator variables for gender, type of degree (general business, specialist or Accounting
and Finance), length of degree (3-year or 4-year) were added to align the analysis more closely with a
similar study done by (Angus and Watson 2009).
116

Elena Fitkov-Norris and Becky Lees
The average grade attained by student in the end of year exam is 60%, the formative test grade is
65% and the majority of students attempted all 4 formative tests (Table 1).
Table 1: Number of formative tests attempted.
Valid
Number of Formative Tests Attempted
Frequency Percent Valid Percent Cumulative Percent
0 10 2.2 2.2 2.2
1 16 3.5 3.5 5.6
2 31 6.7 6.7 12.3
3 109 23.6 23.6 35.9
4 296 64.1 64.1 100.0
Total 462 100.0 100.0
A slightly larger proportion of male students (51.5%) sat the exam. The majority of students (53.5%)
have registered for a general business degree (i.e. Business Management, Business Administration,
Business Studies), 33.5% have enrolled on specialist degrees (Marketing, Human Resource
Management) and 13% of students in the sample have enrolled on an Accounting and Finance (A&F)
degree. Accounting and Finance students are required to have achieved at least a grade B at GCSE
Mathematics, while all other students have achieved the lower requirement of at least a grade C at
GCSE Mathematics. The degree type would be used as a control variable for prior attainment in
Mathematics. Its effect may be somewhat mitigated as there could be students who enrolled on a
general or specialist business degree who may have attained at better GCSE Mathematics grade,
however, gathering information about students exact entry qualifications poses practical and ethical
challenges.
The indicator variable for degree length (3-years or 4-years) depending on whether the student
intends to do a work placement or not, is included as a control for the extrinsic goal orientation of the
students (Midgley et al. 1998). Students who enrol on the 4-year version of the degree would be
motivated to impress potential employers with their year one grades, while students on the 3-year
version have no such incentive as the year graded do not count towards final degree classification
under current regulations. The majority of students in the sample (69%) enrolled on the 3-year version
of their degree.
The mean, median, and mode age of the students is 19, with 79% under the age of 21.
3. Research design
Retrospective multivariate regression methodology was used to investigate the potential predictive
impact of number of formative tests attempted and the test grades on end of year exam performance.
Exam grade (%) was entered as the dependent variable, while participation in number of formative
tests and average formative test grade (%) were input as independent variables, together with
additional indicator variables for gender, degree type (general, specialist or A&F) and degree length
(3-year or 4-year). As mentioned earlier degree type is used a proxy for prior attainment in
Mathematics, while degree length is a proxy for students extrinsic motivation. The analysis was
carried out using PASW Statistics 18.0 and estimated the coefficients for the following equation.
Final Exam Grade = + x Formative Test Grade + x Number of Formative Tests Attempted
+ x Degree Type + x Degree Length + x Gender
This paper sets out to test if the sample data supports the following hypothesis:
The number of formative tests attempted, rather than the grade attained in these is a good predictor
of students’ performance in the end of year exam.
Initial investigation of the data showed that the number of tests attempted and the average formative
test grade are highly correlated, and a collinearity test was performed as part of the regression
analysis. This identified high multicollinearity between number of formative tests attempted and
formative test grade as the Variance Inflation Factors (VIF) for both number of tests and test grade
were greater than 2 (both in the region of 3.4) and their respective condition indices are greater than
15 which are the suggested as benchmark values for the presence of multicollinearity in PASW
statistics. Collinearity is a problem that leads to the estimation of unstable regression coefficients,
which should not be interpreted. Removing either number of tests or the test grade from the model is
117

Elena Fitkov-Norris and Becky Lees
not an option, as the objective is to compare the impact of the two variables on exam grade.
Therefore, the coefficient estimates for the regression function were estimated using a ridge
regression approach.
Ridge regression has been identified as a suitable approach for dealing with multicollinearity and
identifying both stable regression coefficients and significant variables (Marquardt and Snee 1975).
Ridge regression introduced a penalty term to the standard regression equation with a coefficient
lambda, which acts as bias in the estimates, and often significantly reducing the variance of the
estimated model coefficients. The regression equation is then run for different values of lambda and
an optimal value for lambda is chosen by using standard cross validation or bootstrapping techniques
(Marquardt and Snee,1975; Hoerl and Kennard, 1970).
PASW Statistics provides the ridge regression technique in combination with optimal scaling and
categorical regression. This approach has the added advantage of identifying optimal scales for the
nominal and ordinal variables in the model. The ridge regression analysis was run with a .623
bootstrap estimate that collected 50 samples. The optimal value for lambda was 0.56. The results
were validated by running the analysis at several random starting points all producing very similar
solutions.
Table 2 shows the optimal scaling (transformed values) of the independent variables as identified by
PASW Statistics.
Table 2: Transformed values for independent variables
Variable Scale Discretization Transformed values
Exam grade Numeric N/A Numeric
0 tests = − 4.57
1 test = − 3.42
Number of Test Ordinal N/A
2 tests = −.81
3 tests = .09
4 tests = .39
Formative Test Grade Ordinal Intervals length 10 10 equal intervals
General = −.31
Degree Type (general, specialist or A&F) Nominal N/A
Specialist = −.51
A&F = 2.58
Degree Length (3- year or 4- year) Ordinal N/A
3-year = −.67
4-year = 1.49
Gender Nominal N/A
Male = −.97
Female = 1.03
Table 3 shows that the transformation of formative test grade and number of tests attempted
increased the tolerance of the two independent variables, and collinearity diagnostic tests confirm that
collinearity is no longer a problem. Therefore, the estimated coefficients can now be interpreted. Zero
order correlations confirm the correlation between the transformed independent variables and the
transformed exam grade with formative test grade is moderate to strong (r = .582), while the
correlation between transformed exam grade and transformed number of tests attempted is moderate
(r = .366). The partial correlations confirm that formative test grade is the most significant predictor of
variance in exam grade (20%), if the effect of all other variables is eliminated from both the end of
year exam grade and the formative tests grade. All remaining predictors explain a small amount of
variance in exam grade once the effects of all other independent variables are removed - around 2%
each for degree type and degree length, and less than 1% each for gender and, surprisingly, number
of tests attempted. This suggests that the formative test grade is the most significant predictor of end
of year exam performance as the proportion of variation in exam performance explained by the
number of tests attempted is also explained by the other independent variables included in the model.
The results from the ridge regression analysis presented in Table 4 show that overall the independent
variables explain 36% of the variance in exam grade, with all independent variables being statistically
significant. Although none of the coefficients are particularly large, formative test grade is the most
significant predictor of exam performance, with one standard deviation increase in formative test
grade yielding a .307 increase in the standard deviation of exam grade. Number of tests attempted
and degree type are second in terms of their impact on exam performance with one standard
deviation increase in number of tests and degree type leading to around .11 increase in the standard
118

Elena Fitkov-Norris and Becky Lees
deviation of exam performance. As a nominal variable, the coding of degree type needs to be
interpreted alongside the optimal coding recommended by PASW.
Table 3: Correlations and tolerances
Correlations and Tolerance
Correlations Tolerance
Zero-
After
Before
Order Partial Part Transformation Transformation
Degree type (general, specialist, A&F) .221 .175 .141 .930 .950
Degree length (3-year or 4-year) .112 .114 .091 .988 .991
Formative test grade .582 .454 .404 .588 .311
Number of tests attempted .366 .034 .027 .606 .315
Gender .088 .097 .078 .982 .973
Dependent Variable: Exam grade
Table 4: Ridge regression coefficients (penalty .56, .632 bootstrap estimate with 50 samples).
Coefficients
Standardized Coefficients
Bootstrap (1000) Estimate
Beta of Std. Error df F Sig.
Degree type (general, specialist, A&F) .112 .022 2 25.900 .000
Degree length (3-year or 4-year) .062 .023 1 7.628 .006
Formative test grade .307 .018 8 299.015 .000
Number of tests attempted .108 .020 4 28.523 .000
Gender .049 .023 1 4.455 .035
Dependent Variable: Exam grade
As it can be seen from Table 2, increasing the level of prior achievement in mathematics of students,
from GCSE Grade C for general business and specialist degrees to Grade B for Accounting and
Finance students, leads to an increase in the expected exam performance. The impact of degree
length and gender on exam performance is rather small. The data suggests that increasing the
degree length from 3 years to 4 years leads to an almost negligible increase in the standard deviation
of exam grade. The effect of gender (moving from male to female) is similar to the effect of degree
length in both direction and magnitude.
The interaction effects between the independent factors, namely degree length (3-year or 4-year),
gender and degree type (general, specialist or Accounting & Finance) and the dependent variable
(end of year exam performance) were also examined. The only significant interaction occurred
between gender and degree length. While the mean exam grade for females did not depend on
whether they had chosen a 3 or a 4-year degree, the mean exam grade for males who chose the 4year
version of their respective degree was significantly better than the mean grade for males who
chose the 3-year version. This suggests that degree length controls for the intrinsic motivation in male
students better that than in female students.
There also seems to be some weak interaction between gender and degree type. Although not
statistically significant, the mean final exam performance of male students on the Accounting &
Finance degree was better than females, while on the specialist and general business degrees,
females outperformed their male counterparts. This suggests prior attainment in mathematics is
perhaps more significant in predicting future success in male students. Further, qualitative research,
may shed further light on these interaction effects and their relationship to overall performance.
4. Results and discussion
The results show that the most significant predictor of end of year exam performance for this sample
of students is formative test grade, followed by number of formative tests attempted and the students’
prior achievement in mathematics which have small, but significant impact. Gender and degree length
119

Elena Fitkov-Norris and Becky Lees
(3-year or 4-year degree) all have negligible effect on exam performance, although they are all
statistically significant. Overall the, variables explain approximately 36% of the variation in end of year
exam performance.
These finding are in line with previous findings that frequent participation on formative assessment
has a positive impact on student summative achievement (Yorke 2005; Wininger 2005; Angus and
Watson 2009). However, the data do not support the hypothesis derived from Angus and Watson’s
(2009) study that it is participation in, rather than performance in frequent formative assessment that
is the most significant predictor of student performance. The formative test grade achieved has a
larger influence (.316) on end of year exam performance than number of tests attempted (.109).
Although direct comparison with Angus and Watson’s (2009) findings is difficult since they did not
report their standardised regression coefficients, it is clear that the magnitude of the test participation
coefficient estimate is much larger. This could be partly due to the fact that the current study attempts
to evaluate the incremental impact of test participation, with test participation rates ranging from 0 to
4, while Angus and Watson (2009) presented test participation as an indicator variable (0, if student
attempted up to and including 3 tests, and 1 if the student attempted all 4 tests). In addition, their
study does not seem to have encountered the problem of collinearity as they did not report any
collinearity diagnostics. Although they considered marks achieved by students in previous
mathematics courses rather than the actual online test grade in their study it would have been helpful
to see those. Although not presented here, a collinearity diagnostics test was run for the collinearity
between formative test grade and number of formative tests attempted, coded as an indicator variable
at the same scale as Angus and Watson (2009) and these indicated that collinearity is still present.
As expected, the impact of previous mathematics attainment on end of year exam grade is positive
and this finding is inline with the conclusions reached by Angus and Watson (2009). This is further
confirmed by carrying out an Independent Samples Kruskal-Wallis test which showed that there is no
significant difference in the mean number of formative tests attempted across the different degree
types (general, specialist or A&F). Despite this, Accounting and Finance students attained
significantly better results in both formative test grades and the end of year exam grade, than both the
specialist and general business students. There was no significant difference in the formative test or
end of year exam performance of students enrolled on a general business or a specialist business
degrees. Furthermore, interaction between degree type and gender suggests that prior attainment in
mathematics is more significant in predicting future success on a quantitative module for male
students, rather than females.
The main impact of gender on end of year exam performance is very small, with female students
performing slightly better than male students. Gender is not a significant differentiator of performance
in the formative tests, however the evidence suggest that girls tend to participate in significantly
higher number of tests than boys when an Independent Samples Mann-Whitney U Test is carried out.
This seems to lend support to the positive impact of participation in formative assessment on
performance, as other research has suggested that females tend to under perform in quantitative
subjects (Hargreaves et al. 2008).
The additional variable included in this study - degree length, which reflected whether a student is
expecting to do a placement or not, has a very small impact on end of year exam performance, with
students on the 4-year version of the programme performing slightly better than students on the 3year
version. This difference in performance is statistically significant as confirmed by an Independent
Samples Mann-Whitney U Test. The variable was included as an indicator to students’ extrinsic
motivation and the results suggest students who choose the 4- year version may be motivated to do
better. This is particularly evident for male students who seem to perform significantly better than the
male students on a 3- year version of their respective degrees. This could be because they are aware
that good year one results could contribute to getting a placement. On the other hand, it is possible
that students who choose the 4-year version of a programme have greater motivation from the onset
to make the most from their university experience. This point warrants some further investigation.
5. Conclusions
This paper set out to establish whether the data sample from a year one quantitative methods module
in a UK business school supports the hypothesis that it is the number of formative tests attempted
rather than the formative test grade that is a better predictor of end of year exam performance. Ridge
regression analysis, taking into account additional indicator variables such as gender, prior attainment
120

Elena Fitkov-Norris and Becky Lees
and degree length was used to estimate the impact of number of tests attempted and formative test
grade on exam performance. The results suggest that this data set does not support the hypothesis
that it is participation, rather than performance in formative assessment that is a better predictor of
student performance in end of year exams. However, there is evidence that, although small, the
impact of participation in formative assessment on summative end of year assessment is significant
and positive and therefore, there is some merit in using online formative tests as part of a quantitative
methods module. The study confirmed that prior attainment in a quantitative module is also a
significant predictor of future attainment, with weak evidence that this relationship is more pronounced
for male students. In addition, the analysis suggested that motivation plays a significant part in overall
achievement and this is particularly relevant for male students. The factor interaction effects suggest
that practitioners need to be keenly aware of the different motivational levels of male students who do
not have prior achievement in mathematics and may lack strong intrinsic motivation as they are at the
greatest risk of underachieving on a quantitative module.
The results demonstrate the complex nature of the relationships between formative online test
participation and end of year exam performance and highlight the need for further more in-depth
research in this area.
Acknowledgements
The authors would like to thank Dr David Martland for his helpful suggestions and advice.
References
Angus, S.D. and Watson, J. (2009) “Does regular online testing enhance student learning in the numerical
sciences? Robust evidence from a large data set”, British Journal of Educational Technology, 40, No. 2, pp
255–272.
Armellini, A. and Aiyegbayo, O. (2009) “Learning design and assessment with e-tivities”, British Journal of
Educational Technology, 41, No. 6, pp 922–935.
Bartram, B. and Bailey, C. (2010) “Assessment preferences: a comparison of UK/international students at an
English university”, Research in Post-Compulsory Education, 15, No. 2, pp 177–187.
Black, P. and Wiliam, D. (1998) “Assessment and classroom learning”, Assessment in Education: Principles,
Policy & Practice, 5, No. 1, pp 7–74.
Dermo, J. (2009) “e-Assessment and the student learning experience: A survey of student perceptions of eassessment”,
British Journal of Educational Technology, 40, No. 2, pp 203–214.
Dobson, J.L. (2008) “The use of formative online quizzes to enhance class preparation and scores on summative
exams”, AJP: Advances in Physiology Education, 32, No. 4, pp 297–302.
Dunn, K.E. and Mulvenon, S.W. (2009) “A critical review of research on formative assessment: The limited
scientific evidence of the impact of formative assessment in education”, Practical Assessment, Research &
Evaluation, 14, No. 7, pp 1–11.
Hargreaves, M., Homer, M. and Swinnerton, B. (2008) “A comparison of performance and attitudes in
mathematics amongst the “gifted.” Are boys better at mathematics or do they just think they are?”,
Assessment in Education: Principles, Policy & Practice, 15 No. 1, pp 19–38.
Hoerl, A. E., and Kennard, R. W. (1970) “Ridge regression: applications to nonorthogonal problems”,
Technometrics, 12, No. 1, pp 69–82.
Kibble, J.D. et al. (2011) “Insights Gained from the Analysis of Performance and Participation in Online Formative
Assessment”, Teaching and Learning in Medicine, 23, No. 2, pp 125–129.
Marquardt, D.W. and Snee, R.D. (1975) “Ridge regression in practice”, American Statistician, 29, No. 1, pp 3–20.
Midgley, C. et al. (1998) “The development and validation of scales assessing students' achievement goal
orientations”, Contemporary Educational Psychology, 23, pp 113–131.
Murtonen, M. and Lehtinen, E. (2003) “Difficulties experienced by education and sociology students in
quantitative methods courses”, Studies in Higher Education, 28, No. 2, pp 171–185.
Taras, M. (2005) “Assessment–summative and formative–some theoretical reflections”, British Journal of
Educational Studies, 53, No. 4, pp 466–478.
Wininger, S.R. (2005) “Using your tests to teach: Formative summative assessment”, Teaching of Psychology,
32, No. 3, pp 164–166.
Yorke, M. (2005) “Formative assessment in higher education: Its significance for employability, and steps
towards its enhancement”, Tertiary Education and Management, 11, No. 3, pp 219–238.
121

Alleviating Design Silence in Design Science Research: a
Proposal of a Design Method
Francis Gacenga, Aileen Cater-Steel, Mark Toleman and Wui-Gee Tan
University of Southern Queensland, Toowoomba, Australia
Gacenga@usq.edu.au
Aileen.Cater-Steel@usq.edu.au
Mark.Toleman@usq.edu.au
Wui-Gee.Tan@usq.edu.au
Abstract: Information Systems (IS) design science literature offers a plethora of findings on various aspects,
such as the general steps in design science, problem identification, objectives of solutions, and evaluation of the
artefacts. However, there appears to be a dearth of guidance on the design of the artefact itself. Information
Technology Service Management (ITSM) practitioners report challenges in measuring and reporting the
performance of ITSM. This area is identified as significant and lacking in research. We are developing a
framework that can be used for performance measurement of ITSM investments in organisations. The ITSM
performance measurement framework (PMF) will provide a basis of standardisation and performance comparison
for organisations implementing ITSM. The focus of this paper is the design methodology for the PMF framework.
Our work considers literature from IS design science as well as disciplines outside IS design. Previous IS
researchers developed frameworks that guide the IS design research process but do not provide details of the
design process. We extend their work by narrowing the focus on the design step found in IS design science
approaches giving a detailed treatment of the design step and delve further into design literature outside IS
design science. Design is, of course, a very creative endeavour and may not be solely process driven. Few
articles in IS design science address in detail or give any guidance on design per se. Using a design process
from outside IS design science advocating ‘designerly ways of knowing’ we apply the integrative Matching
Analysis Projection Synthesis approach. We integrate an IS design science research framework with a macro
cycle of analysis-projection-synthesis and a micro cycle of research-analysis-synthesis-realisation to create an
approach for designing the PMF. Steps for evaluation and communication are also described. The design
process proposed may be useful for other IS researchers seeking guidance on how to design the artefact in an IS
design science project.
Keywords: design science research, it service management, performance measurement framework, mixed
methods research, matching analysis projection synthesis approach
1. Introduction
In a research project to develop a performance measurement framework (PMF) for IT service
management (ITSM) we employ a multi-paradigmatic and multi-method approach that includes design
science research (DSR). Our review of Information Systems (IS) design science literature did not
uncover specific advice on the design step that is required in most DSR frameworks. The objective of
this paper is to highlight the silence on design in DSR and to propose a design approach for our
particular project. As well as being the research project that motivated the need to identify more
concrete and detailed guidance for the design step in DSR, the ITSM PMF serves as an example
application of DSR and the design approach proposed.
ITSM is a customer-oriented approach used by IT practitioners to manage IT operations organised
around IT services. ITIL® is the most widely used ITSM framework and is based on a library of books
that offer “best practices” for ITSM. ITIL is a lifecycle-based, process-oriented framework that
organizations can use to create, design, deliver and maintain customer-focused IT services.
This project addresses an area that has been identified as significant and lacking in research. ITSM
practitioners report challenges in measuring and reporting the performance of ITSM. We are
developing a framework that can be used for performance measurement of ITSM investments in
organizations. The framework will provide a basis of standardization and performance comparison for
organizations implementing ITSM and could be used to show a relationship between ITSM investment
and benefits from this investment. Our project develops a PMF for ITSM, an IT service management
tool. The research contributes to IS design theory by describing the “design decisions and design
knowledge that are intended to be manifested or encapsulated in an artifact, method, process or
system” (Gregor 2002). The research contributes to the existing ITSM body of knowledge by
addressing the gap that currently exists in ITSM performance measurement (Lahtela et al. 2010),
performance measures and methods.
122

Francis Gacenga et al.
The focus of this paper is on the design methodology for the ITSM PMF. The project uses a multiparadigmatic
and mixed-methods approach based on behavioural science and design science
paradigms. The mixed-methods approach is suitable to the project as it addresses the theoretical
challenge using science and, based on the empirical knowledge gained, develops a practical solution
using design science.
We use the Information Systems Design Research (ISDR) approach as proposed in Peffers et al.
(2008) as a foundation. An overview of the design research process model and behavioural science
research methods are shown in Figure 1.
Figure 1: Initial design science method process model (Based on Peffers et al. 2008)
In stage one we begin with a systematic literature review followed by a survey. In stage two we
conduct case studies and develop the ITSM PMF artefact. In stage three we evaluate the prototype
PMF and enhance it through further design and development.
The paper is structured as follows. First the IS design science literature is reviewed. The gap in the
literature is explained, followed by our proposal to overcome the gap/limitations. We describe how the
proposed design approach is implemented, evaluated and communicated. Finally, a summary and
implications are provided in the conclusion.
2. Review of IS design science literature
According to Hevner et al.’s (2004) article, “design science creates and evaluates IT artefacts
intended to solve identified organizational problems”. The study by Gregor and Jones describes
design science as a sub-strand of a collection of constructive research approaches with a common
emphasis of the central role of the artefact (2007). Peffers et al. (2008) prescribe six processes for
design science: identify problem, define objectives of a solution, design and development,
demonstration, evaluation, and communication. A case for leveraging design theory to improve the
transparency and rigor of design research is demonstrated by Piirainen and Briggs (2011) who
integrate the framework in Hevner et al. (2004) and Peffers et al. (2008) as well as the design theory
in Walls et al. (1992) with that offered in Gregor and Jones (2007). Patas and Goeken’s article
suggests interplay between behavioural and design-oriented research can be improved and draws a
distinction between empirical and theoretical knowledge as well as non-artefact-centric and artefactcentric
knowledge (2011).
A review of the IS design literature provides guidance on how to organize IS design research. We
found that literature was available on design science research steps (Carlsson 2006; Hevner and
Chatterjee 2010; Hevner et al. 2004; March and Smith 1995; Offermann et al. 2009; Peffers et al.
2008; Piirainen et al. 2010), problem identification (Wieringa 2010), objectives of a solution (Carlsson
2007), evaluation (Cleven et al. 2009; Pries-Heje et al. 2008), project management (vom Brocke and
Lippe 2010) as well as evaluations of studies using design science (Samuel-Ojo et al. 2010). Design
science studies providing the actual specifics of the design step are summarized in Table 1 which
123

Francis Gacenga et al.
extends Peffers et al. (2008) and Offermann et al. (2009). In a number of design science studies the
design and build steps are not clearly demarcated.
Table 1: Design step in the is design science literature
Author(s) Design & Development Focus
Cole, Purao, Rossi & Sein (2005) Build (model, instantiate)
Hevner, March, Park & Ram (2004) Iterative search process, artefact
March & Smith (1995) Build
Nunamaker, Chen & Purdin (1990-1) Understand the studied domain, application of
relevant scientific and technical knowledge,
creation of alternatives, and synthesis and
evaluation of proposed alternative solutions
Takeda, Veerkamp, Tomiyama & Yoshikawa (1990) Suggestion, development
Vaishnavi & Keuchler (2008; Vaishnavi and Kuechler
Suggestion, development
2009);
Walls, Widmeyer & El Sawy (1992) Design method, meta design
Offermann, Levina, Schonherr & Bub (2009) Design artefact, literature research
Peffers, Tuunane, Rothenberger, & Chatterjee (2008) Design and development
Sein, Henfridsson, Purao, Rossi & Lindgren (2011) Building, intervention and evaluation
We considered the approaches summarized in Table 1 for the design step of our project but each
lacks detailed guidance. The build step offered in Cole et al. (2005) includes the steps “model and
instantiate” and advocates using software engineering principles such as effective tools and reuse.
Advice on the design step offered in Hevner et al. proposes an iterative search process that would
result in the artefact. The article concedes that “given the wicked nature of many IS design problems it
may not be possible to determine the relevant means, ends or laws” and suggest that a way out, “is to
search for satisfactory solutions” (2004).
Emphasis on building the artefact is highlighted in March and Smith, though the article does not
provide details on what “build” entails. The article states that “build refers to the construction of the
artefact, demonstrating that such an artefact can be constructed” (1995).
Advice on systems design in Nunamaker and Chen (1990) states that “design involves the
understanding of the studied domain, the application of relevant scientific and technical knowledge,
the creation of various alternatives, and the synthesis and evaluation of proposed alternative
solutions”. They did not give details on how to apply relevant scientific and technical knowledge in
creating the alternatives or how to synthesize them.
The process model in Peffers et al. (2008) is synthesized from process elements found in seven
studies taken from IS and other disciplines. In the model, design and development “includes
determining the artefact’s desired functionality and its architecture and then creating the actual
artefact” (Peffers et al. 2008). The article further suggests that moving from objectives to design and
development requires “knowledge of theory that can be brought to bear in a solution” but does not
explain how this can be done.
A computable design process model for CAD systems in Takeda et al. (1990) presents a descriptive
model, a cognitive model and a computable model. The cognitive model proposes a five-step cycle
that includes “suggestion” and “development”. Suggestion entails generating “key concepts that solve
the problem” and development “to construct candidates for the problem from the key concepts using
various types of design knowledge” (Takeda et al. 1990). The article further identifies object and
action levels as distinct levels in the designer’s mental activity. The approach was adapted in
Vaishnavi and Keuchler (2008) and we also find it useful as a conceptual guide though it does not
address the practical design issues we faced.
An elaboration on IS design theory such as the systems development lifecycle (SDLC) with emphasis
on building and testing IS theories is provided by Walls et al. (1992). Their design science research
framework proposes design method and meta design in the design stage. However, it lacks detail on
how to conduct these steps.
124

Francis Gacenga et al.
Based on the work in Peffers et al. (2008) a three-process framework entailing problem identification,
solution design and evaluation is proposed in Offermann et al. (2009). The article considers Matching
Analysis Projection Synthesis (MAPS) but concludes that “for IS design science, it seems that
methodology hasn’t advanced as far as to propose different methods for each process step” (2009).
The article then proposes, in its IS design research framework, a design step split into artefact design
and literature search. Details are not provided on how to perform the artefact design.
Our work, like that in Peffers et al. (2008) and Offerman et al. (2009) considers literature from IS
design science as well as outside IS design. The approaches in Peffers et al. and Offerman et al.
develop frameworks that guide the IS design research process and do not delve into the details of the
design process. The action design research method proposed in Sein et al. (2011) based on action
research advocates evaluation performed “in authentic settings” using the design steps of building,
intervention and evaluation. The article describes building the artifact and does not distinguish this
from design.
To summarize, IS design science literature delivers a rigorous and systematic process on the general
IS design research approach but offers little guidance on the design and development step at the
heart of creation of the artefact. Design is a creative endeavour and may not be solely process driven.
Few articles in IS design science describe the design step in detail or give much guidance on design
per se perhaps because design needs much 'right brain' as well as ‘left brain’ activity (Tovey 1984)
and tends to be context specific. However there are design principles that have been espoused over
the years.
3. Design gap in IS design science
Design is defined as “a goal-directed thinking process by which problems are analysed, objectives are
defined and adjusted, proposals for solutions analysed, objectives are developed and the quality of
those solutions is assessed” (Roozenburg and Eekels 1995). The gap in IS design science is
identified by Offermann et al. who state, “artefact design is a creative engineering process. Not much
guidance is provided in IS literature” (2009). The gap is described as the “lack of design foundations
in the axiomatic statements of the formal sciences or the empirical approaches of the natural sciences
nor the hermeneutic techniques of the humanities” (Jonas 2007). The subject and object of design
science is design and using the definition of Roozenburg and Eekels, design is “to conceive the idea
for some artefact or system and/or to express the idea in an embodiable form” (1995).
Design science lays emphasis on systematic, testable and communicable methods. However it is
instructive to reflect on the differences between design science and the science of design and
engineering as highlighted in Cross (2002). In our reflections we grappled with the question of
whether design needs to be scientific and whether design science advocates that design is scientific.
These questions are captured by Cross (2002) who observed "a desire to 'scientise' design can be
traced back to the 20th-Century Modern Movement in design”. There were aspirations to produce
works of art and design based on scientific methods of objectivity and rationality. These aspirations to
scientise design surfaced strongly again in the ‘design methods movement’ of the 1960s.
We heed the advice in Osterle et al. on the design step, that “artefacts should be created through
generally accepted methods, be justified as much as possible and be contrasted with solutions
already known in science and business” (2010).
4. Our approach to the design problem
Designer-researchers are viewed as a possible solution to the design problem (Cross 2002). Designer
researchers combine scientific methods with “designerly ways of knowing” (Cross 1982). The article
identifies five aspects of designerly ways of knowing: “Designers tackle ill-defined problems. Their
mode of problem-solving is solution-focused. Their mode of thinking is ‘constructive’. They use ‘codes’
that translate abstract requirements into concrete objects. They use these codes to ‘read’ and ‘write’
in ‘object languages’” (Cross 1982).
We reviewed literature on the design step outside IS design science (Archer 1984; Chow and Jonas
2008; Cross 2002; Eekels and Roozenburg 1991; Roozenburg and Eekels 1995) and identified
detailed design processes that would be useful for IS design science. There is a large body of design
literature and we refer to the works by Bayazit (2004) and Cross (1993) for a summary review of forty
125

Francis Gacenga et al.
years of design research which describe research from the early roots: De Stijil, the Bauhaus; to first
generation design methods: Horst Rittel, Morris Asimov, L. Bruce Archer, Christopher Alexander on
patterns, and Herbert A Simon; to second generation design methods; and finally to scientific
research in design.
According to the research in Chow and Jonas (2008), after the first generation methods were
rejected, the postmodernist attitude of no methods was followed by strong adoption of scientific
methods for design research. The article argues that the current situation regarding methodology in
design research is characterized by unproductive dualisms and proposes a generic process model,
Matching Analysis Projection Synthesis (MAPS). We also considered an alternative design cycle and
process proposed in Archer (1984) and a similar one in Roozenburg and Eekels (1995) which entails
analysis, synthesis, simulation, and evaluation conducted in iterations of specification, design,
properties and comparison. We opted to adopt MAPS as the design guide for the design step in our
project as MAPS is similar to the concepts of ‘the true’, ‘the ideal’ and ‘the real’ matching realism
which is the underlying philosophy of the project. MAPS also presents an integrative design research
medium as well as flexibility.
5. Designing a performance measurement framework for ITSM
In Table 2 we outline how we conducted the ITSM PMF project by applying behavioural science
methods with an IS design science framework that integrates ‘designerly ways of knowing’ in the
design step. This fusion of science and design is achieved in three stages and we outline the
objective, method, paradigm, parent discipline, philosophy, data analysis, purpose, study phenomena,
outcomes and contributions.
Table 2: Fusion of science and design
Research/
Stage 1 Stage 2 Stage 3
Design
Objective Problem identification
and definition of
objectives of a solution
Understand the domain of
ITSM performance
measurement
Evaluation of prototype PMF
Determine ITSM benefits
and performance metrics
Design ITSM PMF
Method Systematic literature
review
Case studies, cross case analysis
ISDR (Peffers et al. 2008) and MAPS synthesis and
Survey
realization (Chow and Jonas 2008; Jonas 2007)
Paradigm Behavioural science Behavioural science Design science
Parent Social science Social science IS strand of the Sciences of
Discipline
the Artificial (Gregor and
Jones 2007)
Philosophy Positivist Positivist/realist Realist
Data
Analysis
Quantitative & qualitative Qualitative & quantitative Expert evaluation
Purpose Exploratory Explanatory Application
Phenomena Organization Organization Problem solving artefact
Outcome Theory building: Theory building: explaining Design and action
analysing and describing and predicting
Contribution Empirical ITSM metrics A contingency theory of ITSM performance
and benefits
ITSM performance
measurement
measurement framework
The macro and micro cycles of MAPS are depicted in Figure 2 and Figure 3 respectively. In the macro
cycle, analysis refers to ‘the true’ how it is today; projection is ‘the ideal’ how it should be; and
synthesis is ‘the real’ how it is tomorrow (Jonas 2007).
126

Francis Gacenga et al.
Figure 2: Analysis - projection - synthesis: the macro cycle of the design process (Source: Jonas
2007)
Figure 3: Research-analysis-synthesis-realization: the micro cycle of the design process (Source:
Jonas 2007)
Jonas (2007) advises that a hyper cyclic generic design process model results from combining the
domains of knowing in the macro model (analysis, projection, synthesis) with the learning phases in
the micro model (research, analysis, synthesis, realisation) as depicted in Figure 4.
Figure 4: Generic, hyper cyclic model of the design process (Source: Jonas 2007)
5.1 Macro cycle of design process
The first step in the macro cycle of the design process, analysis, is undertaken using the findings of
the initial stages of our project and provides us with information of ‘the true’, that is, how ITSM
performance measurement is currently conducted in industry. Integrating the findings of the literature
review, survey and case study we gained empirical knowledge as well as in-depth qualitative
knowledge of the state of the art in ITSM performance measurement.
The second step in the macro cycle, projection, deals with ‘the ideal’ and through the literature review
and content analysis of the case studies we get an understanding of how performance measurement
could be in the ideal. We generate an initial model of the ITSM PMF as it should be, depicted in
Figure 5.
127

Francis Gacenga et al.
The model proposed in Figure 5 can be used by organizations to measure the performance of their
ITSM as well as evaluate the metrics they are currently using for completeness in service orientation,
financial and non-financial perspectives in broad economic terms. The metrics within each perspective
are categorized into service, function, process and technology dimensions. These represent ITSM
function, ITSM process and ITSM technology metrics while the service dimension includes metrics to
account for end-to-end process outcomes. At the industry level, wide adoption of the performance
measurement model would lead to standardization of ITSM performance measurement and enhance
the ability of organizations implementing ITSM to benchmark against other organizations using like
terms.
Figure 5: Model to measure the performance of ITSM
Synthesis, the third step in the macro cycle of the design process is achieved in our project by
working with a panel of industry and academic experts to identify how to contextualize the ITSM PMF.
In this step the framework is compared against the performance measurement practices currently in
place in organizations. The panel of experts contributes to the development of a method to evaluate
the ITSM PMF.
5.2 Micro cycle of design process
In the micro cycle of our project, research involves the data collection through literature review, survey
questionnaires and case studies. Analysis entails using the findings of the qualitative and quantitative
data analysis from the project stage two. Synthesis involves employing the five aspects of ‘designerly
ways of knowing’ described in Section 4:
1. The problem was ill-defined as a PMF had not been previously developed and the dimensions of
the framework were unknown;
2. The focus was to develop a solution to the problem: ITSM practitioners experience challenges in
measuring and reporting the performance of ITSM;
3. A constructive mode of thinking was adopted to develop, populate and implement the PMF;
4. Figure 5 is an example of the visual representation of codes used to translate the abstract
requirements of dimensions to arrange metrics for the metrics catalogue;
5. This model informed the database design (‘object language’) of the repository to store the elements
of the PMF including metrics, benefits, processes, BSC perspective, challenges, and business sector.
Realization includes the evaluation and communication of the prototype.
128

Francis Gacenga et al.
For each phase of the macro cycle (analysis, projection and synthesis), four steps in the micro cycle
were undertaken (research, analysis, synthesis and realization). Table 3 presents a summary showing
the specific design elements for the macro and micro design cycles. We developed a prototype and
stopped at proof of concept with further development possible in future projects. Several iterations on
the design prototype were undertaken.
An ITSM metrics catalogue was developed to populate the ITSM PMF. Proof of concept involved
demonstration of the use of the ITSM PMF. The demonstration aimed to solve the problem of
measuring the performance of ITSM in the case study organizations. This involved using the ITSM
PMF as a guide to selecting relevant and contextualized ITSM performance metrics for the
organization and then generating meaningful ITSM performance reports for the business. The revised
design science method process model is shown in Figure 6.
Table 3: Summary of hyper cyclic model applied to the ITSM PMF project
Macro
Micro Cycle
Cycle Research Analysis Synthesis Realization
Analysis Literature review
findings
Projection Survey and case
study data
Synthesis Interactions with
panel of experts
and ITSM
practitioners
Systematic literature
review
Catalogue of
metrics
Quantitative analysis Cross case
and content analysis analysis
Content analysis Performance
measurement
framework
Communication: journal,
conference and seminar
papers
Demonstration of prototype
Tested prototype,
thesis
Figure 6: Revised design science method process model
6. Evaluation and communication
Evaluation and communication are important requirements in design science research (Cleven et al.
2009; Gregor and Jones 2007). In a detailed evaluation of our artefact we apply the guidance and
fitness-utility model for design science research proposed in Gill and Hevner (2011).
The artefact is evaluated for its usefulness and fitness, on its capability to assist the organization to
select contextualized metrics, generate relevant reports to the organization and provide an integrated
set of ITSM performance metrics. ITSM managers and a panel of experts provided feedback as input
to build the final prototype. The results of the literature review, survey, case study and design
research have been communicated through publications in industry and academic journals and
conferences.
129

7. Conclusions
Francis Gacenga et al.
In summary, the review of the IS design science literature revealed a scarcity of guidance on the
design of the artefact, despite a substantial research effort on other aspects of the field, such as the
general steps in design science, problem identification, objectives of solutions, evaluation of the
artefacts and project management. Our project aimed to develop a framework artefact that can be
used for the performance measurement of ITSM investments in organizations. The project uses the
Information Systems Design Research (ISDR) approach (Peffers et al. 2008). The objective of this
paper has been achieved: the silence on design in IS design science has been highlighted and a
design approach has been proposed and trialled in our project.
We approached the fusion of science and design following the advice from the literature. After
exploring the design gap in IS design science, we identified the MAPS approach from a discipline
outside of IS and used it in our project. Using a design process from the design literature advocating
‘designerly ways of knowing’ we applied the integrative MAPS approach (Chow and Jonas 2008). We
integrated the IS design science research approach with the MAPS model. Based on MAPS, we
applied the macro cycle of analysis-projection-synthesis and the micro cycle of research-analysissynthesis-realization
to create an approach to design the prototype of the ITSM PMF.
We recognize as a limitation to this work that there may be other design approaches from other
disciplines, however, the design process proposed here may be useful for other IS researchers faced
with the dilemma of how to design the artefact in an IS design science project. This multi-discipline
approach improved the rigor of our ITSM project by using a systematic and repeatable approach to
guide the design step in developing the framework. This paper contributes to the body of knowledge
on design science by proposing a design approach to fill the current gap.
Note: ITIL® is a registered trademark of the UK, Office of Government Commerce.
References
Archer, L. B. (1984). "Systematic method for designers", in N. Cross, (ed.), Developments in design
methodology. Chicester: Wiley, pp. 57-82.
Bayazit, N. (2004). "Investigating Design: A Review of Forty Years of Design Research." Design Issues, 20(1),
16-29.
Carlsson, S. A. (2006). "Towards an Information Systems Design Research Framework: A Critical Realist
Perspective"1st International Conference on Design Science in Information Systems and Technology:
Claremont, pp. 192-212.
Carlsson, S. A. (2007). "Developing knowledge through IS design science research: For whom, what type of
knowledge, and how." Scandinavian Journal of Information Systems, 19(2), 75-85.
Chow, R., and Jonas, W. (2008). "Beyond Dualisms in Methodology: An Integrative Design Research Medium
"MAPS" and some Reflections."Undisciplined! Design Research Society Conference: Sheffield, UK, pp. 1-
18.
Cleven, A., Gubler, P., and Huner, K. M. (2009). "Design alternatives for the evaluation of design science
research artifacts"4th International Conference on Design Science Research in Information Systems and
Technology ACM: Philadelphia, Pennsylvania.
Cole, R., Purao, S., Rossi, M., and Sein, M. (2005). "Being Proactive: Where Action Research Meets Design
Research"International Conference on Information Systems (ICIS): Las Vegas, Nevada, USA.
Cross, N. (1982). "Designerly ways of knowing." Design Studies, 3(4), 221-227.
Cross, N. (1993). "Science and design methodology: A review." Research in Engineering Design, 5(2), 63-69.
Cross, N. 2002. Design as a Discipline, the Inter-disciplinary Design Quandary Conference [Online]. Milton
Keynes: De Montfort University. Available: http://nelly.dmu.ac.uk/4dd//DDR3-Cross.html [Accessed 10 June
2011].
Eekels, J., and Roozenburg, N. F. M. (1991). "A methodological comparison of the structures of scientific
research and engineering design: their similarities and differences." Design Studies, 12(4), 197-203.
Gill, T., and Hevner, A. (2011). "A Fitness-Utility Model for Design Science Research", in H. Jain, A. Sinha, and
P. Vitharana, (eds.), Service-Oriented Perspectives in Design Science Research. Springer
Berlin/Heidelberg, pp. 237-252.
Gregor, S. (2002). "Design Theory in Information Systems." Australian Journal of Information Systems(Special
Issue 2002), 14-22.
Gregor, S., and Jones, D. (2007). "The Anatomy of a Design Theory." Journal of the Association for Information
Systems, 8(5).
Hevner, A. R., and Chatterjee, S. (2010). Design research in information systems: theory and practice, New York:
Springer.
130

Francis Gacenga et al.
Hevner, A. R., March, S. T., Park, J., and Ram, S. (2004). "Design Science in Information Systems Research."
Management Information Systems Quarterly, 28(1), 75-106.
Jonas, W. (2007). "Research through DESIGN through research: A cybernetic model of designing design
foundations." Kybernetes, 36(9/10), 1362 - 1380.
Lahtela, A., Jäntti, M., and Kaukola, J. (2010). "Implementing an ITIL-Based IT Service Management
Measurement System", J. Marko and K. Jukka, (eds.), Fourth International Conference on Digital Society:
St. Maarten, Netherlands Antilles, pp. 249-254.
March, S. T., and Smith, G. F. (1995). "Design and natural science research on information technology." Decision
Support Systems, 15(4), 251-266.
Nunamaker, J. F., Jr., and Chen, M. (1990). "Systems development in information systems research"23rd Hawaii
International Conference on System Sciences, pp. 631-640 vol.3.
Nunamaker, J. F. J., Chen, M., and Purdin, T. D. M. (1990-1). "Systems Development in Information Systems
Research." Journal of Management Information Systems, 7(3), 89-106.
Offermann, P., Levina, O., Schonherr, M., and Bub, U. (2009). "Outline of a design science research
process"Proceedings of the 4th International Conference on Design Science Research in Information
Systems and Technology ACM: Philadelphia, Pennsylvania, pp. 1-11.
Osterle, H., Becker, J., Frank, U., Hess, T., Karagiannis, D., Krcmar, H., Loos, P., Mertens, P., Oberweis, A., and
Sinz, E. J. (2010). "Memorandum on design-oriented information systems research." European Journal of
Information Systems.
Patas, J., Milicevic, D., and Goeken, M. (2011). "Enhancing Design Science through Empirical Knowledge:
Framework and Application", in H. Jain, A. Sinha, and P. Vitharana, (eds.), Service-Oriented Perspectives in
Design Science Research. Springer Berlin / Heidelberg, pp. 32-46.
Peffers, K., Tuunanen, T., Rothenberger, M. A., and Chatterjee, S. (2008). "A Design Science Research
Methodology for Information Systems Research." Journal of Management Information Systems, 24(3), 34.
Piirainen, K., Gonzalez, R., and Kolfschoten, G. (2010). "Quo Vadis, Design Science? – A Survey of Literature",
Global Perspectives on Design Science Research. pp. 93-108.
Piirainen, K. A., and Briggs, R. O. (2011). "Design Theory in Practice – Making Design Science Research More
Transparent ", Service-Oriented Perspectives in Design Science Research. Springer, pp. 47-61.
Pries-Heje, J., Baskerville, R., and Venable, J. (2008). "Strategies for Design Science Research Evalutaion"16th
European Conference on Information Systems (ECIS): Galway, Ireland, pp. 255-266.
Roozenburg, N. F. M., and Eekels, J. (1995). Product Design: Fundamentals and Methods, Chichester; New
York: Wiley.
Samuel-Ojo, O., Shimabukuro, D., Chatterjee, S., Muthui, M., Babineau, T., Prasertsilp, P., Ewais, S., and
Young, M. (2010). "Meta-analysis of Design Science Research within the IS Community: Trends, Patterns,
and Outcomes", R. Winter, J. L. Zhao, and S. Aier, (eds.), Global Perspectives on Design Science
Research, DESRIST LNCS: St Gallen, pp. 124-138.
Sein, M. K., Henfridsson, O., Purao, S., Rossi, M., and Lindgren, R. (2011). "Action Design Research." MIS
Quarterly, 35(1), 37-227.
Takeda, H., Veerkamp, P., Tomiyama, T., and Yoshikawa, H. (1990). "Modeling design processes." AI Magazine,
11(4), 37-48.
Tovey, M. (1984). "Designing with both halves of the brain." Design Studies, 5(4), 219-228.
Vaishnavi, V., and Kuechler, W. (2008). "Design Research in Information Systems", in V. Vaishnavi and W.
Kuechler, (eds.), Order: a Journal on the Theory of Ordered Sets and its Applications. Auerbach, pp. 1-393.
Vaishnavi, V., and Kuechler, W. 2009. Design Research in Information Systems [Online]. Available:
http://home.aisnet.org/displaycommon.cfm?an=1&subarticlenbr=279 [Accessed 3 March 2012].
vom Brocke, J., and Lippe, S. (2010). "Taking a Project Management Perspective on Design Science Research",
Global Perspectives on Design Science Research. pp. 31-44.
Walls, J. G., Widmeyer, G. R., and El Sawy, O. A. (1992). "Building an Information System Design Theory for
Vigilant EIS." Information Systems Research, 3(1), 36-59.
Wieringa, R. (2010). "Relevance and Problem Choice in Design Science", Global Perspectives on Design
Science Research. pp. 61-76.
131

Measuring the Students’ Satisfaction With the Romanian
Higher Education Environment: The Case of Blended
Learning System at the University of Oradea
Adriana Giurgiu 1 , Carmen-Alina Popa 2 and Adrian Negrea 1
1 The Department for Distance Learning and Reduced Frequency Education,
Faculty of Economic Sciences, Department of International Business,
University of Oradea, Oradea, Romania
2 The Department for Distance Learning and Reduced Frequency Education;
Faculty of Social-Humanistic Sciences, Department of Education Sciences,
University of Oradea, Oradea, Romania
adrianagiurgiu@gmail.com
carmen_berce@yahoo.com
ngr_adrian@yahoo.com
Abstract: The Romanian Agency for Quality Assurance in Higher Education (ARACIS) has begun its activity in
2005, as an autonomous public institution of national interest. Since then, it has developed many quality
standards for the Romanian Higher Education (RHE) system. The ARACIS’ strategy relies on its assumed
mission to constantly assure and improve quality in the Romanian universities, by following its declared
objectives, such as: improving the external evaluation methodology, in full compliance with the European
Standards and Guidelines for Quality Assurance (ESG); increasing the role of students and employers, as final
beneficiaries, within the process of evaluation and assurance of education quality; creating a quality culture in the
RHE; establishing a permanent partnership with all institutions in the RHE system, as well as with the economicsocial
environment, in order to correlate the RHE with the labour market. ARACIS is carrying out its activity
according to the best international practices, focusing towards quality assurance and evaluation of the RHE, as
part of the European Higher Education Area. ARACIS pursues the external evaluation of the RHE institutions
based on its methodology, concluded in six “Quality evaluation activities guide for university study programs and
for higher education institutions”. From this perspective, the paper presents the way in which the University of
Oradea has adopted and implemented the Part V and VI of the ARACIS “External evaluation methodology,
standards, reference standards and list of performance indicators” into its BL (DLS and RFE) study programs,
focusing on describing the internal procedures and research methods used to measure the students’ satisfaction
with these programs, as well as the way in which these internal procedures have been implemented within the
University of Oradea in the last couple of years and which are the results so far. For this purpose, over 2800
questionnaires applied to all BL (DLS and RFE) students of the University of Oradea in these years will be
described and interpreted, and their results will be used to formulate the conclusions of the paper, for assessing
and evaluating the effectiveness of the students’ experience and satisfaction in such of an environment.
Keywords: research methods, students’ satisfaction, quality management, Romanian Higher Education, blended
learning (BL)
1. Introduction to the RHE quality management system
The Romanian Agency for Quality Assurance in Higher Education (ARACIS) has begun its activity in
2005, as an autonomous public institution of national interest. Since then, it has developed many
quality standards for the Romanian Higher Education (RHE) system, while evaluating all the
Romanian universities on a regular basis, and contributing to the classification of the domestic
universities in relation to their quality in education, research, institutional management, students’
employability, students’ and employers’ satisfaction, internationalisation etc. Since September 2009,
ARACIS is a full member of the European Association for Quality Assurance (ENQA) and is registered
in the European Quality Assurance Register-EQAR. The strategy of the ARACIS relies on its
assumed mission to constantly assure and improve the quality of education in the Romanian
universities, by following and implementing its declared objectives, as follows:
Improving the external evaluation methodology, in full compliance with the European Standards
and Guidelines for Quality Assurance (ESG);
Increasing the role of students and employers, as final beneficiaries, within the process of
evaluation and assurance of education quality;
Creating a quality culture in the RHE;
132

Adriana Giurgiu, Carmen-Alina Popa and Adrian Negrea
Establishing a permanent partnership with all institutions in the RHE system (Ministry of
Education-MECTS, National Council for Scientific Research - CNCS, National Council for Higher
Education Funding - CNFIS, ACPART etc.) as well as with the social and economic environment,
in order to correlate the RHE with the labour market (ARACIS 2006).
ARACIS is carrying out its activity according to the best international practices, which are
implemented its own evaluating methodology, focusing towards quality assurance and evaluation of
the Romanian Higher Education (RHE) system, as part of the European Higher Education Area.
ARACIS pursues the external evaluation of the RHE institutions based on its methodology, concluded
in six “Quality evaluation activities guide for university study programs and for higher education
institutions”, as it follows:
Part I. Study programmes (Bachelor’/Master’s degrees) accreditation external evaluation, Part II.
Institutional authorisation/accreditation external evaluation,
Part III. Academic quality external evaluation of accredited higher education institution,
Part IV. External evaluation of Teaching staff training departments (TSTD),
Part V. External evaluation of the study programs offered through distanceLearning system
(DLS),
Part VI. External evaluation of the study programs offered through Reduced Frequency Education
(RFE).
From this perspective, our paper presents the way in which the University of Oradea has adopted and
implemented the Part V and the Part VI of the ARACIS “External evaluation methodology, standards,
reference standards and list of performance indicators” into its BL (DLS and RFE) study programs,
focusing on describing the internal procedures adopted by the University of Oradea during the last
years, and the research methods used to measure the students’ satisfaction with these BL
programmes, as well as the way in which these internal procedures have been implemented within
the University of Oradea and their registered results so far.
2. The ARACIS external quality management evaluation methodology of the
RHE
2.1 General aspects
As stated by ARACIS (2006), “the mission of the Quality Assurance Agency for Higher Education
(ARACIS) consists of achieving public interest for high quality standards in developing study
programmes and in obtaining qualifications in higher education (HE), in supporting the continuous
improvement of HE quality management”. In this respect, ARACIS has issued the necessary
methodology and procedures of carrying out the external evaluation of university study programmes
in order to accreditation the academic degree cycles, defined in accordance with Bologna Process.
ARACIS has also provided the interested part with an ‘External Evaluation Guide’, which represents in
fact, a work instrument of the higher education institutions, as well as of the external evaluators of the
academic quality, providing specific indicators and practices meant to ensure the quality of every
study programme of the RHE. According to the ARACIS ‘External evaluation methodology’, the
accreditation of an institution of the RHE supposes two successive stages:
The study programmes accreditation is carried out at higher education institutions temporarily
authorised or accredited, for study programmes newly proposed for the 1 st cycle or for the
accreditation of 1 st cycle study programmes;
The temporary authorisation is carried out for every study programme within the 1 st cycleuniversity
studies leading to a distinct university qualification since October 13 th , 2006.
The specific objectives of the ARACIS accreditation process are as follows (ARACIS 2006):
To ensure the university communities, the beneficiaries and the large public that the evaluated
organisation, accredited or authorised to organise an academic study programme, complies with
the minimum quality standards expected from a higher education institution;
To determine the evaluated organisation to regularly self-evaluate and to cooperate in its external
evaluation in order to ensure and increase its quality;
133

Adriana Giurgiu, Carmen-Alina Popa and Adrian Negrea
To determine the evaluated organisation to cooperate with other higher education institutions in
order to achieve, monitor and compare its academic quality;
To identify and publish any functioning attempt of a programme that does not comply with the
minimum standards of academic quality.
The academic evaluation of a study programme as pursued by ARACIS is mainly based on the
following aspects and principles:
The maintenance and continuous improvement of the academic standards;
The flexibility in organising the external evaluation mission, so that the activity of the ARACIS’
expert-evaluators not to disturb the teaching process.
For each evaluation visit (mission), ARACIS appoints a group of expert-evaluators (commission of
expert-evaluators) led by a coordinator, aiming to ensure the quality of the external evaluation
process. Thereafter, ARACIS confirms the higher education institution that the self-evaluation
documents constitute the starting point to begin the external evaluation.
The commission of expert-evaluators has to use the existing information and the documentary proof
of the evaluated study programmes provider. Thus, the internal documents have to be examined by
the evaluators appointed by the ARACIS, as self-evaluation documents, in order to ensure the
transparency of the evaluation process by means of applying the standards and reference standards
published by ARACIS;
The commission of expert-evaluators is monitored by the ARACIS Council throughout the whole
period of training, development and reference of its external evaluation mission.
2.2 Methodological stages for the accreditation of an academic degree study
programmes by ARACIS
The accreditation methodology of a study programme used by ARACIS implies some successive work
stages (ARACIS 2006). First of all, on the basis of the application to start the external evaluation
procedure for the temporary authorisation / accreditation, submitted to the accreditation department of
ARACIS by the education provider, the ARACIS Council decides the starting of the external
evaluation procedure if the following conditions are cumulatively fulfilled:
Together with the application, the higher education provider also submitted the Internal evaluation
report, drawn up in compliance with the provisions of ARACIS, using as benchmarks the standards
typical to the temporary authorisation/accreditation stage;
ARACIS appoints then the commission (group of external evaluators) of 3 experts, having speciality
training in the field of the study programme submitted to the evaluation, led by a member of the
Commission of permanent specialty experts, as part of the team. The results of the verifications are
registered by the members of the Commission of external evaluators in the "Visit record for temporary
authorisation/programme accreditation", which is signed by all the commission members. The "Visit
record for temporary authorisation/programme accreditation" is discussed and approved in the
Commission of permanent specialty experts, on domains. The members of the Commission of
permanent specialty experts draw up the "Temporary authorisation evaluation report" or, if the case,
the "Accreditation evaluation report", where they propose, under signature, the temporary
authorisation or accreditation of the study programme or, if the case, its non-authorisation/nonaccreditation;
The ‘Temporary authorisation/accreditation evaluation report’ is submitted to the Accreditation
Department of the ARACIS, which validates the report by verifying the way the evaluation
methodology was applied, draws up the "Validation report of the Accreditation Department" and
submits it to be discussed and validated in the ARACIS Council, which, on the basis of the
conclusions drawn from the documents' analysis and the debates, draws up the Council's Report
Concerning the Programme Authorisation/Accreditation and submits it to the Romania’s Ministry of
Education with the proposal of temporary authorisation/accreditation or, if the case, of nonauthorisation/non-accreditation
of the study programme. Thereafter, on a regular basis, ARACIS
annually monitors the academic degree study programmes authorised for the temporary functioning.
134

Adriana Giurgiu, Carmen-Alina Popa and Adrian Negrea
3. Literature review on learning and eLearning theories
The purpose of this section is to analyze the problems related to implementing the BL in highereducation,
according to the previous work of the researchers with theoretical study of learning
theories which will permit to understand the learning effort for the teacher and the students, the
effectiveness of each method, the students’ workload or collaborative and social aspects of eLearning
.
3.1 Behaviourism learning theories and eLearning
Behaviourists notice the mind as a “black box”, in the sense that a response to a stimulus can be
observed quantitatively, totally ignoring the effect of thought processes happening in mind, and
highlight four aspects relevant for realizing eLearning courses with respect to the behaviourist in
school:
The learning material should be split into small instructional steps being presented in a deductive
way by means of starting with a rule, category, principle, formula or definition, giving positive
examples to reinforce understanding, and showing negative examples to establish conceptual
boundaries;
Course should be defined sequences of instructions using conditional or unconditional branching
to other instructional units and pre-determining choices within the course.
To maximize learning efficiency, learners may be routed to miss or repeat certain sections based
on the performance on diagnostic tests, or on tests within the sequence of learning activities.
Instructional design may also allow a learner to choose the next instruction out of a set of
activities, giving the learner more control over the learning process.
It suggests to display the required operation, procedure or skill, and to break it down into its parts
with appropriate explanation before learners are expected to copy the desired behaviour. It should
allow building proficiency of learners from frequent review or revision with check tests at strategic
points or repeat practice with feedback.
Instructional design emphasizes low error rate and the usage of remedial loops back through
material if necessary.
This approach suggests a structured, deductive approach to design eLearning course, so that basic
concepts, skills, and factual information can rapidly be acquired by the learners.
Further implications of the eLearning can be emphasised by the concept of drill and practice,
portioning materials and assessing learner’s achievement levels, and giving external feedback.
3.2 Cognitive learning theories and eLearning
Cognitive theories provide a frame of reference by which educators and educational technologies can
analyze the behaviour of the learner and design educational environments where the learners can
construct their own knowledge. Good interactive online instructions facilitate knowledge construction
because the learners have to take initiatives to learn and to interact with other students and instructor
(Murphy, Cifuentes 2001). Furthermore, learners experience the information first-hand, which gives
them the opportunity to contextualize and personalize the information themselves. According to
Heinich et al. (2002), learning is the development of new knowledge, skills, and attitudes as the
learner interacts with information and the environment.
In other words, cognitivists see learning as an internal process that involves memory, thinking,
reflection, abstraction, motivation, and meta-cognition. Cognitive psychology considers the learning
from an information processing point of view, where the learner uses different types of memory during
learning. Sensations are received through the senses into the sensory store before processing
occurs. The information persists in the sensory store for less than one second (Kalat 2007), and if it is
not transferred to working memory immediately, it is lost.
Following important aspects should be considered for higher education system by taking into account
the cognitive theories
The learning process should be enhanced by facilitating all sensors, and matching the cognitive
level of the learner.
135

Adriana Giurgiu, Carmen-Alina Popa and Adrian Negrea
Students need to be motivated to learn by means of learning strategies.
The teaching approach should enforce learners to use their meta-cognitive skills by reflecting on
what they learn, collaborating with other learners or checking their progress (Meyer 1998).
The learning content should contain different real-life situations, so that the learners can tie up to
own experiences and, therefore, memorize things better.
3.3 Constructivism and eLearning
Constructivism means generating new knowledge based on prior experience. The key idea is that
students actively construct their own knowledge, learning being seen as an active mental work, not as
a passive reception of teaching (Woolfolk 1993).
A summary of characteristics of constructivism characteristics are as follows:
Several perspectives and representations of concepts and content are presented and
encouraged.
Goals and objectives are derived by the student or in negotiation with the teacher or system.
Teachers serve in the role of guides, monitors, coaches, tutors and facilitators.
Activities, opportunities, tools and environments are provided to encourage metacognition,
Self-analysis -regulation, reflection & awareness.
The student plays a central role in mediating and controlling learning.
Learning situations, environments, skills, content and tasks are relevant, realistic, and authentic
and represent the natural complexities of the 'real world'.
Primary sources of data are used in order to ensure authenticity and real-world complexity.
Knowledge construction and not reproduction is emphasized.
This construction takes place in individual contexts and through social negotiation, collaboration
and experience.
The learner's previous knowledge constructions, beliefs and attitudes are considered in the
knowledge construction process.
Higher-order thinking, Problem-solving, skills and deep understanding are emphasized.
Errors provide the opportunity for insight into students’ previous knowledge constructions.
Exploration is a favoured approach in order to encourage students to seek knowledge
independently and to manage the pursuit of their goals.
Learners are provided with the opportunity for apprenticeship learning in which there is an
increasing complexity of tasks, skills and knowledge acquisition.
Knowledge complexity is reflected in an emphasis on intangible interrelatedness and
interdisciplinary learning.
Collaborative and cooperative learning are favoured in order to expose the learner to alternative
viewpoints.
Scaffolding is facilitated to help students perform just beyond the limits of their ability.
Assessment is authentic and interwoven with teaching. (Koohang 2009).
Koohang (2009) advanced a model based on constructivism learning theory within the area of
eLearning. The model includes three categories of elements of the Constructivism, such as:
1. The Design of Learning Activities
2. Learning Assessment
3. Instructor’s Roles
136

Adriana Giurgiu, Carmen-Alina Popa and Adrian Negrea
The model is comprised of two categories: the learning design elements and learning assessment
elements. The learning design elements include fundamental design elements and collaborative
design elements. The fundamental design elements are essential for designing learning activities.
According to Koohang, the learners can be influenced in several ways through this model:
Learner-centred model for designing eLearning assignments/activities is to encourage learners to
actively construct new knowledge.
The learners are encouraged to develop their own goals and objectives in solving problems
Learners will begin to be responsible for their learning.
This also encourages interrelatedness and interdisciplinary learning.
Multiple perspectives are presented, covering various content, ideas, and concepts. Social
negotiation among learners is encouraged.
4. The implementation of the ARACIS quality management insurance
methodology at the University of Oradea: The BL bachelor programmes
case
The University of Oradea (UO) is a public accredited higher education institution, which celebrated
230 years of higher education in Oradea in 2010, and having as assumed mission to train specialists
in different areas of sciences and culture, to generate new knowledge, to lead to an economic and
social progress.
The fundamental strategic objectives of the University of Oradea as mentioned in its Strategic
Development Plan for the period 2008-2013 are as follows:
The quality insurance of the educational research and management activities for the recognition
of the UO as an institution of reference at the national level;
The increase of the efficiency and competitiveness of the learning processes;
The development of scientific research according to the strategy and priorities defined at national
and European level to improve the scientific and financial results acquired from this activity;
The increase of internal and European visibility of the UO throughout the enhancement of the
academic collaborations with the social-economic environment, and the compliance of the
educational offer with the market expectations;
The promotion of an efficient management system based on the principles of academic
independence in partnership with the beneficiaries of the educational services;
The insurance of the permanently improved life quality of the students.
In this respect, the UO acts on the 3 levels that define the quality of the educational services:
institutional capability, educational efficiency and quality management.
The System Regulation for the Evaluation and Quality Assurance establishes the quality management
system, including implementation structures: the Evaluation and Quality Assurance Commission,
Quality Council, Body of Internal Auditors (ICA) and the Quality Assurance Department. In February
2007 the ICA was established and its working procedure was elaborated. The Body of Internal
Auditors includes for the moment 38 teachers, and has internally evaluated so far 227 study
programmes (Bachelor and Master), 29 research centres, 122 laboratories and 113 teaching
curricula.
4.1 The development of the quality management system at the University of Oradea
Starting with the academic year 2004-2005, in accordance with the Romanian tendencies and with
the good European practice, within the University of Oradea was created the Department of Quality
Assurance with the mission to apply and to maintain the Management of the Quality System at the
University of Oradea.
The DQA has developed in the academic year 2005-2006 the internal quality rules, the section
"Quality" on the web site of the UO, and the Strategy of Quality for the academic year 2006-2007,
which was based on following three major aspects:
137

Adriana Giurgiu, Carmen-Alina Popa and Adrian Negrea
The necessity of quality evaluation and insurance are based on the 3 fundamental fields:
institutional capacity, educational efficacy and the management of quality;
The University's participation at the process of external evaluation of quality pursued by ARACIS;
The necessity of an internal evaluations of the University of Oradea, at both institutional and study
programmes’ levels, on the basis of the standards and performance indicators established by
ARACIS,
Those were included in The Program of Policies, Strategies and Quality Actions, also elaborated by
the DQA.
Furthermore, internal regulations, adjustments to the demands of the national legislation, and the
setting up of the Quality Evaluation Academic Commission (QEAC) were completed within the UO, so
that at present, besides the QEAC and DQA, the University has also a structure named The Quality
Council (QC) that includes the QEAC of the UO and the officials responsible with quality from the
faculties, departments and services, thus insuring the integrated function of the SEAQ.
In May 2007, the UO has enrolled itself in the External Evaluation of Quality process organised by
ARACIS, as an experimental national pilot program within the RHE system. The institutional
evaluation has been made by ARACIS, but unfortunately, the UO was qualified as an „limited trust"
University, mainly because of its imperfect management that was considered as abusive, lacked of
quality, and ineffective. Nonconformities with the performance indicators established by ARACIS were
identified to 9 study programmes out of the total 27 programmes which were evaluated at that time.
Thereafter, in order to correct the identified nonconformities, an Action Plan has been immediately
released by the Senate of the UO in September 2007, as an Operational Plan for the quality
improvement of administrative, educational and research processes at the UO. Most part of the Action
Plan was also assumed in the Operational Plan for 2008 and some others, in the Operational Plan for
2009.
In May 2008, September 2008, March 2009, and July 2010, reports regarding the state of fulfilling
with the assumed Action Plan were submitted to the ARACIS. In the report completed in July 2010,
already 2 years after piloting, was noticed the fact that over 96% of the assumed actions and
measures in the Action Plan have been accomplished.
Therefore, it was decided to ask ARACIS for another external evaluation process, which took place in
March 2011, and which concluded that the UO is for now a “trustfully” university. This is considered an
intermediary success of the UO, as its strategic objective to be recognize as an higher education
institution of reference at the national level, and therefore, it needs at least to be qualified by ARACIS
as an “high trustfully” university, which is the best possible quality level within the RHE system,
according to the ARACIS scale. So, one more step is needed in this direction, and the UO, through all
its structures (faculties and departments) is committed to proceed with fulfilling all the necessary
quality standards to be considered an “high trustfully” university. For this purpose, the QEAC updates
on yearly basis, the specific regulations of the SEAQ, and elaborates the Policies, Strategies and
Actions Program for Quality.
4.2 The quality management case of the BL bachelor programmes
The Department for Distance Learning and Reduced Frequency Education (DIDIFR) is a structure of
the UO which runs 7 Bachelor programmes in DL system (out of which, 6 are organized in the field of
economics, and 1 in the field of engineering), and 1 Bachelor programme in RFE, organized in the
field of educational sciences, totalizing over 1400 students enrolled in the academic year 2011-2012.
The 7 DL programmes have been accredited by ARACIS as “high trustfully” study programmes after
the external evaluation made in 2009, while the RFE programme is still only authorised to function by
ARACIS, having as target for the year 2012 to get the accreditation.
As a structure of the UO, the DIDIFR is committed to contribute to the improvement of the quality level
of the university, by taking actions meant to ensure and fulfil the standards established by ARACIS.
As a consequence, during the last couple of years, DIDIFR pursued an intensive activity towards
meeting the higher education quality requirements, by taking the following actions and measures:
The implementation of the evaluation of the students’ learning results;
138

Adriana Giurgiu, Carmen-Alina Popa and Adrian Negrea
The periodical evaluation of the teaching staff’s quality, based on the following three components:
self evaluation, student's evaluation and the department colleague's evaluation. The evaluation
and quality assurance of teachers and disciplines procedure has covered - over the years - more
choices, the traditional one being more simple and having only self-assessment component
confirmed/corrected in the department's meeting, undersigned by the chief of the department. The
teachers are acknowledged with respect to their rank in the quality top of the department’s
teaching staff;
The assessment of the study subjects by students and colleagues, a procedure which fixes the
responsibilities, the steps to be followed, the questionnaires to be filled in and the evaluation
modality of the global indicator of performance/receptivity of each teacher, and each discipline.
The students’ evaluation is interpreted by processing the results of the questionnaires by a
representative of the faculty/department's staff not involved in the teaching process (e.g., a
researcher). The procedure recommends two types of questionnaire, leaving at the department’s
disposal to choose one of them. The teacher's self-evaluation paper set up by this procedure is
unique and details the possible activities in the domains: educational, scientific research, national
and international recognition, involvement in the academic life.
Function of the evaluation’s results, there are foreseen some disciplinary sanctions for the teachers
with an average result of less than 60%, warnings addressed to teachers with a result between 60%-
80%, as well as financial benefits awarded to the teachers with a result of 100%.
Providing accessibility to the learning resources is another quality requirements closely followed by
the DIDIFR, as it implies providing material conditions for carrying on didactic activities and for
students' and teaching staff research. Synthetically, the learning resources available for students and
teachers are: over 5 thousands e-books, printed volumes (treaties, monographs, didactic text books),
from all domains for which DIDIFR is allowed to proceed BL activities; on-line access at
SPRINGERLINK specialty magazines Thomson ISI, ScienceDirect, Open Access Journals, Oxford
Journals, Scientific Journals International, Periodicals Index Online, LiteratureOnline, MLA
International Bibliography.Scholarships are offered from own financial resources. An eLearning
platform is also available to the BL students, as well as free on-line access to the most important and
representative national and international magazines and journals in their field of study.
5. Assessing and evaluating the effectiveness of the students’ experience and
satisfaction in the BL environment of the University of Oradea from the
learning and eLearning theories perspective
From the perspective of the way in which the University of Oradea has adopted and implemented the
Part V and the Part VI of the ARACIS “External evaluation methodology, standards, reference
standards and list of performance indicators” into its BL (DLS and RFE) study programs, the research
methods used to measure the students’ satisfaction with these BL programmes, as well as the way in
which the internal quality procedures have been implemented within the University of Oradea and
their registered results so far, was the survey method.
Since survey methods are valid techniques to access individuals’ perception, it has been determined
as an appropriate method for our research, the data being obtained based on questioning
respondents. Over 2800 questionnaires were applied to all BL (DLS and RFE) students of the
University of Oradea in the last couple of years, for assessing and evaluating the effectiveness of the
students’ experience and satisfaction in the BL environment of the UO.
Khan's octagonal framework (Khan 2005) was considered as a guide for our research purposes,
although he did not clarify in any explicit manner the process of developing his model. Nor did he refer
to any of the practical and theoretical knowledge produced in this field.
The respondents were asked a variety of questions regarding the quality of the BL programmes of the
UO. These questions were asked throughout the use of the eLearning platform of the DIDIFR, and
participants were answering the question confidently. The survey was conducted on June 2010 and
again, on June 2011, on all students enrolled in the study programmes offered by the DIDIFR. A total
number of 2817 questionnaires were answered out of a total possible number of 2920. The number of
missing data (2920-2817=103) means a number of questionnaires which had been asked to the
respondents, but were not answered by the respondents for not being available at that time. So we
have ignored such missing records because it does not affect the results at all.
139

Adriana Giurgiu, Carmen-Alina Popa and Adrian Negrea
Ethical considerations were maintained and accepted during the survey.
The questionnaire covered the whole research area. The detailed given responses are available in
the Table 1. The purpose of this table is to give an idea to the readers about the accuracy and
reliability of the empirical study.
Table 1: The questionnaire - students’ perception upon the BL environment at the University of
Oradea (%)
140

Adriana Giurgiu, Carmen-Alina Popa and Adrian Negrea
Figure 1: The perception of resource support factor (%)
6. Results and conclusions
In our investigation we have identified a number of factors which could be described as Khan's main
factors. These factors are considered as a framework for developing BL system within the UO and
were validated by survey. The framework we have found can be used as a background to further
improve the quality of our BL programmes.
Based on results obtained from the questionnaires applied to the BL students of the UO, the following
conclusions can be drawn from the perspective of the learning and eLearning theories presented in
this paper, as follows:
DIDIFR meets mandatory regulatory requirements, listed in the external evaluation methodology
of ARACIS covering: the legal framework for the organization and operation, content of teaching,
scientific research, the material, financial activities, business management and institutional
structures; besides, DIDIFR meets the Koohang’s model requirements, according to the students’
opinions;
At the institutional level, the standards established by ARACIS as well as the behaviourist
conditions are met;
The performance indicators are better than at the previous institutional assessment: on the one
side, no indicator is below the minimum threshold, while 29 of the 43 performance indicators are
above the minimum requirements of the ARACIS; on the other side, by taking into account the
cognitive theories, the teaching approach should focus on enforcing learners to use their metacognitive
skills by reflecting on what they learn, collaborating with other learners or checking their
progress;
The majority of the study programmes at the DIDIFR have been validated as complying with
regulatory requirements, the standards and the targets of the ARACIS during external evaluations
that have been conducted since the last institutional assessment, as follows: 6 out of 6 Bachelor
BL Degree Programs evaluated as "Trustfully" and re-accredited in 2009.
As it concerns the Constructivism, the way in which the DIDIFR accomplishes the characteristics of
the constructivism learning and eLearning theories, as resulted from the students’ survey, are
reflected in Table 2.
The study also reveals the fact that the students still face some challenges resulting from the bad
perception over their studies in the BL context, the lack of pedagogical counselling in their curriculum,
the lack of appropriate resources, the lack of inter-connection with the teaching staff as within the
traditional system, and the impersonal connection with their eLearning platform. The survey ensures
this way a full insight into the quality of the learning and teaching activities provided by the DIDIFR,
along with the quantitative data. The focus of this study was on the students' activities and opinions in
the given BL environment which will contribute to form an interactive knowledge among the different
stakeholders on BL system of the UO, as well as to improve the quality of the existing BL Bachelor
programmes.
The SWOT analysis carried out based on the operating parameters, the students’ opinions and the
current status of the University of Oradea, reflects the following:
141

Adriana Giurgiu, Carmen-Alina Popa and Adrian Negrea
Strong Points: the buildings, the campus' spaces, the documentation sources, the students’ facilities,
the new library (which is of reference among the universities of Romania), the teaching staff, the large
number of teachers undergoing documentation and research stages abroad, the national and
international recognition of research results (grants, scientific papers), the relationship with the sociocultural
and economic environment in the area, the annual investment program, the documentary
facilities (treaties, journals, access to databases), the research stimulation by individuals and teams
with excellent research results, and the web site of the DIDIFR and its eLearning platform.
Table 2: The DIDIFR accomplishments with the characteristics of the constructivism learning and
eLearning theories
Weaknesses: the very low number of researchers that pursues post-PhD activity, small number of
research projects where students are involved, the unfit quality of some of the candidates and
students, the lack of interest of some of the teachers and administrative, the morally and technically
exceeded equipments in some didactic and research lab, the postponing of the administrative
reorganization, the low participation in international networks and consortia, the poor promotion in the
socio-economic environment of the research centres competences, the birocratic barriers.
Opportunities: the adjustment of the educational offer upon opportunity, efficiency and performance
criteria, the administrative reorganisation, according to the European practice, for the increase of
financial efficiency, the improvement of the educational processes by understanding and applying the
142

Adriana Giurgiu, Carmen-Alina Popa and Adrian Negrea
principles and methods of quality management, the increased level of facilities offered by structural
funds projects, including cross-border projects.
Threats: the decreased number of candidates from demographic considerations and their guidance to
other universities in Europe, including Hungary; the failure to apply sound and consistent principles
and methods of quality management; the establishment of subsidiaries of European higher education
institutions in the area; the low level of remuneration, which affect the quality of the teaching staff,
especially the young ones, who are not attract to university career anymore.
In conclusion, the quality of educational and administrative processes at the DIDIFR is much
improved compared to previous institutional and departmental assessment, being for now a
competitive higher education institution, connected to the National and European Higher Education
System attached to academic values, national and regional need, indispensable from the border
balance's perspective in relation to the University of Debrecen (Hungary).
Acknowledgements
At a level of a Bachelor programme developed within the BL environment, the research method
proposed within this paper can be used to enhance and assure quality in BL. The paper may also
benefit through its in-depth analyse to all interested parts in measuring students’ satisfaction with their
Higher-Education programs developed in a blended learning system, as well as to all education
providers attempting to assess and evaluate effectiveness of the student experience in such of an
environment.
References
ARACIS (2006) Part I - Study programmes (Bachelor’s and Master’s degrees) accreditation external evaluation,
[online], http://www.aracis.ro/en/procedures/ (accessed during April 2010 - December 2011)
Heinich, R., et al. (2002) “Instructional Media and Technologies for learning”, 7th edition, Merrill Prentice Hall
Kalat, J. W. (2007) “Introduction to psychology”, Pacific Grove, CA: Wadsworth- Thompson Learning
Khan, B. H. (2005) ELearning quick checklist, Hershey, PA: Information Science Pub.
Koohang, A. (2009) A learner-centred model for blended learning design, International Journal of Innovation and
Learning, 6(1), 76-91
Meyer, R.E. (1998) Cognitive, metacognitive, and motivational aspects of problem solving, Instructional Science,
26(1-2), 49-63
Murphy & Cifuentes L. (2001) Using Web tools, collaborating, and learning online, Distance Education, 22(2),
285-305
Woolfolk, A. E. (1993) “Educational psychology”, Boston: Allyn and Bacon
143

The Educational Scorecard: A More Appropriate
Evaluation?
Vivienne Griggs¹, Michelle Blackburn² and Joanna Smith¹
¹Leeds Business School, Leeds Metropolitan University, Leeds, UK
²Sheffield Hallam University, Sheffield, UK
v.griggs@leedsmet.ac.uk
m.blackburn@shu.ac.uk
j.a.smith@leedsmet.ac.uk
Abstract: Against a backdrop of public sector cuts, increasing university fees and high youth unemployment, we
are facing challenges in Higher Education to demonstrate the value of our courses. Assessing the value of
learning, however, is not straight forward. This paper reports on a study of evaluation processes on a postgraduate,
professionally accredited diploma delivered at a selection of post 1992 universities. The driver for the
study was the premise that current evaluation processes do not fully demonstrate the value of the course nor
take into account the needs of multiple stakeholders. We would argue that we cannot truly assess accountability
and comparability without engaging a range of stakeholders, not only in soliciting their views on the outcome of
the learning but also in the design and implementation of evaluation processes. The project included
benchmarking University evaluation processes and conducting a dialogue with stakeholders. The study adopted
a qualitative management research approach, involving: a review of current practice, comparison with a sample
of equivalent courses, and consultation (in the form of focus groups and semi-structured interviews) with a
sample of students and employers. The paper discusses findings and proposes recommendations for future
evaluation procedures in the design of an ‘educational scorecard’ for the course. Kaplan and Norton’s ‘balanced
scorecard (1996) concept was adapted to provide a mechanism to represent and balance the needs of different
stakeholder groups in the education process. The ‘educational scorecard’ presented in this paper is developed
specifically for the post-graduate diploma in Human Resource Management (PDHRM) at Leeds Metropolitan
University. However, the model is flexible and may be adapted for other HE institutions and courses.
Keywords: evaluation, HRD, higher education, stakeholders, balanced scorecard
1. Introduction
The plethora of research available suggests extensive critique of evaluation of learning in academic
settings. Loveland (cited in Haan, 2010) claimed there have been over 2,000 studies in student
evaluations of teaching during the past seventy years. Therefore it is inappropriate to suggest a failure
of activity. The challenge remains, however, about what, how and when we seek to evaluate.
It is clear that today's universities have a wide range of stakeholders all with complex needs. In
attempts to create a framework for measuring delivery against stakeholder requirements, researchers
(e.g. Karathanos & Karathanos (2005); Beard (2009)), have sought to draw upon the work of Kaplan
and Norton (1996) on the 'balanced scorecard' and attempted to create a framework that will measure
performance of an academic institution. All of these writers have tended to accept the Kaplan and
Norton (2006) 4 box framework. Barnth et al (2011) however, sought to create a sector appropriate
model albeit driven by a financial management perspective, and specified 'research', 'teaching' and
'service' as the three perspectives for measurement. This paper seeks to establish whether there is an
appropriate 'educational scorecard' for the post-graduate HRM course through an evaluation of the
needs of varying stakeholder groups.
A key challenge for evaluation of a part-time post-graduate, professionally accredited, course is the
scope and complexity of stakeholders. Sarrico et al (2010) found that the existence of multiple
stakeholders was often ignored in HE evaluation. It could be argued that we cannot truly assess
accountability and comparability without engaging a range of stakeholders in the design and
participation of evaluation procedures. Many evaluation procedures focus solely on student feedback
but this means that the data collected are not triangulated with that of other sources. Sproule (cited in
Nygaard and Belluigi, 2011) goes so far as to argue that the exclusive use of student feedback to
evaluate teaching and/or courses is an ‘anathema to the academic mission’.
The current climate for Higher Education presents a challenging time for Universities. The economic
conditions coupled with Government policies to reduce public-sector funding but seemingly increase
regulation, present both a squeeze on student numbers and a more competitive market place. Chris
Woodhead, former chief inspector of schools, claims vice chancellors have ‘turned a blind eye to
144

Vivienne Griggs, Michelle Blackburn and Joanna Smith
these issues of quality’ and suggests higher tuition fees will encourage students to demand a better
service (Answer the Question, 2012) This exerts pressure on universities and ultimately course
leaders and lecturers, who come under greater scrutiny to demonstrate added value at an acceptable
cost, and to respond in a constructive and timely manner to student demands. The Chartered Institute
of Personnel and Development (CIPD) suggest it is particularly critical in the current economic climate
to ensure a good return on investment for learning initiatives as many companies will have increased
focus on costs (CIPD, 2011). The majority of students on the course under investigation are part-time,
professional students, and therefore we face similar pressure to demonstrate the value of investment
in the course to both students and employers.
This paper unfolds as follows: first we cover a brief review of issues concerning evaluation procedures
in HE, then the research approach for the study is explained, the findings are then explored for each
respondent group under three key themes, finally a proposal for an ‘educational scorecard’ is
presented.
2. Evaluation procedures in HE
The CIPD annual Learning and Talent Development survey (2011) found post-course evaluations or
‘happy sheets’ to be the most commonly used method of learning evaluation (93% of relevant
organisations)this is also the approach found in academic institutions. This approach in education is a
standardized form generally focused on students’ perceptions of teaching rather than their actual
learning. It may provide useful feedback on the teaching experience but if the aim is to improve
attainment of learning outcomes, the focus should be factors that facilitate student learning (Denson
et al, 2010). Similarly Nygaard and Belluigi et al (2011) suggests that when evaluation is based on
de contextualised learning, it ignores the learning processes of students, providing information on
perceptions ultimately like a popularity test. This is supported by research by Frick et al (2010) who
claim course evaluations traditionally used in higher education have few items that are empirically
related to learning achievement.
Powney and Hall (1998) argue that student satisfaction questionnaires are counter-productive, partly
because they are underpinned by an unproven assumption that satisfied students get better results.
Concerns have also been raised by numerous researchers about what influences the results (see for
example, Francis (2011) who cites studies showing students attitudes to a course before the start of
the semester, prior interest and field of study all have an impact on ratings.) The impact that such
monitoring initiatives have in actually improving the quality of teaching has also been questioned.
Denson et al (2011) cite a study that found only 3–10% of instructors reported making major changes
in their teaching as a result of course evaluations. However, some researchers do claim a positive
relationship between course grades and student ratings of course satisfaction. Svanum and Aigner
(2011) suggest this relationship is well established but not straight forward so that controversy
remains about the interpretation and magnitude of this association, together with its implications.
Additionally, Frick et al (2010) report on meta-analyses of studies that have examined the correlation
of the relationship between items such as ‘‘this was an outstanding course’’ and student achievement,
finding a moderate correlation. Nevertheless, whilst not doubting the existence of the relationship,
Frick et al (2010) stress that it does not indicate how to improve teaching. As a whole these results
imply that the reliability and validity of measurement tools require further examination. Similarly, the
extent to which evaluation takes account of the strategic direction of the Business School, the factors
that influence student attainment and the requirements of those paying the fees needs exploring.
Concerns about the prominence of student satisfaction measures relate to the ‘what’ of evaluation
procedures in higher education, but criticisms have also been directed at the ‘when’ or timing of
evaluation, and the ‘how’ evaluation studies are conducted. Firstly addressing the timing issues,
formal procedures are often at the end of a module or course. Whilst there may be a clear rationale
for such summative evaluation, the multiple purposes of evaluation, such as, learning (Easterby-
Smith, 1994) may not be facilitated in this way. Nygaard and Belluigi (2011) suggest effective practice
would involve conducting data collection from students at a time when reflection could positively
impact or facilitate their learning. This supports views of a more strategic approach that incorporates
evaluation throughout the learning cycle rather than solely at the end. Secondly, the standardized
nature of processes utilized by many institutions may not reflect the diversity of subjects, students,
stakeholders and learning outcomes of different disciplines. Construction of a tailored, course specific
approach may yield more meaningful results. We suggest the ‘educational scorecard’ may provide the
framework in which this could be developed.
145

3. Research methodology
Vivienne Griggs, Michelle Blackburn and Joanna Smith
An exploratory approach was adopted based on inductive reasoning. The researchers had a good
appreciation of the breadth of issues from the literature review and their own experience. The
subsequent primary research was aimed at engaging stakeholders in issues relating to the design of
evaluation procedures. There were two key stages in the data collection. First; a benchmarking
exercise which sought to review current evaluation practices in a number of comparable courses and
meet three key objectives;
draw on best practice from other institutions,
share creative ideas and
learn from their success
The second stage sought views of two stakeholder groups: students and employers to identify their
requirements. Clearly, there are other stakeholder groups, such as the university quality department,
government funders and the professional body; for the purpose of this study their requirements were
represented by the formal regulatory and validation processes.
For both data collection exercises a non-probability sampling method was used. This relied on
convenience or opportunity sampling, drawing on a sample of the researchers’ contacts. The purpose
of the research was explained to participants and they were given the choice to be named in the
report or remain anonymous. Data collection took place by telephone interviews, focus groups and
questionnaires depending on the respondent group. To enable consistency for analysis, a structured
questionnaire or feedback sheet was used for each group, covering the same key issues, informed by
the literature review. In summary the respondents were:
5 course leaders from Post 1992 universities offering a similar professionally accredited
postgraduate courses in the same discipline
14 students on such a course at Leeds Metropolitan University
4 employers who currently or had previously sponsored students on such courses
As an initial exploratory exercise, the research was intended to focus on the priorities above, which
were often missing from existing research. The scope of this study did not allow a more
comprehensive or representative sample to be surveyed. The researchers acknowledged this
limitation but felt the nature of the investigation was such that any views from the defined population
were equally valid to contribute to the existing body of knowledge. This limitation does mean that any
findings will be tentative and may require more rigorous testing through a deductive approach before
implementation. However as our findings suggest an adaptable model for specific institutions and
courses, this may not be essential.
4. Findings
The findings provide an insight into the views of university and stakeholder group participants. They
are specific to the university and course context under investigation and are not taken to be
representative of the whole population as per the inductive approach described above. The research
focussed around three key themes:
Evaluation procedures undertaken
The purpose of such evaluation procedures
The involvement of different stakeholders in the process
These were subsequently analysed to give a composite view of evaluation of the course being
investigated and so inform the design of our ‘educational scorecard’.
4.1 Evaluation procedures
4.1.1 Benchmarked universities
Unsurprisingly, many universities surveyed undertook very similar processes but referred to them by
different names. These can be categorised as: course surveys, meetings with students / student
representatives, evaluation integrated into course content and involvement of stakeholders
(Employers, professional body and External Examiners).
146

Vivienne Griggs, Michelle Blackburn and Joanna Smith
With regards course surveys, for two institutions the surveys were centrally developed with little
opportunity to influence content at course level. The third took part in the HEA Post Graduate Taught
Experience Survey (PTES) characterised by the respondent as "the post-graduate equivalent of the
National Student Survey". As with the institutional surveys the questions were generic but information
could be broken down to course level. The reliability and validity of the surveys was unknown but still
proved helpful, as one respondent said "The processes can provide a useful insight into issues that
are important to students but it is difficult to assess how representative they are".
Pertaining meetings with students, 3 of the 5 respondents used student representatives as a source
of evaluation. One institution held a twice-yearly 'board of studies' the benefit of which "depends upon
who is chairing the board and the trail of actions". A second held twice-yearly 'staff/student liaison
committees’; the third had annual 'enhancement and development meetings' Follow up on actions
was seen as a key to credibility by users of this approach.
4 of the 5 universities conducted meetings with entire course cohorts. However, such meetings don’t
always work, one university with a post graduate forum rated it as neutral as "only a few from the
course partake in it".
Finally concerning integration, two universities identified how they had integrated an evaluation
process into their course content. The first institution required students to run an end of course forum
attended by employers and teaching staff as part of assessed skills development. A second institution
used a novel approach where the staff/student liaison committee "is chaired by one of the students
and the course manager is in attendance and it is a democratic process”. This approach was used to
help students "learn about working with representatives and the consultative committee approach."
Involvement of stakeholders is addressed below.
4.1.2 The student perspective
In a review of existing procedures, students were generally aware of both informal and formal
methods of evaluation and recognized the importance of evaluation procedures for course
development. Although one student commented ‘There is no point in conducting an evaluation if the
college is not going to do anything with the information provided. I think a good Learner Evaluation
form tells the students how the information will be used and when and where the findings will be
published.’
The range of evaluation procedures cited included: module evaluations, online discussion forums,
verbal/email feedback, assignment/exam results, university league tables, student surveys,
engagement in lectures and pass rates. One student with disabilities suggested measures of diversity
and how we deal with students with disabilities or special requirements should be incorporated into
evaluation procedures. Some concerns were expressed regarding the completion of in class
evaluations in the presence of the tutor. One student suggested that despite a broad range of
procedures existing, there may be a lack of triangulation in analysis of the information.
The value of learning was described by students in terms of transfer of learning to the work context,
either in their day to day roles or in terms of career progression and development. However, there
were mixed views about the University’s role in the transfer process and whether this was the
university's responsibility or that of the students themselves. For example, some students are in low
level jobs and felt they did not have an opportunity to transfer learning at the moment but would hope
to do so as they progressed in their career. They suggested this could be assessed through a
longitudinal evaluation (12 months and 3 years later), looking at career progression after completion
of the course and surveying their organizations/sponsors. Diverse opinions were also expressed
regarding the challenges of measuring return on expectation. It was acknowledged that the potential
multiplicity of expectations could be problematic, particularly in a course governed by professional
body requirements and university constraints.
4.1.3 The employer perspective
The first noteworthy finding is that of the 4 employers surveyed, there was very little employer
evaluation of the courses on which they were sending students. There appeared to be greater
confidence that the universities’ quality assurance processes were satisfactory and fit for purpose.
However, this meant that opportunities to evaluate transfer of learning were potentially missed.
147

Vivienne Griggs, Michelle Blackburn and Joanna Smith
The response to what employers might look for in a postgraduate, professionally accredited, course
provided wide-ranging answers including: cost, location, time and flexibility.
4.2 Purposes of evaluation procedures
4.2.1 Benchmarked universities
All respondents were asked to identify the purposes to which they put the outcome of their evaluation
exercises. The most frequently mentioned outcomes were:
To provide information to enhance the course
To measure student satisfaction
A formal requirement of the Quality process
To provide feedback on methods of delivery
To measure if expectations have been met
Other purposes identified in the evaluation literature, such as comparing staff and courses (for staff,
current and future students), measuring success rates, learning achievement, learning transfer and to
provide information for staff feedback, appraisals, evaluation of teaching and teaching research; were
considered areas outside of the concern of a course leader. Respondents were however concerned to
say that these factors did however have a role to play for other stakeholders in the institution.
The various institutions displayed different attitudes to the evaluation process: for a number it is a box
ticking, ranking and compliance driven process driven from the centre. For others it is about having a
very positive faculty attitude to quality and having the centre there in a supportive rather than a
‘policing’ capacity. From all respondents there is the genuine desire to improve the student
experience by gathering appropriate feedback.
Feedback had led to developments on a number of the courses, but not usually as a consequence of
the rather blunt university wide surveys. It is difficult to measure the extent of change that had taken
place as a result of evaluation procedures. This is because all of these accredited courses had
undertaken a professional body-driven review within the last year which has resulted in extensive
course revisions.
4.2.2 The student perspective
Students proposed a number of concurrent purposes of the course evaluation. The most highly rated
purposes were:
To measure if learning objectives have been achieved
To measure if expectations have been met
To measure student satisfaction
To provide feedback to staff
4.2.3 The employer perspective
Respondents all agreed that feedback to staff and achieving learning objectives were very important.
Opinion differed most markedly in the extent employers believed universities should be measuring
student success rates, helping future students select their courses, measuring the transfer of learning
and providing financial information to evidence value for money or return on investment.
The results suggest a trust in universities and their evaluation processes to validate the training on
which employers choose to send their staff. Aspects such as teaching and student learning, typically
of ‘pedagogic concern’, score more consistently as important to employers than the more ‘business
concerns’ of financial information and transfer of learning.
148

Vivienne Griggs, Michelle Blackburn and Joanna Smith
4.3 Involvement of different stakeholders
4.3.1 Benchmarked universities
Perhaps surprisingly, for vocational, professionally accredited courses, engagement with employers
on a regular basis was very limited. One institution used a 'research network' of industry specific
practitioners as guest speakers, a second engaged through 'placements' both of which arguably, are
not about direct course evaluation. A third said employer involvement was every 5 years at periodic
review. Of the final two, one is hoping to achieve employer engagement through the new forum, the
other said that this level of engagement happened at a business school level not at the course level.
Two institutions also mentioned the professional body and external examiners as stakeholders of the
process. It seems that the nature of the relationship, focused on development as well as compliance,
was key to securing a positive approach to the evaluation process and its outcomes.
4.3.2 The student perspective
The respondents perceived students, teaching staff and employers as stakeholders but did not
suggest any other stakeholder groups. Students were seen as the primary ‘customer’. The role of
employers was less clear, they were seen as sponsors, additional customers or even irrelevant to the
university as the employer/student relationship was separate to the student/university relationship.
One student commented ‘many employers just pay for the course as part of their commitment to
Learning and Development and really their expectations in terms of learning outcomes and results are
not as high as those of the students’. This separateness was echoed in the relationship between
student and parents (or other family sponsors).
4.3.3 The employer perspective
All respondents, except one agreed that students, employers and teaching staff should be involved,
one respondent also suggested line managers as distinct from a training or HR department. The way
universities should collect feedback varied only slightly with most suggesting questionnaires and
surveys and some suggesting focus groups. There was a strong view from employers that they should
be engaged in evaluation at the end of the course and up to a year after a student has graduated,
echoing the student perspective in 3.1.2 above. Many suggestions relating to students and teaching
staff reflect common practice across the universities surveyed for this research. (As cited in section
3.1.1)
All respondents agreed that students were the primary customer, two noted the employer and one
cited parents. Parents could be extended to refer to all family members who have a stake in the
student’s investment in time and energy and their ultimate academic success.
5. The way forward
The findings supported the general tenet of the importance of evaluation in HRD literature, with all
participants recognising the requirement for, and value of, course wide evaluations. The
benchmarking data demonstrated ways that formal university procedures, which risk losing "some of
the richness of data”, could be supplemented by specific course level procedures to provide greater
depth of information. Where these were integrated with teaching on the course, they also overcame
the problem of low response rates and limited student engagement with the process.
The range of evaluation methods cited by both students and course leaders suggests that lots of
information is collected in both formal and informal ways. However, the extent to which data is utilized
in a coordinated manner could be improved. Students appeared to assume a more integrated
approach than actually occurred. In practice, the results of different evaluation techniques were not
combined into a meaningful whole, leading to a piecemeal rather than strategic view of evaluation.
This leads to a reactive approach where small changes are made based on how students feel rather
than improvements linked to higher attainment, echoing the concerns of Denson et al (2010) above.
Our conclusion is that the course evaluation for the course investigated, should include the following:
A strategic focus (linked to professional body and university strategy)
Clearly stated and communicated purpose and outcomes
149

Vivienne Griggs, Michelle Blackburn and Joanna Smith
Integration into the course as a whole not just completed at the end
Inclusion of appropriate stakeholders
Engagement of students, in order to provide greater depth of information, through customized
course level procedures that enhance rather than duplicate the standardised university
approaches
Triangulation of data from different methods to ensure a more balanced basis for decision making
Where possible, benchmarking against internal and external data
As mentioned previously in section 1, researchers have utilized the Balanced Scorecard as a
framework for academic institutions. The studies generally report on the application of the scorecard
at organizational level. Our proposal is a scorecard tailored to the specific requirements of the course
being evaluated. In order to achieve this we recommend the development of a course specific
‘Educational Scorecard’. The balanced scorecard provides a framework against which long term
performance can be evaluated. It identified 4 perspectives; 'financial', 'customers' 'internal business'
and 'learning and growth' (Kaplan and Norton 1996). When applying the scorecard, individual Key
Performance Indicators are captured under one of the four headings and these are measured and
managed regularly, sometimes as part of a management dashboard Translating this into a university
environment is not simple, as has been reported above regarding the lack of an integrated approach
between relevant stakeholders. If universities are to survive in the increasingly competitive
marketplace where the views of such stakeholders are arguably critical, then accurate measurement
and timely responses may be the key to gaining high student satisfaction scores and other success
measures.
Below you will find our proposed Educational Scorecard for the course investigated for this paper. As
with Kaplan and Norton (1996) there are 4 areas which map the priority areas for course evaluation
and management. Each perspective contains some examples of content but the lists are not
exhaustive. Recommendations from students, one priority area, who took part in the research,
supported the development of more flexible data collection, such as online discussion forums to
enable them to provide timely feedback at any stage of the course and respond to comments made
by other students. However, this may lead to the creation of unrealistic expectations around
immediate change. Additionally if students are unaware of university procedures and compliance
requirements, another priority area, it could mean they feel their desires for immediate and significant
change are being ignored rather than pursed through rather lengthy processes and channels. Also as
reported in 3.1.1 not all students participate in such processes, which could make data collected
invalid due to limited quantity or skewed by those students with the most to say.
It was generally supported by all respondents that evaluation should have a student centered focus.
This would identify the levels of student satisfaction, if learning objectives have been achieved and
whether expectations have been met. However the learning objectives may be understood differently
by each stakeholder e.g. the student, employer or professional body and so feature in differing priority
areas on the scorecard. More disparate views about other themes such as transfer of learning and
return on expectation received, according to our model, could also feature in different areas. However
this is exactly the purpose of the educational scorecard, to capture such variations in a flexible but
structured model, balancing purpose and needs accordingly.
Figure 1: The educational scorecard
150

Vivienne Griggs, Michelle Blackburn and Joanna Smith
6. Significance to wider audience
This study has focused upon one course in one university, drawing on data from similar courses
elsewhere and relevant stakeholders. Its findings are not meant to be automatically applied more
widely to different courses in other institutions. However, the Educational Scorecard is designed to be
adaptable, with scope to vary the titles and content of each quadrant.
The complexity of different audiences, even within the same stakeholder group, means views may
differ. Embracing this diversity through involving different stakeholders in the evaluation process
design and implementation may lead to a more meaningful collection of data. Better dialogue between
stakeholders at the ex-ante, interim and ex-post (Devins and Smith (2010)) stages, may help. It
should be noted that collection of evaluation data is not the end of the evaluation process. The
importance of action planning is essential to lead to continuous learning and improvement, by all
stakeholders. Practical consideration needs to be made as to how all relevant stakeholders’ views can
be captured to provide manageable data for courses and universities to analyse and implement
improvements. The Educational Scorecard contributes to the solution.
References
Barnth, R. J., Mc Gee C. H., and Cataldo II, A. J. 'Failure to Benchmark: An Out-of-Balance Scorecard at an
Academic Institution, Management Accounting Quarterly, Winter 2011, Vol. 12 , No.2, pp. 49 - 56
Beard, D. F. (2009) 'Successful Applications of the Balanced Scorecard in Higher Education', Journal of
Education for Business, Vol. 84, No. 5, pp. 275-282
CIPD (2011), Learning and Talent Development, Annual Survey Report 2011
Denson, N; Loveday, T; Dalton, H. (2010) ‘Student evaluation of courses: what predicts satisfaction?’ Higher
Education Research & Development, Aug2010, Vol. 29 Issue 4, p339-356
Devins, D. and Smith, J. (2010), ‘Evaluation of Human Resource Development’. In: Gold, J et al, Human
Resource Development: Theory and Practice. Palgrave Macmillan, chapter 7.
Easterby-Smith, M. (1994) Evaluating Management Development, Training and Education (2 nd ed), Aldershot,
Gower Publishing p.19
Francis, C. (2011) ‘ Student Course Evaluations: Association with Pre-course Attitudes and Comparison of
Business Courses in Social Science and Quantitative’, North American Journal of Psychology, 2011, Vol. 13
Issue 1, p141-154
Frick, T; Chadha, R; Watson, C; Zlatkovska, E. (2010), ‘Improving course evaluations to improve instruction and
complex learning in higher education.’, Educational Technology Research & Development, Apr2010, Vol. 58
Issue 2, p115-136
Haan, P. (2010), ‘Business Students’ Perceptions of Course Evaluations', College Student Journal, Dec2010,
Vol. 44
Kaplan, R.S. and Norton, D.P. (1996) 'The Balanced Scorecard - Translating Strategy into Action',
Harvard Business School Press, Boston, MA
Karathanos, D. and Karathanos, P. (2005) 'Applying the Balanced Scorecard to Education' Journal of Education
for Business, March/April 2005, pp. 222-230
Likert, R. (1961) ‘New Patterns of Management’ New York, McGraw Hill.
Nygaard,C and Belluigi, D. (2011) ‘A proposed methodology for contextualised evaluation in higher education’,
Assessment & Evaluation in Higher Education Volume 36, Issue 6, pages 657-671
Powney, J. and Hall, S. (1998) ‘Closing the loop: The impact of student feedback on students’ subsequent
learning, Edinburgh: The SCRE Centre.
Sarrico,C; Rosa, M; Teixeira, P; Cardoso, M. Minerva: (2010) ‘ Assessing Quality and Evaluating Performance in
Higher Education: Worlds Apart or Complementary Views?’ A Review of Science, Learning & Policy, March
2010, Vol. 48 Issue 1, p35-54
Svanum, S. and Aigner, C.‘ The influences of course effort, mastery and performance goals, grade expectancies,
and earned course grades on student ratings of course satisfaction.’, British Journal of Educational
Psychology, Dec2011, Vol. 81 Issue 4, p667-679
Woodhead, C. (2012) ‘Answer the Question’, The Sunday Times, 29 January 2012
151

Appreciative Inquiry as an Alternative Method to Assess
Service Quality in a Special Library
Elrita Grimsley
Information Service on Higher Education, Centre for Teaching and Learning,
University of the Free State, Bloemfontein, South Africa
Grimslmj@ufs.ac.za
Abstract: In the negative economic climate of today, special libraries are a luxury that few organisations can
afford. The special library needs to demonstrate its value to the organisation it belongs to and needs to justify the
continued existence thereof. The users of special libraries are a specific group of people that need information
that is utilitarian, pragmatic and immediate. The need arose to investigate how users of a special library
experienced the quality of the service. Being a small special library other evaluation methods to discover the level
of service were not appropriate to use in this context. Appreciative Inquiry is explored as a method to delve into
the level of service encountered by users of this special library. The assumptions and principles that guide
Appreciative Inquiry as methodology are mentioned. Appreciative Inquiry as methodology is grounded in peak
experiences. Appreciative Inquiry has a 4-D cycle (discovery, dream, design, destiny), to gather information but
only the discovery phase of Appreciative Inquiry is used in this study. For this case study a generic Appreciative
Inquiry interview protocol was adapted to discover the quality of the service experienced by the users of the
library. Data was collected from a representative group of users of the library. Narratives of the best experiences
of users are explored to demonstrate what is giving life to the library. The data revealed three themes, empathy,
professionalism and responsiveness to needs of the users, as life giving forces. These findings demonstrate the
quality of the service that is delivered to users of the library. While generalisations are difficult to draw from such
a small study, a recommendation is made that librarians can use Appreciative Inquiry as methodology to discover
what the life giving forces are in their libraries. For the user of the special library, the service experience is short
lived, but can leave a lasting impression. A positive service encounter can tribute to the overall well being of the
librarian, as well as the user of the library and contribute to strengthen the value of the special library to the
organisation it belongs to.
Keywords: special libraries; appreciative inquiry; service quality; higher education; service delivery; academic
libraries
Introduction
The special library in a South African context has its own set of challenges. In the negative economic
climate of today, special libraries are a luxury that few organisations can afford. The special library
needs to demonstrate its value to the organisation it belongs to and needs to justify the continued
existence thereof. It is necessary to create awareness of the exceptional contribution that special
libraries make to the organisation that they belong to, to attract more users and to improve the scope
of their services (Muller 2007:113).
Staff of the special library also need a wide set of competencies, like management skills, personal
skills and professional skills that contributes towards a unique environment. A further challenge to
special librarians especially in the learning organisation is to be at the forefront of information
provision, proactively or in response to needs. To supply a unique service, staff of the special library
has to adapt to various trends and developments in the field they are working in, technology
developments and personal education.
The Information Service on Higher Education (ISHE) at the University of the Free State in
Bloemfontein, South Africa, is a special library that addresses the information needs of academia and
students studying and teaching in higher education and is subjected to deliver answers to questions
as quickly as possible. New information, trends and statistics in higher education as an ever changing
environment, demands that staff of the service need to be well informed and have the necessary
knowledge to deliver a quality service.
Changes in the academic environment in which ISHE functions, created the need to inform
management about the quality of the service delivered by this special library.
Researching the concept of service quality, the quote from the article of Nitecki (2000:259) describes
it as:
152

Elrita Grimsley
‘Service quality deals with those expectations that the library chooses to meet, and
satisfaction is more of an emotional and subjective reaction to a time-limited event or the
cumulative experiences that a customer has with a service provider.’
In another article, Nitecki (1996:181; 188) writes that
‘Identifying new ways to conceive of quality and to monitor its attainment are current
challenges for academic libraries’ and ‘... that greater attention is needed to provide quality
library services, but that there is not a clear concept of what that means nor how to monitor it
within academic libraries.’
SERVQUAL is a well known application to measure quality of services and can also be applied to
libraries. However, although useful this instrument has not been accepted without some criticism.
SERVQUAL does not measure satisfaction but rather the gap between expectations and actual
service in a library. In other words, SERVQUAL focus on the process of service delivery, and not the
outcomes of the service encounter (Nitecki 2000:266).
When moving to a small special library with a small staff, and a small number of users, SERVQUAL is
not an appropriate instrument to measure the quality of service. In this article the potential of
Appreciative Inquiry (AI) as an alternative method to explore the outcomes and quality of a service
encounter in a small special library, is discussed. Sullivan (2004:219) believes that the promise of AI
offers a compelling alternative as a quest to find the best possible situation in libraries.
AI as methodology to gather information about the outcomes of a service encounter of users of a
special library provides a new and exciting method of doing research as AI is not often used to study
service quality in libraries. Through the information gathered, other librarians and information workers
can apply AI to investigate service delivery in libraries. Searching through literature, I could not find
any applications of AI to measure the service quality in a special library.
The aim of the article is to report on the effectiveness of AI as an alternative method to evaluate the
quality of the service being delivered in a special library. Through exploring AI methodology as a data
collection tool an appreciative environment is created where the unique qualities of the staff of the
service can be appreciated rather than being critiqued. Rather than seeing the problems, focussing on
the positive, creates an environment where good relations with users of the service can provide a
stimulating academic atmosphere.
In the consequent parts of the article, the methodology of AI will be discussed. In the discovery phase
of this methodology, it will be explained how a generic AI interview protocol were used to ask frequent
users of ISHE to relate their positive experiences of the service. It will be explained how through
analysing these narratives, the unique qualities that give life to ISHE, were determined. Conclusions
are drawn from the findings and some recommendations are made.
Special libraries
Qualities of special libraries
Special libraries have a more specific clientele than other types or traditional libraries and deal with
more specialised information. The idea is that the special library under scribes the mission of the
sponsoring organisation. The collection and services are then more targeted towards the specific
needs of their users.
The main tasks of the special library should be mirrored by performance indicators such as:
A collection that serves the needs of the customers.
Current needs more that future needs.
Speed and accuracy.
Proactive delivery of relevant information to users.
Customised user services like personal profiles, alerting services and selective dissemination of
information.
Efficient background services (Poll 2007:3).
153

Elrita Grimsley
Because of the specialised nature of the service, user’s needs and wishes as well as user’s
satisfaction with the collection and services offered are of the utmost importance.
The Librarian in the special library
Two core competencies that the librarian in the special library need are identified as:
Professional competencies that include the librarian’s knowledge of relevant information resources,
access, technology and management and the ability to apply this knowledge to provide a high quality
service that entails:
Managing information organisation.
Managing information resources.
Managing information services.
Applying information tools and technologies (Muller 2007:109 -110).
Personal competencies that relates to the special librarians’ attitudes, skills and values that enable
them to contribute effectively and positively to their organisation, clients and profession. These skills
include generic personal skills; to be innovative; a positive work behaviour; communication skills;
leadership skills and to be adaptive and flexible (Muller 2007:110; Yamazaki 2007:57; Patridge
2010:270).
A huge responsibility also lies with the special librarian to take ownership of his or her own education
to be at the forefront of information provision, proactively or in response to own and users’ needs. It is
imperative that the librarian in the special library stays an active learner and purposefully read and
study the discipline or field of study that the special library covers (Muller 2007:111; 116).
Shumaker and Talley (2009:55) published a report on embedded librarianship. Their
recommendations support the opinion of Muller that the librarian’s knowledge of the customer domain
may require relevant academic degrees.
Another author mentions that the relationship between the librarian and the customer is much closer
in special libraries. The librarian may have to interpret what is best for the user. A customer may need
something, but are not necessarily aware of it, thus making needs more difficult to detect (Quinn
1997:363).
Service delivery in special libraries
Quinn (1997:359) indicates that library users must be viewed as customers, because they justify the
existence of the library. To offer a quality service means that the librarian must be able to view
services from the customer’s point of view, because it is the customers that define value. He also
states that “... the manner in which librarians treat users, and their behaviour and communication
style, may be as important to users as the accuracy of the answers they are given” (Quinn 1997:360-
361).
Needs of the user of the special library are usually utilitarian and pragmatic, they need information
that specifically answer a question and usually they need the answer quickly. They are searching for a
specific piece of information rather than a broad overview. They often have deadlines to meet and are
often unsure about what they are supposed to find (Freese and Freese-Brislin 1996:11).
Literature repeatedly states that it is difficult to measure the quality of service as it is concerned with
customer’s expectations and perceptions of quality. For this same reason, standardisation of quality is
complicated. In contrast to products, services are usually short-lived as they only last as long as the
activity lasts. Many can declare that quality is either absent or present, but no single definition of
quality is commonly accepted. Satisfaction has an emotional and subjective reaction to a time-limited
event that a customer has with a service-provider (Yeo 2008:268; Nitecki 1996:181).
Quality can only be maintained through a consistent and conscientious effort and a continued need to
pursue excellence (Yeo 2008:267).
154

Elrita Grimsley
Poll (2007:3) is of the opinion that quality in the special library will be mainly defined by speed,
actuality and accuracy of the information provided by the service. Special emphasis is placed on userorientation
and user-friendliness.
According to a study done by Kandampully (1998:438-439) it is the service personnel in an
organisation that are crucial to the emotional connection and development of a long-term relationship
with the customer. The interaction between the customer and the provider of the service has been
identified as the critical factor that determines and defines the service to the customer. To connect
emotionally with the customer and win the customer’s loyalty and trust creates loyalty. But, service
loyalty precedes customer loyalty. Thus the service must at first be excellent to impress the customer.
Quinn (1997:359) confirms this statement by stating that the manner in which librarians treat users,
and their behaviour and communication style, may be as important to users as the accuracy of the
answers they are given.
The following quote from Quinn (1997:367) links to the overall aim of this article:
“It is interesting to note that even though many librarians spend at least some part of their
day involved in public contact, there appears to be no component of library school
training that gives librarians instruction in how to handle customers.”
Appreciative inquiry
Background of appreciative inquiry
AI is a relatively new research approach. The methodology was developed by David Cooperrider in
the 1980’s. AI is both a process and philosophy (Watkins 2009:2).
AI is mainly used in the field of organisational development. It is a process that inquires into, identifies
and develops the best in organisations to create a better future. Instead of focusing on problems, the
focus is placed on what is working well in a particular organisation. Thereafter, the members of the
organisation envision what might be if the best occur more often. The members then engage in a
dialogue to find out what is needed (a highly participative process) both in terms of tasks and
resources to bring about the desired future. Finally, the desired changes are implemented (Coghlan
2003:5-6).
Theoretical foundation
Theoretical foundations of AI can be found in social constructionism which implies that meaning is
generated through language. It is through language and shared meaning that reality is created. The
power of this positive image and its implications on staff of an organisation creates a positive,
energised environment for staff to perform at their highest level.
It is also a form of action research focusing on the exploration of an organisation’s life giving forces
(Scheel 2007:32).
Assumptions and principle
To understand AI, it is important to understand the five principles that underpin the theoretical basis of
AI.
1. Constructionist principle – Organisations are living, human constructions where words create
worlds .
2. Principle of simultaneity – Change begins the moment we ask a question.
3. Poetic principle – Organisations are social constructions, we can choose what to study in an
organisation.
4. Anticipatory principle – The image of the future guides the current behavior and actions of the
system.
155

Elrita Grimsley
5. Positive principle – The more positive the question, the longer lasting and successful the change
effort will be (Norum 2008:102-105).
Process: The 4-D cycle of appreciative inquiry
The process which is used in AI to collect information is called the 4-D cycle. This process consists of
the following:
Discovery – Conduct an inquiry into the topic and assemble the stories and key ideas that come
out of the inquiry.
Dream – Generalise the discoveries into an image of how the organisation would function if what
you have discovered were fully alive in the present.
Design – How could the organisation be structured to reflect the best of what has been
discovered and created in the previous phases?
Destiny/Delivery – Sustaining and strengthening the capability of the whole system. The circle is
completed and the whole organisation begins to innovate to align with what came out of the
dream and the design phases (Dunlap 2008:26; Watkins and Stavros 2009:22).
A visual presentation of the four core principles of AI is presented in Figure 1:
AFFIRMATIVE
TOPIC CHOICE
Appreciative Inquiry “4-D” Cycle
Source: Cooperrider and Whitney 2008:34.
Figure 1: The AI cycle used in this research
Method
ISHE is a special library on higher education. It serves students studying in higher education and
lecturers teaching higher education programmes as well as doing research. Courses include
assessment and moderation, teaching, learning, e-learning, policies, leadership and management,
program development, quality assurance, adult learning and development, postgraduate supervision,
mentoring, administration and support, learning design, facilitation of community service and a
research component that entails the writing of a mini-dissertation.
A new Centre for Teaching and Learning (CTL) was established on the campus. ISHE forms an
integral part of CTL but the main focus has shifted to deliver an academic teaching and learning
service to all the teaching staff on the campus of the University of the Free State.
Through evaluating the outcomes of the service experience of users of ISHE, the management of
CTL can be informed of the unique qualities of ISHE and thus strengthens the value and contributions
that it makes towards CTL as well as the University of the Free State.
Choosing an instrument to obtain this information was problematic. SERVQUAL is a well known
measurement instrument. It was developed in the early 1980s to measure and evaluate service
quality. It measures the gap between customer expectations and experience (Cook and Thompson
2000:248). Limitations of the SERVQUAL model when applied to academic libraries, is often
mentioned. One of the limitations mentioned by Quinn (1997:367) is the fact that the service quality
model was originally developed for use in a commercial environment. Cook and Thompson
156

Elrita Grimsley
(2000:255) also warn that some caution must be exercised in interpreting results from SERVQUAL
when applied to libraries.
Buttle’s (1996:8; 24) critique of SERVQUAL mentions a number of theoretical and operational issues
which should be of concern to users of the instrument. Important to the scope of this article is the
statement that “SERVQUAL focuses on the process of service delivery, not the outcomes of the
service encounter” Buttle (1996:10).
This last statement opens the door for the application of AI to find out how users of ISHE experience
the outcomes of the service delivered to them. Only the Discovery Phase of the 4-D cycle of AI was
used in the Interview Protocol to discover and identify the best of what is in ISHE. Discovery involves
to find out what is worth valuing, positive capacity, what is the successes and the most alive and vital
moments when users visit ISHE. Every question asked, is positive.
In every organisation there is something that works well. Discovering these strengths can be a starting
point to discover how to create more excellence (Cooperrider, Whitney and Stavros 2008:8-9). Other
advantages of this discovery are a clarified or enhanced sense of identity, renewal of group energy
and improved working relations.
The research design and data collection method
This is a case study employing a qualitative approach and AI design.
Reed (2007:141) describes a case study as “... phenomena that are explored in relation to the
contexts in which they occur, in other words, what happen should be seen in relation to where it
happens”. This description fits the overall design of the study as a case study.
Before beginning the 4-D Cycle of AI it is important to select the topic or focus of the inquiry. What we
choose to study is likely to become our new reality (Hammond 1996:32-33). To identify a core
element that gives life to ISHE, the way in which users of ISHE experience the delivery of the service,
was decided upon. A trademark of AI is the positive questions that are asked during the inquiry.
A generic AI interview protocol (Cooperrider and Whitney 2008:310) was used. This interview protocol
was adapted to inquire how regular users of ISHE experience the service when they use the facility.
Only the discovery phase of the 4-D Cycle is covered by this interview protocol.
The target group for collecting information was selected on the basis of persons using ISHE once a
week. These persons were identified as five lecturers working in the same building where ISHE is
located. These persons are lecturers in Higher Education Studies, a postgraduate programme
provided by the University of the Free State.
Apart from lecturers using ISHE, a group of five students studying towards their master’s degrees or
doctoral degrees in higher education, were also identified as regular users of ISHE.
The interview protocol was electronically distributed to these 10 persons. In line with the AI
requirement, research participants were encouraged to tell stories about how they experienced ISHE.
The interview protocol is shown in Figure 2.
Data analysis and dissemination of information
When the data was gathered from the chosen participants (9 narratives in total were returned), the
process began to make sense of the data. “Sense-making is making sense of the themes and
patterns discovered in the interviews, lifting up meaningful metaphors and stories that give momentum
to organizational success” (Cooperrider, Whitney and Stavros 2008:116).
When the data was gathered from the chosen participants (9 narratives in total were returned), the
process began to make sense of the data. “Sense-making is making sense of the themes and
patterns discovered in the interviews, lifting up meaningful metaphors and stories that give momentum
to organizational success” (Cooperrider, Whitney and Stavros 2008:116).
157

Elrita Grimsley
INTERVIEW PROTOCOL
1. BEST EXPERIENCE (Discovery)
Tell me about the best experience you had when you made use of the Information Service on Higher
Education (ISHE)? Describe the event in detail. When describing the event, keep the following
questions in mind:
What were you doing?
Who was involved?
What happened?
What was the outcome?
What difference did it make towards your research?
What made it an exciting experience?
What is special?
2. VALUES
What do you value most about the service?
What do you value most about the staff?
3. CORE VALUES (Life giving)
As you think about the service, what do you think is the one core value
that makes the service unique and special?
Source: Cooperrider and Whitney 2008:310.
Figure 2: The interview protocol.
Data analysis and dissemination of information
In sense-making, a diversity of approaches can be used. There is no single right way to analyze the
data. It is important to find creative ways to organize, listen too, understand and draw conclusions
from the data. Data can be coded under key themes. The researcher must look for common threads,
to identify themes, narratives and stories that give life to the organisation. Attempts must be made to
discover “the best of what is” (Cooperrider, Whitney and Stavros 2008:117).
Data can also be reduced and displayed in many forms, e.g. tables, pictures, storybooks, newsletters,
diagrams or charts. The search for one perfect method to make sense of data is not the point
(Cooperrider, Whitney and Stavros 2008:117).
The data for this research was captured in written form. Working through the data, the following two
quotes were noticed:
A: “My experience of the Information Service on Higher Education is one of people who
care about me as a user, who always helps me in a friendly and professional way.”
B: “Through the constant flow of new publications received via ISHE I have often had
“aha” experiences when finding new resources that opened up fresh, innovative avenues
in the development of a fairly new subject.”
The most compelling story is reported as the following:
C: ”I have seen my students changing from confusing and anxious students into
confident novice researchers because of the assistance provided by the ISHE staff in
retrieving relevant documents for their research projects.”
From the data, the following themes could be identified:
Empathy: The users of ISHE noticed that they were treated with empathy. Words used to describe the
encounter are: friendly, helpful, warm, caring, patient, supportive, encouragement, emotional support.
Professionalism: The users of ISHE noticed the professionalism with which they were treated. Words
describing the term are: commitment, precision, trustworthiness, specialization, quality, effective,
professional expertise, knowledge of subject, qualified, integrity, trustworthiness, courtesy.
Responsiveness: The users of ISHE noticed the way that their queries were dealt with. Words
describing the response to queries are: effective, prompt, timely, helpful, fast, thorough, speedy,
efficiency.
158

Discussion
Elrita Grimsley
With the discovery phase of AI as applied through the interview protocol, the best experiences of
users of ISHE could be identified. Because the experiences are recalled at the best moments it
happened, the best of what is could be discovered. The narratives of persons A, B and C reflect a
positive, live giving encounter in ISHE. It is through language and shared meaning that this reality is
created.
From the data, the themes that were identified, empathy, professionalism and responsiveness,
captures what is worth valuing, the positive capacity, the successes and the most alive and vital
moments of the service rendered at ISHE. AI as a research methodology is thus grounded in peak
experiences.
These facts link back to the mentioned literature that the interaction between the customer and the
provider of the service has been identified as the critical factor that determines and defines the
service to the customer. To connect emotionally with the customer, especially in a special library
where the relationship between the librarian and the customer is much closer, creates a positive,
energised environment where staff strives to deliver an excellent service. This demonstrates that not
only does staff of ISHE put the customer first, but also that AI as a research methodology puts people
first.
AI as a research methodology revealed what gives life to ISHE. Capturing the past through best
encounters that users of ISHE experienced, the future is build on the positive experiences of the
users of ISHE. The excellent service as a positive and life giving force of ISHE reflects the anticipatory
principle of AI that a positive image creates positive action. The power of this positive image creates a
positive, energised environment for the staff of ISHE to perform at their highest level. The excellent
service must thus continue to give direction and life to ISHE.
Recommendations can be made to librarians of special libraries that AI can be used as a research
methodology to discover what the life giving force in their libraries is. It should also be noted that while
librarians are not trained in client service, they should be aware that treating users of libraries with
empathy, professionalism and prompt service delivery, provides the user of the library with a positive
client experience.
Conclusion
The quality of the service in a special library can truly only be justified by the user. Treating the user
with empathy, professionalism and being responsive to the needs of the users contribute to the quality
of the service. The service experience is short lived but can make a lasting impression on the user. If
this is a positive experience to both parties it contributes to make the librarian feels appreciated and
motivated to provide even a better service. In this way the librarian not only becomes a person of
special value to the user of the special library, but also to the organisation or institution that the
special library belongs to.
With excellent service value is added to the continued existence of a small special library like ISHE.
Especially with the new demands to the service, it can contribute to a high level of excellence and
thus contribute to the success of the specific centre it serves, as well as to the institution as a whole.
AI as a research methodology opened the way for the researcher to evaluate and realise the
exceptional value that ISHE contributes towards CTL, as well as the University of the Free State.
References
Buttle, F. (1996) “SERVQUAL: Review, Critique, Research Agenda”, European Journal of Marketing, Vol. 30, No.
1, pp 8-32.
Coghlan, A.P., Preskill, H. and Catsambas, T.T. (2003) “An Overview of Appreciative Inquiry in Evaluation”, New
Directions for Evaluation, No. 100, pp 5-22.
Cooperrider, D.L., Whitney, D. and Stavros, J.M. (2008) Appreciative Inquiry Handbook for Leaders of Change,
2 nd ed., Berrett-Koehler Publishers, San Francisco.
Dunlap, C. (2008) “Effective Evaluation Through Appreciative Inquiry”, Performance Improvement, Vol. 47, No. 2,
pp 23-29.
Freese, E. and Freese-Brislin, L. (1996) “Special Skills for Special Librarians: Is Current Training Good
Enough?”, Innovation, No. 12, pp 10-15.
Hammond, S.A. (1996) The Thin Book of Appreciative Inquiry, CSS Publishing Company, Plano.
159

Elrita Grimsley
Kandampully, J. (1998) “Service Quality to Service Loyalty: A Relationship Which goes Beyond Customer
Services”, Total Quality Management, Vol. 9, No. 6, pp 431-443.
Muller, B. (2007) Challenges Facing Special Libraries in South Africa IN Bothma, T.J.D., Underwood, P. and
Ngulube, P. (Eds), Libraries for the Future: Progress and Development of South African Llibraries, LIASA,
Pretoria.
Nitecki, D.A. and Hernon, P. (2000) “Measuring Service Quality at Yale University’s Libraries”, The Journal of
Academic Librarianship, Vol. 26, No. 4, pp 259-273.
Nitecki, D. (1996) “Changing the Concept and Measure of Service Quality in Academic Libraries”, The Journal of
Academic Librarianship, May, pp 181-190.
Norum, K.E. (2008) Appreciative Inquiry as Conversation IN Jenlink, P.M. and Banathy, B.H. (Eds), Dialogue as
a Collective Means of Design Conversation, Springer, New York.
Parasuraman, A.Z., Zeithaml, V.A. and Berry, L.L. (1985) “A Conceptual Model of Service Quality and its
Implications for Future Research”, The Journal of Marketing, Vol. 49, No. 4, pp 41-50.
Patridge, H.M., Menzies, V., Lee, J. and Munro, C. (2010) “The Contemporary Librarian: Skills, Knowledge and
Attributes Required in a World of Emerging Technologies”, Library and Information Science Research, Vol.
32, pp 265-271.
Poll, R. (2007) “Quality Measures for Special Libraries”, Paper read at 73 rd IFLA General Conference and
Council, Durban, South Africa, August.
Quinn, B. (1997) “Adapting Service Quality Concepts to Academic Libraries”, The Journal of Academic
Librarianship, September, pp 359-369.
Reed, J. (2007) Appreciative Inquiry: Research for Change, Sage, London.
Scheel, R. And Crous, F. “Leveraging Organisational Cultural Capital”, South African Journal of Industrial
Psychology, Vol. 33, No. 1, pp 29-37.
Shumaker, D.A. and Talley, M. (2009) Models of Embedded Librarianship. Final Report, Special Libraries
Association, Alexandria VA.
Sullivan, M. (2004) “The Promise of Appreciative Inquiry in Library Organizations”, Library Trends, Vol. 53, No. 1,
pp 218-229.
Watkins, J. M. and Stavros, J.M. (2009) Appreciative Inquiry: OD in the Post-Modern Age IN Rothwell, W.J.,
Stavros, J.M., Sullivan, R.L. and Sullivan, A. (Eds), Practicing Organization Development: A Guide for
Leading Change, 3 rd ed., John Wiley, San Francisco.
Yamazaki, H. (2007) “Changing Society: Role of Information Professionals and Strategy for Libraries”, IFLA
Journal, No. 33, pp 50-56.
Yeo, R.K. (2008) “Brewing Service Quality in Higher Education: Characteristics of Ingredients that make up the
Recipe”, Quality Assurance in Education, Vol. 16, No. 3, pp 266-286.
160

Generating a new Interview Method
Yayoi Hirose 1 , Kiyoshi Itao 2 and Tomohiro Umeda 3
1 Japan Science and Technology Agency, Tokyo, Japan
2 Tokyo University of Science, Tokyo, Japan
3 Nara Women’s University, Nara, Japan
hirose@sen.jst.go.jp
itao@rs.kagu.tus.ac.jp
umeda@cc.nara-wu.ac.jp
Abstract: This study aims at generating a new interview method of qualitative research, appropriate for
understanding social interactions in a super-aging society. Several countries are now faced with an aging
problem. In particular, the Japanese population took a downward turn in 2005, and Japanese society is now
comprised of 21 percent of over 65 year-old people, which Japan has never experienced. The working population
of Japanese society is expected to decrease rapidly, and society will be largely occupied by elderly people in the
near future. Discussion is now being held on how to generate a sustainable super-aging society. Due to this
situation, the national government initiated public nursing care insurance ten years ago, and a large number of
senior businesses have been generated, including elderly care services, hospices, nursing home services, and
home help services. However, the care business for the elderly has the severe problem that a lot of care workers
quit their jobs because of psychological stress. One of the reasons is that care workers are not motivated to
provide better care services because they cannot interact with the care receivers very well. For example, care
workers cannot understand whether the elderly really enjoy their current circumstances. The elderly who can
express their intentions can enjoy their care services, such as listening to music. However, the care receivers may
become unwilling or hesitant to make requests of those providing care, as they do not want to be thought of as
troublesome or a nuisance. For example, the care receivers sometimes may not feel comfortable saying, “I want to
go to bathroom. Please help me.” In order to investigate this factor, interview or survey is regarded as an important
method. However, several senior people cannot necessarily express their own opinions, and especially senior
people who require nursing care might have difficulty in speaking. Several helpers state that they are concerned
whether senior people are really satisfied with the present situation: They might be stressed by being watched via
the Internet all the day. This study aims at generating new research methods for obtaining more detailed
information regarding human emotional factors by utilizing sensor technology. It can improve the weaknesses of
qualitative research past discussions have pointed out, and develop the validity of collected data and more
objective analysis of collected data in qualitative research. As the first step for a new research method, the study
uses two types of sensing device. These can be used as a complimentary or an additional source of interview or
survey data, as the data obtained by the sensor technology sometimes suggests in what emotional condition the
senior people stated certain things. By obtaining the emotional condition data of senior citizens, it becomes
possible to generate business which senior citizens really want. Exploring the emotions of those in care in detail
will connect the emotional gap between care receivers and care workers, and promote communications between
each group. By focusing on the service to what senior citizens want, plenty of sustainable senior business can be
expected.
Keywords: interview method, qualitative research, sensing technology, emotional factor
1. Introduction
The care business for the elderly suffers the severe problem that a lot of care workers quit their jobs
because of psychological stress. It is a huge problem for Japan, which expects to become a
super-aging society in the near future. As the aging population grows rapidly, the short-handedness of
care workers will also grow into a serious problem. Japanese society now needs to deal with this
problem. The survey says that one of the main reasons that care providers feel stressed is that they
cannot understand whether they are providing appropriate care services to care receivers and whether
care receivers are really delighted by their services (Care Work Foundation, 2005). While Veterans
take a seat-of-the-pants approach, a lot of care providers of several years experience still wonder what
to do. On the other hand, as a Japanese sociologists points out based on her research (Ueno, 2011),
care receivers may also hesitate to make requests of care workers. In other words, the care service
needs to solve both sides of the misunderstandings and discrepancies, and promote human
integration.
161

2. Research method
2.1 Analyzing emotional factors
Yayoi Hirose, Kiyoshi Itao and Tomohiro Umeda
In providing elderly care services, understanding human emotional factors is paramount. There is a
research tool for measuring human emotions, such as AdSAM. However, this tool is for marketing
research, and mainly for measuring a large range of human emotions (AdSAM , 2010). In order to
analyze human emotional factors, researchers mainly use written documents. For example, AsSAM
can be used when analyzing whether targeted consumers make purchasing decisions and how they
feel when they see an advertisement. On the other hand, nursing care for the elderly requires
one-to-one based care and offers the elderly emotional support.
Zarit (1980, 1990) creates a specific scale for a care provider’s burden, “Zarit Caregiver Burden
Interview: ZBI”, which includes physical, psychological, and economical burdens of care providers
brought by supplying care services. This scale is composed of 22 questions regarding the burden of
care services, and questionees choose one of the five scales. It is useful to clarify what kind of burden
a care provider has and what factor is the most severe for care providers. However, in the super-aging
society, interaction between care providers and care receivers is paramount (Ueno, 2011), and it is
necessary to understand when and how people feel at a specific situation, such as during maintaining
hygiene and meal time. This will bring about a more sustainable aging society, where both understand
each other’s feelings and are motivated to communicate.
Thus, in order to provide sustainable care services, it is necessary to collect data on how people feel in
various situations. One-to-one based care based on specific situations requires consideration of that
what senior people feel is different depending on individual experiences and emotional conditions,
which are socially and historically constructed through experience. For example, it depends on the
social value they belong to whether senior people can easily express what they want. In some
societies which value low-assertiveness, senior people may not be encouraged to express their needs
for care services (Hartog, 2004). They may feel stressed to listen to music which they are not
interested in every day and is different from their conventional life style.
Past literature claims that the qualitative approach is appropriate for an inquirer involved in sustained
and intensive experience with participants, as it focuses on participants’ perceptions and experiences
by using multiple methods that are interactive and humanistic (Creswell, 2003; Fraenkel & Wallen,
1990; Merriam, 1998; Thomas, et al., 2005). In the process of qualitative research, several new
aspects emerge by interpreting and analyzing data. This is because qualitative research develops a
description of an individual or setting, and interprets and concludes personal or theoretical new
meaning (Bogdan & Biklen, 2003; Thomas, et at., 2005). Thus, qualitative research requires the
interpretation of the settings from multiple perspectives. Based on these arguments, in order for care
providers to consider and sometimes create new services favorable for various types of elderly person,
the qualitative approach is appropriate as it clarifies favorable care services for each elderly person.
In order to conduct qualitative research, the researchers collect data based on open-ended
observations, interviews, documents, which include e-mails, scrap books, and audiovisual materials,
including photographs, videotapes, computer software and film (Creswell, 2003). Among these data,
qualitative researchers have often used written documents of interview data when they analyze when
and how people think (Akiyama, et. al., 2006). When analyzing the interview data, qualitative
researchers analyze the documents line by line, following the approach outlined by Strauss and Corbin
(1998) (Easterby-Smith, 2008). Even if the voice of interviewees is recorded and researchers analyze
the emotional state of the interviewees, analysis was based only on written records (Remenyi, 1998).
Past discussions point out there is a bias the qualitative researchers bring to the study (Creswell, 2003;
Remenyi, 1998). The analysis of written accounts is not enough to understand interviewee’s subjective
experience, based on the argument that qualitative researchers seek to understand the context or
setting of the participants through “visiting this context and gathering information personally” (Remenyi,
1998). For elderly care, researchers need to consider people’s contextual differences, as some of the
elderly may hesitate to clearly request what they really want to do. Sometimes, even if they do not
enjoy care services, they may say, “I am enjoying this service.” The problem is when a care receiver
makes this statement, the interviewer cannot make the assessment that they are not as happy as they
maintain. Therefore, researchers need to seek more multidisciplinary research methods in order to
162

Yayoi Hirose, Kiyoshi Itao and Tomohiro Umeda
objectively analyze what people really think based on a multilateral approach. This problem may affect
the following interpretive processes where the researchers aim at understanding why certain things
occur. In order to validate the accuracy, qualitative researchers have used member-checking and
triangulation of different data sources. However, even these methods cannot help avoiding subjective
biases completely. More objective methods based on scientific research can provide further validation
of accuracy.
2.2 Digital tool
Qualitative researchers have been discussing the utilization of various types of technological devices.
Brown (2002) introduces multimedia databases which can capture, store and retrieve digital data.
Several articles point out Maxqda, ATLAS-ti and NUD*IST as tools for digital data analysis
(Easterby-Smith, 2008; Lewins, et al., 2007). Knoblauch et al. (2008) discuss how the development of
information technology enables researchers to generate new visual research methods, including
interpretative analysis of video and photography.
However, there is little discussion among qualitative researchers regarding analysis of human
emotional and physical condition by technological devices. Although audiovisual materials are
convenient for researchers to directly understand their reality, problems have been pointed out, such
as difficulty of interpretation (Creswell, 2003). This study aims at analyzing human emotional factors by
adopting two technological devices. Both of them use advanced sensing technology and aim at
analyzing emotional condition by collecting data inside the human body, such as human voice and
heart beat during the interviews. The devices are developed based on scientific research. Thus,
assessment of emotional condition is based on more objective data rather than a researcher’s
assessment, which is based on the researchers’ skill.
3. Research method and analysis
Based on these arguments, this study aims at analyzing “how they felt?” more objectively by
conducting interviews. The study focuses on the stages of collecting data and interpreting the collected
data, and is just the first stage of generating new interview and analytical method. The study may
enable researchers to offset the weakness of conventional qualitative researchers, which are pointed
out in past discussions (Merriam, 1998), as it can enable the collection of more valid interview data and
analyze interview data in a more objective way. In this study, researchers compare the interviewee’s
statement with their emotional conditions and support the information which written documents cannot
assess. Thus, the study selects the interview data of general people who can express what they feel,
as it is impossible to compare the interviewee’s statement with their emotional factors of interviewees
who cannot express their opinions.
3.1 Sensibility technology ST emotion (ST)
The first device is a product of AGI Japan Inc. “Sensibility Technology ST Emotion”, voice analysis
software and system of emotion estimation, which measures emotional parameters in speech. This
software assesses what emotional factor is prominent, anger, joy, sorrow, and calmness when a
person states a certain thing. The software uses power, fundamental frequency (F0), and their
transition as well as speed and intonation of speech to analyze the voice, as F0 and power refer to the
vibration of vocal codes and may be an expression of natural human emotion (Mitsuyoshi et al., 2007).
The system detects changes of feeling or emotion that a person does not notice, and it can detect
more than 70% of human emotion (Tokuno et. al, 2011). The study analyzes interview data using ST
technology and compares it with the analysis based on written documents of interviewee’s statements.
The researchers obtained guidelines from AGI for interpreting the outcome of the ST analysis. Based
on Mitsuyoshi (2008) and the guidelines, the researchers interpreted the interview data.
3.1.1 Analysis based on coding
The researchers selected three interviewees. The first person is a male care worker, with a wife and 8
month old girl. The second person is his wife, who used to be a care worker. She quitted her job after
marriage and childbirth. The third person is a female care worker, with a husband and a child. They
were interviewed about how they feel about their present life.
On analysis of how a care provider and his wife think about their present situation, one theme emerged
for each person,
163

1. Satisfied with his present life:interviewee 1
Yayoi Hirose, Kiyoshi Itao and Tomohiro Umeda
1-1. Satisfied with their daughter’s smile.
Her smile is really great. I like both the smile of the elderly and children (Interviewee 1)
1-2. Satisfied with the elderly smiles.
I like the elderly’s smile. I see their smiles a number of times so I cannot quit my job. I like
their smile. (Interviewee 1)
2. Anxious about the future: interviewee 2
2-1 concerned about their daughter in the future: Interviewee 2
I am anxious about whether I will be able to do financially for my daughter what my
parents did for me. (Interviewee 2)
3. Satisfied with her present life: Interviewee 3
3-1: Satisfied with her job
I like the elderly. (Interviewee 3)
I enjoy my work. I think this is my true vocation, because I enjoy my work.
(Interviewee 3)
My boss asked me, “Do you like your job?” and I said, “Yes. I like it.” (Interivewee 3).
Based on this initial coding, a researcher may think that interviewee 1 is satisfied with his present
situation as he can look at his daughter’s and elderly people’s smiles, while his wife is financially
concerned about her daughter because of her husband’s low income. Interviewee 3 is satisfied with
her present situation as she likes interaction with the elderly.
3.1.2 Analysis based on the ST technology
On the other hand, the ST displays the two people’s feelings in more detail. Although interviewee 2
certainly talked about her concern about her daughter in the future, her emotional condition was
“CALM” during this statement. Rather, she clearly presented “SORROW” when she explained about
their financial situation.
Compared with people who graduated from university and work at a business office, in
our work, care providers must work much harder.(CALM) But our income is too low.
(SORROW).
We do not have discretionary money after childbirth. (SORROW). Our common hobby is
bringing up our child. (CALM)
The ST can propose different messages from the analysis of interview documents. Based on the
analysis of the written interview documents, a researcher cannot understand whether the interviewee 2
just told the truth about her financial situation, or is unsatisfied with her present life. On the other hand,
the ST suggests that his wife is more disappointed about the situation due to her husband’s low salary,
as she does not have discretionary money. She certainly may have strong feelings about the low
income of care providers despite their hard work. Tokuno et. al. (2011) show that strong stress
increases the feeling of “sorrow” and decreases the feeling of “joy” Therefore, based on the ST
analysis, it can be said that she may be stressed with her present financial situation rather than her
daughter’s future. On the other hand, interviewee 1 did not display “SORROW” during the interview,
although he explained about his low salary. This implies that while his wife is highly concerned about
her husband’s low salary, he is not so disappointed and takes it coolly.
The ST also has the possibility to clarify the difference in human satisfaction between Interviewee 1
and Interviewee 3. Interviewee 1 consistently presented “CALM” whatever he was asked. Even when
he explained about his job and his favorite elderly smile, his emotional condition was calm. This implies
that he is very relaxed with the smile of the elderly and his daughter. The main reason that he
continues to be a care worker was that he can be relaxed with their smile.
164

Yayoi Hirose, Kiyoshi Itao and Tomohiro Umeda
Interviewee 3 shows more “JOY” for the elderly. In particular, she felt delighted when she talked about
how she decided on her job. She explained that as a high school student, she wanted to be a children’s
nurse, and changed her mind when she experienced voluntary practical training for elderly care.
During this talk, she displayed JOY to a certain degree. After that, when she stated “I also like the
elderly,” she displayed a stronger joyful condition. This suggests that she may really like the elderly
and her job. Furthermore, she displayed “JOY” when she stated, “I realized that there are many
misunderstandings about my job,” and talked about the process of how she attained her work-life
balance from the initial biased idea that it is impossible to balance both. Based on analysis of
documents, it is not clear whether she is satisfied with her working style, while the ST displays she is
extremely satisfied with her work-life balance. This may imply that the main reason that she continues
her job is that she is more motivated by interaction with the elderly under a work-life balance.
Table 1 summarizes the analysis of the three interviewees.
Table 1: summary of analytical difference between coding and ST
Analysis based on statement Analysis based on the ST
device
Interviewee
1
Interviewee
2
Interviewee
3
Satisfied with his present life
I like the elderly’s smile. I
see their smiles numbers of
times so I cannot quit my
job. I like their smile.
Concerned about her
daughter in the future.
I am anxious about whether I
will be able to do financially
for my daughter what my
parents did for me.
Satisfied with her present
life, as she likes interaction
with the elderly.
I think this is my true
vocation, because I enjoy
my work.
Neutral with his low salary.
We have not been
purchasing clothes (CALM).
Stressed with the husband’s
low salary.
We do not have discretionary
money after childbirth.
(SORROW).
Feel joyful to do her job.
I also like the elderly. (as well
as children)(JOY)
I realized I was being biased
towards my job. (JOY)
Final analysis
He is satisfied with his
present life, and not so
stressed with his family’s
financial situation.
She is not satisfied with her
present life. She is stressed
with her husband’s low
salary and concerned about
her daughter.
She is satisfied with her
present life, as she enjoys
her job and her present
work-life balance.
3.2 WHS-1
The data shows that the ST software sometimes includes some errors. For example, when interviewee
1 looked at his daughter, the ST interpreted his sigh as “ANGER” and then his laughter as “JOY”.
Based on his comments and smile, it is obvious that he feels JOY when he looks at his daughter. The
ST also sometimes mistakenly interprets “STRESS” for a joyful condition, when people laugh in a loud
voice. Thus, the ST cannot analyze all of the voice data. In order to analyze emotional condition during
talking more precisely, this study adopts the second device, WHS-1, which is a small-sized portable
sensing device. It measures human biological information, such as heartbeat, which includes wave
shape and cycle of human heartbeat, temperature of human body surface, and triaxial rate of
acceleration, and monitors a human biological condition based on specific analysis by PC software.
This sensing device is produced based on the past research which shows that LF/HF, Low Frequency /
High Frequency, conducted by spectral analysis of heart beat variability, is high when sympathetic
nerves activate (Takada, 2005). As this device mainly clarifies a general trend of emotional condition
over several hours, while the ST assesses the emotional condition phrase by phrase, it has been
mainly utilized for analysis of severe depression. Therefore it is necessary to clarify to what a degree
this device shows emotional condition over a short period of time, such as 2-3 minutes. In order to
assess the usefulness of this device as a complimentary source of interview data, the researchers
measured a person’s heart beat and worked out LF/HF figures during the conversation and analyzed
human emotional conditions. The examinee is female, 23 years old. She wore the sensing device, and
talked with the researcher. The researcher sometimes had a very pleasant conversation with her, such
as her hobbies and sometimes strongly ordered her to do so much work within a very short time.
165

Yayoi Hirose, Kiyoshi Itao and Tomohiro Umeda
Table 2 shows the difference of LF/HF figure among her calm condition, relaxed talking and stressful
talking. It shows each average figure during a temporal sequence of three minutes. The comparison
shows that the LF/HF figure is relatively higher at the stressful condition than relaxed talking and calm
condition. During the calm condition, the LF/HF is lowest. The data shows that when the examinee
enjoyed relaxed talking, she was in a certain level of excited condition. As the emotional trend is
different between the three conditions even within 3 minutes, it can be useful to complement the
analysis of interview documents.
Table 2: Comparison of LF/HF figure between calm, relaxed, and stressful conditions
Calm condition Relaxed talking Stressful talking
average 3.36 6.56 8.52
Standard deviation 1.23 2.95 3.73
3.3 Mixed analysis
Finally, the researchers conducted an interview regarding when a care provider felt stressed and
relaxed and analyzed the interview by coding, the ST device, and WHS-1 device. The interviewee is a
female, 43 years old, used to be a care provider. The researcher asked her to look back at her job
experience and explain about her stressful and relaxed experiences. Based on the initial coding,
several codes which present stressful occasions and relaxed occasions came out. The researchers
compare these codes with the ST results and WHS-1 results. Based on these results, the ST and
WHS-1 well support stressful factors based on the coding. For each code, the result of the ST analysis
will be explained in the following sections based on AGI’s guidelines and Mitsuyoshi (2008). Transition
of LF/HF measured by WHS-1 during the interview is shown in Figure 1, while it is impossible to show
the transition of all figures of the ST results due to space limitation. Figure 2 shows one part of the ST
result. By and large, the ST displays “ANGRY” very frequently. This implies that the interviewee was
very stressed about her experience as a care worker. During the interview, she looked back at her
experience and presented stress, complaints and anger. On the other hand, the ST rarely displayed
CALM while she was talking, while WHS-1 sometimes displayed a relaxed condition of the interviewee
(Figure 1). The following explains several stressful and relaxing factors based on three types of
analysis.
LF/HF
45
40
35
30
25
20
15
10
5
0
14:16:48 14:24:00 14:31:12 14:38:24 14:45:36 14:52:48 15:00:00
Zone 1
Zone 6
Figure 1: Transition of LF/HF during the interview
1. Stressful factors
Zone 7
Zone 2 Zone 3 Zone 4 Zone 5
1-1 Required heavy labor in order to avoid fatal accident despite insufficient physical ability
Giving a care receiver a bath is the heaviest labor among care work, such as standing the
elderly and moving them to the wheelchair. I was scared to think if I mistakenly broke their
clinical instruments, including the tube of respiratory apparatus. I was really scared to give
these care receivers a bath. I was so scared and wondered what to do if I broke the
instruments. I cannot do anything to avoid theses types of accidents as the elderly are so
heavy.
166

Strongest factor
Calm
Anger
Joy
Sorrow
Laugh
Excitement
Yayoi Hirose, Kiyoshi Itao and Tomohiro Umeda
Figure 2: ST result which displays a stressful condition
The first stressful factor that came out during the interview was heavy labor required to avoid a fatal
accident for the elderly. WHS-1 also displayed high LF/HF during this section of over 20 seconds (zone
1 in Figure 1). The second half of Figure 2 shows the ST result of this section. During this section, based
on the ST, she felt angry with high excitement, but her joyful feeling was also sometimes high. Based on
the AGI guideline, it can be said that the interviewee felt joyful as the interviewee listened to her and
understood her difficulty. Considering the interviewee’s statement it implies that she wanted to
emphasize heavy labor, such as giving care receivers a bath.
1-2 Care receivers’ negative reaction to her services
Giving the care receivers who hate taking a bath was really difficult. Some of them strongly refused
and sometimes told lies that they took a bath yesterday. But their bodies smells badly. So I tried
to persuade them to take a bath. But if they completely refused, I reported, “they refused.”
During this section, the ST displayed ANGRY and sometimes JOY with a certain level of excitement.
Considering the interviewee’s statement and AGI guideline, this implies that the speaker may be a little
bit fed up with the situation and wanted the interviewer to understand her hardship. The LF/HF figure is
also extremely high for 25 consecutive seconds (zone 2 in Figure 1). Thus, based on coding, it can be
said that this care worker is stressed with the difficulty of persuading care receivers to do necessary
things. Furthermore, the ST and WHS-1 suggest that she may also be fed up with the elderly.
1-3 human relationship with incompatible colleagues in her work site
Ummm…human relationship in the care home… Yes. There are several troublesome care
providers. I think they started care provider jobs because they cannot be working people. I was fed
up with them.…One of them talked about herself that she used to resist going to school in a meeting
room where there were a lot of complete strangers existed.
During this section, the ST displayed that the speaker felt ANGER and SORROW. This implies that the
speaker sometimes felt angry and sometimes got bothered by the situation. The LF/HF figure was also
extremely high and this condition continued over one minutes while talking about this topic (zone 3 in
Figure 1)
1-4 Need to take a responsibility for the accident that could not be avoided
Even if I just observed the situation where care receiver exercised and fell down, I needed to take
responsibility, and was required to write a report about why this happened. I was required to watch
the care receiver at that time. But my colleague stood by me and said “Ah it cannot be avoided.”
Based on the analysis of the interview document, the next stressful factor for her can be necessity to
take responsibility for an accident which could not be avoided. During this section, the ST displayed that
the speaker mainly talked with ANGER, and sometimes also mixed with JOY. Based on the AGI
analysis, this may imply that the speaker talked with anger and self-assertiveness. The WHS-1 shows
that as LF/HF is very high and the high level continued for over 90 seconds (zone 4 in Figure 1), she
may feel stressed.
167

Yayoi Hirose, Kiyoshi Itao and Tomohiro Umeda
1-5 not understanding care receiver’s physical condition
I was disappointed when a care receiver discharged his bowels in the bath. I needed to restart the
whole process of giving a bath from scratch, and washing the bath was also heavy work. I felt if they
gave me even a small sign. Some of the elderly did this as a joke….I do not think they were
malicious…ummm.
During this section, based on the ST, the care worker felt angry and subsequently sorrowful. The
LF/HF figure was also high and the high level continued for one minute (zone 5 in Figure 1) This
implies that the care worker felt angry that the care receiver did what the care worker did not expect,
but was stumped about not understanding the care receiver’s condition, which brought about this
situation.
2. Relaxing factors
2-1 communication with the care receivers
I was happy when I interacted and communicated with the elderly. The most delightful thing was that
I was sometimes able to communicate with the care receivers, even if they suffered Alzheimer’s
disease. As they suffered from the disease, they sometimes forgot what they said just after
speaking. But sometimes, I heard what I did not know, like a war story.
During this section, LF/HF remained at a low level for one minute (zone 6 in Figure 1). This means that
she was relaxed during talking about her delightful things. On the other hand, the ST suggests that she
felt angry and joyful. During this section, she imitated the care receiver’s voice or colleague’s voice.
Thus, this imitation may sometimes be interpreted as ANGRY.
2-2 human relationship with compatible colleagues in her work site
When our care center held a barbeque party, we saw a lot of beef on our table, even if the elderly
would not eat beef. But after the party, the beef was gone…(laugh) Who ate it! Maybe the male care
providers (Laugh)
During this section, the ST suggests that she sometimes felt JOY and sometimes ANGRY. But it was
obvious for the interviewer that she talked about joyful experiences at her work site. Thus, it can be said
that her laugh was interpreted as ANGRY. It was confirmed by the low LF/HF figure for one minute
(zone 7 in Figure 1).
4. Conclusion
This study discussed the possibility of analyzing emotional factors of interviewees by using sensing
devices. In particular, the researchers focused on the stages of collecting data and analyzing interview
data and discussed the possibility of generating new methods of collecting data based on higher levels
of validity and interpreting human emotional conditions more objectively. The study suggests that
utilizing the sensing devices enables more objective analysis regarding interviewee’s emotional
conditions, while conventional methods cannot avoid a certain degree of researcher’s subjective bias.
The result of the mixed analysis is summarized in Table 2. The results suggest that using sensing
devices enables researchers to obtain a closer view of the reality of the interviewees. For example, ST
sometimes displays that the interviewee stated her stressful work situation self-assertively, such as a
need to take responsibility for care receiver’s accidents. This enables the researchers to think about
why this interviewee became self-assertive. She may not have any friends or family who listen to and
understand her situation, and this may further aggravate her feelings of stress towards her job.
Table 2: Summary of the mixed analysis
Initial codes ST analysis WHS-1 analysis
1-1 Stressed with heavy labor Wanted to complain about the
Highly stressful
required
heavy labor to the interviewer
1-2 Stressed with negative Wanted to complain about her
Highly stressful
reaction to her services difficulty and fed up with care
receivers
1-3 Stressed with human
relationship with
incompatible colleagues
Felt angry and got bothered Highly stressful
1-4 Stressed with need to take a Felt angry and sometimes
Highly stressful
responsibility for care
receiver’s accidents
self-assertive
1-5 Stressed with not Felt angry and stumped about Highly stressful
168

Yayoi Hirose, Kiyoshi Itao and Tomohiro Umeda
understanding care
receivers’ physical condition
Initial codes ST analysis WHS-1 analysis
2-1 Communication with care
Includes some errors Highly relaxed
receivers
2-2 Human relationship with
compatible colleagues
Includes some errors Highly relaxed
It can be said that generally the two types of sensing device can play a supportive role in analyzing
emotional factors for interview research. Using these devices can contribute to the development of
qualitative research in terms of strengthening validity of collected data and more objective data
analysis. However, considering the several errors, further related discussions and detailed data are
necessary for future research.
Acknowledgements
We would like to thank Dr. Mitsuyoshi, the president of AGI, whose comments were of great value for
our study.
References
AdSAM (2010) “Sample Survey Results Summary”, (online), Presented at American Marketing Association’s 2010
Marketing Research Conference, Atlanta.
http://www.adsam.com/about/adsams-survey-on-the-american-marketing-assocationss-2010-marketing-res
earch-conference (accessed on 17/01/2012)
Akiyama, M., Kayama, M., Takamura, S.(2006) “A Study of the Burden of Caring for Patients with Amyotrophic
Lateral Sclerosis (MND) in Japan”, British Journal of Neuroscience Nursing, 2(1), pp38-43.
Bogdan, R. C & Biklen, S. K. (2003). Qualitative Research for Education: An introduction to Theories and Methods
4th edn.. New York: Pearson Education group.
Care Work Foundation (2005) A Survey Report Regarding Care Worker’s Stress.
Creswell, J.W.(2003) Research Design: Qualitative, Quantitative, and Mixed Method Approaches, second edition,
Sage Publications, Inc.
Easterbay-Smith, M., Thorpe, R. and Jackson, P.R. (2008) Management Research, 3 rd Eds Thousand Oaks,
CA:Sage.
FraenkelJ., and Wallen, N. (1990) How to design and evaluate research in education, McGraw-Hill Pub. Co. New
York.
Hartog, D.N.D. (2004) “Assertiveness”. In House, R.J., Hanges, P.J., Javidan,M.,Dorfman, P.W., and Gupta, V.
(Eds.) Culture, Leadership, and Organizations: The GLOBE study of 62 countries, SAGE Publications.
395-436.
Knoblauch, H., Baer, A., Laurier, E., Petschke, S. and Schnettler, B. (2008) “Visual Analysis. New Developments
in the Interpretative Analysis of Video and Photography”, FORUM: QUALITATIVE SOCIAL RESEARCH,
Volume 9, No. 3, Art. 14, September.
Lewins, A., and Silver, C. (2007) Using Software in Qualitative Research: A Step-by-Step Guide, London: SAGE
Publications.
Locke, L.F., Spirduso, W.W., and Silverman, S.J. (1987) Proposals that work: A Guide for planning dissertations
and grant proposals 4 th edision Thousand Oaks, CA Sage publications Inc.
Merriam, S.B. (1998) Qualitative research and case study applications in education. San Francisco Jossey-Bass.
Mitsuyoshi, S. Lecture material regarding the ST at the University of Tokyo University (Japanese)
(http://www.agi-web.co.jp/docs/Univ-Tokyo.pdf)
Mitsuyoshi, S.(2008) , “Generating Virtual Self and Emotion by using Sensibility Technology”, in Fukuda,S., and
Watanuki, K (eds) Sense, Emotion and Robot: Challenge to man-machine interaction, Kogyo Chosakai,
Tokyo.
Remenyi , D., Williams, B., Money, A., and Swartz, E (1998). Doing Research in Business and Management: An
Introduction to Process and Method, SAGE publications.
Strauss, A.L. and Corbin, J. (1998) Basics of Qualitative Research: Techniques and Procedures for Developing
Grounded Theory, 2nd edn. Thousand Oaks, CA:Sage.
Takada, Y., Takada, M, and Kanayama, A (2005) “TheSignificanceof LF component and HF component which
resulted fom Frequency Analysisv of Heart Rate and the Coefficient of the Heart Rate Variability-Evaluation
of Autonomic Nerve Function by Acceleralion Plesysmography”, Health Evaluation and Promotion, 32 (6)
Thomas, J., Nelson, J., and Silverman, S. (2005) Research Methods in Physical Activity, 5th edn. Human Kinetics
Publishing.
Tokuno, S., Tsumatori, G., Shono, S., Takei, E., Yamamoto, T., Suzuki, G, Mitsuyoshi, S., and Shimura, M.(2011).
“Usage of Emotion Recognition in Military Health Care”, DSR/IEEE Defence, Science & Research
Conference , 2011.
Ueno, C.(2011) Sociology of Care: Welfare Society based on Rights of the Party, Tokyo, Ohta Publications
WIN Human Recorder Home Page (http://www.winhr.co.jp/)
Zarit, S.H. (1980) “Relatives of the impaired elderly: Correlates of feelings of burden”, Gelontologist, 20, 649-655.
169

Yayoi Hirose, Kiyoshi Itao and Tomohiro Umeda
Zarit, S.H. (1990) “The memory and behaviour problems checklist 1987R and the Burden Interview”. (Technical
Report) Pennsylvania State University Gerontology Center: University Park PA.
170

The new Role of Marketing Research Toward
Sustainability: A Value-Driven Perspective
Somdee Hongphisanvivat
Faculty of Commerce & Accountancy, Thammasat University, Bangkok,
Thailand
somdee@tu.ac.th
somdeejtu@yahoo.com
Abstract: The increasingly turbulent environment calls for a new role for marketing research to employ the
holistic research approach and shift its focus on abilities to learn key knowledge for creating superior value and
sustainability. This paper addresses such holistic views using multiple qualitative methods to obtain wider and
deeper knowledge/information from professional experts, academicians and executives of various industries
experiencing in marketing areas. The findings support the proposed framework revealing the new marketing
research requires three significant changes from the original conception of market research: 1) incorporating
experience and active collaboration between researchers and decision makers into research process to set the
key knowledge most critical to sustainable advantages; 2) employing multiple research methods most useful and
applicable to build capabilities in superior learning key knowledge; and 3) enriching interpretation for insightful
understanding and facilitating superior decision and proper action. Further study illustrating a case study of the
most successful public company in Thailand retail trade industry gives more insights on how to apply holistic
perspective. Following the new research approach, the company employs seven forces focusing on periodical
study of consumers and employees, improved information technology and data management, and continually
building good relationship and strong collaboration with all stakeholders. These assist in developing core
capabilities and dynamic capabilities to faster identify key knowledge, better learning for in-depth understanding
and proper decision on an effective implementation of superior holistic value creation of stakeholders. The
company’s net profits margin, return on equity (ROE) and return on investment (ROI) outperformed the industry
average during the year of 2004-2009 and expected to continue in 2010-2012. These evidences confirm the well
applied holistic orientation able to sustainable competitiveness and long-term business success.
Keywords: new marketing research, holistic value creation, competitiveness, sustainability, business success
1. Introduction
Marketing research has been a critical tool for helping management make decisions and explore new
opportunities (McDaniel and Gates 2010). In particular, market research provides market knowledge
essential to develop marketing strategy (Smith and Fletcher 2004) and implement market orientation
for creating superior customer value to enhance business performance (Slater and Narver 1994).
However, this is insufficient for sustainable development.
First of all, there is a fundamental shift of effective marketing from market-driven into value-driven
market as the next source of sustained competitive advantages (Woodruff 1997) and continued
success (Best 2009). In the highly turbulent economy, core knowledge/capabilities of all related
information are required as a primary source of creating superior value and sustainability (e.g. Maria
and Marti 2004). Eventually, firms’ resources and capabilities tend to offer a more durable strategy
than the market-based orientation for company success in the long-run (Porter 1985, 1996).
Therefore, marketing research has to concentrate on developing knowledge/capabilities for unique
value-driven creation.
Second, there is a need to broaden the theoretical domain of value-driven based on market
orientation to holistic orientation. Market orientation alone is inadequate to create superior customer
value toward sustainability when dealing with dynamics (Grunert et al. 2005). In contrast, firm
capabilities, especially those of management and employees are increasingly critical to sustained
advantages and organization success as their abilities affect customer value (e.g. Mittal and Sheth
2001). Moreover, the empowered society and ecological imperatives force firms to take a broader
view of value creation for various stakeholders to achieve sustainability (Berthon 2010; Maigman et al.
2005) Hence, marketing research has to enhance knowledge/capabilities driving stakeholder value.
Third, creating superior value successfully requires an understanding of the interrelation between the
firm (value-in-offering) and the customer or other important stakeholders (value-in-use) in the value
creation process (Ngo and O’Cass 2010). Thus, marketing research has to shift its focus on valuedriven
decision research since findings able to improve decision in continually creating and delivering
171

Somdee Hongphisanvivat
superior value are central to business success (Carson et al. 2005; Ganeshasundaram and Henley
2007).
Finally, the significant changes in marketing management over the past three decades urge
managers to pursue more duties, particularly on relationship marketing, value creation, knowledge
management and social responsibility (Gok and Hacioglu 2010). This fosters marketing research to
employ a new holistic approach to provide the broader, richer and fuller alternatives of information for
facilitating decision to develop strategic value-driven sustainability (Belz and Peattie 2011; Hamlin
2000; Smith and Fletcher 2004).
Addressing the various needs of new holistic marketing research, this paper attempts to explore three
important gaps: first, examine the new research approach toward value-driven sustainability. Second,
investigate the background and propose the conceptual framework of the new research. Finally,
present and discuss the empirical results followed by managerial implications and future research
recommendations.
2. The new research approach toward value-driven sustainability
The new research approach referred to the ability to provide and learn key knowledge for superior
management decision and proper action is a new source of advantage and business success. This
new approach is emerged due to the high dynamics addressing the need of new marketing
management toward value creation as the heart of sustainable advantages (Gok and Haioglu 2010;
Ngo and O’Cass 2010). Evidence indicates market leaders across diverse industrial sectors from high
technology at Dell Computers to grocery and household products at Tesco stores have outperformed
because of enhancing knowledge faster and better than the competitors. Further, sustaining
superiority in learning through continually searching for new insights and better understanding the key
driver of superior value creation, in particular customer value has proved to be one of the most critical
corporate competencies (Smith and Fletcher 2004).
However, increasing competencies through superior knowledge requires far more than traditional
research. The holistic approach is needed (see Figure 1) to build insight understanding, sensing and
anticipating the rapid change as well as value innovation for sustainable superior performance.
Building capabilities in superior
leaning key knowledge
Identifying key knowledge/
marketing intelligence
Value-Driven
Sustainability
Increasing superior decision
and proper action
Figure 1: The new role of marketing research toward value-driven sustainability
Figure 1 highlights the three critical steps of the new marketing research to achieve value-driven
sustainability:
1) Identifying key knowledge or marketing intelligence most relevant to sustainable advantages and
business success through experience base aligning with the active collaboration between researchers
and decision markers to set the right target of key knowledge and reporting. This will ensure the
research contribution and practical consequence (Carson et al. 2005; Carrion et al, 2004; Hamlin
2000). Moreover, identify and prioritize key knowledge increase powerful information and decrease
complexity in decision (Ganeshasundaram and Henley 2007; Smith and Fletcher 2004). For instance,
the key knowledge of value-driven sustainability mostly involves the prominent principles exploring the
key driver to superior value creation, such as market orientation (Narver and Slater 1990), strategic
resource orientation (Barney 1991), and ecological orientation (Sheth and Parvatiyar 1995).
172

Somdee Hongphisanvivat
Meanwhile building competitive advantages primarily emphasize unique differentiation and/or cost
leadership (Porter 1985).
2) Building capabilities in superior learning key knowledge through employing multiple research
methods and/or integrating qualitative and quantitative research most useful and applicable for action
research. The dynamic economy results in demanding more insight information to in-depth
understanding of the complex phenomena. This leads to a need of employing holistic research
approach to fill the gap of imperfect data and required richer context to which interpretative research
and multiple qualitative methods become important (Carson et al. 2005; Smith and Fletcher 2004). As
such, marketing research has to shift its focus on developing capabilities in learning to utilize key
knowledge faster and better than competitors to achieve value-driven sustainability.
3) Facilitating superior decision and proper action through multiple interpretations of key knowledge
for insightful understanding and effective implementation of value-driven sustainability. It is, thus,
necessary to incorporating management intuition and the researchers’ skills into the data analysis for
enhancing the interpretation power. The engagement of researchers with the end decision makers is
also needed for better handling and gaining more meaningful information (Carson et al. 2005; Hamlin
2000; Smith and Fletcher 2004). This reflects the greater imperative of qualitative research to
marketing management decision and application.
3. Background and conceptual framework of the new research toward valuedriven
sustainability
3.1 Background
There is no single principle able to explain completely the new role of marketing research focusing on
holistic perspective to develop and utilize key knowledge for creating value-driven sustainability.
Intensive reviewed literature suggests at least three principles. Marketing orientation and strategic
resource orientation are used as the foundation concepts while ecological orientation is added to be
adequate for sustainable development.
3.1.1 Market orientation toward value-driven sustainability
Market orientation has been an important concept of developing superior customer value for
achieving superior performance. Applying this construct successfully, it is necessary to well manage
the market information which remains inadequately assessed (Foley and Fahy 2009). This
corresponds to Kohli and Jaworski (1990) placing emphasis on implementing market oriented through
generating, disseminating and responding to market information effectively. Several studies indicate
market orientation has positive relationship with profitability and sustainable advantages when
capabilities are taken into account (e.g. Subramanian and Stranddholm 2009; Vorhies and Morgan
2005). Accordingly, market orientation in the new research perspective has to emphasize the abilities
to learn and use the key knowledge about consumers and competitors to guide for better creating
customer value.
Benchmarking as a market-based learning process from best practice also provides key driver
sources of superiority, novelty and interdependency among marketing capabilities which help
managers to plan and control their impact on marketing system and continued improvement (Camp
1995). Therefore, strong market capabilities based on sound benchmarking information and learning
result in effective market orientation to superior customer value and sustainable advantages (Slater
and Narver 1995; Vorhies and Morgan 2005).
3.1.2 Strategic resource orientation toward value-driven sustainability
The resource-based view focusing on distinctive resources difficult to imitate and non-substitutable is
key driver of sustained advantages and organizational success (Barney 1991). In particular, core
knowledge/capabilities are the most critical to formulate and implement sound strategy for super value
creation (Maria and Marti 2004; Porter 1996). Further, intangible resources tend to offer a more
durable strategic basis, value creation and firm’s performance than the market orientation (e.g.
Sullivan 2000). Specifically, the capabilities of management and employees are the most important
strategic resources as they appear to be the heart of wealth creation and profit generation to longterm
sustainable advantages and success (Pfeffer 1995; Sharkie 2003).
173

Somdee Hongphisanvivat
Therefore, human resource management has to adopt the new focus on competitive advantages by
increasing human knowledge and skills to effectively use the superior resources rather than just
possessing them. This corresponds to the activity-based view stressing that the sustainable
advantages depend on how to employ the distinctive resources to build superior value because it is
people crucially creating superiority and sustainability (Porter 1985; Sharkie 2003).
The distinctive resources not only perform a positive relationship with market orientation but create
specific capabilities, in particular the market-sensing and dynamic capabilities. These enable firms to
learn and anticipate the changing market environments including interpreting and capturing the
market insights which assist in creating superior value to sustainable advantages and performance
(Mengue and Ash 2006; Olavarrieta and Friedmann 2007).
Additionally, the company has to employ internal core capabilities aligning with core activities through
well managing information intelligence (Quinn 1992). Intellectual and/or service activities are able to
maintain at ‘best in the world’ to gain long-run advantages while relying on the capabilities of external
suppliers for non-strategic and non-core activities (Maria and Marti 2004). As such, firms need the
cooperative utilization of the resources and capabilities of other companies and institutions to
continued building internal core competencies placing attention to develop the human unique
capacities unobservable advantages. Consequently, marketing research in the new perspective has
to shift its focus on superior learning internal distinctive resources innovatively for effectively creating
stakeholder value to achieve a more sustainable performance (Doz and Hamel 1998; Sharkie 2003).
3.1.3 Ecological orientation toward value-driven sustainability
The emerged ecological imperatives become the most globally critical issue and significantly influence
business survival and success (Smith and Sharicz 2011). Marketing’s role in satisfying consumers
through quality of consumption goods and services is insufficient. It has to take responsibility towards
society and ecological sustainability to serve and save the planet for a better quality of life primarily
through value creation (Kotler, Roberto and Lee 2002).
Further, marketers have to shape consumers to appropriate choices for meeting the current needs
without sacrificing the future needs. At the same time, they have to provide products and services
able to improve the environment quality (Borland and Paliwoda 2011). This socio-ecological concept
is broader than marketing orientation and necessary to enhance the ability of value-driven
management for decreasing the environmental consequences on human life and society. Therefore,
marketing research in the new perspective has to broaden market knowledge to include
socioecological information for guiding a better decision on social responsibilities since companies
able to build sustainable competitiveness together with sustainable ecology will gain a more promising
superior performance and success (Johnson 2009).
However, achieving sustainable ecology requires the integration of economic, environment and social
aspects at multi-level contexts throughout the organization (Belz and Peattie 2011). It needs at least
three different considerations: 1) take a broader stakeholder, especially social groups, 2) employ a
new and better adapted approach focusing on social marketing mix, and 3) better use of resources
and capabilities to develop sound strategies and implement them with respect to best practice
management (Christmann 2000). Accordingly, marketing research has to concentrate on abilities to
better learn and utilize the key socio-ecological knowledge to create superior social value suited to
each context of society’s demand and balanced various stakeholders’ needs. This is critical to
enhance a more promising wealth for society and secure lasting human welfare (Sheth and Parvatiyar
1995).
3.2 Conceptual framework
The above review and theoretical background lead to a proposed conceptual framework as shown in
Figure 2. This framework explores the relationship among holistic research perspective, holistic value
creation and sustainability outcomes. The holistic research perspective is composed of three critical
considerations: identifying key knowledge, building capabilities in superior learning key knowledge,
and facilitating superior decision and proper action. The holistic value creation employs three
principles focusing on market orientation, strategic resource orientation and ecological orientation in
terms of capabilities to learn and use key knowledge effectively for creating appropriate value. These
will result in sustainability outcomes composed of superior stakeholder value regarding unique
174

Somdee Hongphisanvivat
differentiation and/or cost leadership which in turn able to sustainable competitive advantages and
long-run success.
Holistic Research
Perspective
Identifying Key
Knowledge
• Market information
• Resource information
Building Capabilities in
Superior Learning
• Distinctive capabilities
• Dynamic capabilities
Facilitating Superior
Decision & Action
• Value-driven decision
• Value-driven application
Holistic Value
Creation
Market Orientation
• Learn and use key
market knowledge
• Learn and use
benchmarking
Strategic Resource
Orientation
• Learn and use key
resource knowledge
• Learn and use network
Ecological
Orientation
• Learn and use socio-
ecological knowledge
• Learn and use best
ti t
Sustainability
Outcomes
Superior
Stakeholder
Value Creation
• Unique
differentiation
C
Sustainable
Performance
• Sustainable
advantages
• Long-run success
Figure 2: Conceptual framework of the new marketing research toward value-driven sustainability
4. Research methodology and findings
This study attempts to explore the new role of marketing research focusing on holistic perspective
composed of three critical combinations different from normal research which are essential for
developing holistic value creation toward sustainability. The aim of the work is theory building together
with practicality. Therefore, the multiple-qualitative approach is suitable (Carson and Caviello 1996).
4.1 First study: In-depth interview
The project followed the critical steps of the new marketing research. First, reviewed literature
intensively to find out the under studied areas of value creation and identify the key
knowledge/information needed. Second, proposed the initial framework and research methodology
based on experts and academic opinions experiencing in marketing areas to obtain and use key
knowledge to find out what is the meaning, importance, theory/concept, and application of the holistic
approach to value creation between customer value and other important stakeholder value (employee
value, shareholder value and social value) crucial for superiority and sustainability. Third, organized a
national conference regarding holistic value marketing to get comments and suggestions from
professional experts, academicians and businesses specialized in marketing to enhance relevant
knowledge involved with value-driven sustainability. Fourth conducted an in-depth interview using a
special telephone technique through an individual connectivity with 30 executives earning the MBA
degree and more than 10-year working experiences from various industries in Thailand to get insight
understanding for guiding a better development and application of holistic value creation and new
research approach.
The findings of the 26 usable responses accounting for 86.7 per cent indicate most respondents refer
customer value in terms of cost-benefit trade-off similar to the definition of prior research. More than
70 per cent of them agree to use the holistic perspective as a source of advantage and sustainable
superior performance.
The results support the proposed framework addressing the need of new holistic marketing research.
Creating customer value is the most crucial component (accounting for 31.0 % of the total value
creation) followed by employee value, shareholder value and social value representing 24.4%, 22.6%,
175

Somdee Hongphisanvivat
and 22.0% respectively. This reflects Thai businesses realize the imperative influence of other
stakeholders’ value, especially those dominating the high potential for building strategic capabilities,
such as employee and social value. Therefore, firms have to broaden market information and concept
from micro to macro level since the macro environments have been changed faster and more volatile
affecting business survival and success.
However, most respondents tend to misunderstand how to create effective value of multiple
stakeholders. Most ignore to explore the real need of employees and society based on market
orientation and strategic capabilities concept especially when creating social value. This is why most
firms place less emphasis on identify key knowledge about socio-ecology and human resource, the
most critical drivers to long-run sustainable advantages, in order to superior learning and using such
knowledge for building distinctive and dynamic capabilities to achieve better decision on value-driven
sustainability.
The results reveal most Thai managers have less knowledge of building an appropriate social value
despite its incremental role to enhance firm value rather than image. For instance, one of Thailand’s
top-three furniture exporters has donated a dinning set of little defect furniture to a selected high
school in up-province but the receivers (teachers and students) do not feel appreciate since they have
got enough tables and chairs. Their most preference is to get a scholarship for higher education,
especially in the area of forest conservation and value-added wood processing. If the manager has
searched for their real need, he/she would sell instead of donating the defect furniture to set-up
funding for scholarship. This will create more firm value in the long-run if the graduate scholar can
help promote the company image and becomes the valuable human resource to enhance strategic
capabilities crucial for sustainable advantages and performance.
Further findings suggest to employ new research approach emphasizing the mix methods between
qualitative and quantitative research. However, the holistic concept is quite new in Thailand where few
managers explicitly recognize its relevant to build superiority and sustainability. As such, multiple
qualitative rather than quantitative methods are advised to obtain more insightful information for indepth
understanding and making a proper decision on superior value creation.
All respondents agree to use the four qualitative techniques composed of behavioral-based approach
(focus group, in-depth interview), knowledge-based approach (brain storming, seminar/conference)
experience-based approach (expert opinion, case study) and relationship-based approach (individual
connectivity, network/partnership) as shown in Figure 3. The integration between knowledge-based
and behavioral based is chosen most because of their ability to get more reliable data through a timecost
saving mode. Case study is also very interesting if it is a real and up to date one. Consequently,
the new holistic marketing research is evidentially supported to provide more in-depth, precise, and
creative information intelligence with the focus on capabilities in learning relevant knowledge to guide
sound decision on superior creation of stakeholder value toward sustainability.
Behavioral Approach
(focus group, in-depth interview)
Knowledge-Based Approach
(brain storming, seminar/conference)
Multiple
Qualitative
Methods
Experience-Based Approach
(expert opinion, case study)
Relationship-Based Approach
(individual connectivity, network)
Figure 3: The new marketing research focusing on multiple qualitative methods toward value- driven
sustainability
The findings also describe each research method differently suited for creating each value
component. For instance, focus group seems more appropriate than in-depth interview for quick data
collection of new ideas and opinions on creating unique value if needed a quite larger sample size.
However, it is worth using survey method when exploring consumer satisfaction regarding the general
firm value. Meanwhile salespersons should conduct in-depth interview with the target consumers
when needed information to faster create and effectively implement customer value.
176

Somdee Hongphisanvivat
Building employee value, on the other hand, requires key information from observation of employees’
behavior and their interaction during meeting and working. Similarly, it is better to create shareholder
value based on information from observation during meeting and/or using focus group to obtain more
in-depth understanding for superior value-driven alternatives. On the contrary, building social value
needs information from expert opinion, brain storming and/or participation action research (let
participants involve through the whole research process to get the actual information with a more
promising implementation).
However, the better alternative is to employ multiple research methods since each has strength and
weakness depending on different situations and objectives of the information needed. For instance,
focus group requires expert moderator and unbiased questions. Brain storming is effective when
information providers are ready and willing to give the real data. Case study will provide useful
information when it is up to date, reliable and related to the research issue. Relational-based
approach, the new method proposed in this study, can give more in-depth and most precise
information but takes much more time and requires a certain close relationship with the information
providers in regard to unbiased research process.
Importantly, the findings indicate the most relevant criteria of choosing research approach is its ability
to provide the fact or valid information for proper action. To achieve sound market research and obtain
useful information for practicability, salespeople, customer service personals or involving persons
have to take active role on data collection under supervision of researchers or cooperation with them.
Specifically, it is crucial for researchers to work closely with managers through the whole research
process to help identify and committed utilize key knowledge of superior decision and proper action
toward value creation.
It is, thus, critical to identify relevant information about which components are key drivers of
sustainable advantages and performance, then develop abilities to continually improve them over time
with respect to the environmental changes. Crucially, successful implementation of holistic value
creation requires fairness, honesty and high commitment from top executives to avoid conflict of
interests as well as enhance harmonization among various stakeholders. Therefore, the new research
approach is necessary to provide enough insightful information, key knowledge and build strategic
capabilities in superior learning for better understanding the real needs of stakeholders as well as
making superior decision on a more speed and effective value-driven creation toward sustainability.
Finally, the study explores the appropriate assessment of sustainability in terms of superior
stakeholder value creation and performance in order to keep pace of the required strategic
capabilities consistent with the new environmental changes. Most respondents advise to use both
qualitative and quantitative measures. They all agree to employ satisfaction, happiness, and
collaboration with the company as the key-based assessing the customer value, employee value, and
social value. Meanwhile, dividend payment or return on equity (ROE), sales/revenue growth and profit
should be the major-based measurement of shareholder value.
In conclusion, this study provides a major contribution to highlight the new role of marketing research
shifting its focus on building capabilities for superior learning key knowledge, the most under-explored
area of value-driven sustainability. Beside the need of multiple research techniques and broaden the
theoretical domain of value-driven aspect, it proposes the new dimension of qualitative method, the
relationship-based approach to gather most insightful information through individual connectivity
and/or network/partnership. Since holistic research is the new powerful driver to sustainable
development. However, there is no clear support whether the proposed framework addressing this
new approach will lead to business success. Therefore, further study is conducted using a case
method to illustrate a success public company applying the holistic perspective.
4.2 Second study: Case method
A real case study of CP All, the most successful public company in Thailand retail trade industry is
demonstrated because it is in the high dynamics and competitive environments, has various key
players involvement, and has more details including financial figures available at the stock exchange
of Thailand (SET).
177

Somdee Hongphisanvivat
CP All has bought the license for the 7-Eleven trademark from the Southland Corporation, USA in
1988 to operate convenience store in Thailand. The company’s core services focus on food and
beverages accounting for 72.3 per cent of its business and serves about 6.1 million shoppers daily in
2009 increasing by 10.9 per cent from 2008. The company has applied the holistic perspective to
build superior value based on seven critical forces to competitive success: (1) focus its philosophy on
dual orientation (we desire to see smile from customers through our happy employees); (2) set up its
vision regarding society (we serve convenience to all communities); (3) employ both qualitative and
quantitative methods to conduct its own annual study about consumers and employees including ad
hoc research for action decision; (4) keep records and fast responses to all key information affecting
core abilities and core activities, such as, complaints, meeting and interacting between important
insiders and outsiders, industry insights and expert opinions, new knowledge, trends and
benchmarking from local/global partners, lecture and training series focusing on innovative knowledge
and learning abilities from smart teachers/trainers; (5) emphasize share learning, team work, and
competency enhancement using plan-do-check-action model for best practice; (6) invest in
information technology and database management to better find different needs and behaviors varied
to various locations for improving store services and operations; (7) establish good relationship and
strong collaboration with all key players and stakeholders in regard to offer a fair balanced value
attainment to get insight information and in-depth understanding essential for developing dynamic
capabilities to maintain advantages and superior performance.
The holistic research approach assists the company in getting quicker key knowledge and faster
learning to utilize it effectively and efficiently for building and retaining core capabilities to consistently
offer unique differentiated services and lower operating costs through its largest network of retail trade
over the country. This drives the company to successfully implement holistic value creation. CP All
can maintain its leadership earning ROE of 19.39%, ROI of 15.53% and net profit margin of 0.44 per
cent above the industry average exhibiting –19.14%, -10.43% and –85.09% respectively in the year of
2004-2009. This superiority is expected to continue in 2010-10212 estimating ROE of 39.6%-47.8%
far higher than the average. These evidences confirm applying holistic orientation properly enables
firm to achieve sustainable advantages and business success.
5. Conclusions, implications and future research
5.1 Conclusions
This study provides great contributions to shed insight the new role of marketing research to focus on
building capabilities for superior learning key knowledge, the most under-explored area as a new
source of sustainable advantages and business success. The findings support the proposed
framework addressing the need of holistic approach with the three valuable suggestions.
Firstly, pursuing the three critical steps of new marketing research: 1) identify key knowledge most
critical to sustainable advantages; 2) build capabilities in superior learning and utilizing key knowledge
through multiple research methods most useful and applicable; and 3) facilitate superior decision and
proper action through multiple interpretations for insightful understanding and providing better
alternatives most suited to the rapid environmental changes.
Secondly, applying the three critical steps through the three integrated principles: 1) market
orientation, 2) strategic resource orientation and 3) ecological orientation concentrating on abilities to
learn and use key knowledge to gain differentiate or cost advantages crucial for sustainable superior
performance.
Thirdly, utilizing multiple qualitative rather than quantitative methods suited to the holistic concept quit
new in Thailand. Also, the integration between knowledge-based and behavioral based among the
four relevant qualitative techniques should be considered to get a more reliable result with a time-cost
saving. Coincidentally, it is better to use a case study research as it becomes the dominant method to
explore how to build blocks of knowledge and experiences in sustainable development.
Further study illustrating the real case of CP All, the success public company in Thailand retail trade
industry highly supports applying the new holistic research approach suited to the dynamics leads to
sustainable advantages and business success. Following the new approach based on the 7 forces,
CP All can obtain faster key knowledge and performs better share learning it to develop the core
178

Somdee Hongphisanvivat
capabilities including dynamic capabilities driving to effectively implement holistic value creation of
superior stakeholders value. This results in the company’s performance (ROE, ROI and net profit
margin) outperformed the industry during 2004-2009 and expected to be higher than the average in
2010-2012.
5.2 Managerial implications
Successful in applying the holistic research perspective, firms have to shift the focus from concrete to
continuous collection of more meaningful data, from specific to all related information needed for
building strategic capabilities, and from separate to collaborate utilized information among the
decision markers and researchers. It is critical to hire skill researchers and understand the managers’
need about the key knowledge of value creation, realize the imperfect data and the necessity to adapt
the holistic perspective suited to the different corporate visions, types of industry, and the dynamic
environments. Valid and reliable holistic research remains important but abilities to learn and use
relevant knowledge faster and better than competitors are more imperative to value-driven
sustainability.
Further suggestion is to well train managers and employees for superior learning key knowledge,
building network and relationship of getting more in-depth knowledge from insiders and outsiders, and
encouraging active collaboration between researchers and decision makers focusing on research
targeting, hypothesis or proposition development, data analysis and interpretation, and practical
implementation of research results.
Finally, in the highly rapid changes in environment, it is more crucial to realize which holistic research
approach is most suitable for quicker and lower cost field study with various alternative findings to
effectively superior creation of stakeholder value. Pursuing more data driven based on case study,
longitudinal examinations aligning with observation to yield insight the internal and external
environments leading to superiority and sustainability should be more concern for long-run business
success.
5.3 Future research
To provide wider application and more contributions, future research should further explore the
following critical issues: 1) what are the optimum qualitative methods to successfully explore and
apply the holistic value-driven sustainability; 2) what are the appropriate mixed methods between
qualitative and quantitative research to provide relevant information suited to superior value-driven
sustainability; 3) how to use the multiple qualitative research to develop a parsimonious model of
value-driven sustainability with testable hypotheses; 4) what are more effective measures to evaluate
the new role of marketing research suitable for value-driven sustainability; 5) what are key
knowledge/information needed to create value-driven sustainability across industries, situations and
the nations; and 6) What are additional concepts needed to better explain and apply the holistic
approach.
References
Barney, J.B. (1991) “Firm Resources and Sustained Competitive Advantage”, Journal of Management, Vol 17, pp
99-120.
Belz, Frank-Martin and Peattie, Ken. (2011), Sustainability Marketing, John Wiley & Sons Ltd., United Kingdom.
Berthon, Bruno. (2010) “Responding to the Crisis: Redefining Corporate Value”, Corporate Governance, Vol 10,
No. 4, pp 354-359.
Best, Roger J. (2009), Market-Based Management: Strategies for Growing Customer Value and Profitability, 5 th
ed., Pearson Education Inc., Upper Saddle River, New Jersey.
Borland, Helen and Paliwoda, Stan (2011) “Marketing Responsibility in An Era of Economic and Climate
Change”, Marketing Intelligence & Planning, Vol 29, No. 1, pp 49-62.
Camp, Robert C. (1995) Business Process Benchmarking: Finding and Implementing Best Practices, ASQC
Quality Press, Milwaukee, WI.
Carson, D. and Coviello, N. (1996) “Qualitative Research Issues at the Marketing/Entrepreneurship Interface”,
Marketing Intelligence & Planning, Vol 14, No. 6, pp 51-58.
Carson, David, Gilmore, Audrey, Perry, Chad and Gronhaug, Kjell (2005), Qualitative Marketing Research, Sage,
London.
Carrion, G., Gonzalez, J. and Leal, A. (2004) “Identifying Key Knowledge Area in the Professional Services
Industry: A Case Study”, Journal of Knowledge Management, Vol 8, No. 6, pp 131-150.
Christmann, P. (2000) “Effects of Best Practices of Environmental Management on Cost Competitiveness: The
Role of Complementary Assets”, Academy of Management Journal, Vol 43, No. 4, pp 663-880.
179

Somdee Hongphisanvivat
Doz, Y.L. and Hamel, G. (1998), Alliance Advantage, Harvard Business School Press, Boston.
Foley, Anthony and Fahy, Fohn (2009) “Seeing Market Orientation Through A Capabilities Lens”, European
Journal of Marketing, Vol 43, No. 1/2, pp 13-20.
Ganeshasundaram, Raguragavan and Henley, Nadine (2007) “Decision Research Correlates Directly With Better
Business Performance”, Marketing Intelligence & Planning, Vol 25, No. 1, pp 45-65.
Gok, Osman and Hacioglu, Gungor (2010) “The Organizational Roles of Marketing and Marketing Manager”,
Marketing Intelligence & Planning, Vol 28, No. 3, pp 291-309.
Grunert, K.G., Jeppesen, L.F., Jespersen, K.R., Sonne, A.A., Hansen, K., Trondsen, T. and Young, T. (2005)
“Market Orientation of Value Chains: A Conceptual Framework Based on Four Case Studies from the Food
Industry”, European Journal of Marketing, Vol 39, No. 5/6, pp 428-455.
Hamlin, Robert P. (2000) “A Systematic Procedure for Targeting Market Research”, European Journal of
Marketing, Vol 34, No. 9/10, pp 1038-1052.
Johnson, Robert L. (2009) “Organizational Motivations for Going Green or Profitability versus Sustainability”, The
Business Review, Cambridge, Vol 13, No. 1, pp 22-28.
Kohli, A.K. and Kaworski, B.J. (1990) “Market Orientation: The Construct, Research Propositions and Managerial
Implications,” Journal of Marketing, Vol. 54 (April), pp. 1-18.
Kotler, P, N. Roberto and N. Lee (2002), Social Marketing: Improving the Quality of Life, 2 nd ed., Sage, Thousand
Oaks, CA.
Maignan, I., Ferrell, O.C and Ferrell, L. (2005) “A Stakeholder Model for Implementing Social Responsibility in
Marketing”, European Journal of Marketing, Vol 39, No. 9/10, pp 956-977.
Maria, Jose and Marti, Viedma, (2004) “Strategic Knowledge Benchmarking System (SKBS): A Knowledge-
Based Strategic Management Information System for Firms”, Journal of Knowledge Management, Vol 8,
No. 6, pp 31-49.
McDaniel, Carl, Jr. and Gates, Roger. (2010) Marketing Research Essentials, 7 th ed., John Wiley & Sons, Inc.,
NJ.
Mengue, B. and Ash, S. (2006) “Creating A Firm-level Dynamic Capability Through Capitalising on Market
Orientation and Innovativeness”, Academy of Marketing Science Journal, Vol 34, No. 1, pp 63-73.
Mittal, B. and Sheth, J.N. (2001), Value Space: Winning the Battle for Market Leadership, McGraw-Hill, New
York.
Narver, John C. and Slater, Stanley F. (1990) “The Effect of A Market Orientation on Business Profitability”,
Journal of Marketing, Vol 54, October, pp 20-35.
Ngo, L.V. and O’Cass, A. (2010) “Value Creation Architecture and Engineering: A Business Model Encompassing
the Firm-Customer Dyad”, European Business Review, Vol 22, No. 5, pp 496-514.
Olavarrieta, S. and Friedmann, R. (2007) “Market Orientation, Knowledge-Related Resources and Firm
Performance”, Journal of Business Research, Vol 61, No. 6, pp 623-630.
Pfeffer, J. (1995) “People, Capability and Competitive Snuccess”, Management Development Review, Vol 8, No.
5, pp 6-11.
Porter, M.E. (1985), Competitive Advantage, Free Press, New York.
Porter, M.E. (1996) “What is Strategy?”, Harvard Business Review, November-December, pp 61-78. Quinn, J.B.
(1992) Intelligent Enterprise, The Free Press, New York.
Sharkie, Rob (2003) “Knowledge Creation and Its Place in the Development of Sustainable Competitive
Advantage”, Journal of Knowledge Management, Vol 7, No. 1, pp 20-31.
Sheth, J. and Parvatiyar, A. (1995), “Ecological Imperatives and The Role of Marketing”, in Polonsky, M. and
Mintu-Wimsatt, A. (Eds), Environmental Marketing: Strategies, Practice, Theory and Research, The
Haworth Press, New York, pp 3-20.
Slater, S.F. and Narver, J.C. (1994) “Market Orientation, Customer Value, and Superior Performance”, Business
Horizons, Vol 73, No. 2, pp 162-167.
Slater, S.F. and Narver, J.C. (1995) “Market Orientation and The Learning Organization”, Journal of Marketing,
Vol 59, No. 3, pp 63-74.
Smith, D.V.L. and Fletcher, J.H. (2004), The Art & Science of Interpreting Market Research Evidence, John Wiley
& Sons Inc., NJ.
Smith, Peter A.C. and Sharicz, Carol. (2011) “The Shift Needed for Sustainability”, The Learning Organization,
Vol 18, No. 1, pp 73-86.
Stoelhorst, J.W. and van Raaij, E.M. (2004) “On Explaining Performance Differentials: Marketing and the
Managerial Theory of the Firm”, Journal of Business Research, Vol 57, No. 5, pp 462-477.
Subramanian, R., Kumar, K. and Strandholm, K. (2009) “The Role of Organizational Competencies in the Market-
Orientation-Performance Relationship: An Empirical Analysis”, International Journal of Commerce and
Management, Vol 19, No. 1, pp 7-26.
Sullivan, H.P. (2000), Value-Driven Intellectual Capital, John Wiley & Sons, Inc., New York.
Vorhies, Douglas W. and Morgan, Neil A. (2005) “Benchmarking Marketing Capabilities for Sustainable
Competitive Advantage”, Journal of Marketing, Vol 69, January, pp 80-94.
Woodruff, R.B. (1997) “Customer Value: The Next Source for Competitive Advantage”, Journal of the Academy of
Marketing Science, Vol 25, No. 2, pp 139-153.
180

Student Satisfaction Formation: Linkage Between
Information Satisfaction and College Choice Satisfaction
Nurlida Ismail 1 , Faridah Hj Hassan 2 and Nooraini Mohamad Sheriff 2
1
Taylor’s University, Subang Jaya, Malaysia
2
Universiti Teknologi MARA, Shah Alam, Malaysia
nurlida.ismail@taylors.edu.my
faridah387@salam.uitm.edu.my
nooraini@salam.uitm.edu.my
Abstract: Satisfaction level that students have in their college choice can have a long term impact on the
continued existence of the college. Hence, student satisfaction is considered an important performance indicator.
This study investigates the relationship between information characteristics (comprises of information on college
attributes and information accessibility), information satisfaction and college choice satisfaction. Structured
questionnaire was employed and self- administered survey approach was used to gather data. Two types of
analysis were presented. Firstly, descriptive statistics was employed to describe the basic features of the data for
each item in a construct. While the second analyses involved the establishment of the measurement model as
well as testing the model fit for each construct using structural equation modeling (SEM). The findings of this
study provided a very interesting discovery. It strongly pointed out that information satisfaction plays an important
role in mediating the relationship between information characteristics with college choice satisfaction. The result
of the study indicated very strongly that information characteristics have no direct impact on college choice
satisfaction however; the indirect effect of information characteristics onto college choice satisfaction through
information satisfaction is very sizeable (95% confidence interval: 0.534, 0.802). Hence, ultimately the real
challenge of marketers of higher education institutions is to ensure that information is disseminated effectively
and should result to students being satisfied with such information. As a conclusion, marketers must device their
promotional strategies effectively to make sure that potential students are satisfied with the information provided
through the various promotional sources.
Keywords: information characteristics, information satisfaction, college choice satisfaction
1. Introduction
Satisfaction level that students have in their college choice can have a long term impact on the
continued existence of the college. It has been proven that choice satisfaction is linked to
improvement in retention and loyalty rate (Beerli, Martin and Quitana, 2004; Helgesen and Nesset,
2007) and is considered an important performance indicator (Chan and Chan, 2004). Higher
education, however, represents a completely new and unfamiliar service experience for students. The
advancement in the education services environment has resulted in dramatic changes to the
landscape of higher education worldwide and issues pertaining to in-flow and out-flow of students can
be very complex. Hence, information on characteristics affecting student satisfaction is essential to
marketers. The focus of previous studies on customer satisfaction aimed at understanding satisfaction
as a post purchase phenomenon. However, in addition to post purchase satisfaction, it is pertinent to
also acknowledge that satisfaction with information may possibly occur before a choice is made. It is
reasonable that before a customer can be satisfied with the purchase, one must foremost be satisfied
with the information obtained regarding the product/ service. It is during the information search stage
that marketers are able to provide the right incitement to influence students regarding their
attributes/qualities and create a reaction on needs and wants of potential students leading to positive
motivation to a choice decision.
Though, the pursuit of students’ search for information on various institutions has been much easier
and faster, it has generated a problem of oversupply of information among education institutions.
Hence, effective marketing and well-informed communication techniques are considered necessary to
influence students not only during their search process but also during evaluation and choice process.
With many colleges to choose from, understanding information characteristics influencing these
students information satisfaction is significant for administrators of higher education institutions in their
marketing management decisions.
Information satisfaction creation could most likely be a major consequence of a purchase behavior. It
was indicated that customers perceived different levels of satisfaction with various types of
information that was sought and provided by different sources (Li, So, Fong, Lui, Lo and Lau,
2011).This suggests that in addition to comprehending students’ information characteristics, effort to
181

Nurlida Ismail, Faridah Hj Hassan and Nooraini Mohamad Sheriff
find out how satisfied students are with the information acquired should be of crucial concern. In
effect, the marketing approaches and students’ satisfaction with the information acquired might also
have the capacity to influence the choice made, hence choice satisfaction. Information satisfaction
provides a better impact and greater insight to the study of students’ college choice decision and
ultimately their choice satisfaction. This study aims to investigate the relationship between information
characteristics, information satisfaction and college choice satisfaction.
2. Literature review
It was argued that students’ expectations and satisfaction were to a great extent shaped by the
information acquired through various sources (Halstead, Hartman and Schimdt, 1994). And the
evaluation between students’ expectations and product performance perceptions were based on the
information on product attributes/ qualities. Hence, it can be concluded that information gathered from
various sources regarding attributes of various colleges was the basis of students’ evaluation and
judgment. As a matter of fact, it was revealed in a study by Haji Hassan and Mohamad Sheriff (2006)
that external marketing stimuli was perceived to be the most dominant determinant in stimulating
students’ need to pursue their study at higher education institution. The external marketing stimuli
mentioned were information regarding college attributes and these attributes among others include
quality of programs, quality of lecturers and quality of physical resources. Chapman (1981) also
indicated college attributes as important information and were essential criteria for institution choice.
According to Wagner and Fard (2009) in their study on Malaysian students’ intention to study at a
higher education institution, it was found that information on college attributes has a significant
relationship with students’ intention to pursue higher education.
In addition, due to the massive information provided by various colleges on their attributes, the
information accessibility difference among these colleges may also influence the students search
process. Accessibility differences may influence how the information is used and thus affect choice
outcome. For example, colleges for which information is difficult to acquire or information from websites
that is difficult to retrieve and comprehend may be ignored and not considered for evaluation of
choice. Students may not only prefer information that can be easily obtained but also one that they
can comprehend. Hence, in addition to information on college attributes, accessibility issues are of
particularly relevant for marketing communication decisions.
Bruce (1998) and Spreng, MacKenzie and Olshavsky (1996) strongly advocated that information
satisfaction is an important variable in the judgment of the performance of product used or service
employed. They further asserted that the expectation consumers have concerning a product or
service is not only depending upon the information gathered. This expectation is also dependent on
the feeling of satisfaction with the information they have acquired. Therefore, understanding and
identifying information on college attributes that influence students satisfaction with the information
acquired will definitely be of great value. This is because by recognizing these attributes marketers
will be able to develop better communication and targeting strategies. Especially in education service,
which is intangible and is normally associated with high perceived risk (Murray and Schlacter 1990),
being satisfied with the information sought enable customers to reduce the level of uncertainty and
enhance the quality of choice made.
The satisfaction with information acquired will in turn enhance students’ choice satisfaction. Admitting
the fact that college choice is the ultimate aim, nonetheless, students’ satisfaction with their chosen
college is also as important and crucial. Satisfaction is considered an important performance indicator
(Chan and Chan, 2004; Selnes, 1993). Customer satisfaction is of importance as it creates new
customers and retains current ones. It was proven that customer choice satisfaction was linked to
customer retention (Anderson and Sullivan, 1993; Helgesen and Nesset, 2007). And customer loyalty
and retention are perceived to be key components of improving market share and enhancing the
value of organization (Beerli et al., 2004; Rust and Zahorik, 1993). Peters (in Rust and Zahorik, 1993)
pointed out that the cost of new customers may be five times more costly than that of retaining them.
Reichheld (1996) indicated that loyalty reduces the need to incur customer acquisition costs. Since it
is costly to attract new customers, therefore, retaining existing customers can be very crucial.
Satisfied customers will remain loyal and become walking advertisement as through word of mouth
(WOM) they talk favorably to others about the service or product.
Hence, in-depth knowledge concerning information characteristics and satisfaction of students is
important for education institutions to understand the nature of this process and make better
182

Nurlida Ismail, Faridah Hj Hassan and Nooraini Mohamad Sheriff
judgment. The above discussion on information characteristics, information satisfaction and choice
satisfaction led to the development of the following hypotheses:
H1 There is a significant relationship between information characteristics and information
satisfaction.
H2 There is a significant relationship between information satisfaction and college choice
satisfaction.
H3 Information satisfaction is a significant mediator in the relationship of information
characteristics to college choice satisfaction.
Figure 1 represents the proposed theoretical framework of information characteristics, information
satisfaction and college choice satisfaction model. And Table 1 summarizes the conceptual definitions
of the variables/ constructs examined in this study.
Figure 1: Framework of the study
Table 1: Conceptual definitions
Constructs/variables Definitions Sources
Information characteristics
College attributes
Accessibility
Information satisfaction
College Choice Satisfaction
Characteristics/features of college such as
lecturers’ quality, location, costs and
financial aids, program availability,
international recognition and soft skill
development. These are antecedents that
influence students’ satisfaction.
The extent to which information is available
and reachable to the client in a format that
the client can use.
Further classified into physical, functional
and intellectual accessibility.
A subjective satisfaction judgment of the
information used in choosing a product.
This feeling of contentment will occur when
one’s expectation are either met or
exceeded.
Student’s positive reaction that comes
about when an institution provides an
agreeable level of fulfilment in which the
institution performance met the student’s
standard/ expectation.
183
Athiyaman (1997);
Bourke (2000); Burns
(2006); Chapman (1981);
Cubillo, Sanchez and
Cervino (2006); Qureshi
(1995); Turner (1998)
Bettman (1979)
Connelly Rich, Curley
and Kelly (1990).
Spreng et al. (1996)
Assael (2001); Oliver
(1980)
Ismail (2008)

3. Methodology
Nurlida Ismail, Faridah Hj Hassan and Nooraini Mohamad Sheriff
This study aims to quantify relationships between variables and to generalize results hence structured
questionnaire was employed and self- administered survey approach was used to gather data. Similar
approach was used by Helgesen and Nesset (2007) and Letcher and Neves (2010) in their study on
student choice satisfaction on higher education institution. The choice of indicators was guided from
similar work done by previous researchers as shown in Table 2.
Table 2: List of constructs and sources
Constructs No. of items/
indicators
1 Information Characteristics
College Attributes
23
Sources
Cubillo et al. (2006); Kusumawati (2010); Wagner and
Fard (2009).
Information Accessibility 9
Connelly et al. (1990)
2 Information Satisfaction 5 Lectcher and Neves (2010); Li et al. (2011); Oliver
(1980)
3 College Choice Satisfaction 5 Lectcher and Neves (2010); Mai (2005); Oliver (1980)
Total 42
Subjects were international students in various private higher education institutions in Malaysia.
Colleges were randomly selected and a proportionate sampling was employed whereby the subjects
were divided according to the international students’ population and status of the colleges. By
employing proportionate sampling, it ensured that the sample size drawn from each group was
proportionately represented. This approach was good as it has high statistical efficiency (Cooper and
Schindler, 2011 and Saunders, Lewis and Thornhill, 2009).The categorizations of colleges according
to their status are university, branch campus, university college and college: The questionnaires were
distributed employing three methods based on the type preferred by the officers from the respective
colleges. Questionnaires were either hand-delivered, mailed together with a self-addressed stamped
envelope or a soft copy of the survey sent via electronic mail. The number of questionnaires
distributed to each college was based on the international students’ population of the respective
colleges. A proportionate number of questionnaires (at least ten percent of the college’s international
students’ population) were given to the officers-in-charge. Finally, out of more than eighty colleges
contacted only thirty-two colleges responded and handed over the completed survey within the
stipulated time. This whole process took about six months to complete. Out of the 1,915
questionnaires distributed only 620 were collected.
Based on Table 3, the result indicated that sixty-five percent (403 students) of the respondents were
male and 35 percent (217 students) of the respondents were female. A majority of the respondents
were majoring in the areas of social sciences (57 percent, 356 respondents). The remaining 43
percent (264 respondents) of the respondents were majoring in the area of sciences.
Table 3: Descriptive statistics of respondents’ profile
Gender:
Male
Female
Degree Specification:
Science
Social science
Frequency Percentage
Data of 620 samples was checked for major outliers and as a result of data cleaning process, only five
cases were identified as outliers and these cases were dropped. The remaining 615 cases were
found to be fit for further analysis. Two types of analysis were presented. Firstly, descriptive statistics
was employed to describe the basic features of the data for each item in a construct. While the
second analyses involved the establishment of the structural model as well as testing the model fit for
each construct using structural equation modeling (SEM). Prior to constructing the overall structural
model it was necessary to evaluate the goodness of fit measures for each unobserved
variables/constructs independently as each construct has multiple-indicators (Anderson and Gerbing,
1988). Table 4 shows the indices that were used to measure the goodness-of-fit for this paper. It also
consists of the threshold value for each index. Consequently, modification indexes (MI) were used to
184
403
217
264
356
65%
35%
43%
57%

Nurlida Ismail, Faridah Hj Hassan and Nooraini Mohamad Sheriff
alter models to achieve better fit. A few rounds of re-specification were done to improve the validity of
each unobserved variable. Though there is no absolute ruling on changing a particular parameter
however, the indication is that the decision made on the basis of the MI must be theoretically justified.
Table 4: Indices to measure the goodness-of-fit for this study using SEM
Goodness of fit indices Indicator Source
1 Chi-square and Chi-square/df : as the test
for model discrepancy
Less than three (0.9) Byrne (2009); Hair,
Anderson, Tatham and Black
(2005); Joreskog and
Sorbom (1989)
3 Adjusted Goodness of Fit Index (AGFI): More than 0.9 (>0.9) Byrne (2009); Hair et al
(2005); Joreskog and
Sorbom (1989)
4 Comparative Fit Index against the null More than 0.9 (>0.9) Bentler (1990); Byrne (2009);
model (CFI)
Hair et al (2005)
5 Tucker Lewis Index against the null model More than 0.9 (>0.9) Byrne (2009); Hair et al
(TLI)
(2005); Tucker and Lewis (
1973)
6 Root Mean Square Error Approximation Less than 0.08(

Nurlida Ismail, Faridah Hj Hassan and Nooraini Mohamad Sheriff
4.2 Structural model of the study
Figure 2 reveals the structural model for this study. The figure indicates that the model has three main
constructs (college choice satisfaction, information satisfaction and information characteristics).
Information characteristics were categorized into two sub-domains: attributes and accessibility. And
attributes were measured by professional skills and development, issues on education institution, cost
and financing and lastly academic programs. Figure 2 also indicates that all the goodness-of-fit
statistics met the appropriate acceptable value and the model fit was acceptable. The Chi-square/df
was below 3.0, other fix indices were more than 0.90 and RMSEA was less than 0.08.
IC = Information characteristics, ATT = Attributes, ACS = Accessibility, SAT = Information satisfaction,
Choice = College choice satisfaction
Figure 2: Structural model of the study
In addition, based on Figure 2, the factor loading (L) for items in information characteristics construct
ranged from 0.67 to 0.83, 0.69 to 0.81 for information satisfaction construct and 0.76 to 0.87 for
college choice satisfaction construct. As suggested by Hair, Anderson, Tatham and Black (2005), in
the case of multivariate analysis with social sciences, factor loading above 0.50 is required. Hence,
when factor loading is more than 0.50, the percentage of variance explained (L 2 ) will naturally be
more than 0.25. Figure 2 reveals the L 2 values of all the items of the constructs for this study
exceeded the minimum value. The lowest L 2 value for information characteristics construct is 0.45
indicating that the item was explaining 45 percent of the construct. While the lowest L 2 value for
information satisfaction and college choice satisfaction is 0.48 and 0.58 respectively. The result
indicated that the overall fit for the model was considered acceptable and appropriate for further
analysis.
4.3 Analysis of hypotheses
As presented in Table 7, findings of this study indicated that the first two hypotheses (H1 and H2) were
supported and were in congruent with previous literatures. It can be concluded that information
characteristics (beta = .851) is an important predictor of information satisfaction and information
satisfaction (beta = .781) in turn is an important predictor of college choice satisfaction.
Table 7: Research hypotheses result
H1
H2
There is a significant relationship between information
characteristics and information satisfaction
There is a significant relationship between information
satisfaction and college choice satisfaction
*Significant level at 0.05
Std.Est C.R. P Beta Result
.448 5.931 .001* .851 Supported
.045 15.885 .001* .781 Supported
Figure 3 presents the hypothesized information satisfaction and choice satisfaction model that is
assessed. In the structural path of the final model the two endogenous variables are information
satisfaction and college choice satisfaction and the exogenous variable is information characteristics.
186

Nurlida Ismail, Faridah Hj Hassan and Nooraini Mohamad Sheriff
Since information satisfaction is both exogenous and endogenous it is also called the mediator
variable.
Figure 3: Finalized framework for this study.
H1 There is a significant relationship between information characteristics and information
satisfaction.
The result strongly supported this hypothesis (beta = 0.851, p = 0.001). Both information on college
attributes and accessibility of information were of importance in determining the strong agreement of
satisfaction with the information. Hence for students, the more important the college attributes were to
them the higher their information satisfaction would be if the information gathered met their
expectation. As for information accessibility, according to Daugherty and Ellinger (1994), information
availability, hence accessibility and clients’ responsiveness are positively associated. Brown (2002)
and Lederer, Maupin, Sena and Zhuang (2000), further confirmed that the usage of information is
highly dependent on the ease of finding and understanding the information itself. With respect to the
result, it can be suggested that these international students were not able to physically view the
institution hence the ease and accessibility of information were very important. When information was
readily accessible, these students were able to respond and react better and this translated to greater
client fulfillment and satisfaction in their information search prior to making a choice.
H2 There is a significant relationship between information satisfaction and college choice
satisfaction.
Result of the study strongly support this hypothesis (beta = .781, p = 0.001) demonstrating that there
is a positive direction of relationship between information satisfaction and college choice satisfaction.
Result reported is consistent with findings of previous researchers (Bruce, 1998; Halstead et al., 1994
and Spreng et al., 1996). Hence, it can be explicated that students’ college choice satisfaction is
positively dependent on the information satisfaction. Spreng et al. (1996) examined individual’s overall
satisfaction with a purchase and found that individual feeling of satisfaction with the choice made is
reliant on two antecedents; performance of the products or services and information satisfaction.
Hence, this further strengthens the explanation of the relationship between information satisfaction
and college choice satisfaction. Evidences from this study pointed out that the more the students are
satisfied with the information, the more likely the students will be satisfied with their college choice.
H3 Information satisfaction is a significant mediator in the relationship of information
characteristics to college choice satisfaction.
A further examination was done to test the mediating effect of information satisfaction based on 1,000
bootstrap resamples. Based on Table 8, the value of zero (0) does not fall within the intervals.
Table 8: 95% confidence interval for mediating effect based on 1,000 bootstrap resamples
Information Characteristics
College choice satisfaction [0.534, 0.802]
In addition, the regression coefficient for information characteristics and college choice satisfaction is
shown in Table 9. The P value of the tested relationship is more than 0.05. Hence, there is no
evidence of direct effect of information characteristics on college choice satisfaction. Based on Table
8 and Table 9, results strongly indicated that information satisfaction was a significant mediator in
information characteristics to institution choice satisfaction relationship.
Table 9: Regression coefficient for information characteristics and college choice satisfaction
C.R
Std.Est
P
.
Information Characteristics → College Choice Satisfaction .445 1.93 .084
187

Nurlida Ismail, Faridah Hj Hassan and Nooraini Mohamad Sheriff
5. Conclusion and discussion
With increasing competition among various education institutions both locally and globally, these
institutions need to cautiously re-examine their promotional and communication strategies. Higher
education institutions are facing environmental challenges that call for the development of new
marketing approaches. Strategically marketers and administrators must acknowledge factors that
have significant impact towards students’ choice decision. It is of main concern for marketers and
administrators of education institutions to provide their potential students with sufficient and relevant
information to support their decision. As pointed out by Eagle and Brennan (2007), relying on
fundamental marketing concepts, it becomes evident that once institutions identify students’ needs
and wants, the task of satisfying these needs and wants becomes more feasible. The findings of this
study provided two very interesting discovery. Firstly, findings of this study indicated that information
characteristics are determinants of information satisfaction and consequently students’ choice
satisfaction is dependent on information satisfaction. Secondly, result strongly pointed out that
information satisfaction plays an important role in mediating the relationship between information
characteristics with college choice satisfaction. The result of the study indicated very strongly that
information characteristics have no direct impact on college choice satisfaction. However, the indirect
effect of information characteristics onto college choice satisfaction through information satisfaction is
very sizeable (95% confidence interval: 0.534, 0.802).This significant finding implied that the
challenges of marketers are not only to focus on providing information on vital college attributes but
as important is also on strategies on how information should be disseminated appropriately and
effectively. As postulated by Cardozo (1965), customer expectation about a service or product to a
certain degree may be influenced by marketer controlled information disseminating techniques (such
as advertisement, catalogues and brochures). Hence, ultimately the real challenge of marketers of
higher education institutions is to ensure that information is disseminated successfully and should
result to students being satisfied with such information. Result of this study strongly suggested that
information satisfaction is a condition for the formation of student’s college choice satisfaction.
References
Akaike, H. (1973) “Information Theory and an Extension of the Maximum Likelihood Principle”, In B.N Petrov & F.
Csaki (Eds). Second International Symposium of Information Theory (pp. 267 – 281). Budapest: Acadamiai
Kaido
Anderson, J.C. and Gerbing, D.W. (1988) “Structural Equation Modelling in Practice: A Review and
Recommended Two-step Approach”, Psychology Bulletin, Vol 103, pp 411 – 423.
Anderson, E.W. and Sullivan, M.W. (1993) ”The Antecedents and Consequences of Customer Satisfaction for
Firms”, Marketing Science, Vol 12, No. 2, pp 125 – 143.
Assael, H. (2001) Consumer Behavior and Marketing Action, 6 th edition, Thomas Learning
Athiyaman, A. (1997) “Linking Student Satisfaction and Service Quality Perceptions: The Case of University
Education”, European Journal of Marketing, Vol 31, No.7, pp 528-540.
Beerli, A., Martin, J. D. and Quitana, A. (2004) “A Model of Customer Loyalty in the Retail Banking Market”,
European Journal of Marketing, Vol 38, No. 1 / 2, pp 253 – 275
Bentler, P.M. (1990) “Comparative Indexes in Structural Models”, Psychological Bulletin, Vol 107, pp 238 -246
Bettman, J.R. (1979) An Information Processing Theory of Consumer Choice, Addison-Wesley Publishing
Company Inc.
Bourke, A. (2000) “A Model of the Determinants of International Trade in Higher Education”, The Service
Industries Journal, Vol 20, No.1, pp 110-138.
Brown I.T.J. (2002) “Individual and Technological Factors Affecting Perceived Ease of Use of Web-Based
Learning Technologies in Developing Country”, Electronic Journal on Information Systems in Developing
Countries, Vol 9, No.5, pp 1 – 15.
Browne, M. W. and Cudeck, R. (1993) “Alternative Ways of Assessing Model Fit”, in Bollen, K. A., and Long, J. S.
(ed.) Testing Structural Equation Models, Newbury Park, CA: Sage
Bruce, H. (1998) “User Satisfaction with Information Seeking on the Internet”, Journal of the American Society for
Information Science, Vol 49, No. 6, pp 541-556.
Byrne, B.M (2009) Structural Equation Modeling with AMOS: Basic Concepts, Applications, and Programming,
Routledge
Burns, M.J. (2006) “Factors Influencing the College Choice of African-American Students Admitted to the
Institution of Agriculture, Food and Natural Resources”, Thesis presented to the Faculty of the Graduate
School University of Missouri-Columbia.
Cardozo, R.N. (1965) “An Experimental Study of Customer Effort, Expectation, and Satisfaction”, Journal of
Marketing Research, Vol 11, Aug, pp 244-249.
Chan, A.P.C and Chan, A.P.L. (2004). “Key Performance Indicators for Measuring Construction Success”,
Benchmarking: An International Journal, Vol 11, No. 2, pp. 203 – 221
Chapman, D.W. (1981) “A Model of Student College Choice”, The Journal of Higher Education, Vol 52, No.5,
September - October, pp 490-506.
188

Nurlida Ismail, Faridah Hj Hassan and Nooraini Mohamad Sheriff
Connelly D.P., Rich E.C., Curley S.P. and Kelly J.T. (1990) “Knowledge Resource Preferences of Family
Physicians”, Journal of Family Practices, Vol 30, No.3, March, pp 353 – 359
Cox, D.F. (1967) Risk Taking and Information Handling in Consumer Behavior, D.F Cox ed, Boston: Harvard
University Press.
Cooper, D.R. and Schindler, P.S. (2011) Business Research Methods, 11th edition, McGraw-Hill International
Edition
Cubillo, J.M., Sanchez, J. and Cervino, J. (2006) “International Students’ Decision-making Process”, International
Journal of Educational Management, Vol 20, No. 2, pp 101-115.
Daugherty, P.J. and Ellinger, A.E. (1994) “Information Accessibility: Customer Responsiveness and Enhanced
Performance”, International Journal of Physical Distribution and Logistics Management, Vol 25, No.1, pp 4-
17.
Eagle, L. and Brennan, R. (2007) “Are Students Customers? TQM and Marketing Perspectives”, Quality
Assurance in Education, Vol 15, No. 1, pp 44-60.
Evans, M.M, Foxall, G and.Jamal, A. (2009) Consumer Behaviour, 2 nd edition, John Wiley & Sons Ltd.
Everitt, B.S. (2002) The Cambridge Dictionary of Statistics, 2 nd edition, Cambridge University Press
Hair, J.F., Anderson, R.E., Tatham, R.L., and Black, W.C. (2005) Multivariate data analysis, 6th edition,
Englewood Cliffs, NJ: Prentice Hall
Haji Hassan, F. and Mohamad Sheriff, N, (2006) “Students’ Need Recognition for Higher Education at Private
Colleges in Malaysia: An exploratory Perspective”, Sunway Academic Journal, Vol 3, pp 61 – 71
Halstead, D., Hartman, D. and Schimdt, S.L. (1994) “Multisource Effects on the Satisfaction Formation Process”,
Journal of the Academy of Marketing Science, Vol 22, No.2, pp 114-129.
Helgesen, Ø. and Nesset, E. (2007) “What Accounts for Students’ Loyalty? Some Field Study Evidence”,
International Journal of Educational Management, Vol 21, No. 2, pp 126-143.
Ismail, N. (2008) “International Students’ Satisfaction in their Choice of a Private Higher Education Institution”,
Proceeding of the 9 th International Business Research Conference, Melbourne, Australia.
Kusumawati, A (2010) “ Privatisation and Marketisation of Indonesian Public University: A Systematic Review of
Student Choice Criteria Literature”, Proceeding of the Indonesian Student International Conference,
Thinking of Home while Away:The Contribution of Indonesian Students Studying Overseas for Education in
Indonesia, Melbourne, Australia.
Lederer, A.L., Maupin, D.J., Sena, M.P. and Zhuang, Y. (2000) ”The Technology Acceptance Model and the
World Wide Web”, Decision Support System, Vol 29, pp 269 – 282.
Letcher, D.W and Neves, J.S (2010) “Determinants of Undergraduate Business Student Satisfaction”, Research
in Higher Education Journal, Retrieved from http://www.aabri.com/manuscripts/09391.pdf
Jöreskog, K.G and Sörbom, D. (1993) LISREL 8: Structural equation modeling with the SIMPLIS command
language, Scientific Software International / Erlbaum
Li, P.W.C, So, W.K.W, Fong, D.Y.T, Lui, L.Y.Y, Lo, J.C.K and Lau, S.F. (2011) “The Information Needs of Breast
Cancer Patients in Hong Kong and Their Levels of Satisfaction with the Provision of Information”. Cancer
Nursing, Vol 34, No.1, January/February, pp 49-57.
Mai, L.W (2005) “A Comparative Study Between UK and US: The Student Satisfaction in Higher Education and
its Influential Factors”, Journal of Marketing Management, Vol 21, pp 859-878.
Murray, K.B. and Schlacter J.L (1990) ”The Impact of Services Versus Goods on Consumers’ Assessment of
Perceived Risk and Variability”, Journal of the Academy of Marketing Science, Winter, Vol 18, No.1, pp 51-
65.
Oliver, R.L. (1980) “A Cognitive Model of the Antecedents and Consequences of Satisfaction Decisions”, Journal
of Marketing Research, Vol 17, November, pp 460-469.
Peters, T. (1988) Thriving on Chaos, New York, NY, Alfred A Knopf.
Qureshi, S. (1995) “College Accession Research: New Variables in an Old Equation”, Journal of Professional
Services Marketing, Vol 12, No. 2, pp 163-170.
Reichheld, F.F. (1996) The Loyalty Effect: The Hidden Force Behind Growth, Profits, and Lasting Value, Harvard
Business School Press, Boston, MA.
Rust, R.T. and Zahorik, A.J. (1993) “Customer Satisfaction, Customer Retention, and Market Share”, Journal of
Retailing, Vol 69, No. 2, pp 193-215.
Saunders, M., Lewis, P. and Thornhill, A. (2009) Research Methods for Business Students, 5 th edition, Financial
Times Prentice Hall.
Selnes, F. (1993) “An Examination of the Effect of Product Performance on Brand Reputation, Satisfaction and
Loyalty”, European Journal of Marketing, Vol 27, No. 9, pp 19 – 35
Spreng, R.A., Mackenzie, S.B. and Olshavsky, R.W. (1996) “A Reexamination of the Determinants of Consumer
Satisfaction”, Journal of Marketing, Vol 60, July, pp 15-32.
Steiger, J.H. and Lind, J.C. (1980) “Statistically Based Tests for the Number of Common Factors”, Paper
presented at the Psychometric Society Annual Meeting, Iowa City, IA, USA.
Tucker, L.R and Lewis, C. (1973) “A Reliability Coefficient for Maximum Likelihood Factor Analysis”,
Psychometrika, Vol 38, pp 1-10
Turner, J.P. (1998) “An Investigation of Business Undergraduates’ Choice to Study at Edith Cowan University”,
Unpublished research report, Edith Cowan University, Perth
Wagner, K and Fard, P.Y. (2009) “Factors Influencing Malaysian Students’ Intention to Study at a Higher
Education Institution”, Paper presented in: E-Leader Conference Kuala Lumpur, 5-7 January, Retrieved
from http://www.g-casa.com/PDF/malaysia/Wagner-Fard.pdf
189

Weaving the Threads of Reflexivity: Coming to Terms With
Grounded Theory Research
MacDonald Kanyangale and Noel Pearse
Rhodes Business School, Rhodes University, Grahamstown, South Africa
M.Kanyangale@ru.ac.za
N.Pearse@ru.ac.za
Abstract: Researchers are ultimately responsible for the knowledge they produce as well as how that knowledge
is produced. Reflexivity can contribute to making better knowledge claims and improving research practice.
However, novice grounded theory researchers may experience practical difficulties and confusion in practicing
and deepening the process of reflexivity in a rigorous way. Over the years, different versions of grounded theory
have emerged, each with their own set of ontological and epistemological assumptions. This has placed great
demands on the novice grounded theory researcher, who has to cautiously and appropriately, exercise reflexivity
in their research work. The paper first focuses on understanding the Straussian grounded theory method and the
concept of reflexivity. It then explores the implications of reflexivity from the perspective of the role of the
researcher in: conducting the literature review, theoretical sampling, data collection and analysis; as well as the
implications of a grounded theory researcher exercising reflexivity. In doing so, the article explores the
practicalities of entrenching the concept of reflexivity within the method of grounded theory. The paper concludes
that merely focusing on theoretical and analytical aspects of reflexivity is insufficient to adequately weave threads
of reflexivity into a grounded theory research process. In addition to theoretical and analytical aspects, this paper
calls for researchers using grounded theory to also give attention to the social, inter-subjective dimension of
reflexivity.
Keywords: reflexivity, reflection, grounded theory, theoretical sensitivity
1. Introduction
Rennie (1998:114) aptly posits that grounded theory “turned method upside down: instead of using
data to test theory, they were used to develop it”. Since the two sociologists - Glaser and Strauss -
discovered the grounded theory method in 1967; it has evolved and been used in many fields, as
diverse as anthropology, nursing and management. Despite the diffusion of grounded theory, there
remain four central aspects which underscore the uniqueness of the grounded theory method in
developing a tight, well integrated, inductive theory that accounts for the pattern of behaviour. These
aspects are: (a) the grounded theory is emergent from empirical data rather than from inferences or
existing theories; (b) theoretical sampling and constant comparison enable theory generation through
systematic and analytical procedures; (c) memo writing enables the formulation and revision of theory
throughout the process; and (d) the research process is flexible, and fluid.
Furthermore, grounded theory is not a linear research endeavour, but rather concurrent, iterative and
integrative, with data collection, analysis and conceptual theorising occurring in a simultaneous
fashion until such a time the theory is fully generated (Glaser and Strauss 1967:48-49). Over the
years, grounded theory has evolved into three main versions, namely the original version by Glaser
and Strauss (1967); the Glaserian approach (1992) and the Straussian approach (1990). Another
variation in the conducting of grounded theory is associated with Charmaz (2000) who introduced the
constructivist grounded theory method. Close examination of constructivist grounded theory exposes
critical similarities and limited substantial distinctions from the Straussian version. It is on this premise
that the constructivist grounded theory method proposed by Charmaz (2000) is not discussed or given
further attention here.
This evolution of the grounded theory method has raised debate and suspicion about using grounded
theory in terms of its rigour, and robustness. This has further added to the confusion experienced by
novice researchers. In this regard, divergences in the ontological and epistemological tenets of these
emerging versions of grounded theory, and also conflicting opinions and unresolved issues on the
nature and process of grounded theory contributes to the confusion of new grounded theory
researchers. This is particularly evident in terms of their personal ontological and epistemic stance,
how to do grounded theory, and how to deepen reflexivity.
Grounding a theory in data is the core aim of grounded theory as a method, rather than verifying
extant theory as traditional quantitative methods seek to do (Glaser and Strauss, 1967:1). It has been
argued that for a long time the grounded theory literature has focused mainly on the analytical
190

MacDonald Kanyangale and Noel Pearse
process (Strauss and Corbin, 1990; Hall and Callery, 2001). It is only more recently that attention has
turned to reflexivity in grounded theory (McGhee et al, 2007). This growing scholarly attention on
reflexivity is exposing a number of research issues that are central and worrisome to a potential
grounded theory researcher (McGhee et al, 2007). Inevitably, researchers proposing to utilize this
method would therefore be wise to consider the conflicting opinions and evaluate claims about the
evolution of grounded theory in terms of existing recommendations for the use of the method. This
article explores the relevance, and implications of reflexivity for grounded theory researchers,
especially those adopting the Straussian version.
The paper discusses the Straussian grounded theory method partly in comparison with the Glaserian
grounded theory, prior to introducing reflexivity and exploring some thoughts on the implications of
reflexivity to the Straussian version of generating a grounded theory.
2. Overview of Straussian grounded theory
Barney Glaser and the late Anselm Strauss, both sociologists had differences in their respective
attempts to elaborate and clarify crucial tenets of grounded theory method which they actually cofounded
(Glaser and Strauss, 1967). These differences have led to Glaserian and Straussian schools
of the grounded theory method. Strauss and Corbin (1990:23) clarify grounded theory as being:
“inductively derived from the study of the phenomenon it represents. That is, it is
discovered, developed, and provisionally verified through systematic data collection and
analysis of data pertaining to that phenomenon”.
Subsequently, Glaser (1992) attacked the Strauss and Corbin (1990) version of grounded theory for
departing from and ignoring 90 percent of what he alleged was the original grounded theory method.
Among the various differences between Straussian and Glaserian grounded theory method,
verification has been illuminated as probably one of the major contentious issues (Dey, 1999).
Verification is characterised by Strauss and Corbin (1990:187) as:
...built into each step of the process. Though not testing in a statistical sense, we are
constantly comparing hypothesis against reality (the data), making modifications, and
then testing again. Only that which is repeatedly found to stand up against reality will be
built into the theory.
Glaser dismisses verification on the basis that “there is no need to verify the theory against data
unless it is not based on data, but already goes beyond it” (Dey, 1999:21). Thus, verification is the
next step for theories that are already generated from data.
The two originators of grounded theory method also differed on the use of the coding instruments and
the paradigm model which are unique to the Straussian grounded theory method. These are criticised
as pre-determined schemes for forcing theory, too programmed, and relying much on preconceived
prescription (Bryant and Charmaz, 2010:9). Glaser (1992) objects to the practices of detailed or
minute active analysis and interpretation of data word by word and coding the meaning found in
words or group of words. As such, Glaser (1992) criticizes the Straussian approach as another
method, which should be termed “full conceptual description” and not grounded theory at all.
Additionally, the detailed practices (1) shift the researcher`s primary focus from data to adherence of
complex process of systematic coding and procedures; and (2) also affects the creativity and flexibility
essential to discover theory (Glaser (1992).
On the contrary, advocates of the Strausian approach uphold that the paradigm model is userfriendly,
and helps researchers to draw statements of relationship between the phenomena, context,
intervening conditions, actions/inaction strategies and consequences (Strauss and Corbin, 1990:74).
The elaborate techniques (e.g. axial coding and paradigm model) are not prescriptive, suggesting a
densely codified operation. In fact they are “there to assist with analysis but should never drive the
analysis in and of themselves” (Strauss and Corbin, 1998:58). Strauss and Corbin recommend
reflexivity while Glaser (1992) warns against reflexivity paralysis, which he asserts is self-destructive,
stifles productivity, and unnecessarily distracts the researcher from constant comparative analysis. In
fact, he further claims that reflexivity is not needed, as constant comparative analysis will expose
researcher bias in the research process.
191

MacDonald Kanyangale and Noel Pearse
With all these key divergences and tensions within grounded theory, the question is how does the
researcher exercise reflexivity correctly?
3. Unpacking reflexivity
Historically, reflexivity as a method is traceable to the domain of anthropology, and emerged as a
response to the critique of ethnocentrism prevalent in ethnography. As a way of addressing
ethnocentrism, the main focus was to situate the researcher as neutral, unbiased and not making any
judgement, but rather understanding as comprehensively as possible the participants’ experiences,
way of knowing and being (Pillow, 2003). As such, the researcher`s own values, assumptions, and
biases were expected to be always under control and in check as part of a rigorous pursuit of
neutrality and objectivity in collecting, analysing data and writing up the research. However, the
questioning of how a researcher attains this, and the effects of power imbalances in research
relations were enough to justify a revisiting of the notion of neutrality and objectivity in qualitative
research. Simultaneously, this raised significant questions and placed a critical focus on researcher
subjectivity and how the research relationship altered the doing of research. According to Pillow
(2003:273), this is about “how does who I am, whom I have been, who I think I am, and how I feel,
affect data collection and analysis?”
Sometimes, reflexivity is defined as “self-awareness and an awareness of the relationship between
the investigator and the research environment” (Lamb and Huttlinger, 1989:766). This acknowledges
that the position (including role and identity), interest, and values of the researcher exert some
influence on the research relationship and findings, either intentionally or unintentionally. That is, the
researcher forms part of the social context under study. In this regard, reflexivity is an active,
intersubjective process, which has learning aspects, while reflection is less interactive and sometimes
connotes introspection of an isolated mind. Reflection as a verb is associated with “introspection”. An
emphasis on “retrospection” and “self-critical evaluation” is evident in some of the definitions of
reflexivity. For example, Johns (2000:61) defines reflexivity as “looking back and seeing self as a
changed person”.
Finlay (2002) considers reflexivity and reflection as opposite ends of a continuum. In this vein,
reflection is ‘thinking about’ whereas reflexivity at the other end is more active than just reflection.
Reflexivity is characterised as more immediate, continuing, monitoring, dynamic and subjective selfawareness.
Thus, reflexivity by a researcher involves (a) continuous evaluation of subjective
responses, (b) intersubjective, dynamic relationships (c) thoughtful, conscious self-awareness, and (d)
deliberate attention to the research process itself. Robson (2002) term these as reflective activities,
which form part of the entire research process.
Some definitions of reflexivity are susceptible to emphasis on the retrospective and not the
spontaneity of reflexivity. For example, reflexivity is defined as “sensitivity to the ways in which the
researcher and the research process have shaped the collected data, including the role of prior
assumptions and experience, which can influence even the most avowedly inductive inquiries” (Mays
and Pope, 2000:51).Other definitions of reflexivity emphasize the mental activity rather than the social
or intersubjective aspect of reflexivity. For instance, Powers and Knapp (1995:143) define reflection
as “mentally reviewing and going over and over the details of the lived experience under study,
viewing it from different perspectives, and trying to capture meaning ...”. The paper acknowledges
that, reflexivity as a concept is hard to pin-down and scholars do not agree on what reflexivity means,
nor do they provide many practical demonstrations for novice researchers of how to achieve it, or how
to use it as a guiding standard in the research process. In this paper, the definition of reflexivity by
Robson (2002:22) is adopted. Robson (2002: 22) suggests that reflexivity is:
…an awareness of the ways in which the researcher as an individual with a particular
social identity and background has an impact on the research process.
Reflexivity is a methodical process of on-going questioning of the doing of research and also learning
about self as a researcher, research participants, others, and the research context which illuminates
meanings about personal, theoretical, ethical, and epistemological aspects of the research process.
Aptly, Flood (1999: 35) asserts that “without some degree of reflexivity any research is blind and
without purpose”.
192

4. Weaving reflexivity?
MacDonald Kanyangale and Noel Pearse
As a point of departure, Strauss and Corbin (1990:18) identify a range of skills required by a
qualitative researcher, which includes the ability to:
step back and critically analyse the situation, to recognize bias, to obtain valid and reliable data,
and to think abstractly. To do these, a qualitative researcher requires theoretical and social
sensitivity, the ability to maintain analytic distance while drawing upon past experience and
theoretical knowledge to interpret what is seen, astute powers of observation and good
interactional skills.
Practically, it is crucial to understand the way in which a Straussian grounded theorist would engage
in reflexivity so as to enhance the theory that is generated.
4.1 Role of the researcher
The role of a grounded theory researcher depicted in literature is not clear and straightforward partly
due to ontological and epistemological differences. This has triggered some confusion. Essentially,
Charmaz (2000) is among scholars who suggest that Strauss and Corbin (1990) shifted grounded
theory away from a positivistic ontology towards a constructionist paradigm. Further, this implies that
the nature of reality is rather local, and that there are multiple perspectives of that reality. Thus, the
knower is subjectively linked and part of what can be known. This requires what other scholars have
termed self-reflexivity by the researcher, characterized by dynamic, continuous self-awareness,
conscious reflection and response to the research relations and process. While Strauss and Corbin
(1990) advise that researchers learn about an actor`s interpretation and perspectives, it is not clear
whether this excludes the interpretation of the researcher, or if the participant`s interpretations or
perspective are incorporated into those of the researcher or vice versa. This raises questions about a
relativist view of the world, in terms of whether the perspectives of the researcher and participant are
accorded similar status or not. In doing reflexivity, a grounded theory researcher further grapples with
whether to promote the position of mutuality between researcher and participant or not. This reflects
the extent in which the voice of the researcher is evident in the final product (Ghee, Glenn and
Atkinson, 2007:335). However, what is clear is that the researcher should show fidelity to participants`
everyday realities for the theory to be related to what actually goes on in their real life (Strauss and
Corbin, 1990:23). This allows the plurality of perspectives by research participants to emerge as part
of reality.
The grounded theory researcher has to demonstrate fidelity to the world view of research participants.
This should be manifested through the researcher`s thinking as represented in memos The
interpretive role of a researcher recognises the sensitivity required of a grounded theory researcher as
a social being, creating and recreating social process rather than reproducing, mirroring or reflecting
social reality. This is supported by Schwandt (1994:18) who believes that a qualitative researcher
provides his or her “construction of the constructions of the actors one studies”.
The roles that a researcher takes may be multiple, conflictual or complementary and also dynamic.
For example, practitioner research may be conducted by a researcher who is an insider (i.e. an MBA
student conducting research in his/her organisation using peers and junior employees as participants
or a nurse-researcher investigating a phenomenon to do with nursing students or patients) which may
have implications in terms of the expert power, professional or positional powers. Actual or perceived
power relations outside or within the research context may impact on the researcher –participants
relationships. From a reflexivity perspective, a simple declaration of any non-research or research
related power relations in practitioner research is not enough without exploring its real impact on the
researcher-participant relationship. There is a need to explore when and how the behaviour and
perceptions of participants and the researcher are influenced by the historical, hierarchical or vertical
relationships outside the research relationship. The other roles of the researcher may be durable
beyond the research period as they relate to work. These roles may be overlapping and not easily
demarcated between work and research such that they bring dynamic and unique influence between
the participants and the researcher. The multiplicity, fluidity and role identity problems have been
identified and experienced by practitioner researchers such as nurse researchers and those
investigating their own organisations. Power differences may inhibit the research relationship (i.e.
participants giving conventional answers to impress the manager-researcher; changes in role identity
of a nurse caring and also studying the very same patients). As a possible solution Hobson
(2004:305) suggests that if the “nurse in the researcher cannot be removed then the skills should be
193

MacDonald Kanyangale and Noel Pearse
utilised rather than hidden”. This illustrates not only the potential of a researcher having multiple roles,
but also that the researcher is not detached or simply mirroring reality, but rather engaged with the
social context and has social skills useable in the research process to construct the reality. A
grounded theory researcher adopting a Straussian version of grounded theory needs to be clearly
conscious about his/her role(s), continuously monitor and respond to how these are influencing the
research process According to feminist research, a researcher may identify herself closely with
participants, or make an effort to match the characteristics of participants (i.e. in terms of dress,
language, mannerisms).This may serve to amplify or address the obvious power imbalances between
researcher and participant. Alternatively, the Straussian grounded theory researcher has options to
position his/her role as (a) an insider; (b) as an insider, detached, but appearing not too different from
participants; (c) a collaborator, meaning that the researcher and the informants are partners using
their own experiences and reflections to induce meaning. That is, women studying women and
women`s issues as reflexivity is not only about producing knowledge that helps in gaining insight into
how the social world works but also how this knowledge is produced ; or (d) actually appear more of
an outsider with informed consent to interactively explore the experience of participants. While some
scholars suggest these roles of a researcher, Reed and Procter (1995) contend that a researcher may
occupy three possible roles. Thus, as (a) an outsider with no experience, and visitor to the area of
study, (b) a hybrid, referring to a researcher who is familiar with a research area and is researching
the practice of other practioners, and also (c) as insider who is a practitioner-researcher, who is
investigating his or her own practice and that of known colleagues in the same field.
4.2 Theoretical sampling
Theoretical sampling involves identifying further sources and collecting additional data to further
develop the emerging theory, while simultaneously engaging in data coding and analysis (Glaser and
Strauss, 1967:45-77). In pursuing theoretical sampling, the researcher is more reflexive about what is
emerging from data, the theoretical purpose and theoretical relevance of what incidents need to be
sampled further as a way of theory refinement. This is done to generate theory to the fullest extent,
develop as many properties of the categories as possible, and relate categories to each other and to
their properties as well (Glaser and Strauss, 1967:49).
Besides the emphasis on theoretical aspects emerging from data analysis and incidents, there is
limited advice on how a researcher should interact with people who are holders of incidents or
experiences. Clearly, the researcher-participant interaction is expected to be the means of getting,
confirming or disconfirming data to saturate various categories and the core category (Glaser and
Strauss, 1967). There is a risk that the social dimension of reflexivity, which is more intersubjective,
and complements sensitivity at theoretical level, may be a taken-for-granted aspect of reflexivity
(Baker et al. (1992). Sampling in grounded theory is cumulatively or incrementally driven by what is
emerging from the data and the evolving theory. This directs the researcher to what data to get next
and from where, but does not give advice on intersubjective reflexivity to manage the social
interaction, either when seeking the consent of research participants, or during collection of data
(Glaser and Strauss, 1967:45). Sampling and data analysis processes are more pronounced while the
dynamic social process of a researcher interacting and nurturing a relationship with research
participants is taken for granted. This may pose a threat to the relationship of the researcher and
research participants who have experiences that are useful to cumulatively develop a grounded
theory (Baker et al., 1992). It is very common in grounded theory research to conduct follow-up
interviews with participants, implying that social interactions at any stage of data collection are very
important. This poses political, ethical, interpersonal and institutional challenges which may impact on
the reflexivity of a grounded theory researcher. The audit trails left by the researcher using grounded
theory tend to be mostly analytical in focus and de-emphasize the social dimension and the reflexivity
of the researcher. Both during and after interviews, there is a greater need for the researcher to focus
on his/her assumptions about the world, knowledge, and learning based on a reflection on the social
and political nature of the research situation encountered. The social, political, and ethical questions
being encountered, informs the researcher of the characteristics to look for in the next informants and
improves the conducting of “reflexivity in-the- moment” during subsequent interviews.
4.3 Literature review
Conflicting views on the use of the literature review are relevant to the researcher in terms of ensuring
open mindedness and avoiding the threat of being prematurely driven by preconceptions obtained
from reading the literature (Strauss and Corbin, 1990:50). For instance, Glaserian grounded theory
194

MacDonald Kanyangale and Noel Pearse
discourages a detailed literature review initially, as a way to prevent the researcher from being
constrained, contaminated or inhibited by concepts from literature in terms of creativity in theory
generation. Preconceptions are conceived as unwanted mental baggage that may restrain the mind of
the researcher and limit intellectual flexibility and freedom to explore a phenomenon (Strauss and
Corbin, 1990: 27-28). However, an initial literature review may be important to ascertain the paucity of
extant knowledge and justify the pursuit of grounded theory. Strauss and Corbin (1990:50) argue that
an initial review of technical literature is important for theoretical sensitivity, the stimulation of key
research questions, and as a source of the research problem. It also provides for clarity of thought,
raises awareness about how to avoid the methodological, ethical and conceptual pitfalls that occurred
in previous studies, and helps the researcher to discover the contours of the existing knowledge base,
so as to justify the need for a grounded theory (Strauss and Corbin, 1990: 35).
In the Straussian grounded theory method, both the literature and the researcher’s past experience
helps to theoretically sensitize the researcher to understand the vocabulary of participants, to guide
initial observations, to focus on self-questioning, to refine the questioning and questions posed to
participants, to stimulate questions during analysis and to speed up the process of data analysis
(Strauss and Corbin, 1990:51-54). There is however, the danger that the literature may drive the
researcher to “force the data through the pre-existing conceptual framework”. The Straussian
grounded theory researcher is therefore open minded, and subjects his or her background experience
to scrutiny, to remove prejudice. As such, he/she is not detached, nor claims to be empty headed or a
tabula rasa. Strauss and Corbin (1998:49) caution that “familiarity with relevant literature can enhance
sensitivity to subtle nuances in data, just as it can block creativity”.
It is very crucial for a novice researcher to note that the emphasis on theoretical sensitivity is geared
towards theoretical refinement, elimination of bias from literature, avoidance of forcing of theory from
data while the subtle nuances of the social interaction between the researcher and the participants
are not equally pronounced. In this way, reflexivity is a process by the researcher and involves selfcritical
reflection, self-inspection of potential sources of biases and their control, theoretical
predispositions and how these serve as analytical resources (Swandit, 2007:260). The analytic
procedures promote a continuous interaction with data (collecting and asking about data) and create
theoretical sensitivity (Strauss and Corbin, 1990:41). Thus, literature, experience brought into the
research situation, and analytic techniques being used in the research process are sources of
theoretical sensitivity, which enhances the awareness of the researcher to the subtleties of meaning
in data, helps him to periodically step back and ask questions, and also promotes an attitude of
scepticism (Strauss and Corbin, 1990:41-45). These elements of reflexivity focus on the researcher
and actual data, in line with the key notion that “any theoretical explanations or categories brought to
the research situation are considered provisional until supported by data” (Strauss and Corbin,
1990:45). The major concern of this type of theoretical sensitivity is to achieve a balance “between
that which is created by the researcher and the real” (Strauss and Corbin, 1990:47). Further use of
the literature after the theory generation provides further validation of the accuracy of and illumination
of findings (Strauss and Corbin, 1990:52-53).
Research participants are active social beings whose interactions in the research milieu may be
influenced by their own previous reading of the literature, as well as their personal experiences.
However, there is limited focus of reflexivity on the research participants as active social beings
interacting based on their own previous knowledge and experience The advice by Strauss and Corbin
(1990:56) is that a researcher can use and read relevant published material during all phases of the
research as long as one does not become “captive of any of them” to disentangle the tight ties of the
researcher to data. A novice grounded theory researcher should seek to expand this linkage to allow
a focus on the reflexive actions of participants and not be restricted to the already collected data.
4.4 Data collection and analysis
Reflexivity also entails how a researcher behaves to encourage participants to be open with him or
her. It is likely to be difficult for those being researched to be open if their reflexivity informs them that
the researchers themselves are not willing to be open. Important aspects of these interactions include
the political aspects of seeking consent at organisational and individual levels where participants may
be interviewed several times, ethical requirements of the research, and reciprocal expectations (i.e.
emotions, benefits to research participants) which impact on the research relations between the
researcher and the researched. The originators of the grounded theory method ignored this area on
195

MacDonald Kanyangale and Noel Pearse
the assumption that data collection is discussed in many other ‘excellent books on doing field
research work” (Strauss and Corbin, 1990:59).
Grounded theory researchers engage in constant comparison of incidents, and concepts to
categorise them according similarities and differences, to saturate categories and to also understand
what is happening in the data. Generally, qualitative researchers use some mechanisms such as
keeping a research log, researcher field notes, and research journals as part of the practice of
reflexivity in the field. This provides a working history of unfolding processes and thoughts of a
researcher during the research process. Specifically, grounded theory researchers write memos
throughout the research process to explore hunches, ideas, and thoughts in search of the broader
conceptual explanation for the process, action and interaction. Memo writing as reflexivity suggests
that the researcher is a lone mind, reflecting on reflexivity and what is emerging from the data
collection and analysis. In this respect, theoretical sensitivity echoes “the attribute of having insight,
the ability to give meaning to data, the capacity to understand, and capability to separate the pertinent
from that which isn’t” (Strauss and Corbin, 1990:42). Analytical techniques such as flip-flop and both
close-in and far-out comparisons stimulates theoretical sensitivity to open up new ways of thinking,
break through assumptions, and uncover specific dimensions about the phenomenon being studied.
Additionally, the “waving of red flag technique is “used to see beyond the obvious in data” by being
sensitive and questioning what seems to be the common view point, shared assumptions or
behaviour which signal need for a more closer probing. While theoretical sensitivity, constant
comparison and waving the red flag are inclined towards analytical aspects, these techniques need to
be equally intensified during the intersubjective, conversational interviews when gathering data and
not only when analysing data. The lack of intensity in the use of the techniques in data collection is
perpetrated by Strauss and Corbin (1990:94) who assert that these “techniques are to be used only
as aids to analysis, not done with every field note or interview”. The novice grounded theory
researcher has to question the reflexivity implications, for instance of using all or any of the
techniques, namely flip-flop, waving red flag, and comparisons during interviewing process in the field
and also using these to analyse data. Reflexivity is valuable throughout the entire research process
and not only during data analysis or field data collection.
The grounded theory researcher is concerned about what he or she should be reflexive or sensitive
about. Through axial coding, the paradigm model is a mechanism which helps researchers to focus
on developing and testing statements of relationship or connections between the phenomena, context
in which phenomenon is embedded, intervening conditions, actions/interaction strategies and
consequences to reflect the social context (Strauss and Corbin, 1990:74). The development of
relationships using the paradigm model is distinctively analytical and serves to establish links between
sub-categories, categories, properties, and dimension for purposes of category specification. The
aspects of the paradigm model and statements of relationships, linkages between them are subjected
to constant comparison and verification against data.
Additionally, epistemological reflexivity in grounded theory involves recognition of participants by
according them opportunity to comment, challenge or disagree with any aspect of the emerging
theory until it is relevant, fits and works for them. However, some researchers caution that participants
may not adequately validate a conceptual-level theory which transcends descriptive accounts as it
synthesizes perspectives from different participants from different contexts. It is even more
unfortunate that participants do not have access to the full range of data, or the thinking of a
researcher which has been captured and changed in several memos. It is therefore important for
researchers not to overemphasize member validation. Overemphasis on member validation may
simply encourage reporting of findings at a simple descriptive level, so that members can easily
validate the analytic categories rather than develop explanatory power at a conceptual level. As such,
the novice researcher has to be continuously aware of this potential pitfall during data collection and
analysis, and also in the validation of findings.
5. Conclusion and implications
Reflexivity in most grounded theory literature seems to focus primarily on the researcher, despite the
reality that research participants as active human beings also undergo reflexivity. As such, there is a
need for the novice Straussian grounded theory researcher to actively incorporate and respond to
research participants “in-the-moment” of social interactions in the field. This dynamic reflexivity
requires that researchers to actively focus on the social and intersubjective aspects that influence the
researcher and the researched. It is our contention that this should complement the analytical and
196

MacDonald Kanyangale and Noel Pearse
theoretical procedures and practices of a researcher to enhance reflexivity (i.e. writing journals, field
notes, memo, and member validations).
Furthermore, we argue that interpersonal and social judgment of active human beings, especially
research participants in the social construction of knowledge should not be overlooked as it is a key
issue of reflexivity in the research process. In this regard, social sensitivity “in-the-moment” of
constructing knowledge and not just reflection after knowledge construction is very important. This
upholds that reflexivity cannot be comprehensively as well exhaustively anticipated and reinforced by
the prescriptive, analytical, and theoretical procedures alone. A novice grounded theory researcher
needs to be aware that reflexivity is not about the researcher as a solo mind reflecting on “what is
going on in data” and producing rich decision audit, or critical evaluation of how the researcher might
have influenced the research. Actually, reflexivity goes beyond a self-critical evaluation to embrace,
genuine sensitivity to others, in terms of when and how these influence or get influenced in the social
context as the research process unfolds.
In this respect, the paper strongly advises novice Straussian grounded theory researchers to
emphasize both the social and analytical dimensions of reflexivity which embraces critical selfawareness
on one hand, and recognition of how and when research participants influence the
research process on the other hand. In this way, a Straussian grounded theory researcher needs to
weave threads of social, theoretical and analytical reflexivity, not as a static tool to just mirror
confession but rather as a genuine catharsis that increases the legitimacy or validity of the research.
The Straussian grounded theory researcher needs to be fully conscious that he or she is intimately
involved in both the process and product of research.
References
Bryant, A. and Charmaz, K. (2010). The Sage Handbook of Grounded Theory. London: Sage.
Baker, C., Wuest, J. and Stern, P.N. (1992).Method slurring: The grounded theory/phenomenology example.
Journal of Advanced Nursing, Vol.17, pp 1355-1360.
Charmaz, K. (2000). Grounded Theory: Objectivist and Constructivist Methods. In: Denzin, N.K. and Lincoln, Y.S.
(eds). The Handbook of Qualitative Research .Thousand Oaks, CA: Sage.
Dey, I. (1999). Grounding Grounded Theory-Guidelines for Qualitative Inquiry. London: Academic Press.
Finlay, L. (2000).Pearls, pitch, and provocation. “Outing” the researcher: the provenance, process, and practice
of reflexivity. Qualitative Health Research, Vol.12, No.4, pp531-545.
Flood, G. (1999) Beyond Phenomenology: Rethinking the Study of Religion. London:
Cassell.
Glaser, B.G. and Strauss, A.L. (1967). The discovery of grounded theory: Strategies for qualitative research.
Chicago: Aldine Atherton Inc.
Glaser, B.G. (1992). Emergency versus Forcing: Basics of Grounded Theory Analysis. Mill Valley, CA: Sociology
Press.
Hall, W.A and Callery, C.P (2001) Pearls, pitch, and provocations. Enhancing the rigor of grounded theory:
incorporating reflexivity and relationality. Qualitative Health Research, Vol.11, No.2, pp257-272.
Hertz, R. (1997). Introduction: Reflexivity and voice. In: Hertz, R. (ed.) Reflexivity and voice. Thousand Oaks,
CA: Sage.
Hobson, D (2004).Split personalities: Role conflict between the nurse and the nurse researcher .Nursing Times
Research, Vol.9, No.4, pp305
Johns, C. (2000).Becoming a Reflective Practitioner. A reflective and Holistic Approach to Clinical Supervision.
Oxford, Blackwell Science.
Kelle, U. (2005). Emergence versus Forcing of Empirical Data? A Crucial Problem of Grounded Theory
Reconsidered. Forum: Qualitative Social Research, Vol.6, No.2 Article 27. Available from: http://bnresolving.de/urn:de:0114-fqs0502275.[Accessed
on 1 November, 2009].
Lamb, B. and Huttlinger, K. (1989).Reflexivity in Nursing research. Western Journal of Nursing Research, Vol.11,
No.6, pp765-772.
Mc Ghee, G., Marland, G.R, and Atkinson, J. (2007) Grounded theory research: literature reviewing and
reflexivity. Journal of Advanced Nursing, Vol. 60, No.3, pp334-342.
Melia.K. M. (1996). Rediscovering Glaser. Qualitative Health Research, 6(3):368-378.
Mills, J., Bonner. A. and Francis, K., 2006. The Development of Constructionist Grounded Theory. International
Journal of Qualitative Methods, Vol. 5, No.1 article 3. Available from:
http://www.ualberta.ca/¬iiqm/backissues/5_1/pdf/mills.pdf. [Accessed on 2 November, 2008].
Parker, L. D. and Roffey, B. H. (1997). Back to the drawing board: Revisiting grounded theory and the everyday
accountant’s and manager’s reality. Accounting, Auditing and Accountability Journal, 10(2):212-247.
PartingtoN, D. (2000). Building grounded theory of management action. British Journal of Management, Vol. 11,
No. 2, pp 91-102.
197

MacDonald Kanyangale and Noel Pearse
Pillow, W.S. (2010) Dangerous reflexivity: Rigour, responsibility and reflexivity in qualitative research. In:
Thomson, P. and Walker, M. (eds) The Routledge Doctoral Student Companion. Psychology Press and
Rout ledge, London
Pillow, W.S. (2003) Confessions, Catharsis or Cure? Rethinking the use of reflexivity as methodological power in
qualitative research. International Journal of Qualitative Studies in Education, Vol.16, No.2, pp175-196.
Powers, B.A. and Knapp, T.R. (1995).A Dictionary of Nursing Theory and Research. Thousand Oaks, California,
Sage.
Reed, J. and Procter, S. (1995). Practitioner research in context. In: Reed, J. and Procter, S. (eds) Practitioner
Research in Health Care: The Inside Story. London, Chapman and Hall.
Rennie, D.L. (1998). Grounded theory methodology: The pressing need for a coherent logic of justification.
Theory and Psychology, Vol.8, No.1, pp101-119.
Robson, C. (2002).Real World Research (2e).Blackwell, Oxford.
Schwandt, T.A. (1994). Constructivist, Interpretivist approaches to human Enquiry. In: Denzin, N.K. and Lincoln,
Y.S. (eds). Handbook of Qualitative Research. Sage Publications. Thousand Oaks, CA: Sage.
Strauss, A. and Corbin, J, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques.
Newbury Park: Sage.
Strauss, A. and Corbin, J. (1998).Basics of qualitative research: Techniques and procedures for developing
grounded theory (2e). Thousand Oaks, CA: Sage.
Swandt, T.A. (2007).Qualitative Inquiry: A Dictionary of Terms. Thousand Oaks, CA, Sage.
198

The use of the Time Diary Method to Explore Academic
Time Management: Insights From an Australian University
Branka Krivokapic-Skoko 1 , Roderick Duncan 2 and Kerry Tilbrook 1
1
School of Management and Marketing, Charles Sturt University, Bathurst,
Australia
2
School of Finance and Accounting, Charles Sturt University, Bathurst,
Australia
bkrivokapic@csu.edu.au
rduncan@csu.edu.au
ktilbrook@csu.edu.au
Abstract: Academics and universities have an interest in tracking the tasks and workloads of academics in the
areas of teaching, research and administration, but do academics and their employers know how many hours a
week an academic engages in particular tasks? We discuss the on-going development of an electronic time diary
tool to measure an academic’s teaching, research and administrative tasks. Our preliminary findings suggest that
time spent communicating with students is now a significant portion of an academic workday. Academics work
long hours interrupted by the demands of students as customers coupled with increasing accountability and
compliance within universities. We find that academics value aspects of their work which foster self-direction and
creativity in both teaching and research activities.
Keywords: academics and time management; higher education management; time diaries
1. Introduction
The introduction of learning technologies such as subject websites, forums and emails has resulted in
changes in the tasks expected of academics in running their subjects. Changes in the student cohorts
entering universities have meant a change in teaching practices as the new generation of students
have higher expectations of communication with academics. University student profiles are changing
because:
current students are less likely to be pursuing their studies uninterrupted by paid work obligations
changes to admissions standards means that universities are admitting students with varied
levels of preparation for university and arguably without adequate remediation interventions in
place, and
Introduction of the new VLE (virtual learning environment) has contributed to a climate where
students expect academics to be contacable24/7 (Anderson et al. 2002; Tennant M, McMullen C
& Kaczynski D (2010).
These fundamental workplace changes give rise to several questions that universities and academics
have strong interests in answering. What tasks are academics doing to run their subjects? How long
are academics spending on those tasks to run their subjects? The answers given by past generations
of academics may have little relevance in today’s universities.
The aim of this research was to assess the use of the time by academics, so in the next stage of this
research we can develop and pilot a time diary tool for measuring academics’ activities and use of
time across teaching, research and administration. This tool would assist academics in their own time
management, help academic leaders manage their academic staff and enable universities as
institutions to better leverage their resources to achieve quality outcomes in teaching and research.
2. Academics’ activities and use of time
Previous researchers have given some emphasis to the impact of new technologies on academics’
activities and time (Marginson & Considine 2000; Anderson et al. 2002; Hull 2006; Blackmore &
Sachs 2007). While the educational possibilities of informational and communication technologies for
online learning, collaborative teaching and other new forms of teaching are enormous, one of the
downsides of these new technologies is the way that new technologies impacts academic workloads.
As Tennant et al. (2010, p. 8) argued in their book on teaching and learning in higher eduction,
dealing responsibly with students’ emails takes a large amount of time, much more than student
contact in the so-called “pre-electronic” era. Accurately measuring the tasks and time required for
teaching a subject can assist universities in evaluating their teaching practices and also in juggling
199

Branka Krivokapic-Skoko, Roderick Duncan and Kerry Tilbrook
other academic tasks such as scholarship and research. Equally, an opportunity for academics to
reflect on their own use of time will support academics in evaluating their own teaching practices and
professional activities.
While information technology has many advantaged for the academics it has also the potential to
greatly expand the workload and the time needed to be spent on a variety of the new activities. The
preparation of computer-enabled learning takes more time for academics due to the increased need
for individual learning and training to master these techniques (Anderson et al. 2002; Hill 2006). Time
spent dealing with student emails has also increased greatly with the proliferation of means of
electronic communication. Yet this is invisible work (Anderson et al. 2002, p.13; Soliman 1999) which
is very difficult to capture in existing academic workload documents. Eighty-four percent of
respondents in the Anderson et al (2002) survey agreed that time spent on email communication was
increasing. There is also an expectation by students that academics can answer their enquiries 24/7,
which is part of the overall trend of commodification of the higher education industry with students
portrayed as ‘customers’ (Slaughter & Leslie 1997; Marginson & Considine 2000) leading to the
syndrome of the ‘24 hour professor’ (Chronicle of Higher Education, May 31, 2002, cited in Anderson
et al. 2002, p.18).
Tennant et al. (2010, p. 135) argue that academics now struggle with controlling the impact
technology is having upon the work day and upon personal time and that “technology has made
academics hyper-accessible. Many academics feel they now have to be hyper-responsible.” As
Buckholdt and Miller (2009. p 3) argued “ the idea of 40 hour work week has become even less
relevant in a society organized by information technology”. Some other changes in the education and
training environment, like the ones listed bellow, have led to work intensification for academics
(Dearlove, 1998; Jarvis, 2001; Ogbonna & Harris, 2004; Kinman & Jones, cited in Hull, 2006; Baty,
2005b, cited in Hull, 2006; Blackmore & Sachs, 2007, Jensen and Morgan 2009):
Technological duress - the introduction of new IT systems that academics are expected to master,
such as Interact and MSI;
Devolution of administration - the transfer of tasks that had traditionally been done by
administration to academics, such as managing travel expenses under ProMaster and entering
subject outlines under MSI;
Multiplication of campuses - daily travel between campuses is a feature of the workload of some
academics;
Uncoordinated escalation of bureaucratic process - the creation of entirely new systems and
requirements for academics, such as entering publications in the centralised research output data
base and completing annual performance management reports;
Increasing partnerships with outside organisations which require academic administrative support;
Casualisation - full-time academics find themselves supporting casual staff in tasks that casuals
are unable or untrained to perform, such as drawing up exams or engaging with administration;
Increasing external accountability pressures from government funding bodies e.g. ERA and
quality assurance requirements coupled with the requirement to generate self-funding from
entrepreneurial activities with insufficient infrastructure.
In many surveys of academic staff in the United States, Britain, Australia and New Zealand there was
an overwhelming picture of work intensification, higher stress levels and growing disillusionment with
the changing content of academic work which seems part of a global trend (OECD 1998). In analysing
how new technologies are transforming academics McShane (July 2006) also identified that apart
from significant increase in the workload coupled with the introduction of the new technology related
tasks students now expect academics to available almost 24/7. A survey 1,100 academics from 99
UK universities (Kinman and Jones, 2004) revealed high levels of academic stress, where nearly a
half of the respondents reported that that they were under strain, over two thirds find the work
stressful and almost 80% believe the status of profession is in decline. McLean (2006) and
Pachnowski and Jurcyk (2003) emphasised the level of stress and dissatisfaction among distance
educators at USA universities, caused by ‘round the clock’ nature of distance education.
Some commentators also discuss the seeming mismatch between the aspirations of academics and
the managerial expectations of the universities (Marginson & Considine 2000). All universities
potentially face the same risk of producing high levels of job dissatisfaction, alienation and
200

Branka Krivokapic-Skoko, Roderick Duncan and Kerry Tilbrook
disengagement for academics (Anderson et al. 2002; Coaldrake & Stedman 1998; Ogbonna & Harris
2004).
There is very little research on work intensification that actually measures this in a quantitative way.
Most existing research is in survey form which gives an interesting account of respondent’s
perceptions, attitudes and experiences but very little evidence in terms of a detailed time-breakdown
of these tasks. Many survey results discuss an increase in numbers of hours worked (Anderson et al.
2002; Coaldrake & Stedman 2000; Jensen & Morgan 2009; Vardi 2009) but the individual tasks are
not analysed in detail. Correspondingly, time management for academic leaders and heads of schools
is mentioned as important but it is not really quantified or treated in any depth (Ramsden 1998). One
of the problems is how difficult it is to measure the inter-relationships and interdependencies of these
learning and teaching tasks and their overlaps. Another problem is the subjectivities in terms of
academics’ responses to this allocation of work and their subsequent activities. These activities are
valued according to personal preferences (Wolf 2010) and as such are difficult to evaluate as they
also entail other complex questions such as staffing levels and administrative processes (Vardi 2009).
As early as 1999, a survey by McInnis concluded that “we are perhaps at a critical point for the
academic profession where the amount of hours worked, and the diffusion and fragmentation of tasks
seriously threatens the quality of both research and teaching” (McInnis 1999, p. 63, cited in Anderson
et al. 2002, p. 10). Perhaps this is even more critical with ever increasing demands for accountability
and devolution of academic workload from centres/senior executive, placing academics increasingly
in the position of harried middle-managers trapped between the demands of management and the
students they teach.
3. A time dairy method
A time diary approach, which is also referred to as a ‘micro behaviour’ approach to survey research
(Robinson & Bostrom 1994), can be used for collecting self-reports of an individual’s daily behaviour
(Robinson 1999). It is widely recognised (Kitterød 2001) that studies based on time diaries are one of
most reliable and valid data sources concerning time use. By using the time diaries it will be possible
to report activities as they naturally and sequentially occur in the daily work of academics, and
therefore explore actual behaviour. Time diaries also allow participants to record experiences of their
daily activities, and Bolger, Davis & Rafaeli (2003) documented therapeutic outcomes from this selfreflective
data collection process.
Using time diaries has been endorsed by researchers in educational leadership and management
(Morrison 2007). Time diaries have been used to examine the ways in which heads of schools and
faculties manage the time for professional activities (Earley & Fletcher-Campbell 1989). A time diary
method has been used to explore internet use (Ishii 2004; Collopy 1994; Nie & Hillygus 2002) and has
been seen as an ideal method to detect the new and unanticipated activities, such as the use of new
communication technologies (Robinson & Bostrom 1994). This approach has a potential for
professional development and reflective practice on the use of the academics’ time. Another
advantage of the time diary method is that it can measure simultaneous usage patterns, particularly in
the context of understanding academics’ use of the internet in their teaching, research and
administrative tasks.
The ways in which academics manage their time for profession activities was explored by the use of
the time dairy method in the empirical context of a regionally based Australian university. Combined
with reflective statements from the diaries and the focus groups, the time diary records were
collected, aggregated, collated and analysed in order to get a better understanding of the workday
experiences. The project used a seven-day work schedule instrument (Harms & Gershuny 2009) and
purposive rather than random samples of the participants, as has been done in the vast majority of
the time diary studies (Bolger et al. 2003; Pentland et al. 1999; Nie & Hillygus 2002).
Research participants were fulltime academics from different faculties across Charles Sturt University,
NSW, who were asked to make a continuous recording of their daily working activities. As the vast
majority of the time dairy studies, this research has involved purposive rather than random sample of
the participants, and the academics volunteered to take part in this research. Because participant
motivation in his case was high and the researcher – participant rapport was obviously good was and
already established we expected a low rate of faked compliances (Amie et al (2006). A fixed-time
schedule was used, where the spacing of intervals was an hour, which is considered to capture the
201

Branka Krivokapic-Skoko, Roderick Duncan and Kerry Tilbrook
change process in daily work activities (Bolger et al. 2003). A continuous recording allowed
participants to determine freely the time when one activity ends and another begins. The first set of
the diaries was open-ended and once the participants identified and captured specific types of
activities we developed pre-coded diaries (Harms & Gershuny 2009). We used the random-hour
technique (Pentland et al. 1999) in which research participants were reporting on a smaller segment
of the day. The participants were requested to maintain a schedule of entries every 3 hours as a
minimum. Initially, we decided to employ paper and pencil diaries which is still the most commonly
used approach in time diary research (Bolger et al. 2003). Because this method places substantial
time demands on the participants, participants were given incentives in the form of small individual
research funds.
The time diary sheet to be filled out (a shortened version is contained in Appendix 1) was a 24 hour
time diary and included the following main categories of daily activities:
Communicating with students
Subject administration
Subject preparation
Subject delivery
Research
Service and general administration
These activities were further broken down on the time sheets so that the research activity included
hours spent on grant preparation, reviewing articles for journals, filling out ethics forms, research
meetings with co-authors and other activities associated with research.
Both main activities and possible parallel activities are captured in time diaries. Parallel tasks (often
called secondary activities), such as searching the internet for appropriate video clips to use in
teaching while answering student emails, were recorded. As academics often do tasks
simultaneously, registration of the parallel activities provided the most complete picture of academics’
time use. Where multiple activities were reported in an hour, the academic’s time was assumed to
have been split equally between the reported activities.
Apart from capturing the amount of time academic tend to spend on different daily activities we also
use the time diary method to explore the subjective evaluations of the academic’s daily experience. A
three unit measurement scale was used to self-record the level of effectiveness and emotional
satisfaction felt by the academic at the end of a working day. Exploratory regression analysis was
used for empirical assessment of the relationships between the amount of time spent on particular
activities and reported levels of efficiency and emotional satisfaction.
The protocol for filling out the time dairies was given to the participants in February 2011, and
subsequently one focus group was held in April 2011 in order to obtain comments on the initial data
collecting, after which the categorization in the time diary of some activities were expanded and other
activities were contracted so as to better match the daily experiences of the participating academics.
During the first two weeks of the research project we ran a pilot test on the small group of academics.
We maintained ongoing contacts with the participants by acknowledging that filling out the time diary
sheets was time-consuming and that the project relied on the good will of the academics involved.
4. Results
An initial database of time diaries was compiled during the testing of the paper version of the diary
tool. Eight academics from three Faculties - Business, Science and Arts - from Charles Sturt
University filled out 278 daily diary sheets. All eight academics held positions as Lecturers and Senior
Lecturers with both teaching and research responsibilities. None of the eight had significant university
leadership roles. Of the 278 diary sheets submitted, 265 sheets revealed information about
academics’ use of time, and 223 sheets revealed information about academics’ self-evaluations of
their emotional response to their work and also the work effectiveness of their days. Table 1 reports
summary statistics on the 265 sheets revealing academics’ use of time.
Average hours of a working day reported by the academics were 9.27 which included both weekdays
and weekends. Academics reported working very long days in some periods as well as a significant
202

Branka Krivokapic-Skoko, Roderick Duncan and Kerry Tilbrook
number of weekends. There were large variations in activities across days with the reported standard
deviations of activities being greater than the values of the average hours for those activities. The
diary sheets revealed few days with large blocks of time devoted to single activities, rather academics
split their hours and days across multiple activities.
As a proportion of total working hours reported by the academics, the delivery of subjects - teaching
of lectures or tutorials, conducting labs or delivering seminars - took the least amount of time at only 9
percent of academic time. The activity which consumed the largest proportion of academic time was
research at 26 percent of reported hours, followed by service and general administration at 21
percent. Direct communication with students through electronic or non-electronic means consumed
16 percent of academics’ reported hours.
Table 1: Summary of daily activities
Hours per day
Activity Average (Hours) Std. Dev. (Hours) Percentage of
Average Day (%)
Communicating with
students
1.49 1.91 16.0
Subject administration 1.12 1.89 13.4
Subject preparation 1.38 2.02 14.8
Subject delivery 0.81 1.57 9.0
Research 2.47 3.26 25.8
Service and general
administration
1.99 2.56 21.0
Total 9.27 2.79 100.0
The participating academics were asked to rate their day emotionally on a scale of “Good”, “OK” and
“Poor”, as well as rating how effective they believed their day was. There was a high degree of
correlation between the two ratings with a correlation of 0.44 between the emotional rating and the
effectiveness rating of the day. Academics were more likely to give a higher positive affect and
effectiveness ratings on reported weekends than on weekdays. Possibly this difference between
weekdays and weekends was due to fewer interruptions and more self-directed activities on the
weekends.
A probit regression was conducted on the emotion and effectiveness ratings using the STATA 11
(StataCorp 2009) to ascertain the activities which made the participating academics more likely to
indicate a “Good” emotional rating for that day. Table 2 reports the results of the probit analysis on the
proportion of each activity for that day and the self-assessed effectiveness rating the academic
indicated for that same day. There are positive and statistically significant relationships between the
probability of an academic indicating a “Good” day and the proportion of the day spent on subject
administration and also on research. None of the other activities had a statistically significant
relationship with the emotional rating for the day given by the academic, although the proportion of the
day devoted to communicating to students and to service and general administration had a negative
but statistically insignificant impact on the probability of given a “Good” rating for the day. The
effectiveness rating of the day given by the academic had no statistically significant relationship with
the activities undertaken in the day.
Table 2: Results of a probit analysis on the probability of an academic reporting a “Good” day
emotionally
Dependent variable: Probability of experiencing a “Good” day emotionally as an academic
Proportion of day spent on: Coefficient Standard Error
Communicating with students -0.76 0.70
Subject administration 1.97** 0.56
Subject preparation 0.55 0.57
Subject delivery 0.58 0.64
Research 1.19** 0.45
Effectiveness rating of day 1.46** 0.20
**p

Branka Krivokapic-Skoko, Roderick Duncan and Kerry Tilbrook
Note: Proportion of day spent on service and general administration was dropped as an explanatory
variable as time variables sum to one for the day.
5. Discussions and reflections
The results from the analysis of the pilot time diary data have confirmed for these academics many of
the findings made in earlier surveys of academics (Anderson et al. 2002; McInnis 1999; Jensen &
Morgan 2009). Academics have long workweeks with highly fragmented working days spent meeting
the demands from students and administration while attempting to perform their core tasks of
teaching, research and service.
The academic workweek is 37.5 hours per week which is supposedly 7.5 hours per workday. The
participants in the pilot study were found to be working far longer weeks, including weekends and
hours well outside the core day. If the time spent communicating with students is considered part of
the teaching tasks of academics, in a typical day academics were found to be working 6.8 hours per
day to perform their teaching and general administration roles. Consequently, academics can only
meet their research target by working a longer day of 9.3 hours per day.
The academics in the study worked very fragmented days, because the academics frequently
reported engaging in three to six different tasks in a one hour period. The inability of academics to
devote large blocks of time to individual tasks and the constant interruption of activities is similar to
the “vicious work-time cycle” in software engineers’ workdays found by Perlow (1999), where the
fragmentation of activities significantly reduced the productivity of the engineers. Communication with
students, either directly through conversation, phone calls and emails or indirectly through electronic
forum posting and announcements, was found to consume 1.5 hours per day or 16% of the
academic’s total time.
We found that the academics in the study were more likely to give a positive emotional rating for the
day if they were engaged in subject administration or research tasks for that day. Academics working
on weekends were also more likely to give a positive emotional value for the day. A possible
explanation for these findings is that academics are happier when they are engaged in self-directed
tasks, such as conducting their own research or grading papers.
Usage of the time diary method raised the question about reliability of the data collection process,
such as the extent to which participants comply with instructions, particularly if they were completing
daily time dairies retrospectively. As Collopy (1996) noted there are biases in retrospective selfreports
of time use for work –related tasks because there is a tendency for respondents to overstate
the amount of time they spend on a particular task. Self-assessment of the time spent on daily
activities academics spend on research, teaching, administration may be overstated and this has to
be considered in discussing the findings from this research. Obviously, there is the retrospection error
occurring when the paper and pen diary is not at hand (Bolger et al. 2003). We believe that because
of the highly-motivated participants (academics are increasingly concerned and critical of the
workload policies at Australian universities) the data collection was reasonably accurate. For future
research on this topic, electronic diaries (Green 2006) can be used to record the time of entering the
data into daily records therefore verifying compliance of the participants. Also, the experience
sampling method (ESM) which is a signal-contingent method of data collection (Hektner et al. 2007)
may be used to address this problem. Although ESM is an effective research tool for analysing time
use it is more difficult and expensive to administer, and therefore it was not considered practical to be
used in this project. Nonetheless, being aware of the strength of ESM in assessing the quality of
experience at work we will use ESM in future studies to assess academics’ subjective experiences
during workdays.
6. Appendix 1: Sample time diary for partial day
Faculty: How was your day emotionally? Good Ok Poor
Day: How effective were you today? Good Ok Poor
Communicating with
students
By email
By phone/Skype
5am 6am 7am 8am 9am 10am 11am 12noon 1pm 2pm 3pm
204

By
forums/announcements
By face-to-face
Subject administration
Marking assignments
Marking exams
Entering gradesheet
Setting up Interact
Setting up MSI
Moderation
Subject co-ordination
Subject preparation
Developing lectures
Developing tutorials
Developing exams
Developing online
content
Developing materials
Subject delivery
Delivering
lectures/seminars
Delivering tutorials/labs
Research
Research administration
Reviewing/refereeing
Reading literature
Writing
Grant preparation
Meetings for research
Thinking/planning/general
research
Supervision of PhD/DBA
students
Service and general
admin
Committee attendance
Course/discipline
administration
Community/professional
engagement
Professional
development/training
General
admin/internet/email
Travel for ____________
Conversations with
colleagues
Other (include category)
References
Branka Krivokapic-Skoko, Roderick Duncan and Kerry Tilbrook
Anderson D, Johnson R & Saha L (2002) Changes in academic work, DEST Publications, Canberra.
Blackmore J & Sachs J (2007) Performing and reforming leaders: Gender, educational restructuring, and
organizational change, SUNY Press, Albany NY.
Bolger N, Davis A & Rafaeli E (2003) Diary methods: capturing life as it is lived, Annual Review of Psychology 54:
579-616.
Buckholdt, D.R. and Miller, G.E. (2009) Faculty Stress. Rutledge, Taylor & Francis Group, London and New York.
Coady T (ed) (2000) Why universities matter, Allen & Unwin, St Leonards Sydney.
205

Branka Krivokapic-Skoko, Roderick Duncan and Kerry Tilbrook
Coaldrake P & Stedman L (1998) On the brink: Australian universities confronting their future, University of
Queensland Press, Brisbane.
Clark B (1998) Creating entrepreneurial universities: organizational pathways of transformation, Pergamon Press,
Oxford.
Collopy F (1996) Biases in retrospective self-reports of time use: An empirical study of computer users,
Management Science 42(5): 758-767.
Dearlove J (1998) The deadly dull issue of university administration? Good governance, managerialism, and
organising academic work, Higher Education Policy 11: 59-79.
Earley P & Fletcher-Campbell F (1989) The time to manage? Department and Faculty Heads at Work, Nelson,
Windsor.
Green A, Rafaeli E, Bolger N, Shrout P & Reis Hl. (2006) Paper or plastic? Data equivalence in paper and
electronic diaries, Psychological Methods 11 (1): 87-105
Harms T & Gershuny J (2009) Time budgets and time use. German Council for Social and Economic Data,
RatSWD Working paper series, No. 65, Berlin.
Hektner J, Schmidt J & Csikszentmihalyi G (2007) Experience sampling method: Measuring the quality of
everyday life, Sage Publications, Thousand Oaks CA.
Hill JY (2006) Flexible learning environments: Leveraging the affordances of flexible delivery and flexible
learning, Innovation in Higher Education 31: 187-197.
Hull R (2006) Workload allocation models and “collegiality’ in academic departments, Journal of Organizational
Change Management 19(1): 38-53.
Ishii K (2004) Internet use in Japan: A time-diary method. Paper presented at the conference on Information
Statistics of Internet: Measurement: Analysis and Applications, August 19-20, 2004, Hong Kong.
Jarvis P (2001) Universities and corporate universities, KoganPage, London.
Jensen AL & Morgan K (2009) Overload: The role of work-volume escalation and micro-management of
academic work patterns in loss of morale and collegiality; The way forward, National Tertiary Education
Union.
Kinman, G. and Jones, F. (2004) Working to the Limit. Association of University Teachers, UK.
Kitterød RH (2001) Does the recoding of parallel activities in time use diaries affect the way people report their
main activities?, Social Indicators Research 56: 145-178.
McLean, J. (2006) Forgotten Faculty: Stress and Job Satisfaction Among Distance Educators. Online Journal of
Distance Learning Administration, Vol. 9, No 2
McShane, K. (July 2006) Technologies transforming Academics: Academic identity and online teaching.
Unpublished PhD, Faculty of Education, University of Technology, Sydney
Marginson S (2000) Research as a managed economy: the costs. In Coady T (Ed.) Why Universities Matter: A
conversation about values, means and directions, pp186-213, Allen & Unwin, Sydney.
Marginson S & Considine M (2000) The enterprise university, Cambridge University Press, Cambridge.
McInnis C (1999) The work roles of academics in Australian universities. Evaluations and Investigations
Programme Higher Education Division, Department of Education, Training and Youth Affairs.
Morrison M (2007) Using diaries in research, in Briggs A and Colemna M (2006) Research methods in
educational leadership and management, (2nd edn) pp295-317, Sage Publications, Thousand Oaks CA.
Nie N & Hillygus DS (2002) The impact of internet use on sociability: Time - diary findings, IT & Society 1(1): 1-
20.
OECD (1998) Redefining tertiary education. OECD Publications, Paris.
Ogbonna E & Harris L C (2004) Work intensification and emotional labour among UK university lecturers: An
exploratory study, Organization Studies 25(7): 1185-1203.
Pentland W, Harvey A, Lawton M & McColl M (Eds.) (1999) Time use research in the social sciences, Springer,
New York.
Perlow L (1999) The time famine: Toward a sociology of work time, Administrative Science Quarterly 44(1): 57–
81.
Pachnowski, L.M. and Jurcyk, J. P. (2003) Perceptions of Faculty on the Effect of Distance Learning Technology
on Faculty Preparation Time. Online Journal of Distance Learning Administration, Vol. 6, No 3
Ramsden P (1998) Learning to lead in higher education, Routledge, London and New York
Robinson J & Bostrom A (1994) The overestimated workweek? What time diary measures suggest, Monthly
Labour Review, August 1994:11-23.
Robinson J P (1999) The time-diary method; structure and uses, in Pentland W et al (Eds.) Time use research in
the social sciences, pp47-87, Kluwer Academic/Plenum Publishers, New York.
Slaughter S & Leslie L (1997) Academic capitalism: Politics, policies and the entrepreneurial university, John
Hopkins University Press, Baltimore.
Soliman I (1999) The academic workload problematic, HERDSA Conference, 12-15 July, 1999.
StataCorp (2009) STATA: Release 11, Statistical Software. College Station TX: StataCorp LP.
Tennant M, McMullen C & Kaczynski D (2010) Teaching, learning and research in higher education. Routledge,
New York and London.
Vardi I (2009) The impacts of different types of workload allocation models on academic satisfaction and working
life, Higher Education, 57: 499-508.
Wolf A (2010) Orientations to academic workloads at department level, Educational Management Administration
& Leadership 38(2): 246-262.
206

“Why Should I Care?” Facilitating Learning in Higher
Education
Maria Kyriacou and Panayiotis Constanti
University of Nicosia, Nicosia, Cyprus
maria07@cytanet.com.cy
constanti.p@unic.ac.cy
Abstract: Higher education institutions primarily focus on curriculum development for those courses that will best
enable their students to connect with and master the discipline of their choice; and knowledge-creation that will
expand the academic horizon. It is assumed that both teacher and student are well aware of their main role i.e.,
the teacher presents a lecture that is cohesive and comprehendible; the student attends lectures, studies the
material, submits required coursework, and participates in classroom discussions. Besides these roles, another
more implicit aspect influences the actual learning experience. The student forms impressions about the course
and the lecturer, while her/his ability to study and assimilate the course material is impacted by clues garnered
from the lecturer. Subsequently, teacher behaviour, whether intentional or unintentional, can influence the
student’s learning strategies and subsequent academic success or failure. The teacher has to develop strategies
for addressing this issue, not only to eliminate the educational problems derived from both his/her intentional and
unintentional interactive behaviours, but also to leverage this awareness in order to enhance the student’s
academic progress and prospects. Therefore, the teacher needs to introduce and develop strategies that are
perceived by the student as enhancing a caring and respectful learning environment. Utilizing focus group and
individual student interviews, we set out to answer the following six questions: Have you ever had the feeling that
a faculty member did not care about you and your learning in a course? What did the faculty member do or not
do to give you that feeling? What did you do as a result of that feeling? Have you ever had the feeling that a
faculty member did care about you and your learning in a course? What did the faculty member do to give you
that feeling? What did you do as a result of that feeling? The findings suggest that the teacher’s caring
disposition, can have a significant impact on the student’s learning strategies, and inspire her/his engagement
and learning.
Keywords: student, faculty 1 ,teacher, lecturer, instructor, pedagogical caring
1. Introduction
Higher education institutions primarily focus on two main objectives; curriculum development for those
courses that will best enable their students to connect with and master the discipline of their choice,
and knowledge-creation that will expand the academic horizon. (Goldin and Katz 2008; Remler and
Pema 2009). Within this paradigm it is assumed that both teacher and student are well aware of their
main role, that is, the teacher presents a lecture that is cohesive and comprehendible, while the
student attends lectures, studies the material, submits required coursework, and participates in
classroom discussions.
Acknowledging that faculty behaviour influences student learning (Buttner 2004; Burton and Dunn
2005), we set out to investigate student perceptions regarding the “caring” faculty member and the
behaviours which can either enhance or hinder student engagement during faculty-student
interactions. The findings suggest that faculty members’ behaviour can have either positive or
negative impacts on students, thereby capturing their imagination and enhancing their academic and
personal prospects or negative impacts leading to student withdrawal and diminished learning and
development respectively.
2. The literature
2.1 Cooley’s “looking-glass self”
The student forms impressions about the course and the lecturer, while her/his ability to study and
assimilate the course material is impacted by clues garnered from the lecturer. This activity is not
limited to the classroom setting, but according to early sociologist Charles Horton Cooley, it is a
general human behaviour extending to all social settings (Cooley 1902; Howard 1989). According to
Cooley (1902), we develop an image through our interactions with others. As a result, this self image
is not created just by direct reflection of our own personal abilities, but also from our impressions
1 The terms faculty, lecturer, teacher and instructor are used inter-changeably throughout the text
207

Maria Kyriacou and Panayiotis Constanti
about how other people view us. This “looking-glass self” is the result of our social interactions
comprising three phases.
Firstly, we envision how we present ourselves to other individuals such as, teachers, employers,
family and friends, and then imagine how others perceive and evaluate us, for example, as being
smart, extrovert and fun to be with, or dull. As a consequence of these impressions we develop
feelings about ourselves such as, respect or shame (Cooley 1902; Howard 1989), so that students
will perceive themselves as either bright and intelligent, with an ability to study and achieve a
commendable grade, or they may be uncertain of their skills and competencies, being confused and
ultimately disappointed. Classroom interactions impact students’ self-image as they reflect on the
teacher’s evaluation of their performance and their self-evaluation that they are intelligent, hard
working and capable, is either reinforced or rejected.
Cooley’s “looking-glass self” theory has an important implication. The perception about the “self” is not
objectively formed by what others actually think of an individual, but from that individual’s own
impressions of how others perceive her/him. As a result, students may form wrong self images based
on incorrect perceptions of what others, such as teachers may think of them.
2.2 Expectancy theory: The “pygmalion effect”
Rosenthal and Jacobson’s (1968) experiments regarding the positive impact that teacher
expectations can have on student outcomes have been validated and well documented (see for
example Jussim 1991, 1992 and Andrews et al 1997). This self-fulfilling prophecy is based on
Rosenthal and Jacobson’s students’ IQ test results reported to other teachers. The teachers were
informed that five students had recorded unusually high scores, and that these five randomly chosen,
anonymous students would probably “outshine” their contemporaries by the end of the year. When
the class took an IQ test at the end of the year, all of the students had increased their IQ scores, while
the scores of the five “spurters” (nicknamed by Rosenthal and Jacobson) had improved significantly
more than the rest. Rosenthal and Jacobson concluded that the teachers’ behaviour towards and
expectations of their students had impacted overall performance.
2.3 Caring and pedagogical respect
The discourse on student impressions and the self-fulfilling prophecy outlined above have implications
for the faculty-student interaction. Hawk and Lyons (2008) inform us that faculty behaviour, either
intentional or unintentional, can influence students’ academic progress. In order to address this issue
faculty will need to “care about the learning of their students and must respect them” (p. 317). More
specifically Hawk and Lyons inform us that,
...the likely consequences of not doing so are lower levels of effective student learning,
lower levels of course and instructor ratings by students, higher attrition rates in higher
education and more jaded attitudes towards higher education and learning by those who
do manage to make it through.
(2008: 317).
Noddings (1984; 1992; 2003) and Rice (2001) defined caring as the process of helping the other
person develop, and providing an accommodating context for her/him to grow in order to reach her/his
full potential. Mayeroff (1971) cited eight characteristics an individual must possess in order to care:
1. Know the recipient: understanding the other’s needs; having the competency to act in
response to them appropriately; knowing the other person’s strengths and limitations; the
care giver must have “explicit and implicit knowledge” of ways to assist the student to
grow.
2. Patience: allowing the student to develop in her/his own time and manner.
3. Honesty: an essential part of caring; the teacher’s openness to correction in case
she/he makes a mistake; a tendency not to be afraid to learn from her/his mistakes.
Seeing the other as she/he really is, and not as they would like to be or feel it must be.
4. Trust: having the confidence that students will find their own ways to accomplish their
objectives; the ability to provide a learning-friendly climate; to learn what works and what
does not, from students’ reactions.
208

Maria Kyriacou and Panayiotis Constanti
5. Humility: ready to learn more about others and what caring involves; she/he is
authentic; taking pride in a job well done.
6. Courage: the strength and ability to venture into the unknown.
7. Hope: belief that students will grow, and achieve self realization as a result of their
care giving.
8. Alternative rhythms: checks for understanding; utilising a variety of methodologies to
aid comprehension.
(pp. 19-32).
Noddings (1984; 1992; 2003) contribution to the field is significant, demonstrating that caring and
caring relationships are essential aspects of education (Smith 2004). She suggests that the carer has
to gain a better understanding of the person being cared for, and is essential if caring is to happen
because the carer must first determine the other’s personal and physical situation, in order to be able
to decide on suitable action. In order to complete the caring circle, the person being cared for must
recognize that the carer actually cares for her/him, and when this recognition occurs, Noddings (1984)
explains that caring is “completed in the other” (p. 4). If the person being cared for does not recognize
that the carer truly cares, then the process of caring malfunctions and does not have the equivalent
results.
Furthermore, she describes the concept of ethical caring, distinguishing it from natural caring where
the latter occurs when a person engages in an act of caring because she/he wants to, for example,
helping a friend out when this friend is in trouble, as an act of love. Ethical caring occurs when a
person engages in an act of caring not because that person wants to, but because that person “must”
care. In other words, a person would rather escape the effort and emotional distress associated with
helping another person out and do something more pleasant and joyful, but nevertheless that person
would act caringly because she/he judges that being caring is the right way to act in relation to other
people. Natural caring is the base of ethical caring, which is a reflection of the ideal individual we want
to be.
Rice (2001) highlights two key aspects influencing learning, “a sense of belonging and students’
perception of being cared for” (pp.103-104), concluding that “a teacher may be perceived to know
everything about the subject he or she teaches, but if he or she does not act in a caring manner,
students may report learning less from that teacher” (p.105). Hawk and Lyons (2008) summarise the
ethic of care as “a reflective and action-oriented process about learning of the other and
demonstrating relationship behaviour that seeks to recognize, value, trust and develop the other” (p.
320). Pedagogical caring therefore, involves aspects of an ethic of care:
The focus is on the development of the student as a caring human being and the student
as learner and performer…one who needs to attain particular knowledge, gain skills,
develop attitudes and dispositions, and demonstrate competence. Clearly, the instructor
has the opportunity to model and exemplify caring behaviour.
(Hawk and Lyons 2008: 322)
Offering timely feedback, supportive encouragement and treating the student with dignity are
fundamental elements of pedagogical respect which can significantly enhance a student’s
development (Applebaum 1996; Hawk and Lyons 2008).
2.4 Disruptive faculty behaviours
Academic research focusing on negative faculty behaviours that hinder effective learning has
interested a number of scholars Kearney et al 1991; Farley-Lucas and Sargent 2007). Kearney et al
(1991) developed twenty eight categories of faculty misbehaviour, grouped into three dimensions,
incompetence, indolence and offensiveness. According to Farley-Lucas and Sargent (2007)
incompetence includes “going off topic, language differences and accents, lack of interaction, fixing
exam questions during tests, focusing on irrelevant assignments, unclear assignments” (pp. 7-8).
Indolence includes “a general lack of professionalism, making noise in the classroom, using cell
phones, inappropriate use of humour and jokes, apathy and ignoring students, personal quirks,
lateness, eating and drinking in class, not dressing professionally and lack of preparation” (p. 6), and
offensiveness includes “put-downs, unreasonable or arbitrary rules, favouritism or prejudice, verbal
abuse and sexual harassment” (p. 7).
209

3. Methodology and rationale
Maria Kyriacou and Panayiotis Constanti
The literature informs us that faculty need to address the issue of pedagogical caring and respect
systematically not only to eliminate inherent educational problems derived from their unintentional
class behaviour, but also to utilise this knowledge to their advantage in order to enhance students’
academic prospects. A dearth of investigations into the ethic of care in higher education, and, to our
knowledge, none conducted in the private higher education setting of Cyprus, motivated us to mirror
the Hawk and Lyons (2008) study.
According to Morgan (1997: 12), focus groups rely on “interaction within the group based on topics
that are supplied by the researcher,” and this interaction within the group is what distinguishes focus
groups from interviewing a number of individuals at the same time. The main benefit for social
research is that focus groups portray respondents’ “attitudes, feelings, beliefs, experiences and
reactions” (Gibbs 1997: 363) in a manner that would not be depicted by the use of other methods
such as questionnaires. Since the area of pedagogical caring and respect in higher education relates
to groups in classroom interactions, the selection of a focus group was considered as the optimal
choice for our investigation. Focus group discourse can be dominated by an individual or individuals,
or views may not be fully confidential due to peer pressure (Gibbs 1997). For this reason four of the
students whose voice was somewhat muted in the focus group, were also telephone interviewed in
order to mitigate these limitations.
Eleven local students on the MBA programme took part in the focus group, enabling the discussion,
which was tape-recorded and later transcribed, to be conducted in their native language of Greek.
Seven of them were male, and six of the eleven had completed their undergraduate studies abroad.
Their responses were based on their experiences at both undergraduate and graduate level.
The discourse on student impressions via Cooley’s ‘looking-glass self’, and Rosenthal and
Jacobson’s ‘self-fulfilling prophecy’ have implications for the faculty-student interaction. We therefore,
set out to address the following questions (adapted from Hawk and Lyons,2008):
Have you ever had the feeling that a faculty member did not care about you and your learning in a
course?
What did the faculty member do or not do to give you that feeling?
What did you do as a result of that feeling?
Have you ever had the feeling that a faculty member did care about you and your learning in a
course?
What did the faculty member do to give you that feeling?
What did you do as a result of that feeling?
4. Findings and discussion
4.1 Differences in male and female accounts
When the accounts of respectful instructor behaviors were examined by student gender, we found
that females were more vociferous regarding tutor support and subsequent caring behaviours
informing us that they considered it a fundamental element of effective teaching. In contrast, several
of the male students expressed the view that where support was not forthcoming they would cope on
their own, by investing more of their time in independent study and additional reading.
4.2 Negative faculty behaviours
All of the students had experienced a faculty member who appeared not to care about their learning
(Q 1 and Q 2) which was manifested in uninteresting lectures, e.g. “lack of interaction,” “just reads the
slides” or “sits behind the desk and lectures.” These activities led to boredom and sleepiness making
it difficult to remain focused, particularly on the evening courses. The fact that some faculty were “not
giving an enthusiastic lecture” was perceived by students as “disregard for both the course and
themselves.” These behaviours echo the incompetence paradigm coined by Kearney et al (1991).
Giving the impression of being unapproachable or “keeping a distance” is a more evident indication
according to students as to whether the lecturer cares about them or not, while lecturers discourage
210

Maria Kyriacou and Panayiotis Constanti
participation when providing feedback that is brief and curt. It is not unusual for some faculty
members to become defensive if students’ opinions are contrary to their own, or “when asking if
students understood, and if just one or two students reply that they did, the lecturer will not bother
with the rest of the class, and go on to the next topic.” Along similar lines, a behaviour described as
lack of respect for students is not listening carefully enough to understand what the student’s question
was and then to provide an answer on an unrelated topic. The student asking the original question
reported feeling frustrated; the rest of the class forms a negative impression, while the lecturer is
unaware that something is wrong if students decide not to speak up.
Being unapproachable is an indication of whether the lecturer cares about them or not, and is
manifested in behaviours such as, keeping a distance from students, or failing to encourage student
participation, and in spite of this, some lecturers are accused of favouritism. Comments to emerge
from the focus group included:
“Lecturers prefer a few good students and as a result these students get better treatment
than the rest of the class,”
“Lecturers tend to provide better answers to questions when coming from preferred
students,”
“Answers are more complete and more arguments are put forward to make the point
clear,”
“The lecturer checks whether the student understood or not, and puts more effort in
making him/her understand.”
Often, queries from weaker students are either answered very briefly or simply ignored, begging the
question, does the lecturer consider them unimportant? We were also told that lecturers also provide
better evidence when they disagree with a point raised by a favoured student, as they attempt to
redirect them in the correct direction. For the weaker and less favoured individuals, the lecturer’s
response tends to be shorter and curt. Preferred students are believed to be given more opportunities
to speak in class than the rest of their classmates, perhaps because lecturers are interested in these
students’ opinions or believe that their answer will be correct and well supported. This strategy may
help the rest of the students better understand the issue being discussed. Whatever the reason,
students reported that there would be nothing wrong with such an action, if only other students were
encouraged to speak more often too. Kearney et al (1991) grouped this type of behaviour under
“offensiveness,” a category that also includes “put-downs,” “humiliating students in front of the class
insult and publicly embarrasses them” (p. 322).
“How many books did you read on this subject?”
“I’ve read several, so you do not know more than me on this subject.”
“This is how it is because I say so.”
The foregoing comments illustrate the rare cases of putting students down, which is
described as not having patience with weak students who are a little slow on the up-take,
or using irony and sarcasm for wrong answers or different views. Our focus group was
adamant that the lecturer should provide guidance and help whenever it was sought and
to not “play favourites.”
Regarding unprofessional behaviour outside the classroom, the majority of students held the opinion
that some faculty members (intentionally or unintentionally), discourage students from visiting them in
their office hours. Factors such as lecturer approachability and concern about the quality of answers
provided in class, as well as treating students with respect or not, determine whether a student will
choose to visit a lecturer on her/his office hours. Students who tried to visit some faculty members,
later described as not caring, reported cases of what they described as “unprofessional behaviour”.
Visiting faculty members during their office hours to discover that they were not there was not
perceived by students as important, as they understood that she/he had been called away urgently.
Continued absence, however, was not justified, as was not returning phone calls, not getting replies to
their emails, or getting late responses. For example, a student had emailed with an examination query
only to get a reply a few days after the examination date.
211

Maria Kyriacou and Panayiotis Constanti
4.2.1 Coping
Regarding Q 3: “What did you do as a result of that feeling?” the majority said “they did nothing for
fear of retaliation”. However, there was a unanimous view that they would “give low scores to the
faculty member on the class evaluation form.” The main motivator for carrying on regardless was
economic: “I’ve paid my fees and I can’t afford to stop,” and then suggesting that they would continue
by adopting learning strategies, “I spend more time learning at home,” and “I don’t feel comfortable
asking the lecturer for help”. Seeking peer help and advice was another coping strategy employed by
students who reported having a small or extended network of friends in the class or at a more
advanced level. This coping strategy, invaluable for some students, could not be employed by
freshmen or students who did not have a supportive network of friends at the university. Finally,
almost all students would avoid taking any more classes with the same faculty member, unless they
had no choice.
4.3 Positive faculty behaviours
Responses to Q 4 and Q 5 revealed that all of the students in the focus group reported that they had
come across a faculty member who expressed a genuine interest in them and their learning of the
course, at least once. They also revealed constructive faculty behaviours that inspire students and
affirm the work of Hawk and Lyons (2008).
The first faculty behaviour perceived by students as caring is giving enthusiastic lectures, e.g. “You
can tell by his manner that he is enthusiastic about his topic”, or “She loves what she does”. Admitting
fallibility was considered a positive behaviour by our focus group. Several students told us that “it’s
better if the teacher says I don’t know and I’ll get back to you on that,” accepting the fact that the
lecturer cannot be “all-knowing” and preferring an honest response rather than to be “fobbed off” with
a flimsy answer or excuse. Lecturers who are considered by students as more easygoing and
approachable make them feel more relaxed, providing a safe environment for asking questions in the
classroom. Students are asked if they have any questions, but the way they are treated after posing
the question (appraisal respect) is what is critical. In the words of one student, “a caring lecturer
listens to the question carefully, and if the student can’t express the question clearly, the lecturer
would repeat the question in his own words to see if the question had been understood”. The focus
group reported that a lecturer who made a difference to them would answer even an elementary
question in an objective manner. There were occasions where “a lecturer asked to see a student
individually to give him extra help”. This “feeling-safe-to-ask-questions” environment is, according to
our focus group, of vital importance in forming the impression that the lecturer cares about them and
their learning:
“They are guiding the student…the person with the knowledge is not necessarily a good
instructor…there are many with good knowledge but cannot transfer the knowledge…to
have communication skills maybe, so that the student feels comfortable to communicate,
to go to the instructor and to communicate, this is very important.”
Care is also manifested in whether lecturers know the students’ names as is the belief in their
potential. According to the students, “some teachers don’t even know our names; how do they know
me if they don’t even know that?” They reported that they try harder in order to fulfil this belief, which
is also in accordance with Rosenthal and Jacobson’s “self-fulfilling prophecy” (1968).
Student care also comes in the form of lecturers’ sincere interest regarding the class’ understanding
and learning, and is manifested when she/he conscientiously enquires as to whether the lesson or
topic has been understood, in order to then repeat the material by utilizing other, more appropriate
examples and illustrations:
“The best thing for me is to be able to interact with the instructor, to ask questions and for
the instructor to respond. If I read a text I still need to be able to talk with the instructor.
There are questions I need to ask, to be given different examples because I may not
understand the ones in the book...when I was given something to read I was able to ask
the teacher for help and he gave me further clarification and examples,”
“It’s much easier for me to learn if I can understand the material…with some real life
examples combining theory and practice...it’s better,”
One of the respondents intimated that the learning experience is collaborative:
212

Maria Kyriacou and Panayiotis Constanti
“You should go to class prepared so that you can discuss something that you don’t
understand. The effective instructor tells you from the beginning what to expect, what the
course outline is and to follow it.”
The use of humour is perceived by students as an effective means of breaking the ice, and is
especially valued in three-hour evening classes where students reported often being tired and having
difficulties maintaining their focus. They reported that inspiring lecturers employed a periodic use of
humour to get their attention, and then redirect it to the course material. Our focus group was
unanimous on this, refuting Farley-Lucas and Sargent (2007), who suggest that students did not
consider the use of humour as appropriate faculty behaviour.
4.3.1 Student responses to these behaviours
Responding to Q 6: “What did you do as a result of that feeling?” our sample reported:
“I can ask questions without feeling scared of the teacher”,
“I enjoy the class and the course more than others”,
“I am learning and understanding more”,
“I’ll find out if he’s teaching any other classes and enrol in his class”.
5. Limitations
We acknowledge that our data is both limited and culture bound by being based in the Cypriot higher
education environment, which prohibits the findings’ generalisability. In addition, we acknowledge that
the paper focuses only on the teacher’s inputs and that educational outcomes are the result of
student/teacher collaboration; this relationship assumes that both actors play a significant role in the
outcomes’ achievement. We suggest that research on the manner in which students behave within
this context, and within other cultures, could provide further insight into this relationship.
6. Implications and concluding thoughts
The lecturer-student interaction is a complex web of multi-faceted elements worthy of consideration
by each stakeholder concerned about caring in higher education. Our investigation suggests that
students consider a number of factors associated with the educational process when evaluating the
overall learning experience. Pedagogical caring and respect are an integral part of the students’
higher education experience which can either enhance, or impede their learning and development.
Faculty members delivering “enthusiastic lectures,” that provide a “safe learning environment” free
from favouritism and “put-downs,” who are approachable, get to know their students and available in
order to provide timely and constructive feedback have a special place in the hearts of students.
Those who consider their lecturers as purveyors of pedagogical caring and respect will be more
engaged and have higher aspirations, exhibiting a mature, academic approach to their learning, with
realistic outcomes and objectives. Alternatively, the disengaged student is less likely to participate or
withdraw entirely from the entire learning experience. Faculty members’ caring behaviour is likely to
have a significant impact on their students such as, capturing their imagination and enhancing both
their academic and personal prospects.
Even though our investigation and analysis has focused on the teaching fraternity, we do not exclude
the significant input and impact that student behaviours have on the overall learning experience.
Therefore, we believe that our investigation brings to the fore further, and perhaps more pertinent
questions regarding the tripartite, student-faculty-management relationship requiring further enquiry
and investigation.
Last but not least, operating in an increasingly corporate paradigm, and driven by new accountability
and performance systems, the teaching profession is witnessing the erosion of academic freedom
(Hebson et al 2007,) where education administrators espouse ideals such as “quality in education”
with the emphasis on “the student education experience”. Meanwhile, in the contemporary higher
education classroom, where the student has become the customer, education managers’ “theories-inpractice”
run counter to their espoused, (lofty?) goals. How do we reconcile the oxymoron of
management’s obsession with league tables and targets that include increasing numbers of student
cohorts, and their inherent teaching challenges; and where faculty tenure decisions are based
213

Maria Kyriacou and Panayiotis Constanti
primarily on quality and quantity of research output, while teaching excellence is being subjugated to
the basement of administrators’ lists of priorities?
References
Andrews, T., Wisniewski, J. and Mulick, J. (1997) “Variables influencing teachers decisions to refer children for
school psychological assessment services”, Psychology in Schools, Vol 34, No. 3, pp 239-244.
Applebaum, B. (1996) “But That Is Not What I Mean Criticizing With Care and Respect”, Philosophy of Education,
[online], http://www.ed.uiuc.edu/EPS/PES-Yearbook/96_docs/applebaum.html [Accessed 2 November
2009].
Burton, B. K. and Dunn, C. P. (2005) “The Caring Approach and Social Issues in Management Education”,
Journal of Management Education, Vol 29, p 453.
Buttner, E. H. (2004) “How Do We "Dis" Students? A Model of (Dis)Respectful Business Instructor Behavior”,
Journal of Management Education, Vol 28, p 319.
Cooley, C. H. (1902) Human Nature and the Social Order, Scribner, New York.
Farley-Lucas, B. and Sargent, M. (2007) “’Checking-Out Mentally’: Faculty Misbehaviours and Impact on
Students”, Improving University Teaching 2007 Annual Conference, Jaen, Spain, [online],
http://www.iutconference.org/2007/pdfs/Papers/Farley-Lucas.pdf [Accessed 15 November 2009].
Gibbs, A. (1997) “Social research update, Issue 19: focus groups”, Department of Social Medicine, Bristol
University, pp 363-364.
Goldin, C. and Katz.L. F. (2008) The Race Between Education and Technology, Belknap Press of Harvard
University Press, Cambridge, MA.
Hawk, T. F., and Lyons, P. R. (2008) “Please don’t give up on me: when faculty fail to care”, Journal of
Management Education, Vol 32, No. 3, pp 316-338.
Hebson, G., Earnshaw, L. and Marchington, L. (2007) “Too emotional to be capable? The changing
nature of emotion work in definitions of ‘capable teaching’”, Journal of Education Policy, Vol 22,
No. 6, pp 675-694.
Howard, M. C. (1989) Contemporary Cultural Anthropology, 3rd ed. Scott, Foresman, Glenview, IL.
Jussim, L. (1991) “Social perception and social reality: A reflection-construction model”, Psychological Review,
Vol 98, pp 54-73.
Jussim, L. (1992) “Teacher Expectations II Construction and Reflection of Student Achievement”, Journal of
Personality and Social Psychology, Vol 63, No. 6, pp 947-961.
Kearney, P., Plax T.G., Hays L.R. and Ivey, M.J. (1991) “College Teacher misbehaviors: What students’ don’t like
about what teachers say or do”, Communication Quarterly, Vol 39, pp 309-324.
Mayeroff, M. (1971) On Caring, Harper Perennial, New York.
Morgan, D.L. (1997) Focus groups as qualitative research. 2nd Ed. Sage, London.
Noddings, N. (1984) Caring, a feminine approach to ethics & moral education. University of California Press,
Berkeley.
Noddings, N. (1992) The Challenge to Care in Schools: An Alternative Approach to Education, Advances in
Contemporary Educational Thought series, Vol 8, Teachers College Press, New York.
Noddings, N. (2003) Happiness and Education, Cambridge University Press, Cambridge.
Remler D.K. and Pema, E. (2009) “Why do Institutions of Higher Education Reward Research While Selling
Education?” National Bureau of Economic Research, Working Paper No.14974, May, [online],
http://www.nber.org/papers/w14974 [Accessed 2 December 2009].
Rice, C. (2001) “Teacher’s level of care”, Education, Vol 122, No. 1, pp102-106.
Rosenthal, R., and Jacobson, L. (1968) Pygmalion in the classroom: Teacher expectations and pupils intellectual
development, Holt, Rinehart and Winston, New York.
Smith, M. K. (2004) “Nel Noddings, the ethics of care and education”, The encyclopaedia of informal education,
[online], www.infed.org/thinkers/noddings.htm [Accessed 30 October 2009].
214

Establishing and Maintaining Focus in Your Research:
Promoting the use and Effective Implementation of Focus
Group Methodology in a ‘Real-Life’ Research Study
David Lamb
Edith Cowan University, Joondalup, Perth, Western Australia
d.lamb@ecu.edu.au
Abstract: Within the academic research literature there is a lack of information and guidance on utilising focus
group methods, yet it is a research method that is highly adaptable and can be used in combination with a range
of other methods at different stages of the research process. In this paper, the use of focus groups in research
will be discussed, with reference to a focus group involving pre-birth and post-birth couples to examine how
significant life events, such as ‘first time pregnancy’ and ‘first time parenthood’, impacted upon their access,
opportunity and experiences of leisure as an individual, couple and as a family.Focus groups are an underutilised
qualitative tool in business/management research and during the paper a range of benefits will be
highlighted, with regards to gaining information about participants’ experiences and views on a specific topic.
Focus groups are an adaptable research tool and relatively easy to organise and can be used at all stages of the
research process. It is a qualitative method of enquiry that is similar to group interviewing, but most importantly
the interaction between focus group participants is the key to producing insightful data. Focus groups help
participants to express their own perceptions, attitudes and viewpoints about a topic in which they are personally
involved and in which they have experience and knowledge. Furthermore, focus groups allow participants to
discuss a specific issue in which they have some common interest, with like minded people in a non-threatening
and relaxed environment. As a researcher, it is important to be aware of some of the ‘pitfalls’ of using focus
groups and strategy to ensure effective management and be clear in your purpose for using this method.
Preparation is critical, as there are many tasks that need completion before the focus group meeting can
commence. Similar to any other qualitative methods of research, focus groups can be used for a variety of
purposes, but the researcher needs to be mindful off the advantages and disadvantages of using this method, in
relation to other methods of research. In this paper, the important role of recruiting participants, formulating a
strategy for framing questions and the role played by the moderator in managing the focus group discussion will
be emphasised. Suggestions will be given, to highlight a number of ways in which the data produced can be
analysed and utilised for reporting purposes. In concluding, the key issues outlined in the paper will be re-visited
and the reader will be encouraged to seriously consider using this method as part or in combination with other
qualitative and quantitative methods.
Keywords: focus groups, group interviewing, methods, research, family leisure
1. Introduction
This paper is based upon research undertaken that has involved the use of focus groups as a key
methodological tool in combination with other research methods in exploring various dimensions of
family leisure in New Zealand. In addition, the literature on focus group methodology has been
evaluated in determining a number of key issues in focus group research, that are worthy of further
exploration. As a result, with specific reference to focus groups, this paper will: justify the use of
qualitative method; define what they are; why they are used; what challenges are involved in using
them; consider the planning and recruitment requirements; outline the questioning strategy; provide
advice on managing and moderating them and suggest a range of evaluation techniques to analyse
the data.
2. The qualitative method
Qualitative research stresses the importance of the participant’s viewpoints and use of this method
encourages focus group participants to describe and explain their experiences in their own words.
Becker (1998) notes, that in order to understand the behaviour of research participants, you must
capture their point of view. One way of achieving this, is by adopting a qualitative framework.
Qualitative research is much more concerned with in-depth understanding and detail, where there is
evidence of considerable diversity in research use of methods and philosophical perspectives (Denzin
& Lincoln, 1994). My methodological approach in using focus groups in research is associated with
interpretive design involving cognitive and effective components to encourage the research
respondent to be active in the research enquiry. Jones (1997) indicates that this avoids adhering to
ideas pre-determined by the researcher. The diversity of approaches, epistemologies and
understanding associated with qualitative research offer a useful range of methods to research
peoples’ views and opinions (Tesch, 1990).
215

David Lamb
My focus in the research conducted on family leisure was concerned with in-depth analysis of a
specific issue in a specific locality in what Patton (2002) refers to as centering the research. My
prerogative was to understand the context and real meaning of family leisure by capturing and
understanding the individual’s perspective to secure what Denzin & Lincoln (1994) refer to as ‘rich
qualitative data’. The primary motive in using focus group method was to understand and see the
world through the eyes of those being studied, working primarily in a qualitative style, interpreting and
defining everyday issues and settings in novel ways utilising an ethnographic context and approach
(Morrison 1986). It is acknowledged that some researchers would argue that the downside to
qualitative research is that closeness to the research participants could potentially lead to bias
(Creswell, 1998). In order to avoid research bias within the focus group research involving families
and their leisure, the participants were encouraged to speak freely from their own perspective and in
their own words. Also, a series of prompting questions were used with the purpose of encouraging
them to talk and discuss issues, and their words were recorded in full using a digital audio device
(Powell et al, 1996).
3. What are they?
Focus groups are an underutilised method and have been defined in many different ways depending
on the research orientation adapted. Krueger (1988) describes focus groups as a form of organised
discussion involving a number, where people are concerned with human attitudes and perceptions
relating to concepts, products, services or programs developed in part by interaction with other
people. Whereas, Powell et al (1996) define focus groups as a form of collective activity, that
capitalise on the interaction within a group. Goss & Leinbach (1996) offer a more generic definition
describing focus groups, simply as a social event and Morgan (1997) argues that focus groups are
just another form of group interviewing, whereas Merton (1987) is more concerned with focus group
technique in describing them as a set of procedures for the collection and analysis of qualitative data.
The definition adopted for the pre-birth and post-birth study on family leisure, could best be described
as an organised discussion with a selected group of people on a specific topic, with the aim of gaining
several perspectives on the same topic.
4. When and why are they used?
The use of focus groups in contemporary research can be classified into three types. They can be
used as the primary means of data collection or as preliminary data in a primarily quantitative study or
be used in multi-method research to add to data through other qualitative methods such as
observation and interviews. Therefore, focus groups can be used for a multiplicity of purposes,
including at the commencement and at the end of a research project and as a result are highly flexible
and can be used throughout the research process. Focus groups allow for probing to ascertain clarity
and comprehension and generate a lot of data in a relatively short space of time. Compared to most
other research methods they require minimal expense and do not consume a lot of time. During focus
groups the facilitator (moderator) is able to stimulate dialogue and elicit a multiplicity of views on a
similar topic and uncover potential new insights and ‘fresh’ ideas on a research topic or issue,
previously undiscovered. Within, a focus group discussion it is possible to uncover attitudes, feelings,
beliefs and experiences direct from those participants involved, within a context they understand and
which has meaning to them.
5. Planning and recruitment
Focus group participants can be recruited through a number of sources and in the study on family
leisure, participants were sourced from personal contacts, antenatal classes and parental coffee
groups and people involved in these groups encouraged their friends to take part, referred to by
Kruger (1988) as the ‘snowballing’ technique. Before attending the focus group meeting, participants
were sent a research information sheet explaining the purpose and objectives of the research, which
is considered good practice in focus group research (Tolich & Davidson, 1999). The information
provided detailed the nature of their expected involvement and information detailing how the data
would be collected, analysed and utilised for research purposes only. To record the conversation
during the pre-birth and post birth focus group studies, a digital recorder was placed in the middle of
the group and seating arranged in a circle format, to encourage participation by all (see figure 1,
below).
216

Note-taker
Participant
Participant
Participant
Figure 1: Room set up for focus groups
David Lamb
Moderator
Digital
Recording
Device
Participant
Participant
Participant
Participant
Before organising a focus group meeting, it is advisable to compile a checklist with the intention of
better planning and monitoring the focus group, that helps the researcher deal with eventualities (see
table 1, below). During the focus groups meetings with pre-birth and post-birth couples, the main
concern was to put people at ease, to make them feel comfortable and provide a welcoming
environment to stimulate conversation by asking them a limited number of questions, (as indicated in
table 2, below) During, the focus group meetings, my role as moderator was to encourage discussion
by firstly explaining the purpose of the research and then facilitate what Stewart & Shamdasin (1990)
refer to as, someone else’s discussion and interaction. The role adopted encouraged conversation to
flow, but at the same time allowed participants to contribute answers to the questions raised, in any
way they felt appropriate. This required effective listening skills and direction of the discussions and at
the same time avoiding pre-judgment of feedback or indication of strong approval or gestured
agreement (Morgan, 1997). This was a difficult and challenging role to play and took great powers of
concentration. The key role of a moderator, according to the focus group literature is to facilitate a
warm and friendly environment to make participants ‘feel at home’ to create an environment where
conversation flows, by stimulating interaction amongst the group by encouraging all participants to be
involved in building rapport. Essentially, this requires a moderator with experience, effective
communication and listening skills and with the ability to listen and think at the same time and adapt
to the situation at hand.
In order to focus my attention on the moderator role during the pre-birth and post-birth focus groups,
an experienced note taker was assigned to take detailed notes of the proceedings, just in case there
were any technical problems with the recording equipment and to allow the facilitator to focus on
stimulating discussion within the group. The note taker was discreetly located behind the moderator to
observe and take notes, and used shorthand to detail the contributions that participants made at the
meetings. This enables the moderator to focus attention on asking key questions and encouraging all
participants to contribute ‘good conversation’ (Krueger & Casey, 2000).
Table 1: Checklist for focus group studies, adapted from Krueger & Casey (2000)
CHECK ITEM
1 Contact participants by phone two weeks before focus group
2 Send each ‘likely’ participant research information sheet
3 Once acceptance to engage, send details of focus group location and time process.
4 Give participant reminder phone call the day before the session
5 Questions should flow in a logical order
6 Key questions should focus on the critical issue (family leisure)
7 Limit the use of ‘why’ questions
8 Use think back questions if needed and probe and clarify if necessary
9 Logistics - Room should be appropriate for group size and comfortable with seats/tables
10 Arrive early to arrange the room to encourage all participants to contribute to discussions
11 Check background noise, is not an issue when recording
12 Have name labels for each participant
13 Site digital audio recorder in a suitable location (in the middle of group)
14 Don’t give comments to signify approval and avoid giving your personal opinions
15 Practice questions and know them ‘off by heart’ and
16 Seat experts and talkative participants next to moderator
17 Seat shy and quiet participants directly across from moderator
18 Bring spare audio recorder and spare memory card in case of problems
217

David Lamb
The focus groups were recorded using a digital audio device and within a short period of time, the
audio files were transcribed verbatim in full, into a word document file for further analysis. Once
printed, the transcripts were read several times, in order to firstly identify and then check categories or
key themes which arose from the data within the context of this study.
6. Questioning strategy
In designing questions for a focus group study, a number of considerations need elaboration and
consideration. Within the focus group literature, it is suggested that only 4 or 5 questions should be
used and memorised by the moderator, as this allows for ‘snowballing’ of ideas. Prompting can be
used to elicit further information or clarification by the moderator, but as in all good qualitative
research practice, the questions must first be piloted. In designing questions words such as WHAT,
WHERE and HOW should be used, whereas the word WHY, should be avoided as this makes people
defensive and close up. In terms of sequencing questions, the moderator should move from the more
generic to the more specific and finish off, with an ending question. Therefore, questions should be
sequenced, logical, pre-determined and open-ended to allow for flexibility, to encourage people to
reply to the questions asked in some detail. The first question is best designed to encourage informal
conversation within the group, acting as an ‘ice breaker’ to encourage people to talk with each other
(Guba & Lincoln, 2005). This leads to a number of more focused questions and transition questions to
introduce each topic of the focus group study. These questions can then be facilitated by the use of
probes and pauses.
In the qualitative literature, this is referred to as ‘open questioning’ (Morgan, 1997) which provides an
opportunity for participants to respond, with less restriction (Gubrium & Holstein, 2006). For example,
the question, “Can you tell me about your leisure?” used in family leisure research with pre-birth and
post-birth couples was an exploratory question that elicited a wide range of responses, but was also
useful to introduce the research topic and encouraged people to talk freely during the rest of the focus
group meeting (see table 2, below). This is an effective strategy In terms of questioning style, as a
funnelling technique, where generic questions are asked first, before more specific questions are
covered, after which, more generic questions are asked. The same questions were put to both the
pre-birth focus group and the post-birth group with the exception of one question, which asked the
pre-birth group to discuss their leisure pre-pregnancy and during pregnancy, whereas the post-birth
group were asked about their leisure before and after they became parents. They were also asked
about their expectations and experiences of pregnancy and early parenthood (just post-birth). The
questions, highlighted in table 2 below, acted as probes, to stimulate discussion, where the key role of
the researcher was one of facilitator to encourage informality and talking with purpose (Grbich, 2007).
Table 2: Focus group questions (family leisure research)
Question 1 Can you tell me about your leisure?
(Probe for detailed feedback, reasoning and understanding of concept)
Question 2 As a couple and an Individual is there a difference between your leisure now and before
you had children? (Probe how and in what ways?)
Question 3 As a couple and an Individual how has having children impacted upon your leisure?
(Probe how and in what ways)
Question 4 Are there any important factors that restrict and constrain your leisure?
(Probe, why? Or why not? Some factors may be more important than others?)
Question 5 Are there any important factors that restrict and constrain your leisure?
(Probe or re-phrase question, if necessary)
7. Challenges involved
Within the qualitative research literature, a number of challenges have been recognised that
researchers must overcome in using focus groups. Focus groups can be time consuming as they
require significant amounts of time in pre-planning and there are practical difficulties in assembling
and recruiting participants. Once assembled, participants involved may offer little input or they say
virtually ‘nothing’ and look disinterested or in some cases, an individual within the group might
become too talkative. As a moderator, it is important to encourage participants to ‘air’ their views, but
be wary of a participant dominating proceedings. On the contrary if a participant looks bored and
disinterested, the moderator must try to engage them. Group rules established at the outset should
help clarify that all participants should have the opportunity to contribute and that anything they say
will be valued and could be important.
218

David Lamb
Avoiding bias can be problematic, as this requires an experienced moderator to consider the context
and culture of the group. Data interpretation can be difficult and tedious, as so much data is produced
and the data may not apply to the target population or cause difficulties, when generalising to the
‘whole’ population. Assembling the focus groups can take a lot of time, effort and energy and although
participants may initially give their consent to take part, they still need further encouragement,
persuasion and reminders. To avoid this happening in the family leisure research project regular email
contact was maintained with participants and shortly before the focus groups took place, a number of
personal phone calls were made, the day before the focus group took place, to re-affirm their
involvement and to encourage their full commitment to the study.
During, the recruitment stage of the focus group study on families, many practical problems have to
be overcome. For example, identifying a suitable venue, time and date to suit everybody can cause
difficulties and requires extensive consultation and negotiation and as a result, can be time
consuming, but is essential. In terms of the family leisure study, many couples had to arrange
babysitters for the evening of the focus group studies so that both partners could be present. Many of
the post-birth couples involved had very young children, under the age of two years and chose to
bring their child with them, which caused some difficulties with noise levels in recording their input
with the digital audio device. Some participants had to be contacted on several occasions to confirm
their attendance, especially couples in the pre-birth focus group who were at the latter stages of their
pregnancy. Pre-birth couples in their third trimester and post-birth couples who had recently given
birth had difficulties in making time and space to be involved in the focus group studies. As a result, it
took a lot of time and negotiation to secure a time that suited all couples. However, through regular
and constant communication with participants their involvement was achieved. Male partners involved
needed more encouragement and persuasion, whereas most females were keen to be involved.
Within, the research study involving families a number of focus group participants were worried about
the confidentiality of their input, but reminding them of how the data would be managed and
accessed, served to put their minds at rest and helped re-confirm their continued involvement. Some
individuals were concerned that they might be discussing highly personal and sensitive matters with
people they might not know in the focus groups. Consequently, some individuals needed reassurance
of the processes involved and the fact that the research findings were confidential. For this
purpose, they were given further guidelines on how the focus group conversations would be managed
and were reminded that their comments could not be identified in the write up of the discussions.
Furthermore, if need be, the focus group participants could view the focus group transcript before it
was finalised. These assurances seemed to put their mind at rest and more importantly encouraged
their full involvement. Also, a number of individuals questioned the benefits of their involvement in the
family leisure study, but a number of key informants used their influence, to secure their continued
involvement in the focus group studies. At the meetings with both pre-birth and post-birth couples, the
intention was to put them at ease, to make them feel comfortable and provide a welcoming
environment to stimulate conversation by asking them a limited number of questions, as indicated
earlier in table 2. In focus group studies it is important to encourage discussion by firstly explaining
the purpose of the research and then facilitating what Stewart & Shamdasin (1990) refer to as,
someone else’s discussion and interaction. The moderator role here is to encourage conversation to
flow, but at the same time allow participants to contribute answers to the questions raised, in any way
they felt appropriate. This requires effective listening skills and direction of the discussions, but at the
same time the need to avoid pre-judgment of feedback or indication of strong approval or gestured
agreement (Morgan, 1997). This is a difficult and challenging role to play and takes great powers of
concentration.
One of the methodological issues that potentially limit the use of focus group studies relates to the
nature of focus groups. Although, they often work well to gain a foothold into a research subject and
can help indicate the way forward in terms of issues and topics worthy of further exploration at the
interview stage, they may not always, yield a positive dynamic. As Long (2007) has reported, some
individuals within the focus group may feel uneasy and intimidated within the group, and as a result
feel inhibited from expressing their true feelings and emotions. Also, they may be unwilling to express
perspectives that are divergent to the majority opinion. As, a note taker was utilised during the focus
group studies, conducted with families, it was then possible to focus on the role of the moderator and
involve all participants in dialogue and at the same time manage the conversations to ensure that
everybody present, had their say. This role is crucial in to facilitate all those involved to be active and
motivated to contribute to the discussions that unfold. Nonetheless, as a moderator, one should be
219

David Lamb
wary of personal involvement and be aware when the group needs stimulating to talk and when to
stop. If a topic has reached saturation point and the same content is being repeated. The key purpose
of the focus groups studies with pre-birth and post-birth couples was to ensure that all voices were
heard and consensus on an issue did not lead to participants experiencing a lack of conviction and
confidence in what they had to say.
8. Evaluation/analysis of data
The focus group data for the pre-birth and post-birth study on family leisure was analysed using the
principles of grounded theory and involved firstly reading the data and then coding the data through
three key stages, using the ‘words’ of the participants or as Currie (2004, p.228) remarks
“interpretations of their own social reality”. My first task in analysing the data was as Long (2007,
p.146) suggest, “to read everything you have and then read it again, in order to familiarise yourself
with the data”. The first stage of this process involved ‘open coding’ where the dominant themes were
highlighted within the text (transcript) and given a title. This stage of analysis is also referred to as
‘cluster coding’ (Glaser & Strauss, 1967). The second stage involved ‘axial coding’ or ‘pattern coding’,
where the themes from stage one were collapsed into a smaller number of themes. This was an
interpretive process, where the themes or patterns of data were matched together. The third and final
stage of analysis involved selective coding, which meant that a smaller number of themes identified at
stage two were selected for analysis purposes. The selective codes (themes) then formed the basis
for structuring the findings. Once, the data from the focus groups had been fully collated, participants
were given the opportunity to read and check the feedback they gave. This is a useful strategy for
participants to check the accuracy and validity of the data and is referred to as ‘member checking’ in
the qualitative literature (Patton, 2002).
Analysis of focus group data takes place, whilst the group is in session, where the moderator is
looking for inconsistencies in the data and helping participants comprehend what is being said by
summarising and probing for understanding. Within hours of the focus group, the moderator is
encouraged to listen to the audio recording with the field notes and within days commence the task of
transcribing the data to look for trends and patterns in the data. In analysing the data, the moderator
needs to consider the words in terms of their meaning, variety and phrase used. The context of what
is said needs interpretation and moderators should be mindful of the extensiveness of comments and
the passion and emotiveness displayed and in writing up the data, quotes become a power toll of
expression to exemplify key points.
9. Conclusion and recommendations
The focus group study with pre-birth and post birth couples was extremely useful in deciphering
further lines of enquiry and helped determine specific themes that would require further investigation
at the interview stage of the research study on family leisure. In addition, as Kitzinger (1995) and
Krueger (1988) have both noted, the focus group method is a useful vehicle to explore a range of
ideas that become apparent during the literature review. This focus group study allowed participants
to be involved in a manner they wouldn’t commonly experience, to uncover and create knowledge
unique to experiences of family leisure. The focus group study generated rich and versatile data,
which would be difficult to obtain using other research methods. Although, the planning and
organising of the focus groups required a lot of time and effort, regular and constant communication
was the key to elicit their involvement in the study they. During the focus group studies participants
were also, able to build on one another's responses and come up with ideas they might not have
thought of, in an interview situation and provided and produced ‘rich’ data in participants' own words,
which helped the researcher develop deeper insights into different aspects of family leisure.
As a result, of this study, it is recommended that the number of focus groups used, be extended to
include couples at different stages of pregnancy and at different stages of parenting throughout the
family lifecycle. In the future, researchers in family leisure need to encapsulate the lived experiences
and complexities of family life for all family members. Obviously, that includes children, but in this
study for ethical and practical reasons children were too young to be involved. Nonetheless, seeking
the opinions of older children may help in understanding how leisure is planned and practiced and
how decisions about family leisure are made and processed. The geographical basis of this study
could be extended to include different urban areas to include other major cities in New Zealand, such
as Auckland and Wellington and to compare urban and rural based families. Also, if resources could
be made available there is potential to use focus groups to compare family leisure in different
countries, such as the UK and Australia.
220

David Lamb
To develop this focus group study, it is recommended that a broader range of family types need to be
involved in future research to include: same sex couples; married step families; de facto unions;
multigenerational families; non-European families and single parent families. Also, the analysis of
family types needs to include families from different socio–economic backgrounds, in particular more
financially disadvantaged family groups. Furthermore, there is a lack of research evidence detailing
the experience of extended family networks and on generational perspectives and on the voices of
extended family members, who may play, important supportive roles in family leisure.
References
Allen, J., O'Toole, W., Harris, R., & McDonnell. I. (2010). Festival and special event management (5th Ed.).
Brisbane, QLD: John Wiley & Sons.
Anderson, J.A. (1988). Cognitive styles and multicultural populations, Journal of Teacher Education Vol. 39, No.
1, pp. 2-9.
Becker, H.S. (1998). Tricks of the Trade: How to think about research while you’re doing it.. Chicago: University
of Chicago Press.
Creswell, J.W. (1998). Qualitative Inquiry and Research Design: Choosing Among Five Traditions. Thousand
Oaks, California: Sage.
Currie, J. (2004). Motherhood, Stress and the exercise experience: freedom or constraint? Leisure Studies.
23(3), 225-242.
Denzin, N.K. & Lincoln, Y.S. (1994). Handbook of Qualitative Research. London: Sage.
Goss, J.D., & Leinbach, T.R. (1996). Focus groups as alternative research practice. .Area. 28(2), 115-123.
Glasser, B., & Strauss, A. (1967). The Discovery of grounded Theory: Strategies for Qualitative Research.
Hawthorn, NY: Aldine De Gruyter.
Grbich, C. (2007). Qualitative Data Analysis: An Introduction. Thousand Oaks. CA: Sage.
Guba, E., & Lincoln, Y. (2005). Paradigmatic controversies, contradictions and emerging confluences. In N.
Denzin & Y. Lincoln (Eds), The sage handbook of qualitative research (Third edition ed.) London: Sage.
Gubrium. J.A. & Holstein, J. A. (2006). Couples, kids and family life. Oxford, UK: Oxford University Press.
Jones, I. (1997). Mixing qualitative and quantitative methods in sports fan research. The Qualitative Report,
available at http:/www.nova.edu/ssss/QR/QR3-4/jones.htm2 (accessed January 20, 2012).
Kitzinger, J (1995). Introducing focus groups. British Medical Journal. 311: 299-302.
Krueger, R., A. (1988). Focus groups: A Practical Guide for Applied Research (3 rd Edition) London: Sage.
Krueger, R. A., & Casey, M., A. (2000). Focus groups: A Practical Guide for Applied Research (3 rd Edition)
Thousand Oaks, California: Sage.
Long, J. (2007). Researching leisure, sport and tourism, The essential guide.. London: Sage Publications.
Merton, R.K. (1987). The focussed interview and focus groups. Public Opinion Quarterly, 51, 550-566.
Millard, L.J. (1995). Focus groups. In G.M. Breakwall, S. Hammond, and C. Fife- Shaw. Research Methods in
Psychology. London: Sage
Morgan, D. L. (1997). Focus groups as qualitative research. Thousand Oaks, CA: Sage.
Morrison, M. (1986). Methods in Sociology. London: Longman.
Patton, M. Q. (2002). Qualitative Research and Evaluation Methods. New Jersey: Erlbaum and Associates.
Powell, R.A., Single, H.M., & Lloyd, K.R. (1996). Focus groups in ‘mental health research: enhancing the validity
of user and provider questionnaires. International Journal of Social Psychology. 42(3), 193-206.
Stewart, D.W. & Shamdasini, P.N. (1990). Focus Groups: Theory and practice. London: Sage.
Tesch, R. (1990). Qualitative Research analysis types of software tools. New York: Falmers.
221

Issues and Challenges of Evaluating Digital Divide Projects
Keri Logan and Barbara Crump
Massey University, Wellington, New Zealand
k.a.logan@massey.ac.nz
b.j.crump@massey.ac.nz
Abstract: A requirement of receiving public funding for community based projects is that an independent
evaluation be undertaken within a certain time period. Depending on stakeholders’ perspectives, the evaluation
report is used as an indicator of the worthiness of the project for future funding, identification of issues that need
addressing or, in the case of small impact evaluations, may provide an estimate of the likely outcome of a
planned project. Thus, organisations often use the report for learning, monitoring and accountability. Over the
past decade we have been successful in being commissioned to evaluate four digital divide projects situated in
different New Zealand contexts. The Requests for Proposals (RFPs) briefly define the goals for the project which
have both technical and social objectives, state the required method which is usually quantitative, and the
timeframe for completing the evaluation. A feature of the RFPs has been the lack of clarity in the stated outcomes
and in some of the objectives. The projects involved multiple stakeholders who each bring their own perspectives
and expectations for the evaluation that are, at times, conflicting. In this paper we identify the issues that have
arisen in evaluations that meet these expectations and that are useful to the stakeholders. In order to resolve
these challenges we believe the most critical factor is open and frequent communication that encourages the
building of trust relationships with all stakeholders. Reflection of our experiences in meeting the various issues
and challenges have improved the quality of our evaluative practice which other evaluators may find useful. We
have learned to approach every project from, as Patton (2008) says, a logical, concrete and observable
perspective which is often very different from that of the stakeholders with whom we are interacting, and for
whom we are evaluating.
Keywords: evaluation, digital divide, community projects
1. Introduction
Formal systematic evaluation of government-funded digital divide projects has been a feature of four
community computing initiatives we evaluated over the past decade. The projects have been in the
form of planned social intervention with the aim of encouraging the use of computing by sectors of the
population considered to be in danger of alienation from the knowledge society (a state known as the
digital divide). Addressing the digital divide was the goal of the New Zealand government’s Digital
Strategy, launched in May 2005 with the aim of creating a digital future for all New Zealanders
whether they be at home, school, work or in the community. The focus was on spreading the benefits
of information and communications technology (ICT) right across the economy. The community sector
was one of the three “agents of change” and a contestable fund of nearly $60 million was made
available for projects that would support grassroots initiatives to build ICT skills in communities and
regions.
A requirement of the funding was an independent, formal evaluation for the primary purpose of
accountability. The evaluation was to assess the project’s viability and to ascertain whether it met
project objectives. The evaluations also had the potential to provide useful feedback to the users.
Many of the Requests for Proposals (RFPs) prescribed the expected outcomes and the type of
evaluation required by the funders. The client (the body that was successful in winning the
competitive funding round) often emphasised a quantitative approach, usually in the form of a survey,
as the most appropriate method. When a summative evaluation was required adopting a quantitative
approach had the potential of not meeting the needs of other stakeholders who often were the joint
funders.
This paper reports our experiences in meeting the issues and challenges of evaluating four digital
divide projects in a large New Zealand city. We begin by briefly discussing evaluation approaches
followed by discussion of the method and the main challenges and issues we encountered in dealing
with multiple stakeholders in the evaluation activities of planning, designing, implementing and
reporting. We then conclude by identifying key critical factors which guided our evaluation practice.
2. A brief evaluation review
A holistic evaluation approach emphasises the need to assess the merit and worth of a project and
commonly addresses four questions: “What needs to be done? How should it be done? Is it being
222

Keri Logan and Barbara Crump
done? Did it succeed?” (Stufflebeam 1971: 1). Stufflebeam’s (2007) CIPP model (context, input,
process, product) provides a comprehensive framework that highlights the necessity of taking into
account the context in which a programme takes place, evaluating the programme’s goals,
investigating inputs, examining the implementation and measuring the outcomes, both positive and
negative. However he notes (2001: 91) that it “is virtually impossible to fully and unequivocally assess
any program’s ultimate worth” and emphasises the focus should address questions regarding merit,
worth, probity and significance.
The timing and purpose of evaluations influence the approach taken; that is, whether the evaluation
should be summative or formative (or has elements of both). Scriven (1991: 340) is credited with
being the first to use the terms summative and formative. He describes summative evaluation of a
programme as being about judgement, “conducted after completion of the program and for the benefit
of some external audience or decision-maker … The aim is to report on it [the program] ... not to
report to it.” The people to whom the evaluation results are reported then make decisions relating to
whether the programme should be expanded, continued, continued with some changes or maybe
even discontinued. This feedback usually takes the form of a one-off formal report.
Formative evaluation is different in that the focus is on improvement of the programme rather than the
passing of judgement (Patton, 2008). Consequently, data is gathered regarding the strengths and
weaknesses of an intervention or programme. The main users of formative evaluation results are the
programme staff and sometimes the participants themselves. In formative evaluations qualitative data
is usually collected over a period of time and fed back to the staff on a regular basis creating a cycle
of reflection and improvement. However, both Patton (2008) and Scriven (1991) note that it is
important not to treat the formative/summative dichotomy as being the same as the distinction
between process (evaluating what happens between the inputs and outputs) and outcome (evaluating
the end results of a project) evaluation. Scriven clearly states that summative evaluations are not
solely about outcomes nor are formative evaluations a type of process evaluation. Chen (2004, 2005)
in describing his stage model suggests that programme improvement and learning should be the
emphasis in the early stages of a programme’s life, but that there should be a shift to a summative
focus later; although he does say it is not necessarily a lateral process. He proposes a practical
taxonomy that consists of four stages and suggests that evaluation choices should be driven by the
stage at which a programme is being evaluated. Therefore both formative and summative issues can
be addressed within a single evaluation.
A further complementarity to evaluation is monitoring, considered by Hatry, Wholey & Newcomer
(2004: 671) as “an important subset of evaluation”. Monitoring is often the role of the funding agency’s
representative, who ensures that the funds are spent correctly and that the project is kept on track.
The monitor seldom has evaluation skills and is mainly focused internally on the progress of the
programme implementation. The progress reports that result from monitoring may be used to improve
the project so long as it is sufficiently flexible to adapt to changing conditions (Annecke, 2008;
Scriven, 1991). Monitoring and evaluation are systems that are designed for both managerial and
accountability functions (Patton 2008).
Patton (2008) notes that there are a myriad of challenges to effective, useful evaluation, some of
which we experienced in our evaluations. Among these are issues involving ethics, quality,
engendering commitment of funders and decision makers who often do not know what evaluation
involves and can set unrealistic deadlines, provide limited resources and have “gross misconceptions
about what can actually be measured with precision and definitiveness” (p. 56).
This brief overview provides a background to the approaches and focus that we adopted when
evaluating the four projects. All projects had multiple stakeholders and the intent was for a utilisationfocused
programme evaluation, noted by Patton (2008: 37) as one that focuses on “intended use by
intended users” who should be able to apply the findings and implement the recommendations in a
useful fashion. Utilisation-focused evaluation is the systematic acquisition and assessment of
information to provide useful feedback to the authority so they can assess each project’s viability and
whether it is accomplishing what it is meant to accomplish (Patton, 2008). The results may be used as
an indicator of the worthiness of the project for future funding. They are important for the providers
and other stakeholders as a means of identifying any problems and addressing them where
appropriate. Thus, in some instances organisations use the report for both learning and
223

Keri Logan and Barbara Crump
accountability. In the next section we identify and discuss the method adopted for the four evaluations
followed by the major challenges we encountered in evaluating the projects.
3. Methodology
The mixed-method approach adopted for the four evaluations incorporated aspects of a partnership
model involving stakeholders who brought multiple perspectives. This cooperative relationship
involved individuals and groups such as the local council, charitable trust, a corporate and volunteers.
Typically, the trust representatives and volunteers contributed to the research design. Methods used
included a survey, in-depth interviews with key stakeholders, on-going ‘conversational’ interviews with
computer users and weekly observations of the computer rooms. The Sequential Explanatory Visual
Model (see Ivankova, Creswell and Stick, 2006) provided guidance as to the sequencing of method,
definition of procedure and the resulting product. The same two researchers worked together on all
four projects. However for the larger project where evaluation was for a longer time period, four
researchers shared data gathering tasks. All contributed to the reports which were submitted to the
contractor and other stakeholders. The next section identifies and discusses the challenges and
issues which arose during our evaluations.
4. Project challenges
Funding for the four projects was supplied from the central government’s Digital Strategy fund and the
local city council. A multinational computer corporation and charitable trusts also contributed funding
and resources. The RFPs to which we responded related to expectations and requirements stated in
those documents. To ensure that the method, process and measures met stakeholder needs and
interests, elements which Patton (2008) describes as characteristics of utilisation evaluation, we
adopted a partnership approach in engaging relevant stakeholders. The RFPs set out in detail the
expected outcomes, time of completion, and, for some, the evaluation approach. These documents
presented many challenges to achieving an effective and useful evaluation. We begin with a
discussion of how we dealt with the unrealistic expectations defined in the RFPs.
4.1 Unrealistic expectations stated in RFPs
The RFPs were, in most cases, based on information provided in the funding applications. This was a
competitive process and usually just one third of applications were successful. The funder looked for
meritorious, practical projects promoting the adoption of information technology (IT) within a socially
inclusive environment that would foster community. An application that held the promise of providing a
service or a computing centre that would ‘make a difference’ to people’s lives was therefore attractive
and the RFPs not only addressed the technical, practical computing requirements but placed
emphasis on making a ‘difference’ to people’s lives. However some of the measures of success
stated in the RFPs were unlikely to be achieved given the constraints also stated within these
documents.
Some of the measures of success defined in the RFPs were also very unrealistic, particularly those
for evaluating the implementation of an ICT network in a youth centre. The overall goal for the project
was stated as: to increase positive opportunities for youth development so that young people can
become resourceful, responsible, socially aware, interactive and positive. The goal related to one
basic premise, namely: Is the community ICT hub meeting the needs of its community of interest
(youth)? To satisfy this premise the RFP specified external outcomes under the four categories of
Connection, Personal Development, Information and Third Place (a term defined by Oldenburg
(1991) to describe a non-threatening place where regulars gather and which has a strong sense of
social inclusion and connection). Each of these categories had five to seven specified “measures of
success”. For example, “reduction in youth ‘drifting’ on city streets”, “reduction in youth issues and
problems”, “IT and literacy skills are developed and enhanced”.
Evaluating some of these broad expected outcomes was not possible within the time available, just
four months from January to May. New Zealand’s long summer break is within that time during which
the youth centre was closed. When it did re-open many secondary school students were still on
holiday. As students were the target sample this meant that the time for data collection was
effectively reduced to just over two months. To measure whether or not a “sense of community is built
and sustained” would need ongoing observations and interviews over a lengthy period of time, thus
some of the RFP requirements for this project were unrealistic given the time frame within which the
evaluation was expected to be undertaken.
224

Keri Logan and Barbara Crump
Additional to the time problem was the scope. For example, to identify that literacy skills were
developed and enhanced would require in-depth investigations establishing a young person’s literacy
level (likely involving his/her teachers) and then, after attending the youth centre and using the ICT
facilities, an assessment of literacy. Even then many variables could account for a difference in the
pre and post assessments.
Reducing the evaluation scope to realistic levels that would still indicate accountability and be useful
to stakeholders required much negotiation. We had various meetings over a number of weeks with
the client with whom we had the contract, the centre’s manager and youth workers where we
discussed the problems as we saw them, asked for input and then submitted our draft proposal that
re-framed the objectives and expected outcomes. Our aim was to encourage each stakeholder to say
what they would like to get out of the evaluation. Once we had consensus we submitted our final
proposal which was accepted and by using a mixed method approach we were able to meet most of
the stakeholder expectations for the evaluation.
4.2 Other issues
4.2.1 Pressure for a positive evaluation
Very often the client wants a “politically advantageous study performed” (Stufflebeam, 2001: 91). The
client who commissioned our evaluations was the director of a charitable trust which was dependent
on project funding for its continued existence. Therefore the trust had a vested interest in positive
evaluations of the projects with which it was involved. While there was no overt pressure for our
evaluation to report positive outcomes we were aware of the need to maintain our independence. We
clarified our role right at the beginning, with meetings that involved not only the client but, where
possible, with the staff of the newly-implemented project. We listened carefully to what they wanted
to find out from the evaluation and explained that for reasons of validity and credibility we would report
our findings as accurately as possible, given that all participants, including the evaluators, have their
own values and agendas. We advised that they would receive a draft report for comment before the
final report was submitted. By adopting an open and communicative approach we were able to build
trust relationships that contributed to the reliability of the data gathered.
In several of the projects the client, in the pre-evaluation discussions, emphasised that the core
funders (typically central and local governments) were interested in “hard facts”. The client was
insistent that statistical data, therefore, be a feature of the final report. Other stakeholders such as the
marketing manager of a multinational corporate was interested in the social and community impact
the project would have as their company’s commitment and participation was predicated on meeting,
and being seen to demonstrate, social responsibility to local community. The challenge then was in
being able to negotiate a shared understanding of what and why the evaluation was being undertaken
and to satisfy the wishes of these different stakeholders. This was achieved by adopting a mixedmethod
approach where the qualitative data was particularly useful, especially in establishing why
there was a low user-uptake of one newly-implemented project.
4.2.2 Ethical issues
An example where a negative occurrence posed an ethical dilemma for us was with the digital divide
project aimed at the whole community of an inner city suburb. The project involved a computer centre
that was established in a room of the city council’s largest high-rise housing complexes. The aim for
the project was to narrow the digital divide within the suburb and had specific objectives relating to
social and human capital. (See Crump (2006) for a full analysis of the longitudinal study). A
polytechnic provided course materials and paid two volunteers to run computer familiarisation and
application software lessons. The volunteers were paid per student; the higher the enrolment, the
greater the pay.
We monitored the frequency of use of the computing centre in two ways: collecting data from the daily
log where students would sign in and recording class attendance during our twice-weekly
observations. We noticed that the log recorded near-full classes which did not tally with the numbers
we counted during our observations. Eventually, one of the coordinators admitted to inflating the
enrolment numbers by entering fictitious names so that they could not only retain their paid
employment status but increase the size of their pay. False data was therefore submitted to the
polytechnic staff responsible for managing the teaching programme in the computing centre.
225

Keri Logan and Barbara Crump
Once we became aware of the true situation we were faced with a dilemma: should we respect the
trust that had grown between us as evaluators and the paid volunteers and keep quiet about actions
we considered to be fraudulent and unethical or should we inform the polytechnic? After discussions
with an ethics expert we contacted the polytechnic manager responsible for the teaching programme
and informed her, without naming anyone, that we believed there was a discrepancy in the numbers
attending the lessons and those indicated in the register. We left it up to her to take further action (of
which there was none). We also indicated to a senior staff member of the multinational corporation
that we had misgivings as to the accuracy of the numbers participating in the training. However, they
were not receptive to hearing ‘bad news’. They were focused on the ‘good news stories’ appearing in
local newspapers demonstrating their social responsibility.
We have since reflected on our actions and feel ambivalent as to whether we did enough considering
the morality and ethics of the situation. Should we have acted differently? Should we have taken the
matter further and informed other stakeholders such as council officers? At the time, we justified our
approach as our contract was to evaluate the usage of the centre by residents; the classes were not
part of the evaluation, albeit they were delivered in the centre.
4.2.3 Evaluation type
The evaluation type was typically a mix of formative and summative as stakeholders were interested
in both improvement of the project and judgement of their worth. In one of the projects a summative
evaluation only was possible because of the limited timeframe. The summative report contained
recommendations; hence there were elements of judgement as well as implications for improvement.
We found that the distinctions between evaluation types blurred and the evaluations had some
elements of both, indicating the dynamic nature of evaluating projects (Datta, 2006).
Clarification of evaluation type (formative, summative, monitoring) was needed before formalising the
contractual agreement and was not always a straightforward process. For each project on-going
communication and negotiation with the key stakeholders eventually determined what was to be
evaluated and the purpose and use of the evaluation. Patton (2004) highlights in his discussion of
utilisation based evaluation the necessity of developing a working relationship with stakeholders in
order to negotiate an evaluation method that is suitable for the individual project and the time over
which an evaluation takes place influences this working relationship. The digital divide project that
focused on an inner city suburb began with a one-year pilot project evaluation but evolved into two
projects lasting for three years. This long term evaluation presented problems of a different type to
those of a shorter duration (the shortest of just four months). Because the major funder was the city
council the people with whom we were dealing were constantly changing and there was little
continuity in the overseeing of the project. This constant change of staff meant that the original
rationale for the council’s funding for, and commitment to, the project became obscured over time.
Towards the end of the evaluation period council staff had no ‘ownership’ of the project.
In the project involving the establishment and delivery of a subsidised, mobile, affordable and
sustainable ICT technical support service the evaluation type was decided after discussion with key
stakeholders. The client who had invited us to evaluate this one-year pilot project also stressed the
importance of a quantitative evaluation (for a report to the core funder). However, the focus for the
project initiator and champion was utilisation. The champion was very keen to incorporate a
monitoring aspect to the project and invited one of the evaluators to be an ex officio member of the
steering group which met each month. At the meetings evaluation activities and preliminary evaluation
results were reported, additional to the core business of project management. After several
discussions with the stakeholders it was agreed that the evaluation would result in an interim report (a
formative evaluation that would enable learning and contribute to the monitoring aspect of the project)
and a final report (summative evaluation) to meet the funder’s (a government department)
requirements. The ongoing monitoring approach better served those internal to the project. Thus the
spotlight moved from a strong accountability and measurement approach primarily paying attention to
external stakeholders (Patton, 2008) to one that included a monitoring focus with regular feedback to
the commissioning body.
4.2.4 Need for flexibility
A further issue which we encountered was the need to be flexible and change our plans when
necessary, demonstrating the dynamic nature of evaluation (Datta, 2006). For example, when it
226

Keri Logan and Barbara Crump
became apparent in one project that the numbers subscribing to the mobile computing service were
much less than had been anticipated we amended the research design to include a non-subscriber
survey involving telephone interviews to gauge reasons and potential barriers for NFPs not
subscribing. Similarly in our first evaluation involving the housing estate and use of the newly-
implemented ICT suite in the youth centre, low patronage by the residents prompted us to change the
evaluation design to include a ‘No Usage’ survey. Analysis of the resulting data broadened our
understanding and provided more useful results than would have been the case if we had adhered to
our original design.
4.2.5 Funding
In another project the application for funding provided misleading information. An ICT communityfocused
trust was granted funding from the local city council to contract us for a pre-formative
(Scriven, 1991) evaluation to identify whether there was a need by the not for profit sector for
implementing an ICT-based shared services framework. The purpose of the project, if implemented,
would be to standardise requirements for compliance, day-to-day management, strategy and
governance and marketing. In this application it was stated that our university would undertake an
independent evaluation (a) at a price that was completely unrealistic and (b) without consultation with
the university or ourselves as the proposed evaluators. The local body funding was granted and only
then was the university approached. Because the funding allocated to evaluation was so little
extensive negotiations with senior university management were required in order to waive some of the
university costs and enable the evaluation to take place. Had the evaluation been unable to proceed
then the trust would have been placed in the embarrassing situation of receiving money under false
pretences. Situations such as this are caused through naivety and time pressure for submitting
applications, rather than any intention to mislead the funding agency. This instance demonstrates that
programme funders and decision makers who request an evaluation often do not understand what
evaluation is and the time and costs involved.
4.2.6 Stakeholder ‘buy-in’
We recognised that stakeholder buy-in was important if an evaluation was to serve a useful purpose
(Patton, 2008) and worked hard to try and ensure this was the case in all projects, not always with
success. In the instance of the youth centre ICT project, where the evaluation was on a limited
budget, it was agreed (during the proposal stage) that the manager of the centre, and the youth
worker would administer the survey to computer users. Although initially indicating their willingness
and interest in doing this they became apathetic over time. Data collection was sporadic (the survey
was available for six weeks) and required several reminders from the client to ensure surveys were
completed. We understood that the project staff felt the evaluation had been imposed on them and
they possibly felt threatened by it. Although we could not be sure, we suspected that the youth
workers had filled in some of the forms themselves in order to increase the number of survey forms.
5. Discussion and conclusion
The range of evaluation issues we have discussed in this paper confirms Patton’s (2008: 56)
contention that there are “any number of challenges: political intrigues, unrealistic deadlines, limited
resources, gross misconceptions about what can actually be measured with precision and
definitiveness”.
We believe that precision and definitiveness is an ideal that can seldom be achieved if we are to
complete an evaluation that meets the requirements, wishes and needs of the various stakeholders.
Based on reflections of our experiences we believe a key critical factor is to approach every project
from, as Patton (2008) says, a logical, concrete and observable perspective which is often very
different from the stakeholders with whom we are interacting, and for whom we are evaluating. Such
a perspective can be achieved by open and frequent communication with all the stakeholders. It is
essential that prior to any contract being signed the purpose of the evaluation is negotiated and
clearly understood and agreed to by all stakeholders. This may take some time, especially when the
outcomes are unrealistic or there are numerous stakeholders, each with their own agenda. Without
this support and agreement, the evaluation is unlikely to go smoothly, or to be useful or used for
decision making on completion.
Building a sense of trust between those evaluating and those being evaluated is critical. Constant
interaction and discussion with project staff helps to ensure successful buy-in. It is also helpful if they
227

Keri Logan and Barbara Crump
have input into the negotiations regarding the type of evaluation and the purpose. This will contribute
to mitigate any antipathy that may arise when people feel they are being monitored and/or evaluated.
Being sensitive to the context and place (Weiss, 2004) is a further critical factor. How we address one
challenge cannot be automatically applied in another situation, even although it may appear similar.
The stakeholders are different, the project is different and our values and insights are continually
changing according to the place from which we are viewing the situation. As Patton (2008: 155) says
when discussing his Grand Canyon hikes “I return different from when I entered. Not always different
in the same way. But different.” By continually reflecting on our actions during and after every
evaluation we form new understandings that inform our actions in different situations (Schon, 1983).
Reflection and offering “a menu of possibilities” (Patton, 2008: 37) will go a long way to achieving a
successful evaluation.
References
Annecke, W. (2008) 'Monitoring and Evaluation of Energy for Development: The Good, the Bad and the
Questionable in M&E Practice'. Energy Policy, 36: 2839-2845.
Chen, H.-T. (2004) 'Theory Driven Evaluation '. In M.C. Alkin (ed.) Evaluation Roots. Trading Theorists’ Views
and Influences. Thousand Oaks, CA: Sage.
Chen, H.-T. (2005) Practical Program Evaluation: Assessing and Improve Planning, Implementation, and
Effectiveness, Sage, Thousand Oaks, CA.
Crump, B. J. (2006) 'Institutions, Community, and People: An Evaluation of a Longitudinal Digital Divide
Experience'. In D.H. E. Trauth, T. Butler, B. Fitzgerald & J. Degross (ed.) Social Inclusion: Societal and
Organizational Implications for Information Systems. pp. 333-346, Boston: Springer.
Datta, L. (2006) ‘The Practice of Evaluation: Challenges and New Directions’. In I. Shaw, J. et al. (eds.), The
Sage Handbook of Evaluation. Sage Research Methods Online [Electronic].
Hatry, H., Wholey, J. and Newcomer, K. (2004) 'Other Issues and Trends in Evaluation'. In J. Wholey., et al.
(eds.), Handbook of Practical Program Evaluation. pp. 670-684, San Francisco, CA: Jossey-Bass.
Ivanakova, V., Creswell, J. W. and Stick, S. L. (2006) ‘Using Mixed-Methods Sequential Explanatory Design:
From Theory to Practice’, Field Methods, 18 No 1: 3-20.
Oldenburg, R. (1991) The Great Good Place, Cafes, Coffee Shops, Community Centers, Beautiy Parlors,
General Stores, Bars, Hangouts, and How They Get You through the Day, Paragon House, New York.
Patton, M. Q. (2004) 'The Roots of Utilization-Focused Evaluation'. In M.C. Alkin (ed.) Evaluation Roots: Tracing
Theorists' Views and Influences. pp. 276-292, Thousand Oaks, CA: Sage.
Patton, M. Q. (2008) Utilization-Focused Evaluation, Sage Publications, Inc., Thousand Oaks, CA.
Schon, D. (1983) The Reflective Practitioner, Basic Books, New York.
Scriven, M. (1991) Evaluation Thesaurus, Sage Publications, Inc, Newbury Park, CA.
Stufflebeam, D. (1971) 'The Relevance of the CIPP Evaluation Model for Educational Accountability '. Journal of
Research and Development in Education, 5: 19-25.
Stufflebeam, D. (2001) 'Evaluation Models '. New Directions for Evaluation, 89: 7-98.
Stufflebeam, D. (2007) ‘CIPP Evaluation Model Checklist’. [online]
http://www.wmich.edu/evalctr/archive_checklists/cippchecklist_mar07.pdf (accessed on 12 January 2012).
Weiss, C. (2004) 'Rooting for Evaluation: A Cliff-Notes Version of My Work'. In M. Alkin (ed.) Evaluation Roots:
Tracing Theorists' Views and Influences. pp. 153-168, Thousand Oakes, CA: Sage.
228

‘Thing’ Qualities: Grounded Theory Method, Discourse
Analysis and Semiotics Approaches
Arminda Lopes
Instituto Politécnico de Castelo Branco, Portugal
aglopes@ipcb.pt
Abstract: This paper articulates the concerns about adopting various research methodologies of analysing
verbal, non-verbal and images data such as transcripts of designers discourse, and small group dialogues. A
combination of some of the principles of Grounded Theory Method (GTM), Conversation Analysis (CA) and
Semiotics during the research data collection and data analysis were applied. All those were about the meaning
we make with words and images. The case studies used to gather data were obtained from two different
research networks: The Leonardo Network on Culture, Creativity and Interaction Design and The White Rose
Network for Affective Communication in Consumer Products and Exhibition Design. GTM was developed by
Glaser and Strauss in 1967 and it has evolved as a method in slightly different directions. In this research the
basic concepts and procedures settled by the cofounders are followed but in position with the constructivist
Grounded Theory using it as a heuristic strategy. A partnership between the researcher and designers enabled a
mutual construction of meaning during interviews and conversations and a meaningful reconstruction of their
stories into a grounded theory model. The CA principles embody a theory of meaning-making embracing with a
theory of human behaviour and human culture. The basic of CA is comparison as such in GTM. Semiotics is
commonly defined as the science of signs and has to do with symbols and meanings. Just as semiotics, the
study of signs, helps us to understand practices of description, which in turn reveal how meaning is
communicated, so does it enable us to understand the creativity of the designer, design being a method of the
communication of meaning. In addition, the decision for the use of semiotic analysis was based on the need to
understand the nature of drawings and artworks themselves, and the multiple relationships between them and
designers as well as the designer’s values and beliefs through the artworks inherent signs. Grounded theory
method, conversation analysis and semiotics were the approaches to the discovery and explanation of patterns
that changed over time. The main concepts of these methodologies and their interrelations were useful in the
formulation of the attained research theory as it will be presented in the paper.
Keywords: grounded theory method; discourse analysis; conversation analysis; social semiotics; multimodal
semiotics
1. Introduction
The main goal, in this paper, is to present the complementarity of the data analysis using three
different methodologies: Grounded Theory Methodology, Conversation Analysis and Semiotics with
special concern on the objects produced by designers. The adoption of an interpretative paradigm of
research inquiry and the choice of the three methodologies was informed on the similar analytic
perspectives and on the decision that a variety of approaches suited the present research.
Grounded theory is typically presented as an approach to do qualitative research. Grounded theory
research begins by focusing on an area of study and gathers data from a variety of sources, including
interviews and observations. Once gathered, the data is analysed using coding and theoretical
sampling procedures. Then, theories are generated, with the help of interpretive procedures. The
major purpose of grounded theory is to start with the data and use them to develop a theory.
The progenitors of GTM were Glaser and Strauss, in the sixties, during their studies in the healthcare
domain. They developed systematic methodological strategies during research conversations and
observations until the development of theories from the research grounded in data. The major
strategy they use is a general method of comparative analysis. Then, in 1990, Strauss and Corbin
further developed the GTM adding systematic approaches and validation criteria. They defined
grounded theory method (GTM) as a qualitative research method that uses a systematic set of
procedures to develop an inductively derived GT about a phenomenon, highlighting that GTM is an
analytical approach based on grounding the analysis in the data that have been gathered and
inductively reaching conclusions from these data. Later, Charmaz' work initiated the development of a
more profoundly justified base for the method: The constructivist form engaged with the problematic
issue of data and the active role of the researcher in the process of conceptual development.
Whatever its strengths, weaknesses and ambiguities, the popularity of GTM grew and the method has
become truly used in qualitative research. The goal in this research is, although following the
constructivist branch, to see GTM as a tool assessed in terms of usefulness for the particular tasks
229

Arminda Lopes
and applications of this research. Thus, whether researchers follow Glaser, or Strauss (and Corbin),
or Bryant and Charmaz, or any other variants, the key issue becomes the extent to which their
substantive research produces conceptual innovations and theoretical insights that prove useful. This
approach suited as a tool to analyse discourse and behaviours the fundamental concern of
conversation analysis.
Conversation analysis was the second advocated methodology used. Conversation analysis and
discourse analysis are approaches to study the social world. However, there is a variety of theoretical
perspectives defining Discourse Analysis (DA): it has been used as a generic term concerned with
language in its social and cognitive context (Brown and Yule 1983; Coulthard 1977; Van Dijk 1985);
as a description of studies focusing only on linguistic units above the level of the sentence (Stubbs
1983); as the correct term for research concerned with cohesion - cohesion concerns “the
components of the textual surface, that is the ‘text-syntactic’ connectedness” (Titscher 2000:22) which
means that the sequence of linguistic elements in a text follows grammatical rules and dependencies.
All the functions that are applied to create relationships between surface elements are categorised as
cohesion; and connectedness across sentences or turn-taking (Titscher 2000; van Dijk and Kintch
1983) and to cover developments from structuralism and semiotics (Foucault 1971; Pecheux 1982),
among others. In this study, discourse will be used in an open sense, following Gilbert and Mulkay
(1984) to cover all forms of spoken interaction, formal and informal, and written texts of all kinds. The
focus is on practice and context.
The method and procedures used, in this study, for the analysis of discourse was conversation
analysis a sub set of DA looking of people talking.
Finally, semiotics was considered to analyse the produced artefacts/artworks. Semiotics is primarily
concerned with the meaning of signs and symbols in language. The essential idea is that words/signs
can be assigned to primary conceptual categories, and these categories represent important aspects
of the theory to be tested. The importance of an idea is revealed in the frequency with which it
appears in the text. From a methodological standpoint, social semioticians analyse images according
to three main meta-functions. The study follows Halliday (1978) in recognising three main kinds of
semiotic works which are always performed simultaneously. Halliday calls this kind of work metafunctions.
Kress and van Leeuwen (1996) have extended this idea to images, using a somewhat
different terminology: representational, interactive and compositional. The figures in this paper will be
analysed within this perspective and also following Olesen et al. (2001) who consider that to describe
a ‘thing’ we need to describe its qualities and the ‘context of the things’.
In this paper, by the research data analysis obtained from two research networks to arrive to a theory,
it will be understand further about what designers say and how they behave during the design
process. Conversely, it will be found that there are some entities that form the elementary ‘building
blocks’ of a thing (artefact/artwork): lines, shapes, colours, forms, purpose, meaning and
relationships. Finally, by analogy with the three methodologies, all of them focus on language use and
production of meanings; texts are deconstructed in basic elements: codes, words, signs.
2. The three methodologies purposes
The main reason to justify the decision of presenting artefacts/artworks description within semiotics is
because they are expressions with different signs on different levels of design language. Semiotics is
considered a rehearsal that predisposes practices of description that consecutively affects practices
of communication of meaning, and being design a practice of expression of meaning, semiotics help
to understand the manifestation of designers’ creativity. Conversely, the use of semiotics analysis was
based on the need to understand the nature of drawings and artworks and the multiple relationships
between it and designers as well as the designer’s values and beliefs through signs. Despite
semiotics can be treated as both an underlying philosophy and a specific mode of analysis, in this
paper it is used as the latter.
Another form of semiotics is "conversation analysis." In conversation analysis, it is assumed that the
meanings are shaped in the context of the exchange (Wynn, 1979). The researcher immerses
himself/herself in the situation to reveal the background of practices.
Conversation analysis (CA) was the adopted method to analyse the use of conversational practices
that simultaneously analyse action, meaning, context, and inter subjectivity. The goal in this analysis
230

Arminda Lopes
was to find out how interactivity was achieved through dialogue considering especially what roles
speakers took on, how they positioned other participants into particular roles, how turn taking and
topic change occurred and the different kinds of feedback strategies that participants used. The
interactional features of conversation: turn-taking, adjacency pairs and repairs were considered.
GTM was followed as methodological guidelines for text and pictures analysis. Conversely, it was
suitable for the adoption of multiple methods of data collection, as it was the case. Comparison
between data, codes and categories permitted to understand relationships between verbal, nonverbal
and pictures’’ meaning.
A brief reflexion about the main concern of each methodology is followed.
2.1 Grounded theory method
Grounded Theory (GT) is a general method to use on any kind or combination of data, and it is
particularly useful with qualitative data (Glaser 1998) (Denzin 1994) argues that GT is only one of
several different qualitative research methods available to those conducting exploratory research. The
basic idea of the grounded theory approach is to read (and re-read) a textual database or
observations of behaviour, such as interactions and label variables (called categories, concepts and
properties) and their interrelationships.
The GTM approach, particularly the way Strauss develops it, consists of a set of steps whose careful
execution is thought to "guarantee" a good theory as the outcome. It involves two phases in the
analysis of qualitative data. Data fragments are compared in order to derive general descriptors
(concepts; categories) which catch their analytically relevant properties. The second phase is used to
elaborate, refine and reduce results of the first phase. This involves connecting codes in the form of
hypothesized propositions specifying conditions. Open coding is the part of the analysis concerned
with identifying, naming, categorizing and describing phenomena found in the text. Essentially, each
line, sentence, paragraph etc. is read in search of the answer to the repeated question "What is this
about? What is being referenced here?"
GTM procedures are equally applicable to non-textual data, but a central importance is attributed to
text as data material in the form of interviews, transcripts, observers’ notes, book, articles, etc. The
most well-known application of GTM is text analysis which in discourse analysis is viewed as a
manifestation and result of a particular combination of factors.
2.2 Conversation analysis
The core of conversation analysis is about close observation of the world (Sidnell 2010). It is about
discovering things we don’t already know and to understand the social order of talk. CA focuses on
people’s talking. For Potter et al. (1987:80) it is concerned with how “the contributions of different
speakers are meshed together in conversations and the way different types of actions – blaming,
greetings, excuses – are produced and managed”. CA is characterised by the view that how talk is
produced and how the meanings of that talk are determined as the practical, social and interactional
accomplishments of members of a culture.
The analysis of conversation was made through taking sequences of conversation and finding rules
and procedures to be used to generate the orderly features to be examined. The identified rules were
brought into relationships with other data to be tested and explicated. The aim was to establish
categories of orderly, or not, of behaviours and discover the rules that participants used to construct
the interactions. This analysis was articulated with GTM concepts and semiotics interpretations of the
artefacts/artworks produced.
2.3 Semiotics
The need to develop theories and descriptions of semiotics has been a concern of several scholars.
Saussure anticipated the need for a science aimed to study the life of signs within society (Saussure
1974:16); while Barthes (1957/1972:112) claimed for the ‘development of a semiological science’.
We consider that semiotics open the meanings that we understood from everything that surround us.
This argument bond to consider Halliday (1978) insight that language realizes and reflects in its
231

Arminda Lopes
internal organization the social functions for which it serves. Social semiotics concerns pictures
analysed as a meaningful text, as social semiotics is interested in deconstructing a text to identify the
elements that make up its structure. Its fundamental aim goes beyond the simple understanding of the
structure of relationships and differences that characterise a given sign system. According to Iedema
(2001) deconstructing a visual text, in a systematic manner, is a means to subject its meaning to
critical analysis.
The social semiotic approach led to the multimodal studies of O’Toole (2010), O’Halloran (2010),
Kress and van Leeuwven (2006). According to van Leeuwen (2005: preface) “multimodal social
semiotics is concerned with the way people use semiotic ‘resources’ both to produce communicative
artefacts and events and to interpret them (…) in the context of specific social situations and
practices”.
In the semiotic dimension of Haraway’s conception of reality, (1997), the social relationships include
non-human as well as humans as socially actors. In her point of view, the term semiotics should be
taken, in a broad sense, to express the textual and symbolic dimensions of the world as they go into
fields of practice.
The approach followed for the data analysis, in this paper, is a social semiotic as well as a multimodal
social semiotics one. It allows deconstructing texts into three main types of meaning.
3. The study
Five case studies were used to gather data for this research. They were obtained from two different
research networks: The Leonardo Network on Culture, Creativity and Interaction Design and The
White Rose Network for Affective Communication in Consumer Products and Exhibition Design.
The research work was the study of interdisciplinary art and technology teams designing interactive
artworks. Teams were composed of participants with various professional backgrounds, for example
researchers from the fields of art, design, IT, computer science, engineering, architecture, cultural
studies, and media studies.
The case studies, as the main source of information, comprised meetings, observations, visioning
workshops, interviews and the analysis of artefacts. The artists’ study and the researcher reflections
aimed to contribute to the development of a common language and methodology between
technologists and designers. Firstly, an overview of each case study describing tasks and moments
observed which were recorded on video camera was given to understand who designed what, when,
how, where and why. Then, some of the cases study’s interviews were referenced to clarify and or
complement some details in the data collected. The involvement of artists and scientists brought to
the design practice different perspectives.
Objects are elements of the design process as well as people. An object is an intersecting point in a
network of relations (Hjelmslev 1943) cited in Mattozzi, 2007; and an object is a “body” - a semiotic
figure - (Fontanille 2004) cited in Mattozzi, 2007. It was considered to structure the semiotic analysis
of objects so that it should be possible to account for the ways in which an object (artwork) can
signify. The goal is the revelation of the relevance given to the artworks analysis compared with the
designers’ speech and behaviours which is the reason for considering the object as actors in the
design and dialogue interactions.
The ability of an object to communicate social relationships depends on the relationships which
emerge from its use and also in the way the object is itself a way of communication and how it gets
users to interact with it. Haraway together with Latour (1988, 1993) and others consider that humans
and artworks are intricately bonded. Thus, not only were signs considered vital keys to unlock the
meaning of artworks, the conveyance of those meanings and the designer’s input, but to consider
what the designers actually said and how they behaved during the design process, and to describe
the outcome they produced was regarded as of high import in this research.
4. Results
This section outlines the results from the data analysis of the research. The strategy followed was,
firstly, by using GTM guidelines: transcripts of interviews were analysed and coded; then, categories
emerged, which were grouped to generate concepts. Finally, a theory comes up when the data
232

Arminda Lopes
analysis was saturated. This approach gave meaning and allowed the research understanding of the
data analysis.
Next sections present the results from the application of conversation analysis approach; the
semiotics approach for artwork analysis and the grounded theory.
4.1 Conversation analysis approach
Within conversation analysis, there were some relevant questions to be answered in the description of
conversation: Why did the designers speak one at a time? How did designers know when to change
turns? How did designers know when to initiate new topics? How did designers know it is appropriate
to interrupt? How could a designer complete another speaker’s utterance? How did designers
recognise when a speaker wants to close a conversation?
The answers to these questions were: In conversation there are no pre-set rules, for who talks when
or for how long wasn’t defined. Designers seemed to respect themselves and they made use of turn
taking to take the floor. Transitions from one turn to another with no overlap were regular. The current
speaker selected the next speaker by addressing a question or speakers self-selected in starting to
talk. A new topic was introduced when, for example, the last speaker ended with a sentence like:
“Okay it’s done. Now (…”); or the next speaker introduced the new topic by himself, by using a
question or even a declarative sentence. Frequently, one speaker talked at a time but it happened
that, by the use of gaze or eye contact, and also through some gestures, a speaker interrupted
another. In some cases they made overlaps. Another speaker’s utterance was completed, usually, by
interruption or by waiting for their turn to complete the first speaker’s utterance. The length of
conversation was not specified in advance. When a speaker wanted to close a conversation they said
something like: “Well”, “Okay” or direct information “You need to leave”, “We’ve finished”.
These answers could be given by anyone in any conversation. However the goal was to apply
conversation analysis rules to design teams’ dialogue – a specific type of an individual performing
specific tasks - to show that when they are producing an artefact, they behave in the same way as
any individual behaves in doing any task.
Verbal and non-verbal behaviour go hand in hand, often simultaneously, both either together or apart.
They have a significant impact on whether or not people achieve their objectives with other people.
These behaviours allowed us to better understand the collaborative environment created in both
processes (design and dialogue) and also to understand each designer’s culture.
Concerning non-verbal behaviours, it is considered that all the behaviours had more to do with the
designers’ personality and the influence of the meeting’s contexts. The relation between gestures and
some aspects of personality depended on several different procedures: some gestures reflected a
prevailing emotional state, or a style of behaviour; designers could control and manipulate their
behaviours, and they could even produce the opposite gesture to their true state; some designers
gesture style was, partly, a result of his/her cultural and professional, background, age, and sex or
even health or fatigue state. Some designers made unconscious movements which could be
considered as being closely integrated with the content of the speech itself.
4.2 Artwork description
The artefacts/artwork analysis is an exploratory process involving visual analysis of a dialogue
through a series of drawings that were made in a dialogue context.
The artefacts developed are drawings and words, from this point, all called texts following Kress et al.
(1996:231) who argued that “texts are material objects which result from a variety of representations
practices that make use of a variety of signifying systems, each of which contributes to the meanings
of the text in its own particular way”.
Having as starting point Halliday’s functional grammar, Kress and Leeuwen have provided a grammar
of visual design which operates through semiotic systems taking into account the patterns of
experience, social interaction and ideological positions. The metafunctions of this grammar of visual
design have been adapted to Halliday’s functional terminology: ideational → representational,
interpersonal → interactive, textual → compositional (see table 1).
233

Table 1: Meaning analysis
Arminda Lopes
Representational Meaning Compositional Meaning Interactive Meaning
Syntax (qualities of the artwork - Semantic (forms, purpose, and Pragmatic (relationships).
lines, shapes, colours and textures)
and Materials
meaning)
Figure 1 was taken from the video recorded during a case study workshop as an example of data
analysis. People split in groups and each group answered, in different ways the proposed challenge.
In the group composition a diversity of designers’ background was observed. This group was formed
by designers from a variety of national cultures which influenced in the way they talked and in the
considerable amount of ideas each team presented until a consensus was achieved defining the one
to follow. The challenge was conducive to creativity in the way that the designers found solutions. All
the drawings were executed within a limited time and also using a limited range of media in order to
make their productions as simple as spontaneous as possible. They were executed using paper and
pens, with only black, blue and red ink. Some “drawings” were cut out magazine pictures, some
sketches were accompanied with words. During the presentation and evaluation of each drawing,
arguments were presented to better explain the proposals, which contributed to the other groups
interpretation and understanding.
Figure 1: Dad waiting in a hospital with injured boy
The interpretation and analysis of the artefacts meaning was made considering the syntax, semantic
and pragmatic functions presented on table 1.
Figure 1 a) and b) exemplifies that aiming for a single product was a concept too narrow, so designers
needed to look at the wider context and get away from function. They are sequential phases of the
design process for the same proposal. The first one was made in a previous round of exercises and
the second one was the visual result of that planning process.
Representational meaning – Syntax: The story is “Dad waiting in a hospital for five hours with injured
six year old boy” and the designers suggested a slogan “A waiting room that you won’t want to leave”
which means a room with communication facilities: internet connections, telephones, a blue tooth or a
bleeper system which would enable those waiting to be able to wander around the site without
worrying that they would miss a call for the next phase of the medical process.
There are three different coding agendas: linear, shape and spatial organization. Lines are straight
shapes – straight/curve, symmetrical/asymmetrical, opened/closed; positions – vertical/oblique,
upright/inverted. These contrasts reveal two contrasting parts, that which designers represented by
drawings and that which they found in magazines (technology: games, toys, phones, and computers).
Physical comfort was proposed by pieces of furniture and the entertainment one achieved by
technology as a symbol of a consumerist society.
The objects’ drawings are in coloured ink, and only one is in blue, establishing the room’s comfort and
the connection with the outside world. So, there are two kinds of spaces: inside – a closed circle given
234

Arminda Lopes
by the position of each element, while the outside is represented by the door at the top and the
communicative media which link to the outside.
Compositional meaning – Semantic: There are a considerable amount of emotional feelings
expressed by the words meanings that can be read from the narrative. Two axes are presented: time
and space. Time refers both to physical time and to a psychological one. The former concerns the
time passing while waiting in the closed space of a hospital; the latter refers to the amount of feelings
that appear and disappear during the physical waiting period and by the closure and openness which
are represented by three different spaces: closed, opened, desired. Those can be connoted with the
inside world, outside world and the desired world. The viewer can draw, through lines and shapes,
both of the places.
In this text, the main message evoked through the written words is one of anxiety and frustration. A
clear amount of negative feelings arose from the story – some interacting with the space and others
interacting with the situation. All of these feelings belonged to the same axis: distress, boredom,
displeasure, impatience, irritation, annoyance, preoccupation, sadness, pain, closure, etc.
The text is organised into three columns. Each column has an emotional reference: the top of the left
hand one connotes with loss, the top of the central one referring to frustration and the top of the right
hand one representing boredom. Then, the each column bottom shows the emotional details of each
head. The central post-it represents the story. The hand written notes, surrounded by ovoid forms on
the background are the goals to attain.
The sketches composition is made through proportional and non-proportional forms. For example, the
size of the users head is bigger than the sofa or the table, which seems to underline the main focus of
the text: the users comfort and the communication with the outside world. Pictures from a catalogue
represent the desired objects to have in the room. Some pieces are well defined while others are
sketched.
Entertainment is represented by male associated activities expected to assist including pool, televised
football, “scalextric” etc with plenty of other toys to keep the patient amused. There is a bar with a
‘blokey’ non-medical ambience represented by the glass and the pictures of a bar with bottles. The
notion of comfort is mainly represented by the sofa which is in a different colour and the room
decoration: a painting on the wall and the desk oil lamp.
Interactive meaning – Pragmatics: the designers’ message seems to be straightforward “A waiting
room you don’t want to leave”. All the decorations and appliances pieces show the contrast with the
text motif and establish a relationship between producer and receiver. The interaction between the
inside world and the outside is expressed by the drawing with a tree and a flower in the upper left
hand side. This room would provide occupation and important contact with the outside world as
desired.
4.3 Grounded theory - conceptual Insights
Thinking about concepts of theory in Grounded Theory is not straightforward. However, after
categories saturation steps, it was decided to track an interpretative definition of theory which
emphasises understanding rather than explanation. We interpret our designers’ meaning and actions
as a starting phase for analysis. Then, we observed what they were saying and how they behaved
during the design process. Using the comparative method, a theory emerged from this comparative
work.
Four key concepts within the design process were identified: communication,
collaboration/cooperation culture, and creativity (Figure 2). The designers communicated their ideas,
demonstrated by the proposed interpretation of the outcome and supported by the fact that there is
some information about society and the designers’ background represented in the artwork, which is a
reflection of the way they expressed their ideas. The designers collaborated in a true partnership
manner, adopting a common and shared methodology and not a juxtaposition of individual
approaches. They exchanged experiences aiming at a better understanding of the situation at hand.
Cultural clues were identified in the way each team’s member presented ideas and behaved during
the design process. Creativity occurred through the application of some actions from different areas of
235

Arminda Lopes
knowledge; the thinking process was a method revealing this creativity. The proposed challenge was
contributing to creativity in the way the designers found the solution to the problem at hand.
Figure 2: Four Cs in design and dialogue processes
Finally, following Charmaz (2006), in a constructivist position, the priority was placed on the
experiences and relationships with the participants in the design process and on meanings and
actions they constructed. At this stage, a framework was considered.
The framework of design, i.e. design as dialogue, was, mainly, to be used as a tool which facilitates
the view of designers’ work as dialoguers; to provide guidelines to analyse and interpret design; to
improve design practices. The framework tries to answer some questions, for example: How might
each dialogue feature be relevant in the design/dialogue process? How might verbal and non-verbal
behaviours have influence on the design process? What are the main types of meaning to consider in
analysing artefacts? What are the features that allow contextual information in the interaction
process?
5. Discussion
The concept of communication transverses all the process of design but the way the designers
communicated was better understood by the way they used language. Four Cs were found during the
design and dialogue processes: communication, (through which designers transmit their ideas and
maintain the dialogue), collaboration, (the method used to show the team’s engagement) and
creativity (presented in the way designers participate in the process and also along it), in addition the
culture being reflected in the produced artefact/artwork.
Answering the questions considered by conversation analysis, it can be said that sometimes a
speaker positioned another participant into a particular role when he wanted more information to be
added and the other was the expert in a specific subject. Designers adapted their speech according to
the person they were talking to and also according to the point behind the speech. The transcript’s
use of language functions, during conversations helped to understand verbal behaviours variations
through the way designers influenced and interacted with the kind of language used and how they
made use of associated functions which language fulfils in different situations. Designers, in general,
showed respect for the opinions they were hearing, indicating they cared both about the person
speaking and about what s/he was saying. Some gestures of empathy were made to indicate the
appreciation about what had been said.
The results of the artwork analysis which were written up provided only tentative conclusions about
the relationship between the artworks and dialogue. The artworks simultaneously indicated and
constituted the design process carried out in different situations, with different goals and by people
with different backgrounds.
Both conversations and artwork practice were understood in this research as forms of dialogue
intimately linked through meaning and, as meaning and interpretation are inherent in the relationship
236

Arminda Lopes
between the verbal and the visual, the practical and theoretical elements involved in the investigative
process of the research were mutually dependent.
The artworks as tangible results of practice were shown to facilitate a profound form of
communication that went beyond conventional language, and can therefore be understood as
dialogical phenomena that both derive from and embody the emotional and intuitive reasoning.
The framework permitted to interpret that: designers presented their assumptions and opinions. The
way they carried this out was by stating not by defending against somebody who had another opinion;
they contributed their knowledge to the dialogue in progress: it was a process of sharing; Suspension
involved attention, listening and looking and it also involved exposing a designer’s reactions,
impulses, feelings and opinions in such a way that they could be reflected back to the others in the
group; Inquiry and reflection was made by asking open-ended questions as a form of exploration of
assumptions and beliefs; Reflection provided the opportunity to review and connect with what had
been said. Reflection also provided the opportunity to slow down and collect their thoughts. Good
listening was both an active and passive skill, it took an effort to really hear and digest what was being
said.
6. Conclusions
This paper explored what happens when designers within multidisciplinary teams produced an
artefact/artwork. In investigating this topic, the research applied and integrated a wide range or
research methods and techniques in the analysis of data from observed case studies. Three
methodologies were used: Grounded theory, conversation analysis and semiotics.
Amongst other contributions, this research provided a framework, which is out of the scope of this
paper, for analysing design as dialogue derived from the data analysis of the case studies. The
researcher started to analyse the data and then to build the framework. Then, using the Grounded
Theory research process, which is interactive and open-ended, the framework was used as a tool to
analyse the final data, i.e., initially the data supported the creation of a framework and the framework
served to use as a model to analyse the final data.
The analysis of the verbal and non-verbal data was carried out through the use of Conversation
Analysis approach.
According to the analysed artefacts/artworks, designers are in general preoccupied with social
interaction solutions. Design teams have conversations during the design process and during the
conversations designers discuss their ideas, and it is apparent they come from different angles –
differing levels of experience, differences in age and profession, differences in abilities to deal with
technical or non-technical tasks, among others. In the artwork/artefact development, in some cases,
designers worked on separate pieces of the work and at the end they are joined together. The
solution to allow ideas exchange, to find connections and meaning, and interests in a real
collaborative environment is to use dialogue as a tool.
The three methodologies suited for the study since they permitted to find meaning and understanding
through verbal, non-verbal and image data.
References
Barthes, Roland, (1957): Mythologies, translation 1972 by Jonathan Cape, Ltd.
Charmaz, Kathy (2006), Constructing Grounded Theory, A Practical Guide Through Qualitative Analysis, SAGE
Publications, London, Thousand Oaks, New Delhi.
Coulthard, M. (1977), An Introduction to Discourse Analysis, Essex: Longman.
Denzin, Norman K. and Lincoln, Yvonna S. (eds), (1994), Handbook of Qualitative Research, Thousand Oaks:
Sage.
Foucault, M. (1971), Orders of Discourse’ Social Science Information, 10:7-30.
Gilbert, G. N and M. Mulkay, (1984), Opening Pandora’s Box: A Sociological Analysis of Scientist’s Discourse,
Cambridge University Press.
Glaser, Barney, (1998), Doing Grounded Theory – issues and discussions – Mill Valey, CA: The Sociology Press.
Halliday, M.A.K, (1994), Introduction to Functional Grammar, 2nd Edition, London: Edward Arnold.
Haraway, Donna, (1997), Modest_Witness Second_Millennium: FemaleMan meets OncoMouse, feminism and
technoscience, New York; London: Routledge.
237

Arminda Lopes
Ledema, R. (2001) “Analysing film and television: a social semiotic account of Hospital”: An Unhealthy Business,
In T. van Leeuwen & C. Jewitt (Eds.), Handbook of Visual Analysis (pp. 183-206) London: Sage.
Kress, Gunther; van Leuwen, Theo (2006): Reading Images – the Grammar of Visual Design, 2nd edition,
London: Routledge.
Mattozzi, Alvise, (2007), Design Semiotics in Use: Se Fun International Seminar, June 6-8, 2007 from
http://www2.uiah.fi/sefun/DSIU_papers/DSIU%20_%20Mattozzi%20_%20Model%20for%20semiotic%20an
alysis.pdf (accessed on 24/01/12).
O'Halloran, K. L., Tan, S., Smith, B. A., and Podlasov, A. (2010), 'Challenges in Designing Digital Interfaces for
the Study of Multimodal Phenomena', Information Design Journal, 18(1), 2-12.
O'Toole, M. (2010), The Language of Displayed Art (2nd ed). London and New York: Routledge.
Potter, J. and M. Wetherell, (1987), Discourse and Social Psychology, London: Sage.
Saussure, Ferdinand de (1974): Course in General Linguistics (trans. Wade Baskin) London: Fontana/Collins.
Sidnell, Jack, (2010), Conversation Analysis: An Introduction, John Wiley and Sons.
Stubbs, Michael, (1983) Discourse Analysis – the Sociolinguistic Analysis of Natural Language, Basil Blackwell,
Oxford.
Titscher, et al, (2000), Methods of Text and Discourse Analysis, Sage Publications Ltd.
Van Dijk, T.A, (1985), Handbook of Discourse Analysis, Vols 1-4, London, Academic Press
van Leeuwen, T. (2005), Introducing Social Semiotics. London: Routledge.
238

Are Employment Tribunals a Barrier to Justice? A Mixed
Methods Approach
Jonathan Lord
University of Salford, Manchester, UK
j.d.lord@salford.ac.uk
Abstracts: Two publications from the British Chamber of Commerce and Chartered Institute of Personnel and
Development stated that the Employment Tribunal System (ETS) is broken due to a number of reasons, including
the amount of vexatious claims and the abuse of the system by claimants. The statements by the two employers
organisations instigated an exploratory analysis of unfair dismissal claims from 1998 – 2011.The intention of the
study is to analyse employment tribunals in respect of its purpose and whether the system is broken as purported
by employers organisations. Through analysing users’ perceptions of the Employment Tribunal Service, the
intended outcome is to demonstrate that although there are problems with the system, it is not a barrier to justice
and that it actually needs to be modernised to ensure that the service is accessible for people who have a dispute
with an organisation. Employment Law is an extremely political and controversial issue at the moment, therefore
this research will analyse whether the current employment tribunal system is effective, and most importantly
identify potential changes to improve the system. A mixed method approach is unusual within a law discipline and
would yield information that could be used to shape government thinking. This paper analyses the possible use of
a mixed method approach to studying the ETS.
Keywords: mixed method, employment law, triangulation, employment tribunal, convergence model
1. The context of the research
The Employment Tribunal System (ETS), as it has been known since 1997, is an integral aspect to
the application of employment law, and has facilitated a fluctuating increase in claims since the ETS
was established in 1997. Figures from the ETS annual reports show 218,100 cases were taken to
tribunal in 2010 – 2011 in comparison to 236,100 cases in 2009 – 2010 and 151,028 cases in 2008 –
2009. Although there was a 20% reduction in accepted tribunal claims in 2008 -2009, largely
attributable to the airline employees as well as falling number of equal pay and sex discrimination
claims, the actual number of single claims has risen in comparison to previous years, leading to
additional workload for the tribunals. The table below outlines the figures from the creation of the ETS:
Table 1: Employment tribunal claims
Year No. of Applications No. of Single Claims(One
claim can have multiple
jurisdictions)
April 1997- March 1998 80,435 80,435
April 1998 - March 1999 91,913 148,771
April 1999 - March 2000 103,935 176,749
April 2000 - March 2001 130,408 218,101
April 2001- March 2002 112,227 194,120
April 2002- March 2003 98,617 172,322
April 2003- March 2004 115,042 197,365
April 2004- March 2005 86,189 156,081
April 2005- March 2006 115,039 201,514
April 2006 - March 2007 132,577 238,546
April 2007 - March 2008 189,303 296,963
April 2008 - March 2009 151,028 266,542
April 2009 - March 2010 236,100 392,700
April 2010 – March 2011 218,100 382,400
Industrial Tribunals (as they were originally known) were initially established under s.12 of the
Industrial Training Act (1964) for the purpose of considering appeals by employers against training
levies imposed under that Act. Through the researcher’s own experiences and a review of the current
239

Jonathan Lord
literature, it has been noted that the tribunal system has evolved over the last five decades into a
complex and formal process that has detracted from its origins as an informal mediation service.
1.1 Importance of the study
Tribunal figures have shown that the use of the service has increased substantially over the last
decade in spite of legislative procedures being introduced, which have been designed to reduce
recourse to tribunals. The service is an integral aspect to employment law and also an important
mediator in disputes between claimants and respondents (including prospective workers). Therefore it
is essential that this service supports and benefits all individuals involved so that a resolution can be
made in the most efficient, effective, reasonable and inexpensive way possible.
This research will complement existing literature as well as contribute towards the factors that will
shape, evolve and develop the model and processes of the ETS. In particular this study will differ from
previous ones in that it will not focus purely on empirical data, collected as part of a governmental
research series, but will explore the real thoughts and opinions of users of the service. As outlined in
chapter two, previous research has tended to focus predominantly on the statistical aspect to the ETS
or how claimants are treated inappropriately. This study will extend beyond the standard analysis of
the ETS and provide a unique insight into the affects and ramifications of tribunals on the employer /
employee relationship.
2. The employment relationship and the law
The relationship between the employer and the employee has changed considerably since the middle
of the 19 th Century, with the commencement of the Industrial Revolution and with the emergence of
capitalism facilitating this change. Burgess (1980) commented that this has resulted in the change of
relationship between employer and employee, shifting from traditional forms based on status e.g.
servitude, to one that forms the notion of a contractual relationship, ‘although workers were not as
‘free’ as portrayed, they were clearly emancipated compared to conditions suffered under previous
regimes’ (Thomason, 1984:2). Historically, markets had not penetrated the pre-industrial skilled
trades, which worked on a customary basis of wage bargaining rather than aligning this with the
demands of the markets. Therefore traditional craft skilled workers had more control over their own
wage bargaining process and had limited restraints, on exploitation of their labour, due to the customs
and practice of the trade over the decades.
With the increase in factory based production within key sectors, such as textiles and engineering,
this enabled employers to reframe the principles of determination of pay, based on the market
conditions (Burgess, 1980).
3. The introduction of ‘law courts’
The Donovan Commission had severe reservations regarding the efficiency, effectiveness and also
appropriateness of Industrial Tribunals, stating that, ‘The Multiplicity of jurisdictions is apt to lead to
waste, to frustration and to delay’ (The Royal Commission ..1965 – 1968: para 570).
This example supported the commission’s recommendations:
…that industrial tribunals should be enlarged so as to comprise all disputes between the
industrial worker and his employer… not only overcome the present multiplicity of
jurisdictions …also…produce a procedure which is easily accessible, informal, speedy
and inexpensive, and which gives them the best possible opportunities of arriving at an
amicable settlement of their differences” (The Royal Commission on Trade Unions and
Employers’ Associations 1965 – 1968: para. 572).
The Donovan Commission seemed to have a utopian vision of Employment Tribunal’s (ET’s)and how
they should be designed, developed and delivered. This vision may have concentrated too much on
the future and failed to address the underlying current problems with ETs. To understand Donovan’s
recommendations, it is appropriate to identify and evaluate the key elements of their report. The
recommendations can be segmented into the following sections:
Easily Accessible
Informal
Speedy
240

Inexpensive
Easily Accessible
Jonathan Lord
A key aspect of the Donovan Commission’s recommendations and problems with the current system
was the accessibility of hearings. The Donovan Commission envisaged ‘that they will be operating in
all major industrial centres and thus easily accessible’ (The Royal Commission on Trade Unions and
Employers’ Associations 1965 – 1968: para. 548). This not only had the premise that both the
employer and the employee could access a tribunal in their area rather than some considerable
distance away, but would also speed up the process of resolving the dispute.
Informal
The Donovan Commission’s ideal for having an ‘informal’ ET structure was divided into two segments.
Firstly, they wanted the whole ET to have a more informal approach, from the application process to
the hearings and judgements. Secondly the Commission wanted an informal approach to the
resolution of disputes, through having a pre-hearing. This sought to have the obvious benefit of
resolving the dispute in a setting away from the more formal ET.
Speedy
The idealology of a speedy ETS is linked to the area of developing an informal approach, as a rapid
resolution of the dispute would be addressed through an informal pre-hearing. It can also be linked to
the process of an ET hearing being informal, as ET ‘court rooms’ are far less formal than other court
systems. For example there are no wigs or gowns and the rooms, where disputes are heard, are
considerably less imposing. At criminal courts for example, both parties are expected to present their
case and challenge the case of the opposite side. At the commencement of ET’s, the Chair was
expected to support either side in ensuring all of the relevant facts were presented and communicated
clearly. Through an ‘informal’ process, employees with a dispute would be able to apply to an ET and
have this resolved (either to the benefit of the employee or employer) in a reasonably speedy
timeframe, in comparison to other courts.
Inexpensive
As well as the ideal of ETs being reconciled in a speedy fashion, the Donovan Commission envisaged
them to be inexpensive. What the Donovan Commission were referring when they stated
‘inexpensive’ is subjective, but can be interpreted that it would be inexpensive for the parties involved,
the employee and employer. Therefore it is possible to interpret this recommendation this way as the
Donovan Commission had made a number of proposals which would have severely increased the
cost for the government, such as the recommendation to create more ET hearing ‘courts’, across a
increased number of towns and cities.
3.1 The employment tribunal service
Previous sections in this study have discussed the creation and development of ETs, therefore this
section will concentrate on the impact and effectiveness of the service. Lord Wedderburn’s speech in
the House of Lords (Wedderburn, 2009) regarding the composition of the ETS is evidence that the
discussion around the service is still prevalent and necessary to ensure the service continues to
develop and fit the needs of a modern employment relationship.
3.2 Impact of tribunals on the way employers manage relationships
The use of employment legislation and the ETS has always been viewed disparagingly by employers.
Evans et al., (1985) states that although a fairly muted response was received in 1971, when unfair
dismissal was first introduced, employers were more vociferous when legislation and the remit of ET’s
were extended in 1975. Objections raised at the time, which look very familiar in today’s workplace,
included they would be a deterrent to job creation, tactics would be used to avoid legislation (for
example temporary contracts) and an increase in bureaucracy. A study at the time by Clifton and
Tatton-Brown (1979) countered this notion and in fact demonstrated that they had little or no impact
on firms. In the study, 4% of companies suggested that this had an impact upon their business, and
rated lowly in a list of main difficulties in running a business. However, an interesting factor of this
241

Jonathan Lord
study was that it reiterated that small businesses were largely ignorant of the law. Although, over the
last four decades, this has changed, with data from the Workplace Industrial Relations Survey (2004),
finding that 76% of companies had a disciplinary and grievance procedure in place. This can be
associated to the introduction of a raft of employment legislation, a rigid process for following
disciplinary and grievances, and the threat of a costly visit to an ET.
As previously discussed, the original master-servant contract and the fundamental inequity in that
relationship has resulted in an attempt to try to and rectify the disparity. The establishment of
employee rights and the removal of legal constraints on collective bargaining have changed the
nature of employment relations. Although legislation has been in place since the late 19 th Century, it
was not until the 1970’s, and the creation of ETs that protection for employees was realistic. Gennard
and Judge (2010:273) relate the protection back to the Industrial Relations Act (1971) stating that:
…the act gave individual employees the right, for the first time, to complain to an
industrial tribunal that they had been unfairly dismissed. The 1971 act was a turning-point
in the relationship between employer and employee. The relative informality of the then
industrial tribunals and the fact that access to them did not depend on lawyers or money
meant that for many employee the threat of dismissal without good reason disappeared
or diminished.
Gennard and Judge (2010:273) go on to counter this by warning:
…this does not mean that employees cannot be unfairly dismissed. They can. The law
has never removed from management the ability to dismiss who it likes, when it likes,
and for whatever reason it likes. All that has happened since 1971 is that where
employers are deemed to have acted unreasonably and unfairly in dismissing
employees, they can be forced to compensate an individual for the consequences of
those actions.
The commentary by Gennard and Judge provides a salient discussion around the impact and
influence of the ETS on how employers manage and relate to staff members. Despite the protection
and threat of a tribunal, employers are still dealing with employees in a fashion that was in effect prior
to the creation of tribunals. Employers still argue that the relationship with their employees should
remain private and any disputes should not be dealt with in a public arena Rowlinson (1993).
An ET therefore will measure whether ‘natural justice’ has taken place. Employment law has
formalised the ideologies of natural justice, into a rigid framework that employers have to adhere and
therefore natural justice has been supplanted into employment legislation, whereby the action of the
employer in dealing with the employee is measured just as highly, if not more so, than the actual
actions or non actions of the employee. The threat of an ET has resulted in employers not being able
to act ‘instinctively’, but forcing them to follow set procedures. This is also applicable for employees
who are also pressed into following set, rigid procedures before being able to apply to an ET or have
their potential award reduced by up to 25%. The employee may feel uncomfortable in raising the
issue with their employer due to the possible implications and therefore would prefer the tribunal to
deal with the issue(s). The ETS therefore have tried to regulate the employment relationship, which in
some cases can be detrimental to both parties.
3.3 Impact of tribunals on employees in seeking to resolve conflict
Although it can be argued that with the advent of ETs a new raft of ‘disputes’ have arisen, some
commentators have linked the rise in applications to tribunals over the last four decades to the decline
in trade union power (Shackleton, 2002). Drinkwater and Ingram (2005) argued the use of tribunals
was a reaction to the purification of collective action, due to the difficulty experienced by trade unions
in being able to organise and act collectively. A clear accusation is that tribunals offer a new avenue
of resolution for the same conflict, which would have previously been voiced through collective action.
There is a clear correlation between the increase in ET applications and a decrease in trade union
membership. Numerous writers have substantiated this and agreed that the fall in trade union
membership and therefore a relaxation in collective agreements, has been affected by the
individualisation of the employment relationship (Kelly, 1998; Brown et al., 1998: Hawes, 2000). With
the decrease in trade union membership, employees have therefore used ETs to resolve any disputes
they may have. However, the nature of resolving conflict through a trade union or an ET is radically
242

Jonathan Lord
different. For example, an ET will ask the claimant what remedy or remedies they are seeking, with
three options:
To get their old job back and compensation (reinstatement).
To get another job with the same employer and compensation (re-engagement).
Compensation only.
There could be a change in this idea though, as Gibbons (2007), states there is a growth in the
number of multiple tribunal cases involving a single issue (or set of issues) which are or have affected
a number of employees in the same workplace. Gibbons (2007:98) attributes the increase in multiple
cases, ‘directly to collective issues, with the rate of such cases fluctuating substantially because it is
heavily influenced by large scale disputes’. Gall and Hebdon (2008) labelled this ‘semi- collective
action’ and is estimated to cost employers £3 billion and predicted to rise to £5 billion with base pay
liabilities and future wage bills. The recent case of Birmingham City Council being taken to an ET,
compounds this theory. In the case of Baker and others v Birmingham City Council (ET
1305819/2006), more than 4,000 female workers won their equal pay claim in an ET. The female
employees were employed across 49 different job, which were traditionally dominated by females,
including cleaners, carers, cooks, care assistants and teaching assistants, and were on the same pay
grade as men. They complained of being excluded from bonuses worth up to 160% of their basic pay
which were paid to men. The workers were represented by different organisations, for example 900
were represented by Steton Cross Solicitors. Historically, a dispute like this would have involved trade
union negotiations and possible industrial action. The fact that the amount of compensation sought
was large (£200 Million) resulted in the employees taking action which they knew was the only option
open to achieve the compensation sought.
Batstone et al’s (1977) assertion that strikes were a tactical extension of ‘collective opposition’ can
therefore be assimilated to the use of ETs, as in the case of Birmingham City Council, when an initial
grievance was raised and refuted regarding the issue of equal pay. The application to the tribunal
service is the obvious next step in the tactics of employees to resolve their dispute. Metcalf (2003)
disagrees with this, and concludes that the use of tribunals is a necessity due to the restrictions
placed on trade unions by the Conservative government (1979-1997):
…the strike threat…was weakened by a succession of laws which permitted a union to
be sued, introduced ballots prior to a strike, and outlawed both secondary an unofficial
action (Metcalf, 2003:175).
The policies of the Conservative government continued with the successive Labour governments
(1997- 2010), which decided to continue with the same policy, stating that:
…there will be no going back, the days of strikes without ballots, mass picketing, closed
shops and secondary action are over” (Department for Trade and Industry, Fairness at
Work, 1998:98).
With Labour (1997-2010) continuing with restrictive practice policies on trade unions, ETs have
developed into the only viable option for employees to try and resolve their disputes. The explosion in
individual employment rights (Dickens, 2000) has now swept into the realm of employment relations,
‘which had previously been a matter of voluntary determination’ (Dickens and Neal, 2006:7). With the
implementation of the Statutory Disciplinary and Grievance Procedures (2004), employers and
employees were effectively forced into channelling disputes into a set, rigid process. Even though this
has been repealed after the Gibbons Review and replaced with the semi-mandatory Acas Code of
Practice (2009), (implemented through the Employment Act, 2008) organisations and employees are
still advised to follow the process formalised in the Statutory Disciplinary and Grievance Process
(2004), with the added threat of awards at tribunals being reduced or increased by 25% if the correct
processes are not followed.
It has been suggested though that the impact of the procedural changes, brought in through the
Employment Act (2002), has downgraded rather than enhanced procedural fairness (Hepple and
Morris, 2002). Although the procedures have been updated to mirror the Acas Code of Practice
(2009), Hepple and Morris (2002) still argued that even though these set processes are in place,
employers could follow the statutory procedure and could escape a finding of unfair dismissal on the
grounds that procedural defects would have made ‘no difference’ to the decision to dismiss. Knowing
this, employees may feel that applying to a tribunal would therefore be futile. Kelly (2008) argues that
the overall growth in tribunal claims indicates rising levels of discontent at work. However, only a
243

Jonathan Lord
minute fraction of grievances that could result in a tribunal claim, actually result in this action being
taken, with only a quarter advancing to an actual hearing. Brendan Barber from the Trade Union
Congress (TUC) is quoted in EIRO (2001:9) as stating:
I am fed up listening to employers griping about a so called compensation culture.
Tribunal claims do not arise because sacked workers are ‘having a punt’. Only around
30,000 claims a year go to a full tribunal hearing. Meanwhile, as many as three-quarters
of a million times a year employers get away with actions that could land them in a
tribunal. That is the real scandal.
There is an argument that the sector in which a person works can affect the application to a tribunal.
Dickens (2000:34) bluntly states that:
…the number of people employed in those parts of the private services sector, where the
management of employment relations is often conducted in a harsh and arbitrary manner
by small employers and unions are weak, is generating more claims.
Another problem with ETs involves the stigma attached to the person who has submitted an
application against their employer. As tribunals publish most case information in the public arena
(except for ‘restricted reporting’ cases) it is possible for prospective employers to determine whether
someone has submitted tribunal applications previously, which would the ‘warn them off’ and not
employ them. The TUC’s (2008) commission on vulnerable employment reported that workers:
…are stopped by fear that even though they might have the strongest case in the world,
once the next employer who they got finds out they took the previous employer to the
tribunal, the chances of getting a job go out of the window (TUC 2008:132).
3.4 The costs and benefits of tribunals
Analysing the costs associated with the tribunal system is a difficult and controversial task
(Shackleton, 2002) and there is a need to distinguish the various diverse notions of cost. An analysis
of the tangible direct cost involved in operating the ETS is outlined below and in accordance with
guidance from Shackleton (2002), a further analysis of other direct costs (i.e ACAS) and also
compensation, compliance and indirect costs’ will be provided during the secondary analysis.
Table 2: Employment tribunal service income and expenditure
2006/07 2007/08 2008/09 2009/10
£000's £000's £000's £000's
Income £22 £35 £12 £0
Administrative Staffing Costs £15,338 £14,630 £15,091 £15,781
Accommodation costs £13,415 £14,100 £12,791 £14,777
Administrative overheads
Hearing costs (including Judicial
£11,556 £11,229 £11,084 £14,291
Salaries / Fees) £30,762 £35,051 £36,915 £39,541
Total Expenditure £71,049 £74,975 £75,869 £84,390
4. Research design
In selecting the most appropriate research design the following three elements will be considered:
The research problem
Personal experiences
Audience
Certain types of social research problems require specific approaches (Creswell, 2009). If a concept
or phenomenon needs to be appreciated due to little or no research being available, then the most
appropriate approach would be qualitative methods. If the problem prompts the following:-
The identification of factors that that influence an outcome
The utility of an intervention or
Understanding the best predictors of outcomes
244

Jonathan Lord
Then the quantitative approach is the most appropriate method. The mixed method approach would
be used to ‘understand’ a problem in a clearer way and better way, as it utilises the strengths of both
approaches. Therefore the research problem should influence the whole design of the study.
As this study is supervised and assessed by academics in the chosen topic area and also in research
practices, the approach should be centred around the requirements of assessment, the areas of
research required within the research centre and also for potential researchers in the future, who may
use this study to develop new fields of analysis and develop the topic even further.
4.1 Research process and framework
Within the research process two questions need to be addressed, firstly, what will be the methodology
or methods of the research and secondly, what is the justification for the choice of methodology and
methods utilised?
To answer these questions, Crotty (1998:3) has developed four questions and themes, which are:
What methods do we propose to use?
What methodology governs our choice and use of methods?
What theoretical perspective lies behind the methodology in question?
What epistemology informs this theoretical perspective?
4.2 Research design - the triangulation design
Denzin (1978:291) defined triangulation as “the combination of methodologies in the study of the
same phenomenon” and is based on the triangulation metaphor from navigation and military strategy
which, according to Smith, (1975), uses multiple reference points to locate an object’s exact position.
Within the field of social sciences, triangulation can be attributed to Campbell and Fiske (1959), who
devised the theory of ‘multiple operationism’. An example of how this work within the field of
management is through appraisals. A manager’s effectiveness could be assessed through a one to
one interview, an evaluation of performance record and a ‘peer review’. Although the focus remains
on the effectiveness of the manager, the method of data collection varies. Jick (1979) used the
triangulation strategy to study the effects of a merger on employees over a period of fourteen months.
To achieve multiple viewpoints, three observational approaches were utilised; feelings and
behaviours, direct and indirect reports, obtrusive and unobtrusive observations. The main reason for
using the triangular design, is for the researcher to, either directly compare and contrast qualitative
and quantitative information or to substantiate or develop quantitative results with qualitative data.
Creswell and Plano Clark (2007) have devised a triangular design which demonstration how
quantitative and qualitative methods are used within the same timeframe and adopt equal weighting.
Quantitative
Interpretation based on
Quantitative and Qualitative
Results
Qualitative
Figure 1: The triangulation design
The triangulation design has been labelled the ‘concurrent triangulation design’ by Creswell and Plano
Clark (2007) due to its single phase nature and timing. To best understand the problem or
phenomenon, the research will collect qualitative and quantitative data separately but in tandem, and
then proceed to merge the two data sets. The merging of data can be done through interpreting the
data collectively or by “transforming data to facilitate integrating the two data types during the
analysis” Creswell and Plano Clark, 2007:64). An example of this type of research can be viewed in
Jenkin’s (2001) single phase study regarding ‘Rural Adolescent Perceptions of Alcohol and other
Drug Resistance’. Qualitative data was collected in the form of focus groups and quantitative data
through semi-structured questionnaires. The two data sets were then interpreted by relating the
qualitative results in the qualitative findings.
245

Jonathan Lord
Mixed Method Approach - The convergence model approach
Figure 2: Mixed method approach: The convergence model
5. Future research activity
This piece of research will be used as a platform to build on the findings and knowledge gained from
the information produced from the study. The researcher has a specific interest in employment
legislation, in particular the practical implementation of legislation into work places. Therefore the
researcher envisages further research being carried out around this topic area, with a number of
themes identified for future research:
The use of technology within the tribunal system
The changing requirements of employment legislation, in relation to the different permutations that
people are now employed or contracted for work
The impact of the Equality Act (2010): once a relative active period has passed, the researcher
intends to analyse, through tribunal judgements, how this huge piece of legislation has affected
employers and employees.
References
Barker and others v Birmingham City Council (ET 1305819/2006)
Batstone, E. I. Boraston and S. Frenkel (1977) Shop Stewards in Action: The Organisation of Workplace Conflict
and Accommodation, Oxford: Blackwell.
Burgess, K. (1980). The Challenge of Labour (1st ed.). St Martin's Press.
Campbell, Donald T. and D. W. Fiske., 1959 "Convergent and discriminant validation by the multitrait -
mutimethod matrix." Psychological Bulletin, 56: 81 -105.
Clifton, R. and Tatton-Brown, C. (1979), Impact of Employment Legislation on Small Firms,
Department of Employment Research Paper No. 6.
Creswell, J (2009). Research design: Qualitative, quantitative, and mixed methods approaches. London. Sage
Pub.
Creswell, J. W., and Plano Clark, V. L. (2007).Designing and conducting mixed methods research. Thousand
Oaks, CA: Sage Publication
Crotty, M. (1998). The foundations of social research. London: Sage.
246

Jonathan Lord
Denzin, NK. (1978). Sociological Methods. New York: McGraw-Hill
Dickens, L. (2000) ‘Doing more with less: Acas and individual conciliation’, pp.67- 92 in W. Brown and B. Towers
(2000) Employment Relations in Britain: 25 years of the Advisory, Conciliation and Arbitration Service,
Oxford: Blackwell.
Dickens, L., & Neal, A. C. (2006). The changing institutional face of British employment relations.
Kluwer Law International.
Donovan, Lord (Chairman) (1968) Report of the Royal Commission on Trade Unions and Employers'
Associations, London: HMSO .
Drinkwater, S., & Ingram, P. (2005). Have Industrial Relations in the UK Really Improved? Labour, 19(2), 373-
398.
DTI. (1998). Fairness at Work.
Employment Act 2008
Equality Act 2010
European Industrial Relations Observatory 2001
Evans, S., Goodman, J. and Hargreaves, L. (1985), Unfair Dismissal Law and Employment Practice in the 1980s,
Department of Employment Research Paper No. 53
Gall G, and Hebdon R. Conflict at work. In: Blyton P, Bacon N, Fiorito J, Heery E, editors.The SAGE Handbook of
Industrial Relations. Los Angeles: Sage; 2008. p. 588-605.
Gennard, John, & Judge, G. (2010). Employee Relations (5th ed.). CIPD.
Gibbons, M. (2007). A review of employment dispute resolution in Great Britain. Database (p. 64).
Hawes, W. (2000), “Setting the pace or running alongside? ACAS and the changing employment relationship”, in
Towers, B. and Brown, W. (Eds), Employment Relations in Britain: 25 years of the Advisory, Conciliation
and Arbitration Service, Blackwell, London, pp. 1-30.
Industrial Relations Act 1971
Jenkins, J.E. (2001). Rural adolescent perceptions of alcohol and other drug resistance. Child Study Journal,
31(4), 211- 224.
Metcalf, D. (2003). 'Trade unions', pp.170-190 in R Dickens, P Gregg and J Wadsworth (eds). The Labour Market
Under New Labour. Basingstoke: Palgrave-Macmillan.
Smith, H. W. 1975. Strategies of Social Research: The Methodological Imagination. Englewood Cliffs, NJ:
Prentice Hall.
Shackleton, J. R. (2002). Employment tribunals: their growth and the case for radical reform (p. 117). Institute of
Economic Affairs.
Thomason, G. (1984). A Textbook of Industrial Relations Management (1st ed.). Institute of Personnel
Management.
Wedderburn, L. (2007). Labour Law 2008: 40 Years On. Industrial Law Journal, 36(4), 397-424.
247

Towards Bridging the Quantitative–Qualitative Divide in
Business and Management Research
Hendrik Marais
Graduate School of Technology Management, University of Pretoria, Pretoria,
South Africa
hc.marais@up.ac.za;
marais.stnetwork@gmail.com
Abstract: The paper addresses the methodological commonalities linking quantitative and qualitative
methodologies. It offers a three dimensional framework of research methodology that spans the assumed divide
and shows that quantitative and qualitative research approaches can be mutually complementary and offer a
more nuanced approximation of the truth. The framework, Research methodology in 3D, is based on a metamethodological
analysis and consists of three root dimensions, namely, firstly, four constituent components
presented in all forms of research (concepts, questions/hypotheses, observation/measurement and
communication). The second dimension of the framework consists of the core set of criteria that all scientific
research must comply with (reliability, validity, ‘objectivity’ and replicability), each may be expressed under
different labels and in different forms depending on the specific research tradition. The third dimension represents
the basic goals of research, viz. to describe, to explain or to understand. A three dimensional framework emerges
when the above dimensions are combined. The framework shows some of the tangent planes between
quantitative and qualitative scientific research and suggests possible complementarity between research
approaches. The framework furthermore serves at least the following functions: firstly, as map to assist students
in charting their way through a maze of new concepts. Secondly, the framework is a useful planning instrument
for comprehensive and complex research programmes. Thirdly, the framework can be utilized as a useful guide
in evaluating research proposals on the one hand and research reports, on the other.
Keywords: research methodology, quantitative research, qualitative research, mixed methods
1. Introduction
According to Bryman, “combining quantitative and qualitative research has become unexceptional
and unremarkable in recent years” (2006: 97). His content analysis of over 200 articles showed that
the main reasons offered by authors for combining those two research methodological approaches
were ‘enhancement’ (augmenting of either quantitative or qualitative findings), sampling (to facilitate
the sampling of respondents or cases) and triangulation (cross-validation).
Against the background of Bryman’s observation, it is interesting to note that an analysis (see
appendix 1) for the present paper of the 78 papers presented at the European Conference on
Research Methodology (ECRM) in 2011 showed that 21% used a multi-methodology design, 41%
were qualitative only and 29% were quantitative only (see Ashwin, 2011, for the proceedings).
Clearly, while multi-methodology research seems to be on the ascent, qualitative studies still
predominate, at least at the ECRM in 2011. In this regard, one is reminded of Blumberg, Cooper and
Schindler’s comment, “Many scholars show a strong preference for either type of study. However,
these preferences more likely reflect their own capabilities and experiences than a general idea about
which type of research is more useful” (2008: 192). The present paper offers a three-dimensional
framework of research methodology based on complementarity rather than irreconcilability, which
could further facilitate the incorporation of combinations of qualitative and quantitative research
approaches in research and thus offer a more nuanced approximation of the truth. The paper,
however, does not deny the value of good mono-methodological research irrespective of the
methodological orientation or the important differences between the two approaches in, for example,
the researcher’s position in the research setting, theoretical orientation, research design, scope,
information-gathering techniques and others set out by, for instance, Denzin and Lincoln (2005: 11–
12).
There are many reasons why attention should be focused on complementarity between research
approaches. Three methodological reasons include, firstly, the convergent and discriminant validation
approach (e.g. Campbell and Fiske, 1958) and the parallel qualitative requirement of triangulation
(e.g. Denzin, u.d.; Denzin and Lincoln, 2005) which highlight the benefits to validation of incorporating
different methodological approaches into a study. A second justification for multiple methodologies is
the growing importance of mixed-method designs, “a third methodological movement…as a pragmatic
way of using the strengths of both movements” [i.e. quantitative and qualitative], as Tashakorri and
248

Hendrik Marais
Teddlie described mixed methods (2003: ix). A third challenge to research methodology – and project
management – emerges from the growing importance of multi-, inter- and especially transdisciplinary
research (cf. Hadorn et al., 2008), as researchers address increasingly complex scientific and socially
relevant problems, partly in response to the dynamics of science, technology and innovation policy,
including international scientific collaboration programmes, and governments’ expectations that
publicly funded research should address national and even international problems, i.e. return on
investment (cf. OECD, 2007; Rosenfield, 2009). Fourthly, the possibility of constraints on public
funding of universities in many countries, could stimulate multi- and interdisciplinary research as a
strategy to ameliorate the effects of cuts in research funds. This is even more so in less industrialised
countries, such as South Africa, where some academic departments and disciplines have to
amalgamate. And finally, “As such, MMRD [mixed methods] may dislodge ossified positions primarily
maintained by identity politics associated with QL [qualitative] and QN [quantitative] methods” to quote
Bergman (2011: 275).
2. Main argument
The paper argues that national and institutional science policies increasingly challenge researchers to
address complex socially relevant problems, and that such problems often require at least a
multidisciplinary research design. The paper further argues that although caution is generally required
in utilising different methodological approaches in the same study (see, e.g., Cameron, 2011, for
some of the challenges), this should not necessarily inhibit researchers from venturing into the
territory of multiple methodologies to find more comprehensive answers and solutions. It is further
suggested that representations of quantitative and qualitative methodological approaches are often
ideal types developed by philosophers of science and strong proponents of either side, but that they
seldom occur in pure form in everyday research. Finally, the paper argues that there are indeed
interfaces and sufficient communalities between accountable methodological approaches to justify
multiple approaches, provided that this is done within an explicit methodological framework
3. Objectives of the study
The main objective of the study was to develop a framework that could be used as an instrument to
identify the interfaces between different methodological approaches and thereby utilise a broader
spectrum of approaches than would otherwise be the case.
4. Information base of this paper
A brief note on the process that was followed in drafting this paper may be appropriate. The material,
analyses, interpretations, conceptions and results reported in this paper are the result of critical
reading of publications on research methodology, interactions with colleagues, active self-initiated and
commissioned research, teaching of research methodology courses, research supervision, and
research evaluation ranging from research proposals to completed research reports. In one sense,
this paper represents a qualitative insider perspective on the subject, but in another sense it reflects a
modest attempt at integrating in a fair way a wide spectrum of published knowledge on the process of
research.
5. Conceptual framework
At the risk of offering redundant information, definitions of key concepts as used in this paper are
provided:
Research: The uncovering of the truth about a phenomenon (and/or its relationship with other
phenomena) by means of scientific methods. The term ‘research’ as used in this paper equates to
scientific research and includes qualitative research.
Science: The accumulated published information generated by means of scientific methods, on
the validity of which the peers in the particular field have reached consensus; science and
scientific research include all science cultures.
Social science: The subset of science that focuses on human, social and cultural behaviour,
including business and management studies
Qualitative research: Research that approaches phenomena from the insider’s or subject’s
perspective to understand it in its natural context; this approach uses qualitative indicators such
as words, stories, pictures and other communicative representations as non-numerical symbolic
249

Hendrik Marais
information on phenomena; its methodologies are normally less formalised, rigid, specific and
explicated, but more comprehensively recorded.
Quantitative research: Research that approaches phenomena from an outsider’s perspective to
explain and predict the phenomenon under study in isolation; this approach uses numerical
indicators of abstract concepts; its methodology is normally relatively formalised, rigid, crossreferenced
and explicated, but more parsimoniously recorded by means of statistics.
6. Root dimensions of research methodology
To make comparisons between different methodological approaches first requires the identification of
a limited set of necessary and sufficient properties, attributes or characteristics of the scientific
research process that could serve as bases of comparison. From phenomenological and systems
perspectives, and based on analyses of textbooks on research methodology, research publications
and relatively wide research experience, a limited number of root dimensions that qualify research as
‘scientific’ were identified. These were further developed into a framework that would not define
scientific research as a discreet point in the knowledge space – implying an all-or-nothing character –
but would acknowledge that research can take a range of legitimate forms, in other words suggesting
the existence and promotion of the utilisation of multiple approaches to generating scientific
knowledge. A careful reading of methodological publications yields further conditions, but they can
mostly be subsumed under one of the three root dimensions or are, upon closer inspection, particular
to a very special research design. The root dimensions are summarised below, and a conceptual
framework, Research Methodology in 3D, is offered in a later section.
The three root dimensions emerged from the conceptual analyses:
Constituent components of the scientific research process: concepts/constructs,
hypotheses/theses, observation and communication of the research
Epistemic criteria that scientific research should comply with: reliability/dependability,
validity/credibility, objectivity/confirmability and replicability
The primary purposes for which research is normally undertaken: description of a phenomenon
and its relationship with other phenomena, explanation of causal relationships between
phenomena, and in-depth understanding of phenomena.
6.1 Root dimension 1: Constituent components of the scientific research process
Most, if not all, human behaviour, ranging from everyday behaviour, through the arts, to scientific
research, consists of at least the following four generic components (cf. e.g. Marx, 1963; Mouton,
1996):
Linguistic and para-linguistic representations of phenomena (words, concepts, constructs,
symbols of all types): Differences in the nature of concepts as used by qualitative and quantitative
researchers have been debated by prominent scholars (see Mouton, 1996, for a useful summary),
but it is important to note the obvious: concepts represent a necessary component in both
qualitative and quantitative research traditions, although the labels and exact criteria may differ.
Concepts should meet the criterion of explication and operational specificity.
Questions about phenomena (conjectures, theses, hypotheses): Hypotheses (and their variants)
serve to direct research and are key components of both qualitative and quantitative research. A
hypothesis or thesis should be testable, and more specifically ‘rejectable’, to be useful.
Observation of phenomena (sensory perception, experience, measurement): Observation can
range from participant observation to relatively sophisticated measurement of behavioural or even
psycho-physiological responses. The criterion of control equates to ‘objectivity’, meaning that the
researcher shows her/his autonomy in the observation process and is not merely ventriloquising.
Communication of what has been observed (general discussions, news reports, scholarly
publications): This component is seldom included as a basic component of research, but since the
earlier definition of science refers to ‘published’ information on which ‘peers have reached
consensus’, communication is elevated as a key component of scientific research process. This
position is strengthened by the common observation that whether the research qualifies as
‘scientific’, as well as the quality of the research study, are taken to be reflected by the journal –
more specifically, the impact factor of the journal – in which the work was published or the
conference at which it was presented!
250

Hendrik Marais
The first three constituent components were originally summarised in this form by Marx as being
‘basic elements of theory construction’ (1963: 10). Specific criteria applicable to each of the
components differentiate between forms of behaviour (e.g. distinguishing between a short story, an
investigative journalism article and a research report). Each of the four components represents a
continuum, and different research methodological approaches need not be, and are not, located at
the same point on each of the continua. The four components are shown in Figure 1 as continua, with
the scientific criteria on the far right of each and the arts on the extreme left of each continuum (cf.
Marx, 1963: 11; Mouton and Marais, 1988: 157).
Figure 1: Constituent components and criteria of scientific research (after Marx, 1963: 11)
The following brief notes on aspects of Figure 1 may be important in the context of this paper:
The four constituent components are, of course, not independent of each other, and can perhaps
be described as the results of a conceptual unfolding analysis (to use a concept coined by
Coombs) of the core components of human behaviour in general and research in particular.
The representation in Figure 1 assumes that scientific research is a special form of human
behaviour, the main difference between it and other forms of human behaviour (e.g. everyday
conversations, different forms of art, and investigative journalism) being the specific content given
to the italicised criteria in that figure. Also, the representation offered in Figure 1 allows for a
range of social research methodological approaches – in general, qualitative research such as
phenomenological studies would tend to be located to the left of experimental psychological
studies, which would normally be closer to the far right of the continuum.
Each of the components is expressed as a continuum – rather than an absolute point in some
idealistic semantic space – with the obvious implication that different research methodological
approaches could be located along the various continua. In Figure 1, physics is theoretically
positioned at the far right as the ‘ideal’, but this does not preclude, for example, history
somewhere to the left and closer to the middle of the continuum.
The key criterion for scientific research is italicised. Implicitly or explicitly, scientific research
strives to be as specific as possible with regard to its concepts, to deal with testable research
questions/hypotheses, not to engage in random observation or measurement, and to accept that
which has been passed as credible by peers in a particular research field.
Generally speaking, and referring to good research, experimental research (especially laboratory
research), would be closer to the right extreme of the four components, while qualitative research
would tend to be located somewhere to the right of the midpoint of each of the continua.
Scientific research takes place when these four components, each complying with the scientific
criterion, interact in a systematic way.
6.2 Root dimension 2: Epistemic criteria that scientific research should comply with
The second root dimension of the framework is represented by the core set of criteria that all
research, whether quantitative or qualitative, must comply with in order to qualify as scientific, as
defined in section 4. Figure 1 reflects a process criterion for each of the four components of scientific
research, namely: concepts should be operationalised as specifically as possible, preferably in terms
of empirical referents; hypotheses should be formally testable; observation should be as controlled as
possible; and, finally the communication of the findings should be subjected to peer review. These
process criteria, however, do not specify the actual quality required of scientific research as the
251

Hendrik Marais
search for truth. For this, one should drill deeper into the nature of the scientific research process, as
reflected in methodology textbooks and courses, as well as criteria applied to the assessment of
dissertations/theses, journal articles, and conference papers. As supplement to own conceptions and
research practices, selected textbooks on research methodology - quantitative and qualitative
research - were scanned to identify the primary criteria for scientific research (e.g. Alasuutari,
Bickman, and Branner, 2008; Babbie and Mouton, 2001; Blumberg, Cooper and Schindler, 2008;
Mouton, 1996; Mouton and Marais, 1992; Remenyi, Williams, Money and Swartz, 1998; see Marais,
Pienaar-Marais and Gathua, 2011, for further sources consulted). These and other textbooks
generally agree on two important issues. Firstly, both quantitative and qualitative methodologies
subscribe to the same core set of epistemic criteria and, secondly, different terms are used by
proponents of those methodologies to label the criteria (see Table 1). The criteria are, however, in the
words of Babbie and Mouton, “all bootstrap conceptions of sort. Although we should strive with
everything in our power to do truly valid, reliable, and objective studies, the reality is that we are never
able to attain this completely” (2001: 276), but, one might add, should pursue this in all research
undertaken and explicitly report on it in all presentations. Table 1 offers an overview of the key
criteria, using ‘bilingual’ nomenclature for the criteria, definitions and indicators, as normally used by
quantitative and qualitative researchers respectively.
Table 1: Key epistemic criteria
Quantitative research Qualitative research
Criterion Meaning and procedure Criterion Meaning and procedure
Reliability
Same findings upon replication?
Test-retest & interrater reliability
Dependability;
trustworthiness;
consistency
Similar context yields similar
findings? Inquiry audit
Compatibility between respon-
Internal
validity
Measured what intention was?
Experimental control; statistical
triangulation
Credibility
dents’ and reported perceptions?
Prolonged engagement; member
checks; quality record; narrative
triangulation
External
validity
Generalisability to population?
Random sampling
Transferability
Applicable to other cases and
contexts? Purposive sampling;
detailed descriptions of process
Findings not function of biases of
‘Objectivity’
Reflecting own views?
Control over subjective factors
Confirmability
researcher? Audit trail; trust &
rapport with subject; intersubjectivity
Replicability
Can next researcher replicate the
study? Peer reviewed publication
Replicability
Clear description of procedures?
Appropriate peer-reviewed
publication
Table 1 highlights, among other things, the following four points:
In the first place, the epistemic criteria, normally associated with quantitative research, have their
equivalents in qualitative research (also see Golafshani’s, 2003, overview. It would be reasonable to
define each of the criterion continua as unidimensional. In other words, each of the four epistemic
criteria can be used to assess the quality of research irrespective of whether it be quantitative or
qualitative. In the second place, notwithstanding the ‘universality’ of the epistemic criteria, it should be
noted that the ways of determining whether they have been met, differ between qualitative and
quantitative research, and are expressed differently (hence the reference to ‘bilingual nomenclature’
earlier), but that should not detract from the equivalence between the two sets. There is a greater
degree of standardisation of the criteria in the quantitative than the qualitative approaches. In the third
place, meeting the criteria – in each methodological approach – sets the limits of the extent to which
the studies involved qualify as scientific, and it therefore becomes an imperative in each study to
report the outcomes of testing the reliability/dependability, internal validity/credibility, external
validity/transferability and ‘objectivity’/confirmability of the study. (Even some quantitative survey
studies omit to report the reliability and internal validity of the survey questionnaire used in the
particular study, which implies conclusions like ‘We are 95% certain that we can generalise our
findings to the relevant population – unfortunately we are not sure what we are generalising!’) In the
fourth place, the fact that all the cells in Table 1 could be populated supports the argument offered
earlier in this paper, namely that there is sufficient complementarity between quantitative and
qualitative approaches to justify the argument that the utilisation of methodological approaches from
both could contribute to a more nuanced approximation of the truth.
252

Hendrik Marais
6.3 Root dimension 3: Primary purposes for which research is normally undertaken
Even a superficial reading of reports on research projects shows that a recurring characteristic that
differentiates research projects is the goal of the project, or the so-called ‘teleological function’ of
research. Different lead researchers and senior academics classify research designs in a wide variety
of ways (e.g. Blumberg, Cooper and Schindler, 2008: 196). These classifications are generally useful,
but use different facets of research as bases, such as research design, research setting, and
information-collection techniques, to mention three. The Dutch social-psychologist, Van Leent, as far
back as 1965, offered a useful and, it is suggested, still relevant classification of the purpose of
research that captures the teleological dimension, while allowing for linkages between quantitative
and qualitative research. (Perhaps, this linking of quantitative and qualitative research should come
as no surprise, since Van Leent wrote during the period of the flowering of phenomenology in
Northern Europe!).
Van Leent proposed a three-dimensional typology of the social sciences in general, and social
psychology in particular. More specifically, he argued that the teleological function was a meaningful
differentiator between various research traditions pursued in the social psychology of the day, and he
identified the following basic goals of research:
6.3.1 Descriptive research
Many social scientific projects, including business and management studies, have the primary goal of
describing a phenomenon (e.g. how many instances of the phenomenon exist in a population as
found in a survey, such as marketing research). Normally, inductive theories (e.g. typologies) emerge
from such research.
6.3.2 Explanatory research
A relatively small percentage of social scientific research tries to explain the exact relationship
between phenomena (i.e. determine cause–effect relationships), and an even smaller portion of those
uses one of the traditional experimental designs to show the effect of the independent variable or
cause on the dependent variable (Campbell and Stanley, 1963). In quantitative research, hypotheticodeductive
theories normally follow from this type of research design.
6.3.3 Understanding research
Describing the incidence or properties of a phenomenon (e.g. how many jobs have to be cut for a
business to survive) and/or to explaining the causal relationships among phenomena (e.g. the
reasons for the need to reduce the workforce significantly) – important as they may be – does not tell
us what it means to a loyal and hardworking employee to learn that one has been declared
redundant. Qualitative research approaches, such as phenomenological, ethnographic and related
designs and methods, can often contribute to in-depth understanding of the experience, retrenchment
in this case. The fact that descriptors of qualitative research (such as insight and understanding from
the perspective and experience of the subject) bring this goal of research quite well to the fore.
6.3.4 Summary of teleological dimension
Normally, but not always, quantitative research approaches take the form of descriptive or explanatory
studies, while qualitative studies are more focused on the in-depth understanding of one or more
phenomena. Van Leent’s original typology of social psychology – and, as this paper implies, the larger
part of research methodology in the social sciences in general – can be represented by the three axes
of a cube (Figure 2).
253

Hendrik Marais
Figure 2: Three fundamental goals of scientific research (after Van Leent, 1965)
7. The framework: Research methodology in 3D
A three-dimensional framework emerges when the three root dimensions are joined up, as shown in
Figure 3.
Figure 3: Research Methodology in 3D
Figure 3 reflects the following salient aspects:
The framework can be considered as a decision-making tool.
Each of the dimensions summarises the building blocks that constitute scientific research and
simultaneously shows the equivalencies between qualitative and quantitative research.
Each of the dimensions can be – and normally is – used individually as specifications in planning,
undertaking and/or assessing a project of limited scope.
However, the framework offers a ‘joined-up’ space that facilitates the planning of complex multi-,
inter- and transdisciplinary research projects, especially if the boundaries between qualitative and
quantitative approaches have to be crossed in the same venture. The framework highlights the
elements that initially have to be identified and the criteria that have to be specified. Experience
shows that the routes an interdisiplinary academic research project and a commissioned
transdisiplinary one would follow, would not be exactly the same, but both would have to account
for each of the checkpoints in the cube, i.e. components and process criteria, epistemic criteria
and research goal.
Planning of such complex research projects requires the plotting of each of the research approaches
to be utilised (quantitative and qualitative) on each of the three root dimensions to determine their
compliance with the three sets of scientific requirements (components, criteria and research goals). A
useful tool for this purpose would be a radar or spider chart, being a two dimensional representation
of multivariate information (e.g. Wikipedia, 2012).
254

Hendrik Marais
Conversely, the framework could be useful in assessing all the elements included in the project
design of a comprehensive multi-methodological research project.
Finally, the framework could be used as a set of planning specifications irrespective of whether
the researcher’s orientation is qualitative or quantitative.
8. Potential contribution of the framework
The paper shows that the root dimensions constituting the framework comply with expositions of
research methodology found in most standard textbooks on this vast area of study, but go beyond
some by, firstly, explicitly relating the three root dimensions and their subsets to form an interlinked
framework. Secondly, the paper assumes and demonstrates salient tangent planes between
approaches to scientific research that have traditionally been assumed to be mutually exclusive,
namely quantitative and qualitative research. Thirdly, the framework perhaps goes further than many
traditional presentations by demonstrating the possible complementarity between research
approaches. Finally, the framework further addresses latent imbalances that emerged from the
modest survey of the papers presented at the ECRM 2011 Caen conference (see endnote).
The framework, as it stands, has its limitations. In the first place, the paper certainly has a quantitative
methodological bias. Secondly, it has been developed and utilised in a behavioural and social
sciences context only and has not been subjected to a thorough cross-disciplinary peer review
process. Thirdly, an inherent limitation of the framework is that it is selective to the extent that it
incorporates what its developers regarded as the most important and salient elements. Other
scholars, depending on their own methodological preferences, might argue that other elements
should have been incorporated in the framework. Fourthly, the relative complexity of the Research
Methodology in 3D framework as represented in Figure 3 should be noted, but developing radar or
spider charts would largely compensate for this factor. These and other limitations are receiving
further attention, e.g. experiments with radar charts.
9. Functions of the framework
The root dimensions of the framework have served as a framework for the author as an active
researcher and teacher for many years, but in a linear rather than integrated mode. This paper is the
first attempt at linking the three so-called root dimensions to form an integrated three-dimensional
framework. The root dimensions, individually and in combinations, have proved to be facilitating
mechanisms in teaching, project planning and evaluation.
9.1 Teaching of research methodology
The root dimensions and framework have proved useful as a map to assist teachers in systematising
the logic of the research process at public universities (with relatively inexperienced students) and
private universities (where students tend to come from professional occupations) and assisting
students in charting their way through what some experience as a maze of new concepts and
seemingly contradictory criteria.
9.2 Project and research programme planning
The framework has proved useful as a planning instrument for complex research programmes – e.g. a
programme matrix of 40 individual projects from which overarching integrated conclusions and
recommendations were required (see HSRC, 1987: 3–8). Clearly, the root dimensions, individually
and in combination, can serve equally useful project planning functions when critical questions on the
scientific constituent components, scientific criteria and purpose of the project have to be considered.
9.3 Evaluation of research proposals and reports
The combination of root dimensions, as reflected in the Research Methodology in 3D framework, can
serve as a useful assessment guide in evaluating research proposals and reports. In this regard, the
framework represents a logical set of evaluative dimensions that can be checked, irrespective of the
specific methodological approach of the researcher. In practice, this means that each of the first two
root dimensions should be transformed to a set of four summative rating scales, while the third root
dimension would normally consist of two subscales measuring the appropriateness and the
correctness of application of the chosen research approach, producing 10 three or four-point rating
scales in total. (A paper elaborating on this function is in preparation.)
255

9.4 Other functions of the framework
Hendrik Marais
A number of further functions can be listed, such as serving as a common framework for debates
between proponents of different methodological orientations within the social sciences and between
the social and natural sciences as well a framework for the identification of apparent communalities
and differences between, for instance, scientific research and the creative arts.
10. Conclusions
None of the individual bits of information offered in this paper should come as new knowledge to the
experienced researcher and perceptive teacher of research methodology. The paper is an integration
of existing knowledge, as the dates of some of the references should show! The paper may open
three perspectives for some readers – in varying degrees for advanced research students, active
researchers and teachers of research methodology. Firstly, broad agreement on the definition of
scientific research and its essential properties could take the sting out of the main, and often
unproductive, debate between proponents of quantitative and qualitative approaches to research.
Secondly, most social scientific research (including business and management fields) is multi-faceted
and requires multi-method designs, which would include quantitative and qualitative approaches. This
would apply even more strongly where there might be practical, economic, human and related
consequences related to the eventual implementation of the research findings. Thirdly, the emerging
framework, Research Methodology in 3D, brings to the fore the interaction effect between the three
seemingly independent root dimensions of research, thereby offering a conceptual instrument to
relate quantitative and qualitative approaches to research.
The following conclusions seem justified: Firstly, typologies of research approaches, e.g. qualitative
and quantitative research, are often the results of conceptual exercises and not of empirical analyses
of real published research reports and consequently do not necessarily represent research-aspracticed.
Secondly, the framework offers a conceptual tool that could be used in finding interfaces
between methodological approaches, whilst at the same time highlighting the unique contribution of
each. Finally, perhaps the time has come to seek complementarity between approaches rather than
spend lots of time trying to defend one’s own approach and losing sight of the potential contribution by
other approaches – even within the own paradigm.
Acknowledgements
Financial and other support of the University of Pretoria’ Graduate School of Technology
Management is gratefully acknowledged. Tinus Pretorius (UP) and Magdal Pienaar-Marais (NRF) are
thanked for their constructive comments on a draft of the paper and the copy-editing done by Robyn
Arnold deserves a special word of appreciation.
Appendix 1
The results of a content analysis of all 78 Caen 2011 papers are summarised in Table 2.
Table 2: Distribution of papers presented at ECRM 10
Category
Research papers
Qual (%) Quan (%) Multi (%) Qual (%)
Conceptual papers
Quan (%) Multi (%) Plan (%)
Total
Paper 19 (29) 19 (29) 4 (6) 8 (12) 0 10 (15) 6 (9) 66
PhD 8 1 0 2 0 0 1 12
Total 27 20 4 8 0 10 7 78
Notes: 1. Research papers: Reports on completed research projects. 2. Conceptual papers: Those
that considered theoretical and methodological underpinnings of research or set out project plans
(Plan) and did not report a research project. 3. Qual: Papers that dealt only with qualitative research;
Quan: Papers that dealt only with quantitative research; Multi: Papers that covered both qualitative
and quantitative research
References
Alasuutari, P. Bickman, L. and Branner, J. (2008) Handbook of social research methods. Sage, Thousand Oaks.
Ashwin, M. (ed.) (2011) Proceedings of the 10 th European Conference on Research Methodology for Business
and Management Studies, Normandy Business School, Caen.
Babbie, E. and Mouton, J.M. (2001) The Practice of Social Research, Oxford University Press, Cape Town.
256

Hendrik Marais
Bergman, M.M. (2011) “The Good, the Bad, and the Ugly in Mixed Methods Research and Design”, Journal of
Mixed Methods Research, Vol. 5, No. 4, pp 271–275.
Blumberg, B., Cooper, D.R. and Schindler, P.S. (2008) Business Research Methods, McGraw-Hill, Berkshire.
Bryman, A. (2006) “Integrating quantitative and qualitative research: How is it done”, Qualitative Research, Vol. 6,
No. 1, pp 97–114.
Cameron, R. (2011) “The five Ps of mixed methods research”, In M. Ashwin (ed.), Proceedings of the 10th
European Conference on Research Methodology for Business and Management Studies, Normandy
Business School, Caen, pp 76–83.
Campbell, D.T. and Fiske, D.W. (1958) “Convergent and discriminant validity by the multitrait-multimethod
matrix”, Psychological Bulletin, No. 56, pp 81–105.
Campbell, D.T and Stanley, J.C. (1963) Experimental and Quasi-experimental Designs for Research, Rand-
McNally, Chicago.
Denzin, N.K. (u.d.) What is triangulation? http://jthomasniu.org/class/Handouts/triang-denzin (retrieved
2012.02.26)
Denzin, N.K. and Lincoln, Y.S. (2005) Handbook of Qualitative Research, Sage, Thousand Oaks.
Hadorn, G.H., Hoffmann-Riem, H., Biber-Klemm, H.S., Grossenbacher-Mansuy, W., Joye, D., Pohl, C.,
Wiesmann, U. and Zemp, E. (eds) (2008) Handbook of Transdisciplinary Research, Springer, Bern.
Golafshani, N (2003) Understanding reliability and validity in qualitative research. The Qualitative Report 8(4):
597-607 (http://www.nova.edu/ssss/QR/QR8-4/golafshani.pdf; retrieved 2012.02.26)
HSRC (Human Sciences Research Council) (1987) The South African Society: Realities And Future Prospects,
Greenwood Press, New York.
Marais, H.C., Pienaar-Marais, M. and Gathua, S. (2011) On the Promotion of the Utilisation of Research Findings
by Graduate Students in Management Studies: A South African Perspective. Paper presented at the 11 th
European Conference on Research Methodology, Caen, 20-21 June.
Marx, M.H. (ed.) (1963) Theories in Contemporary Psychology, Collier-MacMillan, London.
Mouton, J. (1996) Understanding Social Research, Van Schaik, Pretoria.
Mouton, J. and Marais, H.C. (1988) Basic Concepts in the Methodology of the Social Sciences, HSRC, Pretoria.
OECD (2007) OECD reviews of innovation policy: South Africa. Paris: OECD.
Remenyi, D., Williams, B., Money, A. and Swartz, E. (1998) Doing research in business and management: An
introduction to process and method. Thousand Oaks, Sage.
Rosenfield, P.L. (2009) “The Potential of Transdisciplinary Research for Sustaining and Extending Linkages
Between the Health and Social Sciences”, Social Science and Medicine, Vol. 35, No. 11, pp 1343–1357.
Tashakorri, A. and Teddlie, C. (eds.) (2003) Handbook of Mixed Methods in Social and Behavioral Research,
Sage, Thousand Oaks.
Van Leent, J.A.A. (1965) Sociale Psychologie in Drie Dimensies (Social Psychology in Three Dimensions), Aula,
Utrecht.
Wikipedia (2012) Radar chart. http://en.wikipedia.org/wiki/radar_chart (accessed 2012.02.28)
257

A Model Development Strategy to Determine Factors That
Influence Knowledge Retention in Organisations
Ellen Caroline Martins¹ and Nico Martins²
¹Organisational Diagnostics, Johannesburg, South Africa
²Department of Industrial and Organisational Psychology, University of South
Africa, Pretoria, South Africa
orgdia@iafrica.com
martin@unisa.ac.za
Abstract: The wave of knowledge loss and attrition that organisations face in a world of layoffs, retirements,
staff turnover and mergers gave rise to this study. The purpose of the paper is to describe the research
methodology that was followed, which focuses on theoretical and empirical research, including multiple
multivariate analysis techniques to develop a model that would explain the factors that could enhance or impede
tacit knowledge retention. An understanding of the research methodology can assist other researchers in utilising
this process for future model development and it could assist organisations in coping with a world of losing
knowledge by retaining critical tacit knowledge. The preliminary literature review on knowledge retention revealed
the theoretical approach to be followed, namely a multi-disciplinary approach focusing on knowledge
management, organisational behaviour and organisational development. A theoretical model was developed that
identifies the organisational and behavioural factors to be considered when addressing the issue of knowledge
loss. This theoretical model focused on human input factors considering knowledge loss risks and behavioural
threats that could cause knowledge loss and strategy implementation risks. A quantitative empirical research
paradigm using the survey method was followed to determine the extent to which these factors impact knowledge
retention. The survey was conducted electronically and on paper in the water supply industry. The exploratory
factor analysis technique was used to explore the factor structure underlying the variables. The internal reliability
of the variables in the newly proposed factor model was determined. Nine key factors were identified through
principle component factor analysis of which knowledge behaviour, leadership and strategy implementation had
the highest factor loadings. The next step was to follow a model development strategy using the structural
equation modelling (SEM) technique to find the best fitting model. After rigorous model testing three final models
were compared. The regression model that forms part of the SEM process confirmed that there are relationships
between most dimensions, which are in line with the theory. A further multiple regression analysis was performed
to determine the degree to which different dimensions predict knowledge retention. In summary it can be
concluded that the research methodology chosen for this research study, underpinned by SEM, can be utilised to
firstly confirm the influencing factors of knowledge retention. Secondly SEM enables the development of a
theoretically justified new model to determine the relationship between strategy implementation, knowledge
behaviour and knowledge retention.
Keywords: principal component factor analysis, structural equation modelling, model development strategy, tacit
knowledge loss, knowledge retention, knowledge behaviour
1. Introduction
Knowledge loss has become a critical factor that could make organisations vulnerable in difficult
economic times as well as during thriving economic growth periods when competition is rife. All
organisations face the risk of losing knowledge, which could affect their sustained competitive
advantage (Martins & Martins, 2011). There is growing concern in the business and organisational
sector that organisational knowledge can be lost through the exit of employees due to changing
workforce demographics such as an aging workforce, more competitive recruiting and faster turnover
in younger people (Nicholson, 2008). These factors gave rise to this research study. The research
was aimed at developing a structural equation model of knowledge retention that would explain the
factors that could enhance or impede tacit knowledge retention. In this paper the research
methodology that was followed is described, supported by a brief discussion of the research
outcomes.
A preliminary literature review revealed that little research has been conducted in the field of
knowledge retention with a behavioural focus. However, a vast amount of literature was found on
knowledge, knowledge management, organisational challenges and solution-driven knowledge
retention initiatives, and the organisational behaviour discipline as such, thus facilitating the
application of the relevant concepts to knowledge retention from an organisational and behavioural
perspective.
258

2. Research problem statement
Ellen Caroline Martins and Nico Martins
Many organisations seem to be in denial about knowledge loss, and the leaders in these
organisations do not acknowledge the threat and impact of lost knowledge (Foster, 2005:28). Many
management teams find that attracting, developing and retaining a knowledgeable work force is a
major issue, but many are unaware of the scope of the problem or potential solutions, such as
retaining knowledge through knowledge management solutions (American Productivity and Quality
Centre, 2002:6). If organisations do not seriously address the issue of knowledge loss and attrition by
implementing a knowledge retention strategy, this could have detrimental effects on their business
success and survival.
In order to determine the focus of a knowledge retention strategy, it is necessary to understand the
organisational factors that would enhance or impede knowledge retention. Knowledge is gained by
different manifestations of behaviour such as learning, sharing and transferring the acquired
knowledge to human beings. Certain enhancing or impeding organisational and behavioural
conditions that prevail in organisations may cause this knowledge to either be lost or retained.
The problem investigated in the current research can be formulated as follows:
What behavioural and organisational factors could an organisation consider to combat the increasing
knowledge loss and attrition that are affecting organisations?
3. Theoretical perspective
The research problem to be investigated seems to relate to different fields of study. Considering the
context of the problem investigated in this research, namely the organisational and behavioural
factors that could influence knowledge retention, the theoretical scope of this research could be
broadened to a multidisciplinary approach relating to the disciplines of knowledge management,
organisational behaviour and organisational development.
The concepts of knowledge loss and knowledge retention fit into the discipline of knowledge
management. Although there has been immense interest in knowledge management, the field is
complex and there does not seem to be any consensus on further development. Scholl and Heisig
(2003) conducted a first global Delphi study to assess the art of knowledge from a theoretical and
practical point of view and to study the future of knowledge management. The outcome of this study
revealed that the emphasis is on human factors and a shift from an information technology
perspective to a behavioural science perspective.
Organisational behaviour as an applied behavioural science builds on a number of behavioural
disciplines such as psychology, sociology and anthropology. It is concerned with what people do and
how their behaviour affects the organisation (eg employment turnover, productivity, human
performance, work and management) (Robbins, 2005).
The purpose of involving organisational development is the assumption that knowledge retention
needs to be managed as an organisational improvement strategy. French and Bell (1999) describe
organisational development as the applied behaviour science discipline dedicated to improving
organisations and the people in them by making use of the theory and practice of planned change.
4. Literature review on knowledge retention
The intention was to accurately describe the phenomena through the literature study. A generic tool
for academic work that contextualises arguments, namely contextualised theory building, was found in
the literature and deemed to be a useful tool to guide the discussion on the contextualisation of
knowledge. Contextualised theory building is a research methodology that focuses on the close link
between the question of why the issue of knowledge is important, epistemological assumptions of
knowledge, knowledge appearances and knowledge applications. The term “epistemology” refers to
the investigation of fundamental assumptions of knowledge (Venzin, Von Krogh & Roos, 1998) or the
status and nature of knowledge – how we know the world (Dick & Ellis, 2006) or the philosophical
enquiry of knowledge (Nonaka & Takeuchi, 1995).
A thorough study was conducted on the epistemological theories and models of knowledge to obtain
clarity on the many different epistemological conceptualisations found in the literature. Sixteen
259

Ellen Caroline Martins and Nico Martins
different theories and models (by no means a complete number) of knowledge in the knowledge
management, organisational behaviour and organisational development fields were studied. The
theories and models seemed to be context specific (Martins & Martins, 2011).
It appeared that in the current research, the “bridging of epistemologies as possession of knowledge
and knowing as action”, developed by Cook and Brown (2002), was a suitable approach to follow.
Key factors in this research are tacit knowledge possessed by individuals and groups and
organisational knowledge as well as knowing (in other words, how people do their knowing – implying
human action and interaction with the world). The context of the current research was from a
knowledge loss and knowledge retention perspective.
The appearance (manifestation) of knowledge relates to the different forms that knowledge can
assume (categories and types of knowledge) and identifying hidden appearances of knowledge
because it is wrapped up in related constructs (eg competence, invisible assets, intangible assets,
strategic assets, organisational memory, capabilities and skills) (Venzin et al, 1998).
In terms of the application of knowledge (exploring how the concept of knowledge is applied in this
research), the constructs “knowledge” and “knowledge retention” were described from a knowledge
management perspective and an organisational behaviour perspective drawing from literature in
these two study fields as described above.
“Knowledge” is defined as the knowledge (expertise) that exists in the heads of people, their work
experience and applying their knowledge in the work situation, which if lost to the organisation, could
be detrimental to the functioning and competitive advantage of the organisation.
“Tacit knowing” was described as the knowledge that resides in people’s minds and their experience
that is not easily documented. It relates to expertise and skills that were developed over time and
manifests in the behaviour of individuals in their jobs, working in teams, interacting with external
stakeholders, and so on.
“Knowledge retention” is action oriented and is defined as maintaining, not losing, knowledge that
exists in the minds of people (not easily documented) and knowing (experiential action manifesting in
behaviour) that is vital to the organisation’s overall functioning (Martins & Meyer, 2012).
The organisational and behavioural factors that inhibit or promote tacit knowledge retention were
derived from the literature study. The organisational factors relate to whose knowledge and what type
of knowledge may be lost to the organisation that could undermine the organisational strategy
(Delong, 2004). Knowledge behaviour that could cause loss of knowledge in people’s minds (ie tacit
knowing) refers to the behaviours of learning, knowing, creating, sharing, transferring and applying
knowledge. The organisational behaviour model adapted from Robbins (2005) was used to provide a
framework for identifying the behavioural factors that would influence knowledge retention.
Knowledge retention was added as a human output factor that would be influenced by the
organisational behaviour factors at organisational, group and individual levels.
5. Theoretical model of tacit knowledge retention
A theoretical model based on the literature study of the organisational and behavioural factors was
developed. A model can be described as a simplified, partial representation of the dynamics of a
phenomenon by displaying the main elements in a process. The most obvious aspects of the model
are emphasised (Mouton & Marais, 1990; Robbins, 1998).
The theoretical model developed in this research identifies the factors that need to be considered in
addressing the issue of knowledge loss. A condensed version of the theoretical model is displayed in
figure 1.
The two main focus points of the model are the external forces of change and the human input
factors. The external forces refer to factors such as the nature of the workforce, economic shocks,
competition and a world recession that could influence knowledge retention in organisations. The
human input factors refer to the carriers of knowledge pertaining to identifying the tacit knowledge
loss risks of whose knowledge and what type of knowledge needs to be retained. The knowledge
behaviour needs to be demonstrated to contribute to knowledge retention. The behavioural threats
260

Ellen Caroline Martins and Nico Martins
manifesting from demonstrating the knowledge behaviour could cause knowledge loss, whereas
behavioural enhancers could affect the retention of critical tacit knowledge. In turn, these behavioural
enhancers or threats could impact the manifestation of the knowledge behaviour. All these factors
could impact on the implementation of the organisation’s strategy. Identifying the strategic risks of
knowledge loss is therefore imperative. A holistic approach would imply that the information
technology infrastructure is also taken into consideration.
Source: Martins and Martins, 2011
Figure 1: Factors that influence knowledge retention
It can be concluded that identifying the risks and enhancing or impeding factors would indicate to the
organisation where to focus its efforts to retain knowledge and enable it to design and implement a
knowledge retention strategy that would ultimately contribute to knowledge retention.
6. Empirical research methodology
In this research, the purpose of the empirical research was to test and validate the model developed
in the literature study and develop a new model on the basis of the findings of the empirical study.
The empirical research design, research method and statistical analyses are discussed below.
6.1 Research design
The survey method was deemed to be the most appropriate empirical research method to obtain this
research aim. It is a cost-effective method compared with, say, conducting interviews and focus
groups, and was agreed to and accepted by the organisation in the water supply industry in which the
survey was to be conducted, in terms of feasible given time, resource and organisational constraints
(Brewerton & Millward, 2001). Furthermore, quantitative data could be used to conduct multivariate
statistics in an attempt to develop a new model based on the empirical results and compare it to the
theoretical model.
The empirical research method steps are consequently described in terms of questionnaire design,
sample design, survey administration and statistical analyses explaining briefly how the validity of the
research findings could be enhanced.
6.2 Questionnaire design
The questionnaire was designed in a paper and an electronic format using a Likert-type 5-point scale
with categories ranging from strongly disagree to strongly agree and some demographical questions.
Following the advice of Booysen (2003) the researcher went through a rigorous process of statement
formulation in several draft versions to finally construct the questionnaire that was pretested.
261

Ellen Caroline Martins and Nico Martins
A group of specialists in the organisational behaviour and knowledge management disciplinary fields
of the research pretested the questionnaire. The next phase of the pretest process was completed by
experts of the same type of population for which the questionnaire was intended.
6.3 Population and sampling
The survey was conducted in the water supply industry. The non-probability sampling method, namely
purposive sampling (Brewerton & Millward, 2001) was chosen to compile the sample. Purposive
sampling is appropriate when the researcher wishes to select unique cases that can provide special
information (Uys & Puttergill, 2003).
The population was limited to supervisory, management and senior levels of operational staff. The
reasoning here was that they would have a sound understanding of knowledge retention behaviours,
influencing factors and the strategic impact that knowledge loss could have on their organisation. The
size of the population based on these sample specifications was 1 070 in the participating
organisation.
6.4 Survey administration and response rates
The survey was conducted electronically to be completed by individuals who had access to computers
and on paper in facilitated group sessions. Survey completion was anonymous and completely
voluntary.
According to Church and Waclawski (1998), a response rate of somewhere between 30 to 85% can
be expected. Of the 1 070 possible observations, 455 observations were usable after the data was
cleaned, which is 42.5% of the population specified for the research.
Several guidelines were followed to determine whether the number of responses were sufficient to
ensure a high degree of accuracy when conducting multivariate statistics, such as factor analysis and
SEM. The number of respondents needed for statistical analysis complied with the following
guidelines:
A minimum of at least five times as many observations (based on 455 observations received and
88 statements in the questionnaire) required for factor analysis.
A minimum of at least 100 or more observations needed for multivariate statistics (Hair,
Anderson, Tatham & Black, 1995).
Neuman’s (2000) estimate of a 30% sample ratio of a small population of 1 000 (ie 300
respondents) required for a high degree of accuracy.
It can be concluded that the sample was sufficient to enable the researcher to achieve the research
aims.
6.5 Statistical analysis
Statistical analyses were conducted to address the research aim of determining statistically the
enhancing and impeding organisational factors that influence knowledge retention through exploratory
factor analysis, compiling a SEM model to validate the theoretical model and determine whether any
new constructs emerged.
6.5.1 Exploratory factor analysis
The exploratory factor analysis was conducted with SPSS version 17.0. The first step in the process
was to extract the factors based on the data collected from the respondents for each item in the
questionnaire. The first specification of the principal component factor analysis (PCFA) produced a
reasonably acceptable factor model of eleven (11) factors. Two factors consisted of only one item
each and these were eliminated. The factor loadings were investigated, and re-specification of the
factor model was computed by returning to the extraction stage, extracting factors and reinterpreting.
Items that had scores lower than .400 were removed. In total, nine items were removed from the
knowledge retention questionnaire. The VARIMAX rotational method, which seems to be the
approach that provides a clearer separation of factors (Hair et al, 1995), was used to conduct the
factor rotation.
262

Ellen Caroline Martins and Nico Martins
Three different criteria were used to determine the number of factors to be extracted in the second
factor analysis, namely:
The scree test which showed a change in direction after the fourth factor and the point at which
the line seemed to straighten was after the ninth factor.
Latent root criterion (eigenvalues) and percentage of eigenvalues which revealed that although 14
components appeared to have an eigenvalue greater than 1.00, which is considered to be
significant, the extraction sum of squared values and the rotation sum of squared values indicated
that nine factors accounted for 64.61% of the total variance.
The factor structure did not produce a dependent variable to measure knowledge retention. Rowe
(2006) contends that most theories and models in applied psychosocial research are formulated in
terms of latent variables (or hypothetical constructs) that are not directly measurable or observable.
As a means of data reduction, it is acceptable to compute latent or composite variables, such as
knowledge retention, from several observed indicators (or response items), each requiring responses
in Likert-type ordered categories. The researcher combined the relevant items into a composite
variable by extracting from the existing questionnaire, the variables/items that would measure
knowledge retention. In the SEM process knowledge retention was used as a latent variable to
determine whether the model will fit, indicating that the variables chosen to represent knowledge
retention are valid indicators of knowledge retention.
The total communality obtained by adding the individual sums of squares for each of the ten factors is
51.062, which represents the total amount of variance extracted by the factor solution (Hair et al,
1995). This indicates that the factor solution accounts for at least one-half of the variance of all the
variables.
The interpretation of the refined second factor analysis produced the following factors:
Knowledge behaviour
Strategy implementation and values
Leadership
People knowledge loss risks
Knowledge attitudes and emotions
Power play
Knowledge growth and development
Performance management
Organisational support and encouragement (Martins & Meyer, 2012)
Knowledge retention
The Cronbach alpha was used to determine the internal reliability of items in each of the ten factors.
The overall Cronbach alpha coefficient obtained for the knowledge retention questionnaire was
.975803 for the total of 79 items. Since the total value was above .7, the instrument (scale) can be
deemed to be reliable (Pallant in Castro, 2008). The reliability coefficient of the factors appears to vary
between .954460 and .721514. The Cronbach alpha of knowledge retention was .859876.
It can be concluded that the internal consistency (reliability) of the overall knowledge retention
questionnaire and the factors are consistent in what they are intended to measure. If multiple
measurements are taken, the reliability measures will all be highly consistent in their values (Hair et
al, 1995). Rowe (2006) proposes that some form of confirmatory factor analysis (CFA) should be
applied to ensure that indicator variables associated with any one latent variable are valid indicators
of the trait. CFA is part of the SEM process.
6.5.2 SEM
SEM has been described as a collection of statistical techniques that allows examination of a set of
relationships between one or more independent variables, and one or more dependent variables,
either discrete or continuous in both independent and dependent cases (Tabachnick & Fidell, 1983).
Hair et al (1995) describe SEM as a “Multivariate technique combining aspects of multiple regression
263

Ellen Caroline Martins and Nico Martins
(examining dependence relationships) and factor analysis (…) to estimate a series of interrelated
dependence relationships simultaneously.”
SEM in the current research can be defined as described by Hair et al (1995) above. The purpose is
not hypothesis testing, but to confirm the exploratory factor structure and determine multiple
relationships between the constructs. Application of this technique enabled the researcher to produce
a new model based on the empirical research that was compared to the theoretical model.
Three distinct SEM strategies can be followed namely:
confirmatory modelling strategy (not the best method to prove the proposed model if it has an
acceptable fit, but only confirms that it is one of several possible acceptable models);
competing models strategy (as a means to evaluate the estimated model with alternative models
bringing the researcher closer to a test of competing “theories”); and
Model development strategy (combination of confirmatory and exploratory approach) (Garson,
2009; Hair et al, 1995).
The SEM model development analysis was undertaken using the AMOS statistical program (version
18.0) to complete the model development strategy by developing different models. Several different
models were tested using SEM procedures such as a multiple regression model, a model with
covariance and one without covariance. The next set of models was tested using the correlation
matrix. All models were found to be deficient. Alternative models were tested on the basis of the
theory and changes to the structural and/or measurement models suggested by the SEM modification
indices. Three models were selected to be compared with one another in order to select the best
fitting model. The model that produced the best goodness-of-fit indices tested knowledge behaviour’s
influence on strategy implementation and on knowledge retention (including relationships between
most exogenous variables). The fit indices are displayed below in table 1:
Table 1: Goodness-of-fit indices with coefficient values for models 1, 2 and 3
GOODNESS-OF-FIT
CRITERION
Model 1:
Knowledge
behaviour’s influence
on strategy
implementation
Model 2:
Knowledge
behaviour’s influence
on knowledge
retention
Model 3:
Knowledge
behaviour’s influence
on strategy
implementation and
knowledge retention
[including
relationships between
most
exogenous variables]
MEASURES OF ABSOLUTE FIT
Chisquare
(CMIN)
2
χ
P
Goodness-of-fit
(GFI)
INCREMENTAL FIT MEASURES
Normed fit
index
(NFI)
Incremental
fit index
(IFI)
Comparative
fit
index
(CFI)
887.064 .000 .710 .676 .684 .682
849.989 .000 .719 .690 .697 .696
155.805 .000 .937 .943 .948 .947
Note: Conventional cut-off: Good fit is indicated by GFI>= .90; NFI, IFI and CFI>= .90
(Garson, 2010)
264

Ellen Caroline Martins and Nico Martins
The best fitting model 3 indicated a direct causal relationship between strategy implementation and
knowledge retention and knowledge behaviour and knowledge retention, but knowledge behaviour
also has a causal relationship with strategy implementation. The model is displayed in figure 2.
Figure 2: Model of knowledge retention
The estimated results of prediction in this model, when interpreting the regression model with strategy
implementation as an endogenous (dependent variable), indicated that organisational support
(estimate of 1.09) has more of an impact on strategy implementation than people knowledge loss
risks (estimate of .46), and both combined explain 59.3% of the variance. Strategy implementation
(estimate of .63) explains more of knowledge retention than knowledge behaviour (estimate of .09),
explaining 47.8% of the variance, and both combined (strategy implementation and knowledge
behaviour) explain 75.2% of the variance. Knowledge attitudes and emotions (estimate of .94) explain
more of knowledge behaviour than power play (estimate of .43), leadership (estimate of .39),
knowledge growth and development (estimate of .15) and performance management (estimate of
.07). This implies that strategy implementation (influenced especially by organisational support and
encouragement) would have a stronger effect on knowledge retention than knowledge behaviour.
Knowledge attitudes and emotions would have an extremely strong effect on knowledge behaviour
(figure 2 and table 2 below).
Table 2: Squared multiple correlations
Knowledge behaviour
Estimate
.478
Strategy implementation .593
Knowledge retention .752
The regression model that forms part of the SEM process confirmed that there are causal
relationships between most dimensions, which are in line with the theory. Intercorrelations in SEM
indicated that they are all significant with p values below .05 at the .00 (two-tailed) level. This confirms
the theory that they are for the most part intercorrelated to a great extent.
265

Ellen Caroline Martins and Nico Martins
6.5.3 Multiple regression analysis
In order to obtain further confirmation of the conclusions drawn from SEM model 3, multiple
regression analysis was performed to determine the degree to which different dimensions predict
knowledge retention.
The multiple regression analysis offered significant support for most of the causal relationships,
particularly of both knowledge behaviour and strategy implementation on knowledge retention. The
intercorrelations between most of the dimensions were confirmed by both structural equation model 3
and the multiple regression analysis.
7. Comparison of theoretical model and SEM model
In comparing the third structural equation model and the theoretical model, it can be concluded that a
new dimension, organisational support and encouragement, emerged that has a direct causal
relationship with strategy implementation. Strategy implementation in both the theoretical and SEM
derived models had a direct causal relationship with knowledge retention. Knowledge behaviour
remained the same as the knowledge behaviour dimension in the theoretical model. Knowledge
behaviour had a direct causal relationship with strategy implementation and knowledge retention.
The individual, group and organisational levels disappeared in the SEM model. The behavioural
threats of the theoretical model were replaced with a new set of factors of which the leadership
dimension remained the same in both models. The new set of factors refers to knowledge attitudes
and emotions (with a strong influence on knowledge behaviour), power play, knowledge growth and
development and performance management.
8. Conclusions
The research methodology chosen for this research produced a sound theoretical foundation of the
factors that influence knowledge retention from a knowledge management, organisational behaviour
and organisational development perspective. In the empirical study sufficient data was collected to
conduct multivariate statistical analyses. The exploratory factor analysis produced a reliable factors
structure. The SEM model development strategy enabled the researcher to test and confirm the
influencing factors of knowledge retention and develop a new theoretically justified model to
determine the relationships between strategy implantation, knowledge behaviour and knowledge
retention.
It can be concluded that the SEM model produced a more streamlined factor structure that would be
easier to interpret than the theoretically derived model, which consists of a number of dimensions and
sub-dimensions. It would appear that if enhancing behavioural factors are in place, knowledge
behaviour could improve, which in turn would enhance knowledge retention and strategy
implementation.
This research methodology could enable other researchers to apply the same methods and
techniques to contribute to research and solving other organisational challenges.
References
American Productivity and Quality Centre. (2002) “Retaining Valuable Knowledge: Proactive Strategies to Deal
With a Shifting Work Force”, [online]. Consortium Learning Forum Best-practices Report,
http://www.apqc.org/knowledge-base/documents/retaining-valuable-knowledge-proactive-strategies-dealshifting-work-force-
Babbie, E. (1998) The Practice of Social Research, 8th edition, Belmont, California, Wadsworth.
Booysen, S. (2003) “Designing a Questionnaire”, in Intellectual Tools: Skills for the Human Sciences, edited by D.
Rossouw. 2nd edition, Van Schaik, Pretoria, pp 127–142.
Brewerton, P. and Millward, L. (2001) Organizational Research Methods: a Guide for Students and Researchers,
Sage, London.
Castro, M.L. (2008) The Relationship Between Organisational Climate and Employee Satisfaction in a South
African Information and Technology Organisation, MA dissertation, University of South Africa, Pretoria.
Church, A.H. and Waclawski, J. (1998) Designing and Using Organizational Surveys: a Seven-Step Process,
Jossey-Bass, San Francisco, California.
Cook, S.D.N. and Brown. J.S. (2002) “Bridging Epistemologies: the Generative Data Between Organizational
Knowledge and Organizational Knowing”, in Managing Knowledge: an Essential Reader, edited by S. Little,
P. Quintas and T. Ray, Sage, London, pp 68–101.
266

Ellen Caroline Martins and Nico Martins
DeLong, DW. (2004) Lost Knowledge: Confronting the Threat of an Aging Workforce, Oxford University Press,
Oxford.
Foster, L. (2005) “Confronting the Global Brain Drain”, Knowledge Management Review, Vol 8, No 5, pp 28–31.
French, W.L and Bell, C.H. (1999) “Organization Development: Behavioral Science Interventions for Organization
Improvement”, 6th edition, Prentice-Hall, Upper Saddle River, New Jersey.
Garson, G.D. (2009) “Structural Equation Modeling”, [online] from Statnotes: Topics in Multivariate Analysis,
http://faculty.chass.ncsu.edu/garson/PA765/structur.htm.
Garson, G.D. (2010) “Structural Equation Modeling Example Using WinAMOS: the Wheaton Study”, [online],
from Statnotes: Topics in Multivariate Analysis, http://faculty.chass.ncsu.edu/garson/PA765/structur.htm.
Hair, J.F, Anderson, R.E, Tatham, R.L and Black, W.C. (1995) Multivariate Data Analysis, 4th edition, Prentice-
Hall, London.
Martins, E.C. and Martins, N. (2011) “The Role of Organisational Factors in Combating Tacit Knowledge Loss in
Organisations”, South African Business Review, April, Vol 15 No 1..
Martins, E.C. and Meyer, H.W.J. (2012) “Organisational and Behavioural Factors that Influence Knowledge
Retention”, Journal of Knowledge Management, Vol 16, Iss 1.
Nicholson, N. (2008) “Empower the Next Generation”, Communication World, Vol 25, No 2, pp 14–18.
Robbins, S.P. (1998) Organizational behavior: Concepts, Controversies, Applications, 8th edition, Prentice-Hall,
Upper Saddle River, New Jersey.
Robbins, S.P. (2005) Organizational Behavior, 11th edition, Pearson Prentice-Hall, Upper Saddle River, New
Jersey.
Rowe, K. (2006) “The Measurement of Composite Variables from Multiple Indicators: Applications in Quality
Assurance and Accreditation Systems – Childcare”. ACER, August, pp 1–6.
Scholl, W. and Heisig, P. (2003) “Delphi Study on the Future of Knowledge Management: Overview of Results”,
in Knowledge Management: Concepts and Best Practices, 2nd edition, edited by K. Mertins, P. Heisig and
J. Vorbeck, Springer, sl, pp 179–190.
Tabachnick, B.G. and Fidell, L.S. (1983) Using Multivariate Statistics, Harper and Row, New York.
Uys, T. and Puttergill, C. (2003) “Sampling”, in Intellectual Tools: Skills for Human Sciences, edited by D
Rossouw, 2nd edition, Van Schaik, Pretoria, pp 107–116.
Venzin, M, Von Krogh, G. and Roos, J. (1998) “Further Research into Knowledge Management”, in Knowing in
Firms: Understanding, Managing and Measuring Knowledge, edited by D. Kleine, G. Von Krogh and J.
Roos, Sage, London, pp 26–66.
267

Searching for a Third way: Self-Justification
John Mendy
University of Lincoln, UK
jmendy@lincoln.ac.uk
Abstract: Over the years, researchers have been exploring ways of dealing with the problem of justification,
partly to simplify and facilitate teaching its fundamental role and partly to improve the quality of research-inpractice.
Traditionally, justification (or justifying one’s research) is linked to depersonalisation, formalised in terms
of criteria such as reliability and validity. The need for different forms of justification has been keenly felt due to
some difficulties with this notion. The question arises: how can one depersonalise when one deals with
situatedness? The arisen situation has led to a plethora of alternative criteria for justification, such as credibility,
transferability, confirmability or recoverability and even usability. Each of these notions carries different
interpretations according to the research purpose(s). My focus in this paper is to search for a solution of the
problem of justification (or justifying) in an area where it has received insignificant attention yet: an area where
people construct boundaries to their experiences by cooperating in some task. Here justification can and needs
to be explored as if new. In this area justifying one’s research is not aimed at sentences that traditionally seek to
represent ‘a’ or ‘the’ reality. Here, people construct boundaries to their experiences by cooperating in, and
thereby learning from, some task rendering the question of justification one of identifying which task boundaries
are to be preferred and learnt from. The area seems to have exploded in the literature in the last decade or two.
An earlier recognition of this approach was Taylorism, as part of the tradition of representing some reality. What
is being treated as if new is the realisation that representation has to be replaced by something else, as follows.
What people are able and wish to contribute to and learrn from, depends on what the collective task is and vice
versa. If people’s willingness to contribute to their development and that of the collective task is frustrated, they
start to resist. In consequence, the problem of justification becomes one of linking the level of individual and that
of collective experiences such that each level learns from and maintains the other even when challenged. If the
latter is to happen, attempts should be made to demonstrate the link’s uniqueness. This means that the process
of discovery and protecting each level’s learning and maintainability become justified and are combined in terms
of a process of self-justification. The justification process is exemplified by the results of a study concerning the
behaviour of employees in 4 organisations facing radical changes because of external challenges. It is described
how certain individuals are able to take initiatives superior to their managers’. Their activity may be called
‘employees as producers’. They become autonomous learners and contributors. Interestingly, they become able
to link personal to collective development and thereby to contribute so their companies become more responsive
to new challenges. It is argued that this means that they strive to increase the quality of the link between
individual and collective contributions to each other’s learning and thereby self-justify that link. This type of
justification is based on the equivalence between finding a unique link between individual and global
observations, similar to the case of representation. Comments are added to contextualise the problem of finding
this unique link. In one it is pointed out that links between non-observational contributions are more severely
restricted and more difficult to achieve than links between observational contributions, traditionally implemented
using reliability and validity.
Keywords: research, justifying, coherence, procedure and learning
1. Introduction
1.1 Aim of paper
Over the centuries, science in general and researchers in particular have been struggling to deal with
the teaching and implementation of a significant problem: justification. My focus in this paper is to try
to invent a procedure designed to solve this problem by talking about variety to help clarify this
resurgent problem so we don’t have to spend more resources in future research. The language of a
production procedure is chosen, although other forms (e.g. design, approach, “superstructure” (Vahl
1997: 2)) have been used in other contexts. While some procedures have studied organisations (or
the world) from the outside by using language to represent that ‘world’ in a depersonalised way, I
intend to write about this “craving for justification” (Miller 2007: 4) from an insider’s position. In the
latter, the researcher is part of the situation being studied whilst there is some detachment between
self and the problem one is intending to solve in ‘world’ in the former. In the latter, his/her actions
influence what he/she produces and is influenced by participants’ intentions. When one adopts the
inside perspective, language becomes a structuring factor or if one wishes, research becomes (and
is) a problem of what language we use to deal with a justification problem. The latter should take into
account what boundaries its participants are willing to construct and thereby contribute to and to the
final product/tool etc. The relation between the inside and the outside, between the researcher and
his/her ‘world’ (as the link between structure and content of language) is mapped. Such mapping is
268

John Mendy
anticipated to help us recognise what contributes to people’s actions and what does not. It is further
aimed at discovering where preferences lie in the individual and collective boundaries we carve out of
problems and what contributes to the quality and hence the protection of the mapping. In facilitating
the visibility of the linkage between these elements, we are not restricted to the predictability of
outcomes or even their representation but their anticipation and situatedness. Practical procedures
are provided. These should identify relevant theories and the steps taken in their structuring. The
production procedure should help study respondents’ statements, justify the data as used, the steps
needed to collect these and what approach(es) may be developed as (a) proposed alternative(s) of
doing research. It is proposed such procedures will help solve the justification problem.
The empirical data was collected before (in 2006, 2007) and after (in 2011) the current global
economic crisis from managers and non-managers whose companies had to reorganise to survive.
Using their experiences, the paper captures what individuals choose to contribute to collective tasks
(as their chosen boundaries), what they see as the value of their contributions to deal with the link
between the individual and collective level of functioning. In this sense employees take up active
research functions, in relation to responding to outside influences. People’s contributions should
identify the link between individual and collective objectives, what resource develops from this linkage
and what added value could be provided for future research, learning and development. The mapping
provided in the justification of results is anticipated to provide its uniqueness in the added quality in
that which has not been added yet. It is anticipated the use of employees’ contributions forms part of
a procedure to study companies from an ‘in-here’ position. The results as justified can then be
classified as new forms of research through a new self-justificatory process. The link between
individual and collective performances is treated as unique in a process of self-organisation. This
could provide a resource, which is channelled towards the development of self-maintaining collectives
(see also HEA UKPSF) that constitute “ways of establishing equivalences among beings” (Boltanski
and Thēvenot 2006: 9) as a more inclusive, a more liberating form of organisation, of teaching and
learning about research.
Justification has to contain a number of elements, as follows:
Selecting organisations that show variety;
Selecting management and non-management roles to inform the justificatory outcome(s);
The situation(s) where participants’ constructed experiences are interpreted as part of a ‘meta’
story of individuals contributing to a collective’s task (Rorty 1991). The uniqueness of the results
means mapping has to be unique, i.e. undisturbed by external influences.
The justificatory outcomes should demonstrate a procedure that produces collectives that ‘strive’
to increase their maintainability (i.e. sustainability) through their actions i.e. by increasing
members’ learning and adaptability. Such variety can act as a resource to solve problem
situations in a coordinated way (-employees as producers). The quality of channelling variety
resides in the way individuals’ activities are transitioned onto collective objectives such that this
transition maintains each level even when further challenged.
Checking whether the name ‘employees as producers’ belongs to the activities being described
helps minimise speculative interpretations through a self-checking process.
Helping collectives to develop to the point where they have achieved maintainability between
flexible individual contributions and collective performances to the point of recognising their
progress as a self-recognisable, self-organising and self-justifying collective.
1.2 Historical context of justification
Justification can be traced back to the Protestant Reformation period (C16th.) as a major area of
religious contention. Over time, it came to be associated with one’s method of acting judiciously and
therefore being ‘atoned’ with the law. Recently, theories of justification (e.g. coherence, reasoning,
validity etc) have focused on whether our beliefs and values mirror ‘a’ or ‘the’ ‘real’ world in a
depersonalised way in our quest for new forms of knowledge and understanding. The latter attempt to
address the important question: ‘how do we claim to know and understand what we know and
understand?’ Some of the main opponents of justification are “Bartley (in Bunge 1964)” and Popper
(1934; 1959 trans.; 1980 ed.), who propose “critical rationalism”. Over time, other proposals have
varied between depersonalisation and situatedness. One could lay claims to discovery if it could be
269

John Mendy
shown that something unique has been added from one’s observations and the process justified.
Certain discoveries may lead to inventions (e.g. of products, procedures).
Section 2 deals with challenges, 3 presents the justificatory arguments, while 4 accounts for its
outcomes. Discussions are presented in section 5 and the conclusion in 6.
2. Challenges
To help simplify the justification problem other authors (see Lincoln and Guba 1985; Guba 1990;
Reason and Hawkins 1988; Hacking 1981) have attempted to justify their work by their analogy with
the notion of a paradigm as defined by Kuhn (1962). A group of people interact to solve problems until
deviations appear that require ‘revolutionary’ solutions. It is claimed this procedure enhances further
research competence and where deviations may appear the group ‘strives’ to justify the use of
resources by comparing new findings with existing procedures and theories. Structural changes are
needed in collecting (additional) observation material or a ‘new’ language is designed to solve a
problem emerging from an ‘older’ problem (Whyte, Wilson and Wilson 1969). This could happen
during stages considered ‘deviations’ to science and research when increased variety may pose
further questions to already existing forms of knowing how, of teaching and learning to solve
problems. This may lead to subversion and sabotage or to innovation and increase of collective
competence (see Forsyth 1916; Chase and Simon 1973; Broadbent 1977; Kendall 1982; Frensch and
Funke 1995).
2.1 Dealing with challenges
When organising material, some studies were noted close to the traditional Cartesian procedure
(finding a single link between experiences and description through a reduction of differences of what
the ‘real’ thing is) and others closer to the non-traditional Darwinian (or Darwinian-like) procedure
(where behaviours being observed are closer to the procedure). The first is based on
exhausting/saturating the link between experience and description (finding the widest set of
observations and the most succinct representation). This implies creating a boundary around the
experiences by studying them as if they were an object, so the link between ‘object’ and ‘a’ or ‘the’
‘reality’ remains stable (see Star and Griesemer 1989). In the more complex Darwinian-type
procedure, people construct their own boundaries and can react to or resist the procedure and its
justification by creating more variations in their behaviour. Examples include ‘wicked problems’
(Hardin 1968; Rittel and Weber 1973).
To solve the problem of what justification procedure to adopt, it seems advisable to collect additional
(primary) data (September to November 2011), particularly on companies that continue to be
significantly challenged - to identify whether members’ experiences show variety at complex
behavioural levels (i.e. other than the level where variety is reduced and represented). Research
procedures of how to study variety and resolve the justification problem are presented (see 2.1.1 and
2.1.2).
2.1.1 Procedure 1: Reducing variety
In that part of the literature studying people’s behaviours, machine-like or Cartesian procedures
dominate, relating people’s actual behaviours to preferred end-states. The adopted procedure
includes number variables, the relations between which constitute the problem. These tend to be
used to define the parameters involved (Fayol 1949; Etzioni 1961; Beer 1966). Individuals are
assumed either not able to choose whether to behave like the procedure, implying the possibility of
prediction or to choose to do so (see case of the Hawthorne effect; Landsberger 1958 and Taylor
1947). Those involved may experience restrictions to their freedom and to what they can contribute,
as they are treated as if they were object-like representations of ‘reality’. When they do not, their
behaviour is observed as variety or “overload” (de Zeeuw 1996: 19).
2.1.2 Procedure 2: Controlling variety
The search for a more uplifting procedure still fixates the behaviour of the controllers (e.g.
representational research) in a non-inclusive way. For example, researchers may organise, develop
and teach strategies that define, thereby control, the roles of participants, the potential links between
the roles and what results are predicted.
270

John Mendy
Overstepping of roles (and variety) will, however, increase over time (and become better visible, e.g.
via conflicts) when participants have to adapt their performance against the predicted outcomes.
When this happens managers (as some researchers) may revert to procedure 1 (i.e. increase the
control so contributions and behaviours are predictable and represent-able (Marx and Engels 1985
Ed.; “Weber (in Roth and Wittich1979 eds)”; Tjosvold 2007).
“Espoused values” (Schein 1985: 17, 244) make it possible for individuals to maintain variety by
guarding some personal flexibility (Schein 1985: 17). The ‘value model’ is sometimes claimed to help
achieve the intended change in behaviour as the operational procedure constrains the former to
‘espoused’ values. However, many other values play a part in research and in learning, as they do in
the continuation of a company or a university (e.g. the values of resisting the ‘espoused’ production
procedure by constructing one’s preferences). These preferences cannot be controlled unless through
a more complex procedure.
3. Procedure 3: Self-justification
To deal with the problem of control in research-in-practice one needs a more complex process of
justification. Not only are strategies of delineating members into a boundary with roles sufficient; the
boundary itself becomes changeable in a wider group. It can be optimised to support the collective
when performing activities. This makes the boundary the result of a system of cooperation (Gramsci
1971; Axelrod 1984). It identifies what members of a collective may see others in their environment to
be doing and conveys to others what they too are doing (in terms of objectives and preferences). This
type of process allows one to observe and report a wider range of values – allowing individual
learning and contributions to be quite varied. Individuals become free to self-construct what they like
to include or exclude. Members become aware of what their freedom is for (Jakupec and Garrick
2000).
3.1 Argument 1: On the level of research
Justifying research and striving for higher quality is not a trivial feat. Attempting to simplify the process
is also not simple. Two arguments need clarification. Firstly there is need to change the research
game from the traditionally representative and depersonalised, Cartesian type to the ‘newer’
Darwinian-like forms of organising. Here, situations and contexts do matter. Justifying by induction
has been refuted most convincingly by Popper (1980) – except of course for the trivial case when
what is being described contains only a finite number of elements, the links of which can be observed
and represented. Difficulties arise when it is not known whether the number is infinite or finite. Not
surprisingly, research based on reducing people’s preferences (through representation) tended to
have been among the earliest (see 2.1.1).
3.2 Argument 2: On the level of participants
Secondly, the Darwinian-like procedure suggests that the memories of contributors involved in
change situations and how they might construct their preferences to bring the process of learning
under their control be explored– rather than observe or ‘problematise’ developments as an outsider.
Some associated difficulties are discussed.
Four sets of employees (51 in all) from 4 private-sector organisations intended to emphasise
differences in roles and behaviours were studied. The organisations had these categories.
Interviewees had different nationalities, from different geographical regions, with different values.
They included people of European, African and Asian origins.
Questions were equally as varied and dealt, firstly, with changes in working practices and roles, how
communication was carried out via what language construction. Secondly, they referred to constraints
employees had experienced in performing tasks prior to and after the challenges, what form these
had taken, how they had constructed their experiences of the parameters of managerial relations.
Respondents were invited to add any other constructions (of preferences) they considered vital to
their interactions.
One of the difficulties is that people will be in the process of changing the construction of the internal
language of the company by contributing to the discussions that the procedure generates. The
previous language may not have gone and the new one may not have been fully constructed yet.
There may be significant differences in the ways the differing languages are interpreted and
271

John Mendy
understood by respondents, thereby increasing variations. For example, responses from interviews
may emphasise doubts about the challenges (and express feelings of loss), rather than reveal what is
being constructed in the new company.
Challenges facing the companies are summarised hereunder:
Table 1: Summary of Company Challenges
In 2004, Longhurst faced the challenge of implementing its decision to expand to new
counties in the UK. The company had to devise new divisions of labour, changes in roles
and competences and had to induct new employees from European countries.
In 2005, Eden had to make ‘reasonable adjustments’ to their facilities in order to facilitate
easy access and free movement of physically and/or mentally disabled people or those
with other forms of disability. It had to face increasing resistance to some of its activities,
e.g. building on brown or green sites.
In 2004, Laurens Patisserie faced a relatively fast increase in the demand for patisserie
products (cakes and pies) from supermarkets. This consequently put pressure on
employees’ workloads. The company had to induct non-English staff, e.g. from European,
East European, African and Middle Eastern origins.
In 2005, Prospects incorporated Connexions and had to manage the transfer from the ‘old’
company (Connexions) to the ‘new’ one (Prospects). This included adapting staff to
Prospects’ cultural norms and value systems, agreeing with union members and relocating
staff.
When discussed with managers, they agreed challenges were varied and substantive. Responding to
each challenge was seen as requiring considerable effort, implying individual and collective (re)constructions
and adaptations. It was expected employees would attempt to vary their contributions
by constructing ‘small’ stories, linked to the ways they construct boundaries for a task and to the
company’s overall adaptation.
4. Employees’ statements
Respondents’ constructions initiated when the activities to deal with the challenges were announced.
The story (and the six subsequent steps below) did not differ between the companies. The responses
collected at the previous and recent interviews were treated as if they could be compared and after
their comparison, appeared to tell the same complex story, except step 6. Yet, there has been some
change from the ‘old’ to the new situation as employees started to interpret the new changes in
behaviour as self-justified change and, therefore self-maintained change. Previously people were
innovative because of the threats and because of management imposition to deal with the threats
(see step 2 below); now they are innovative as a collective part of the new challenges (see step 6
below).
1. There was an awareness of the need for changes, for example, from a situation of ‘easy come and
easy go before to get the work done quicker... now, it is more professional and that has changed the
chemistry of the company’.
2. Managers’ imposition of discipline and punishment was considered tantamount to ‘an invasion of
employee territory’.
3. This led to employees’ resistance of formalised relationships, where ‘communication has become
the biggest challenge’.
4. Employees were left ‘feeling handcuffed’ as they tried to develop ‘training’ initiatives for the
collective good.
5. Employees feel the need to be valued but experience obstacles as ‘to change the culture becomes
extremely difficult’ and ‘the company has struggled to get staff from Newark’.
6. Employees begin to interpret tasks in their own way, to take initiatives that they do not report to the
managers, in ways they consider beneficial to their company, as some employees wanted ‘to have
responsibility on the way things are going’ and felt ‘wound up if someone tells me what to do’. They
272

John Mendy
felt ‘able to make the department innovate even when the manager did not understand what was
being done’.
At the sixth step participants begin to take their own initiatives. They direct their preferences as if to
innovate and ‘integrate learning’ (Tinto 1975). These were part of their ways of dealing with the
reorganisation’s challenges. Now we can talk about the quality of the work as a different, somewhat
special way of justifying research from the rather traditional, representational and Cartesian approach.
Responses suggest that workers can generate variety; that they are resilient despite attempts to
control their preferences and are able to get around their managers by the sixth step. Step 6 is
therefore better than step 2 in terms of portraying how a collective can deal with challenges to their
survival. The form of reporting adopted here shows how to move from step 2 to 6 in a systematic way
by developing interactions which have to satisfy 2 criteria:
1. The criteria of step 6 where people take initiatives thereby creating further variety;
2. A way of moving to step 6 systematically and justifying employees’ actions. Employees can judge
whether step 6 activities are good or bad and can change if needs be. The role of research is to
support going to step 6 by supporting experiences and interactions in the direction of step 6 and
possibly beyond.
This form of observing and reporting procedure seems sufficient to identify what employees produce
when externally challenged. However it does not identify the way they look at the future and the way
they take responsibility for events beyond their direct control. A more powerful, more complex
procedure beyond step 6 is needed to see what can be achieved.
4.1 Employees as producers
Having identified the variety of behaviours and preferences that some interviewees have the ability to
initiate around tasks and their usefulness, one might wish to identify some way or approach to
stimulate these. Listening to a story might be effective. One objection would be, however, that a story
usually does not self-justify the behaviour and the preferences it tells about, for example in terms of
expected advantages. There is no indication, for example, that those who develop the required
managerial behaviour in the story are less likely to be fired or relegated to less advantageous roles.
The notion of employees as producers of their competence seems to respond to this problem. It
identifies that those who develop certain behaviours attain some form of power – the power to vary
their contributions and thereby justify their actions against threats, such as obstacles raised by the
managers (or colleagues who subvert) or external ones such as acting as both carer and manager for
people with complex disabilities (e.g. dementia). It might be objected again, unfortunately, that this
does not provide the kind of justification that might be sought by managers or those who seek to
control employees. It turns the story into a tool to achieve an objective (being able to do some
activity). Something more uplifting as well as supportive is required: a more complex level of
justification. One possibility would be to think of a procedure within which an understanding of how
someone who actually wishes to enact the story by acting as a ‘witness’ (see Nevejan and Brazier
2011; Nevejan 2009), as well as create the competence this requires might proceed. Employees-asproducers
capture this type of understanding and justification. When I repeatedly compared the
procedure to the six-step-story developed from the interviews, I came to the same conceptemployees-as-producers
of their learning and adaptation.
‘Employees-as-producers’ indicates a new type of procedure (allowing for the construction of
boundaries that contain far greater developmental preferences and variety than was previously
possible in representational procedures or even in the ‘values-model’). The first 2 procedures appear
to lead to recognition systems in the Cartesian way (sections 2.1.1 and 2.1.2). If they can be shown to
contain the relevant variety of experiences in daily life, they may allow for future recognition – if this
leaves the procedure untouched (i.e. with no remaining variety). The procedure leading to the
construction of employees-as-producers is intended to help create collectives that perform some task
or action and have the freedom to vary their contributions and preferences. This is done through a
process of self-organisation and self-justification, without their survival being disturbed or threatened
or, if one wills, a collective becoming closed to further changes. This implies that the constructions
and the contributions of the actors in the collectives constitute a resource for further learning and
development (Kolb 1974; Fineman 2000; Bauman 2004). What constitutes people’s idea of the
273

John Mendy
collective task is recognised from the outside as a collective act even when the procedure adopted in
its reporting shows differences in language used amongst participants and their interpretation of that
language. This process seems to observe and is referred to as providing understanding of the idea
that previously could be identified only dimly and partially by the story.
The procedure invented here serves as a double recognition system. Further checking may increase
its quality. What seems to be of particular value is that the procedure contributes a valuable resource
to support employees in a process of self-checking or self-justification that maintains collective
survival. The concept allows to identify what contributes to its own viable implementation in
organisational situations and hence when it has achieved high quality: i.e. when the collective stays
undisturbed by further challenges. This characteristic makes it highly attractive as an alternative to the
traditional procedure (see sections 2.1.1 and 2.1.2). Implementing the procedure implies that people
(as participating members of organisations) become able to choose what objective to construct. This
makes checking the concept part of its implementation: one cannot develop a procedure that
constructs an object function if one does not interact and ensure that individual activities help realise
the collective act. The procedure contributes in two valuable ways: it serves as a constraint on what
people are able to contribute to tasks and facilitates the effectiveness and changeability of their
contributions when considered necessary. Therefore, it provides a framework for the observation and
the language for reporting of the behaviour in local, context-dependent situations that goes beyond
those of representations, stories and interactions. It is self-justificatory.
5. Discussion
The interviews are interpreted as presentations of employee experiences and as expressions of a
more general linguistic structure. The variety of the responses was taken to indicate that employees
were varying their contributions and thereby adapting individual preferences to collective
reorganisations, “acting as representatives in their turn” (Boltanski and Thēvenot 2006: 3). The sixstep-story
supports and is supported by some elements of the literature (e.g. on management’s
attempts to reduce differences, on individual and collective learning and development and selfjustification).
The procedure reported what direction the statements suggested that employees wished
to modify the constructions of their experiences. Respondents could be said to have ‘researcher-ship’
and ‘knowing-in-action’ (Schon 1983). To enable this type of act, the statements were structured and
reported in a language that tells the ‘meta’-story of how to invent a procedure that helps justify
combining preferences.
In addition to the procedure reported here, other approaches have been developed to approximate
the justification of what one is exploring in case it is unknown what amount of individual and collective
preferences is involved. These may be categorised into three types. The first one is to consider the
procedure’s internal coherence. If, for example, a next step only makes use of a previous step and
nothing is added from outside the step, the results are considered justified. This often is called the (a
posteriori) utility approach (Brody 1984; Finlay 2008). The second is to consider external coherence.
This means that research is taken to add something ‘wider’ to an initial problem area, e.g. a frame via
which to identify related data, or a pattern to predict or anticipate new phenomena, or an environment
that needs to be controlled to be able to achieve a procedure’s objectives. This is often called
justification by design. The third is to position the work done in terms of some ‘debates’, or of what
seems close, ‘traditions’. This type of justification raises the additional problem of the entanglement of
the languages being studied – the language of its coordination and the language of the debates.
These include discussions whether the results are due to the use of quantitative or qualitative
methods, whether they can be used with some confidence even if the design is executed with limited
resources, whether what is added represents ‘a’ or ‘the’ ‘reality’ and so on. None guarantees that
checking will be exhaustive as “defections are not avoided” (de Zeeuw 1996: 20). The alternative
research proposition developed in the paper is a form of language, whose structure determines or
produces the research content. If research is conducted following step 2, it would produce data or
reports for management to impose on employees further, thereby help to justify the curtailing of
employees’ variety. At step 6, this will not suffice as differences in interpretations are permitted while
still allowing space for collective action. The procedure leading to employees-as-producers (post step
6) was found to have sufficiently mirrored both the name as well as the naming of the variety that
employees choose to contribute to collective performances for their maintainability.
274

6. Conclusion
John Mendy
To meet the paper’s objective, the procedure I invented included arranging for interviews with people
familiar with at least some organisations experiencing radical restructuring. The procedure and its
elements were identified as were possible outcomes and checking for quality. Combining these
elements as an alternative way to study organisations from a situated perspective is proposed.
Further work is required to deal with emerging multi-faceted variety from a group of interacting
participants (e.g. higher education policy-makers, tutors, university authorities and students,
managers and non-managers) especially where and when the division of labour, roles and
interactions have been loosened to a point where a larger range of conflicting values, opinions, myths
and assumptions become operational. Further work is also considered necessary to capture the next
level of research development beyond the sixth step of operational procedure. In other words, what
happens to participants (e.g. employees-as-producers etc) once they have achieved the learning, the
skills for collective ‘researcher-ship’ competence? On an individual level, what (or who?) do I
become? These, and related questions/problems could provide the bases for more complex research
formulations and further research on the processes of research, processes of discovery and problemsolving
as well as processes of self-justification. The work continues...
References
Axelrod, R. (1984) The evolution of cooperation, Basic Books, New York.
Bauman, Z. (2004) Work, consumerism and the new poor, Open University Press, Maidenhead.
Beer, S. (1966) Decision and Control, Wiley, New York.
Boltanski, L. and Thēvenot, L. (2006) Justification: economies of worth (C. Porter trans.), Princeton University
Press, Princeton.
Broadbent, D.E. (1977) ‘Levels, hierarchies and the locus of control’, Quarterly journal of experimental
psychology, Vol. 29, pp 181-201.
Brody, P.J. (1984) ‘In Search of instructional utility: A function-based approach to pictorial research’, Instructional
Science, Vol. 13, No. 1: pp 47-61.
Bunge, M. (1964 ed.) The Critical Approach to Science and Philosophy, The Free Press of Glencoe.
Chase, W.G. and Simon, H.A. (1973) ‘Perception in chess’, Cognitive Psychology, Vol. 4, pp 55-81.
Darwin, C. (1858) ‘On the tendency of species to form varieties and on the perpetuation of varieties and species
by natural means of selection’, Journal of the proceedings of the Linnean Society of London, Zoology Vol. 3:
pp 46-50.
Descartes, R. (1837) Discourse on Method [Translated from French: Discours de la mếthode], Penguin Books,
London.
De Zeeuw, G. (1996) ‘Three Phases of Science: A Methodological Exploration’, Centre for Systems and
Information Sciences, University of Humberside, ISBN186050 250 0
Etzioni, A. (1961) A Comparative Analysis of Complex Organisations: on power, involvement and their correlates,
Free Press, New York.
Fayol, H. (1949) General and Industrial Management, Pitman Publishing, New York.
Fineman, S. (2000) Emotions in Organisations, Sage, London.
Finlay, S.M. (2008) ‘Towards profitability: a utility approach to the credit-scoring problem’, Journal of Operational
Research Society, Vol. 59: pp 921-931.
Forsyth, P.T. (1917) The Justification of God: Lectures for Wartime on a Christian Theodicy, Independent Press,
London.
Frensch, P. and Funke, F. (1995) Complex problem-solving: the European Perspective, Laurence Earlbaum
Associates, Hillsdale, New Jersey.
Gramsci, A. (1971 Ed.) Selections from the prison notebooks of Antonio Gramsci (Q. Hoare and G. Nowell-Smith
eds. and trans.), Lawrence and Wishart, London.
Guba, E.G. (1990) The paradigm dialogue, Sage, Newbury.
Hacking, I. (1981) Scientific revolutions, Oxford University Press, New York.
Hardin, G. (1968) ‘The Tragedy of the Commons’, Science, Vol. 162, No. 3859: pp 1243-1248.
Jakupec, V. and Garrick, J. (2000 ed.) Flexible Learning, Human Resource and Organisational Development,
Routledge, London.
Kendall, P.C. (1982 ed.) Advances in cognitive behavioural research and therapy (Vol. 1), Academic Press, New
York.
Kolb, D. (1974) Experiential Learning: Experience as the source of Learning and Development, Prentice Hall,
London.
Kuhn, T.S. (1962) The Structure of Scientific Revolutions, University of Chicago Press, Chicago.
Landsberger, H.A. (1958) Hawthorne revisited, Ithaka, Cornell University.
Lincoln, Y. and Guba, E.G. (1985) Naturalist inquiry, Sage Publications, Newbury Park, California.
Marx, K. and Engels, F. (1848) The Communist Manifesto, (S. Moore 1985 ed. and trans.) Penguin,
Harmondsworth.
Miller, D. (2007) ‘Overcoming the justification addiction’, Lecture delivered at University of Bergen, Socrates-
Erasmus 2006/07 Teacher Mobility Programme, funded by Commission of European Communities.
275

John Mendy
Nevejan, C. (2009) ‘Witnessed presence and the YUTPA framework’ Sprouts, PsychNology Journal, Vol. 7, No.
1: pp 59-76. ISSN1535-6078.
Nevejan, C. and Brazier, F. (2011) ‘Witnessed Presence in Merging Realities in Healthcare Environments’,
Studies in Computational Intelligence, Vol. 326: pp 201-227. doi 10.1007/978-3-642-16095-0_11
Popper, K.R. (1980 Ed.) The Logic of Scientific Discovery, Unwin Hyman, New York.
Reason, P. and Hawkins, P. (1988) Human Inquiry in Action: Developments in New Paradigm Research, Sage,
London.
Rittel, H. and Weber, M. (1973) Dilemmas in a General Theory of Planning, Policy Studies, Vol. 4: pp 155-169.
Rorty, R. (1991) Consequences of Pragmatism, Harvester, London.
Roth, G. and Wittich, C. (1978 Eds.) Economy and Society: an outline of interpretive sociology, University of
California Press, Berkeley.
Schein, E.H. (1985) Organizational Culture and Leadership, Jossey-Bass, San Francisco.
Schon, D.A. (1983) The reflective practitioner: how professionals think in action, Arena, Aldershot.
Star, S.L. and Griesemer, J.R. (1989) ‘Institutional Ecology, Translations and Boundary Objects, Amateurs and
Professionals in Berkeley’s Museum of Vertebrate Zoology’, Social Studies of Science 19, 1907-39, pp 387-
420, doi: 10. 101177/030631289019003001
Taylor, F. (1947) Scientific Management, Harper and Brothers, New York.
The UK Professional Standards Framework, The Higher Education Academy, The Higher Education Academy,
York, UK.
Tinto, V. (1975) ‘Dropout from Higher Education: A Theoretical Synthesis of recent Research’, Review of
Education Research, Vol. 45, pp 89-125.
Tjosvold, D. (2007) ‘Cooperative and Competitive Goal Approach to Conflict: Accomplishments and Challenges’,
Applied Psychology, Vol. 47, No. 3: pp 285-313.
Vahl, M. (1997) ‘Doing research in the social domain: concepts and criteria’. In Systems for Sustainability,
People, Organisations and Environments, Stowell, F.A., Ison, I.R., Plenum Press, New York/London.
Whyte, L.L.; Wilson, A.G. and Wilson, D. (1969) Hierarchical Structures, American Elsevier, New York.
276

Research Philosophical Debates and Classifications:
Students’ Dilemma
Marcia Mkansi, Edwin Asiamah Acheampong, Baomi Qi and Kondal Reddy
Kondadi
University of Bolton, UK
mm1lia@bolton.ac.uk
eea1lis@bolton.ac.uk
bqi1@bolton.ac.uk
k.r.kandadi@bolton.ac.uk
Abstract: Research philosophical debates and classifications such as ontology, epistemology, and anxiology are
rife and causes dilemma to research students in establishing its relevance to subjects areas and discipline. A
number of studies have used different descriptions, categorisations and classifications of research paradigms
and philosophies in relation to research methods with overlapping emphasis and meanings. This has resulted in
tautological confusion of what is rooted where, according to whom, how it informs future studies and its impact on
those that are subject to its application especially, research students. A discussion that may provoke research
and paradigm advocates from their peripheral standpoint to become concerned about students’ is required.
Eventually, this may result in the development of a planned, systematic framework, and procedure that show
some consensus to bail research students from these dilemmatic debates. This paper briefly reviews, discusses
and analyses the debates and classifications through literature. Then, assesses how it impacts on research
students’ understanding and choice of research approach through case studies based on two North West
Universities in the UK. The students are divided into different faculties and subjects groups to allow for
comparison and to obtain different perspectives and understanding of these impacts. Interviewed PhD research
students were purposively sampled; and the questions were structured for comparability purposes. The
overarching aim is to provide a mapping of these research philosophical debates and classifications, provide their
clear distinctions and especially their implicit impacts on research students’ understanding and decision of
research approach for their projects. Although, the case studies offer detailed understanding of students’ impacts,
the findings are not conclusive to all students in the UK; and the focus on faculty and subjects groups is of major
limitation. Therefore, recommends future studies to employ a survey and narrow the inquiry to subjects, research
problem, philosophy and faculty of the study.
Keywords: research philosophies, debates, students
1. Introduction
The difficulty in conducting research today is navigating through the research philosophical debates
relating to epistemology, ontology, doxology that those who made it are unlikely to be affected by it. A
number of studies (Saunders et al., 2009; Ritchie and Lewis, 2003; Guba, 1990; Guba and Lincoln,
1989) have used different descriptions, categorisations and classifications of research paradigms and
philosophies in relation to research methods with overlapping emphasis and meanings. This has
resulted in tautological confusion of what is rooted where, according to whom, how it informs future
studies and its impact on those that are subject to its application especially, research students. The
mounting debates have taken a distinctive turn that causes dilemma to research students in
establishing its relevance to subject areas and disciplines. This paper briefly reviews, discusses and
analyses the debates and classifications from literature reviewed. The dilemma will be examined in
this research through a series of case studies conducted with research students from two different
North West Universities in the UK. The students are divided into different faculties and subjects
groups to allow for comparison and to obtain different perspectives and understanding of these
impacts. Structured interviews are used to collect data. The overarching aim is to find out whether and
how research philosophical debates and classifications impacts on understanding and decision of
research approach by PhD students’, and provides a mapping of these debates as well as their
classification and distinctions.
2. Research philosophical stance debates
Proponents of research philosophies (Saunders et al., 2009; Guba and Lincoln, 1994; Becker, 1996)
have engaged and displayed their knowledge and beliefs in what appeared and got interpreted as
paradigm “wars”. Although their definitions of ontology, epistemology, and anxiology have a common
theme with a bit of different meaning and emphasis; there seem to be no consensus in the
classification and categorisation of these paradigms. A thorough literature review (Saunders et al.,
2009; Will et al., 1997) of these philosophies and research methods has overlapping evolutionary
277

Marcia Mkansi et al.
process. The evidence ranges from the original presentation, description and categorisation outlined
amongst other major philosophical advocates (Burrell and Morgan, 1979; Guba and Lincolm, 1989;
Guba, 1990) to recent philosophical scholars (Saunders et al., 2009; Ritchie and Lewis, 2003). For
example, Ritchie and Lewis‘s (2003) description and classification of ontological and epistemological
stances is different from that of Saunders et al. (2009). Ritchie and Lewis’s (2003) ontological
perspective include realism; materialism, critical realism, idealism and relativism; and the
epistemological perspective include positivism and interpretivism. Hence, Saunders et al.’s (2009)
and Guba and Lincoln (1994) indicate a perspective that views philosophies (i.e. positivism, realism,
interpretism, and pragmatism) from an ontological, epistemological, axiological stance. An even
overlapping classification of these philosophies is that of Guba and Lincolm (1989); which links
positivism, post–positivist, and constructivist to critical realism. A further example in the context of
philosophy is pragmatism which, is thought to have evolved from realism described as Peirce realism
by Thayer (1981). These philosophies are not entirely different. Put differently, they all share a
common set of assumptions, and their commonalities identify these philosophies as examples of
broader philosophies. However, whilst they share critical assumptions, they emphasize very different
implications of those assumptions. And while they all focus on explaining methodological differences
in research, they adopt different categorisation and classification. Given these differences, it is not
surprising that these philosophical debates have generated a dilemma for research students. Indeed,
recognising these impacts to students can help organise this ongoing philosophical debates and can
help explain differences amongst the different philosophical stance.
3. Philosophy and research approach debates
There is much ongoing debate on where a particular method (i.e. qualitative, quantitative or mixed
method) is rooted or founded in relation to these philosophies (Johnson and Duberley, 2000; Bryman,
1984; Morgan and Smircich, 1980; Caelli et al., 2003). Even Guba and Lincoln (1994) acknowledged
the ongoing patents of paradigm to research approach. Few exemplary discussions of these debates
are provided on table 1 below:
Table 1: Philosophical debates for: Mixed methods, qualitative and quantitative approach
3.1 Philosophical debates for neither of the approaches
Whilst the scholars battle out on the foundation of philosophies or paradigms to research approach,
Johnson and Onwuegbuzie (2004) argue that the differences in epistemological beliefs and logic of
justification do not dictate what specific data collection and analytical methods to utilise, therefore,
should not prevent the exploitation of any methods. In support of this, Guba and Lincoln (1994) stress
278

Marcia Mkansi et al.
that the rooting of approaches is secondary to paradigm and do no limit the use of either qualitative or
quantitative from any research paradigm. Differences of trivial nature paid to philosophies in relation
to research approach are greatly discussed by Becker (1996).
4. Philosophy and field of study debates
Then follows the debate on which philosophy is for social sciences or natural sciences? The survey
findings by Orlikowski and Baroundi (1991) and Alavi and Carlson (1992) , for example, report that the
philosophies found to be widely applied and more popular in Information systems was positivism, and
links the philosophy to the field of social sciences (Steinmetz, 2006). But, Polit and Beck (2008)
associate positivism to natural sciences. In a different view, Bryman (1984) associates positivism to
social research that applies natural science. This leads to subject rooting to different fields of study,
for example, Information system’s roots is found to be overlapping with different field of studies such
as social sciences, computer science, and business studies (Orlikowski and Baroundi, 1991;
Hirschheim, 1985; Steinmetz, 2006). Hence the debate boarders on whether quantitative is for natural
sciences (Polit and Beck, 2008; Steen and Roberts, 2011), and the case of qualitative approach in
social sciences as is widely prescribed by Babbie (2007).
5. Philosophy and subject debates
The philosophical foundation debate gave rise to contradicting arguments as to which philosophy is
best for a particular subject. For example, three different philosophical views have been identified for
information systems (IS), namely, critical social theory (Ngwenyama and Lee, 1997; Orlikowski and
Baroundi, 1991). Pragmatism (Agerfalk, 2010; Goldkuhl, 2008); critical realism (Hjorland, 1998;
Dobson, 2002). Yet, the survey findings by Orlikowski and Baroundi (1991) and Alavi and Carlson
(1992) indicate that the philosophy found to be more popular in Information systems is positivism.
5.1 Pragmatist view to the subject of information system
Goles and Hirschheim (2000) brought pragmatism into Information systems. The importance of
pragmatism to information systems has been acknowledged amongst others by Agerfalk (2010); and
Baskerville and Myers (2004). The emphasis by these scholars is that information systems is often
seen as pragmatic discipline with a prominence on practical research, theory and practical
implications.
5.2 Critical social theory view to the subject of information system
The connection between CST and IS had been launched from two fronts: firstly, relationship between
science, theory and practice; and the social action and meticulous knowledge in which it is based
(Alvesson and Willmott, 1992). Klein and Hirschheim (1993) emphasised the relationship of
pragmatisim and IS to work, social and interaction. Examples of CST application to IS research
include, amongst others, studies by Alstyne and Brynjolfsson (2005); Orlikowski and Baroudi (1991).
5.3 Critical realism view to the subject of information system
Several scholars (Scott, 2007; Carlsson, 1989; Barrett, 2010; Dobson, 2002) have endorsed CR as
an epistemological stance for information systems. The emphasis is towards its ability to address
natural and social sciences, which offer a platform for use of variety of methods. Hence, its ability to
take a realist stance by pointing out the limitations of positivism and interpretivism individually, and
subsumes critical social theory (Mingers, 2004; Wikgren, 2004).
6. Research terminology debates
Advocates of research methods (Srivastava and Rego, 2011; Saunders et al., 2009; Khotari, 2006)
have used different terminologies that are contradictory one to another, which leave students
staggering as to which is which, and why there is no consensus in the scholarly fields. Amazingly,
even Johnson and Onwuegbuzie (2004) picked on some of this confusion; which was on the
reference or treatment of epistemology and method as synonyms. Yet, not much thought is given to
how it impacts on students, who widely apply and rely on these scholars’ guidance for research
directions and clarifications.
279

Marcia Mkansi et al.
6.1 Examples of terminologies on research approaches
A number of studies have used different descriptions of the main research approaches with common
themes, categorisations and overlapping emphasis, which is, in fact, qualitative and quantitative. For
example, quantitative is also known as empirical research (Hinchey, 2008); deductive, explanatory
(Saunders et al., 2009; Engel and Schutt, 2005). On the other hand, qualitative is also known as
exploratory research (Neelankavil, 2007; Engel and Schutt, 2005; Bernard, 2006); inductive
(Saunders et al., 2009; Engel and Schutt, 2005); and formulative (Khotari, 2006). Whilst the latter
scholars acknowledge the similarity of these approaches, some other scholars treat these approaches
separately. For example, Hinchey (2008) clearly indicates that empirical research is quantitative, but,
Srivastava and Rego (2011) described empirical and quantitative as separate types of research
approaches. In a thorough review of these terminologies, it appears rather, to have amounted to a
tautological confusion of these different descriptions of research approaches. This is because all other
types of research approaches are variations of either one or combination of qualitative, quantitative,
and the combination of the two often referred to as mixed method or multi methods depending on the,
field of study (i.e. social science, natural science), or the source of scholarly reference (i.e. Saunders
et al., 2009; Khotari, 2006, etc). In support of the fact that there are only three distinct approaches to
research is Bryman (2006).
6.2 Examples of terminologies on research approach, techniques and strategies
Morgan and Smirch (1980) consider qualitative research as an approach rather than technique. But,
the latter is referred to as paradigms by Cluett and Bluff (2006) cited in Steen and Roberts (2011).
Newman and Benz (1998) use strategies and approaches interchangeably as reference for qualitative
and quantitative in the same book; whilst Sauders et al. (2009) refer survey, experiment, observation,
ethnography, archival, grounded theory and case study as research strategies.
7. Research design and methodology
This research adopted a qualitative approach and considered case study as strategy. This was mainly
due to the strength of case study in answering ‘how’ questions and providing in-depth understanding
of phenomena as widely described by Yin (2003). Structured interviews were considered as
appropriate technique for extracting comparable findings, as echoed by Khotary (2006). The study
focused on two North West Universities as case studies to provide in-depth perspectives of how
research philosophical debates and classifications impact on PhD research students’ understanding
and decision of research approach. The aim was addressed by:
Reviewing literature in order to map different philosophical debates by different scholars, from
which the structured questions for interview purposes were derived.
Purposively sampling PhD research students from different subjects, faculties, and Universities,
and used both personal and telephone methods of interviews.
The purpose of focusing on two different University, PhD research students, diverse subjects, and
varied faculties was to provide comparable multi-perspectives of impacts to understanding and their
decision of research approach. Saunders et al. (2009) assert that this kind of approach is good for
establishing whether the findings of one case occur in another case. At best, the limit to two
Universities and the overall approach helped to eliminate some of the fervent criticisms against case
study relating to control, access, time, and travel and analysis cost as expressed by among other
scholars (Gable, 1994; Rowley, 2002; Darke et al., 1988). The selected cases in the North West are:
The University of Manchester and University of Bolton. The usefulness of studying a limited number of
cases in-depth have been discussed and expressed by John and Onwuegbuzie (2004).
7.1 Case analysis
The study collected at first data that aimed at providing profiles of 15 PhD students that took part in
the cases, from which comparison can be made for each faculty. The purpose for profiling was to gain
insight into the extent of understanding of research philosophical debates and classifications by each
PhD students and to find clues as to how the students decide on research approach. This aided in
providing evidence and determining similarities and differences in levels of understanding and
decisions on research approaches. The total response rate was 9 (60%) PhD students’ for Bolton
University, from which 7 were in their final year, 1 in second year, 1 first year student. Manchester
University’s response rate comprised of 6 (40%) students, from which 5 were in third year and 1
second year PhD students’. Respondents came from 60% male and 40% female students from
280

Marcia Mkansi et al.
diverse subjects groups which comprised of: business, science, engineering, music and theology. The
breakdown of students faculties are: 53.3% social sciences, 13.3% humanities, 20% science, and
13.3 engineering (table 2). Data was coded and analysed through SPSS which allowed for some
quantification, and mapping of different approaches chosen by PhD students in each faculty, and
subjects; whilst enabling a point of in-depth comparison and contrast of impacts to understanding and
decision making.
Table 2: Faculty distribution of respondents
Students’ faculties Percent Valid Percent Cumulative Percent
Social Sciences 53.3 53.3 53.3
Humanities 13.3 13.3 66.7
Sciences 20.0 20.0 86.7
Engineering 13.3 13.3 100.0
Total 100.0 100.0
7.2 Case results: PhD students from Bolton and Manchester
The interview canvassed opinions on research philosophical debates in relation to its varied
classifications, research approach, field of study, subject, and research terminologies debates.
7.2.1 Philosophical stance debates findings
This part of the study highlights key issues, which PhD students’ have towards research philosophies,
debates and its classifications. Of particular interest were the perceptions of understanding and
relevance of research philosophies in choosing a research method. The findings indicate similar
patterns of confusion in the classification and understanding of research philosophies by PhD
students across different faculties and year of study. Whilst 73.3% of students indicated awareness of
different classifications and debates surrounding research philosophies, 80% reported that it was
difficult and confusing to understand. This came as a sharp contrast from 6.7% that thought research
philosophies was easy and straight forward to understand. The students’ grounds for difficulty and
confusion are based on: ambiguity, different descriptions, and categorisation of same philosophy.
Could this be confirming Silverman’s (2010) dangerous view of research dichotomies?
A disconcerting finding is the fact that 33.3% PhD research students from science and engineering
faculties were less concerned and not much knowledgeable on the philosophical debates and
classifications, and felt it was less crucial in their studies. This perception is more inclined to be in disaccordant
with the rest of other faculties’ students (66.7%), who indicated a disentangled view
describing them variously as, necessary but irrelevance vis-à-vis relevant but unnecessary, or
necessary and relevant. The views of science students combined with those that were thorn in
between unnecessary and relevance is in accordance with Becker’s (1996) unnecessary view of
differences paid to philosophies in relation to research approach. Of greater interest however, was the
students’ rationale for associating with research philosophies. A significant percentage (33.3%) mix of
students from science, humanities, and social science claimed to have been influenced by their
supervisors towards a particular philosophy. The break down of which is: 6.7% from social science
towards interpretivism, 6.7% from science towards positivism, 13.3% from science towards critical
realism, and 6.7% from humanities towards interpretivism. Even though 13.3% indicated a kind of
negative enforcement towards the philosophies, it will be highly risky to assert any positive or
negative influence by supervisors. Therefore, the question of supervisors’ influence towards a
particular philosophy remains a research question. Whereas, the other mix of 33.3%, from social
science (20%), science (6.7%), and engineering (6.7%), who espoused interpretivism, CR, and CST
claims to be influenced by their research approach.
The students’ primary view of research approaches to philosophies creates intellectual dissonance,
that is, in contradictory belief to Guba and Lincoln’s (1994) secondary view of approaches rooting to
philosophies. Only 20% social science students indicated a combination of self belief and research
approach as their motive towards interpretivism. However, assert that their philosophical stance
changes depending on the research problem and approach. Clearly, there is a huge disparity in these
students’ view and understanding of philosophies to that described by the majority of philosophical
advocates (Saunders et al., 2009; Ritchie and Lewis, 2003; Polit and Beck, 2008). A minor view by
6.7% interpritivist from social science claim to be influenced by field of study, whilst 6.7% from
engineering could not relate with any of the philosophies due to lack of knowledge.
281

Marcia Mkansi et al.
7.2.2 Philosophy vs. research approach debates findings
The majority of students 46.7% indicated that qualitative approach can be viewed from any
philosophical stance (table 3). Hence, 53.3% linked qualitative approach to some kind of philosophical
stance that is interpretivisim, critical realism, and or critical social theory; with the majority of the
former break down gravitating towards advocates of interpretivisim for qualitative approach (McNabb,
2008; Denzin and Lincoln, 2005).Yet, quantitative and mixed method were not linked to a particular
philosophy by the majority. The total of 66.7% students indicated that quantitative approach can be
viewed from any philosophical stance, and 73.3% showed similar perception about mixed methods. In
summary, the majority of students could not link a particular approach to a particular philosophy.
There seem to be inclination towards books that are in similar view to Johnson and Onwuegbuzie
(2004); Guba and Lincoln (1994), who do not limit the use of approaches from any philosophical
stance. Reasons for their views of philosophies seemed to emanate from research books for the
majority of 73.3% students. Since majority of students indicated awareness towards different
classifications and debates, but their views of philosophies leaned towards books in line with Guba
and Lincoln (1994) and Johnson and Onwuegbuzie (2004), would this imply that students found less
confusion and more flexibility in the non attachment of philosophies to research approaches?
Although there is patterns of confusion in understanding philosophies, the was almost universal
agreement that research problems inform the research approach by students, which is indeed in line
with the majority of research methods advocates (Sauders et al., 2009; Khotari, 2006; Polit and Beck,
2008). Research students’; disagree, in part, to the manner in which the advocates of each
philosophy put them forward. The core reason was because all of the philosophies represent different
but complementary dimensions of a research philosophical stance. But lack consensus, which rather,
leaves students’ in dilemma.
Table 3: Philosophical stance for qualitative approach
Phil. Stance for qualitative
approach Percent Valid Percent Cumulative Percent
Valid Interpretivist 26.7 26.7 26.7
Critical Realist 20.0 20.0 46.7
Critical Social
Theorist
6.7 6.7 53.3
Any philsophy 46.7 46.7 100.0
Total 100.0 100.0
Table 4: Philosophical stances for quantitative approach and mixed methods
Phil. stance for
quantitative approach Percent Valid Percent Cumulative Percent
Valid Positivist 13.3 13.3 13.3
Critical Realist 20.0 20.0 33.3
Any philosophy 66.7 66.7 100.0
Total 100.0 100.0
Phil. stance for mixed
methods Percent Valid Percent Cumulative Percent
Valid Critical Realist 20.0 20.0 20.0
Realist 6.7 6.7 26.7
Any philosophy 73.3 73.3 100.0
Total 100.0 100.0
7.2.3 Philosophical debates vs. field and subject of study findings
Although 60% students’ indicate awareness of different philosophies endorsed for their field of study
and subject area, there was minor uniform connection of a single philosophy to field or subject area.
Individuals’ descriptions are moderated by research problem, values, and supervisors, a subsequent
of which is the contrasting descriptions and confusion in understandings of philosophical relation to
research. For example, out of the 33.3% science and engineering students, only 6.7% linked
quantitative to positivism. This is in sharp contrast to the gospel preached by advocates of positivism
(Polit and Beck, 2008; Alvesson and Skoldberg, 2009); who assert that positivism is the philosophical
stance for science and engineering. It further confirms earlier reports by these students that
philosophical stance was less relevance to their research. An arising question is whether
philosophical stance is not drummed much to science and engineering students in comparison to
282

Marcia Mkansi et al.
other faculties? Or the gospel preached by advocates of positivism to science and engineering is not
heard or applied?
7.2.4 Research terminologies debates findings
The majority of students’ 66.7% description of research terminologies seems to be in favour of books
that describe research terminologies similarly to Saunders et al. (2009), as compared to those that
use the terms interchangeable like Newman and Benz (1998). However, a significant minority 33.3%
give a disconnection that is neither right nor wrong subject to references, between research
proponents’ and students, the impact of which is directly related to the proponents’ different
descriptions of same research terminologies. In the final view, 66.7% students’ calls for consensus,
training and clarity that would facilitate diversity between students’ and proponents’ of research, with
33.3% questioning philosophical relevance to their studies.
8. Conclusions and recommendations
PhD students, regardless of their year of study, subject and faculty, held a universal perception of
confusion and difficulty in understanding research philosophical debates and classifications. The fact
that almost every PhD students interviewed showed some form of dilemma towards research
philosophical debates and classification, beckons a need for consensus by philosophical advocates.
The contradictory perception between research students and philosophical advocates symbolize an
alarming disparity, which research institutions and scholars, for the credibility of research, must
consider taking decisive actions. This study has puts forth a thought provoking discussion pertaining
to the impacts of the debates on students’ understanding and choice of approach, and encapsulates
the debates as evidenced in the literature reviewed. However, the onus remains with research
methods and paradigms’ advocates to develop frameworks and devices that would bail research
students from the dilemma, and rather provide systems that help students’ get a grasp of a difficult
field.
References
Agerfalk, P. J. (2010) ‘Getting pragmatic’. European Journal of Information Systems Vol.19, No. 3, pp. 251-256.
Alavi, M. and Carlson, P. (1992) ‘A Review of MIS Research and Disciplinary Development’. Journal of
Management Information Systems, Vol. 8, No. 4, pp. 45-46.
Alvesson, Mats and Willmott, Hugh (1992) Critical Management Studies. Sage Publications Ltd.
Alvesson, M. and Skoldberg, K. (2009) Reflexive Methodology: New Vistas for Qualitative Research. Sage.
Alstyne, Marshall and Brynjolfsson, Erik (2005) ‘Global Village or Cyber Balkans? Modeling and Measuring the
Integration of Electronic Communities’. Management Sciences, Vol. 51, No. 6, pp. 851-868.
Babbie, E. (2007) The practice of social research.Thomson Wadsworth.
Barrett, M. (2010) ‘Exploring critical realism as the theoretical foundation of mixed-method research: evidence
from the economics of IS innovations’. Working Paper Series 3/ 2010. Cambridge.
http://www.jbs.cam.ac.uk/research/working_papers/2010/wp1003.pdf [11 March 2011].
Baskerville, R. and Myers, M. D. (2004) ‘Special Issue on Action Research in Information Systems: Making IS
Research Relevant To Practice-Foreword. MIS Quarterly, Vol. 28, No. 3, pp. 329-335.
Becker, H. S. (1996) The Epistemology of Qualitative Research. The University of Chicago Press.
Bernard, H. R. (2006) Research methods in anthropology: qualitative and quantitative approaches 4th Ed. Alta
Mira Press.
Bryman, A. (2006) ‘Integrating quantitative and qualitative research: how it is done?’. Qualitative Research, Vol.
6, No. 1, pp. 97-113.
Bryman, A. (1984) ‘The debate about Quantitative and Qualitative Research: A question of Method or
Epistemology? The British Journal of Sociology, Vol. 35, No. 1, pp.75-92.
Burrell, G. and Morgan, G. (1979) Sociology Paradigms and Organizational Analysis: Elements of the Sociology
of Corporate Life. London: Heinemann.
Caelli, K.; Ray, L.; Mill, J. (2003) ‘Clear as Mud: Toward Greater Clarity in Generic Qualitative Research’.
International Journal of Qualitative Methods, Vol. 2, No, 2.
Carlsson, B. (1989) Industrial dynamics: technological, Organisational, and structural changes in industries and
firms. Kluwer academic publishers.
Darke, P.; Shanks, G.; Broadbent, M. (1988) ‘successfully completing case study research: combining rigour,
relevance and pragmatism’. Information systems Journal, Vol. 8, No. 4, October, pp. 273-289.
Denscombe, M. (2007) The good research guide: for small-scale social research projects, 3rd Edition. The
McGraw Hills, 2007.
Denzin, N. K. and Lincoln, Y. S. (2005) The Sage Handbook of Qualitative Research3rd Ed. Sage Publications
Inc.
Dobson, P. J. (2002) ‘Critical realism and Information systems research: why bother with philosophy?’.
Information Research, Vol. 7, No. 2, January.
Engel, R. J. and Schutt, R. K. (2005) The practice of research in social work, Volume 1. Sage Publication, Inc.
283

Marcia Mkansi et al.
Gable, G. G. (1994) ‘Integrating case studies and survey research methods: an example in information systems’.
European Journal of Information Systems, Vol. 3, No. 2, pp. 112-126.
Goldkuhl, G. (2008) ‘What Kind of Pragmatism in Information Systems Research?’. AIS SIG Prag Inaugural
meeting, Dec 14, pp. 3-8 2008. Paris.
Goles, T. and Hirschheim, R. (2000) ‘The Paradigm is dead, the Paradigm is dead…long live the paradigm: The
legacy of Burrell and Morgan. The International Journal of Management Science, Vol. 28, pp. 249-268.
Guba, E. G. (1990) The Paradigm Dialog. Sage.
Guba, E. G. and Lincoln, Y. S. (1994) Competing Paradigms in Qualitative Research, Handbook of qualitative
research. Sage.
Guba, E. G. and Lincoln, Y. S. (1989) Fourth Generation Evaluation. Sage.
Hinchey, P. H. (2008) Action Research Primer. Peter Lang Publishing, inc.
Hirschheim, R. (1985) ‘Information System Epistemology: An Historical Perspective’. Proceedings of the IFIP WG
8.2. Working Conference on Research Methods in Information Systems. Armsterdam.
http://areadocenti.eco.unicas.it/virili/TerracinaRW/Kit/HirschheimISEpistemology.pdf [17 August 2011].
Hjorland, B. (1998) ‘Theory and metatheory of Information Science: A new interpretation’. Journal of
Documentation, Vol. 54, No. 5, pp. 606-621.
Johnson, B. R. and Onwuegbuzie, A. J. (2004) ‘A Research Paradigm Whose Time Has Come’. Educational
Research, Vol. 33, No. 7, pp. 14-26.
Johnson, P. and Duberley, J. (2000) Understanding Management Research: an introduction to epistemology.
Sage Publications Ltd.
Klein, H. K. and Hirschheim, R. (1993) ‘The application of Neohumanist principles in information systems
development’. Proceedings of the IFIP WG8.2 Working Group of Information Systems Development.
Armsterdam. http://www.cis.gsu.edu/~dtruex/Klein_Papers/KleinhirschheimIFIP82930517.pdf [11 March
2011].
Khotari, C., R. (2006) Research Methodology: methods and techniques. New Age International (P) Ltd., 2 nd Ed.
McNabb, D. E. (2008) Research methods in public administration and nonprofit management: quantitative and
qualitative approaches 2nd ED. M.E. Sharpe, Inc.
Mingers, J. (2004) ‘Real-izing information systems: critical realism as an underpinning philosophy for information
systems’. Information and Organisation Vol. 14, No. 2, pp. 87-103.
Morgan, G. and Smircich, L. (1980) ‘The case for qualitative research’. The Academy of Management Review,
Vol. 5, No. 4, pp. 491-500.
Neelankavil, J. P. (2007) International Business Research. M.E. Sharpe, inc.
Newman, I. and Benz, C. R. (1998) Qualitative-Quantitative Research Methodology: Exploring the Interactive
Continuum. Board of Trustees, Southern Illinois University.
Ngwenyama, O. K. and Lee, A. S. (1997) ‘Communication Richness in Electronic Mail: Critical Social Theory and
the Contextuality of the Meaning’. MIS Quarterly, Vol. 21, No. 2, pp. 145-167 (1997). Abstract.
http://www.jstor.org/pss/249417 [6 March 2011].
Orlikowski, W. J. and Baroundi, J. J. (1991) ‘Studying Information Technology in Organisations: Research
Approaches and Assumptions’. Information System Research, Vol. 2, No. 1, March, pp. 1-28.
Polit, D. and Beck, C., T. (2008) Nursing Research: Generating and Assessing Evidence for Nursing Practice.
Lippincott Williams and Wilkins.
Ritchie, J. and Lewis, J. (2003) Qualitative research practice: a guide for social science students and
researchers. Sage.
Rowley, J. (2002) ‘Using Case Studies in Research’. Management Research News, Vol. 25, No. 1, pp. 16-27.
Saunders, M.; Lewis, P.; Thornhill, A. (2009) Research Methods for Business Students. Pearson Education
Limited, 5th Ed.
Scott, D. (2007) ‘Resolving the quantitative-qualitative dilemma: a critical realist approach’.International Journal of
Research & Method in Education, Vol. 30, No. 1, pp. 3-17.
Srivastava, T., N. and Rego, S. (2011/ 1958) Business Research Methodology. Tata McGraw-Hill.
Steen, M. and Roberts, T. (2011) The Handbook of Midwifery Research. Wiley-Blackwell.
Steinmetz, G. (2006) ‘Review Forum The politics of Method in the Human Sciences: Positivism and Its
Epistemological others’. Canadian Journal of Sociology Online, pp. 632.
Thayer, H. S. (1981) Meaning and Action: A critical history of pragmatism. Hackett Publishing Company, 2 nd Ed.
Wikgren, M. (2004) ‘Critical realism as a philosophy and social theory in information science?’ Journal of
documentation, Vol. 61, No. 1, pp. 11-22 (2005).
Will, F. L.; Westphal, K. R.; Macintyre, A. (1997) Pragmatism and Realism. Rowman and Littlefield.
Yin, R. (2003) Case Study Research: Design and Methods 2 nd Ed. Sage.
284

The Question of Mixed Methods Suitability to RBV
Research: A Literature Review
Marcia Mkansi, Baomin Qi and Gillian Green
University of Bolton, UK
Mm1lis@bolton.ac.uk
Bq2@bolton.ac.uk
G.green@bolton.ac.uk
Abstract: Resource-based view (RBV) is a theory with inherent methodological challenges resulting in calls for
mixed methods (referred to as plurality). The notion of testability in relation to measuring intangible resources is a
central question in RBV and generates methodological concerns. These methodological concerns led other
scholars to think and suggest that mixed methods may be useful in eradicating the concerns. Although increasing
attention is being given to plurality in RBV research, much of the plurality application and outcomes in RBV
studies is enigmatic. This paper attempts to determine whether plurality is subject to the use of RBV or upon
research problem under study. In doing so, the paper briefly reviews and discusses the methodological
challenges raised previously, and the principles of mixed methods. It further determines if previous studies
encountered the said methodological challenges. And eventually, the basis under which mixed methods was
considered appropriate for RBV studies. A sample of scholarly papers drawn from previous RBV studies is
reviewed. The plurality applicability and suitability to RBV are thus scrutinized in terms of aims of the studies
(RBV dimensions measured), research approach, methodological challenges encountered in measuring RBV
(limitations), and recommendations by the studies (suggestions). The paper concludes with discussions of cases
and suggestions of areas where plurality is particularly well-suited to research using the RBV approach.
Although, the discussions indicate the perspectives of viewing plurality to the RBV studies; its heavy reliance on
literature is a major limitation. Subsequently, assumptions have been made as the authors of previous studies
were not interviewed. However, the study indicates the prominent resourcefulness of literature in identifying
problems and contribution to solutions. The paper recommends that the applicability and outcome of mixed
methods in RBV, in relation to the methodological challenges will be successful as more research advance its
concepts and regularly apply it.
Keywords: mixed methods, RBV, methodological concerns
1. Introduction
In recent years resource-based view (RBV) studies and research have received many calls for
methodological concerns (Priem and Buttler, 2001). Unfortunately, methodological concerns still affect
RBV researchers especially the testability of intangible resources. The proponents of RBV studies
(Hoskisson et al., 1999; Jick, 1979; Barney et al., 2011) have highlighted the need for future
researchers to apply mixed methods to eradicate these concerns and to provide a broader justification
of RBV. Yet, RBV research that has applied mixed methods to eliminate the concerns is enigmatic.
Instead, the majority of studies on RBV used either qualitative (Ellis-Chadwick et al., 2007; Runyan et
al., 2007; Clulow et al., 2003) or quantitative approach (Lockett and Thompson, 2001; Zhuang and
Lederer, 2006; Raduan et al., 2009; Ekeledo and Sivakumar, 2004) other than mixed methods. The
challenges inherent in the scholars’ studies seem to differ from one case to another and are not
always associated with the challenges outlined by previous scholars (see table. 2). Exceptionally,
Barney et al. (2001) used dual approach in his assessment of RBV since 1991; however, vaguely
described the process and shortcomings of the dual approach in that study. Rather drew the multiple
approach recommendations from the study’s findings of the assessment of challenges inherent in
different RBV research. This research seeks to take a step towards a rigorous critique and
contemplates clarification pertaining mixed methods to RBV research by addressing four elemental
questions: Are mixed methods suitable to all RBV research? Is the call for plurality to RBV research
not too broad? Is plurality subject to the use of RBV or upon the research problem and resource
under investigation? Is mixed method approach enough to avoid the methodological challenges
warned of by Barney (2001)?
This research approached the four questions by collecting, reviewing, and scrutinizing a sample of
research articles that applied RBV, discussed methodological concerns, and made calls for plurality.
The different methodological approaches applied in RBV literature were contrasted from a suitability
point of view to identify the relevance or irrelevance of mixed methods and methodological concerns
encountered. RBV literature was further classified into the aim of their study, research approach,
methodological challenges, limitations, and recommendations by the studies in order to analyse the
285

Marcia Mkansi, Baomin Qi and Gillian Green
suitability and relevance. The importance of questioning the suitability of mixed methods to RBV
research have a propounding effect to the progress of RBV, and at best related to Priem and Buttler’s
(2001) description of sociological view status of idea and scientific progress. The broadness analysis
of the plurality calls to RBV might particularly add value in narrowing the calls as some approaches
may be best suited towards measuring some RBV research objectives than mixed methods. To a
certain extent this may cast a light to instances whereby methodological approach may be determined
by the resource and research problem of the study rather than the application of RBV. Hence,
identifying the level and cases at which mixed methods may be particularly suitable to RBV research
methods, may provide a vital platform for future researchers to focus on methods that can reduce the
methodological concerns. Assessing whether mixed method is enough to avoid all the methodological
concerns and challenges, might assist in directing theory building and research; whilst simultaneously
clarifying the potential contribution of mixed methods to RBV research.
2. Resource-based view: Definitions and clarifications
To avoid the sometimes narrow definition of RBV, the paper offers insight and clarification into the
application of the terms, as well as the appropriate definitions. RBV suggests that resources
possessed by a firm are the primary determinants of its performance, and these may contribute to a
sustainable competitive advantage of the firm. RBV views the firm, not the industry, as the source of
competitive advantage (Ekeledo and Sivakumar, 2004); at a given time, it could be defined as
tangible and intangible assets tied semi-permanently to the firm (Caves, 1980 cited in Wernerfelt,
1984). Assets are anything tangible or intangible the firm can utilize in its processes for creating, and
offering its goods and services to the market; whereas, capabilities are continuous patterns of actions
in the utilisation of assets to create, produce and offer products to a market (Wade and Hulland,
2004). Measuring RBV resources (i.e. assets and capabilities) is complex, especially where an
intangible (capability) resource is involved. Subsequently, a number of critics questioned the
testability of these resources and raised several questions and concerns, amongst which have been
discussed on table 1 below.
2.1 The RBV methodological concerns
According to Madhok’s (1997) cited in Ekeledo and Sivakumar (2004), a resource-based approach
has good explanatory abilities. However, despite the strengths of the approach, other scholars (Priem
and Buttler, 2001; Tokuda, 2005) warn of the methodological challenges inherent in measuring
resources (e.g. intangible resources), which generate concerns about the testability of the RBV.
Detailed critiques paid to RBV in relation to methodological concerns are greatly discussed by
previous scholars (Kraaijenbrink et al., 2010; Priem and Buttler, 2001; Tokuda, 2005). A list of key
methodological concerns and description are provided in table 1.
3. The tenets and principles of mixed methods
Mixed method research is a synonym for traditional qualitative and quantitative research (Johnson
and Onwuegbuzie, 2004) or multiple approaches (Bryman, 2006). Previous scholars (Johnson and
Onwuegbuzie, 2004; Saunders et al., 2009) state that mixed methods allows both approaches
(qualitative and quantitative) to be combined, including research techniques, concepts or languages
into a single study. The emphasis is that the mixture of qualitative and quantitative research helps to
interpret and understand the complex reality of a given situation. Thus, offering a logical and practical
alternative that overcomes the weakness and debates of mono-methods studies (qualitative and
quantitative approach). According to Saunders et al. (2009) and Johnson and Onwuegbuzie (2004),
the inquiry process of this approach utilises induction (discovery of patterns), deduction (testing of
theories and hypotheses), and abduction (discovery of set of explanations). This type of inquiry was
given as an example of reducing RBV methodological concerns by Barney et al. (2001). The key
fundamental factor in applying mixed method approach is that the researcher can apply methods that
maximises the best chance of obtaining useful complementary outcome rather than contradictory
outcome.
4. The basis under which mixed methods was considered appropriate for
eradicating RBV concerns
As alluded to by prior RBV scholars (Wernerfelt, 1984; Beard and Sumner, 2004; Helfat and Peteraf,
2003; Fahy, 2000) and other RBV critics (Tokuda, 2005; Priem and Buttler, 2001) measuring RBV
resources (i.e. assets and capabilities) is complex and subject to several methodological concerns.
286

Marcia Mkansi, Baomin Qi and Gillian Green
Table 1: Resource-based view methodological concerns
In a bid to eradicate the concerns, Barney et al. (2001) analysed research findings that used different
approaches in their studies to measure different resources which were intangible in nature. One
approach involved the use of proxies to measure movies studios, which allow for large sample
empirical investigations, but subject to concerns about construct validity (Godfrey and Hill, 1995). The
other one, involved the assessment of observables (referred to as higher-order intangible resource)
that utilised a latent construct detection capabilities of structural equation modelling, and facilitated
hypothesis testing and effect size estimation (Barney et al., 2001). And lastly was the consideration of
ethnography and participant observation’s fertile depictions of organisational phenomena, but not
adept enough at generating empirically robust conclusions of intangible resource described by Rouse
and Daellenbach (1999). The approaches were said to be almost successful in their respective
measurements, but not without some concerns (Barney et al., 2001). It was the realisation of each
approach’s strength and weakness, which led other scholars (Hoskisson et al., 1999; Barney et al.,
2001) to encourage future RBV studies to incorporate multiple approaches. However, this
encouragement may be misleading if consideration is not given to the types of resources that may try
alternative approaches.
5. Methodology design: Assessment of mixed methods suitability to RBV
The literature considered to assess the suitability of mixed methods to RBV were all randomly
selected; peer reviewed, and all published between 2000 to 2009. The rationale behind focusing only
on publications from 2000 was to attempt to provide critical, current knowledge that may have had the
likelihood of being impacted on by the mounting methodological concerns, and or had the opportunity
to react to the concerns as previously encouraged. The bullets points below outline the processes
taken in selecting previous RBV research.
287

Marcia Mkansi, Baomin Qi and Gillian Green
Considerable thought included literature published from 2000 by some RBV representatives
authors (i.e. Barney et al., 2001 and Fahy, 2000) as identified by Priem and Buttler (2001). The
purpose was to gain current insight and recommendations regarding the methodological
concerns.
All of the research papers selected and analysed concerned RBV, though it measured different
resources in different sectors. This was to allow for different perspectives of the methodological
concerns that might have been encountered in different fields or sectors of study.
The plurality applicability and suitability to RBV are thus scrutinized in terms of aims of the studies
(here referred to as RBV dimensions measured), research approach, methodological challenges
encountered in measuring RBV (limitations), and recommendations by the studies (suggestions).
Most of the research papers reviewed were extracted from the journal and academy of
management review, mainly published by ABI/INFORM, Sage, Emerald, MCB University Press,
Elsevier, etc. The publishers appeared to be containing a spectrum of research in RBV.
Table 2: Analyses of previous RBV studies
5.1 Previous RBV studies
The RBV studies reviewed indicate differences from one case to another based on the dimensions
measured and other, unknown factors. The majority of studies on RBV used either qualitative (Ellis-
Chadwick et al., 2007; Runyan et al., 2007; Clulow et al., 2003) or quantitative approach (Lockett and
Thompson, 2001; Zhuang and Lederer, 2006; Raduan et al., 2009; Ekeledo and Sivakumar, 2004)
other than mixed methods. The challenges inherent in the scholars’ studies seem to differ from one
case to another, and not always associated with the challenges outlined earlier above in table 1.
Exceptionally, Barney et al. (2001) used dual approach in his assessment of RBV since 1991;
however, vaguely described the process and shortcomings of the dual approach in that study. Rather
drew the multiple approach recommendations from the study’s findings of the assessment of
challenges inherent in different RBV research. A thorough review indicates the dimensions measured
288

Marcia Mkansi, Baomin Qi and Gillian Green
in terms of RBV and research objectives to be the major dictator towards the research approach and
challenges incurred thereof, other than the use of RBV. For example, Ellis-Chadwick et al. (2007)
measured web based IT as a resource in grocery e-retailing using a qualitative approach and the
challenge incurred was the lack of access to managers than RBV related.
6. Discussions
6.1 Is mixed method approach suitable to all RBV research?
The review of Barney et al.’s (2001) research paper point out clearly that the call for incorporating
multiple approaches was based on the analysis of studies that have applied different approaches that
was to some extent good in measuring intangible resource but still had minor concerns. Without
withstanding the good deal of analysis made by these studies, this study questions the consideration
and identifies a lack of thorough consideration into different types of intangible and tangible resources
that may be best suited by multiple approaches. This study calls for the identification of specific
resources that may be particularly best measured by mixed methods. In certain contexts; the former
will serve as a helpful first step in establishing boundaries for (and contributions of) mixed methods.
This research agrees, in part, with the recommendations and encouragement of mixing research
methods in RBV. But, this may be so much in cases wherein a resource under study may be what
Wernefelt (1984) calls two sides of the same coin, involving several components of assets (tangible
and intangible) that are dependent on each other or capabilities, thereby raising methodological
concerns (i.e. testability). This inevitably is because other approaches may be positioned in
alternative ways and that each of these research approaches can generate important, but different
insights (Bryman, 2006; Becker, 1996). That is, typically, there is no ‘one best way’ of measuring
resource-based view resources, and whatever approach is deemed appropriate by the researcher to
the aim of the problem under study involves emphasizing some insights at the expense of another
approach.
6.2 Is the call for plurality to RBV research not too broad?
The tendency of mixed methods calls towards future RBV research that is all inclusive, might have,
and still may make it more difficult to establish methodological and testable boundaries. Although
there have been differences in the way resources are defined (Barney, 1991), the quotation below
asserts the aspect of inclusiveness better:
“We encourage future scholars to craft studies incorporating multiple approaches
(Barney, 2001; Hoskisson et al., 1999)”.
The former legitimately dogmatise the use of mixed methods to RBV, which rather restricts or
constraints future RBV researchers. Looked differently, this notion that future RBV research can use
mixed methods suggests that other methodological approaches for measuring RBV with certain
categories of resources might be methodologically void. Whilst, other categories of resources might
inherently be easy for researchers to measure and manipulate using a different approach other than
the mixed methods.
6.3 Is plurality subject to the use of RBV or upon the research problem/ resource
under study?
The proponents of RBV (Barney et al., 2011; Barney et al., 2001; Hoskisson et al., 1999) assert that
mixed methods may be useful in eradicating methodological concerns, without placing careful
attention on which resources, where, and how that may be useful. The lack of plurality boundaries
contributes less to future RBV researchers. If plurality is subject to the application of RBV, then
proponents of RBV must be clear regarding the resources and level at which mixed methods can
adequately measure and eradicate the concerns. Those interested in measuring if plurality can
eradicate all RBV methodological concerns can apply mixed methods to their study. However, those
attracted by specific resources of sustainable competitive advantage may adopt the approach that
can best measure the resource and use research methods guides as offered by research methods
scholars (Saunders et al., 2009; Khotari, 2006) to help achieve the goals of their study. This is
because, what makes a good research is not the plurality nor the different types of research
approaches, but rather, the assumptions they share (Bryman, 2006; Becker, 1996). The
methodological concerns from a particular research approach may not be based on the approach per
se, rather by how the method was utilised and the ability of the researcher. Put differently, the
methodological concerns experienced in a particular study may be based on how the research
289

Marcia Mkansi, Baomin Qi and Gillian Green
approach was utilised, timing and application of techniques. This is because, exactly, the same
approach if applied for different purposes or differently by different researchers; it may provide a
distinctive set of methodological concerns and results.
6.4 Is mixed method approach enough to avoid the methodological challenges?
Giving due regard to the importance of choosing the right research approach to address the aim of
the study (Saunders et al., 2009); any of the research approach may probably have a profound
implication to the reduction of methodological concerns. Contextualised differently, different
methodological concerns might entirely arise even if a multiple approach has been applied. This is
consistent with research views by Becker (1996); Bryman (1984). Looked at differently, it could be
that plurality would not have been encouraged had it been previously applied and tested. In this case,
will there ever be an ideal, unified approach that can widely and implicitly be accepted as a proper
measure of intangible resources? Maybe, this can only be answered as more and more research
shares their findings of plurality to measuring RBV resources. On another perspective, an ideal unified
approach to measure RBV may not be all that educative and contributively to literature and scientific
progress. Instead, an understanding that different approaches can be used to measure intangible
resources in several different ways, and that the way the approach is chosen depends mostly on the
resource to be measured as well as the empirical context of application, may be the best suggestion
to future RBV scholars. An alternative approach by RBV researchers may be: first, to identify the
resources to be measured; secondly, decide on research approach that may yield the highest
possible results and minimize the concerns; thirdly, find similar studies that have used the approach;
and lastly, identify any limitations encountered by those studies and integrate a better research
strategy and techniques that can minimise the concerns.
7. Recommendations and conclusions
Given the latter discussions, how best can mix methods (plurality) be positioned in measuring
intangible resources and eradicating RBV methodological concerns? This study disagrees, in part,
with Barney et al.’s (2011) and Hosskisson et al.’s (1999) call for multiple approaches to future RBV
research. Instead, the research suggests several conclusions concerning the use of mixed methods
to future RBV. RBV researchers need clear methodological guide and integrated framework that
recommend resources that may be best measured using mixed methods in pursuit of their research
goals and purposes. This is mainly because; there is research evidence by previous RBV scholars
(Ellis-Chadwick et al., 2007; Runyan et al., 2007; Clulow et al., 2003) that indicate the instrumentality
of mono-methods studies in solitarily producing research outcomes without experiencing any of the
widely outlined methodological concerns warned of in RBV. Not to say that, the studies did not
experience other methodological concerns and even made plurality recommendations. However, their
methodological concerns and suggestions seemed to be the kinds of methodological concerns that
may arise from any study utilising any type of approach. In a nutshell, there are weaknesses and
strengths in each approach and one might be best suited for a particular resource under study. In
support of this view, other scholars (Becker, 1996; Bryman, 1984) stress that neither one of the
research methods is better than the other; instead one might be more useful than the other in a given
study. Therefore, in order to eradicate the said RBV methodological concerns without creating any
other concerns, meaningful prescriptions of methodological approaches and boundaries appropriate
for a set of RBV resources should be offered. Mixed methods may be appropriate for measuring
resources that are mainly two sides of the same coin (intangible depending on tangible resources),
whereas mono-method may be appropriate for measuring the process by which one of the resources
is developed. Although, this research indicates the perspectives of viewing plurality to RBV studies; its
heavy reliance on literature is a major limitation. Subsequently, assumptions have been made
because not all RBV research papers were reviewed, and the authors of previous studies were not
interviewed. However, the study indicates the prominent resourcefulness of literature in identifying
problems and contributions to solutions. The paper recommends future studies to include more
attributes of analysis in similar studies.
References
Barney, J. B. (2001) ‘Resource-based theories of competitive advantage: A ten-year retrospective on the
resource-based view’. Journal of Management, Vol. 27, No. 6, December, pp 643-650.
Barney, J.; Ketchen, Jr. D. J.; Wright, M. (2011) ‘The future of resource-based view theory: Revitalization or
Decline’. Journal of Management, Vol. 37, No. 5, September, pp 1299-1315.
Barney, J.; Wright, M.; Ketchen, Jr. D. J. (2001) ‘The resource-based view of the firm: Ten years after 1991’.
Journal of Management, Vol. 27, No. 6, December, pp 625-641.
290

Marcia Mkansi, Baomin Qi and Gillian Green
Beard, J. W. and Sumner, M. (2004) ‘Seeking strategic advantage in the post-net era: viewing ERP systems from
the resource-based perspective’. Journal of Strategic Information Systems, Vol. 13, No. 2, July, pp 129-150.
Becker, H. S. (1996) The Epistemology of Qualitative Research. The University of Chicago Press.
Bryman, A. (1984) ‘The debate about Quantitative and Qualitative Research: A question of Method or
Epistemology’. The British Journal of Sociology, Vol. 35, No. 1, pp 75-92.
Bryman, A. (2006) ‘Integrating quantitative and qualitative research: how it is done?’. Qualitative Research, Vol.
6, No. 1, pp 97-113.
Castanias, R. P. and Helfat, C. E. (1991) ‘Managerial Resources and Rents’. Journal of Management, Vol. 17,
No. 1, March, pp 155-171.
Clulow, V.; Gerstman, J.; Barry, C. (2003) ‘The resource-based view and sustainable competitive advantage: the
case of a financial service firm’. Journal of European Industrial Training, Vol. 27, No. 5, pp 220-232.
Ekeledo, I. and Sivakumar, K. (2004) ‘International market entry mode strategies of manufacturing firms and
service firms: A resource-based perspective’. International Marketing Review, Vol. 21, No: 1, pp 68-101.
Ellis-Chadwick, F.; Doherty, N. F.; Anastasakis, L. (2007) ‘E-strategy in the UK retail grocery sector: a resourcebased
analysis’. Managing Service Quality, Vol. 17, No. 6, pp 702-727.
Fahy, J. (2000) ‘The resource-based view of the firm: some stumbling-block on the road to understanding
sustainable competitive advantage’. Journal of European Industrial Training, Vol. 24, No. 2/3/4 pp 94-104.
Foss, N. J. (2005) Strategy, Economic, Organization and The Knowledge Economy: The Coordination of Firms
and Resources. Oxford University Press.
Godfrey, P. C. and Hill, C. W. L. (1995) ‘The problem of unobservables in strategic management research’.
Strategic Management Journal, Vol. 16, No. 7, November, pp 519-533.
Helfat, C. E. and Peteraf, M. A. (2003) ‘The Dynamic Resource-Based View: Capability Lifecycles’. Strategic
Management Journal, No. 24, pp 997-1010.
Hoskisson, R. E.; Michael, A. Hitt; Wan, W. P.; Yiu, D. (1999) ‘Theory and research in strategic management:
Swings of a pendulum’. Journal of Management, Vol. 25, No. 3, pp 417-456.
Jick, Todd, D. (1979) ‘Mixing Qualitative and Quantitative Methods: Triangulation in Action’. Administrative
Science Quarterly, Vol. 24, No. 4, pp 602-611.
Johnson, B. R. and Onwuegbuzie, A. J. (2004) ‘A Research Paradigm Whose Time Has Come’. Educational
Research, Vol. 33, No. 7, pp 14-26. Educational Research.
Khotari, C. R. (2006) Research Methodology: methods and techniques. New Age International (P) Ltd., 2nd Ed.
Kraaijenbrink, J.; Spender, J. C.; Groen, A. J. (2010) ‘The resource-based view: A review and assessment of its
critiques’. Journal of Management, Vol. 36, No. 1, January, pp 349-372.
Lockett, A. and Thompson, S. (2001) ‘The resource-based view and economics’. Journal of Management, Vol.
27, No. 6, December, pp 723-754.
Priem, R. L. and Butler, J. E. (2001) ‘Is the Resource-Based “View” A Useful Perspective for Strategic
Management Research’? Academy of Management Review, Vol. 26, No. 1, pp 22-40.
Raduan, C., R.; Jegak, U.; Haslinda, A.; Alimin, I., I. (2009) ‘A Conceptual Framework of the Relationship
between Organizational Resources, Capabilities, Systems, Competitive Advantage and Performance’.
Research Journal of International Studies, No. 12, October, pp 45-58.
Reed, R. and DeFillipi, R. J. (1990) ‘Causal Ambiguity, Barrier to Imitation, and Sustainable Competitive
Advantage’. Academy of Management Review, Vol. 15, No. 1, January, pp 88-102.
Rouse, M. J. and Daellenbach, U. S. (1999) ‘Rethinking research methods for the resource-based perspective:
isolating sources of sustainable competitive advantage’. Strategic Management Journal, Vol. 20, No. 5,
May, pp 487-494.
Runyan, R. C.; Huddleston, P.; Swinney, J. L. (2007) ‘A resource-based view of the small firm: Using a qualitative
approach to uncover small firms resources’. Qualitative Market Research: An International Journal, Vol. 10,
No. 4, pp 390-402.
Saunders, M.; Lewis, P.; Thornhill, A. (2009) Research Methods for Business Students. Pearson Education
Limited, 5 th Ed.
Srivastava, R. K.; Fahey, L.; Christensen, K. H. (2001) ‘The resource-based view and marketing: The role of
market-based assets in gaining competitive advantage’. Journal of Management, Vol. 27, No. 6, December,
pp 777-802.
Tokuda, A. (2005) ‘The Critical Assessment of the Resource Based View of Strategic Management: The Source
of Heterogeneity of the Firm’. Institute of International Relations and Area Studies, Ritsumeikan
International Affairs, Vol.3, pp 125-150.
Wade, M. and Hulland, J. (2004) ‘Review: The Resources-Based View and Information Systems Research:
Review, Extension, and Suggestions for Future Research’. MIS Quarterly, Vol. 28, No. 1, pp 107-142
(2004).
Wernefelt, B. (1984) ‘A Resource Based View of the Firm’. Strategic Management Journal, Vol. 5, N0. 2, pp 171-
180.
Wright, P. M.; Dunford, B. B.; Snell, S. A. (2001) ‘Human Resource and the Resource Based View of the firm’.
Journal of Management, Vol. 27, No. 6, December, pp 701-721.
Zhuang, Y. and Lederer, A. L. (2006) ‘A resource based view of electronic commerce’. Information and
Management, Vol. 43, No. 2, March, pp 251-261.
291

Solving Problems: A Living Theory Approach to Research
Development
Graham Trevor Myers
Durban University of Technology, Durban, South Africa
graham@dut.ac.za
Abstract: This paper gives an explanatory account of my living theory of research methodology in fostering and
sustaining student research in Universities of Technology. Grounded in my research to eliminate the frustrations
and sense of hopelessness of students, I explain how I have developed research methodology processes to help
them develop their research skills so that each may contribute to the development of knowledge in research
methodology and their particular field of interest. Using the concept of praxis and shared praxis (Groom 1991), I
show how I have nurtured communities to become communities of transformational development. I explain how I
have developed a living theory (Whitehead, 2008) of research methodology to encourage growth in such
communities and how my own experiences have contributed to the emerging epistemology of practice. The
importance of my research is grounded in my contribution to the educational advancement of students through
developing their research skills so that they may become co-authors of their own living theories of practice and,
through developing their own community of praxis and shared praxis they may give expression to their research
skills.
Keywords: praxis, shared praxis, research methodology processes, communities of transformational
development, living theories, teaching research methods
1. Prologue
I am a teacher first and a researcher second. Starting this paper with ‘I’ as the central character, this
journey takes me from some place deep in my past where everything was quantitative, to some place
somewhere in the middle, on ‘a’ way to an end. I use the indefinite article rather than the definite
article ‘the’, because I am not sure which path I am going to take. This is just a part of my journey
which I look back at, and reflect on hoping that the places your journey has taken you resonate with
mine, or remind you of the place where you are on your road at present, while perhaps for others it
lets you sit smugly and think of the places that I am still to visit and you have already been. As I
wonder through this paper perhaps you can identify some of the learning stations along the way that
you have had to visit; and you can possibly show where there are other places of learning that should
be visited? It may be your journey that I next read, and broaden the knowledge that I have, and
extend the experiences that I can have.
2. Background
Two things happened to me last year. I became very aware that in 12 months I could retire and
needed to lighten the boxes that I would have to carry with me. I opened up the cupboard that I stored
the student proposals and pilot studies in, and started to count one shelf and multiply it by the number
of shelves that I had. The total was in excess of 2 500. This was the result of my work since 1996,
when the polytechnic I worked for had introduced a research methodology course for all fourth year
students who complete their Bachelor’s degree in Technology before they enter the Masters level of
study.
The second thing that happened was that I was asked by a department other than my own, to run my
research methodology course for them for a supposed ‘20 students’. It was not more than three
weeks and my class had topped 110 students. The course administrator informed me that the
students had heard that there was someone who understood research and they came and registered.
What an honour, but what a lot of work.
I now need to cover a few words and concepts at the outset that many who read this may not have
come across. For those of you, who, like me, are treading gingerly on this Action Research path for
the first time as well, may need some markers in the sand within which to place this paper. Those of
you who are further down the road of discovery, I hope that they bring back some memories, some
even painful as you trudged along alone, because this path can only be traversed alone as you build
that temple which you will present unsullied to the great architect of the universe.
The first of these is the term Praxis (process) which according to the Journal of Praxis in Multicultural
Education is “learning through a dialectical union of reflection and action” (2010:1). I understand it as
292

Graham Trevor Myers
the practising of an idea or method. An un-cited source quotes Aristotle as believing that there were
three basic activities of man. These were “Theoretical” – the goal being determining truth; “Poetical” –
whose goal is producing; and “Practical” – the goal of which is Action (Praxis) or practical knowledge.
Kolb in his 1984 work defines praxis as “informed, committed action”. Freire (1970) takes this a step
further and indicates that it is reflection on the world, in order to transform it.
For the second enlightenment, Jack Whitehead (2008:103) defines a ‘living theory’ as “an explanation
produced by an individual for their educational influence in their own learning, in the learning of others
and in the learning of the social formation in which they live and work”. The question that is usually
posed is “How do I improve my practice?”(Whitehead and McNiff 2006). But in this case I need to
reflect on “How did I improve my practice? To comply with the living theory methodology I need to
determine if what I did, my praxis, can be generalised into my living theory for my future use and
application in praxis.
3. The beginning of research methodology as a subject
A single text book existed at that time in 1996, on research methodology, within my grasp, and from it
a structured course in Research Methodology was devised by the university. But I found that the
students did not understand what I was doing and what they had to do, and hated the subject with a
passion. They were getting good theory but this was not working. I found I had to distance myself
from the traditional or conventional ways of observing, thinking and responding to the subject material
and the students. I found a solution in a phrase I heard somewhere in a speech. It encouraged one as
a researcher to see all others as “village idiots”, or peculiar in the “different sense”, and this change in
focus would allow me to start observing what was going on. I was crossing over into unfamiliar
epistemological, methodological and ontological space. My outcomes were ill defined so the first goal
was to establish clearly what the outcomes should be, and hope that in doing so I would find a way to
guide young researchers along an exciting and constructive path of discovery. I had to find a form of
educative relationship that would help students become research methodologists. I asked myself
“How can I create a space in which young researchers can grow?”
I had no idea what was going to happen but I knew I had to find a way for the students to succeed
happily in what they were doing and possibly prepare them for further levels of research. I started by
challenging the conventional and accepted approaches to the teaching of Research Methodology by
focusing on the areas of possible improvement. I adopted the perspective – that students had innate
potential and were gifted and able to contribute to the process of learning. This became a
transformational evolutionary process for me (McNiff and Whitehead 2002:56). It was by no means
easy. There were times when nothing seemed to be working. I have since found that this is normal, so
I have been comforted, but at the time it was very scary. Schon (1995: 28) speaks of a ‘swampy
lowland’ from which one tries to make observations. From this vantage point, at the bottom with no
real solid ground around, I thought that the frequently messy and confusing problems that seem to be
of great human concern tend to defy technical solution. I soldiered on though, using trial and error,
and just muddling through to rise above the surroundings to see what it was that I was really grappling
with. I found it really difficult to develop a theory out of my practice and make sense of it, and even
more difficult to communicate these ‘understandings’ such as they were, to others. The formation of
the idea is often not intelligible in the written form. Wherever I stepped, the ground seemed uncertain.
Johnson (1995:69) refers to this experience of confusion as being an inhabitant of a “cloud of
unknowing”. I felt that this description was most apt.
So it is that I now try to make sense of the world I have created and try to place this understanding
within the context of my theory and a cohesive frame. As I move through the myriad of observations,
a theory emerges but other concerns surface and clamour for attention. I am the central character in
this study using a narrative method to analyse, after the event, what has happened. I need to know if
the claims that I have made find reinforcement in theory. The manner in which I have worked has,
after this time, led me to a deeper understanding of my practice. I have developed a process of life
affirming and empowering students. Ensuring the participants are informed and involved at all times.
This has led to communication systems being used in the form of e-mails and mailing lists as well as
face book pages and a leather-bound product that launches them on a further stage in their studies.
To link my practice and my research I need to narrate what I have done over this period of time. The
first observation was that students found qualitative methodologies very difficult. Most of them wished
to pursue such a course of action but got lost in the structure – not knowing what they were doing.
293

Graham Trevor Myers
Their presentations were unstructured and showed no coherence. This led me to develop and
implement a very structured framework within which they could operate and is reported in my
previous article (Myers 2010). Seven guiding assignments were constructed which took each student
past certain stations in their development of a research proposal and a pilot study.
The first was an introduction in which the student defined the problem statement for the work they
wished to design for themselves. To this point it had been necessary for students to focus on topics
which they would meet in their field of study. I changed this. They could do their research on any topic
they wanted to. I was here to teach them the methodology. This often takes a while to humanise them
to a place where they understood that they were allowed to design their own ‘syllabus’ for this subject.
I always refer to a number of past proposals in this section as some of them are very original and
interesting. One was about how the law would deal with umthakathi (Witchcraft). Another was virginity
testing amongst Zulu girls as a preventative method for spreading of AIDS. This led to another study
on virginity testing amongst the Tswana boys which then led to an investigation on female
circumcision amongst certain clans in the Tswana culture. Very few people knew about this and the
literature had to be taken orally – which led to great class discussions. I do not allow the student to
carry on until this chapter meets all the rubric requirements that I distribute with the assignment. My
red pen worked overtime in this period and the load was heavy.
Back to the thinking I went. What if I made their peers mark this before it reached me. This was done
the next year and it lightened the load somewhat but not enough, so the idea was taken a step
further. There were so few research methodologists in the environment that it may be an idea to bring
others into this praxis of developing researchers. There were already two years of graduates who had
been through this system and they could do a first marking of this chapter. So it was that the
community of researchers became a reality. I only had to mark once and allocate a mark, most of
them being almost perfect with minor corrections to be done. Students and their mentors started
judging themselves against the criteria.
The student then has permission to go on to the next step, to take the variables or constructs that
they have decided to deal with, to the library. Here they encountered very good service from the
library staff. This chapter had to be marked and corrected firstly by any person they knew, or they
looked for in a local school or library, who had English as their first language with at least a 3 year
qualification with English as a major (Most of these students had English as a second or third
language). This formed the second person in their network of mentors.
The third chapter, once the second chapter was compliant with the rubric and been to me for marking,
covered the research methodology that they chose form the methods that had been taught in the
weekly theory lesson. This went to the research methodologist mentor they had sourced, for rubric
compliance. Once all three chapters were complete they were submitted in a ring binder with the
questionnaire or list of questions they were going to ask, to serve as their proposal.
Only with my consent, after considering the ethical consequences and perhaps calling the student in
to alter the data collection or the sampling so that it was more acceptable, were they allowed entering
the second and third phase, which were the data collection and the analysis. While they were
collecting this in the time before the mid-year June break they were give the theory in analysis with
particular emphasis on graphs and tables, some descriptive statistics and the basic inferential stats
like correlation and chi squared results. I no longer wanted to see this work and they had to rely on a
‘maths teacher’ they knew in the community to suggest corrections. These mentors often confided in
me that they were helping one of my students and were so involved in the exercise and very excited
about it. Some never said a word but their names were in the acknowledgments and I know they were
given a copy by the student when the project was completed to form part of their own portfolio. Just
the other day I met a colleague from the University of South Africa (UNISA) who “had to inform me”
that he met so many of my students in his master’s class that had learnt their methodology in my
classes.
This final copy was then bound, with Chapter 4 being the analysis and Chapter 5 the conclusions,
recommendations and self-criticism. This last chapter is still not well presented and needs work but
what had started out looking like a simple exercise in developing a research methodology course had
revealed itself as a multi-faceted, multi-layered epistemological and ontological challenge. I require
two copies to be submitted, bound in leather as they would their thesis in any other qualification, and
294

Graham Trevor Myers
they gave a copy to each of their three mentors and those to whom their work is dedicated. They are
so proud of this first book they have written.
This process continued for a number of years but then new students were asking to borrow these
projects, and the second one that had been handed in for archive purposes went missing, I kept the
first copy as part of my record of my practice. I then decided to ask for an electronic copy on a disk.
These also started to pile up in the cupboard. But I was able to electronically distribute good projects
on similar topics to those that students wished to research.
My practice had shown a significant shift in emphasis from imparting knowledge to nurturing an
educative relationship between me and the student as well as the student and a network of significant
others whom had been sourced by the student. The praxis was not only mine but in a joint manner
had shifted to a shared praxis (Groom 1991). The theory and the practice had been linked, theory and
action. The students had become their own embodied living theory of practice, and I had improved my
professional practice as an educator.
This had taken a number of years to follow a Kurt Lewin (1946) series or spiral of steps which
consisted of planning, action and fact-finding about the results of my action. Not yet complete, this
living theory (Whitehead 1993:80) is constructed from my explanation of my educational development
where ethical values are embodied in practice. The questions posed by McNiff and Whitehead
(2005:29) that have been answered in the narrative above are as follows:
What was my concern?
Why was I concerned?
What kind of evidence have I produced to show that I was concerned?
What did I do about it?
What will I still do about it and how?
What kind of evidence have I produced to show that what I did was having an educational
influence?
How have I evaluated that influence?
How did I ensure that any judgement I made was reasonably fair and accurate?
How did I modify my practice in the light of my evaluations?
A culture of engagement had been developed based on a student centred democratic value, but this
had to be recreated every year with a new cohort of ontologically imagination brain-dead students.
They had to be taught that there was another way of ‘being’ in the academic world. One which
engendered the image of going on a journey based on a willingness to challenge their most deeply
held beliefs and assumptions. Their reality had to be lived and thus an alternative consciousness of
doing or praxis had to be developed which in turn influenced my praxis. To assist this, the research of
previous students had to become available to new students.
Back to the real world I had to go, and look at how it was that we as researchers conveyed our work
to others. It was through papers like this one; so the requirement was added for each student to
collapse their double spaced stand-alone chapters which followed the ‘classical’ ‘golden thread’ of
opening each chapter with a synopsis of, what occurred in the previous chapter, what would occur in
this chapter, and what would occur in the next chapter, introduction and conclusion, into a flowing
narrative in the form of a publishable paper.
Where to from here? I now have hundreds of publishable papers of pilot studies, with many
observations and lots of insights by students who have been emancipated from uncritical ideologies,
assumptions, biases, prejudices, paradox and ineffective pedagogies. Students who so desperately
want to take this further, but above this level exists a rigid supervisory system of one person’s
ideology being cloned in the work of their students rather than a shared praxis emerging and a
learning community developing where there is a discourse on power, autonomy and collaborative
vulnerability.
Many research supervisors seem to be greatly stressed by the moments of uncertainty, fear,
confusion, self-doubt and emerging novelty that such a ‘new’ system creates but I can assure them
295

Graham Trevor Myers
that the epiphanies that occur in both their own work and those of their students are worth every
moment of uncertainty. Action research does require on-going commitment as it never ends and the
living lifestyle theory that I have developed is fraught with shaky foundation but only once these have
been interrogated can they be identified, examined and corrected. I have had to hold a lens up to my
praxis and constantly look at it from a new perspective as though it were ‘peculiar’, looking beneath
the surface, beyond labels, stereotypes, paradigms, facades, power and control which abounds in the
university.
So it is that I come to the end of this half-way mark in the development of my personal research
horizon. It is for me a sacred place into which I allow my students, and now you, to enter. It is a place
of nurturing, where an ambience of individuality as well as co-existence exists where research may
flourish and develop for the individual as well as foster the transformation of the community in which
they find themselves.
This year I start another group of student on this journey, moving them from one reality across the
threshold to another. In me however there is a restlessness, and an uprooted-ness which leaves me
searching for a new direction with a possible disillusionment, confusion, insecurity, anxiety and
concern (Turner 1969). I hope that the spirit is still strong enough to do this and I thank you for
listening to my story in which I have tried to solve problems in my teaching and developing of
research in my own small neck of the woods at the southern tip of Africa. I have used a shared praxis
model and I have many students who remember me fondly and write about this in their dedication in
their research presentations.
Thank you for listening.
References
Centre for Multicultural Education Table of contents. 2010. Journal of Praxis in Multicultural Education. Vol. 5
Issue 1 Article 3. http://digitalcommons.library.unlv.edu/jpme/vol5/iss1/3 Accessed on 19 February 2012.
Freire, P. 1970. Pedagogy of the Oppresses. Continuum. New York.
Groom, T. 1991. Sharing Faith. Harper Collins. San Francisco.
Johnson, W. 1995. Mystical Theology: The Science of Love. Harper Collins. London.
Kolb, D. 1984 Informal Educational Encyclopaedia. (http://www.infed.org/biblio/b-explrn.htm) as mentioned in
Wikepedia. 2011. The Free Encyclopedia. http://en.wikipedia.org/wiki/praxis_(process) Accessed
2011/12/08
Lewin, K. 1946. Action Research and Minority Problems. Journal of Social Issues 2(4): 34 -46.
McNiff, J. and Whitehead, J. 2005. Action Research for Teachers: A Practical Guide. David Foulton. London.
McNiff, J. and Whitehead, J. 2002. Action Research : Principles and Practice 2 nd ed. Routledge Falmer. London.
Myers, G. 2010. Research Methodology by Numbers, Proceedings of the 9 th European Conference on Research
Methodology in Business and Management, IE Business School, Madrid, Spain.
Schon, D. 1995. Knowing in Action: The New Scholarship Requires a New Epistemology. Change. November –
December: 27- 34.
Turner, V. 1969. The Ritual Process: Structure and Anti-Structure. Aldine de Gruyter. Hawthorne, NY.
Whitehead, J. 2008. Using a living theory methodology in improving practice and generating educational
knowledge in living theories. Educational Journal of Living Theories, Volume 1(1): 103 – 126.
296

A Journey into Analysis in the Development of a World of
Meta-Science
Graham Myers and Verna Yearwood
Durban University of Technology, Durban, South Africa
graham@dut.ac.za
vernan@dut.ac.za
Abstract: The proposal by Mouton of the three world’s model, places knowledge in three distinct worlds. The first
of these is world 1, and in it resides lay-knowledge which tends to be one dimensional. The world 2 knowledge is
based on the interrelationship of constructs or variables. This he also refers to as the world in which the scientific
search for truth takes place. It implies a relationship seeking process which links various pieces of lay-knowledge
together. The world 3 knowledge he labels the meta-science, or “above” science and links the scientific
knowledge into models or theories which are all encompassing. This model discusses the methods of analysis
that may be used in the different worlds. The first of these would be world 1 in which the type of analysis deals
with the one dimensional nature of the data under analysis using single variable graphs and tables, descriptive
statistics, and the conclusions that can be made from such examination. World 2 data is examined in order to
determine the relationship between the constructs or variables under examination. Starting with two such
variables it may be extended as the number of constructs increases and the complexity increases. It uses crosstabulation
and graphical representation in two dimensions as well as inferential statistics. World 3 data would be
that grand attempt to take all the relationships determined in world 2 and start to put these into a comprehensive
whole. The methods that Glass (2000) explains or attempts to examine, begins with the use of ANOVA but ends
with saying that “more useful and more accurate ways of synthesizing research are required.” The unit of analysis
at this level is the project examined in world 2 and the methods of analysis required need further discussion.
Keywords: lay knowledge, scientific knowledge, the world of meta-science, world 1; world 2; world 3, analysis in
research, meta-analysis
1. Introduction
Each one of us as Research Mythologists tries to bring order to the knowledge that we possess,
arranging it in the most logical way to allow us to progress, unhampered by the intricacy of the
philosophy underpinning what we are doing. There is a trilogy of words put forward by Whitehead and
McNiff (2006) which one needs to understand such discussions. They are Epistemology, Ontology,
and Methodology.
Mouton’s definition of epistemology as the study of the nature and origins of knowledge, also called
the “epistemic imperative” or the commitment that scientists make to find the best approximation of
the world (Babbie and Mouton 1998:642). In his earlier description of the world 3, Mouton defines it as
“’Epistemic’, derived from the Greek word ‘episteme’ which means “‘authentic’ or ‘true knowledge’
Ontology, stemming from the Greek word ontos, means the study of reality or ‘being in the reality’
(Mouton 1996:46). Every fact or relationship we uncover or comprehend must have place or position
in our own knowledge as we attempt to approximate the ‘true knowledge’ talked about above.
The third word we need to look at to complete the trilogy is methodology. These three, are examined
in Action Research as taught by Whitehead and McNiff who advocate an enquiry about “How do I
improve my practice?” This examination of epistemological, ontological and methodological
transformation has helped this paper.
Mouton’s three worlds of knowledge are going to be examined to determine how each world interacts
with authentic knowledge (Epistemic) to determine how it is placed in an approximation of the true
knowledge (Ontology) and the methodology of analysing it.
2. The three world model
Mouton (1996) begins explaining his 3 world model by indicating that all people live in multiple worlds
of knowledge. Each of the ‘worlds’ requires a different kind of knowledge. Each one fits what is known
into the world and analysed in a different way. He implies that they are often not necessarily
interlinked. These worlds are the world of everyday life, the world of science and the world of metascience.
In his 1996 presentation he place “everyday life” knowledge in world 1, Science in world 2
and meta-science in world 3. Van der Schyf (2008) has applied this model to the teaching of
297

Graham Myers and Verna Yearwood
Accounting in South Africa to try to explain how the academic tensions within universities can be
remedied. In this paper it will be attempted with Research Methodology.
In 2008 Mouton reverses the order of the worlds with world 1 being meta-science topping the world of
science (world 2) which rests on world 3 (Everyday Life). In the first model he had everything resting
on meta-science. These worlds have differing complexities. We refrain from calling them levels of
complexity because there is nothing that implies they are hierarchical. They are rather different. The
model put forward by Mouton in both texts shows arrows indicating that the flows of information are
from the meta-science to the world of science and on to the world of everyday life. The opposite will
be used in this paper, implying that from the lay knowledge emerge the scientific studies and from the
scientific knowledge emanate the “critical interests” (Mouton 1996:10) in the world of meta-science.
2.1 World 1
World 1 contains the “lay knowledge” which enables citizens to manage their daily operations. This
knowledge is made up of wise sayings, common sense, savvy, ‘how to do’, or what may be termed
pragmatic knowledge (coming from the factual and practical meaning of the word).
It consists of the “social and physical worlds; the symbolic worlds (worlds of signs and symbols) which
are inhabited by different ‘entities’: individual human beings, collectives, social practices,
organisations and institutions.”….“All of this takes place in different temporal and special frameworks”
(Babbie and Mouton 1998:14).
This knowledge is applied to solving problems, to reach consensus and to live a ‘better life’. It is rich
in meaning and personal symbolism and normally uncritically adopted from a variety of sources.
2.2 World 2
World 2 knowledge is created by observing the phenomena in world 1 as peculiar in the atypical
sense. These phenomena can be political, economic, social, physical etc. The systematic and
rigorous examination of these observation, which occur in world 1 are analysed differently rejecting
the authority of people, which world 1 espouses, but using the authority of evidence, driving for the
search for the external truth.
Remembering that the world of science is not homogeneous, each one develops in in its evolutionary
way by falsification (Popper 1972). Thomas Kuhn (1962) purports that there are periods of normal
science where world 1 observation is interpreted within a certain paradigm, while in periods of
scientific revolution there is a shift to another paradigm. Mouton argues that it is the paradigms within
world 3 that direct the study of science in world 2. Thus an explanation of why originally world 1 lay
above the other two worlds.
2.3 World 3
World 3, according to Mouton is the area where critical reflection and assignment occurs. It is where
one reflects on the knowledge in world 2 (scientific world). The scientific community collectively
examines the presentations within the scientific world and places these in congruent, comprehensible
wholes (Epistemology and Ontology). It is here that the ‘scientists’ reflect on what they are doing and
the methodology they will use to advance the various ‘truths’ like Scientific Truth, Religious Truth,
Moral Truth, Historical Truth etc.
The criticism that one has of Mouton’s explanation on world 3 is that it seems to have two distinct
focuses. The first of these is examining how the paradigm affects the scientific advancement, and the
second is the emphasis that the myriad of scientific findings in the world 2 of science can be
combined to form a whole. It would not then be amiss to imply that world 3 is the place where the
independent findings of world 2 are examined in the light of a greater whole. A meta-analysis question
for Mouton (1998:77) would be “What is the current state of research on Research Methodology?” or
“What are the key debates taking place in Research Methodology?” This leads to the authors making
the assumption that it may be that world 3 is the world of models and theories.
A model is defined as, “a representation of a system that is constructed to study some aspect of that
system or the system as a whole” (Cooper and Schindler 2006:712). The same authors on page 47
298

Graham Myers and Verna Yearwood
indicate that “a theory is a set of systematically interrelated concepts, definitions, and propositions
that are advanced to explain and predict phenomena (facts) (ibid 47).
3. Meta-science and meta-analysis
From the above it is evident in one sense that meta-science is what exists ‘above’ science. It could
either be the driving force for an investigation or the result of the scientific investigations and could be
considered as a hegemony or paradigm driven concept.
Turning to the work of Glass (2000:4), he quotes Michael Scriven’s meta-evaluation in which he takes
the meaning of ‘meta’ to be that which stands ‘behind’ or at the ‘back of’, stemming from the
publication of Aristotle who felt that there was something ‘transcendental’ behind physics which bound
it all together, which was called meta-physics. Teasing this discussion out, he warns that it should not
be seen as some overarching general theory of everything but simply a way of examining the analysis
of the analysis.
The development of scientific truth has backed its way into a corner, says Glass (2000). The method
of peer review is assumed to guarantee truth at the expense of all other forms of verification of
‘deviant’ enquiry. The idea is that if you are not doing as I do then you are not discovering knowledge.
The second criticism that Glass (2000) puts into the arena for discussion is the method of distribution
of findings. Money, time and the space available in journals has led to a stylised reporting method
which gives no raw data but only the statistical conclusions which do not allow studies to be
compared. The variables used are often in different measurement scales and defined differently. This
makes it almost impossible for comparison. The analysis of this in world 3 falters at the foundational
level because of a lack of direction in world 2.
Outcomes based education seems to require that all students at a master’s level gather enough data
to be statistically significant, so that they can examine the relationships between at least two variables
and possibly link it in to the literature. But then the method of gaining access to the volumes of
literature is becoming difficult and expensive as more journals try to become self-funding.
Glass (2000:16) says the following: “Meta-analysis was created out of a need to extract useful
information from the cryptic records of inferential data analysis in the abbreviated reports of research
in journals and other printed sources”. The logic seems to be that as the world 1 data is gathered
(pragmatic data in terms of Mouton 1998) it is analysed in terms of a scientific method which provides
information of a different kind. These findings in world 2 (epistemic data) are then to be analysed to
provide the information for world 3 (critical data)
3.1 A new meta-science
The proposition of this paper is that a hierarchical model of research knowledge be examined. World
1 data is gathered in a one dimensional plain. The emphasis tends to be on “position on a linear
plane” when choosing data to be analysed and the methods to be used for analysis, whereas in world
2 one would look at function and relationship. The world 1 analysis would take place where all that is
required is a reporting of the information and those items which are in close proximity would be
examined (distance). Analysis in world 1 requires observation to gauge distance.
World 2 analysis is a relational look at items which are in more than one dimension. The function of
analysis is to determine if there is a relationship between different ‘constructs’ or ‘variables’.
Measurement places it on a two dimensional plane. The ‘time’ dimension could be thought of as the
attempt of the chameleon to establish bifocal observation in world 1.
If meta-science then is, Glass (2000) implies, is the linking of the research in the scientific world
(world 2).
3.2 The origins of meta-analysis
The objective of creating meta-analysis would seem “to make the findings of research in the social
and behavioural sciences more understandable and useful” (Glass 2000:1). The volume of research
is growing and the traditional narrative approach is becoming cumbersome and unwieldy. Something
new is necessary and this paper attempts to start a process of looking at such a discussion. Stand-
299

Graham Myers and Verna Yearwood
alone ‘studies’ are not contributing to world 3 knowledge. Variable or constructs that are dissimilar
need to be defined to allow for comparison.
If world 1 is statistically descriptive and world 2 is predominantly statistically inferential, then what of
world 3. ANOVA, using multiple variables has been used as an analysis tool, but criticised for using
“apples-and-oranges” or dissimilar data in the analysis. Averages and standard deviations loose much
of the detail of the research findings. Putting them through a second analysis to try and plot complex
results from these figures does not enrich the results. We agree with Glass (2000:11) that there is
very little that inferential statistics can do in meta-analysis to create a meta-science. We require an
analysis that can lead us to a more sophisticated understanding of our world.
So it is that we turn to methods of analysis that we use, and try to determine which could be a positive
method to apply to meta-science.
4. Analysis in the 3 world model
We have noticed that analysis is not fully comprehended by students who have to use it. They tend to
present the graphs and the tables and have no idea what can be examined further. Worse still is, they
do the statistics and hope the answer of 1.47 is understood. They need to determine what this means
and in what context it is understood.
To assist in providing a platform for world 3 analysis, Glass (2000:14) has instigated that data should
be stored in a manner so as to be used by any reader. The progress of students through the research
degrees should be revisited and more emphasis should be placed on interpreting the world 2 data
collected so that additions can be made to the models and theories that predominate in world 3.
4.1 Recommendations for World 1
World 1 analysis deals with the one dimensional nature of the data under consideration. The analysis
used for such examination is in the form of single variable graphs and tables, descriptive statistics,
and the conclusions that can be made from such examination. Descriptive statistics reduce data on a
variable to a few possible summarised numbers like a mean, mode or median (central tendency) and
standard deviation, range, and inter quartile range, (spread). The often discarded box plots of periods
past and leaf and stem presentations are far more enlightening than a bar graph or pie graph.
Texts that have been printed on the subject of analysis tend to present only the theory and very little
of the interpretation. This change should be encouraged so that students are more apt at moving from
the data to the conclusions, using presentations that will facilitate analysis in another world.
4.2 Recommendations for World 2
It would seem that it is in this world that most of the changes need to take place. More emphasis
should be placed on grounding the research in the existing research. Literature searches could be
made easier by creating an interdependent system which links all similar studies together rather than
having them scattered over a myriad of sites which are becoming based on commercial principles
rather than the development of knowledge. This should fall within the ambit of those institutions
whose output is research. Such a system should also be guarded against commercialisation so that it
does not suffer the same fate as the internet.
Major and Savin-Baden (2010) have put forward some ideas on examining this information explosion
in Social Research. Their flow diagram is presented below for two reasons. The first is to gloss over
their 8 stage process of qualitative research synthesis which starts to fit in with the tone of this paper.
The first step being to identify studies related to a research question. If one starts to expand on this
when designing a new information system it may be that research questions could be a variable of
search in a literature repository.
Their third stage is the examination of theories and methods used in each study which rings true with
the suggestion that the research should fit in with the notion of “meta-science” as a development of
overarching bodies of knowledge creating a theory or a model.
300

Graham Myers and Verna Yearwood
The second reason for the incorporation of the diagram in Figure 1 is to demonstrate other forms of
presentation. The traditional analysis in World 2 is to examine the data in order to determine the
relationship between the constructs or variables under examination. Starting with two such variables it
may be extended as the number of constructs increases and the complexity grows. The method of
analysis which will be used is cross-tabulation and graphical representation in two dimensions as well
as inferential statistics. As one starts to go beyond this point the ability to physically represent the
process becomes more intricate and abstract.
Identify studies related to a research
question
Compare and analyse findings of each
study
↓ ↓ ↓
Synthesize findings of each study
→→
←←
→→
Provide a series of recommendations ←←
Collate qualitative studies across a large area
of literature
↓ ↓ ↓
Examine the theories and methods used in
each study in-depth
Undertake and interpretation of findings
across the studies
↓ ↓ ↓
Present an interpretative narrative about the
synthesis of findings
Source: Major and Savin-Baden 2010:11
Figure 1: Process of qualitative research synthesis
The usefulness of this analysis would be suspect in assisting the development of a ‘meta-science’ as
such examinations as hypothesis testing lands up rejecting research questions which when
incorporated in a larger study could move such knowledge development forward.
The level of measurement is also problematic in many studies as it is at variance with what the
mainstream has chosen as the level of accuracy of the measurements. It means that comparisons are
difficult.
The positive outcomes that Major and Savin-Baden (2010:11 - 16) found are worthwhile mentioning
here as they help to open up this subject. They feel that qualitative research synthesis can:
Help contain the information explosion,
Help manage the information explosion,
Help address the problem of knowledge fragmentation,
Help to identify gaps and omissions in a given body of research or within a single article,
Provide a different perspective on questions addressed through quantitative approaches,
Provide ways to advance theory,
Spark dialogue and debate,
Add a depth dimension to qualitative studies,
Allow for development of evidence-based practice and policy,
Create cost-effective research.
4.3 Recommendations for world 3
It is within this section that the most speculation has to take place. World 3 data would be that grand
attempt to take all the relationships determined in world 2 and start to put these into a comprehensive
whole. The methods that Glass (2000) attempts to examine, beginning with the use of ANOVA but
ends with saying that “more useful and more accurate ways of synthesizing research are required”
(Glass 2000: 13) The unit of analysis at this level is the project examined in world 2 and the methods
of analysis required need further discussion and possible development.
By looking back at the deficiencies that we have in our research protocols we have to come up with
more accurate and helpful ways of fusing the information from various studies into a whole.
301

Graham Myers and Verna Yearwood
The first point to mention is that “inferential statistics has little role to play in meta-analysis” (Glass
2000:11) and probability theory linking samples to the population are not useful to meta-analysis
(Glass 200:11). Donald Rubin (1990:155) feels that representation of the population is far less
important than other ‘things’. He feels that the description of the conditions under which data is
collected is far more important and standard or ideal study conditions should be delineated.
The style or research reporting has to be tailored to make it more compatible and comparable. The
method of subjecting research to the scrutiny of peers needs to be refined so that not only those
‘good’ papers which meet the epistemology, ontology and methodology of the reviewer are accepted,
but also those ‘bad’ papers are accommodated because these papers often are written by
researchers who stand in the ‘swampy lowlands’ (Schon1995: 28) where the terrain is often ‘messy’,
‘confusing’ and incapable of technical solution but where actual experience is trying to be understood
by a researcher.
Students in their first year of research should not be taught to see themselves as great thinkers
creating new theories but rather as co-workers collecting information within the grand theory to
support or alter it. Their work should be encouraged to add to a study rather than to create new
studies of their own (Glass 2000: 13).
The management of data should facilitate the reporting of all data and not just the selection that is
required to confirm or reject and hypotheses. Glass feels that this retards the development of the
theory and the models. There is too much data and not enough theory.
Archives of data need to be created and standards of compliance on the defined variables need to be
supported so that data is comparable. There are leaders in research in various fields who are trying to
do this, using teams of international researchers, all linked through chat groups. Glass has a uniform
data base to which all contribute. Action research has a Practitioner-Researcher list serve at
practitioner-researcher@jiscmail.ac.uk where researchers around the globe participate and discuss
action research in education. These facilities however are not advertised enough.
When it comes to planning a world 2 study, to facilitate meta-analysis, the results should be presented
in tabular form and “graphical” form so as to give the maximum amount of information possible. The
constructs or variables being examined should be defined in the same manner as acceptable theory
has done. Going back to ‘older’ forms of data presentation (graphs) would not be amiss, things like
leaf and stem plots, placing actual data on graphs and possibly even attaching the raw data in it
summary form.
Analysis in project management (like diagrams) and the use of marketing metrics could be starting
points to start to redefine methods of data presentation in meta-analysis. Although graphs indicate
relationships between studies in meta-analysis they do not form good sources of data when coming
from world 2.
5. Conclusion
This paper has tried to examine meta-science and meta-analysis in an attempt to facilitate discussion
on the flood of “useless” information in word 2 studies which is not adding to the theoretical base of
the subjects. A focus on integrated and controlled research needs to be examined but this requires
some visionary input and time taken to project manage such co-operations.
There needs to be fewer mounds of data being collected and more analysis and comparison done.
Repositories for data which can be used by other researchers need to be created. Champions need to
materialise to create analysis archives and uniform scales of measurement and definitions within the
theory or model.
References
Babbie, E. and Mouton, J. 1998. The Practice of Social Research: South Africa Edition Oxford University Press.
Cape Town
Glass, GV. 2000. Meta-Analysis at 25 www.salus.edu/nc/vi/..../meta-analysis_09.2006.ppt_ Accessed on 5 April
2012.
Glass, GV. 1976. Educational Researcher Nov. 1976 Vol. 5 no 10 3-8 Primary, Secondary, and Meta-Analysis of
Research
302

Graham Myers and Verna Yearwood
Kuhn, TS. 1970. The structure of scientific revolutions 2 nd ed. University of Chicargo Press. Chircargo.
Major, CH. and Savin-Baden, M. 2010. An Introduction to Qualitative Research Synthesis – Managing the
information explosion in social science research. Routledge. London.
Mouton, J. 1996. Understanding Social Research. Van Schaik. Pretoria.
Mouton, J. 1998. How to succeed in your masters and doctoral studies. Van Schaik. Pretoria.
Popper, KR. 1972. Objective knowledge: an evolutionary approach. Clarenden. Oxford.
Rubin, D. 1990. A new perspective, Chap 14 (pp 155-166 in Wachter KW and Straf, ML. Eds). The future of
meta-analysis. Russel Sage Foundation. New York.
Schon, D. 1995 ‘Knowing-in-Action: The New Scholarship required a New Epistemology.’ Change November –
December 27-34.
Van der Schyf, BD. 2008. The essence of a university and scholarly activity in accounting, with reference to
a Department of Accounting at a South African university. Meditari Accountancy Research Vol.16 No.
2008: 1-26.
Whitehead. J, and McNiff, J. 2006. Action Research Living Theory. Sage Publication. London.
303

Researching Resilience as a Psychological Resource
Capacity: A Possible Predictor of Students' Academic
Experience at a South African University
Rita Niemann 1 and Tina Kotze 2
1
School of Higher Education Studies, University of the Free State,
Bloemfontein, South Africa
2
Department of Industrial Psychology, University of the Free State,
Bloemfontein, South Africa
niemannr@ufs.ac.za
kotzet@ufs.ac.za
Abstract: Gaps in research on predictors of successful academic performance necessitate a search for
indicators which may have implications for students in dealing with stressful and challenging situations in their
academic career. This study focused on resilience, as a psychological resource capacity, as a possible tool to
identify at-risk students in higher education institutions. High drop-out rates among first year students at South
African higher education institutions necessitates this type research. Students face numerous challenges to their
well-being and academic performance during their introduction to higher education, which may prevent them from
actualising their full academic potential. It is, therefore, necessary to explore indicators that may predict the
success of students entering the higher education domain. In this study the possible effect of resilience on the
performance of first year students has been researched at a South African university. Findings in this regard
provided an understanding whether students would be able to pursue their educational goals despite several
difficulties, to adapt to situational discontinuities, overcome disadvantaged circumstances, draw on inner
strengths and skills and to maintain their psychological well-being. The sample for this investigation consisted of
789 convenient sampled first year Industrial Psychology students at a South African university. The Adult
Resilience Indicator (ARI), for measuring resilience was used as measuring instrument, consisting of 52 Likert
scale items and measuring eight domains of resilience: confidence and optimism, positive reinterpretation, facing
adversity, determination, support, negative rumination and helplessness, religion and emotional regulation. The
stepwise multiple regression analysis was used to determine whether resilience can be used as a predictor of
students’ academic success. The results showed that three domains of resilience, namely, positive
reinterpretation, facing adversity and religion could be significant predictors of academic success, while
confidence and optimism, support, determination, negative rumination and emotional regulation did not show a
significant relationship to academic performance. Based on the results of this study, it is evident that students'
ability to deal with adversities, see situations in a positive way and their religious strength may be valuable
predictors of their academic success as they seem to assist students in dealing with the challenges and risks
related to university studies.
Keywords: first year students, academic success, resilience, psychological resource capacity
1. Introduction and background
This paper reports on research into the relationship between resilience as psychological resource
capacity and first year students' performance in view of contributing to the search for academic
predictors in higher education.
The high drop-out rate of students at Higher Education Institutions (HEI) is a worrying situation and
was emphasized by the alarming figures in one of the recent OECD reports (2010) on international
comparisons in Higher Education. The report indicated that the USA has the highest non-completion
rate of the countries involved in the comparison. The USA with its 50% drop-out is followed by New
Zealand, Sweden and Mexico. In South Africa a report by the Human Sciences Research Council
(2008) revealed that about 30% of the students enrolled in South African Higher Education institutions
drop out in their first year of study, of which large numbers drop out after the first semester. If these
figures are interpreted in relation to other recent research findings, which indicate that intelligence and
ability are not the only determinants of students’ academic success (Law 2010:196; Pritchard, Wilson
and Yamnitz, 2007; Ross, Cleland and Macleod, 2006 and Moffat, Mc Connachie, Ross and Morrison,
2004), it implies that numerous first year students, face a number of challenges that hamper their
performance and that they do not possess the necessary coping skills. Summers (2003) proposed
that the identification of at-risk students may contribute to curbing student attrition as strategies for
support and development can be instituted in advance.
304

Rita Niemann and Tina Kotze
Gaps in research on predictors of successful academic experience, as indicated by Peterson (2009)
and Basson (2006) necessitate a search for possible indicators predicting how students will deal with
stressful and challenging situations in their academic career (Snyder, Shorey, Cheavens, Pulvers,
Adams III, and Wiklund 2002: 820). Within the South African context, such research is rather crucial
as the South Africa's general graduation rate of 15% is one of the lowest in the world (OECD, 2010)
In searching for possible academic predictors, this paper reports on an investigation that was
conducted among first year students in South Africa in which students' resilience was correlated with
their academic performance. Resilience was used, because it is a vital psychological resource,
capacitating people to cope with challenges. When students at university fail to cope with their
challenges, it may result in negative outcomes, preventing them from actualising their full academic
potential (Vaez and Laflamme, 2008). In this regard Siddique, LaSalle-Ricci, Glass, Arnkoff, and Diaz
(2006:667) state that research relating to performance in challenging situations, warrants an
understanding of psychological resource capacities. It, therefore, seems meaningful to investigate a
psychological capacity such as resilience, to "keep students on track" when pursuing their educational
goals despite several difficulties (Snyder, et al. 2002: 820).
2. Problem and purpose
The drop-out of students in higher education fuels the lack of graduates entering the labour market, a
serious limiting factor in developing countries, such as South Africa. Consequently, universities have
to grapple with questions such as how can students' potential for success be predicted?
A South African developed measuring instrument, the Adult Resilience Indicator (ARI) was used to
determine whether resilience can be used as a predictor of students’ academic success. In
addressing the problem this paper reports on a study that was focused on
making meaning of the concept resilience,
conducting product-moment correlations and multiple stepwise regressions to determine whether
the resilience, or dimensions thereof, significantly contributes to the academic performance of first
year students, and
Exploring the measurement of resilience as a predictor of students' academic success.
In exploring the relationship between the attributes of resilience and students' academic performance,
the following hypothesis and null hypothesis have been formulated:
Hypothesis (H1): There is a significant correlation between some of the attributes of resilience and the
academic success of first year students in Industrial Psychology at the University of the Free State.
Null hypothesis (H0): There is no significant correlation between some of the attributes of resilience
and the academic success of first year students in Industrial Psychology at the University of the Free
State.
In testing the hypotheses it was necessary to have an adequate sample, yielding data with a small
variance and to use an instrument that is relevant to the context of the target group - these aspects
are addressed later on.
3. Resilience: Making meaning of the concept
According to Arehart-Treichel (2005) the essence of resilience is the ability to "rebound from stress"
and to attain good functioning despite difficulty. Similarly, Wolin and Wolin (1993) define resilience as
achievement of some levels of recovery from a traumatic experience Common characteristics of
resilient individuals include task commitment, verbal ability, intelligence, ability to dream, desire to
learn, maturity, internal locus-of-control, risk-taking and self-understanding (Bland, Sowa and
Callahan, 1994) and therefore, resilience have been frequently associated with positive emotions,
especially in difficult times (Philippe, Lecours and Beaulieu-Pelletier, 2009). Research conducted by
Dunn, Iglewicz and Moutier (2008) and Strumpfer (2003) also showed that individuals with higher
levels of resilience are less vulnerable to burnout, which prevents people from delivering quality work.
This may be a valuable asset for students grappling with academic challenges. These definitions all
describe a resilient person as being able to achieve positive outcomes despite challenges.
305

Rita Niemann and Tina Kotze
According to Grotberg (1995), resilience is important because it is the human psychological capacity
which enables one to face, overcome and be strengthened by adversities of life. On the other hand, a
non-resilient person is described as an individual who succumbs to environmental risk (Rigsby, 1994).
Grotberg (1995) holds the view that all people experience stressful situations that leave them
vulnerable, but whether these individuals are crushed or strengthened by adverse situations, depends
on the person's level of resilience. It is only recently that the focus on academic success evolved out
of this tradition where resilience was linked to positive educational outcomes (Rutter, 1979; Morales,
2008:197). Academic resilience research is thus seen as the study of high educational achievement
despite the presence of risk factors that normally portend low academic performance (Morales, 2008:
197), which implies that a student’s capacity to overcome acute or chronic adversities (Martin &
Marsh, 2009:353) can play an important role in his/her performance at HEI's.
Youssef and Luthans (2007:779) state that there are several unique characteristics that distinguish
resilience from other psychological capacities. Resilience is more linked to the need to take both
proactive and reactive measures in the face of adversity. A person's capacity to be resilient promotes
the recognition and acknowledgement of adversities, allowing the affected individual the time, energy
and resource investment to recover, rebound, and return to an equilibrium point. Proactively,
resilience also allows for the use of setbacks as “springboards” or opportunities for growth beyond
that equilibrium point. However, the focus of resilience goes beyond the additive sum of one’s assets
and risk factors, as it incorporates the adaptation processes and mechanisms that combine assets
and risk factors in a cumulative, interactive pattern. Thus, by developing resilience, students may
overcome the stress and uncertainty they are facing during their first year in the higher education
environment (Luthans, Avey, Clapp-Smith and Li, 2008:822).
In the above discussion about resilience, there are clearly two dimensions: process and outcome.
These dimensions are not only limited to the above conceptualisation, but has also been subject to
several discourses as to whether it should be characterised as an outcome or a process (McCubbin,
2001). In this regard Fergus and Zimmerman (2005:399) describes it well by explaining that resilience
refers to the process of overcoming the negative effects of risk exposure, coping successfully with
traumatic experiences, and avoiding the negative trajectories associated with risks in bringing about a
positive outcome or reduce or avoid a negative outcome. In the context of this study resilience theory
is focused on strengths rather than deficits as it refers to the capacity of students to overcome the
negative effects of adversities.
In this paper the term resilience as an umbrella concept, referring to both outcomes and processes,
will be used. Resilience will particularly be used in terms of academic resilience, referring to students
who are academically successful, despite obstacles that prevent the majority of other students with
similar obstacles from succeeding. The fact that it can include both a process and an outcome, makes
research on resilience as a possible predictor of academic success, rather valuable, because it is
particularly during the process phase when supportive interventions by the institution can contribute to
the envisaged successful outcomes of academic performance.
As students fall in the adult category, it was necessary to explore the measurement of adult resilience.
During the past ten years there were some significant attempts on developing instruments for
measuring adult resiliency, such as the Resilience Scale for Adults (Friborg, Hjemdal, Rosenvinge
and Martinusse, 2003) and the Conor-Davidson Resilience Scale (Connor and Davidson, 2003).
In obtaining the data, reported in this paper, the South African developed Adult Resilience Indicator
(ARI) was used. The ARI was developed and standardised by Visser (2007) in two phases. He first
conducted a qualitative investigation to indentify attributes of resilience and then subjected those
attributes to a principal components factor analysis with a varimax rotation (Visser, 2007:89). Then a
second order factor analysis was run to investigate the relationship among the subscales. The eight
factors with the highest factor loadings (eigenvalues) were confidence and optimism, positive
reinterpretation, facing adversity, social support, determination, negative rumination, religion and
emotional regulation. These attributes have also been used in context to resilience in earlier studies.
The following clarifications of each of the attributes, provides insight into how they are related to
resilience:
306

Rita Niemann and Tina Kotze
Confidence and optimism: Resilient people are confident in their own abilities to overcome the
adversity being faced with. It also refers to having an optimistic view of the future, and to expect that
all will end well (Carver and Scheier, 1999).
Positive reinterpretation: Resilient people have the ability to reinterpret situations in a positive way
and to find some meaning in what has happened by learning from difficult experiences (Friedrickson,
et. al. 2003).
Facing adversity: Resilient people are willing to face their adversity and to have the courage to live up
to it – even when it seems to be unpleasant (Fergus and Zimmerman, 2005).
Support: Resilient people make use of social support, which can be present in the form of friends
and/or family to carry them through difficult times (Lazarus, 1999).
Determination: Resilient people carry on doing something, despite opposition or the encountering of
obstacles and to refuse to accept failure (Seligman, 2002).
Negative rumination: Resilient people do not harbour negative feelings about the difficulties that they
face or engage in long periods of self-pity (Yates and Masten, 2004).
Religion: Resilient people engage in some form of religious practice, such as praying and believe that
they will be helped during difficult times (Lazarus, 1999).
Emotional regulation: Resilient people possess the ability to regulate emotions, specifically negative
emotions (Reivich and Shatté, 2002).
Hence the discussion of how the above attributes were explored in terms of students' success and
whether those attributes can possibly be used as indicators of students' academic performance.
4. Research design
The challenge of improving students' academic success has remained an elusive, yet pressing
concern at universities in SA. Given that a degree is the single greatest hope for a country to ascend
to the relative stability of middle class life, emphasises the need for research-based processes in
search of achievement predictors. Research into resilience as one such a predictor is one way of
addressing this need.
4.1 Participants
A sample of 789 first year students in Industrial Psychology at the University of the Free State, South
Africa, was selected by means of convenience sampling. Convenience sample has no controls to
ensure precision, but in this case it was a useful procedure as the entire population of 1800 first year
students in the course were asked to participate, meaning that the final sample consisted of those
who "selected themselves" (Cooper and Schindler, 2011:397). The researchers had to rest with the
sample size as it contained all the students who were prepared to participate. The majority of the
students were female (57%) and between 18 and 20 years old (80,10%). In terms of home language,
the majority of the students (approximately 53%), spoke an African language, while 35.74% had
Afrikaans and 6.97 English as their first language. The majority of the students (56.16%) were from
the Faculty of Economic and Management Sciences while the rest came from the Faculty of the
Humanities (11.28%) and other faculties (7.73%). The constitution of this sample is a representative
snapshot of the students' gender and language composition at this university.
4.2 Instrument
Visser’s (2007; 2009) Adult Resilience Indicator (ARI) was selected to measure the resilience of the
participants, as it was particularly designed to measure resilience in adults by assessing the internal
and external resources that people tend to draw on in times of hardship and adversity within the
South African context. The ARI that was used for this investigation consisted of 52 Likert scale
questions measuring the eight dimensions of resilience discussed earlier in this paper: (i) confidence
and optimism, (ii) positive reinterpretation, (iii) facing adversity, (iv) support (v) determination, (vi)
negative rumination, (vii) religion and (viii) emotional regulation. The ARI questionnaire was
distributed to the 789 students and was not piloted as the ARI is a standardized questionnaire.
307

Rita Niemann and Tina Kotze
The resilience scores of the first year Industrial Psychology students, which were measured by means
of the ARI, were correlated with their academic achievement in the subject at the end of the first
semester. The results of the first semester were chosen, because students tend to start dropping out
towards the end of and after the first semester.
The value of internal consistency was calculated, yielding the following adaquate Cronbach alpha
coefficients: Confidence and optimism (.78); Positive reinterpretation (.85); Face adversity (.69);
Support (.89); Determination (.71); Negative rumination (.81); Religion (.73); Emotional regulation
(.70).
4.3 Ethical measures
Permission for the project and ethical clearance was granted by the Research Committee of the
Faculty of Economic and Management Sciences The aim of the research, as well as the datagathering
instruments that the students had to complete were discussed with students, so that they
could make an informed decision whether they wanted to participate. Students knew that participation
was voluntary and they were ensured of confidentiality. They were also informed that it would be
necessary to provide their student numbers in order to link their scores on the questionnaires to their
academic marks for the first semester, but that anonymity would be otherwise respected. During the
data-gathering sessions of which the dates were provided to them, the researchers were directly
involved in giving the instructions and distributing the questionnaires. The questionnaires were
collected immediately after the students had completed them and they were thanked for their
participation.
4.4 Data analysis
The data analysis for this paper was generated using EViews7 software. For the purpose of the
statistical analysis of the data, multiple stepwise regressions and Pearson product-moment
correlations were run on all eight dimensions of the various independent variables of resilience in
order to determine which dimensions significantly contributed to successful academic performance. A
p-value of at least 0.05 was used as selection criteria in the model building process.
5. Results and discussion
Tables 1 and 2 describe the results of variance and the stepwise multiple regression analysis to
determine which of the dimensions of the independent variable (resilience) explain the highest or
strongest variance in academic performance (dependant variable).
Table 1 shows a variance of about 4% percent, which can be regarded as significant.
Table 1: Variance explained
R Square Adjusted R Square F statistic (p-value)
0.0420 0.0371 8.5988 (

Rita Niemann and Tina Kotze
From the P-values of the Stepwise Multiple Regressions in table 2 in relation to the variance (table 1)
it is evident there were significant relationships between three attributes of resilience and academic
performance. The three attributes that can be regarded as significant predictors of academic
performance are positive reinterpretation (p

Rita Niemann and Tina Kotze
The results of the hypotheses testing as presented in tables 2 and 3 showed that students'
engagement in positive reinterpretation, ability to face adversity and their religious orientation can be
used as indicators for their academic success. Therefore, H0 was not supported as it has been found
that there is a significant correlation between some of the attributes of resilience and the academic
success of first year students in Industrial Psychology at the University of the Free State. Resilient
students' positive reinterpretation shows that they have the ability to make the best of a bad situation
(negative estimates as indicated in tables 2&3). If their studies become a burden or become too
challenging, they have the capacity to try to turn the situation around into something positive and try
to find some meaning from what has happened to them and learn from the situation. Fraser, Kirby,
and Smokowski (2004) and Richman, Bowen and Woolley (2004) refer to these abilities as 'protective
factors' that help an individual to overcome such challenges for the better. Lösel and Bender (2003)
have been clear that resilience is not a single pattern of those protective factors operating, but rather
multiple mechanisms operating synergistically over time to get to positive outcomes. An important part
of a lecturer’s role can be to encourage students to pursue their goals, notwithstanding hardship that
may come with the effort of studying. This can be accomplished through modeling and direct
reinforcement of students’ efforts, so that they don't get discouraged, but to see the context of the
situation and make sense of it in order to enhance their academic performance.
Two dimensions pertaining to resilience showed a positive estimate with academic performance,
namely, the ability of students to face their adversity (p

Rita Niemann and Tina Kotze
Based on the results of this study, as well as previous research such as that of Morales (2008), it
seems that resilience produces benefits to students in terms of academic performance. The predictive
validity of these constructs in terms of academic performance may portend the educational usefulness
of these constructs, especially in terms of the results' consistency with much of the literature.
Academic success seems to start with focusing upon a desired goal and the thoughts about how one
is going to reach the desired goal. These thoughts of positive outcomes, regardless of adversities and
challenges, provide students with mental plans of action that are focused upon overcoming of
possible obstacles that may interfere with achieving academic success. Students’ thinking, therefore,
should stay focused “on task” so that they become less likely to be distracted by self-depreciatory
thinking and counterproductive negative emotions.
It can be rather valuable for Higher Education Institutions to introduce strategies for support and
development by adopting policy on teaching resilient thinking and skills in terms of goal setting and
formulating strategies to students. Such initiatives have the potential to improve students’ goal
pursuits, not only academically, but probably in all areas of their lives resulting in Higher Education
Institutions delivering graduates who will be able to succeed in life and in their jobs. Resilience
interventions will enhance protective factors such as taking responsibility and transforming stress into
resilience. By adding a repertoire of psychological competencies to first year students' academic and
life skills need to be considered when universities plan student preparations courses.
As most South African universities are currently focused on improving student success rates, future
research which extends this project to include second and third year students would add to resilient
theory and literature. However, within the troublesome high drop-out figures of first year students at
South African universities, this study already makes a valuable contribution. Institutional planners in
the higher education sector have to take cognizance of the value of resilience in students and
reinforce resilience-building skills. Research projects like these can strengthen the work of academic
institutions be developing tools to not only predict academic performance, by also to support students
in bridging the school-university divide towards the attainment of academic success.
References
Arehart-Treichel, J. (2005) Resilience shown in youth protects against adult stress, Psychiatric News, Vol 40 No.
17, Retrieved on March 14, 2008 from http://pn.psychiatryonline.org/cgi/content/ full/40/17/14.
Basson, D. (2006) Die verwantskap tussen enkele veranderlikes en akademiese sukses op universiteit.(The
relationship between a number of variables and academic success at university), Unpublished MA-thesis,
Stellenbosch, University of Stellenbosch.
Bland, L.C., Sowa, C.J. and Callahan, C.M. (1994) An overview of resilience in gifted children, Roeper Review,
Vol 17 No. 2, pp 77-80.
Connor, K.M. and Davidson, R.T. (2003) Development of a new resilience scale: The Connor-Davidson resilience
scale (CD-RISC). Depression and Anxiety, Vol 18. pp 76-82.
Cooper, D.R. and Schindler, P.S. (2011) Business Research Methods, Boston, McGraw-Hill International Edition.
Dass-Brailsford, P. (2005) Exploring resiliency: academic achievement among disadvantaged black youth in
South Africa, South African Journal of Psychology, Vol 35, No.3, pp 574-591.
Duggleby, W., Cooper, D. and Penz, K. (2009) Hope, self-efficacy, spiritual well-being and job satisfaction,
Journal of Advanced Nursing, Vol 65, No.11, pp 2376–2385.
Dunn, L.B., Iglewicz, A. and Moutier, C. (2008) A conceptual model of medical student well-being: Promoting
resilience and preventing burnout, Academic Psychiatry, Vol 32, pp 44-53.
Fergus, S. and Zimmerman, M.A. (2005) Adolescent resilience: A framework for understanding healthy
development in the face of risk, Annual Review of Public Health, Vol 26, pp 399-419.
Frazer, M.W., Kirby, L.D. and Smokowski, P.R. (2004) Risk and Resilience in Childhood, In Mark. W. Frazer (Ed),
Risk and Resilience in Childhood, Washington, NASW Press.
Friborg, O., Hjemdal, O,, Rosenvinge, J.H. and Martinussen, M. (2003) A new rating scale for adult resilience:
what are the central protective resources behind healthy adjustments? International Journal of Methods in
Psychiatric Research, Vol 12 No 2, pp 65-76.
Friedrickson, B.L., Tugade, M.M., Waugh, C.E. and Larkin, G.R. (2003) What good are positive emotions in
crises? A prospective study of resilience and emotions following the terrorist attacks on the United States on
September 11 th , 2001. Journal of Personality and Social Psychology, Vol 84, pp 365-376.
Grotberg, G. (1995) A guide to promoting resilience in children: strengthening the human spirit, Early Childhood
Development Practice and Reflection Series, Retrieved 12 December 2011 from
http://resilnet.uiuc.edu/library/grotb95b.html.
Human Sciences Research Council (HSRC) (2008) High university drop-out rates: a threat to South Africa's
future (Policy Brief), Pretoria, HSRC
Law, D.W. (2010) A measure of burnout for business students, Journal of Education for Business. Vol 85, pp
195-202.
311

Rita Niemann and Tina Kotze
Lazarus, R.S. (1999) Stress and Emotion: A new synthesis, New York, Springer.
Lösel, F. and Bender, D. (2003) Protective Factors in Resilience. In D.P. Farrington & J. Coid, Prevention of
adult antisocial behaviour, Cambridge: Cambridge University Press.
Luthans, F., Avey, J.B., Clapp-Smith, R. and Li, W. (2008) More evidence on the value of Chinese workers’
psychological capital: A potentially unlimited competitive resource? International Journal of Human
Resource Management, Vol 19 No.5, pp 818 – 827.
Martin, A.J. and Marsh, H.W. (2009) Academic resilience and academic buoyancy: multidimensional and
hierarchical conceptual framing of causes, correlates and cognate constructs. Oxford Review of Education,
Vol 35 No. 3, pp 353-370.
McCubbin, H., Thompson, A. and McCubbin, M. (2001) Family measures: Stress, coping and resiliency
inventories for research and practice, Honolulu HI, Kamehameha Schools.
Merdinger, J.M., Hines, A.M., Osterling, K.L. and Wyatt,P. (2005) Pathways to College for Former Foster Youth:
Understanding Factors that Contribute to Educational Success, Child Welfare. Vol 84 No. 6, pp 867-896..
Moffat, K.J., McConnachie, A., Ross, S. and Morrison, J.M. (2004) First year medical student stress and coping
in a problem-based learning medical curriculum, Medical Education, Vol 38, pp 482 – 491.
Morales, E.E. (2008) Academic resilience in retrospect: Following up a decade later, Journal of Hispanic Higher
Education, Vol 7, No.3, pp 228 – 248.
OECD (Organisation for Economic Co-operation and Development), (2010) International comparisons in higher
education. Paris: OECD.
Parker, M., Lee Roff, L, Klemmack. D.L., Koenig, H.G., Baker, P. and Allman, R.M. (2003) Religiosity and mental
health in southern, community-dwelling older adults, Aging and Mental Health, Vol 7, No.5, pp 390-397.
Peterson, V.M., (2009) Predictors of academic success in first semester Baccalaureate nursing students, Social
Behavior and Personality, Vol 34, No. 3, pp 411-418.
Pritchard, M.E., Wilson, G.S. and Yamnitz, B. (2007) What predicts adjustment among college students? A
longitudinal panel study. Journal of American College Health, Vol 56, pp 15-21.
Philippe, F.L., Lecours, S. and Beaulieu-Pelletier, G. (2009) Resilience and Positive Emotions: Examining the
Role of Emotional Memories, Journal of Personality, Vol 77, No. 1, pp 139-176.
Reivich, K. and Shatté, A. (2002) The resilience factor. New York: Broadway Books.
Rigsby, L.C. (1994) The Americanization of resilience: deconstructing research practice, In M.C. Wang & E.W.
Gordon (eds), Educational resilience in inner-city America: Challenges and Prospects, Hillsdale, N.J.:
Lawrence Erlbaum Associates, Inc. pp 85-94.
Ross, S., Cleland, J. & Macleod, M.J. (2006) Stress, debt and undergraduate medical student performance,
Medical Education, Vol 40, No. 6, pp 584 – 589.
Richman, J.M., Bowen, G.L. and Woolley, M.E. (2004) School Failure: An Eco-Interactional Development
Perspective. In Mark. W. Frazer (Ed) Risk and Resilience in Childhood. Washington: NASW Press.
Rutter, M. (1979) Protective factors in children’s responses to stress and disadvantage. In M.W. Kent and J.E.
Rolf (Eds), Primary prevention of psychopathology, Vol 3: Social Competence in children. Hanover, N.H.
University Press of New England.
Seligman, E.P. (2002) Positive psychology, positive prevention, and positive therapy. In C.R. Snyder and S.J.
Lopez (Eds.) Handbook of positive psychology. Oxford: Oxford University Press.
Siddique, H.I., LaSalle-Ricci, H.V., Glass, C.R., Arnkoff, D.B and Diaz, R.J. (2006) Worry, optimism and
expectations as predictors of anxiety and performance in the first year of law school, Cognitive Theory and
Research, Vol 30, pp 667-676.
Snyder, C.R. (1994) The Psychology of Hope: You can get there from here, New York, The Free Press.
Snyder, C.R., Shorey, H.L., Cheavens, J., Pulvers, K.M., V.H. Adams III and Wiklund, C. (2002) Hope and
Academic Success in College, Journal of Educational Psychology, Vol 94, No. 4, pp 820-826.
Strumpfer, D.J.W. (2003) Resilience and burnout: a stitch that could save nine, South African Journal of
Psychology, Vol 33, No. 2, pp 69-79.
Summers, M. (2003) Attrition research at community colleges, Community College Review, Vol 30, No. 4, pp 64-
84.
Vaez, M. and Laflamme, L. (2008) Experienced stress, psychological symptoms, self-rated health and academic
achievement: A longitudinal study of Swedish university students, Social Behaviour and Personality, Vol 36,
pp 183–196.
Visser, W.A. (2007) Daily hassles, resilience and burnout of call centre staff. (Unpublished Philosophiae Doctor
thesis) North West University, Potchefstroom, South Africa.
Visser, W.A. (2009) The development of the Adult Resilience Indicators. Paper read at the Wellness in the
Workplace Conference, 21 – 22 September, 2009. Ilanga Estate, Bloemfontein, South Africa.
Wolin, J. and Wolin, S. (1993) Resilient Self: How survivors of troubled families rise above adversity., New York,
Random House. Inc.
Yates, T.M. and Masten, A.S. (2004) Fostering the future: resilience theory and the practice of positive
psychology. In P.A. Linley and S. Joseph (Eds.) Positive psychology in practice. Englewood Cliffs (NJ):
John Wiley & Sons.
Youssef, C.M. and Luthans, F. (2007) Positive organizational behaviour in the workplace: The impact of hope,
optimism and resilience, Journal of Management, Vol 33, pp 774-800.
312

Cross-Cultural Marketing Research: Neuromarketing and
In-Depth Interview
Maia Ozdemir
Tomas Bata University, Zlin, Czech Republic
ozdemir@fame.utb.cz
maya.ozdemir@yahoo.com.tr
Abstract: What the customers really feel? This question is critical for formulating competitive marketing
strategies. Significance of marketing is out of any doubts. Companies worldwide spend trillions of dollars each
year on advertising, marketing, and product R&D. For decades advertisers, marketers, and product developers
are making an effort to apply social psychology to discover the tools to tap into the consumer’s brain. Some
marketers believe that most of biases occur because of responses changed by the customer’s conscious mind.
Development of technology gives a wide range of opportunities for the new generation marketing research. New
point of view most times gives an advantage in the marketing field. It helps to develop creative strategies. For
decades, marketing researchers tried to find out an excellent answer using various methods such as focus
groups, surveys and so on; sometimes asking consumers directly and sometimes applying an indirect approach,
such as choice-based conjoint. But, neither method is foolproof. In the frame of this paper we will discuss some
weak points of neuromarketing by demonstrating results of our research and give suggestions toward gaining
more precise data. The primary data from Turkey, Czech Republic and Russia were collected using in-depth
interviews. Interviews were conducted among 17 Turkish, 25 Czech and 7 Russian, using simple random
sampling technique. For the purpose of our research we have used Conceptual analysis of the data gained. In
result, we have identified common for each ethnic group patterns. They are Cleanness for Turkey, Nature for
Czech Republic and Heat for Russia. Applying suggested data-gaining method allows researches to have more
creative point of view toward customers’ real feelings.
Keywords: culture, cultural genesis, evolution, in-depth interview, marketing research, neuromarketing
1. Introduction
Culture is very complicated and important factor in human’s life. It influences human’s behavior more
then anything else in the world. It roots in the mists of time when our ancestors struggled with nature
for their lives; and stays in the field of unconscious. Culture consists of many elements, and it’s very
hard to mesure it. That’s why culture is still the “elephant in the room” for academic researchers
(Hollis 2011).
On the other hand, marketers developed strategies like sensitive translations, standardized
questionnaires and analitic routines. Inspite of all these innovations, bias in cultural research area is
still a very important barrier. According to Nigel Hollis, just 1% of 70,000 ads around the world
“actually performed below average when tested in another (cultural) market”. So, culture can be
measured just by more creative methods.
According to Clark, there are two ways of research on residential search and choice. One of them is
basic model of “search” provided by marketing and economics; another is psychological model
focused on choosing among alternatives (Clark 1993).
An attempt of interfacing and comparing different approaches was made by Garling and Golledge in
the book “Behavior and Environment: Psychological and Geographical Approaches”. There are
significant differences in understanding of consumer behavior in different scientific disciplines. They
influence research methodology and analysis.
The behavior-environment interface could be studied from two angles. In some disciplines, such as
geography and economics, people’s behavior in a given environment tends to be seen from the
outside. By contrast, in psychological research decision makers are viewed from the inside. The
research is focused on the decision maker’s goals or values in a given environment (Montgomery
1993).
It is clear that researches studied from angle of non-psychological disciplines such as economics and
marketing are not enough by themselves to predict consumer decision making process as well as
psychological researches by themselves are not enough to do so too.
313

Maia Ozdemir
Development of technology gives a wide range of opportunities for the new generation marketing
research.
The latest technologic discovery in modern marketing research is EEG scanner. Its developer Dr.
Pradeep at the 75th Advertising Research Foundation conference explains: “The skullcap-size device
sports dozens of sensors that rest on a subject's head like a crown of thorns. It covers the entire area
of the brain, so it can comprehensively capture synaptic waves.” Consumer-research firm
NeuroFocus, managed by Dr. Pradeep, “promises something deeper, with unprecedented access into
the nooks and crannies of the subconscious” (Penenberg 2011).
New point of view most times gives an advantage in the marketing field. It helps to develop creative
strategies. However, neuromarketing has a few points that contradict to results of our research. In this
article we will show these weaknesses by presenting our research and make some suggestions
toward future direction of modern marketing research.
2. Problem formulation
Neuromarketing is a quite new method of marketing research. The first attempts to apply
neuroscience to other fields were taken in the middle of the 20th century by the neurologist Paul
MacLean. He has proposed that we have not one brain, but three. It is interesting that similar ideas
can be found in many esoteric spiritual traditions like Kabbalah and Platonism, but MacLean made an
effort to bond three planes of consciousness to physical parts of the brain (U.S National Library of
Medicine 2011). This idea was not wide accepted in the medical field but psychologists are still using
it. Since that time neuroscience and technology developed very quickly, and the appearance of
neuromarketing seems to be very natural. Without any doubts, neuromarketing is a necessary
application for gaining more precise data in marketing research. However, this approach is not
foolproof too.
One of well-known neuromarketing companies is science based consumer-research firm NeuroFocus.
According to NeuroFocus CEO Dr. Pradeep, people’s brains are similar with “some differences
between male and female brains” (Penenberg 2011). However, our research shows that human
brains are “remarkably alike” just within some cultural groups. Differences between groups are still
very important.
The probably cause of Dr. Pradeep’s wrong assumption about the similarity of brains may be
explained by Ian Addie in his article Is neuroscience facilitating a new era of the hidden persuader? :
“By nature of the electrodes being placed on the scalp, EEG can only effectively measure brain
activity at the surface of the brain, while we know that a lot of mental processes, including some of
those responsible for controlling emotion and memory formation, occur deep within the brain and
outside of the reach of EEG. So the current techniques used in neuromarketing are far from ideal for
their purpose. However, even if these techniques were not without their limitations, it is necessary to
have an accurate and comprehensive understanding of brain function if the results are to be correctly
interpreted and completely understood – and in this regard we are also currently lacking” (Addie
2011).
3. Problem solution
Cause of the differences between groups may be based on evolutionary domain. The brains started
to differ in the period when according to the most widely accepted “Out of Africa” Hypothesis; Homo
erectus was spreading across the world. At that time human groups moved into different geography
regions with different characteristics that had huge influence over future development of culture.
There are a number of evidences for impact of geography characteristics on the cultural genesis.
Emergence of science in ancient Egypt was connected to the need for development of irrigation
system for Agriculture, building pyramids, mummification of bodies and so on in the desert climate.
Also, the most widely accepted myth about the origin of man that describes the creation of man by the
gods from the most common material in the area; in different cultures man is created from clay, soil,
wood, coconut, bones of animals or of fish, etc (Ishakova 2003).
The physical differentiation of brain’s parts at that time may be explained in terms of neuroplasticity.
This so deep physical differentiation was an absolutely necessary for survival process of adaptation.
Now it is an inherited base of our brain that is similar just within the certain culture group members.
314

Maia Ozdemir
The idea of transmission of cultural events to later generations is not new. Darwin did believe that
learned in one generation characteristics could be inherited in later generations. According to him, the
process of inheritance was working by transmitting particles, called ‘gemmules’, produced throughout
the body to offspring in the sex cells. Darwin claimed that gemmules hold characteristics specific to
the body part that produces them (Darwin 1877/2004).
According to Stanford Encyclopedia of Philosophy, cultural evolutionists agree that at the level of the
population, cumulative evolution requires that fitness enhancing cultural traits be preserved in the
offspring generation (Stanford Encyclopedia of Philosophy 2011).
The similarity of brains within the culture group was proved either by our following research.
3.1 Methodology of research
3.1.1 Aim of research
The main Aim of our research is extracting from all answers common for each group patterns that
influence decision about buying a new housing.
3.1.2 Sample of research
The research was conducted in Turkey, Czech Republic and Russia. Samples of research are three
ethnic groups: Turkish, Czech and Russian.
3.1.3 Instrument and procedures
Instrument of our research is in-depth interview because of its flexibility and providing more detailed,
unconscious and thus more trustful information.
In order to reach the Aim of research, we conducted in-depth interviews with 17 Turkish, 25 Czech
and 7 Russians using simple random sampling techniques. Number of the interviews depended on
ability to find out some strong common pattern.
The first set of interviews was conducted in Bursa/Turkey. Assistants were selected from Senior
Psychology students in Uludag University.
The second set of interviews was conducted in Zlin/Czech Republic. Assistants were selected from
Senior Management and Economics students in University of Tomas Bata in Zlin.
The third set of interviews was conducted in Moscow/Russia.
In order to increase the reliability of results, the following unified interview protocol was developed for
all culture groups.
Questions for the interview:
The ideal home (what home the participant want to live in? more details about it (plan, odors,
colors))
Childhood memories (the strongest something s/he remember first, may be positive or negative,
some experience about home)
More details about the childhood home (plan, odors, colors)
What makes one place a home?
Guidelines:
a voice recorder will be used during the interview
Main points, participant’s age, sex, education level and occupation will be written after the
interview in MS Word.
During the interview, the most attention must be paid to the following:
315

Maia Ozdemir
something necessary for survival that might be developed in past generations as a product of an
adaptation;
Early childhood memories.
3.1.4 Ethical considerations
At the first part of each interview an interviewer gave all necessary information about research aims,
anonymity and security of all private data.
3.1.5 Data analysis
For the purpose of our research we have used Conceptual analysis of the data gained.
The first of all, participants’ answers were reduced to phrases relative to the Home and a necessity for
survival. Then the phrases were distributed by subcategories that later were combined into
categories. These categories are some common patterns for each of the target groups. Than every
interview was scanned for existence or absence of the pattern found. Concepts were coded even if
they have appeared in different forms but meant similar enough. The coding was conducted manually
writing down concept occurrences. Other irrelevant information was ignored.
3.2 Results of research
The strongest common pattern for Turkish group is Cleanliness (Table 1). 15 of 17 participants used it
or it’s synonyms in their answers.
The strongest common pattern for Czech group is Nature (Table 1). 22 of 25 participants used it or it’s
synonyms in their answers.
The strongest common pattern for Russian group is Heat (Table 1). 6 of 7 participants used it or it’s
synonyms in their answers (Ozdemir 2011).
3.3 Discussing of the results
For understanding the results of our research we must look at common in target populations myths
and points of view about Home. As we can see, there is an influence of the environment over results
of interviews.
Cleanliness in the hot climate is necessary for surviving of the population, especially a nomadic
population. That’s why cleanness takes a critical place in Muslim religion.
Wood and nature is the main point in the culture of Vikings that had spread in the North and the
middle part of Europe. Their first houses were built from parts of their ships. Wooden ships were their
universes. Also, holy wooden carvings took a very important place in the Vikings’ world (Marsadolova
2009).
The center of the house in the cold climate is an oven. Without any doubts, the heat is absolutely
necessary for surviving in this geography region. So, the oven was a starting point of the house
(Marsadolova 2009). It is interesting that in opposite to cleaning traditions in the hot climate, in Russia
it was a bad luck to clean something including own body during the holidays.
Neuroplasticity in childhood is much stronger than in adulthood. And newborn’s brain more likely
makes stronger new connections in accordance with the inherited scheme. So, a question about
younger generations may be raised. Are they in or out of the common patterns found? In the
beginning of the research we were skeptic about using young respondents because of existence of
different youth subcultures. But we had a few teenager respondents who were in the patterns similar
with older ones. This finding proves our hypothesis about the nature of inherited cultural matrix inside
the brain. Individual experiences like youth subcultures create just superficial changes in the brain.
Lack of more younger respondents doesn’t allow us to make a comparison with more younger
generations like children, toddlers and infants.
316

4. Conclusions
Maia Ozdemir
Without any doubts, neuromarketing is one of the best and most technologic research methods; but it
has some weaknesses. Even if increasing the sensitivity of EEG may solve certain problems, the
brain complexity and limitation of our understanding of brain shouldn’t be forgotten. Our research
results could never be found by neuromarketers without applying in-depth interview method.
Neuromarketing, like in-depth interview, allows reaching unconscious parts of the brain, however,
main patterns should be found by in-depth interviews. On the other hand, neurofeedback can prove
the results of the in-depth interviews. For example, in our case respondents from different culture
groups may have the similar test with just pictures of different houses. Absence of words and their
translations increases the reliability and trustfulness of the results.
In conclusion, it is clear, that neuromarketing and in-depth interviews may complete each other and
draw together more accurate picture of what the customers really feel.
Table 1: Allocation of categories and subcategories from the responses
Category Subcategory Phrases
32 years old woman: “…light and white walls are just what I want, light
Light and white give me a feeling of cleanness.”
Cleanliness
Nature
Heat
References
Light colors
30 years old woman: “…purity and cleanness relax me, dark colors
strangle me, I prefer white.”
45 years old man: “…white gives a feeling of space.”
White 30 years old woman: “…white sofa and white ground, because it
relaxes me.”
46 years old woman: “My ideal home should be a little house with
Garden a huge garden. I used to play with my friends in the wood or in the
nature.”
38 years old man: “I will like to have a home that is close a river with a
Green lot of greenery, hills and mountains. I like nature and therefore I
believe having a home close to nature will be perfect for me.”
42 years old woman: “The first thing I remember is our beautiful
Flowers, garden garden. My mother loves flowers so she had the most beautiful
garden from all of neighbors.”
56 years old man: “At the front yard there are a lot of flowers. Now in
winter you can’t see their beauty, but when you see them in spring, it
is a symphony of colors that are playing there.”
67 years old woman: “My garden is where I go to not hide but shelter
Garden as a myself from my responsibilities and the environment. A garden is
shelter completely natural, made by my hands, and contains the raw
emotions of frustration, enjoyment, and serenity.”
Warm house 57 years old woman: “It was a little warm house.”
47 years old woman: “Winter you must be more closed so I prefer the
House (warm) apartment; there was a small window and when I had turned off lights
versus winter I watched snowing and fell asleep… winter you are always inside.”
80 years old woman: “...some of us were sleeping on the oven; it was
Warm oven very nice and warm. Most of all I loved to sleep on the oven. It was
very warm there.”
60 years old man: “Outer entrance hall is used to keep the house
warm. Imagine a Russian oven; wooden loft (a bunk) was built at a
same height at a 1-meter distance from it. It was high because it is
warmer there. So, my first memory is jumping from oven to loft and
back with my friends.”
Addie I, Is neuroscience facilitating a new era of the hidden persuader? International Journal of Market Research,
Vol. 53, Issue 3, pp. 303-305.
Clark W.A.V., Search and Choice in Urban Housing Markets. Behavior and Environment: Psychological and
Geographical Approaches. Editors: Garling T and Golledge R.G., Elsevier Science Publishers B.V., 1993.
Darwin, C., The Descent of Man, Second Edition, with an introduction by Adrian Desmond and James Moore,
London: Penguin, 1877/2004.
Holls N, Globalization in context, Journal of Advertising Research, March 2011 Supplement, pp. 37-40.
317

Maia Ozdemir
Ishakova, Исхакова О.А., Зарождение и становление культуры. Культурогенез. Культура древних
цивилизаций. Московский автомобильно-дорожный институт, 2003 (author’s translation).
Marsadolova, Марсадолова Т.Л., Воплощение модели мира в традиционных жилищах славян и
скандинавов, Культурогенез и культурное наследие. Сборник научных трудов, Санкт-Петербург,
Астерион, 2009 (author’s translation).
Montgomery H., The Choice of a Home Seen From the Inside. Behavior and Environment: Psychological and
Geographical Approaches. Editors: Garling T and Golledge R.G., Elsevier Science Publishers B.V., 1993.
Ozdemir M, Marketing management in construction industry: cultural differences in consumers´ housing choice,
Problems of Management in the 21st Century, Vol.1, 2011, ISSN: 2029-6932.
Penenberg A, They have hached your brain, Fastcompany.com, September 2011, pp. 85-125.
Stanford Encyclopedia of Philosophy, [online], Website: http://plato.stanford.edu/entries/evolution-cultural/
(accssesed on 27.10.2011).
U.S National Library of Medicine, [online], Website: http://oculus.nlm.nih.gov/cgi/f/findaid/findaididx?c=nlmfindaid;cc=nlmfindaid;view=reslist;subview=standard;didno=maclean534
(accssesed on
27.10.2011).
318

Utilising Assessment Tasks to Support the Research of
MBA Students
Noel Pearse
Rhodes Business School, Rhodes University, Grahamstown, South Africa
N.Pearse@ru.ac.za
Abstract: For a variety of reasons related to the nature of the degree and its delivery, as well as the
characteristics of the students enrolling, the research component of MBA degrees has proven to be a significant
challenge for many students. Based on the literature, it can be argued that well designed formative and
summative assessment tasks can provide the scaffolding that students require to progress with their research.
Engaging in reflective practise and focusing on the research component of the MBA degree offered at the
Rhodes Business School over a period of ten years, the goals of this paper are to (1) describe the use of
assessment tasks, and (2) to analyse the effects of these tasks on the ability of students to complete their
research, as well as (3) the effect they have had on the supervision process. Data was collected from
documentation and by purposively gathering responses to specific questions through interviews and email
correspondence with supervisors, and past and current students. A theoretically driven thematic coding process
guided the collection of data and its analysis. In addition content analysis was used to describe the trend in the
use of criterion-referenced assessment. The findings show that an increasing number of assessment tasks were
incorporated into the module over the ten year period. These tasks assisted the student and supervisor in
structuring the progression of the research. It was concluded that overall the inclusion of these assessment tasks
was beneficial to both students and their supervisors, but that more dialogue was needed to clarify what was
expected of supervisors and students when supervising and undertaking these tasks, respectively. However, in
some instances these tasks seemed to place an additional burden upon students, who struggled to meet the
deadlines that were set or were frustrated by them. Further research is recommended in this regard.
Keywords: assessment, student research, supervision, course work master’s degrees, MBA
1. Introduction
According to Rau (2004), South Africa’s Council on Higher Education’s Quality Committee supports
the idea of educators being reflective practitioners (Schon 1983), expecting them to learn from their
teaching practice, engage with educational theory and practice self-evaluation. The teaching and
learning policies of Rhodes University also encourage lecturers to be reflective practitioners as a
vehicle for enhancing the quality of their teaching.
This study looks at the role of assessment within the research process of MBA students at the
Rhodes Business School. Engaging in reflective practise and focusing on the research component of
the MBA degree offered at the Rhodes Business School over a period of ten years, the goals of this
paper are to (1) describe the use of assessment tasks, and (2) to analyse the effects of these tasks
on the ability of students to complete their research, as well as (3) the effect they have had on their
supervision. Tracing the assessment practices over the years, it examines the role of assessment
from the perspectives of students and supervisors.
2. Course work master’s degree research
Course work master’s degrees have arisen in response to the need to update the knowledge of, and
further develop the practice of professionals (Drennan and Clarke 2009). According to Drennan and
Clarke (2009) they have also included a research component, with the intention that professionals
would develop research skills to either understand research results or to undertake their own research
in an independent manner. Despite these intentions, there has been very little scholarly examination
of these research endeavours, and whether they have indeed achieved these outcomes or not
(Drennan and Clarke 2009).
Referring to knowledge, understanding and skills, the Association of MBAs (2012: 7) requires that “all
programmes should ensure that candidates acquire a firm understanding of the major areas of
knowledge including ... business research methods and consultancy skills”. Unlike some other
countries, the South African accreditation of MBA degrees prescribes a relatively large research
component. A Masters-level qualification is required to be at least 180 credits on the National
Qualifications Framework, with at least 60 of these being research related (Department of Education
2007). One credit is equivalent to ten notional hours, implying that students should on average be
spending 600 hours on their research. Current proposals under discussion may see this being
319

Noel Pearse
reduced to 45 credits but still remaining a compulsory component of the degree (Department of
Education 2011).
It would seem though, that course work master’s degrees pose particular challenges for those
students enrolled in them. For example, they require students to engage in learning activities of
greater complexity than that expected of, or experienced at an undergraduate level (Kell 2006;
Department of Education 2007; 2011). Furthermore, the discipline of the research component is very
different to that of the coursework component, requiring an even greater level of independent work. It
also places great demands on students in the area of writing, which is internationally regarded as the
most challenging area of academic endeavour, and one where there are high levels of
underperformance (Drennan and Clarke 2009; Parr 2011).
Compared to other professional master’s degrees, for MBA students the research component poses
additional challenges, given the nature of the degree. The MBA degree is multidisciplinary in its
curriculum design, and has also permitted entry to students from a wide range of disciplinary
backgrounds. By implication, students typically embark on research in what is a relatively new field of
endeavour for them, and one in which they have not received a high degree of specialisation through
the coursework component. It is also more unusual for students to continue research in an area which
was a specialisation in their previous studies. In contrast, other professional master’s degrees tend to
have a comparatively narrower disciplinary focus and tend to build on the professional’s disciplinary
background, thereby enabling a higher level of specialisation in courses.
In addition, in some cases, prior studies have not adequately prepared MBA students to undertake
independent research. In other cases, some students who have done research in other disciplines
[e.g. natural sciences] find the paradigm switch accompanying commerce-related research to be
challenging. Yet other students struggle with the relatively unstructured nature of the research
component, especially in contrast to the course-work component of the degree, which has smaller
and more defined assessment tasks, due according to a prescribed schedule, and “written off” once
handed in. The research component is more extensive, and less structured, requiring students to
figure out their own structuring of their work in consultation with their supervisor, and reworking the
same piece of work many times over.
The challenges with research are compounded by motivational and time challenges, as many MBA
students do not value the research component as much as they value the coursework component.
Many are business practitioners and struggle to see the value of academic research to their business
practice. In addition, some MBA programmes – such as that at the Rhodes Business School - cater
primarily for students who are working fulltime and conduct their studies on a part time basis. Their
studies pose significant challenges from a time management and work life balance point of view.
3. Assessment and supervision
According to Ramsden (1992) assessment can drive student learning, and when feedback is viewed
as a dialogue between the student and teacher (or supervisor, in this case), then higher quality
instruction (or supervision) and learning occurs – including feedback about the mistakes that students
have made. Ramsden (1992) further notes that making expectations clear to students can in fact
promote a self-critical and independent approach in students towards their work.
However, much thought needs to go into the conceptualisation and design of such assessment tasks,
so that effective learning can occur in an incremental manner. According to the South African Higher
Education Quality Committee, assessment in an outcomes-based approach “should be consciously
determined by the end results ... making explicit the learning outcomes ... and then designing
assessment instruments that will effectively test students’ attainment of these outcomes…” (Council
for Higher Education 2003, in Rau 2006: 63). An outcomes based approach to education – including
the way in which assessment tasks and requirements are crafted – can potentially facilitate research.
Lovitts (2007) argues that an outcomes based approach has several benefits to students, including
more actively engaging them in the development of relevant intellectual capabilities and skills,
enabling them to more effectively gauge their own knowledge and abilities, and to monitor their own
progress through self assessment. Furthermore she argues that it will clarify expectations and
improve communications between the student and supervisor, thereby enhancing the quality of the
supervisory relationship.
320

Noel Pearse
Pearson and Kayrooz (2004) identify four roles of a supervisor, namely mentoring, sponsoring,
progressing the candidature and coaching. Of relevance here is the role of progressing the
candidature. This can be defined as “providing guidance on the thinking processes that lead to
successful research outcomes” (Nulty Kiley & Meyers 2009). The concept of scaffolding in education
has its roots in the work of Vygotsky (1962) and is used to refer to the interactional support that is
provided to mediate learning in the zone of proximal development. Assessment tasks represent one
form of scaffolding. A well designed sequence of assessment tasks can facilitate the completion of the
research. It is therefore argued that a more structured approach to supervision facilitates the progress
of students and assists with the process of supervision.
From a leadership theory perspective, if students enter into their MBA being underprepared to
undertake their research, then a directive approach or “telling” leadership style is a more effective
style of supervision, whereby the supervisor provides a lot more structure to the research process,
and detailed guidance to the student (Hersey and Blanchard 1977). This management principle would
support the idea of designing several small assessment tasks with explicit requirements to guide
students in their research endeavours. This would imply that providing in advance, explicit criteria for
assessing the various assessment tasks will support a “telling” supervision style and also helps to
clarify for students what is required in their research. Lovitts (2007) argues for greater transparency in
the assessment of PhD research, suggesting that making the tacit rules for assessment more explicit
will make them more transparent to students, who will then be able to prepare more effectively and
produce better quality work. Detailed assessment guidelines provide students with much needed
structure, thereby facilitating their research endeavours. Furthermore, by providing a number of
assessment tasks that are aligned with the final research dissertation, helps to break the dissertation
into smaller, more manageable components.
While assessment tasks have the potential to provide the scaffolding that students need to undertake
their research, Parr (2011) notes that having done assessment in writing, students and their
educators should be able to work out how to move forward once a gap in performance is observed,
but that a developmental model of writing that would enable this process, simply does not exist.
Consequently, educators need to be more reliant on their own judgement in this regard, emphasising
the importance of reflective practice (Schon 1983). In summary, it is deduced from the literature, that
the provision of several well designed formative and summative assessment tasks and explicit
guidelines for their assessment will facilitate student progress with the research. The aim of this
research is therefore to investigate if this has indeed the case, as perceived by supervisors and
former MBA students of Rhodes Business School.
4. Research design
Data was collected from historical documentation, through analysing samples of student work, and by
purposively gathering responses to specific questions through interviews and email correspondence
with supervisors, and past and current students. Documentation used included course outlines,
emails and course evaluations. The course outlines provide an overview of the course, and inter alia,
incorporate the critical cross-field outcomes and specific intended outcomes; teaching methods;
detailed instructions for all assessment tasks (including the assessment criteria and mark sheets to be
used in each case); and the programme schedule. Given the part time nature of the degree, most
supervision of students takes place by email. The emails and research related assignments and draft
submissions of all of the students that the researcher has supervised over the past ten years have
been stored, and served as a source of data. Course evaluation documentation was also used. As a
routine, a course evaluation is administered after each year’s presentation of the course. This has
mainly taken the form of an anonymous written survey, with students responding mainly to open
ended questions. These qualitative assessments have been complemented with a focus group
evaluation, conducted by someone outside of the Business School, and taking place in the last
teaching block of the Degree. This evaluation is written up as a report to the School. While its scope is
the whole of the degree, it also incorporates information about the research module and the research
dissertation.
This documentation was supplemented with data collected through face-to-face interviews and/or
written responses from five other supervisors and several students, to the questions set out in Table 1
below:
321

Noel Pearse
Table 1: Interview questions for supervisors and students
Questions for supervisors
I am investigating the role of assessment as set out in the MBA Research module and its effects on
students’ progress with the research, as well as on the supervision process. Could you please respond to
the following questions to assist in this research.
1. As part of the process of MBA students undertaking their research, a number of assessment tasks
have been set, namely the literature framework, conceptual analysis, literature review and research
proposal. Please comment on the impact of these assessment tasks on your supervision. In what ways
have they helped? Have they hindered in any way?
2. Referring to the content or details as set out in each of the assessment tasks, are there any changes
you would like to recommend?
3. The course module sets out the assessment form that will be used by examiners to examine the MBA
student’s research. Has this had any influence on your supervision of students?
4. Do you think that the assessment tasks are excessive, appropriate or too few? What changes would
you recommend regarding the number and type of assessment tasks?
5. Any other comment you would like to make about the assessment of the research?
Questions for students
I am investigating the role of assessment as set out in the MBA Research module and its effects on
students’ progress with the research, as well as on the supervision process. Could you please respond to
the following questions to assist in this research.
1. As part of the process of MBA students undertaking their research, a number of assessment tasks
have been set, namely the literature framework, conceptual analysis, literature review and research
proposal. Please comment on the impact of these assessment tasks on how you went about your
research. In what ways have they helped? Have they hindered in any way?
2. Referring to the content or details as set out in each of the assessment tasks, are there any changes
you would like to recommend?
3. The course module sets out the assessment form that will be used by examiners to examine the MBA
student’s research. Has this had any influence on how you went about your research?
4. Do you think that the assessment tasks are excessive, appropriate or too few? What changes would
you recommend regarding the number and type of assessment tasks?
5. Any other comment you would like to make about the assessment of the research?
6. Any other comments you would like to make about research in the MBA in general?
A thematic coding process (Boyatzis 1998) guided the collection of data and its analysis. In addition,
qualitative content analysis was used to describe what assessment tasks had been formulated, and
how they were assessed, and how this had changed over the ten years.
5. Findings
The findings of this research are presented according to the following themes: (1) a description of the
research course and its assessment tasks and how these have changed over the years; (2) student
perspectives; (3) supervisor perspectives.
5.1 Research coursework and assessment
The Rhodes Business School has offered its MBA degree since 2001. It was originally conceived of
as an executive MBA programme with students attending three two week teaching blocks over a
period of three years. More recently the coursework component has been condensed into two years,
with a further six months to complete the research component.
The research process begins with the delivery of the research methods module. This was usually
presented in the second teaching block of the second year but is now been moved to the first
teaching block of the second year of study. As part of this module, the class is given samples of the
work of prior cohorts as illustrations and sometimes to critique and assess for themselves. The
assessment tasks are discussed in class, some in a more cursory way and others – such as the
research proposal – more thoroughly, with students placing themselves in the role of assessor.
It is also during this module that students select their research topics and begin their research
endeavour. At this point students are assigned a supervisor. This schedule implies that the research
process now begins mid way through the coursework component of the degree and runs in parallel
with the latter half of the course component, with a further six months to complete the research, giving
a period of about 18 months supervision. This supervision takes place largely through e-mail
correspondence and telephone calls, with face-to-face meetings encouraged to take place at least
during the teaching blocks when students are on campus.
322

Noel Pearse
There are a number of significant milestones along this research journey. The first set of these are
prescribed by the School, and these take the form of assessment tasks. These assessment tasks
count towards the student’s mark for the module, culminating in a written research proposal that is
tabled at the Higher Degrees Committee of the Commerce Faculty. Once the proposal has been
tabled another set of dates are scheduled, but serve only as guidelines for the student in terms of the
progress that they should be making. The student is encouraged to discuss these latter dates with his
or her supervisor and to formulate a more detailed and structured progression of the field work and
write-up phases of the research.
The summative assessment tasks incorporated into the research methodology module include a
literature framework and a conceptual analysis, preceding a literature review; as well as the research
proposal. Each one of these tasks follow a criterion-referenced form of assessment, and detailed
assessment guidelines have been developed, as well as examples provided, both in class and on the
eLearning system. The literature framework and conceptual analysis are more recent additions as
assessment tasks, and were introduced to further break down the literature review component of the
research. Their introduction caused some anxiety amongst the first cohort who received these
additional assessment tasks as they felt uncertain about what was required and did not have
examples from previous students to look at.
Over the years some of the assessment tasks have seen some changes being made to the task
instruction and assessment criteria, in the light of the feedback received from supervisors and
students and the assessment of their work. For example, the research proposal assessment criteria
have been extended to add several minimum criteria that needed to be met before the proposal could
be tabled for consideration at the Higher Degrees Committee, focusing largely on the technical
requirements of the proposal, such as its length and conformity with the referencing requirements.
The rating scale for scoring the proposals was also adjusted as the scale range from one to four had
had the effect of inflating the results. A zero point was added to the scale.
In addition to these summative assessment components, there are two presentations that are
assessed formatively by fellow students and School Faculty. One of these presentations immediately
follows the research modules and the second takes place after the literature review is complete, but
preceding the writing of the proposal.
5.2 Student perspectives
It would appear that overall; the assessment tasks had served as scaffolding for students. One
student noted that they had given him perspective and helped to clarify the scope of the research.
Another noted that as a set, the assessment tasks provided her “with a logical series of steps,
standards, and knowledge to engage with [her] topic, literature and overall research proposal”. It also
provided her with a foundation for academic writing and referencing, which had been a challenge for
her. Another observation was that “the assessment tasks ... ensure that the learner fully understands
the depth, possible research questions and possible outcomes that the research might present.”
The content and details set out in each of the assessment tasks had been of benefit to students, in for
example clarifying what was required of them, enabling them to measure the standard of their work
and to monitor their progress. As a result they became a point of reference and a type of checklist in
preparing the assignments. However, one student suggested that more time be allocated in class to
discuss the tasks in greater detail, along with their time frames.
While most students viewed all of the assessment tasks as enabling them in their research, one
student was of the view that some of the tasks – notably the literature framework and the conceptual
analysis less helpful, as it did not suit her way of working and therefore she felt it was stifling. It was
also observed that some students were struggling to meet the deadlines for assignments, suggesting
that they had the potential to be an obstacle to the completion of the research, rather than an enabler.
5.3 Supervisor perspectives
In terms of the assessment tasks themselves, supervisors generally indicated that they had assisted
in their research supervision. Regarding the content of the assessment tasks, supervisors reported
that the details set out in the assessment tasks were sufficient and relevant. One supervisor noted:
323

Noel Pearse
“The assessment tasks have helped to structure what seems to be a new and difficult discipline for
many students ... Knowledge production is then a new experience and framing the elements of the
research endeavour does help to structure the research process and provide a logic for what needs to
be done.”
Supervisors confirmed that the assessments had impacted their supervision, guiding and directing
them and their student. They reported that they were very useful in breaking “the final ... output into
manageable component milestones along the journey to the final thesis [serving] as relevant inputs
along the way to a quality final product”. One supervisor raised concerns about the value of the
conceptual analysis, noting that “students have grappled to link and further develop the concepts in
the conceptual framework to a coherent literature review”.
Supervisors concurred that the assessment criteria had assisted “to ground the evaluation, and the
criteria are therefore useful in terms of the guidance given to students on how to set about the
research process”. The number of assessment tasks was deemed to be appropriate by most of the
supervisors, though one suggested that additional tasks would further facilitate the development of
the confidence of students to undertake the research, and their progress.
6. Discussion
Overall, the results of this research supports the deduction from the literature, that the provision of
formative and summative assessment tasks that are seen by supervisors and students to be well
designed and provide explicit assessment guidelines will facilitate student progress with the research.
The findings show that an increasing number of assessment tasks were incorporated into the module
over the ten year period, and that these had been updated from time to time as the result of reflective
practice. The assessment tasks were primarily focused on providing scaffolding to the literature
review as well as to the correct formulation of a research proposal.
While these tasks assisted the student and supervisor in structuring the research progression, in
some instances these tasks seemed to place an additional burden upon students, who struggled to
meet the deadlines that were set. It is debatable though, as to whether the progress of these students
would have been any better in the absence of the assessment tasks. Furthermore, there were cases
where students felt there was inadequate discussion or instruction given on how to undertake some of
the assessment tasks. This was also reflected in requests for examples of good responses to the
assessment tasks from previous cohorts.
7. Conclusion
It was concluded that overall, the inclusion of detailed assessment tasks was beneficial to both
students and their supervisors, but that more dialogue was needed between the lecturer, supervisors
and students to clarify what was expected of supervisors and students when supervising and
undertaking these tasks, respectively.
There was also some evidence that the structured approach adopted in scaffolding the research
endeavour through assessment tasks, did not suit all students. Further research is needed to
understand the reasons for this variance in response. In particular, investigating the effects of the
research ability of students, their study behaviour and learning preferences in this regard would be
insightful. Furthermore, the assessments used in this study were criteria based. Given the wide range
of interpretations that seem to exist about criteria-based assessment (Sadler 2005), closer scrutiny of
the format of assessment tasks is also required.
References
Association of MBAs. (2012). Criteria for the Accreditation of M B A Programmes. On line. Available at
http://www.mbaworld.com/administrator/file_sys/uploaded_files/1299681060-MBA%20critieria_web.pdf
Boyatzis, R.E. (1998). Transforming Qualitative Information: Thematic Analysis and code development,
Thousand Oaks California: Sage Publications.
Department of Education. ( 2007). The Higher Education Qualifications Framework, Higher Education Act, 1997
(Act No. 101 of 1997). Pretoria: Government Gazette.
Department of Education. (2011). The Higher Education Qualifications Framework: As Revised. Pretoria:
Government Gazette.
324

Noel Pearse
Drennan, J. and Clarke, M. (2009). “Coursework master’s programmes: the student’s experience of research and
research supervision”. Studies in Higher Education, Vol 34, No. 5, pp 483–500.
Hersey, P. and Blanchard, K. (1977). Management of Organizational Behavior: Utilizing Human Resources.
Englewood Cliffs, NJ: Prentice-Hall.
Kell, C. (2006). “An analysis of entry-level postgraduate students’ readiness for student-centred, Masters Level
learning”. Learning in Health and Social Care, Vol 5, No. 3, pp 133–141.
Lovitts B. E. (2007). Making the implicit explicit: Creating performance expectations for the dissertation. Sterling,
Virginia: Stylus.
Nulty, D. Kiley, M. & Meyers, N. (2009). “Promoting and recognising excellence in the supervision of research
students: an evidence-based framework”. Assessment & Evaluation in Higher Education, Vol 34, No. 6, pp
693–707.
Parr, J. (2011). “Repertoires to scaffold teacher learning and practice in assessment of writing”. Assessing
Writing Vol 16,pp 32–48.
Pearson, M., and C. Kayrooz. (2004). “Enabling critical reflection on research supervisory practice”. International
Journal for Academic Development, Vol 9 (1), 99–116.
Ramsden, P. (1992). Learning to teach in higher education. Cornwall: TJ International Ltd.
Rau, A. (2004). Supervision: A Foucaultian Exploration of Institutional and Interpersonal Power Relations
Between Postgraduate Supervisors, Their Students and the University Domain. Unpublished Doctoral
Thesis, Rhodes University, Grahamstown, South Africa.
Sadler, D. R. (2005). “Interpretations of criteria-based assessment and grading in higher education”. Assessment
& Evaluation in Higher Education, Vol. 30, No. 2, pp 175–194.
Schon, D. (1983). The Reflective Practitioner: how professionals think. London: Temple Smith.
Vygotsky, L.S. (1962). Thought and Language. Cambridge, Mass.: MIT Press.
325

Using Technology to Support and Monitor Independent
Learners
Martin Rich
Cass Business School, London, UK
M.G.Rich@city.ac.uk
Abstract: This paper focuses on independent learners within a taught undergraduate or postgraduate degree
programme. Typically these are students who are expected to carry out an element of independent work, even
though the primary focus of their studies is formal teaching. They are often uncertain as to what sort of research
skills and methods they should use in independent study, and sometimes receive little guidance on the extent to
which they should adopt formal research methods. Frequently they are placed in a position where they need to
obtain a rapid grounding in the basics of research methods – often at a pace dictated by the requirements of their
independent work and not by the structure usually associated with a taught degree programme. Even as
independent learners, they sometimes want to share ideas with their peers. Sometimes they will need to acquire
very specialised skills which are relevant to their chosen subject, perhaps by contacting an expert in the field.
Sometimes they will have very clear ideas as to how to pursue their chosen topic, and the educator’s role
becomes one of mentoring more than one of tuition. These properties all point towards the use of technology,
such as a virtual learning environment (VLE) to provide materials for this group. In particular, work on the notion
of a personalised learning environment (PLE) is relevant to individual learners who need access to a distinctive
and tailored set of material. While it should be straightforward to put a course in research methods online, a move
towards providing personalised support for individual learners demands a deeper understanding of their
requirements. It also provides an opportunity to encourage independent learners to operate within a loosely
connected community with areas of common interest. While the independent learning process is typically based
on a constructive paradigm, there is sometimes a need for some material to be provided in a more didactic,
transmissive, style. Moreover, use of a VLE offers scope for educators to monitor learners, to respond quickly to
requirements, and to gather knowledge about how a cohort of students operates. Therefore the information
gathered through this application of technology can be used to inform the way that support can be offered in
future. This paper discusses the use of technology at a relatively simple level to support independent learning,
within a range of taught degree programmes. It raises issues about how this could be extended, especially in the
context of changing expectations among students and evolving attitudes to their use of technology.
Keywords: independent learners, constructivist learning, research skills, virtual learning environment
1. Introduction
Independent learning is an important concept within an undergraduate or postgraduate degree
course, even where the primary mode of learning is through formal taught lectures and seminars. The
Higher Education Academy (2012) addresses this particularly in the context of international students
entering the British higher education system, who may have varying expectations of the level of
support and formal tuition that they will receive. Drawing on work by Candy (1991) on lifelong
learning, independent learning here is characterised by the responsibility for learning being placed on
students, and not on educators.
An element of independent learning has always been an essential component of a British honours
degree, typically achieved by a student writing an extended essay, or carrying out a piece of scholarly
work of their own design. Potentially this is a valuable opportunity to practise skills, such as the ability
to carry out research, which are learned elsewhere in a degree course. Furthermore the emphasis in
recent years on student-centred learning has led to a tendency to incorporate activities where
students do assume significant responsibility for their own learning.
A cynical view of the inclusion of independent learning components within a degree course would be
that it could appear as an excuse to reduce the amount of input provided by lecturers. However a
counter-argument would be that there is a tendency to ‘over-teach’ in universities and that, if properly
managed, a reduction in formal teaching could offer scope for students to develop and internalise
their own understanding of a subject. It has been argued (Bligh, 1998) that the interaction between
lectures and other means of instruction within a university is not necessarily well understood.
So one key to effective learning is the provision of effective support, perhaps taking the form of
scaffolding (Wood, Bruner and Ross, 1976) a concept which, while initially presented in the context of
young children’s skills acquisition, can be extended to adult learners. The increasing use of
information technology to provide such scaffolding, together with the increasing expectation among
326

Martin Rich
students that online resources will be provided as part of their learning process, suggest that online
tools could be useful in providing a readily available, while not intrusive, platform for this support.
Although this is primarily written as a conceptual paper, discussing both the use of technology to
support learning and the need to support independent learners, it is grounded in experience of
supervising independent, but directed, work by students and also of use of a virtual learning
environment, principally as an organising tool for resources made available online to students.
2. Embarking on independent learning
This section sets out a simple set of categories covering the type of support from which an
independent learner could benefit. The diagram below represents these as a set of forces contributing
to the student experience, but these are also mapped onto a set of areas represented by three
overlapping ovals.
Figure 1: Categories
Methodological support refers to guiding students through uncertainty as to the type of method that
they should use. In many cases this will extend to guidance about whether students need to use
formal research methodologies in independent work: for example at the undergraduate level and
particularly in vocational subjects it could be argued that formal research methods would be
inappropriate. Even in such cases, students will expect some guidance as to what conventions to use.
This can extend to practical issues such as referencing, or to what exactly needs to be delivered at
the end of the independent learning exercise.
In some cases, students do need to acquire research skills, and this is where these categories of
support become directly relevant to the subject of teaching research methods. Methodological support
becomes most important where students need to acquire the language of research methods rapidly.
For instance they may need to appreciate enough about the distinction between qualitative and
quantitative approaches to make an informed choice of which to use. They may need to learn enough
about the limitations of (for example) carrying out interviews or surveys to ensure that they do not
interpret their results inappropriately.
In the context of a taught degree course, therefore, it is worth noting that methodological support often
does not constitute a thorough training in research methods. It can be tactical, bounded by the
student’s own needs, limited by time constraints, and very closely targeted at the requirements
imposed by a particular piece of independent study.
Data support refers to the need to find, interpret, and identify primary and secondary data to be used
by students. This category does not apply to every example of independent learning, but in the case,
for example of a dissertation undertaken by a management student which uses data about real
organisations, the primary data is instrumental in making the student’s output distinctive. Identifying
suitable data does depend on the choice of methodology, so data support is related to methodological
support. But identifying and finding data is an important enough exercise, and has a sufficiently
different feel to students, to deserve a separate category within this framework.
327

Martin Rich
Data support, in many cases, is closely intertwined with methodological support. But the nature of the
model should be that students draw on more than one category of support in each stage of their
studies as independent learning.
Technical support refers to detailed issues of how to carry out activities that contribute to independent
learning. As with data support, there is a close link with methodological support but it is also useful to
identify the boundaries between the two. Methodological support might help a student to recognise
that, for example, a large-scale quantitative analysis involving the use of regression would be
appropriate to deal with a particular issue. Technical support would help the student to choose
suitable statistical packages to implement this process.
The present author’s own past experience, as an undergraduate student taking a combined course in
mathematics, statistics and computer science, included an instance where technical support would
have been valuable. The independent learning task was to write some code in the Fortran
programming language. While the function of the code was tightly specified by the supervisor, the
process of writing and delivering the code remained the author’s responsibility. Technical support
could have taken the form of guidance as to how to structure and design the code.
Mentoring support refers to the process of monitoring, encouraging, and gradually guiding a student’s
independent learning. This has deliberately been placed at the bottom of the diagram, with the
implication that the student is at the centre, because this is the category which most closely
represents the conventional definition of support. It is also the category where students would expect
to gain support from their peers as well as from educators and one where the concept of students and
educators together as a community is most valuable.
In the model proposed by Garrison et al (2000) of ‘communities of inquiry’ the category of ‘social
presence’ fits most closely with the need to provide mentoring support. More recent developments of
this model have emphasised the nature of social presence (Annand, 2011) and raised the question of
whether the effective management of social presence can contribute measurably to the student
experience.
These four categories of support are located in three overlapping area, represented by ovals on the
diagram.
The external environment covers sources of data, the phenomena which an independent learner
might seek to understand, the constraints on exploring that data, and so on.
Techniques refer to particular activities and processes which an independent learner must follow.
The learning set refers to a community of independent learners and is so called because of its use in
action learning (Revans, 2011). Despite the recent date associated with the publication, Revans
introduced the concept of action learning, and noted the centrality of the set to it, in the mid-twentieth
century. The reissue of his work in recent years is an indication of its continuing relevance,
representing as it does an effective and established model for a group of people who are working
together to effect collective learning.
Membership of a learning set becomes important when a group of people are learning together and
are only loosely tied: they are each working on individual activities but they have some association
with one another and some common learning needs.
Both the adoption of the notion of the learning set, and the use of the term ‘support’, reflect one key
characteristic of independent learning already identified, that responsibility for learning remains with
the student.
3. Bringing technology into the process
The requirement for support, of various types, for independent learners, together with the potential for
the creation of a community of inquiry, indicates that this is a fertile area for the use of information
technology, and particularly online resources, to supplement learning. Online communities work well
where members of a group are loosely tied and can be geographically dispersed, but are
nevertheless connected through one specific area of shared interest. One of the elements of
328

Martin Rich
scaffolding identified by Wood, Bruner and Ross (1976) is ‘reduction in degrees of freedom’ and
online learning can provide tools to make independent learning more directed, where this is
appropriate. A lecturer could offer a set of options to a student, or even provide a set of suggested
readings, or domains in which to carry out further reading, as a way to rationalise the uncertainty
perceived by the student.
Online communities also work well when students – if, as many still are, they are in an environment
where face-to-face teaching through lectures and seminars is the dominant medium of instruction –
engage in activities which take them away from the physical surroundings of the university.
Independent learners should be visiting, or observing, interviewing, or even just reading material
which is available somewhere other than the university itself and online communities can provide a
valuable element of ‘tethering’ them to the university’s resources. By combining involvement with
learning at the university, and involvement in learning in the outside world, they can contribute to the
furtherance of the scholarship of engagement (Bruner, 1996).
Technology is already proving useful in providing background support and instruction. For instance a
student seeking an introduction to the relative merits of quantitative and qualitative research, within
the category identified above as methodological support, would find the video introduction to the issue
by Bryman (2010) to the issues surrounding the two a very valuable resource. While this is part of a
traditional didactic lecture delivered to camera (and, as it happens to an audience at the same time)
independent learners can benefit from being able to watch it at their own pace. And it is worth noting
that the live audience for this lecture comprises students who are about to go on to become
independent learners, as they are about to start writing dissertations. But to do so, with confidence,
they need to be given an introduction to the range of techniques, which is presented in a highly
structured and standardised form. This is an occasion where didactic teaching and independent
learning are complementary to each other.
Most students in higher education now are familiar with the use of virtual learning environments
(VLEs). Personal learning environment (PLEs) set out to provide a resource which is more tailored to
individual requirements than a conventional VLE. It is noteworthy that the rhetoric on PLEs refers to
the importance of lifelong learning (Van Harmelen, 2006), which is also invoked in definitions of
independent learning. A PLE, if correctly implemented, should offer students a highly tailored set of
learning resources. PLEs should be set up as open systems so that students can bring in resources
from different places as needed. But to gain effective benefits from a PLE, a student must have a
deep appreciation of their requirements, and perhaps a clear understanding of the gaps that exist in
their knowledge at the commencement of an exercise in independent learning. In terms of the
categories of support identified above, mentoring support has most potential to steer students towards
a recognition of what they do not know, and what they need to learn.
Use of a PLE is consistent with the emergence of a group of independent learners as members of a
community of practice (Wenger, 2010) united by the need to obtain support and to develop individual
skills. Among the characteristics that Wenger associates with communities of practice are learning
partnerships and learning citizenship. Independent learners, if they are to take responsibility for their
own development, need to develop the attributes of good citizenship and they need to work with one
another to acquire collective knowledge. Wenger devotes some of his analysis to the difficulty in
recognising accountability in social learning communities and, while it is impossible to be prescriptive
about how it should be achieved, ensuring accountability is a challenge for communities of
independent learners.
Learning analytics (Siemens, 2010) have a role to play as well. By its nature, independent learning is
difficult to monitor. If the output from an independent learning exercise is a single substantial piece of
work, then educators may have no significant indication of what progress is being made until the final
product has been submitted. However if technology, such as a PLE, is used to facilitate learning
within a community, data produced by the system can be used too track students’ participation. While
students might have legitimate concerns – which must be acknowledged – that use of such data could
be intrusive, the opportunity exists for educators to, at least, spot instances of non-participation. On a
larger and more complex scale, the account by Dawson (2010) of the use of a large-scale analytic
tool to understand the social learning dynamics created within a cohort of students, is an indication of
the potential of learning analytics for fostering a better understanding of learning processes.
329

4. Practical experience and examples
Martin Rich
Part of the motivation for taking an interest in VLEs, and enhancements of this such as a PLE, arose
from an observation that both educators and students tended to associate VLEs with formal,
structured, learning materials. In the context of supporting independent work, a set of formal classes
about research methods, aimed at a cohort of students who would use these methods in producing a
piece of work independently, would be perceived as a suitable way to use a VLE. Support for the
student while producing the independent piece of work would remain the domain of email and of
personal meetings. Therefore the examples of student support discussed in this section are mostly
instances of support being offered through email.
One exception arose with two students on a part-time taught masters course in management, working
jointly on a project where they applied course concepts to a business where one of them worked. On
their own initiative, they created team spaces within the VLE where they could share documents with
each other and with their supervisor. Both students were used to working collaboratively using
software within their employment and as such the notion of creating a do-it-yourself content
management system, using the resources provided by the university, for their project came readily to
mind. Documents chosen for this principally related to secondary data and to the context of the
project, so in terms of the categories discussed above, this application fitted firmly in the domains of
data support and of the external world.
In other instances the domains are less sharply delineated. One example of this was an
undergraduate student who, because of some complicated personal circumstances, was carrying out
a dissertation some months later than his peers. An extract from an email message from the
supervisor gives an indication of the nature of the support that was appropriate in this case:
‘You are completely right to say that qualitative research and primary data are important
aspects of your approach. You are right also to say that you need to add a conclusion but
you could perhaps explain more about exactly how you plan to get from the interviews to
the conclusion. Are there particular themes that you really need to draw out?’
While this is principally methodological support, discussing the relevance of qualitative approach, and
what the student would need to do to make the dissertation distinctive, there is also a strong element
of reassurance and of steering the dialogue in a particular direction, which could be regarded as
mentoring. Both these elements fitted into the domains of the external world and of techniques. The
nature of this student’s work was such that he did not immediately fit into a learning set. In fact use of
a VLE with asynchronous discussions would have offered him an environment with some of the
elements of a learning set because he would have been able to benefit from other students’
dialogues.
Methodological support can also overlap with data support. This example related to a student building
a project around their own experience of working in projects, and crucially using their own
observations of project management techniques as their primary data:
‘So I think some of your perspective needs to be ”what does somebody trying to
understand a set of practical issues around information systems, and project
management, need to know about research methods?” and “how can the ideas of
researchers whose work you have read help to get some new insights into a process that
I've already been observing for a while?” ‘
In terms of methodological support this relates to an issue where students often expect guidance, but
which was particularly marked in this student’s case. This was the interplay between research
methods and the subject, and the extent to which formal research methods could be built into the
student’s work given constraints of time and of the student’s experience. This example firmly fits the
domains of the external world and of techniques.
Technical support can extend to issues which are relevant to a number of different students, though it
could also cover very specific points determined by one student’s work. A specific example of this
applied to a set of students carrying out individual projects around a theme related to economics.
They were encouraged to use some quantitative data, either as part of a mixed methods approach or
as the core of their work. The students were uneasy about the complexity of quantitative analysis that
was expected of them – for instance whether they should carry out multiple regression. In the event
this was resolved using an elaborate email dialogue involving students, an academic supervisor, and
330

Martin Rich
the course director to establish that multiple regression would be desirable in a purely quantitative
analysis, but could be ‘traded off’ with other possible analytical techniques for students taking a mixed
method approach. This example covers techniques, but also the learning set given that it concerned
several students who would each have different interpretations of the same constraints.
This last instance was one where use of a VLE for discussion would clearly have been beneficial. It
would be possible to put the definitive instructions on the level of analysis required onto the VLE. It
would have been a simple matter to invite the course director into the VLE discussion to adjudicate on
the suitability of particular research methods, given the constraints associated with the course as a
whole and university policies. But this would have required both preparatory work to set up resources
within the VLE, and an encouragement to students to see use of the VLE as part of the process for
this exercise.
Furthermore there are some general barriers highlighted by students when their reluctance to use the
VLE was raised. Some of these stemmed from perceived usability issues, whether these were to do
with a dated look and feel associated with earlier versions of the VLE, or even the extent to which a
later version was still optimised for users with a web browser on a personal computer when students
were more likely to access it using a smartphone or a tablet device. However one should not assume
that these shortfalls in usability are the root causes of any reluctance to use the VLE. Students also
reported a lack of a sense of ownership: if the were to discuss matters they would prefer to do it in
their space and not one provided by the university, and also the sense of email being a familiar and
readily available medium for communication. Even for students familiar with the use of social media in
their everyday lives, the need to learn their way around a new area in the VLE relating to independent
stud could be off-putting.
5. Concluding remarks
This paper has set out some thoughts about the process for support of independent learners,
particularly in a setting where technology such as a PLE is available to facilitate learning. The
process, and particularly the importance accorded to methodological and technical support, offers the
scope to ensure that independent learners can carry out research with as good an understanding of
the research methods as can be expected.
The key messages which could be extracted from this analysis include:
Recognise the four distinct domains of support and ensure that students can draw on each of
them
Understand that there are close connections between the four domains, and that sometimes the
messiest and most intractable problems where students need support occur at the boundaries
between these domains.
Be aware of the extent to which individual learners should take responsibility for their own
learning, and be prepared to hand over responsibility gradually if necessary
Foster the use of social presence and good learning citizenship
Introduce a level of personalisation into technical solutions which are used, even if they do not
strictly constitute a personal learning environment
Consider the use of learning analytics to provide monitoring and understanding of the student
cohort.
References
Annand D (2011) Social presence within the community of inquiry framework. The international review of
research in open and distance learning. 12 (5) retrieved 7 th March 2012 from
http://www.irrodl.org/index.php/irrodl/article/view/924/1855
Bligh D (1998) What’s the use of lectures? 5 th edition. Exeter, Intellect
Boyer E L (1996) The scholarship of engagement. Journal of public service and outreach 1 (1) 11-20
Bryman A (2010) Conducting mixed methods research (video) retrieved 7 th March 2012 from
http://www.youtube.com/watch?v=0lZ3w8pKyJI
Candy P (1991) Self-direction for lifelong learning: a comprehensive guide to theory and practice. San Francisco,
Jossey-Bass
Dawson S (2010) ‘Seeing’ the learning community: an exploration of the development of a resource for
monitoring online student networking. British journal of educational technology 41 (5) 736-752
331

Martin Rich
Garrison D R, Anderson T and Archer W (2000) Critical inquiry in a text-based environment: computer
conferencing in higher education. The Internet and higher education 2 (2-3) 87-105
Higher education academy (2012) Independent learning retrieved 7 th March 2012 from
http://www.heacademy.ac.uk/resources/detail/internationalisation/ISL_Independent_Learning
Revans R (2011) ABC of action learning. Revised edition. Farnham, Gower.
Siemens G (2010) What are learning analytics retrieved 7 th March 2012 from
http://www.elearnspace.org/blog/2010/08/25/what-are-learning-analytics/
Todd M, Bannister P and Clegg S (2004) Independent inquiry and the undergraduate dissertation: perceptions
and experiences of final-year social science students. Assessment evaluation in higher education 29 (3)
335-353
Van Harmelen M (2006) Personal learning environments retrieved 7 th March 2012 from
http://www.cs.man.ac.uk/~mark/PLEs_draft.pdf
Wenger E (2010) Communities of practice and social learning systems: the career of a concept. In Blackmore C
ed Social learning systems and communities of practice 179-198. London: Springer/Open University.
Wood D, Bruner J and Ross G (1976) The role of tutoring in problem solving. Journal of child psychology and
psychiatry 17 (2) 89-100
332

A Methodological Approach to Investigating the Transferral
of Human Resource Management Practice From a
Multinational to the Host Country
Kate Rowlands
Salford Business School, University of Salford, UK
K.E.Rowlands@salford.ac.uk
Abstract: This abstract sets out to examine the extent to which a multinational organisation can transfer their
Human Resource Management policy and practice to a host country subsidiary. In order for the research to reach
the required outcomes it is imperative that the study is set upon robust research methodologies. The kernel to
this research paper is to discuss the introductory approaches to the research in terms of philosophical stance and
the methods best suited to this research as well as setting the piece in context and establishing the background
to this study. The rationale behind my decision to conduct a case study of the approach of a major global retailer
towards their penetration of the Malaysian market. In particular, the study will focus on how they have transferred
Human Resource Management policies which are practiced in the Western society to a developing country
considering the differences between Human Resource Management and International Human Resource
Management
Keywords: international human resource management, multinational, methodology, Malaysia, host country,
human resource management practices
1. Background
The theory of Human Resource Management has been debated for many years however in today’s
global climate where organisations are seeking new and innovative ways of expanding what does the
future hold for Human Resource Management and how will Multinational organisations continue to
develop this practice across their global sites. Recent developments within the HR field by notable
champions such as Perlmutter, 2011 have suggested that the future of International HRM practice lies
with its ability to accept value adding contributions from the host country company through wider
organisational objectives. On the contrary Rowley, (2007) has highlighted that many host country
central sites are uncertain as to the role that they have to play in contributing to HRM practice and
policy when they have an abundance of senior management from the parent company demanding
nothing more from their function than the role that they have traditionally played, follow the leader.
The question could therefore be asked, can the host country develop their own effective and strategic
HRM practice locally?
This study aims to examine the critical issue of which appropriate research methodologies can be
adopted to ensure that the overall research aim is achieved which is; what is the extent to which the
HRM function of the MNC in the host country is able to achieve a strategic vision. If the host country is
not able to do so then what are the implications for both the host and parent company’s?
2. The role of human resource management in the multinational
Human Resource Management (HRM) in today’s economic climate for the MNC is fraught with issues
and disparity. Over recent years there has been a dramatic growth in the number of organisations
directly investing in locations overseas namely East Asia (Lane, 2000). This has been induced
through the liberalisation of international trade and the emergence of economic regions such as the
European Union which have enabled direct investment overseas to both gain and also maintain
competitive advantage through its people that is the human asset (Myloni, Harzing, Mirza, 2004). The
Human Resources and values associated with HRM facilitate the development of the organisational
culture and also the people that operate within influence the overall culture so is it wise to attempt to
transfer the core of the business (Wright et al, 2000). This has led to enormous implications for HRM
in terms of how will the organisation manage workforces in countries which have a bewildering
variety of cultural, socio-economic and political differences and in addition to these complex issues
there is the task of placing an expatriate workforce in a country outside of their experience (Clark,
Knowles, 2003).
The MNC can be described as a vehicle for transporting HRM policy and practice from the parent
country across national boundaries (Stening, 2004) but the underlying issue here is what are the
limitations to this transferal process or is the expectation of the MNC to blindly attempt to function in
333

Kate Rowlands
modus operandi (Ferner, 2000). With the standardisation of technological advancement across the
industrialised sector including production and process and the shortening of product lifecycles MNC’s
are turning towards the HR function to not only support and help but to drive their strategic
organisational competitive advantage. HRM is no longer viewed as a support function but now is an
integral part of the strategic vision for the MNC (Smale, 2007). The limitations to the MNC penetration
into East Asia are both diverse and complex in nature. The MNC must not only respond to national
demands by the host country but also faces the challenge of maintaining a clear strategy
simultaneously in both a local and global range. There are multiple constraints here notably the
resistance from the host country in terms of the concern for having their national economy overrun by
global organisations and the control which the MNC seeks to maintain through enforcing decision
making which is made at a global level without the consideration for the national scope (Hirst,
Thompson, 1999).
There is also a view that the MNC arrives with the objective of continuing with the strategy of a one
approach that is a ‘best fit model’ of HRM practice for their organisation as a whole and not a local
level driven goal which may be courteous in its approach to the host country. In terms of fighting such
a powerful MNC the host country appear to be restricted in their ability to successfully regulate the
activities of the MNC once they are operational within the country (Royle, 2000). An example of such
powerful methods are highlighted by Stenning, (2005) based on a study carried out on the fast food
chain McDonalds whereby he examined the food chain in terms of how they had integrated into a
global market and stated that although the local employment regulations were still applicable to the
organisation “the full impact of such systems is being mediated and undermined by the strategic
imperative of powerful company-based employment systems”. This demonstrates that large MNC’s
may pacify the local arrangements in terms of HRM legislation but they are also driven by a
commanding agenda which destroys convergence and fails to invest in employee relationships in
order to become the primary shaper of the global economy (Ferner, 2003).
3. HRM versus international HRM?
The kernel of this thesis is to address the issues faced by MNC’s in terms of the transferal of HRM
policy and practice in their host country however in order to do so the framework for this approach
must be justified in terms of classifying the HRM model. The MNC holds a unique set of managerial
issues and organisational processes which need to be coordinated to facilitate the management of
employees on a global scale. However, the precarious balance of performing as a local organisation
versus the extent to which their practices resemble those of the parent organisation or global standard
they are striving to attain still remains a concern (Rosenzweig, Nohria, 1994). Based on this
knowledge it is a very critical issue that HRM is an integrated part of the overall strategy for the
organisation and that said Perlmutter (1998) notes that there are three strategic choices for the MNC;
Ethnocentric, Polycentric and Global. However, the issue still remains in question how are these HRM
policies and practice transferred? Do the MNC’s attempt to combine practice using methods which
suit the local institutional environment? It could then be said that MNC’s are operating in a confused
melange of internal differentiation depending on location as to whether they adopt a local
isomorphism (Jain et al, 1998). Based on these assumptions HRM constitutes a major constraint
within an MNC and in addition to these issues there is still the perilous question of how employees
from disparate national, cultural backgrounds are employed and managed successfully (Adler,
Bartholomew, 1992).
4. International HRM model
In comparison to the western approach to HRM there are some notable differences which act as
imperative elements for the success of HRM within the MNC. Generally there are more activities
involved with IHRM (Dowling, Welch, 2004) to include Internationalisation of taxation for employees,
relocation, socialisation, host government relationships and language/translation services. These
points merely underline some of the key attributes linked with the role within the MNC in terms of
ensuring that the people focus is led from a centralised point. It is imperative that the expatriate
employee welfare is paramount and this is of much greater significance than the domestic HRM
model as the success of embedding specific people management practices to realise profits and
viability of business operations for the entire business (Scullion, 2004). Failure to establish these early
routines could indeed be catastrophic in terms of the organisations presence on the international
stage. Other HRM policies may include healthcare, spouse and family relationships support. Some of
these practices are also dependant on the maturity of the MNC. As the MNC develops and grows
334

Kate Rowlands
there is less significance placed on the role of the expatriate and the HR focus changes in line with
current people needs.
There is also the added pressure from the local governments due to the high profile of the MNC
particularly within developing countries. Another notable point for IHRM is the on-going development
and management of the expatriate performance as this differs from the standardised western model
as the MNC hierarchy is essentially flatter, less traditional and operates more on a networking basis.
Managing the performance of expatriates can create chaos as it is often a complex approach in terms
of designing and implementing an appropriate criteria for performance management particularly when
the workforce maybe a combination of parent, host and developing country employees (Shen, 2003).
Other points include wider variables within the decision making process as centralised decision
making cascaded through the MNC can be thinned in terms of the HR function itself. The trend of
moving towards e-enabled HRM and the devolving of the day-to-day practice to line managers within
the host country in addition to flexible methods to the traditional HRM model, all threaten to dilute
decision making practice and more concerning, the role of the HR specialist (Purcell et al, 2009).
5. Transferring HR practice and policy to a developing country
It can be suggested that the term Human Resource Management has acquired increased currency
since the 1970’s onwards in Western countries ( Blyton and Turnbull, 1992). As a result there has
been excessive debate on what exactly is HRM and is it merely Personnel Management in a new tin?
Indeed a whole industry has developed around the term which is led by the concept that HRM can fix
an organisation’s people problems. This has been challenged by authors such as Legge, (1995) who
have questioned the validity of such an approach to people management in organisations today.
However, Townley, (1994) considers that this form of people management has evolved through both
societal change and the need for this approach based on the dimensions of the organisation in
modern day western society. Some of these changes were brought about through economic factors
such as the significant industrial restructuring of the 1980’s alongside the demise of traditional
manufacturing and trade union influence and collective bargaining. The recession of the early 1980’s
also marked a fundamental change in the structure of employment in Britain throughout this period
thus igniting the need for a change in the methods used to manage people. These harder factors
inevitably were accompanied by social shifts and the reduction in union power encouraged employees
to become more self-reliant and according to Armstrong, (2008) HRM practice reflects this whereby
the employer has a relationship with the individual employee. It has also been argued, Kanungo et al,
(1990) that the western way of HRM works and adds value to the organisation so it therefore
understandable that other developing countries are attempting to emulate such works within their own
organisations.
According to Blunt and Jones, (1992) these emulations are usually fractional of ‘best practice’ of the
methods used in western society however, the question is whether or not these can be replicated
where society values and economic climate differ so greatly from their western models. It has also
been argued (Warner, 2002) that the globalisation of management practices has greatly contributed to
the convergence of managerial mindset. Another question for the purpose of this study is how
successfully these management practices converge and whether the diversity demonstrated in
management practice can be maintained through ‘indigenous’ management. This will also test the
validity of HRM practice in terms of whether it is a product of political, economic, social and cultural
aspects or can it be a universal set of functions? According to Pugh and Hickson, (2002) it has been
argued that managerial elements are converging due to a need to pursue similar technological tools
to gain efficiency in production and general operation. Lawler et al, (1995) found no significant
difference between local organisations and multinational companies in terms of HRM practices.
Similarly, in a study of HRM practices of foreign, joint venture and local firms in Sri Lanka, (Mamman
et al, 2006). Based on recent studies (Horwitz et al, 2006) it can be argued that due to this rapid
convergence MNC’s need not change their HRM practices when operating in some developing
countries. According to Mamman, Adeoye, (2010), ‘they can adopt an ethnocentric approach to their
HRM policy and practices but is it that easy?
6. Research design
This study will adopt a case methodology approach. There is clearly the question of why adopt a case
methodology approach without the consideration of a study across a number of organisations? The
case study method as discussed by Stake (1995) notes that the objective of studying one
335

Kate Rowlands
organisation is that it can benefit to understand in depth the complexity and nature of the case study
in question as oppose to a more superficial comparative study across an industry or sector.
A case can be described as a community, location or organisation (Bryman, 2008). The case method
is often associated with qualitative research methods as oppose to quantitative methods such as
statistics and factual data as these can often fail to fulfil the depth of the study. However, that said
there are a number of methods which are qualitative in nature which help provide a case focus of
interest and an intensive and detailed examination such as observation and interviewing. Yin (2003)
distinguishes five variants of a case:
The critical case; whereby the researcher has a well-designed theory and the case is chosen to
support this theory and provide a better understanding.
The extreme or unique case; a common focus in clinical studies providing an intrinsic interest
which makes the case extreme or unique.
The representative or typical case; here a day to day or typical situation is adopted to provide
the case background for the study as they are broad and suit a number of specific questions to be
answered.
The revelatory case; when an investigator has the opportunity to access a particular
phenomenon previously inaccessible for scientific research.
The longitudinal case; this may be chosen as it offers the opportunity to be investigated at two
or more junctures and also so that it can be studied over a period of time.
There are several benefits from adopting the case study approach; firstly this method permits the
author to collect valuable qualitative data without becoming lost in volumes of information. This
method of research provides a carefully planned and crafted study of the ‘real life’ situation of Human
Resource policy and practice in both the U.K and Malaysia. It also provides the author with a detailed
perspective of the study and enables an insight into the events and their relationships which may not
be gained through other larger research methods where valuable insight and data may be lost in
volume.
There are also criticisms of this method which include the notion that there are little grounds for
establishing reliability and generality of findings due to the study being concise. In addition, to the size
of the study there is also the significance that the intense exposure of the case may create biases
towards the findings. It has also been argued that such methods can only be successful as an
exploratory tool.
Moreover the case study method for the purpose of this study allows the author to become engaged
in depth. Due to the nature of the study the researcher is permitted to gain great detail from the case
study which will highlight the rhetoric and reality of HRM practice and policy in this study. Robert Yin
states that the case study is an ‘empirical inquiry that investigates a contemporary phenomenon
within its real-life context; when the boundaries between phenomenon and context are not clearly
evident; and in which multiple sources of evidence are used’ (Yin, 1984, p. 23).
7. Research aim and objectives
In order for this study to remain focused and valuable within its field it is essential to propose at an
early stage an overall aim which will guide the line of enquiry throughout the literature review,
research design and methodology and analysis of data.
Aim:
This research aims to examine the organisational framework of Human Resource Management within
the U.K and Malaysia with a particular focus on how the parent and host company interact to shape
the process and delivery of HRM policy development to meet with the challenges of improving the
working environment for staff.
Research objectives:
To examine the organisational structure of HRM in the U.K by clarifying the legislative framework
and the influences of external factors such as culture and socio-economics.
336

Kate Rowlands
To identify the structure and characteristics of the retail industry through secondary data analysis;
which will then be used to create a typology of HRM practice.
To elicit the views, attitudes and aspirations of Managers over different aspects of HRM practice
and to find out the nature of their influence over changing HRM policies at a local level.
To compare the visions and views of those managers in the U.K to the host company in Malaysia.
To identify the extent to which HRM practice in the host company is used in other parts of the
organisation to create competitive advantage.
8. Proposed methodology: An introduction to the philosophical stance
As an introduction to this section it is imperative to grasp an understanding of what research methods
are. They are detailed by Wong, (2011) ‘as being the rules and procedures which are ultimately
designed by social scientists to guide the enquiry of a problem in a systematic manner’. The
foundations of research within social sciences often lay around two major terms epistemology and
methodology. Both terms are developed from Greek philosophy of knowledge or how we come to
know. Epistemology is concerned with the philosophical stance of how we come to know and the
methodology is the practical application through the use of methods so that we come to know. The
question that underpins epistemology is whether the social world can be studied and examined in the
according to the same principles and processes as the natural sciences (Bryman, 2008). It can be
said by Olsen (2004) that it is of great benefit to the reader if they understand the researcher’s
philosophical position as this helps to situate the reader in terms of both argument and technique.
The foundation of an important philosophical stance in this field is by Crotty (1998) who simplified the
four elements of a research process. Each of these elements denoted as ‘informing’ the next so that
they would act as a flowing line of development throughout the research process. The first of these
elements being epistemology which was described as being, ‘what it means to know’. Within this
element Crotty enlisted objectivism, constructionism and subjectivism alongside their variants. The
second of these elements was theoretical perspective otherwise known as the philosophical stance.
This element includes positivism and post positivism interpretivism; symbolic-interactionism;
phenomenology; hermeneutics. In addition, critical inquiry, feminism and post modernism would sit
here. The third of the elements was characterised by the methodology, detailed here as the plan of
action or strategy. Within this element experiential research, survey research, ethnography,
phenomenological research, grounded theory, heuristic theory, action research, discourse analysis,
feminist standpoint research. Finally, the ‘Methods’ or ‘Techniques’ element. This section includes
sampling, measurement and scaling, questionnaires; observation (participant/non-participant),
Interview, focus group, case study, life history and narrative and this element is not exhaustive here.
These four elements would later act as the foundation for further work in this philosophical line by
Cresswell (2003) who stated that research allows the researcher to; make claims as to what
knowledge is (ontology); explain how they know what they know (epistemology); describe what values
goes into the research (axiology); determine how they want to write about it (rhetoric); and decide on
the processes for studying it (methodology). Clearly, from the standpoint of these two theorists alone
it is clear to understand that the theoretical or philosophical stance taken by the researchers will
clearly influence the researcher’s methodology and ultimately their choice of data collection methods
thereafter. This be can see in the work of Dunne, Pryor et al (2005) which demonstrate the varying
influences as interlocking sets. In the three sets there are; Ethical and Political Issues; Practical and
political issues; epistemological and ontological issues and central to all, the methodology.
9. Key concepts
It has been perceived that for many years within the field of science that experiments and knowledge
have been closely linked to the traditional approaches of positivism where observations and
experiments have justified methods and approaches to analysing knowledge. Positivism can be
described as the method for ‘advocating the application of the methods of the natural sciences to the
study of social reality and beyond’ (Bryman, 2008). This term is heavily dependent on the process of
collating facts and data to prove an objective through the analysis of tested hypotheses to confirm
knowledge.
Empiricism as described by historical authors such as Ayer (1946) and Carnap, 1966) is the theory
that knowledge is indeed derived from ‘sense-experience’ and empiricist-philosophers within the field
treat this science as the only source of genuine knowledge. That is using the human mind as a blank
337

Kate Rowlands
canvas and adopting the stance that we develop knowledge from our sensory experiences. This also
excludes the theory that knowledge can be acquired through entities which cannot be observed and
moreover that knowledge must be testable, factual and observable (Benton, 2001). It can be
determined that ideas must be tested rigorously before it is concluded that they can be considered as
knowledge. This view is clearly contested by theorists such as Feyerabend and Kuhn who claim that
science is a ‘playful learning’ where new meaning and understandings are grasped, attained and then
move onto something else (Williams, May, 1996). They perceive science as an evolving programme
of events and encounters and that there is not one singular best approach to science but a variety of
developing methods suited to the objectives in the context of the study.
There is also another view which is that of interpretivism. This view contrasts with the positivistic
approach as it aims to appreciate the ‘distinctiveness of humans’ as subject matter. It therefore needs
a different approach to assess this distinction within the human being as oppose to the natural order.
It could also be said that this approach takes into consideration the difference in the strategy required
to analyse these differences. Therefore, data can be assessed and differentiated in an interpretative
manner which contrasts to the more rigid approach of positivism.
There are also two ontological considerations within this discussion of the key concepts. The first is
objectivism. This is primarily concerned with the assumption that social phenomena have an
existence which is independent of social individuals. In the purpose of this study for example it can be
said that the organisation is an object which has processes, rules and legislation. The organisation
has a hierarchy, jobs are allocated through a labour system, it has a purpose and organisational goals
and objectives. The individuals within this organisation are constrained within the rigidity of its system
and they obey the rules, do their jobs and strive to achieve the goals but within this they are inhibited
and restricted by the organisation. It can then be said the same of cultures as they restrain and
demand that individuals who have a belief and values must act in a certain manner to exist within the
culture so they socialise within to function or participate fully. This is the objective reality as the culture
and organisation adopt a palpable reality of their own which is external to the individual. The opposing
approach is constructionism or also known as constructivism. This challenges the view that
organisation and culture are pre-given and develops the theory that social phenomena are not merely
shaped through social interaction but that they are continuously evolving and changing through social
interaction. It has also denoted that researchers over the years have developed their own
constructions through their own accounts of their research so they have created their own
interpretation of social reality as oppose to the existence of a definitive.
By contrast the term ‘realism’ exists as an epistemological consideration. There are two
considerations here; Empirical and Critical. The Empirical stance clearly based around empiricism
offers the view that through the application of suitable methods that reality can be understood. For this
study the critical realist position will be embraced. This approach was pioneered by a number of
authors in the 1970’s namely Harre who developed work based on realist philosophy of the natural
sciences and then Hesse’s work around models and metaphors in scientific thinking. These two
authors were also critical in the work developed in 1975 by Roy Bhaskar who is now renowned for the
pioneering theories of ‘critical realism’. This theory underpins the research in social sciences within
this field however there is much discord which still surrounds this approach.
Within the context of everyday life the term ‘realistic’ exists which can be perceived as meaning that
people claim to be realistic when they do not have high expectations of themselves or even the
events they participate in (Benton, Craib, 2001). It could be said that these people have a resigned
acceptance of the way things are within their world and that cannot live up to their hopes and dreams
which contrasts with fantasy, escapism and non-representational forms of expression. However,
Bhaskar within this theory concludes that critical realism takes three distinctive features which are; the
recognition that social science is a social practice and the knowledge of social science is a social
product; the recognition of the independent existence of the objects of scientific knowledge; an
account of scientific experiment and discovery as simultaneously material and social practices in
virtue of the former two elements are sustained (How, 2001).
The Critical stance supports the view that we need to understand the reality of the ‘events and
discourses’ in the social world within which we live through identifying the frameworks which generate
these ‘events and discourses’ (Bhaskar, 1989). The latter of these theories opposes positivism as a a
direct reflection of the reality within which the research is carried out yet critical realism offers the
338

Kate Rowlands
stance that it is merely an aspect of understanding the reality. Positivism relies heavily on the
systematic approach whereas critical realism appreciates implication and terms which may not be in
direct relation to the object of the study.
In the context of this study the critical realism stance is compatible with a variety of research
methodologies as it is can be said that this view is the half-way house between the extremes of
positivism and interpretivism. It would appear that although the philosophical stance may vary, the
depth of the study and the key element is the design of the research itself as there is no right or wrong
philosophical approach. The methods must be carefully formulated to ensure that the optimum
research is reached and that the methods are appropriate for the topic area and the framework which
has been chosen to work within.
10. Research methodology
Thus far a common approach to assessing workplace attitudes within this specific area of HRM
practice has been the use of the survey. This exists in many forms but a common type is whereby
employees are able to respond in a monosyllabic manner. This method is often termed as the
‘additive’ approach. For example the questionnaire may assess the use of HRM practices in an
organisation and therefore the higher the score on the questionnaire then the greater the existence of
HRM practice within the organisation and also positive employee attitudes (Guest, Conway, 2002).
There are also criticisms to this approach such as the ‘simple’ nature of its design as it fails to
examine in depth whether or not the HRM practice is of value, how well it is used and its
dissemination across the organisation (Fiorito, 2002).
A survey will be used to collate data for this study as it is considered to be a reliable and effective
method of data collection particularly when cross examining two sets of organisational data in the U.K
and Malaysia. This will provide a unitary approach which will yield data to support the specific
research questions posed. In addition to the survey it is hoped that the researcher will have access to
participate in interviews to support the survey findings and also to gain a greater depth of
understanding between the two comparative operations of HRM practice within an MNC.
11. Conclusions
To conclude, although this study is still in its infancy it can be perceived by the topics already
examined that there is an opportunity within this field to pursue the research aim and objectives
further. A point which is of notable interest and acts as the catalyst to this study will be the
methodologies adopted in order to achieve the desired outcome for the study. There has been much
critical analysis of the philosophies of research methodology, research rationale and objectives
carried out throughout this paper however whether they are appropriate and relevant will be
determined through the finality of the study itself. There has been much knowledge gained through
the discussion and research of methodologies within the context of this paper and it will be interesting
to review this work in a longitudinal format once the data has been collated to assess and reflect
further on the theoretical research choices made and their appropriateness in the research field itself.
References
Anheier.H.K. (2004) ‘Civil Society: Measurement, Evaluation and Policy’. London: Earthscan.
Ahmad.M.A, (2004) ‘ Malay images in economic affairs: viewing through the lens of a Malaya. Presented at the
second conference on Malay civilisation 20-28 February, Kuala Lumpur.
Ahmed.Z. Julian.C. (2009).’Doing Business In Malaysia’. Thunderbird International Business Review. Vol.51,
No.1. January/February 2009.
Armstrong.M. Brown.D. (2006). ‘Strategic Reward: Making it happen’. London: Kogan Page.
Aycan.Z. (2005). ‘ The interface between cultural and institutional/structural contingencies in human resource
management’. International journal of human resource management, 17 (7): 1083-120.
Bene.C. Lawton.R. Allison.E. (2010). ‘Trade Matters In The Fight Against Poverty’. World Development Vol.38,
No.7, pp933-54.
Blyton.P. Turnbull.P. (1992). ‘Reassessing HRM’. Sage; London.
Braunholtz-Speight.T. (2008). ‘Policy Responses to Discrimination and their Contribution to Tackling Chronic
Poverty’. Chronic Poverty Research Centre. University Of Manchester.
Bryman.A. (2008). ‘Social Research Methods’.Third Edition. Oxford University Press.
Cappelli.P. McElrath.R. (1992). ‘The transfer of HR practices through multinationals’. Paper presented at the third
conference of the bargaining group. (Berkely, 5-6 March)
Coates.B. David.R. (2002) ‘Learning for Change: the art of assessing the impact of advocacy work’. Development
Practice, 12 (3&4): pp. 530-541.
339

Kate Rowlands
Coe.N.M.Wrigley.N.(2007) ‘Host economy impacts of transnational retail: the resrach agenda’. Journal of
Economic geography, Vol.7, pp.341-71.
Country Focus. (July 2006). ‘ Consumer profile – Malaysia’. Asia Pacific Statistical Analysis,No.6-7 2006.
Curran.Z. (2005). ‘Civil Society Participation in the PSRP: the role of evidence an impact on policy choices’. PPA
Synthesis Study. London: Overseas Development Institute.
Dawson.J. (2003). ‘Model of the impacts of retail internationalisation’, in Dawson.J, Mukoyama.M., Choi.S.C,
Larke.R.’The internationalisation of retailing in Asia’. Routledge Curzon, London.
Edwards.M. (2001) ‘Global Civil Society and Community Exchanges: A Different Form of Movement’,
Environment and Urbanisation, 13(2): pp. 145-149.
Edwards.M. (2009) ‘Civil Society’: Second Edition, Cambridge.
Edgar.F, Geare.A. (2005) ‘HRM Practice and employee attitudes: different measures-different results’. Personnel
Review. Vol.34, No.5. Emerald Publishing Group.
Holst.H.P.(2009)’Cultural perspectives on the development of the retail trade, East and West. Asia Pacific
Business Review, Vol.15, No.1, 137-149.
Howell.J. Pearce.J. (2001) ‘Civil society and development: A critical exploration. Boulder: Lynne Rienner
Publishers.
Human.N. (1992). ‘Entrepreneurship In Malaysia. Technology Mara Selangor, Malaysia: The basics of
entrepeneurship.
Kaliappen.S.R.Alavi.R. Abdullah.K. Zakaullah.M.A.(2009)’Spillover effects of foreign hypermarkets on domestic
suppliers in Malaysia’. International Journal Of Retail & Distribution Mnaagement. Vol.37.No.3. Emerald
Publishing Group.
Kanungo.R.M. Jaeger.A.F. (1990). ‘Introduction: The need for indigenous management in developing countries’.
In Jaeger.A.M and Kanungo.R.M: Management in developing countries, London: Routledge, pp. 1-23.
Keegan.W.J. (2002). ‘Globalmarketing management’ (5 th edition). Englewood Cliffs, N.J:Prentice Hall.
Kent.J. (2008) ‘Workers at Tesco paid just 8p an hour’. Telegraph.co.uk.
Legge.K. (1995). ‘HRM; Rhetoric and Realities’. MacMillan, Basingstoke.
Liu.W (2004). ‘The cross-national transfer of HRM practice in MNC’s: An integrative model’. International journal
of manpower, Vol.25, No.6. Emerald Publishing Group.
Malaysian Industrial Development Authority (MIDA), Department of Statistics, Malaysia (2007, May 31).
Retrieved 12.10.2010 from http://www.mida.gov.
Mamman.A. Adeoye.B.(2010) ‘Performance Management in Multinational Companies in Africa: Implications for
transferability of Human Resource Management Innovation to Developing Countries. IDPM, University Of
Manchester,
Marsden.D. (1991). ‘Indigenous Management’. IJHRM, Vol.2 No.1.
Mitlin.D. Mogaladi.J. (2009) ‘Social Movements and Poverty Reduction in South Africa’, SED research paper,
http://www.sed.manchester.ac.uk/research/socialmovements/publications/reports.
Oakley.P. (2003) ‘Strengthening Civil Society: Concept and Approaches’ in B.Pratt, Changing Expectations: The
Concept and Practice of Civil Society in International Development. Oxford: INTRAC: pp.31-42.
Philips.E,M. Pugh.D.S. (2010). ‘How to get a PhD: A Handbook for students and their supervisors’. Fifth Edition.
Open University Press.
Retail Asia (2002). ‘Tesco jostles for position in Malaysia’s hypermarket industry’. Retail Asia, June.
Taylor.H. (2010) ‘Human Resources In Development Working Papers No.2’, IDPM. University Of Manchester.
Thapa.G. Gaiha.R. Imai.K.S. Kulkarni.V.S. (2009) ‘Soaring Food Prices: A Threat or Opportunity In Asia?’ BWPI
Working Paper 69. Brooks World Poverty Institute. University Of Manchester.
The Malay Mail (2006). ‘Tesco opens up more space for local traders’. The Malay Mail.
Townley.B. (1994) ‘Reframing HRM: Power, Ethics and the subject of work’. Sage; London.
Van Rooy.A. (1998). ‘Civil Society as Idea: An Analytical Hatastand?’ in Van Rooy.A, Civil Society and the Aid
Industry. London: Earthscan.
Walker.A. Chakie-Wong.C.(2006) ‘Asean Economic bulletin’. Vol.23.No.23.Dec
Warr.p. (2006). ‘Povert and Growth In South esat Asia’. ASEAN Economic Bulletin. Vol.23, No.3.
Wu.C. Lawler.J.J. Yi.X. (2008). ‘Overt employment discrimination in MNC affiliates’. Journal Of International
Business. Vol.38, pp772-794.
Yin.R.K.(1994) ‘Case Study Research: Design and methods’. Second Ed. Applied Social Research Methods
Series, Vol.5. Sage.
340

Beyond the Glass Ceiling: A Gendered and Cultural
Hospitality Management Discourse on The Advancement of
Women Based on Integrated Research Paradigms
Faith Samkange and Sihle Dingani
Swiss Hotel Management School, Switzerland
fsamkange@shms.com
Abstract: Historically, development politics is littered with leadership failures (Pretorius and Roux 2011). Who
wields power in the higher echelons of leadership? Research though negatively skewed against developing
economies suggests visionary, innovative and competent leadership has never been more urgent and yet
economic leadership remains the prerogative of privileged men (IlO 2010 and Kjeldal 2008 World Economic
Forum 2011). Should leadership competency remain a gendered privilege? Is the continued marginalization of
women in leadership justifiable and affordable given the current economic and political global challenges? While
addressing these questions this paper argues engaging competent leadership is a complex challenge that should
transcend gender boundaries. Research has since established educational, sociological, cultural and legal
barriers perpetuate gender leadership imbalances (IlO 2010 and Kjeldal 2008). However, the applicability of
these factors in the hospitality industry has not been fully explored. The situation in Africa particularly Zimbabwe
warrants attention. Based on a case study of selected hotels, this research assesses the extent to which women
are breaking through the hospitality management glass ceiling … or are they? It further examines the sociocultural
influences emerging and their impact on policy implementation. Personal experiences in community
development and gender empowerment research in Zimbabwe and Africa acknowledge the cultural and political
sensitivity of pursuing gender discourse raising questions: Is there a research methodology equally sensitive?
Can a singular methodology address such sensitivity while embracing the cultural complexity implied? These
methodological challenges implied demand alternative integrated and contextualized research paradigms.
Consequently, this study rejects the fallacies of rigid and prescriptive research perspectives often responsible for
the flawed articulation of political and economic issues in Africa. Flawed articulation of issues impacts negatively
on development interventions (Samkange 2011). This paper examines how critical theory (Marx 1994) blends
with cultural theory (Hofstede 2001) to develop an integrated contextualized research methodology used to
explore the gender development issues identified.
Keywords: gender equality, culture, integration, contextualization
1. Introduction
The Zimbabwean case study examines gender equality initiatives based on cultural values. Cultural
traditions marginalize women in leadership especially at work. Arguably, this marginalization is an
unjustifiable luxury no longer affordable in Africa, a continent continuously struggling for an economic
breakthrough. The history of development economics and leadership in Africa is littered with failures
putting economies at a global competitive disadvantage (Pretorius and Roux 2011). Globalization
assumes a level playing but the reality demonstrates the increasing vulnerability of developing
economies. Maintaining a competitive urge in this environment demands visionary, innovative and
competent leadership otherwise many African states risk nurturing the “third world” status for eternity.
Who is making it to powerful leadership positions? Research though negatively skewed against
developing economies is unanimous regarding the lack of gender visibility in the higher echelons of
power (Linge, Van Rensburg and Sikalieh 2010). Engaging competent leadership regardless of
gender is urgent. Zimbabwean, women left indelible marks in the annals of history competently
fighting alongside men to replace colonialism with egalitarian socio-economic systems. Their
continued marginalization where it matters most…the higher echelons of power is both
counterproductive and unjustifiable. Although gender equity initiatives are evident (Dumont,
Stojanovska and Curveyrs and 2011, Smith 2006) studies have not fully explored gender imbalances
in the hospitality industry with a convincing cultural perspective highlighting the complexity of
researching into this area.
Appropriate intervention is driven by contextualized research. Are the complex socio-cultural and
political dynamics of leadership within the Zimbabwean context well understood? If so, does gender
development intervention research embrace these complexities? Given the much-publicized turbulent
economic, socio-political backdrop, dismantling the rudiments of gender discrimination should be
prioritized to leverage the polarized economy. And yet gender equality discourse has never been so
sensitive and complex. This paper is designed to provoke and intensify gender equality discourse at
the work place as a methodology for change. It articulates the methodological issues arising when
341

Faith Samkange and Sihle Dingani
assessing the extent to which women are breaking through the hospitality management glass ceiling
… or are they? The historical, cultural, political and legal influences emerging and their impact on
policy implementation are explored using an integrated research paradigm. Methodologically,
discourse exposes the socio-cultural challenges impacting intervention. This paper therefore
examines the research methodology applied based on the evidence from the study.
Culture, a fundamental concept in this study, involves the socialization of human beings. It collectively
programmes the mind implanting values systems that distinguish members of one group from another
(Hofstede 2001). Is there any gender-balanced methodology for determining these values systems?
The nature and composition of leadership is critical in this respect because it is the subject of
investigation in this study. There is consensus (Connell 2006, Dumont, Stojanovska and Curveyrs
2011) men dominant leadership with a few exceptions. They, in reality, determine the value systems
handed down through the generations. Schwartz (1992) describes culture as rich complex of
meanings, beliefs, practices, symbols, norms and values that people live by. Tradition demands
adherence to these cultural values thus institutionalizing and perpetuating gender discrimination. This
could explain why men find relinquishing leadership difficult. This study problematized a gendered
discourse, which ultimately seeks to promote the sharing of this power.
2. Population and sampling
The field study undertaken used a purposive sample of 6 luxury hotels in Harare stratified according
to ownership, star rating and location. A total of 30 subjects including executive managers, human
resource managers, female managers and technical supervisors, key informants in commerce and
industry and specific government departments participated in this study. The idea was to broaden the
spectrum of participants in order to capture as many women in management levels as possible. Key
informants included people particularly women in influential positions chosen conveniently.
Management research insists on targeting existing managers, mostly men (Mwana 2002). Can men
articulate gender issues without bias? Selecting the most appropriate research subjects given the
scarcity of women into leadership position is a methodological challenge. Consequently management
research facilitates the institutionalization of gender discrimination by focusing on those in the
forefront of management… the men. This research sought a balanced articulation of issues and
hence problematized the empowerment of women. A contextual analysis of the work situation was
therefore necessary to identify the potential women leaders operating in the peripherals of
management. The deployment of cultural dynamics to propagate the value of gender research and
empower participants to speak out is a significant principle of development research (Desai and
Porter 2006). An open minded and yet creative enquiry was explored. Culturally and ideologically, the
choices undertaken to promote dialogical discourse in this study are philosophical and
epistemological.
3. Philosophical and epistemological underpinnings of the study
The philosophical and epistemological roots of gender development should be examined to
appreciate the methodological choices taken in this study. A growing body of knowledge articulates
the significant linkage between research philosophy and methodological choice (Gray 2009, Bryman
and Bell 2008, Saunders, Thornhill and Lewis 2010). An analysis of the research problem reflects a
specific focus on leadership in the Hospitality industry suggesting scientific and sociological
epistemological underpinnings. This position resonates with Lashley and Morrison (2002) who
contend Hospitality Management borrows heavily from social sciences. The advancement of women
into leadership is a multi-dimensional research variable. Historically Zimbabwean women abandoned
cultural roles and took up arms to liberate Zimbabwe from colonialism. Although the political origins of
power suggest sociological underpinnings, this peculiar Zimbabwean history cannot be ignored.
Proponents of gender discourse agree traditional power politics has been instrumental in
institutionalizing gender discrimination (Wag 2011). This denotes strong anthropological
underpinnings. However the developmental nature of leadership and how access to education and
experience influence the power dynamics in Zimbabwe reflect economical and educational
foundations. Emancipation starts from the mind according to (Desai and Porter 2006). It can be
argued the emancipation of women into leadership is informed by the study of Psychology. The use of
legislation as an emancipatory tool (Government of Zimbabwe 2011) suggests a legal epistemological
grounding.
342

Faith Samkange and Sihle Dingani
Philosophically it is difficult to determine one specific form of knowledge particularly theory to guide
and inform this piece of work. Can one individual researcher posses all the forms of knowledge
essential? The need for collaborative, open minded and creative enquiry under these circumstances
is well documented (Gray 2009, Creswell 2003, Silverman 2009). In reality though, an analysis of
gender development research demonstrates a narrow epistemological perspective and thus so far
failed to embrace the multifaceted epistemological underpinnings of the issues implied. Perhaps this
could explain why despite concerted efforts, women continue to operate in the margins of leadership.
Methodologically no “one size fits all” research approach can embrace the broad philosophical and
epistemological demands of gender development research. This study therefore adopted a gendered
and integrated approach to research to articulate multi-faceted philosophical and epistemological
underpinnings of the gender development issues at stake.
4. Integrating research approach and theoretical framework
Social and cultural theory explains gender differentiation as a phenomenon entrenched in cultural
value belief systems (Schalkwyk 2000, Hofstede 2001, Schein and Mueller 1992). Research
methodology was therefore grounded in these cultural values systems. This is reflected in the choice
of words to use during the conduct of the research, where and how to meet the participants, how to
generate useful discourse, what questions to ask and how to respond to the various challenges
emerging. Research methodology was therefore located within the cultural context of Zimbabwe.
The Cultural Orientation theory (Schwartz 1992), the Corporate Cultural theory (Woolliams and
Trompenaars 2003) and Think Management Think Male theory (Schein and Mueller 1992) explain
societal, organisational and individual values. While these theories provided an insight into cultural
issues and perceptions they were not universal and easy to apply within the Zimbabwean context.
The Cultural Variability theory (Hofestede 2001) was therefore considered.
Cultural variability theory (Hofstede, 2001) assumes people carry ‘mental programs’ developed in
early childhood. These mental programs contain components of national culture expressed in
predominant values. Among these values “Power Distance” and “Masculinity” are significant in this
study. Power Distance emphasizes the distribution of power between top leadership and
subordinates. Masculinity describes the strength of personality and attributes usually associated with
men while downplaying the delicate expressive and caring nature of the feminine role in society. The
strength of Hofestede‘s theory lies in its universality even though McSweeney (2002) challenges the
assumption that every micro-location is typical of national culture. In the Zimbabwean situation it is
true the majority of the people living in the rural areas are less adulterated by other cultures. However
cultural diversity exists in Harare, an urban area where the study was conducted. This geographical
focus encouraged a consistent cultural approach to research. Hofestede‘ s theory is flawed
philosophically because it perceives culture subjectively and yet empirically reduces it objectively to
numerical values. The relevance and accuracy of value measurement in Hofstede‘s theory can be
challenged given the dynamism and complexity of culture as a research variable. However, it is
argued Hofestede‘s work has withstood empirical scrutiny over the years (Jones (2007). This
argument justifies the application of the Cultural Variability theory in this study. However, to counteract
the flaws identified and strengthen the credibility of this research social and critical theory with
particular reference to critical theory (Marx 1994) is also applied.
Critical theory (Marx 1984, Webber 2000) underscores the significance of power distribution and
concurs with the high power distance dimension in Hofstede‘s (2001). It recognizes power is derived
from wealth emanating from ownership of the economic means of production. Critical theory thus
subscribes to the equitable distribution of wealth and power to achieve a classless society. Webber‘s
social stratification theory highlights the linkage between economic wealth and social status. He
reiterates the significance of redistribution of power and wealth (Webber 2000). A social system with
class distinction discriminates and exploits the poor (the proletariat). Most women are devoid of power
because of limited participation in the social economy. They rarely own land and other means of
production. Critical theory in particular Marxism therefore classifies them as proletariat while
condemning their exploitation by the privileged few (bourgeoisie). Although Marxism is criticized for its
weak articulation of sexism and race as forms of exploitation (Belkhir, 2001) it explains the basis of
gender inequality in Zimbabwe. Critical theory emphasizes that the gender playing field has never
been level hence the study targets the insignificant powerless players in management, the women.
Most of these women were hidden behind the powerful man. Indigenous knowledge systems were
343

Faith Samkange and Sihle Dingani
critical in locating these women. Collaboration was fundamental in creating opportunities for tapping
into these systems.
The deductive use of theory was useful in determining levels of advancement and explaining the
factors impeding progress. However this approach was too limited to articulate the casual linkages
between these factors and their impact on policy formulation and implementation. Using these casual
linkages to develop emancipatory strategies for the advancement women into the higher echelons of
power requires a combination of approaches. Consultative and integrated approaches are necessary
to yield better ideas regarding the nature of the intervention programmes most effective. Development
theorists argue action research is most appropriate in this case (Desai and Porter 2006). The
Zimbabwean context presents peculiar political, cultural and developmental challenges. It is therefore
clear one linear approach applied in a fragmented manner could not yield desirable research
outcomes.
5. The research strategy and ethical issues
Addressing gender development issues within the Zimbabwean context, given the broad philosophical
and epistemological underpinnings is a daunting task. Methodologically, the study should
acknowledge the cultural and sociological diversity implied in the research variables. Peculiar
development dynamics dictate cautious and sensitive methodological navigation. Personal
experiences in community development and gender empowerment research in Zimbabwe and
elsewhere in Africa recognizes the cultural and political sensitivity of pursuing gender discourse
(Samkange 2011 and WAG 2011, Masiyiwa 2011). Is there a research methodology equally
sensitive? Can one method embrace such sensitivity and cultural complexity at the same time?
Gender discourse is sensitive because it is emotional and exposes the cultural vulnerability of
individuals. It challenges the rudiments of culture upon which Zimbabwe thrives as a society.
Therefore it is not surprising that until recently, this discourse has been the privilege of a few
educationally and politically empowered individuals particularly women. These proponents of gender
equality are perceived as cultural renegades whose motive is to create cultural and social
disharmony. It is against this background that this research initiative sought to create an environment
in which people can freely pursue gender discourse without being stereotyped.
Politically, Zimbabwe has been in the spotlight for the wrong reasons. Consequently the media is
often accused of blowing issues out of proportion. Any research agenda should be perceived as
genuine and not designed to create bad publicity in a country struggling to manage reputational risk
against a wide range of odds. Research ethics demand best practice to avoid misrepresentation of
issues. Misrepresentation of issues creates ethical dilemmas detrimental to the subjects and the
organisations involved (Creswell 2003). The need to safeguard the interests and well being of
research participants including specific organisations is a fundamental ethical issue. The significance
of ethical etiquette in this case cannot therefore be overemphasized (Gray 2009) hence meticulous
management of the research process is important. Informed consent was sought from the
organisations and the participants. Confidentiality and anonymity was also guaranteed. Briefing
sessions were conducted as part of the feasibility study to sensitize participants on the research
agenda and its execution. Research scholars argue this is best research practice (Bryman and Bell
2009). Transparency demands debriefing is pursued at the right moment to inform interested parties
of the findings, the consequences and benefits accruing. This paper creates an opportunity to
disseminate findings.
Pursuing management research within the Zimbabwean context prioritized effective articulation of the
cultural issues at stake and the related indigenous knowledge systems as part of the research
process (Samkange 2011, Masiyiwa 2011). Integrated and contextualized research paradigms were
considered appropriate in this case. Integration and contextualization mean developing a broader and
holistic research perspective that relates directly to the peculiar research context. Research in this
case underscores the role and place of the local people in creating a better understanding of the
context in which research issues emanate (Silverman 2009). A consultative approach based on
indigenous knowledge systems was undertaken.
The notion of indigenous knowledge systems in this case resonates with a body of knowledge and
practices, norms values and attitudes regarding the Zimbabwean culture. Included in the systems is
not only the language but a set of social and political skills values, knowledge and attitudes required
to understand the Zimbabwean context and how it operates. Development research argues this is
344

Faith Samkange and Sihle Dingani
necessary to perceive issues correctly and determine the most appropriate forms of knowledge and
modes of communication (Desai and Porter 2006, Silverman 2009). Aspects of this research were
therefore participatory in nature. This was essential to encourage the levels of collaboration required
to source the necessary data. Collaborative engagement can facilitate an accurate interpretation of
the research environment. This viewpoint is endorsed by (Cornell 2006) who asserts collaboration
with the right partners can help manage the political and cultural challenges emerging from the
research context. Desai and Porter (2006) concur with this methodological position advising that
working, as a team is certainly more productive than working in isolation. Retrospective analysis of the
research process reflects that collaborative engagement was complementary to the effective
deployment of indigenous knowledge systems and application of cultural, social and critical theories.
This form of integration encouraged informed methodical choices.
6. Research method
Gray (2009) and Creswell (2003) highlight the relationship between research philosophy and research
method. The choice of research methodology in this study was carefully considered after examining
the broad philosophical and epistemological underpinnings of the research problem. The implication is
that research method should embrace the different forms of knowledge (Lashley and Morrison 2002).
In this case research variables are therefore a critical determinant of research method. Part of this
study sought to determine the levels of advancement into the higher echelons of power. The
knowledge sought in this case was factual, objective and required quantification in terms of
advancement levels. Creswell (2003) reasons in such cases quantitative method is appropriate. The
study also examined the factors influencing levels of advancement into leadership positions. This
knowledge is abstract and subjective and therefore not tangible and measurable. According to
Silverman (2009) this provides a rationale for the application of qualitative method. Similar studies in
Kenya have endorsed the use of qualitative method to explore the subjective nature of the barriers
impeding progress in the advancement of women (Dumont, Stojanovska and Cuyvers, 2011).
However, this study went beyond the determination of factors towards the articulation of the casual
linkages emerging with implications for policy formulation and implementation strategy. This was the
most complex part of the research. Fragmented liner methodological approach could not address
such complexity. An inclusive and integrated methodology was therefore necessary. Blended and
Integrated application of quantitative and qualitative methods was necessary in this case. This
integration informs the choice and application of research tools.
7. Instrumentation
Blending cultural and gender theory into indigenous knowledge systems in order to articulate the
complexity of gender development variables was one of the greatest challenges of this research. The
difficulty of determining which research tools, processes, procedures, and techniques would be most
useful is consistent with emancipatory, participatory and action oriented research paradigms
(Silverman 2009). An accurate perception of the research context determines the appropriateness
and relevance of research methodology. The fallacies and pitfalls of prescribing research
methodology out of context are well documented in (Samkange 2011). Methodological weaknesses
account for numerous blotched interventions in developing contexts (Botswana Government 2011). A
feasibility study was therefore undertaken to understand the nature of the research environment and
the circumstances under which specific research tools and procedures could be applied. The focus
was therefore on what works best, and how the process could be managed to maximize the data
generation.
Triangulated data collection processes and procedures including telephony, web-based and personal
face-to-face interaction were deemed appropriate to solicit rich data sets. Questionnaires, formal and
informal interviews and focus group discussions were conducted while hotel websites and
documentation where consulted to maximize data generation. According to (Gray 2009) triangulation
is necessary to check and manage biases while navigating the political and cultural minefields of data
collection. Triangulation therefore highlights the significance of diversity, flexibility and open
mindedness while objectively managing research biases (Porter and Desai 2006). Triangulation of
research tools, processes and procedures and their application in this study is illustrated in the
diagram in figure 1.
345

Faith Samkange and Sihle Dingani
Figure 1: Integration and application of research tools
8. Application of research tools and influence on findings
A historical analysis of government documentation established the development of a robust legal
framework (Zimbabwe Government 2001, WAG 2011 and Manyoni, 2010). Consequently some inroads
towards the advancement of women into government leadership are evident. The Zimbabwean
government boosts of female vice president and vice prime minister and, 9 ministers out of 40.
Determining replication of this development the highest echelons of power in the hospitality Industry
demanded a different se of tools. A questionnaire, observation and a study of the organisational
charts were applied.
A summary of the economic participation of women is shown in table 1. Notably no representation of
women at executive management level is visible. Cooperate websites were consulted to establish
female representation in boardrooms. Out of 6 corporate organisations no female representation was
reflected at chairperson level even though 2 of these corporates had at total of 3 female committee
members. This lack of visibility at the highest echelons of power confirms the existence of the glassceiling
phenomenon (Dumont, Stojanovska and Cuyers 2011 and Connell 2006). Arguably, the
progress evident in government is not replicated in the hotel industry. Findings therefore collaborate
leadership potential remains idle (World Economic Forum 2010).
Table 1: Summary of gender participation in hotels
Gender
Male
Female
Overall
Participation
%
73
27
Managerial
Positions %
71
29
Promotion in
last year %
70
30
Executive
positions %
100
0
Board of Directors
Chairperson positions%
Total 100
100
100
100
100
Formal and informal telephone, Internet based and face-to-face interviews including group
discussions established the significance of socio-cultural barriers despite the existence of a robust
gender development legal framework. These findings are consistent with (Linge, Van Rensburg and
Sikalieh 2010, Mwana 2002). Consistent with Desai and Porter (2006) informal interaction with
participants to encourage illuminative discourse took place in relaxing socio-cultural environments.
346
100
0

Faith Samkange and Sihle Dingani
Popular local cuisine outlets (kwaMereki) were used as springboards for productive discourse.
Zimbabwean culture resonates with collectivism (Hofestede 2001) hence participants were
comfortable in familiar social environments where food is used as a social currency. Female voices
put the tone and mood of gender discourse into proper perspective thus:
Laws “were not enforced” and had no impact on my achievement. “… I overworked myself to prove I
was worthy”. Legislation “ seems to persuade rather than enforce” implementation. Laws should be
“stricter … why not implement quotas?” “I am qualified and experienced but it was difficult to get
promoted into leadership”. “I constantly have to prove I am as capable as my male counterparts ”. As
a woman I had to play “the right politics” to get promotion”. Discrimination persists even though “
labour act forbids discrimination”. The other problem is that “your husband wants you home at a
certain time. Sometimes “you have to forgo promotion… the demands at home do not allow you to
hold leadership positions”.
These few selected voices illustrate policy implementation problems and the socio-cultural issues
emerging as part of the glass-ceiling phenomenon impeding progression into leadership. A linear
methodology could not have captured the levels of advancement while at the same time articulating
the factors including the role conflicts that makes it difficult for women to maintain an enabling work
life balance let alone advance into the higher echelons of power so vividly.
9. The challenges and the opportunities
Integration of the research perspectives and triangulation of process, instrumentation and the
emerging data encouraged effective management and articulation of the complex cultural and political
issues while creating opportunities for cross checking the data generated. This enhanced validity and
reliability by addressing possible researcher biases arising from interaction within a familiar research
context. Integrating research paradigms created a cohesive understanding of the research issues at
stake. Compartmentalization and fragmentation of knowledge not compatible with the complexity and
sensitivity of the research issues was thus avoided (Desai and Porter 2006). The major challenge of
this methodology is associated with high material resource inputs, advanced research skills and time
constraints. Blending the research process with the local cultural systems was difficult. The inherent
political and cultural minefields emanating from the sensitivity of the research context required careful
management. Meticulous planning and preparation including impeccable execution could not be
avoided (Bryman and Bell 2009).
10. Conclusion
Research plays a critical role in dismantling retrogressive and exploitative the cultural monopolies that
institutionalizes gender discrimination and perpetuates powerlessness among women in Zimbabwe.
The think management think male mentality (Schein and Mueller 1992) is inconsistent with the gender
inclusive Zimbabwean liberation history. Zimbabwe needs competent leadership to leapfrog the
delicate economic recovery process. Marginalizing women in sector with great potential for generating
the much-needed foreign currency earning is a luxury no longer affordable. Although cultural and
political sensitivity pose methodological challenges developing research in this respect, a
contextualized and collaborative approach can yield results (Kaufmann 2008). However, political
cohesion could facilitate better coordination of the gender development research required to
dismantle the cultural barriers working against women. This study unearthed glaring gender
imbalances in the high echelons of leadership and examined related factors and their impact on policy
development and implementation based on integrated research perspectives. Further research could
consider a broader sample based on collaboration and integration to create contextualized
interventions that could facilitate greater visibility of women in the highest echelons of power.
Longitudinal action research involving a wider range of interested parties with a broader research
skills base could be pursued.
References
Belkhir, J. (2001) Race, Gender & Class Special Issue on Marxism and Race, Gender & Class, Vol 8, No 2, pp
23-33.
Botswana Government. (2011) Gender Equity, Botswana Centre for Human Rights, Gaborone.
Bryman, A. and Bell, E. (2009) Business Research Methods, Oxford University Press, Oxford.
Connell, R. (2006) “Glass Ceilings or Gendered Institutions? Mapping the Gender Regimes of Public Sector
Worksites”, Essays on Equity, Gender and Diversity”, pp 837-84
347

Faith Samkange and Sihle Dingani
Creswell (2003) Research Design: Qualitative and Quantitative , Mixed Methods Approaches, Sage Publications ,
London
Desai, V. and Porter, R.B. (2006) Doing Development Research, Sage Publication, London
Dumont, M; Stojanovska, N. and Cuyvers, L. (2011) “World Inequality, Globalization, Technology And Labour
Market Institutions” South Africa International Journal of Manpower Vol 32, No3, pp 257-272.
Gray, D. (2009) Doing Research in the Real World of Work, Sage Publications Ltd, London.
Government of Zimbabwe, (2011) Constitution of Zimbabwe, Kubatana Constitution, Harare.
Hofstede, G. (2001) Culture's Consequences: Comparing Values, Behaviours, Institutions, and Organisations
Across Nations, 2nd ed. Sage Publications Inc., London.
Jones, M. (2007) “Hofstede - Culturally questionable”, Faculty of Commerce, Vol1, No1, 1-9.
Kaufmann, G. (2008) “Testing the "Glass Ceiling" Effect on Gender Differences in Upper Level Management: The
Case of Innovator Orientation”, Organisation and Leadership, pp 1-7.
Kjeldal, S. Rindfleish, J. Sheridan, A. (2008) “Gender Inequality in Academia: the Role of Informal Rules and
Practices” Women in Management, Vol 1,no 39, pp1-11.
Lashley,C. and Morrison, A. (edit) (2002) In Search of Hospitality: Theoretical Perspectives and Debates,
Butterworth-Heinemann, Oxford
Leung, K., Bond, M. De Carrasquel, S. Muñoz, C. (2003) “Social Axioms: The Search for Universal Dimensions
of General Beliefs About How the World Functions”, Journal Of Cross-Cultural Psychology, Vol 33, No 3, pp
286-302.
Linge, T. K.; VanRensburg, W. and Sikalieh, D. (2010) “The role of Visibility and Career Advancement Of Women
in Kenya”, The Journal of Language, Technology & Entrepreneurship in Africa, Vol 2, No1, pp1198-1279
Manyoni, F.S. (2010) Women and Youth in Business. Zimtrade, Harare,
Marx, Karl. (1994) "Theses on Feuerbach. in L. Simon, ed., Karl Marx.” Selected Writings, Hackett, New York.
Masiyiwa, E. (2011) Women's Action Group (2011) Kubatana, Harare.
Mwana, H. A. (2002) Think Manager Think Male: does it apply to Zimbabwe? University of Zimbabwe, Harare.
Powell, G. Graves, L. (2003) Women and Men in Management, 3 rd Edition, Sage Publications California
Pretorius, M. and Le Roux, I. (2011) “Successive Failure, Repeat Entrepreneurship And No Learning: A Case
Study”, SA Journal of Human Resource Management/SA Vol 9, No1, pp 23-249.
Samkange, F. (2011) “A Conceptualised and Contextualised Methodology for Interrogating
ComplexTechnological Development Issues”, Proceedings of the 10 Th European Conference on Research
Methodology in Business Academic Publishing Ltd. UK.
Schalkwyk, J. (2000) Culture, Gender Equality and Development Cooperation. Canadian International
Development Agency, Toronto
Schein, V. and Mueller, R. (1992). “Sex Role Stereotyping and Requisite Management Characteristics: A Cross
Cultural Look”, Journal of Organisational Behaviour, Vol 13, No 5,
439-447
Schwartz, S. (1992) “Universals In The Content And Structure Of Values: Theoretical Advances And Empirical
Tests in 20 countries”, Advances in Experimental Social Psychology, Vol 25, no 1, 1 – 165.
Sickels, R. (2004) The 1940s Greenwood Publishing Group Inc., Connecticut.
Silverman, D. ( 2009) Qualitative Research: Theory method and Practice, Sage Publications, London.
Smith, E. (edit) (2006) “Gender Inequalities in Kenya”, Unesco.
Trompenaars, F. and Woolliams, P. (2003) Business Across Cultures, Chichester, Capstone.
Williams, W. Martin, L. (2010) “What Happened to the Glass Ceiling? “ Gender Equity in theWorkplace, Vol 1,
No1, 1-2.
World Economic Forum (2006) “The Global Gender Gap Report 2006” World Economic Forum. World Economic
Forum (2010) “The Global Gender Gap Report 2010”. World Economic Forum
Yeganeh, H. and May, D. (2000) Cultural Values and Gender Gap: A Cross National Analysis,Emerald Insight.
Zimbabwe Ministry of Women Affairs (2011) “Gender and Community Development Progress Report of the
Republic of Zimbabwe” United Nations Economic Commission for Africa.
348

An Amalgamation of Grounded Theory Method and Object
– Orientation Concept: A Practical Approach for Theory
Development
Transmissia Semiawan
Politeknik Negeri Bandung (Bandung State Polytechnic), Bandung, Indonesia
transmissia@jtk.polban.ac.id
Abstract: This paper is about an implementation of Glasser-Strauss’ Grounded Theory (GT) method from the
perspective of Object-Orientation (O-O) concept. The paper will discuss the way in which the author understands
the philosophical as well as the methodological aspects of the method. The philosophical part emphasizes the
concept of theory development process. The theory is an encapsulation of personal insights and its real life
experiences. It is a conceptual abstract process of identifying phenomena in terms of categories, properties and
hypotheses as the elements of the theory. The methodological part explains the way in which the theory – as the
object – is generated by means of constructivism and interpretive approaches of understanding people's mind.
Based on these approaches, the O-O concept is carried out through an abduction process rather than induction
or deduction one with respect to Charmaz’s idea of construction of meaning. The amalgamation of GT method
and O-O concept was carried out to develop a theoretical based Information Management (IM) model for Higher
Education (HE) institutions. The model was developed based on a study of handling complexity of informationbased
activities throughout HE institutions. Since it involves the human nature of knowledge, the model has been
developed with the underlying epistemological assumption. The model encompasses four interrelated
phenomenological categories, which are: Academic Life, Organisational Culture, Work-unit and Communication
Metaphor. As a result of conceptual abstract process the four categories are significantly based on the
fundamental concepts of sharing information, communication, learning and knowledge development, resourceuse
and communication across the organisation. From the perspective of O-O concept, the fundamental
concepts suggest common properties and behaviours that bring about specific characters to each of the
categories so that each of them conceptually denotes a specific paradigm in relation to their designation. Linking
all the paradigms at their properties level, the interrelated paradigms provide a big picture, in terms of a
theoretical framework, of IM in a HE institution.
Keywords: grounded theory method; object orientation concept; theory development, information management;
higher education
1. Introduction
Grounded theory (GT) method has been recognised and progressively used across a wide range of
disciplines and subject areas. Practices of using the method have also been discussed intensively
among researchers since the GT method was developed. For the most part, practice of using the
method has been considered challenging by researchers as it deals with numbers of abstract
processes as well as fundamental theories and concepts. Accordingly it is essential to understand the
underlying philosophical idea of the method before applying it in a study project.
This paper explores the process of generating theory by looking at Glaser and Strauss’ main idea of
‘Insights, Theory Development and Reality’ (Glaser & Strauss, 1967). The author is challenged to
combine object-orientation (O-O) concept to GT method in approaching construction of meaning
process towards the development of theory. The discussion will emphasize the way in which the O-O
concept is made use of in terms of applying and practicing it in the theory development process within
which the construction of meaning from the reality is concerned.
The practice of this amalgamated approach in a study of generating a theoretical based Information
Management (IM) framework for Higher Education (HE) institutions will also be discussed. This
includes the way in which the framework is developed from the data by means of the core process of
comparative analysis through which the data is conceptualised and integrated to form a theory using
the coding process (Strauss & Corbin, 1998). By the use of the coding process, elements of the
theory – categories, properties and hypotheses – are then identified from the data and subsequently
are conceptualised as the theory.
2. An understanding of GT method
The debate regarding the practice of GT method has been happening since the founder of GT method
– Glaser and Strauss – had different perspectives on the process. This has impact on the
development and refinement of the method in relation to its epistemology as the underlying
349

Transmissia Semiawan
philosophical assumption of GT method as a qualitative method. The impact brought GT method in
two epistemologies: positivist and constructivist. Consequently, with the dual root of its epistemology,
the GT method has been adopted in at least two major different approaches with their specific
characteristics.
With reference to the dual root of its epistemologies, GT method is contested and therefore a flexible
concept that researchers may develop and adapt in terms of applying and practicing it in different
ways in the theory development process (Bryant & Charmaz, 2007). Considering this idea, the author
has implemented GT method using constructivist approach but from the perspective of O-O concept.
2.1 The philosophy of GT method
The texts that the author found very useful from the book ‘The Discovery of Grounded Theory’ are in
the last chapter which is about the three basic elements of GT method: Insights-Theory Development-
Reality. To the author, this is the philosophical process of theory development which not only gives
her an understanding of GT method as a whole but also guides her to apply the method at ease. In
general, GT method enables researchers to develop reality into their insights and conceptualise it
towards the generation of a theory and its elements, and then arrive at the theory that will be of use to
others.
To understand the philosophical process of GT method above, Glaser & Strauss (1967) explain a way
of looking at:
how researchers use comparative analysis to develop reality (i.e. experiences) into insights;
how insights then turn out into theory; and
how the theory can be further referred to by other people in building insights
To carry out the above, Glaser & Strauss (1967) again suggest three strategies through which
researchers make adjustment and adaptation to their surroundings. The strategies include:
understanding the existing knowledge that researchers have – at this point the researchers
should be aware of what knowledge they have and should be able to ‘see’ what is going on
(Glaser, 1992, p.22) within the real world around them;
doing ‘cultivating insights’ – this is the way the researchers gain knowledge by gathering
experiences from the world around them and picturing the situation or problems that exist therein;
this strategy together with the first one denote the idea of ‘constant comparative analysis’
applying their enhanced knowledge – at this stage the researchers should be able to position
themselves and explore their knowledge within the external world in order to develop theory with
respect to the real world around them; this point describes the idea of ‘fit, work, relevance, and
modifiable’;
The use of the three strategies which should blur and intertwine continually directs researchers in
developing the theory by means of conceptual abstract process, comparative analysis, and coding
process. Figure 1 shows the author’s comprehension of the philosophical process of theory
development with respect to the use of GT method as a research method within the entire flow of
research work.
2.2 GT method as a research method
As shown in Figure 1, the use of GT method as a research method starts with how researchers find
out the problem(s) by questioning the real world around them and then develop a theory using
comparative analysis by means of the coding process, and the conclusions of the analysed results
present the new theory as the solution to the real world problem.
Questioning the world is the process of applying the first and the second strategy above in order to
understand the world around. By means of this, researchers create data which are based on the real
situation. Using the conceptual abstract process, researchers generate elements of theory from the
data towards the development of a theory.
350

Transmissia Semiawan
Figure 1: Research work using GT method
Comparative analysis is the core process of generating theory. It is used to develop a theory by
generating elements of theory which are categories, properties and hypothesis by means of a
conceptual abstract process through a coding process. It is an analytical tool for transforming insights
into relevant categories, properties, and hypotheses in order to ‘emerge’ theory from data. This point
is very important because it implies that researchers should be able to find theory and its elements
(i.e. categories, properties, hypotheses) within the data, not find data in relation to any predetermined
theory or hypotheses (Glaser, 1978, pg. 38). This is by means of conceptual abstract process through
which researchers should be able to identify a thing as being significant from the data. The author
stress the word ‘data’ as this is also the main issue in qualitative approach by the means of how we
develop/create the data based on people’s experiences and use the data from which we capture
phenomena towards the creation of theory and its elements.
3. The O-O concept
O-O concept has been known well within the computing science discipline particularly in the software
development area. Instead of going into the O-O concept from the perspective of this area, the author
refers to the O-O concept made known by Strauss (1969) and Neuman (2003) who analyse the
concept from the perspective of how a ‘thing’ exists as an ‘object’. The amalgamation of GT method
and O-O concept shows the way Strauss’s and Neuman’s O-O concepts take part in the theory
development process and become involved particularly in the comparative analysis and conceptual
abstract process.
Neuman (2003) explains how our mind brings the object into being through the process of giving a
meaning to it or what he called ‘signification process’. The basic activity of this process is the creation
of ‘boundary between the organism and its environment through a signification process that
differentiates between self vs. non-self’ (Neuman, 2003, pg. 11). This is the way human beings
differentiate an object from other objects by reflecting them upon a set or a class of things which are
symbolised or conceptualised by providing a name and a meaning to the classification process
relevant to the situation.
The objectifying process thus occurs because we the human being chooses to make the ‘thing’
significant or having a meaning or an important effect to others. To do this, one needs to define a
boundary of a ‘thing’ from other ‘things’ around it by constantly looking at the significances of each
such thing and analysing how it differs to the others in terms of its characteristics or properties. And
by using our human abstract conceptual process, we classify and define the ‘bounded’ thing as an
object and we symbolise, code or name it as an abstract entity in terms of its significant
characteristics.
Similar to Neuman’s ‘objectifying’ process, Strauss (1969) defines an ‘object’ as follows:
…any particular object can be named and thus located in countless ways. The naming
sets in within a context of quite differently related classes. The nature or essence of an
object does not reside mysteriously within the object itself but is dependent upon how it is
defined … (pg. 20)
351

Transmissia Semiawan
The direction of activity depends upon the particular ways that objects are classified
(p.21). …it is the definition of what the object “is” that allows action to occur with
reference to what it is taken to be (pg. 22)
The above concepts give the author a comprehensible idea of how the O-O concept relates to GT
method and takes part in the theory development process. The concepts point out a logical
connection between objectifying-process by means of differentiating process and theory-development
using comparative-analysis. The objectifying process which is a ‘process of making a thing exists as
an object’ goes with the theory generating process; whilst the process of differentiating which
differentiates between ‘self vs. non-self’ matches the idea of the theoretical comparison activity or
comparative analysis activity.
In terms of GT method, objects can be seen as the elements of a theory which are categories,
properties and hypotheses. These elements of theory which conceptually and significantly indicate
phenomena can be identified through coding process. Hence, it is reasonable to say that the
objectifying process can be used to understand and guide the coding process in building a theory.
This is by means of conceptualising data through a differentiating process in order to identify
significant phenomena and to categorise or to classify them according to their properties as theory or
elements of theory.
3.1 Categories and properties
A category is ‘a group of events, happenings, objects, and actions / interactions that are identified
conceptually similar in nature or related in meaning’ (Strauss & Corbin, p. 102); and a property is ‘a
conceptual aspect or characteristics of a category’ (Glaser & Strauss, 1967, p. 36; Strauss & Corbin,
1998, p. 102).
To explain about category and property, the author takes an example of an object called ‘Pencil’. A
‘pencil’ is a name given to the object. Before considering and naming the object as a ‘pencil’, one
would think about the meaning of the object which is based on its properties – the attributes and the
behaviour – attached to it. The meaning of this object is one’s reflection to the signification of the
object. This can be defined by considering the object’s shape, function, colour, behaviour,
composition, or other appearances that allow for characterising and differentiating it from other
objects. Hence, the object is differentiated from other objects using its own characteristics. The
characteristics of the object signify the properties of the object; and when other objects have similar
characteristics or properties, then one can classify or categorise them and identify them with a name
or label.
From the ‘pencil’ example, the characteristics of the object are:
composed of a piece of graphite
covered by a wooden or metal shaft or other materials
used for drawing or writing
Hence, the properties that characterised the object are:
Attributes: it has a piece of graphite or other marking material that is erasable inside a wooden or
metal shaft or other materials;
Behaviour/method: [it can be used for] drawing, or writing; at this point the object could be
considered ‘dynamic’ as the outer world of this object request it to act or to function as its
behaviour;
With the above properties, one then classifies or categorises the object in terms of signifying it using
its properties, and may call it as ‘pencil’. Each of the properties may change according to the
circumstances. When other characteristics are identified, i.e. used for scratching, the object would
have a different name with a different meaning relating to the attribute and behaviour of the object.
3.2 Hypotheses
Once the categories and properties are identified and developed in abstraction, hypotheses can then
be put forward. Hypotheses –as the other elements of theory besides categories and properties– are
352

Transmissia Semiawan
‘generally generated from the relations among categories and their properties’ (Glaser-Strauss, 1967,
pg. 39):
As the categories and their properties are identified, researchers should be able to see the common
interactions among the categories in terms of their properties. One way of developing relationships
among the categories is by a generalisation process.
In terms of the O-O model, generalisation happens as some categories have common characteristics
or properties that are inherited, i.e. a type of object ‘A’ is classifiable as a specific form of a larger
category – e.g. a jazz is a kind of music. By means of inheritance, one can differentiate some
categories in a way of hierarchical relationship of super-category against sub-category. The
relationship is called IS-A relationship. Through this relationship one can identify which categories are
more general or more specific in terms of the properties.
Another way to expose an object is by its composition – object contains other object. Take an
example of the human body as an object. It has parts like head, torso, arms, or legs that make up the
properties of the human body – so the human body composes of different parts of the body. However,
each of the parts can be considered as a single object with each of its own characteristics. For
instance, arm has its own characteristics as it has hand, fingers, elbow, wrist, etc. Therefore, through
composition, one might find other categories within a category with more specific properties and the
relationship amongst the categories is defined as Composition relationship.
Another way to create a relationship among categories or classes is by looking at the property
behaviour of the object, and this condition refers to polymorphism. Polymorphism is a way of looking
at an object as a new object by seeing some common set of operation in different ways. For example,
one might name a polygon regarding its geometrical shape and this can be done by reshaping the
polygon based on its number of sides – i.e. 3 sides as triangle, 4 sides as square, rectangle,
trapezoid, parallelogram or rhombus; 5 sides as pentagon; 6 sides as hexagon, etc. As sub-classes
these types of polygon are linked to their super-class using IS-A relationship. However, these types of
polygon are differentiated regarding their different implementation of methods or operations which is
based on the number of the sides. Therefore, the IS-A relationship is built in terms of polymorphism
rather than generalisation.
Based on the details above, objectifying process can be carried out by means of generalisation or
specialisation. Generalisation is also known as induction process as an object is generated based on
the common characteristics or properties that are inherited in other objects.
On the contrary, a deduction approach denotes specialisation through which specific characters or
properties of an object derived from its super-class object.
With an advantage of induction and deduction approaches, there is an abduction approach which
principally calls attention to the constructivist method by means of signifying new characteristics from
the identified objects or phenomena.
With reference to the three basic strategies of using GT method and the fundamental idea of
constructivist assumption, abduction approach allow researches to deepen the conceptual analysis
throughout the characteristics and properties of existing identified phenomena and cultivating their
insights in order to mean new categorical objects.
4. The implementation of GT method using O-O concept
With reference to Figure 1, this section discusses the way in which the author applied the
amalgamation of GT method and OO concept in a study of generating a theoretical based Information
Management (IM) model for Higher Education (HE) institutions (Semiawan, 2008). The discussion
focuses on the comparative analysis within which the data collecting, conceptual analysis through
coding process and building a theory are concerned.
4.1 Data collecting through interview
In order to obtain the data that represent the real situation of information management in a university,
data collection through interviews was conducted.
353

Transmissia Semiawan
To workout the interview questions, some ideas had been put forward to draw out the participants’
attentions and thought. The ideas are related to the participants’ experiences in dealing with the
university’s information resources (including the technologies) and how these support their working
activities. The ideas came out from the author’s insights (Strauss, 1987) and through these the
questions were pointed out in the participant’s particular setting during the interview. This approach
was used with reference to Charmaz’s (2002) approach of ‘symbolic interactionist theoretical
perspective’ through which the author tried to explore and to observe the participant’s thoughts,
opinions, and experiences by building up the questions leading to those ideas.
To some extent the questioning required intuitive skills to go with the participant’s mind in order to
follow their thinking process so that the next questions or ideas to be asked can be very quickly
determined. Therefore, the interview was designed in such a way to ensure the above ideas were put
forward mostly in conversational-like rather than question-and-answer type of interview. Through this,
each of the participants would speak about the ideas with respect to their own opinions or
experiences.
4.2 Analysis and building a theory
Analysis stage is the most important stage during comparative analysis as this stage embraces the
coding process by the means of conceptual abstract process. As the author has experienced, memo
writing is extremely important especially during the comparative analysis stage. Memo writing or
memoing allows the author to illustrate her interpretation over the data either in ‘free-style’ writing or in
a formal analytical statement. This is important because it directs the author to step from the analysis
in to the writing of the theory generating process (Charmaz, 2002). The memo can be structured not
only as textual basis but also in other patterns or symbols such as diagrams, figures or tables. This is
paramount in terms of helping the author to conceptualise phenomena and to capture significance or
meaning therein the data.
For the study, the author used the memo in different type of structures during interview and analysis
phases, which were textual memo and diagrammatical memos. The textual memo was developed and
used to illustrate the objectification process and construction of meaning about a particular topic
within texts. The importance of the object was identified based on the significant points within the
texts which bring out the analysis of an idea of the topic. For example, a signified object /*share-info*/
was identified from number of substantial points, as follows (Semiawan, 2008):
/*share-info*/
publishing the work
letting people know what you’ve done
telling anybody
Creating new information and transmitting information and communicating information through
publications, teaching, etc.
giving people information
working and getting to know people outside your particular area
talking to people from arranged areas
making a positive difference for them
translating throughout everything you do
sharing between staff, students and external stakeholders to create a richness of information
Another way of memoing was diagrammatical memos through which the author together with the
participant analysed and objectified ideas during the discussion. When identifying an idea during the
discussion, the author made sure that the signified idea – drawn as a diagram – was the same as the
one that the participant meant by re-asking and confirming the diagrammed idea to the participant.
The ‘that’s right!’ answer denoted a confirmation that the author and the participant had the same
understanding and identified the same significance to a particular domain context they had discussed.
In view of the above examples, memoing illustrates how the author – on her own or together with the
participants – interpreted the data by means of identifying significances or constructing meaning using
354

Transmissia Semiawan
formal analytical structures (textual based, diagram, etc.). In relation to the objectifying process,
memoing helps the author to find the attributes and behaviours as the characteristics of a
phenomenological object in the data. This leads to generating a theory through categorisation or
classification at the level of the properties and dimension a lot faster and easier.
The stages above were applied iteratively and recursively in order to capture phenomena of
information management in the university’s life and conceptualise it as an abstract theoretical
framework. Within the interview phases, understanding the data in terms of constructing the meaning
across the problem context was undertaken through a conversational type of discussion that
represents collaboration work between the author and the participants. By means of this activity, links
between data collecting and analysing was accumulated towards the concept/theory development
(Charmaz, 2002). Throughout the process of understanding the data, new issues may arise or come
to light which the author and participants may be probing and therefore required them to explore the
context deeper. The phases of the study process above implied the evolvement of building the theory
and were carried out iteratively until the author was ‘saturated’ with the definite theory.
As a result of the analysis stage, five phenomena had been identified: sharing information;
transparency and communication; learning and knowledge development; communication across
organisation; resource-use (Semiawan, 2008). The phenomena were identified as conceptual objects
by means of a conceptual abstract process. Using the conceptual abstract process and abduction
approach, new phenomenon can also be developed by referring it to other phenomena as the
underlying constructs, or using its own properties. For example, sharing information and
communication-transparency were the very basic and essential phenomena that bring in the
occurrence of learning and knowledge development through the relevance of their fundamental
characteristics i.e. make the information available and meaningful to other people and have easy
access to the information so that people can learn and gain knowledge from the information provided.
Regarding abduction approach, all of the phenomena were then becoming fundamental concepts for
generating and constructing the elements of the theory. This was done by means of classificatory and
linking process amongst the phenomena as well as amongst their characteristics and properties (see
Figure 2 and Figure 3).
As the comparative analysis continued and the conceptual abstract process through abduction
approach carried on, four phenomenological categories were identified based on the five fundamental
phenomena above. The four phenomenological categories which were interrelated and
interdependent became apparent as Academic-Life, Organisational-Culture, Work-Unit and
Communication-Metaphor (Semiawan, 2008).
Figure 2: Development of Academic-Life category
355

Transmissia Semiawan
Figure 3: Development of Organisational-Culture category
Academic-Life (Figure 4) symbolises the integrated way of research (RS), teaching-learning (TL), and
service (SC) activities to expose the intellectual capital (IC) of the university. The integrated ways
comprise of:
Knowledge network development – shows a relationship between research and teaching-learning
Institutional development – links teaching-learning with service
Reputation development – illustrates the correlation between research and service
Intellectual capital includes
New knowledge / innovations – come from knowledge development
Knowledge transferred – as the result of institutional development
Solutions – derived from reputation development
Figure 4: Category-1: Academic-Life
Organisational-Culture (Figure 5) comprises of three dimensions within which the academic life is
dynamically floating around: cognitive dimension, resource dimension, process dimension. Each
dimension represents the conduit metaphor of two opposing extremities
Cognitive Dimension: Explicit – Persistence
Resource Dimension: Strategic – Operational
Process Dimension: Mechanistic – Organic
Work-unit (Figure 6) represents an individual, people, units or organization. The work-unit which is
symbolised by a cube illustrates the unit’s capabilities of carrying on all the academic life within the
organisational culture in terms of managing all information assets that embrace all resources in order
to carry out a set of activities and make strategies and plans with respect to their position and function
within the organisation.
356

Figure 5: Category-2: Organisation-Culture
Figure 6: Category-3: Work-Unit
Transmissia Semiawan
Communication Metaphor (Figure 7) signifies the concept of interface management by pointing out a
crossing point among the work-unit relationship. The work-unit performs the work in framework basis.
Individual frameworks are the smallest unit frameworks that cumulatively develop the whole big
organisational framework. The development of unit frameworks requires interface management in
terms of linking them to each other through a communication metaphor. The interface management
through communication metaphor denotes a process of information exchange that takes account of
the process of transmission and reception as well as the sharing process in terms of understanding
the meaning of the information. The presence of communication metaphor among the units develops
the balance of the whole organisational framework in terms of how the work-unit positions itself in
academic life and throughout the organisational culture.
Figure 7: Category-4: Communication-Metaphor
The interrelated and interdependent categories which were signified as the IM framework or model for
HE institutions turned out to be the finding of the study which was symbolised in a form of the ‘Crystal
Maze Inside Out’ (Figure 8) (Semiawan, 2008):
‘Crystal maze’: the work-unit carrying out academic life from the perspective of six sides from the
three cultural dimensions.
‘Inside-out’: the movement of the work-unit relative to cultural dimensions in order to make the
best use of it within the organisation culture.
357

Figure 8: ‘The Crystal Maze Inside Out’
5. Conclusion
Transmissia Semiawan
The use of GT method in research work requires an understanding of some basic elements
underlined in the methodology and how they could be used in terms of generating theory, which
include conceptual abstract process; comparative analysis; and coding.
Comparative analysis is the most important process in the GT method as it involves conceptual
abstract process and the process of construction of meaning. The application of OO concept in GT
method helps the author to find the attributes and behaviours as the characteristics of a
phenomenological object in the data by means of the objectifying process. This leads to generating a
theory through categorisation or classification at the level of the properties and dimension a lot faster
and easier.
References
Bryant, A. and K. Charmaz (2007). Grounded Theory Research: Method and Practices Pre-eminent Qualitative
Research Method - To appear as the Editors' Introduction to The Sage Handbook of Grounded Theory,
SAGE Publication, Inc.
Charmaz, K. (2000). Grounded Theory: Objectivist and Constructivist Methods. Handbook of Qualitative
Research - 2nd Edition. N. K. Denzin and Y. S. Lincoln. Thousand Oaks - London - New Delhi, SAGE
Publication, Inc.: pp. 509-535.
Charmaz, K. (2002). Qualitative Interviewing and Grounded Theory Analysis. Handbook of Interview Research. J.
F. Gubrium and J. A. Holstein. Thousand Oaks, London, New Delhi, SAGE Publications: pp. 675 - 694.
Charmaz, K. (2006). Constructing Grounded Theory - A Practical guide through qualitative analysis, SAGE
Publications Ltd.
Glaser, B. G. (1978). Theoretical Sensivity - Advances in the Methodology of Grounded Theory. San Francisco,
The Sociology Press, PO Box 400, Mill Valley, California, 94942.
Glaser, B. G. (1992). Basic of Grounded Theory Analysis: Emergence vs. Forcing. Mill Valley, California,
Sociology Press.
Glaser, B. G. and A. L. Strauss (1967). The Discovery of Grounded Theory: strategies for qualitative research.
New York, Aldine De Gruyter.
Neuman, Y. (2003). Process and Boundaries of The Mind - Extending the Limit Line. New York, Boston,
Dordrecht, London, Moscow, Kluwer Academic / Plenum Publisher.
Semiawan, T. (2008). “The CrystalMaze Inside – Out” – Information Management Framework for Higher
Education Institutions, A PhD thesis. Leeds Metropolitan University. Leeds. UK.
Strauss, A. L. (1969). Mirrors and Masks – The Search for Identity. Martin Robertson & Co, Ltd., London,
Medicine in Society Series.
Strauss, A. L. (1987). Qualitative Analysis for Social Scientists. Cambridge, Cambridge University Press.
Strauss, A. L. and J. Corbin (1998). Basic of Qualitative Research: Techniques and Procedures for Developing
Grounded Theory. Thousand Oaks, California, SAGE.
358

True Science and Project Management
Miles Shepherd and Roger Atkinson
Bournemouth University Business School, UK
Miles.shepherd@msp-ltd.co.uk
Rogeratkinson@bournemouth.ac.uk
Abstract: This paper is intended to highlight the main ontological and epistemic issues in research into project
management. It builds on an earlier paper, presented at ECRM 11, and identified the problems inherent in
developing Bodies of Knowledge (BoK) and in particular for project management. We noted that these issues
have not been adequately addressed, thus they pose chronic problems when determining approaches to the
structure, detailed contents and the manner in which Project Management bodies of knowledge have been
compiled. In this paper, we investigate the impact these issues have on the development of a theoretical
underpinning for the discipline of project management. The notion that a professional discipline requires such an
underpinning is challenged and a number of issues and alternatives are identified. For example, paradigmatic
views of project management knowledge and how these relate to the construction of bodies of knowledge in
order to identify possible research strategies that may resolve some of the underlying weaknesses and gaps. We
note that project management cannot be defined totally in the Popperian sense of falsification using failure
studies, but in addition requires the outputs from credible rather than significant research outputs in order to
attempt to challenge the fitness for purpose of current (BoKs) and define a much boarder ‘Network of Knowledge
(NoK) in order to enable the profession or discipline of Project Management to be accepted as such. We discuss
the implications of the identified ontological situation of project management in terms research agendas, research
methodologies and on theory building research. The paper concludes by reviewing alternatives to positivistic
views of hypothesis testing and theory building approaches.
Keywords: project management, bodies of knowledge, empiricism, justifications
1. Introduction
We take project management (PM) in its widest sense and so do not separate programme or portfolio
as different domains. Rather, we view the domain to include the management of single projects and
multiple projects in various combinations. We also take projects to be varied in the level of:
complexity, novelty, difficulty, value and duration.
In an earlier paper (Shepherd and Atkinson 2010) we outlined issues related to formal Bodies of
Knowledge (BoKs) and concluded that:
PM as a domain continues to develop in terms of knowledge, practice and techniques;
BoKs will also need to evolve as practice changes;
There remain a number of ontological issues that have yet to be resolved;
Neither inductive nor deductive research processes alone could resolve the knowledge issues of
PM.
In this paper we explore the situation regarding the theoretical base of project management. In the
view of some academics, theory plays an important role in the study of any field. This view seems to
be partly one that indicates a lack of theory is seen as an inhibitor to acceptance of the field as a
legitimate area of study (Turner 2007). Project Management is often not seen as a management
discipline at all, as Turner (2007) highlighted. This in his view, and that expressed in an influential
EPSRC research network (Maylor 2006) is due to the apparent dominance of a systems perspective
that ensures that PM is seen as belonging in the engineering and physical sciences rather than the
domain of social or behavioural science. For many practitioners, this is not a problem since PM has its
roots in enterprises such as the Manhattan Project, major defence contracting and construction.
Others, especially organizational scholars, who might be expected to bring a useful view to bear, may
be inhibited by this. However, Bredillet (2007b) implies that the need for a theoretical base is founded
on the need for an unequivocal ‘field of study’ to delineate the area for which academic researchers
might legitimately operate. Following Audet’s (1986) definition, ‘...the space occupied by the whole of
the people who claim to produce knowledge in this field, and this space is also a system of
relationships between these people competing to gain control over the definition of the conditions and
the rules of production of knowledge’; Bredillet demonstrates PM to be a valid academic field with
requisite knowledge, people who produce the knowledge and relations between people who define
359

Miles Shepherd and Roger Atkinson
the conditions and rules. In effect this gives some legitimacy to researchers and practitioners as well
as professional societies.
There are many views of what constitutes ‘theory’. In a special theme issue of Administrative Science
Quarterly, Weick (1995) drew attention to one of the dictionary definitions of theory as ‘guess,
speculation, supposition, conjecture, proposition, hypothesis, conception, explanation, model’. This
allows a wide interpretation and it seems that far from informing the reader of what theory is, authors
found it difficult to define theory but were better able to say what it is not. However, Whetton (1989)
calls on what Kipling called his ‘six honest men’ (Who, What, When, Where, How and Why) to explain
how theory building takes place. In his view, theories derive from concepts, often include propositions
and hypotheses but are eventually testable. This has been at the centre of PM theory development
over the years.
Even with a wide range of possible candidates for PM theory, there is a rich seam of scholars for
whom there is either no theory (e.g. Cicmil 2006, Cicmil and Hodgson 2006, Turner 1999) or weak
theory (e.g. Koskella and Howells 2002, Williams 2004). Atkinson (1999) stated that there was no
agreement on even the definition of Project management and that could be the discipline’s strength
as it evolves. The International Project Management Association (IPMA), Project Management
Institute (PMI), Association for Project Management (APM) and the British Standards Institute (BSI) all
offer their own definition, albeit with minor semantic differences. Some might consider this variation to
be because whoever ‘owns’ the definition could possibly ‘franchise’ its use and therefore of
commercial interest. If this economic supposition is correct there will remain several definitions of
project management although there are other reasons why this divergence will remain.
The remaining structure of this paper is as follows: First we consider why theory might be considered
important as far as PM is concerned. This is followed by a discussion about four questions, these are:
What is the state of theory development in PM?
Is this theory adequate?
Are alternative approaches available?
How to proceed?
Finally we offer a conclusion to this paper and a brief pointer to the third in this series.
The Importance of PM Theory
As an essentially practical discipline, some might feel that there is no need for an explicit ‘theory of
PM’. This might be correct if the practice of PM was entirely effective but, as Williams (2004) points
out, it is not. There is a long history of failed projects that will be all too familiar to readers. A number
of authors have enumerated a catalogue of poor performance, notably Flyvberg, Holm, and Buhl
(2002), Morris and Hough (1987); Williams (2004) and innumerable NAO reports (see for example
NAO 2011).
For the practitioner, there certainly is a need for some delineation of the field; in other words some
definition of what is and what is not PM. Once this is available, it becomes possible to set out
definitions of practice, educational requirements and training courses (Morris et al, 2006) explain the
need for bodies of knowledge in the commercial world, noting that these give reassurance to the
practitioner, help establish career frameworks and enable training and educational courses to be
developed. As was demonstrated in Shepherd and Atkinson (2011), this is a rather circular
arrangement with more than a little self interest conditioning both the development of Bodies of
Knowledge and their usage. Koskella and Howell (2002) also explain that professional status depends
on mastery of theory as well as mastery of technique, hence, no theory, no profession. As Fugate and
Knapp (1998) comment, reliance on the theoretical is “the single most important factor distinguishing
a profession from a craft.” Other professions such as Law consider Jurisprudence as the basis of their
theory, while theory in economics is based on ‘ a priori’ evidence. The medical profession do not have
a single overall theory, but rather a series of specialist medical departments each with their own
theories and knowledge. Many of these are commonly known using the names including ‘ology’ a
suffix derived from the Greek Logos meaning ‘the study of’, or the ‘speciality in’. Departments such as
Haematology (blood) or cytology (cells) each have their own set of theories while generally staying
within other, broader theoretical settings. .However, these terms are now being extended beyond the
360

Miles Shepherd and Roger Atkinson
medical field and terms such as Oenology (wines) and Ombrology (rain) are being introduced.
Perhaps this is how project management can exist without a single theory while some factors could
be theory driven related to specifics such as risk and quality etc, as ‘the study of’ project management
becomes the new yet-to-be-named ...‘ology’.
More broadly, PM is seen by many organisations as the ideal way to solve many of the complex
issues of the world (e.g. inter alia Frame 1994, Hobday 2000, or Turner 1999). Others challenge this
notion (e.g. Cicmil and Hodgson 2006) on a variety of grounds, not least theory related. Bredillet
(2007b) notes that PM has become important for firms that seek to gain competitive advantage
through building competences. While PM is certainly effective in the delivery of relatively simple
projects, there is a well reported stream of both project and project management failures in the more
complicated and more complex projects, as previously noted. So there is an organisational and
economic need for PM theory. Lack of PM theory is also charged with causing a lack of commitment
to PM methods (Forsberg et al. 1996; Melgrati and Damiani 2002) and slow rate of methodological
renewal (Morris 1994).
2. Current state of PM theory development
Those who claim there is no PM theory really have little to substantiate such a claim. While it is true
that there is no explicit, all embracing theory of PM, many scholars hold that what theory there is
comes from other disciplines. The professional societies are largely silent on the matter of theory and
it has been claimed that existing Bodies of Knowledge are largely declarative, offering no reasons for
their presentation of the discipline (Koskella and Howell 2002). Williams (2004) claims that ‘Project
management presents itself as self-evidently correct (and therefore presumably an explicit espoused
theory is not essential), and provides a normative set of techniques. No justification is necessary, as
there is no question that the PMBOK Guide is correct. In fact, he could have been referring to any of
the main BoKs since as previously noted (Shepherd and Atkinson 2011), there is a large measure of
agreement over main techniques and approaches contained in the major Bodies of Knowledge. True,
there are some differences in their starting point and discussion over the extent that PM is domain
independent but professional societies generally view the practice of PM in broadly the same way
(PMBOK ® , APM BoK, IPMA ICB ® ).
Scholars make various claims about extant PM theory. For Lundin (1995) and Lundin and Söderholm
(1995), it is rational while for Melgrati and Damiani, 2002 and for Packendorf (1995) it is normative.
Koskella and Howell (2002) see it as reductionist while Smyth et al (2006) and Williams (2004) see it
as rational positivist. Overall, extant theory as espoused in the various BoKs is seen prescriptive.
Cooke-Davies et al (2007) claim this is in large part due to the dominant views of ‘the dominance of
various versions of control theory, operations research, or systems theory in the studies of projects
and project management’.
Study of the literature indicates that there have been a number of attempts to develop some form of
overarching theory of PM. Söderlund (2002) identified 7 schools of project management and set out
briefly the underlying epistemic stance. Bredillet (2007a – c, 2008a – c), reporting work commissioned
by the Project Management Institute (PMI) and carried out in conjunction with Turner and Anbari,
identified nine schools and set out his views of the relevant ontological stances. Each school
embraces some specific view of theory. Interestingly, Biedenbach and Müller (2009) used Bredillet’s
schools as a basis for an assessment of paradigms underlying research reported at IRNOP
Conferences. The popularity of the various schools, as reflected in the rankings, shows an interesting
shift in emphasis. Whether this is due to a change in acceptance of the relevant paradigm is not clear
but is a point we will return to shortly.
A third view of PM theory came from Turner (2006a – d) who set out a series of premises and lemmas
in order to ‘derive a structure for project management and identify inherent elements in Project
management...’. This approach is seen as somewhat mathematical (Sauer 2007) but its importance is
recognised since an a-theoretic study of PM is unhelpful, especially in an age of constrained research
funding. Sauer follows Kuhn’s perception (1996) of the role of theory and sees it as helping direct
inquiry onto productive lines. Thus lack of explicit theory hampers the pursuit of ‘programmatic
research’ (Sauer 2007). Interestingly, Sauer sees this programmatic research as part of the
development of the cumulative tradition in the advancement of PM knowledge. Kuhn is quite explicit in
his rejection of this notion that cumulative additions broaden theory and widens acceptance of
particular paradigms (Kuhn 1997 p 7 & 108). Nonetheless, there is a strong case for good PM theory
361

Miles Shepherd and Roger Atkinson
on both political and scholarly grounds. Thus more direction is needed in the research process so that
the major stumbling blocks are addressed as a priority. This was one of the aims of the EPSRC
Network ‘Rethinking Project Management’ (Winter et al 2006). The network had been charged with
the task of defining ‘a research agenda aimed at enriching and extending the subject of project
management beyond its current conceptual foundations.’
Söderlund’s Schools Bredillet’s Schools
Biedenbach & Müller Ranking
1994 2007
1 Optimization Optimization 2= 6
2 Critical Success Factor Modelling 7= 7
3 Contingency Governance 1 3=
4 Behavioural Behaviour 2= 2
5 Transaction Cost Success 4 5
6 Marketing Decision 9 8=
7 Decision Process 5= 1
8 Contingency 5= 3=
9 Marketing 7= 8=
Figure 1: Schools of Project Management and popularity of research bases
A fourth view of PM theory comes from Koskella and Howell (2002) who reviewed the then current
version of PMBOK ® to determine its theoretical stance. They concluded that project management as
practiced today rests on an implicit and narrow theory that must be developed, extended and
enriched. They conceptualised PM as Project and Management, then assigned various theories to
each identified component. Essentially they see PM as a form of production and subject to a series of
theories borrowed from production management as seen in Fig 3 below:
Subject of theory Relevant theories As outlined by
Project Transformation Porter (1985)
Flow Gilbreths (1922)
Value generation Shewhart (1931)
Management Turner (1999a)
Planning Management-as-planning Johnston & Brennan (1996)
Management-as-organizing Johnston (1995), Johnston &
Brennan 1996).
Execution Classical communication theory Shannon & Weaver (1949)
Language/action perspective Winograd and Flores (1986).
Controlling Thermostat model Ogunnaike & Ray (1994)
Scientific experimentation model Shewhart & Deming (1939)
Figure 2: Theoretical analysis (after Koskella and Howells 2002)
Thus it is evident that there is some cause for optimism. Several notable researchers have presented
theory to support the practice of PM even though there seems to be little empirical evidence to
support these efforts. That there is a diversity of schools indicates that the discipline is still at a low
level of maturity compared with the physical sciences but leaves open opportunity for theory
development. The multiplicity of schools also indicates that no theoretical school dominates the
academic study of PM, leaving again, opportunity for the researcher to tackle what Kuhn calls ‘normal
science as problem solving’ (Kuhn 1996 pp 35 – 51).
While deriving some of the theory from management, project management does not appear to have
followed a similar maturity or evolutionary development. Crainer (1998) traced the history for what
were termed Management Fads. Figure 3 illustrates how over approximately a hundred year period
those so called ‘Fads’ switched from what could be categorised as Art to Science. Further it was
found that it was not enough for one Fad simply to be better for it to take over from the previous, the
acceptance of a new ‘Fad’ included the tarnishing of the previous Fad for it to be accepted. However,
Systems Theory and Quality theory for example, still exist as part of management. It is noteworthy
that while Themistocleous and Wearne (2000) identified how the topics of project management
changed over time, indicating how PM was itself evolving, there was no attempt to catalogue those
topics as management did, into Fads. PM has moved on over the last fifty years but there is no
chronological list to indicate how it moved from one major ‘Fad’ to another, if indeed it did.
362

Scientific Management
Operational Studies
Strategic Management
Competitive Forces
Systems Theory
Re-engineering
Science
Figure 3: Management fads (after Crainer 1998)
3. Adequacy of current theory
Miles Shepherd and Roger Atkinson
1900
2000
Hawthorne Studies
Maslow’s Hierarchy
Service
Quality
Empowerment
Competencies
As we have already seen, there is some doubt that existing theory is adequate for the needs of the
discipline. Koskella and Howell (2002) explicitly state that existing theory is narrow and needs
development. Williams (2004) shows that in the light of project overruns, the existing theory does not
address the complexity of modern projects. Winter et al (2006) also claim that PM has ‘long been
criticised for its ‘lack of relevance to practice and, consequently, to improved performance of
projects...’. It is not the adequacy of theory development that is criticised here so much as the
adequacy of the theory that currently is available. We have already noted that PMBOK (PMI 2004)
and indeed other BoKs are declarative. Furthermore, researchers in the critical management school
challenge the validity of BoKs on sociological grounds (e.g. Cicmil and Hodgson 2006). Others
criticise the way project knowledge is represented and the concentration on tools and techniques
(Morris 2002) at the expense of underlying understanding (e.g. Maylor 2001). Thus it seems clear that
many respected researchers and PM scholars regard the adequacy of existing theory to be at best
doubtful and at worst, inadequate.
4. Alternative approaches
There appear to be a number of possible alternative approaches. A brief glance at the literature
shows that the following might be considered as alternatives to a fully developed theoretical stance:
Use of paradigms instead of fully developed theories (Kuhn 1996);
Evolution of Communities of Practice (Wenger 1998)
Networks of Knowledge (Moon 1999).
Kuhn’s work on the advance of science (1997) opened the way to the consideration of paradigms
where groups of scientists ‘gather’ round a set of ideas that represent their specific view of how their
discipline ‘is. Scientific advance is made when someone makes a breakthrough and establishes a
new paradigm that accounts for more of the unknowns of the discipline. Kuhn’s conception is that
paradigms precede theory (Kuhn 1997 p44). Furthermore, such paradigms do not need to be
complete or to explain all aspects of a ‘science’. Kuhn takes this idea further and notes that
paradigms “may be prior to, more binding, and more complete than any set of rules ... that could be
unequivocally extracted from them” (Kuhn, 1996, p46). Furthermore, according to Kuhn, those who try
to extract rules from such a situation usually find it a much more difficult task than simply identifying
paradigms (Kuhn 1997, p3).
Wenger (1999) builds on earlier work and takes the premise that groups of practitioners form
communities that share knowledge and understanding of specialist action through observing each
other and through practice mediated by their peers. Under this view, knowledge is gained by doing;
363
Art

Miles Shepherd and Roger Atkinson
thus knowledge is ‘situated within the community of practice. This perception fits well with the critical
management schools and was specifically addressed in the EPSRC study in 2003 – 2005 (Cicmil et al
2006) where representation of knowledge and how practitioners learned was a fundamental concern.
Moon (1999) provided a method for understanding how theory and practice work together. Using a
metaphorical example, Moon suggested that there is a ‘network of knowledge‘ and this is comprised
of ‘espoused theory and the idiosyncratic processing of espoused theory that has occurred ‘ the latter
being derived from experience and other sources. This view would appear to support an argument for
broadening the representation of knowledge currently embodied as a BoK into a NoK. Many, including
students believe, that the BoK is THE complete source of PM knowledge, a dangerous state, but the
move to a Network of Knowledge could help create a shift away from a BoK to a much more
comprehensive source of knowledge for project management.. Many professions including the
medical, legal and others point towards complete libraries when their body of knowledge is discussed,
compared to the single, slim volumes that PM boasts, as we highlighted elsewhere (Shepherd and
Atkinson 2011).
5. How to proceed?
There is something to be said in favour of each of the alternative approaches outlined in the previous
section. We would argue that if the professional societies are silent on the theory of PM, as Koskella
and Howell (2002) maintain, they would do well to embrace paradigms as an alternative to adopting
specific theoretical stance. These societies are composed, for the most part, of practitioners. Few if
any will become involved with research but many will wish to keep current in their knowledge and to
improve their practice. This basic principle is enshrined as the Continuing Professional Development
programmes in place in all the major PM societies. These practitioners learn their basic trade from
study of paradigms. For as Kuhn (1996 p11) says, ‘the study of paradigms ... is what mainly prepares
the student for membership of the particular scientific community...he there joins men who learned the
bases of their field from the same concrete models, his subsequent practice will seldom evoke overt
disagreement over fundamentals’. Thus enthusiastic endorsement of these paradigms should be seen
by these societies as a critical success factor in their own operation.
We also consider that the use of paradigms fits with the stage of development that the discipline has
reached. As Kuhn (1997) said, (p 35) there is a great deal of puzzle solving to be done by normal
science. The existence of the various schools is evidence that paradigms are ‘out there’ and in active
use. We will explore this aspect, and how it affects research methods in another paper.
Wenger’s ideas of situated learning and communities of practice hold a significant advantage for the
professional societies. Structuring groups to build knowledge and understanding of specialist areas
sounds somewhat like the Specific Interest Groups that exist in several societies. Someone who is
considered an expert in one group, or community, is not necessarily an expert in another group. This
allows a natural way for new entrants to join, learn and progress within a range of communities and
also to hone their skills by doing what professionals have always done – practice their skills. While
this is no long term solution to lack of a theoretical base, it does have the advantage of enforcing
some professional attributes while helping to erect barriers to entry for the societies.
Moon (1999) has developed reflection into learning and professional development and in order to
understand her own area she builds on the work of Schön who demonstrates how theory of practice is
built on the basis of reflection. Schön described how theoretical knowledge could be developed by
using the work of Gray and Frostrom (1991) and Boud, Keogh and Walker (1985) who included
clinical practice to help develop practice theory of nursing. Moon appears to take this a stage further
and discusses the Network of Knowledge where it is described earlier as espoused theory and the
idiosyncratic processing of espoused derived from experience. Moon’s network of Knowledge could
very easily be considered as a contender for the development of a more comprehensive
representation of Project Management knowledge with some coming from evidence based research
while other elements are included via reflection, as the enhancement of clinical theory evolved using
in part new knowledge obtained from reflective practice (Gray and Frostrom 1991).
6. Summary
At the start of this paper following a brief discussion as to why a theory might be considered important
we said we would discuss four questions. First we discussed the state of theory development through
a series of arguments and supporting references which indicated that PM is lacking in theory, in the
364

Miles Shepherd and Roger Atkinson
sense that it can be testable. However, parallels with other professions also indicated that while a
single theory is not essential, in order to be considered a profession some theory for PM certainly is.
Next we discussed whether the theory identified was adequate and the literature showed that it was
not adequate. The third point considered alternative approaches and this included suggestions that
theory could be replaced by the use of paradigms, practice might be enhanced by the exploitation of
situated learning and that PM BoKs could be expanded to be a network of knowledge NoK. Finally, a
brief discussion of how to proceed was offered. This all suggests that while the professions have been
slow to develop theory they should at least consider some alternative paradigms
An earlier paper (Shepherd and Atkinson 2011) identified issues in developing a single, consistent
BoK. Here we set out to build on the outcomes from our earlier paper by examining the role of theory
in the emerging domain of Project Management. We have shown that theory is important for the
discipline since it enables improvement in understanding and increasing effective performance. It is
clear from the literature that the current state of theory development in PM is, at best, weak. It can be
seen that there are varying schools of PM thinking but the stance of the professional societies seems
to be a silent one with all the BoKs adopting a declarative outlook and eschewing formal adoption of a
theoretical position.
There is empirical evidence for the existence of a number of schools of PM thought, reminiscent of
Kuhn’s paradigm concepts. There also seems to be some support for more socially based ideas of
PM understanding through situated learning as put forward by Wenger. The case for broadening the
knowledge conception of project management also suggests that networks of knowledge have a part
to play.
7. Conclusions
In our view the task is now to develop better understanding of project management through further
exploration of the domain, including examination of boundaries, extensions of current thinking and
resolving problems of practice. Project management clearly needs theory; we believe this theory
needs to cover the individual elements of PM but probably not PM itself, in the manner of the way the
medical profession treats its theoretical development. As we have seen, practitioners have a need for
some form of BoK but the current versions remain narrow so there seems scope to expand the
conceptual base by means of a much more comprehensive Network of Knowledge. Finally, there
seems to be merit in the idea of idea a new ‘ology’ for project management as it is a study of, or
speciality in, or art of project management. All these ideas should, we feel, be explored by means of
research. It is here that the application of Popper’s ‘True Science’ will apply. But this raises a number
of issues that we explore in our next paper.
References
Atkinson, R. (1999). Project management: Cost, time and quality, two best guesses and a phenomenon; it’s time
to accept other success criteria. International Journal of Project Management, 17(6), pp. 337–342.
Audet, M. (1986). Le process des connaissances de I’admimstration. In M. Audet & 1. L. Malouin (Eds.), La
production des connaissances de I’administration (pp. 23-56). Quebec: Les Presses de l’Universite Laval.
Boud D, Keogh, R and Walker, D (eds) (1985) Reflection: Turning experience into learning, Kogan Page,
London.
Bredillet, C. N. (2007a). From the editor: Exploring research in project management: nine schools of project
management research (part 1). Project Management Journal 38(2), 3 – 4.
Bredillet, C.N. (2007b). From the editor: Exploring research in project management: nine schools of project
management research (part 2). Project Management Journal 38(3), 3 – 5 .
Bredillet, C.N. (2007c), From the editor: Exploring research in project management: nine schools of project
management research (Part 3)”, Project Management Journal, Vol. 38 No. 4, pp. 2-4
Bredillet, C.N. (2008a), From the editor: Exploring research in project management: nine schools of project
management research (Part 4)”, Project Management Journal, Vol. 39 No. 1, pp. 2-6
Bredillet, C.N. (2008b), From the editor: Exploring research in project management: nine schools of project
management research (Part 5)”, Project Management Journal, Vol. 39 No. 2, pp. 2-4.
Bredillet, C.N. (2008c), From the editor: Exploring research in project management: nine schools of project
management research (Part 6)”, Project Management Journal, Vol. 39 No. 3, pp. 2-5.
Cicmil, S., and Hodgson, D. (2006). New possibilities for project management theory: A critical engagement.
Project Management Journal, 37(3), 111-122
Cicmil, S., Williams, T., Thomas, J. And Hodgson, D. (2006) Rethinking Project Management: Researching the
actuality of projects International Journal of Project Management, 24(7), pp675 – 686.
365

Miles Shepherd and Roger Atkinson
Cooke-Davies, T., Cicmil, C., Crawford, L. And Richardson, K. (2007). We’re not in Kansas anymore Toto:
Mapping the strange landscape of complexity theory, and its relationship to project management. Project
Management Journal Vol 38 No 2 pp 50 – 61.
Crainer, S (1998) The Ultimate business guru book: 50 thinkers who made management., Capstone , Oxford
Flyvberg, B., Holm, M.K., & Buhl, S.L. (2002). Understanding costs in public works projects: Error or lie? Journal
of the American Planning Association, 68, 3, 279-295
Forsberg, K, Mooz, H and Cotterman, H. (1996) Visualizing Project Management. John Wiley & Sons, NeFrame,
J. D. (1994). The new project management: Tools for an age of rapid change, corporate reengineering, and
other business realities. San Francisco: Jossey-Bass.w York.
Gray J and Frostrom S. 1991 Generating theory from practice and the reflective technique in Towards a
Discipline of Nursing G Grey and R Pratt eds, Churchill Livingstone
Hobday, M., (2000) The project-based organisation: an ideal form for managing complex products and systems?
Research Policy 29 pp 871–893
Johnston, R.B. and Brennan, M. 1996. Planning or Organizing: the Implications of Theories of Activity for
Management of Operations. Omega, International Journal of Management. Science, Vol. 24, No. 4, pp. 367-
384.
Johnston, R.B. 1995. Making manufacturing practices tacit: a case study of computer aided production
management and lean production. Journal of the Operational Research. Society . 46, 1174-1183.
Koskella, L. and Howell, G,. (2002) The underlying theory of project management is obsolete. Proceedings of the
PMI Research Conference, 2002. Pg. 293-302.
Kreiner, K. (1995). In search of relevance: Project management in drifting environments. Scandinavian Journal of
Management, 11(4), 335–346.
Kuhn, T. S. (1996). The structure of scientific revolutions, third ed. Chicago: The University of Chicago Press.
Lundin, R. A., and Söderholm, A. (1995). A theory of the temporary organization. Scandinavian Journal of
Management, IJ(4), 437-455.
Lundin, R.A. (1995). Editorial: temporary organizations and project management. Scandinavian Journal of
Management, 11, 315-317.
Maylor, H. (2006) ed: Rethinking project management special issue of International Journal of Project
Management, Vol.24 (4)
Melgrati, A., and Damiani, M. (2002). Rethinking the project management framework: New epistemology, new
insights. Proceedings of PMI Research Conference. Newtown Square, PA: Project Management Institute.
Moon A.J (1999). Reflection in Learning and Professional Development – Theory and Practice. Oxon, Kogan
Page
Morris, P. W. G. (1997). The management of projects (2 nd ed.). London: Thomas Telford.
Morris, P.W.G. (2002) “Science, Objective Knowledge and the Theory of Project Management” Proceedings of
the Institute of Civil Engineering Vol. 150, May, pp. 82 – 90
Morris, P. W. G. And Hough, G. H. (1987) The Anatomy of Major Projects, Wiley and Sons, Chichester.
Morris P.W.G., Crawford, L., Hodgson, D., Shepherd, M.M. and Thomas, J. (2006). Exploring the role of formal
bodies of knowledge in defining a profession – The case of project management. International Journal of
Project Management;24:710–21.
NAO (2011) The cost effective delivery of an armoured vehicle capability. Report by teh Comptroller and Auditor
General. HC1029, 20 May 2011.
Ogunnaike, B. A. and Ray, H. W. (1994). Process Dynamics, Modelling, and Control. Oxford University Press,
New York.
Packendorff, J. (1995). Inquiring into the temporary organization: New directions for project management
research. Scandinavian Journal of Management, 11, 4, 319-333
Porter, M. 1985. Competitive Advantage. The Free Press.
Project Management Institute. (2004) A Guide to the Project Management Body of Knowledge (Third Edition).
Newtown Square, PA: Project Management Institute.
Sauer, C. (2007) What do we want from a theory of project management? A response to Rodney Turner.
International Journal of Project Management Vol;25(1):1–2.
Schön, D. A. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the
professions. San Francisco: Jossey-Bass Publishers
Shannon, C. E. and Weaver, W. (1949). The Mathematical Theory of Communication. (Republished edn. 1962).
The University of Illinois Press, Urbana.
Shenhar, A.J. (1998) From Theory to Practice: Toward a Typology of Project Management Styles, IEEE
Transactions on Engineering Management, Vol. 45, No. 1, February, pp. 33-48.
Shenhar A, and Dvir D. (1996)Toward a typological theory of project management. Research Policy;25:pp607–
32.
Shepherd and Atkinson (2011) Project Management Bodies of Knowledge: Conjectures and Refutations. Paper
presented at ECRM 11, Caen, 2011
Shewhart, W.A. (1931). Economic Control of Quality of Manufactured Product. Van Nostrand, New York.
Shewhart, W.A. and Deming, W.E. (1939). Statistical Method from the Viewpoint of Quality Control. The
Graduate School, The Department of Agriculture, Washington.
Smyth, H. J and Morris, P. W.G. (2007) ‘An epistemological evaluation of research into projects and their
management: Methodological issues’, International Journal of Project Management, 25 (423-436)
366

Miles Shepherd and Roger Atkinson
Smyth, H. J., Morris, P W. G., and Cooke-Davies,T. (2006) Understanding Project Management: philosophical
and methodological issues. Paper presented at EURAM 2006, May 17-20, BI Management School, Oslo.
Söderlund, J. (2002). On the development of project management research: Schools of thought and critique.
Project Perspectives, 8, 20–31.
Themistocleous, G. and Wearne, S.H. (2000) “Project management topic coverage in journals” International
Journal of Project Management 18: 7 – 11.
Turner J.R. (2006a) Towards a theory of project management: the nature of the project. International Journal of
Project Management Vol;24(1):1–3.
Turner J.R. (2006b) Towards a theory of project management: the nature of the project governance and project
management. International Journal of Project Management Vol;24(2):93–5.
Turner J.R. (2006c) Towards a theory of project management: the functions of project management. International
Journal of Project Management Vol;24(3):187–9.
Turner J.R. (2006d) Towards a theory of project management: the nature of the functions of project management.
International Journal of Project Management Vol;24(4):277–9.
Turner, J. R. (1999b). Project management: A profession based on knowledge or faith? (Editorial). International
Journal of Project Management, Vol. 17, No. 6, pp. 329-330.
Turner, J.R. (1999a). The Handbook of Project Based Management. (2 nd Edn) McGraw-Hill.
Weick, K. E. (1995) What theory is not, theorizing is. Administrative Science Quarterly; 40, 3;pp 385 – 389
Wenger, E., (1998) Communities of practice: Learning, meaning and identity. Cambridge, UK: Cambridge
University Press
Whetten, David, A. (1989). What Constitutes a Theoretical Contribution? Academy of Management Review, Vol.
14, No. 4, pp. 490 – 495.
Williams, T. (2004). Assessing and building on the underlying theory of project management in the light of badly
over-run projects. Paper presented at the PMI Research Conference, London.
Winograd, T. and Flores, F. (1986). Understanding Computers and Cognition: A New Foundation for Design.
Ablex, Norwood.
Winter, M., Smith, C., Morris P.W.G, and Cicmil S. (2006) Directions for future research in project management:
The main findings of a UK government-funded research network. International Journal of Project
Management, Vol 24 (8) 638 – 649.
367

Students’ Entrepreneurial Transition in Enterprise
Education: A UK/China Comparison
Jiwei Jenny Shi 1 , Naomi Woodier-Harris 2 and Quan Jin 3
1
University of Central Lancashire, Preston, UK
2
University of Derby, Derby, UK
3
Shanghai Institute of Foreign Trade, China
Jshi5@uclan.ac.uk
naomi.woodier@googlemail.com
jinq@shift.edu.cn
Abstract: This article explores the triggers and barriers of business start-ups amongst business students at two
British and two Chinese universities. It presents the similarities and differences in students’ entrepreneurial
transition in terms of exploring start-up as a career option between the four universities. It helps to further
understand the transitions into student entrepreneurship for further research into establishing an educational
framework to encourage more students’ start-ups. It also discusses the impact of enterprise education in both the
UK and China.
Keywords: entrepreneurship education, entrepreneurial transitions, students entrepreneurship, UK / China,
triggers and barriers, educational framework
1. Introduction and research background
The University of Central Lancashire (UCLan) has a long established and successful reputation of
excellence in employability, enterprise and entrepreneurship education. In 2011, UCLan achieved
several high profile awards including AGCAS (The Association of Graduate Careers Advisory Service)
Employability International Award for Excellence 2011 and was runner up in the 2010 SFEDI national
awards for supporting student start-ups. UCLan also has a long established and extremely successful
record of partnerships in China, which began back in the 1980s.
UCLan has also successfully completed several PMI2 projects (The Second Prime Minister’s
Initiative) which aimed to promote employability, enterprise and entrepreneurship education to priority
countries e.g. China and Nigeria. Each of UCLan PMI2 project focused on promoting employability,
enterprise and entrepreneurship to the different group of students (engineering, fashion and business
students) from a specific discipline. The wealthy of knowledge, the evident demand and benefits of
entrepreneurship education in both the UK and China, and the close relationship between UCLan and
its Chinese partner universities that provided another opportunity for this collaborative research
project into applying best practice of entrepreneurship education both in the UK and China through a
generic educational framework.
This is an action-oriented collaborative research project, which divided into 3 stages as follows:
Stage 1: Understanding Students’ Entrepreneurial Transition
Stage 2: Evaluating Current Entrepreneurship Education
Stage 3: Implementing Current Entrepreneurship Education
This paper presents the findings of the first stage of the research project, which will identify triggers
and barriers of business start-up and students’ attitude and learning needs towards entrepreneurship
at two British and two Chinese Universities. The findings were collected based on an existing
entrepreneurial transition model (Woodier - Harris, 2010) via semi-structured interviews. The transition
model (Appendix 1) states a process of six different stages (Awareness, Interest, Exploration, Career
Option, Engagement and Pre-startup) of entrepreneurial transitions for starting a business. This model
was used to advise, assist and encourage entrepreneurial students to start a business.
The findings answered the following questions:
What are the similarities and differences in university students’ entrepreneurial transition in terms
of exploring start-up as a career option between the UK and China?
What are the key factors to influence Entrepreneurship Education in the UK and China?
What are students’ Needs in Entrepreneurship Education and Further Improvement?
368

Jiwei Jenny Shi, Naomi Woodier-Harris and Quan Jin
The findings will also help to share a deeper understanding of students’ entrepreneurial transitions
and to assess the effectiveness of entrepreneurship education in the later stage.
The project has been undertaken between UCLan, University of Derby, Shanghai Institute of Foreign
Trade (SIFT) and East China University of Science and Technology (ECUST), which was an
extension of a pilot study after the Student Placements for Entrepreneurs in Education Programme
(SPEED). The pilot study was conducted at one of the participating HEI, 3 months following the
completion of the students nine month SPEED programme. The aim was to investigate the affect of
an entrepreneurial programme, such as SPEED on students and their decisions to start, or not start a
business.
The ‘Critical Incident Technique’ that used in the pilot study proved extremely effective, therefore the
method was used together with the entrepreneurial transition model in this collaborative project to
obtain real in-depth insights of students / graduates experiences in enterprise education at each
participated university.
2. Enterprise education in the UK and China
2.1 Why need to improve entrepreneurship education?
This is a time driven by unprecedented change, when being in one of the most severe economic
recessions for generations. The unemployment rate hit the highest level since 2011 with
approximately 25% young people out of work. Students and graduates face increasing difficulties in
starting their careers, in conditions of unprecedented change that they never expected. With a fall in
graduate jobs and a freeze in graduate salaries and high levels of graduate unemployment there are
evidently more challenges that both academics and advisors face in helping students and graduates
explore career decisions and employment prospects.
Research identified that students and graduates appear to be ‘putting off’ starting a business until
after they have gained more experience (Galloway et al., 2005). Many of the reasons for this was; the
time it takes to start a business, develop an idea whilst studying and the need for experience first
before starting up.
In a study by Hannon, (2007) he concludes that there is currently only 7% of students engaging in
enterprise activity. This ranges from attending enterprise workshops through to full degree
programmes at undergraduate and postgraduate level.
Entrepreneurship development is the core to enterprise education, which is central to many
government policies for building a knowledge driven economy and an entrepreneurial culture. As
stated by Hannon, (2006) the role of education is important in building entrepreneurial capability in
tomorrow’s entrepreneurs and leaders. Our economy needs an increasing flow of new knowledgebased
businesses, providing both enhanced and sustainable social and economic value (Birch and
Clement 2006).
However, for progress to be made the educational system has to be orientated towards ‘doing’ more
than ‘thinking’ (Formica, 2002) and traditional teaching has to change to encourage entrepreneurship
(Cooper and Lucus, 2007) and that universities must provide the right environment for student
enterprise by inspiring and motivating individuals to find opportunities, acquire resources and take
action.
2.2 A China - UK comparison
2.2.1 Enterprise initiative and support
With the UK facing unprecedented challenges such as the recent recession, rapid social changes,
and worldwide competition the importance of enterprise is crucial to a countries growth and economic
up-turn. The developments and interest in entrepreneurship have developed worldwide because of
the instrumental role it has on the growth of the economy. As a result of this recognised catalyst for
economic development and job creation, policy makers have started to develop a wide array of
measures to support entrepreneurship education both in the UK and China.
369

Jiwei Jenny Shi, Naomi Woodier-Harris and Quan Jin
In the UK the developments briefly include; Small Business Service (SBS) in 1999, national enterprise
campaigns using Alan Sugar and Richard Branson (successful multi-millionaire entrepreneurs),
establishment of a National Council for Graduate Entrepreneurship (NCGE) in 2004 with the aim of
raising the profile of entrepreneurship and the option of starting a business as a career choice
amongst students and graduates, enterprise modules at undergraduate and postgraduates and
increasingly many Masters in Entrepreneurship. Although enterprise and entrepreneurship has shown
significant developments, and enterprise education has had a growing interest; from a single course
to over 1500 institutions around the world in 50 years (Hannon, 2005) much is yet to be learned about
best practice, what works well and doesn’t and the larger impact of enterprise education on students
and graduates in producing entrepreneurs. There is a noticeable lack of research on graduate careers
in terms of entrepreneurial intentions and development experiences (Nabi and Holden, 2008). It is
suggested that this is important given the increasing numbers of students considering or pursuing
entrepreneurial careers ( Harding, 2007, Holden et al., 2007).
There have also been many regional programmes and initiatives at universities within the UK in
driving forward the enterprise agenda, with graduates being seen as the key to national growth. In
brief some include enterprise clubs / associations e.g. NACUE, business plan competitions, Dragon
Den and Apprentice style activities, enterprise simulations and enterprise games like XING (business
scenario game board) and SimVenture (business computer simulated game) which were all designed
to develop and foster students’ entrepreneurial spirit, mindsets, behaviours and skills.
In China however, although enterprise skills trainings are widely recognised and promoted across
Chinese universities for the purpose of enhanced employability, but entrepreneurship programmes
only have slowly developed in the past 12 years. Enhancing student entrepreneurial capability is
currently one of the highest priorities for the Chinese government (The Ministry of Education, 2010) as
they believe it could help to sustain China’s economic growth. The Government has taken this on
board, planning numerous policies (e.g. first year free tax, free enterprise trainings and increased
business load) to student business start-ups (Cityyouth, 2009).
Based on the model of the first business idea competition in China, the Tsinghua Entrepreneurs Club
(TEC) in 1998 was introduced (Mei and Xu, 2009). In 1999, the Chinese Ministry of Education’s
proposal "face the 21st century Education Promotion making Plans for Action" plans to support
Chinese enterprise education in order to encourage student and graduate business start-ups in
particular in the high-tech sector. Following this in 2002 the Ministry of Education piloted the
entrepreneurship education promotion plan in universities including Renmin University of China,
Tsinghua University, Beijing University of Aeronautics and Astronautics, Shanghai Jiao Tong
University. In 2005, the Central Committee of the Communist Youth League of China (CYLC), the All-
China Youth Federation and International Labour Organization cooperated on establishing KAB
(Know About Business) which was a Chinese entrepreneurship education project for students across
different Chinese universities.
Finally, in 2005 Shanghai set up the first national business start-up fund of 100 million Yuan
(approximately £10 million) for university students every year, piloted to a group of universities,
supporting 230 students’ business start-ups. The government also provides business start-up grants
between £5,000 to £30,000. Zhejiang governments also set up a business start-up fund of 1.1 billion
Yuan to develop more entrepreneurial students (Zhang, 2010). There are many other regional policies
or funding available to student / graduate entrepreneurs across different regions which include
Shanghai, Beijing, Zhejing, Chongqing, Liaoning, Hunman, Fuzhou, Hainan, Henan, Zhengzhou and
Hangzhou.
Despite these achievements, entrepreneurship education in China is still in its infancy compared with
developed countries, like the UK. China faces tough challenges in particular in growing
entrepreneurship, some of these are;
The benefits of entrepreneurship are not being understood properly amongst students, graduates
and academics
Lack engagement from industries and limited financial support from Chinese government
Lack expertise and resources in entrepreneurship education
370

Jiwei Jenny Shi, Naomi Woodier-Harris and Quan Jin
2.2.2 Triggers and barriers of students entrepreneurial transition
Atherton (2007) suggests that the likeliness or propensity of an individual starting a business is based
on the one’s personal experiences prior to the engagement in the new venture. Furthermore,
individuals who had not previously thought about or considered starting a business were stimulated to
do so by an event or influence (such as a friend or colleague starting a business, or losing a job).
There are certain drivers towards business start-up such as; an encounter or interaction with a family
member, friend or acquaintance who had started a business; loss of job which in turn pushes
individuals to explore alternative income opportunities; dissatisfaction with job; and an identification of
an idea or business opportunity (Atherton, 2007).
Previouse studies indicated that the different levels of motivation and skills of the entrepreneur
(Herron and Sapienza, 1992), their attitude, beliefs and abilities could have a great influence on the
decision to enter into a new business (Boyd and Vozikis, 1994). In addition, the decision towards
starting a business is a personal choice that could be different because of the differences in
personality, background, perception of self-employment, role model and network (McStay, 2008).
Family support is also seen as being important support or hinder factor to pursue an entrepreneurial
career (Woodier-Harris, 2010).
Interestingly, there is limited research on Chinese enterprise or entrepreneurship education, with it
only being in its infancy of 12 years and there are fewer research projects conducted regarding the
triggers and barriers of student start-ups in China. One particular paper indicates that 98% of Chinese
students have thought about self-employment (Guangdong Youth Leaders College, 2006); 87% of
Chinese students are interested in the idea of being self-employed (Ren, 2006).
Despite the nation’s ambition, there are approximately 10,000 university graduates unemployed each
year, only 1% of them willingly chose to start a business (Mei and Xu, 2009). In addition, despite the
government effort and students’ passion in business, there are only 30% of student enterprises that
survive within the first five years and only 17% of them make a profit (Yang, 2009).
It is reported that there are many variables preventing many start-up decisions, for example, lack
business experience, funding, support and resources (Yang, 2009). Furthermore, Wang et al, (2004)
suggests that the external environment has not been mature enough to enable student business startups
to flourish because of the slow progress of Chinese enterprise and entrepreneurship education.
However Tsinghua University believes that Chinese students lack the right entrepreneurial attitude,
commitment, creativity and business experience (Yang, 2009).
51job (the biggest national job agency in China) suggested that students primarily started a business
to develop self worth and one of the biggest obstacles reported was funding and lack of practical
experience (51job.com, 2009). Furthermore, it was also reported that a small percentage of
students/graduates have negative influence by their families at the early stages of their decision
making of starting a business (Fan and Wang, 2004).
There are different factors could support or hinder one’s decision on starting a business. Some of
those factors (e.g. personality, family background and influence) that are difficult to be changed, but
other factors (e.g. business experience, identification of idea and opportunity, support and resources)
that could be developed and improved through the entrepreneurship education and trainings. More
importantly, there is important to explore whether there is a positive relationship between intentions to
start a new business and entrepreneurial self-efficacy, and the impact of those triggers and barriers
on entrepreneurial self-efficacy.
3. Research rationale, methodology and process
This research project was divided into the following stages:
Stage 1: Understanding Students’ Entrepreneurial Transition It aims to share a deeper
understanding of students’ entrepreneurial transitions between the participated institutions across
the UK and China and to justify the rationale for further research
Stage 2: Evaluating Current Entrepreneurship Education It aims to evaluate entrepreneurship
education at each participated institution and to assess the effectiveness of entrepreneurship
division amongst entrepreneurial students.
371

Jiwei Jenny Shi, Naomi Woodier-Harris and Quan Jin
Stage 3: Implementing Current Entrepreneurship Education It aims to identify entrepreneurial
capacity in order to propose an educational framework and self-assessment / guidance tool for
both students and academics.
3.1 Rationale
Desirability and entrepreneurial self-efficacy directly influence a student’s decision of starting a new
business; level of students’ previous entrepreneurial experience will influence students’ perceived
desirability of self-employment and entrepreneurial self-efficacy (McStay, 2008)
For the purposes of this research, it is critical to understand entrepreneurial transitions amongst
students before measuring the effectiveness of current entrepreneurship education at participated
universities between the UK and China. It is challenging to influence the start-up process to different
individuals, however, it is possible to identify the commonality and the best practice in the process to
improve current entrepreneurship education both in the UK and China. At the first stage of this
research project, it focussed on understanding more about the triggers and barriers to students’
entrepreneurial transition and their perceptions and learning needs in entrepreneurship education.
This phase of study extends the current body of research in the path to entrepreneurship and the selfemployment
intentions, it is also a critical step to shape future direction of the research.
3.2 Sample and limitation
Research participants are all undergraduate students who have undertaken an enterprise module,
course or programme, which taking them through idea generation to launch a business during their
studies. A sample of ten students was selected from different enterprise initiatives / programmes with
focus of business start-up at each participated university across two semesters in 2010. The
participants were selected only based on their availability and willingness for the first 10 interviews at
each university. This student population provided a total sample of forty interviews. Although this
research was conducted at a relatively small scale which perhaps may not be representative of every
university within the UK and China, the qualitative data collated have allowed the authors to gain a
deeper insight into students’ entrepreneurial transition. Different level of experiences of the
researchers, resources, geographic and cultural differences are the limitations to this research project.
3.3 Method
A wide range of articles from books and journals relating to entrepreneurship education, the rationale,
both China and UK government’s initiative and support, and the triggers and barriers of
entrepreneurial transitions were reviewed before undertaking the project. The project employed
qualitative research technique due to the nature of the project, which attempts to capitalise on the
subjective as a means for understanding and interpreting human experiences (Polit and Hungler,
1995). “Qualitative research is essential to address questions of what, how (process) and why (Baker,
1992), at this stage of the project was aimed to understand what are the triggers and barriers of
students’ start-ups and how they transformed through the entrepreneurial education through students’
perspectives.
According to the perceived research needs, the research process was divided into the following steps:
Step 1: Literature review
Step 2: Design semi-structured interviews
Step 3: Data collection and analysis
Step 4: Make recommendations for future research based on research results and peer debate
The semi-structured interview was divided into four parts. These are as follows:
Part 1: Students Profile (age, sex, experience of enterprise training and etc.)
Part 2: Student Entrepreneurial Transition
Part 3: Students’ Attitude towards Entrepreneurship
Part 4: Students’ Needs in Enterprise Education
The same semi-structured interview questionnaires was used and translated from English to Chinese
for Chinese participants over the two semesters to remove the language barriers between the
372

Jiwei Jenny Shi, Naomi Woodier-Harris and Quan Jin
participants. The data were collected collaboratively by the researches in their own home countries.
An overview of the nature of the study was provided to students and clear instructions were given that
participation was entirely voluntary.
4. Discussion and analysis of semi-structured interviews
4.1 Key findings
The following key findings were highlighted from semi-structured interviews conducted with 40
students from two British and two Chinese universities (Appendix 2). All participants had received
some form of enterprise trainings during their studies and all were either interested in creating a new
start up or had already started.
Key findings of Part 1(Students Profile):
Research data highlighted significant differences between British and Chinese students. These are as
follows:
British students were considerably older (the majority of students are aged between 25-30, with
some aged 35 and over). Chinese students were all under the age of 25.
Interestingly there is approximately equal numbers of male and female British students, whereas
male Chinese students significantly outnumber their female counterparts.
There are more British students in the process of starting a business.
Students with a family business background are more likely to start a business, especially in
China.
The prior entrepreneurial trainings and experiences received are more satisfactory to British
students.
Key findings of Part 2 (Student Entrepreneurial Transition)
The results from the second part of the semi-structured interviews highlighted that the majority of
British students had already started their businesses after undertaking enterprise training, whereas
Chinese students were still at the infancy stages of their entrepreneurial journey.
There are similarities and differences in terms of triggers and barriers at each stage of students’
entrepreneurial transition. These are as follows:
At Awareness Stage, both sets of students perceived a talk or an event to be an influential trigger to
business start-ups. Inspirational role models have the most significant influence amongst students
especially in China. The negative image of self-employment was the major barrier, in particular
amongst Chinese students due to the negative perception of self employment held by family
members.
At Interest Stage, both sets of students reported increasing confidence in starting a business after
studying enterprise. Lack of employment opportunities is another common ‘push’ factor. However,
limited funding plus the fear of failure are major barriers preventing students exploring further
opportunities.
At Exploration Stage, British students believed they were provided with more opportunities to explore
and develop their ideas, whereas Chinese students felt they lack the training and knowledge required
to take their ideas forward. Both sets of students believed enterprise support and extra motivation
could further develop or hinder their entrepreneurial transition.
At Career Option Stage, self-belief is the trigger for both sets of students. Many Chinese participants
were more likely to take the risk compared to their British counterparts. . Lack of practical experience
is the only barrier at this stage to both sets of students.
At Engagement Stage, self-motivation and relevant skills are the key factors to both sets of students
in engaging further enterprise activities. Chinese students sought practical advice and support from
373

Jiwei Jenny Shi, Naomi Woodier-Harris and Quan Jin
their families and friends whereas British students requested assistance and advice from universities
and professional bodies.
At Pre-Startup Stage, the burning desire, the passion for business and the ‘can do’ attitude are the
triggers for both sets of students. The financial aspect of starting a business presents a main issue to
Chinese students due to the cultural differences and attitudes towards borrowing money.
Part 3: Evaluation of Entrepreneurship Education
Key findings of Part 3 (Students’ Attitude towards Entrepreneurship):
Both British and Chinese students reported increased awareness and confidence in entrepreneurship
due to their enterprise education. The knowledge and skills attained help to enhance their
employability and to explore self-employment as an alternative career option. Many students also
believed the practice of entrepreneurship could be applied in a wider context rather than restricted to
business start-up.
The motivation of starting a business is different between British and Chinese students; British
students had a clear understanding of why they wanted to start a business e.g. freedom, total control,
make money and an alternative employment opportunity. In contrast, Chinese students with a family
business background were more likely to take the start-up risk and had a stronger desire for wealth
creation. Interestingly, Chinese students’ attitudes towards entrepreneurship are predominantly
influenced by their family members’ perception on self-employment.
Key findings of Part 4 (Students’ Needs in Enterprise Education):
Enterprise skills training is evidently more advanced in British institutions compared to their Chinese
counterparts. However, limited practical experience and business advice during the entrepreneurial
development stage were common issues at both British and Chinese universities. For British students,
enterprise skills help to build confidence that acknowledged being an advantage for those who are
willing to take leadership in an organisation; entrepreneurial awareness, knowledge / skills and
business work experience are key factors to business start-ups. For Chinese participants, external
factors such as on-going mentoring, funding and government support are more influential to the
success of business start-ups; most of them believed enterprise skills could be outsourced or covered
by in-house expertise if they were to start a new business. Both British and Chinese students
suggested that entrepreneurial awareness, knowledge / skills and the intelligence of know-how are
the key ingredients to successful entrepreneurship.
5. Conclusions and future work
5.1 Similarities and differences in entrepreneurial transitions compared between
British and Chinese students
For both British and Chinese students, similar triggers and barriers have been identified at each stage
of the entrepreneurial transitions process. Self-believe, motivation and ‘can do’ attitude are the key
triggers to a successful business start-up. Limited awareness, confidence, practical experience and
fear of failure are the main concerns. Both British and Chinese students who come from family
background and also received practical advice and support from local business community are the
ones have established a business.
There were more significant number of British students started a business than Chinese students.
British students were given more opportunities to explore and develop their ideas, they were more
satisfied about the received entrepreneurial training and felt more confident to pursue the
entrepreneurial career path. Although Chinese students from family business background expressed
that they were more willing to take risks when opportunities presented, however, most students were
still at the early stages of the entrepreneurial transition cycle because they felt they were not equipped
with the right knowledge and skills. Family influence is a significant factor in supporting or hindering
start-up success amongst Chinese students. Furthermore, financial constraints and the cultural
differences and attitudes towards borrowing money presented another negative influence.
374

Jiwei Jenny Shi, Naomi Woodier-Harris and Quan Jin
5.2 Key influential factors of entrepreneurship education to students in the UK and
China
Based on the interviews results and the peer discussion, the authors realised that both British and
Chinese students have increased their confidence dramatically in starting a business after receiving
entrepreneurship training. Entrepreneurship education has positive influence during and after
students’ entrepreneurial journey, but it is worth noting that success should not be measured solely by
the number of new business start-ups. Regardless of the triggers and barriers of starting a business,
self-efficacy is the most influential factor that is closely linked with start-up decision amongst both
British and Chinese students. The research also shows the students’ experience in entrepreneurship
training programme could hinder or support the start-up decision. There are differences in
entrepreneurship programmes between the UK and China because of many external factors:
Culture
In China, the concept of entrepreneurship is relatively new and perception of self-employment
remains negative to most ordinary Chinese families. Family has significant influence and in some
instances, total control on a student’s career choice.
Purpose
In China, the purpose of enterprise education is enhanced employability. Chinese academics believe
that entrepreneurship is not suitable for every student. In the UK, universities like UCLan believe
enterprise education aims to develop more entrepreneurs and intrapreneurs as well as creating an
entrepreneurial culture.
Systems
Employability service, curriculum design and extra curriculum activities and support are similar
between the two British and two Chinese universities. However, the approaches to enterprise
education are different.
In China, skills trainings is the focus of enterprise education for enhanced employability;
entrepreneurship trainings are provided separately and primarily through non compulsory extracurriculum
activities. In the UK, in particular at UCLan, the basic elements of enterprise are embedded
within curriculum and extra-curriculum enterprise activities; entrepreneurship programmes are
delivered with a flexible approach in mind.
5.3 Students’ needs in entrepreneurship education and further improvement
Higher numbers of student start-ups were identified at institutions with an incubation / hub, where
networks, practical advice and support were provided. Both British and Chinese students believed
enterprise education is not solely about encouraging them to start a business. It should prepare them
for the world of work and business. Most students preferred to gain some business experience after
graduation before venturing into business start up. The mind-set, knowledge and skills developed
through enterprise education will enable them to secure a good position within a company and these
attributes will also help them to consider starting a business later in life. Both sets of students hoped
to gain a deeper insight into entrepreneurship and self-employment, to learn more transferable skills;
not necessarily enterprise skills alone, but more importantly, practical business experience. Potential
student entrepreneurs expressed a desire to attaining more practical business experience and
business start-up guidance. Interestingly, students also reported that enterprise knowledge and skills
were not always deemed a necessity if a viable business opportunity presented itself. Skills could be
outsourced and a pool of skills and people could be built. Most students believed enterprise education
provided them with transferable skills and business acumen, giving them a choice of being an
entrepreneur or intrapreneur or both.
The authors believe the purpose of enterprise education needs to be clarified, based on
understanding the students’ needs. The key findings of this research project suggest the following:
promote the benefits of self-employment and entrepreneurship as early as possible
help students to establish a positive mind-set and attitude towards self-employment
link local entrepreneurial talents to university enterprise programmes
provide supportive networks, practical advice and on-going mentoring
375

establish an entrepreneurial culture
Jiwei Jenny Shi, Naomi Woodier-Harris and Quan Jin
develop students’ entrepreneurial capacity which including awareness, enterprise knowledge /
skills and entrepreneurial intelligence
develop entrepreneurial spirit, self-believe, self-efficacy and ‘can do’ attitude
5.4 Conclusions and future work
Entrepreneurship education in the UK is continuously evolving. The research helped authors to
understand insights of students’ entrepreneurial transitions and their needs in entrepreneurship
education. However, the challenge still remains. Despite there are a range of different factors could
hinder or support the business startup decision, how can we develop more entrepreneurs? What are
the key elements of entrepreneurial capacity that we could help students to develop in order to build
their self-efficacy? Future work will focus on evaluating current entrepreneurship education and
developing students’ entrepreneurial capacity. An educational framework will be developed for
academics and career advisors to raise the awareness of entrepreneurial transitions and to develop
entrepreneurial capacity at the each stage of the transition. A self-assessment tool will be also
developed to guide students to go through the transitions.
6. Appendix
6.1 Entrepreneurial transition model
Decision to put
entrepreneurial
intentions on hold
No business start-up
Awareness Stage
Through friends and family in business
TV Media / events / Travelling
Interest Stage
Short/Long-term Aspirations
Motivated by event/talk
Exploration Stage
Talk to friends& family
Speak with advisors and people in
business
Career Option Stage
Students explore their ideas
further. Conduct research, speak to
people in business, and gather
market research
Engagement Stage
Student’s intentions are to
actively develop and test their
ideas and start a business
Pre-Startup Stage
The students experience business
start-up, the ‘effective’ & ‘noneffective’
incidents make up their
learning experiences.
Business start-up
376
Decision to pursue
entrepreneurial
intentions further
Future business Startup

7. Appendix 2
Jiwei Jenny Shi, Naomi Woodier-Harris and Quan Jin
7.1 Key summary results from semi-structured interview
Students profile
Question UK China
Derby UCLan SIFT SH
1 5% under 25
40% over 35
85% under 25 95% under 25
50% 25-30
30% under 25
40% 30-35
15% 25-30
5% over 35
15% 30-35
2 40% Male
60% Male
65% Male
25% Male
60% female
40% female
35% Female
75% Female
3 55% family
business
55% family business 5% family business 35% family business
4 90% with training 75% with trainings 30% with trainings 70% with trainings
The Stage business startup development
Stage UK China
Derby UCLan SIFT SH
Awareness 80% 30%
Interest 15% 20% 25%
Generating ideas 5% 15%
Career option 5% 10%
Engagement 15% 5% 10%
Start-up 85% 70% 10%
Triggers and Barriers
Triggers UK China
Derby UCLan SIFT SH
Awareness a) friends/family a)friends/family c) Inspirational
a)tv/media
c) event/talk c) event/talk role model
c)Talk/event
c)talk/even
Interest a) study enterprise a)study a)study enterprise c)lack career opp.
c) lack of career enterprise c)lack career opp. c)tv/media
opportunities b)family
influence
Generating ideas c)opportunity d)personal
b)enterprise
presented
drive
workshop&training
d) curiosity
c)opp
presented itself
c)opp presented itself
Career option a) benefits of self- c)self-believe
b)benefits of selfemployment
d)viability of
employment
d)viability of idea idea
c)self-believe
Engagement a) practical advice c)relevant skills
d)self-motivation
d) self-motivation d)selfmotivation
Start-up c) Desire/passion c)desire/passio
c) Desire/passion
d) can do attitude n
d)can do
attitude
a)funding
Barriers UK China
Stage Derby UCLan SIFT SH
Awareness b) security of a)access to info d)negative c)family influence
job d)negative perception perception of d)negative perception
c) family of self-employment self-employment a)access to info
influence
a)family influence
Interest d) other
c)funding
c)funding
c)funding
commitments
c) funding
b)fear of failure b) fear of failure b)fear of failure
377

Generating ideas b) lack of time
a) lack of
Jiwei Jenny Shi, Naomi Woodier-Harris and Quan Jin
commitment
Career option a) risk
c)lack of
experience
Engagement c) lack of selfmotivation
b) lack of
support from
family/friends
Start-up b) overcome
challenges
c) lack of
commitment
References
c)limited support
d)lack of motivation
d)unviable ideas
a)perceived risk
b)lack of support from
family /friends
a)lack of practical
advice/support
c)financial constraints
a)fear
c) limited support
d) lack of motivation
c) lack of practical
experience
a)lack of practical
advice/support
c) financial constrains
Brooks, I. and Weatherston, J. (1997) The Business Environment: Challenges and Changes, Prentice Hall,
London.
Doherty, Noel and Delener, Nejdet. (2001) “Chaos Theory: Marketing and Management Implications”, Journal of
Marketing Theory and Practice, Fall, Vol 9, No. 4, pp 66–75.
Atherton, A. (2007) "Preparing for business start-up: 'pre-start' activities in the new venture creation dynamic"
Journal of Small Business and enterprise development, vol.14, no.3, pp.404 - 417.
Baker, M. (1992) In Other Words. Routledge.
Boyd, N. G. & Vozikis, G. S. (1994) "The influence of self-efficacy of entrepreneurial intentions and actions"
Entrepreneurship: Theory and Practice, vol. 18, No. 4, PP. 63-77.
Birch, C. and Clement, B. (2006) “Creating graduate entrepreneurship through self-employed work placements -
Project SPEED”, paper presented at ISBE 2006, Cardiff.
Cityyough (2009), “Shanghai Graduate Policies in Business start-ups”, available at:
http://www.cityyouth.cn/node/2165 (accessed 4 May 2009)
Cooper, S. and Lucus, W. (2007) “Building Entrepreneurial self-efficacy and intent through education and
experience”, paper presented at ISBE 2007, Glasgow.
Fan, W. and Wang, Z. M. (2004), “The Study of Entrepreneurial Intention and Its Determinants”, Psychological
Science, Vol. 27 No.5, pp. 1087-1090
Formica, P. (2002) “Entrepreneurial Universities: the value of education in encouraging entrepreneurship”,
Industry and Higher Education, Vol. 16, pp. 167-175.
Galloway, L., Anderson, M., Brown, W. and Whittam, G. (2005) The impact of entrepreneurship education in HE,
Report for Business Education Support Team, Heriot-Watt University, Edinburgh.
Guangdong Youth Leader College (2006), “Research Report of University Students’ Entrepreneurial
Capabilities”, Journal of Guangdong Youth Leaders College, Vol. 20 No.63, pp.11-16
Hannon, P. P. D. (2005) The journey from student to entrepreneur: A review of the existing research into
graduate entrepreneurship, IntEnt Conference, University of Surrey.
Hannon, P. (2006) “Teaching pigeons to dance: sense and meaning in entrepreneurship education”, Education
and Training, Vol. 48, pp. 296-308.
Hannon, P. (2007) “Enterprise for all? The fragility of enterprise provision across England's HEIs”, Journal of
small business and enterprise development, Vol. 14, pp. 183-210.
Harding, R. (2007) Graduate Entrepreneurship in the UK: Summary report from GEM UK Data, Birmingham,
National Council for Graduate Entrepreneurship.
Herron, L. & Sapienza, H. J. (1992) "The entrepreneur and the initiation of new venture launch activities"
Entrepreneurship Theory and Practice, vol.17, no.1, pp.49-55.
Holden, R. J., Jameson, S. & Walmsley, A. (2007) "New Graduate employment in SME's: still in the dark?"
Journal of small business and enterprise development, vol.14, no.2, pp.211-27.
McStay, D (2008) An investigation of undergraduate student self-employment intention and the impact of
entrepreneurship education and previous entrepreneurial experience, PhD, ePublications@bond, Faculty of
Business, Technology and Sustainable Development.
Mei, W. H. And Xu, X. Z. (2009), “Problems and Strategies of Entrepreneurship Education in Higher Institutions in
China”, Educational Research, Vol.351 No. 4, pp.67-72
Nabi, G. & Holden, R. (2008) "Graduate entrepreneurship: intentions, education and training" Education and
Training, vol.50, no.7, pp.545-551.
Polit, D.F. and Hungler, B.P. (1995). Nursing Research: principles and Method. 5 th ed. J.B. Lippincott
Ren, G. Y. (2006), “A survey and Research on Establishing Business Quality of Graduates of Colleges and
Universities”, Journal of China Institute of Industrial Relations, Vol. 20 No.5, pp.97-105
The Ministry of Education (2010), “Suggestions of Higher Education in Innovation, Entrepreneurship Education
and students / graduates self-employment”, available at: http://202.123.110.5/gzdt/2010-
05/06/content_1600716.htm (accessed 10 October 2010)
378

Jiwei Jenny Shi, Naomi Woodier-Harris and Quan Jin
Xu, X. Z. and Ye, Y. H. (2010), “Influencing Factors and Promoting Strategies of the Undergraduates’
Entrepreneurial Belief”, Research in Higher Education of Engineering, No. 3, pp.78082
Woodier-Harris, N.R. (2010), “Evaluating the impact of SPEED on students’ career choices: a pilot study”,
Education and Training Journal, Vol. 52, No. 6/7, p 463-476
Yang, Li. (2009), “Survey of business-start-up of university students in Wuhan”, Journal of Wuhan Institute of
Technology, Vol. 31 No. 2, pp. 65-71
Zhang, H. J. (2010), “Predicament and countermeasure of Developing Entrepreneurship Education for University
Students”, Education Exploration, Vol. 226 No. 4, pp.65-68
51job.com (2009), “Students / Graduates Business Start-up Research”, available at:
http://arts.51job.com/arts/79/296711.html (accessed 20 June 2010)
379

Parameter Estimation Using Asymptotic Analogy
Anthony Stacey
Graduate School of Business Administration, University of the Witwatersrand,
Johannesburg, South Africa
Anthony.Stacey@wits.ac.za
Abstract: Conventional analysis of quantitative research data calculates sample statistics from which population
parameters and characteristics are inferred. As a result of using ordinal response scales, much quantitative
research data is subject to discretisation error. This error compounds the well-recognised and quantifiable
sampling error, but is generally overlooked. Discretisation error results in unreliable estimates of population
parameters such as means, standard deviation, correlation coefficients and the like. Hypothesis tests and other
inferences are similarly affected. It will be demonstrated in this paper that the magnitude of discretisation error
may be substantial. A novel methodology will be presented that completely eliminates discretisation error by
treating ordinal data as strictly ordinal, and avoiding incorrectly attributing a numeric property to an ordinal
response scale. In its generic form, the asymptotic analogy methodology involves the creation of a numerical
(agent based) simulation model of individuals’ attitudes, perceptions, opinions, thoughts, feelings, beliefs or
judgements. The initial input parameters of the model are completely arbitrary. Summary statistics are calculated
for the simulation model corresponding to the research sample statistics. The input parameters of the model are
then incrementally adjusted using a goal seeking algorithm which minimises the differences between the sample
statistics and the summary statistics for the numerical model. The numerical model is thus asymptotically
calibrated to the observed research data. Two examples of parameter estimation using asymptotic analogy will
be presented. In the first exposition of the methodology, the means and standard deviations of verbal ordinal
(Likert-type) survey data will be estimated by fitting normal distributions to the raw categorical data. The second
illustration of the methodology will demonstrate the analysis of rank-ordered survey data, thereby resolving a
frustrating and intractable problem for quantitative researchers. Apart from the obvious theoretical benefits of
eliminating discretisation error, the methodology has the practical benefit of not requiring technical statistical
software because the modelling can be implemented using readily available desktop spread sheet software.
Keywords: parameter estimation; scale reliability; agent based modelling, simulation modelling
1. Introduction
Researchers frequently want to evaluate the attitudes, perceptions, or beliefs of a large group of
people. Generally these cannot be observed directly and it is necessary to use indirect methods, such
as a survey. For various reasons, one usually cannot research everybody and researchers survey a
sample, using these results as indicative of the population as a whole. It is well known that this
process of statistical inference is flawed; the sample results are not always an accurate evaluation of
the whole population.
Perhaps the most widely known and analysed flaw in this approach is sampling error. The results
from the sample will depend on the particular sample chosen because of variations in the population.
Sampling error has been adequately described and quantified in the literature.
A less obvious flaw of statistical inference occurs when people respond by choosing one of a discrete
set of options. Feelings, attitudes and the like, do not fit neatly into discrete categories. Classifying
these into categories thus introduces error into the estimates which is generally neither recognised
nor analysed. Sorting and rank ordering items on the basis of some criterion overcomes this problem.
However, rank ordered survey data cannot be analysed using conventional statistical analysis.
It will be shown that these shortcomings of statistical inference from survey data can nonetheless be
overcome. This will be done by creating and calibrating numeric models that simulate how people
respond to surveys.
2. Estimation of population parameters using statistical inference
Statistical inference is a well-established process in quantitative research. The principle is that in
order to determine the statistical characteristics of a population, it is unnecessary to analyse data for
the entire population (Upton & Cook, 2006). Provided certain prerequisites are observed, data
concerning or obtained from a subset of the entire population, that is the sample, represents the
characteristics of the population as a whole with sufficient accuracy and reliability through the process
of inference (Everitt, 2002).One way of representing the process of statistical inference as it pertains
to research is illustrated in Figure 1.
380

Anthony Stacey
Figure 1: The process of statistical inference
Somewhat simplistically, the process of statistical inference can be disaggregated into the following
phases:
Scoping. Firstly the scope of the research is used to define the research population, which is
then used to determine a relevant sample frame. The scope of the research is also used to
identify the population parameters which are relevant to the research, such as means, variances,
correlation and regression coefficients.
Data collection. Data are gathered from the sample typically by means of a survey. This sample
data can be assumed to be representative of the population as a whole provided the sample has
the characteristics of a probabilistic sample.
Analysis. The relevant sample statistics, corresponding with the population parameters which are
relevant to the research are then calculated.
Inference. Relying on the assumption of having a representative sample, the sample statistics
are taken to be unbiased estimates of the relevant population parameters. It is understood that
these estimates of the population parameters are subject to random sampling error. However, the
magnitude of the sampling error can be quantified – it is dependent on the size of the sample, and
the researcher ensures that the sample size is large enough to ensure that the estimates of the
population parameters are sufficiently precise.
3. Ordinal level response data
3.1 Use of ordinal data formats in surveys
Constructs such attitudes, perceptions, and preferences are generally impossible to evaluate through
direct observation. Therefore researchers resort to indirect data sources, typically using some sort of
self-report survey methodology (Blumberg, Cooper, & Schindler, 2005). Another characteristic of
these constructs is that although they are continuous and unidimensional, they do not have any unit of
measurement. This may be one reason that researchers have devised rating scales with verbal
descriptors, such as the ubiquitous scale of agreement developed by Likert (1932).
The convenience of verbal ordinal rating scales, with a relatively limited number of response
categories, is substantiated by Miller (1956) who concluded that humans’ ability to make
unidimensional perceptual judgements is finite and surprisingly poor. It was found that typically
humans are unable to differentiate accurately between more than approximately seven levels of a
particular construct. Hence the popularity among researchers of rating scales of up to seven points.
Although Labovitz (1970) identifies advantages and substantiates the reliability of treating such
ordinal level data as interval data, the author concludes with a note of caution that “Further
exploration and tests are necessary for added confidence in treating ordinal data as if they are
interval” (Labovitz, 1970, p. 523). The author notes that analysing ordinal data as strictly ordinal is the
more conservative approach.
Ironically, the convenience and face validity of using verbal ordinal response scales, generically
referred to as Likert-type scales, is in contrast with the ongoing debate about how such observed data
381

Anthony Stacey
ought to be treated (Carifio & Perla, 2007). Despite the well documented pitfalls (Kaptein, Nass, &
Markopoulos, 2010; Mayer, 1971), it remains common practice to analyse ordinal level data as if they
are interval level. By way of example, in a review of three prominent education research journals,
Harwell and Gatti(2001) found that approximately three quarters of the variables used were ordinal in
nature but treated as interval level.
Svensson (2000) compared the use of continuous and discrete response scales. However, the
research did not explicitly model survey participants’ cognitive process that they follow to classify their
underlying attitudes, perceptions, or preferences into discrete response categories. Discretisation
error is the error resulting from discretising the underlying continuous variables which contaminates
the observed data, compounding the well-recognised and quantifiable sampling error.
3.2 Discretisation error in ordinal response data
As a demonstration of discretisation error, consider a survey of some arbitrary construct using a fivepoint
Likert scale of agreement (from 1 corresponding to “Strongly disagree” to 5 corresponding to
“Strongly agree”). Let the mean level of agreement among the population be the fourth response
category, 4 corresponding to “Agree”, and let the standard deviation of the level of agreement among
the population be equal to one response category. The effect of sampling error can be negated by
considering a sample that is sufficiently (infinitely) large. Now, Table 1 shows the calculation of the
sample mean and sample variance of this arbitrary construct.
Table 1: Calculation of sample mean and sample variance
Response category
Numeric
value, x
Proportion of
responses, p x
x.px [x – E(x)] 2 [x – E(x)] 2 .p x
“Strongly disagree” 1 0.0062 0.0062 8.5672 0.0532
“Disagree” 2 0.0606 0.1212 3.7133 0.2250
“Neither agree nor
disagree”
3 0.2417 0.7252 0.8593 0.2077
“Agree” 4 0.3829 1.5317 0.0053 0.0020
“Strongly agree” 5 0.3085 1.5427 1.1514 0.3552
3.9270 Sx 2 = 0.8432
With the effect of sampling error having been negated, the sample mean is expected to be equal to
4.0000 (corresponding to the “Agree” category) and the sample variance equal to 1.0000
(corresponding to one response category) if they are indeed unbiased estimators of the
corresponding population parameters. However, it is clear in Table 1 that the sample mean and
variance thus calculated are not equal to the expected values. The difference between the expected
values and the calculated values is attributable to discretisation error. If the conventional process of
statistical inference was followed with respect to this construct, then the estimates and inferences
would be patently unreliable. This vindicates the caveat issues by Labowitz (1970).
4. Estimation of population parameters using asymptotic analogy
The process of statistical inference does not take into account the cognitive process that participants
follow in responding to survey items and questions using ordinal rating scales. The process of what is
being referred to as asymptotic analogy explicitly takes this cognitive process into account and
successfully eliminates discretisation error. The process is illustrated in Figure 2 and described as
follows:
Scoping and Data collection. The first two phases of the process are no different from
conventional statistical inference. This is convenient because the methodology can be applied to
existing research designs.
Modelling. The third phase of the process is the development of a simulation model of the
cognitive processes that the participants follow in responding to survey items and questions using
ordinal rating scales. It will be shown that this involves certain assumptions regarding population
382

Anthony Stacey
distributions. However, the simulated population parameters comprising the inputs to the model
can initially be arbitrarily assigned. The behaviour of this numerical model simulates the cognitive
processes of the individual participants, and the model is an analogy of the population.
Evaluation. Depending on the arbitrarily assigned simulated population parameters, there will
initially be a difference between the sample data obtained from participants and the outputs of the
numerical model. It is therefore appropriate to calculate a relevant error statistic that evaluates the
goodness-of-fit between the sample data and the numerical model.
Optimisation. The goodness-of-fit can now be improved by changing the simulated population
parameters. The goodness-of-fit will be optimised when the error statistic is minimised. This can
be achieved by following an algorithm which asymptotically minimises the error statistic by varying
the simulated population parameters. Therefore the numeric model asymptotically becomes an
accurate analogy of the cognitive process that participants follow in responding to survey items
and questions.
Inference. Finally, it is argued that provided the goodness-of-fit between the sample data and the
numerical model is sufficient, then the simulated population parameters in the model are reliable
estimates of the true population parameters.
Figure 2: The process of asymptotic analogy
In summary, the underlying principle of asymptotic analogy is that if a simulation model can be
developed of the participants’ responses such that there is an insignificant difference between the
model itself and the observed sample data, then the input parameters of the model are valid and
reliable estimates of the true population parameters.
Two examples and illustrations of the asymptotic analogy methodology are given in Paragraphs 4 and
5.
5. Ordinal response modelling and distribution fitting
The methodology of asymptotic analogy does not treat ordinal level data as if it is interval and is
therefore not subject to discretisation error. Consider a survey comprising a number of items with the
same verbal ordinal (Likert-type) response formats. Let the survey items be framed as statements and
let the response format be the degree of agreement / disagreement with each item. In order, for
example, to determine statistically significant items, it is necessary to evaluate the means and
variances of all items. However, there is neither any basis nor any need to assume that the response
format is interval scaled (i.e. that the response categories are of equal width).
5.1 Development of the simulation model
The following assumptions will be made in the development of the simulation model:
Equivalent interpretation of response categories across participants. All participants will be
assumed to have a similar interpretation of the verbal descriptors that define each response
category. This assumption is implicit in any survey using verbal ordinal response scales, and is
considered reasonable provided the validity of the response scale has been established.
Consistent interpretation of response categories across survey items. Similarly, participants
will be assumed to have a consistent interpretation of the verbal descriptors that define each
response category throughout the survey. Again, this assumption is implicit in any survey that
383

Anthony Stacey
uses a single verbal ordinal response format, and is considered reasonable provided the survey
design is apposite.
Continuity and unidimensionality of the underlying response variable. This is warranted
given that the underlying response variable (in this example the extent of agreement or
disagreement) does not have discrete values and can vary continuously between two extremes.
Normal distribution of opinions, views, sentiments, or perceptions. This is merely an
assumption of convenience not necessity, and other distributions of the underlying variable may
be assumed if appropriate. This assumption is justifiable on the basis that the underlying variable
does not have an absolute unit of measurement.
Standardised parameters. For convenience, the parameters of the model are standardised (with
an overall mean of zero and a variance of unity) because the underlying variable is
dimensionless.
The cognitive process involved with discretisation of the response variable “Extent of agreement” into
distinct response categories is illustrated in Figure 3. As the extent of agreement moves from the
negative “Strongly disagree” extreme in a positive direction towards agreement, there is an implied
threshold that differentiates and separates a “Strongly disagree” response from a “Disagree”
response. This threshold extent of agreement is labelled τ1 in Figure 3. Similar thresholds, labelled
τ2to τ4, differentiate between and separate the other response categories.
“Strongly
disagree”
τ 1 τ 2 τ 3 τ 4
“Disagree”
“Neither agree
nor disagree”
Extent of agreement
“Agree” “Strongly
agree”
Figure 3: Discretisation of continuous response scale
Input parameters to the model are the population means and variances of the survey items, and the
values of the thresholds. Arbitrary values can be chosen for these parameters, and for convenience it
is recommended that the following values are initially assigned:
τi = for i = 1 to (R - 1)where R = the number of response categories, and
μj = 0 and σj 2 =1 for all j = 1 to S where S = the number of survey items.
The modelling of the discretisation of the continuous responses is illustrated in Figure 4. In this figure,
the left hand chart illustrates the response data for a particular survey item, and the dotted normal
distribution curve on the right hand chart represents the numerical model of underlying opinions,
views, sentiments, or perceptions. Clearly the goodness-of-fit between the response data and the
numerical model is poor, due to the input parameters to the model having been arbitrarily assigned.
Figure 4: Modelling discretisation of continuous responses
384

5.2 Model fitting
Anthony Stacey
The observed number of responses in each response category to each survey item is known simply
from tallying the observed data. The expected number of responses in a particular response category
to a particular survey item is given by the following equation:
where:
i = the response category for i = 1 to R
j = the survey item for j = 1 to S
n = sample size
= the normal distribution function with mean of μ, and standard deviation of σ
μj = the population mean of the j’th survey item
σj =the population standard deviation of the j’th survey item
τ0 = -∞
τR = ∞
This integral can be calculated using the NORM.DIST function in Microsoft Excel 2010 (NORMDIST in
earlier versions). Therefore, a statistic representing the goodness-of-fit between the observed data
and the simulation model can be calculated. It is suggested that the χ 2 statistic of goodness-of-fit be
calculated as follows:
Table 2 illustrates the simulation model with the arbitrarily assigned input parameters for a survey of
company directors’ attitudes towards various elements of corporate governance. Model input
parameters are highlighted in bold typeface. Notice that the p-value is very nearly equal to zero
indicating, as expected, a significant difference between the observed data and the simulation model.
Now the numeric model can be fitted to the sample data by minimising the value of χ 2 . This is
achieved by changing the values of the input parameters τi , μj and σj subject to the conditions that the
overall mean is equal to zero, and the overall variance is equal to unity. That is:
There are various potential algorithms for minimising the value of χ 2 that are beyond the scope of this
article. Suffice it to say that the Microsoft Excel “Solver” add-in proves adequate to optimise the
numerical model.
Table 3 illustrates the simulation model after the goodness-of-fit has been maximised by varying the
input parameters.
Table 2: Illustration of the normal distribution fitting model before optimisation
Survey items
Observed response
Item A Item B Item C Item D Item E Item F Item G
frequencies Oij j = 1 j = 2 j = 3 j = 4 j = 5 j = 6 j = 7
Strongly disagree i = 1 1 2 1 1 1 1 2
Disagree i = 2 3 5 6 1 9 1 9
Neither agree nor
disagree
i = 3 2 3 3 2 4 3 6
Agree i = 4 18 15 13 15 9 13 8
Strongly agree i = 5 1 0 2 6 0 7 0
385

Anthony Stacey
Survey items
Sample size n 25 25 25 25 23 25 25
Population means μj 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Population standard
deviations
σj 1.00 1.00 1.00 1.00 1.00 1.00 1.00
Expected frequencies Eij Item A Item B Item C Item D Item E Item F Item G
Strongly disagree 1.7 1.7 1.7 1.7 1.7 1.7 1.7
Disagree 6.0 6.0 6.0 6.0 6.0 6.0 6.0
Neither agree nor
disagree
9.6 9.6 9.6 9.6 9.6 9.6 9.6
Agree 6.0 6.0 6.0 6.0 6.0 6.0 6.0
Strongly agree 1.7 1.7 1.7 1.7 1.7 1.7 1.7
χ 2 contributions Item A Item B Item C Item D Item E Item F Item G
Strongly disagree 0.27 0.07 0.27 0.27 0.27 0.27 0.07
Disagree 1.53 0.18 0.00 4.21 1.45 4.21 1.45
Neither agree nor
disagree
5.99 4.51 4.51 5.99 3.24 4.51 1.33
Agree 23.66 13.27 8.01 13.27 1.45 8.01 0.63
Strongly agree 0.27 1.67 0.07 11.22 1.67 17.01 1.67
31.72 19.70 12.86 34.97 8.08 34.01 5.15
Thresholds τ1 = -1.50 χ 2 = 146.48
τ2 = -0.50 p-value =

Anthony Stacey
Survey items
Strongly agree 0.07 0.49 0.11 0.00 0.00 0.04 0.00
0.56 0.81 1.27 0.70 0.70 1.85 0.05
Thresholds τ1 = -1.41 χ 2 = 5.93
τ2 = -0.68 p-value = 1.00
τ3 = -0.36
τ4 = 1.42
Figure 5 illustrates the effect of varying the model parameters to maximise the goodness-of-fit for a
particular survey item.
Figure 5: The effect of varying model parameters to maximise goodness-of-fit
The following points are noted regarding the results of the optimisation algorithm:
The better goodness-of-fit between the model and the observed data after optimisation is patently
evident.
The thresholds are not equidistant. The narrowness of the “Neither agree nor disagree” category
is indicative of participants’ decisiveness in responding to survey items.
The overall mean of zero falls into the “Agree” category. This indicates that there was a general
level of agreement with the survey items across all participants.
The p-value of approximately 1 is interpreted that there is not a significant difference between the
model (of the population) and the sample data. Therefore the population parameters (means and
standard deviations) can be reliably inferred from the simulation model.
What originally manifest as a bimodality of the results is not borne out by the analysis. The
apparent bimodality was merely a result of the relative underuse of the “Neither agree nor
disagree” category by participants.
The consequences of this analysis are as follows:
The survey items can now be analysed for significance, identifying those whose population
means are significantly different from the (overall) mean.
Demographic groups can be analysed separately. The assumption of normality necessary to carry
out t-test and analysis of variance (ANOVA) is automatically satisfied by the model. The
methodology lends itself particularly well to the analysis of small samples because of this feature,
rendering the application of non-parametric statistical tests unnecessary.
Should different demographic groups interpret ordinal scaling formats differently (e.g. due to
translation or cultural inconsistencies) this would be readily apparent from the threshold values.
This could facilitate cross demographic standardisation of survey results.
It has been shown (Stacey, 2005) that estimating population means and standard deviations
using the normal distribution fitting algorithm is more valid and reliable than alternative
approaches.
387

6. Analysis of rank ordered data
Anthony Stacey
The use of survey response formats that involve a sort of items into an order based on a specified
criterion is widely used and results in rank ordered data (Diaconis, 1989). Yu (2000) provides a
thorough summary of the many approaches that have been taken to the analysis of rank ordered
data. These approaches often require that all items are sorted so that each participant manifests a
particular permutation of ordered items.
The cognitive limitations identified by Miller (1956) are possibly the reason why it is onerous on
participants to rank order a large number of survey items. The methodology proposed by Stacey
(2006) has the advantage that it accommodates both full and partial rank ordering of items. Although
not presented as such, it will be shown that methodology proposed by Stacey (2006) for analysing
rank ordered survey items is another example of the using the asymptotic analogy approach.
Consider a scenario in which the mean and variance of preferences for a number of items needs to
be evaluated. The American Psychological Association 1980 election data analysed by Diaconis
(1989) is a well-documented example of such a dataset. Table 4summarises the percentages of
voters ranking each candidate in each rank position, including those who omitted to rank all five
candidates when casting their vote.
Table 4: Observed percentage of voters ranking candidate i in position j
Ranking
Candidate j = 1 j = 2 j = 3 j = 4 j = 5 Omitted
i = 1 19% 18% 12% 6% 6% 39%
i = 2 15% 12% 12% 9% 7% 45%
i = 3 26% 11% 7% 7% 9% 40%
i = 4 21% 11% 9% 8% 9% 42%
i = 5 19% 14% 10% 7% 8% 42%
6.1 Development of the rank ordering simulation model
The cognitive process of ranking candidates based on voters’ preferences for each candidate can
easily be modelled numerically. The preference of a voter for each candidate can be considered to be
a continuous variable and the rank position of each candidate can be computed, for example, using
the RANK function in Microsoft Excel. Conceivably the cognitive process of partial ranking of
candidates is that each voter omits a candidate from their vote if their preference for that candidate
does not exceed a notional preference threshold. This too can easily be modelled using a conditional
IF function.
In modelling the cognitive process of rank order voting, although arbitrary, it is reasonable to set the
initial means and standard deviations of preferences to be equal for all candidates. Similarly, the
modelled notional preference threshold can be initialised to the overall mean preference, on the basis
that voters would not vote for “below average” candidates. Using these initial parameter values in a
Monte Carlo simulation model, Table 5 gives the simulated percentage of voters ranking each
candidate in each ranking position as well as those who did not vote for each candidate.
Table 5: Initial simulated percentage of voters ranking candidate i in position j
Ranking
Candidate j = 1 j = 2 j = 3 j = 4 j = 5 Omitted
Estimated
population
mean
Estimated
population
variance
i = 1 20% 17% 10% 4% 4% 45% 0.0000 1.0000
i = 2 20% 17% 10% 4% 4% 45% 0.0000 1.0000
i = 3 20% 17% 10% 4% 4% 45% 0.0000 1.0000
i = 4 20% 17% 10% 4% 4% 45% 0.0000 1.0000
i = 5 20% 17% 10% 4% 4% 45% 0.0000 1.0000
χ 2 = 7543 Voting threshold 0.0000
388

Anthony Stacey
6.2 Fitting the rank ordering simulation model
Fitting of the rank ordering simulation model to the observed data now involves adjusting the
parameters of the model (estimate population means and variances, and voting preference threshold)
to minimise the difference between the model and the observed data. As in the previous example, the
χ 2 statistic can be calculated to evaluate the goodness-of-fit.
In this instance the Microsoft Excel “Solver” add-in is not appropriate because the error function is
discontinuous. The shotgun stochastic search algorithm (Stacey, 2006) is an algorithm that has
proved effective in minimising the error statistic. Table 5 gives the simulated percentage of voters
ranking each candidate in each rank position after minimising the error statistic.
Table 6: Simulated percentage of voters ranking candidate i in position j
Ranking
Candidate j = 1 j = 2 j = 3 j = 4 j = 5 Omitted
Estimated
population
mean
Estimated
population
variance
i = 1 20% 21% 15% 6% 5% 33% 0.0890 0.8548
i = 2 15% 17% 15% 7% 6% 40% -0.0653 0.8204
i = 3 26% 17% 10% 4% 5% 38% 0.0404 1.2259
i = 4 20% 17% 11% 5% 6% 40% -0.0470 1.0913
i = 5 19% 18% 12% 6% 6% 39% -0.0171 0.9916
χ 2 = 2809 Voting threshold -0.1658
The following points are noted regarding the optimised simulation model:
Candidate 1 has the greatest estimated population mean (i.e. is the most preferred candidate)
although Candidate 3 had the greatest number of 1 st rankings. The least preferred candidate is
Candidate 2.
The variance of preference for Candidate 3 is greater than all other candidates.
Although optimised, the goodness-of-fit statistic indicates a significant difference between the
simulation model and the observed data. This suggests that the cognitive processes have not
been fully captured in the logic of the simulation model.
Other aspects of the cognitive process of rank ordering candidates can be modelled in an
enhanced simulation model (such as correlation between preferences for candidates and different
thresholds for voters willing to rank order only one, two or three candidates) but this is beyond the
scope of this article.
7. Conclusion
This article has presented evidence that shows the reliability and validity of conventional statistical
inference pertaining to ordinal survey data to be inadequate. The reliability is questionable because
the analysis of ordinal data exhibits a discretisation error that has historically not been quantified or
resolved. The validity is questionable because the cognitive process required of survey participants is
not modelled when applying conventional statistical inference.
A new paradigm of analysis of ordinal data is suggested. This involves the development of a
simulation model, using readily available desktop software, which models and becomes an analogy of
the cognitive processes followed by survey participants. It is shown that by using an optimisation
algorithm, the model can be fitted to the observed data. It is then argued that, provided there is no
significant difference between the model and the observed data, the model is indeed a true and
accurate analogy of the research population. As such, the relevant parameters of the population can
be reliably inferred from the model’s input parameters.
This process of inference from a numeric analogy to the population does not involve the calculation of
any sample statistics. It is therefore put forward as a fundamentally different paradigm from
conventional statistical analysis and inference. The easy availability of adequate computer power and
suitable desktop software creates the ideal environment for the universal adoption of this new
paradigm for analysis of survey data.
389

References
Anthony Stacey
Blumberg, B., Cooper, D., & Schindler, P. (2005). Business Research Methods. Maidenhead: McGraw-Hill
Education.
Carifio, J., & Perla, R. (2007). Ten Common Misunderstandings, Misconceptions, Persistent Myths and Urban
Legends about Likert Scales and Likert Response Formats and their Antidotes. Journal of Social Sciences,
3(3), 106-116.
Diaconis, P. (1989). A Generalization of Spectral Analysis with Application to Ranked Data. The Annals of
Statistics, 17(3), 949-979.
Everitt, B. S. (2002). The Cambridge Dictionary or Statistics (2nd ed.). Cambridge: Cambridge University Press.
Harwell, M., & Gatti, G. (2001). Rescaling Ordinal Data to Interval Data in Educational Research. Review of
Educational Research, 71(1), 105-131.
Kaptein, M., Nass, C., & Markopoulos, P. (2010). Powerful and Consistent Analysis of Likert-Type Rating Scales.
Paper presented at the 28th International Conference on Human Factors in Computing Systems, Atlanta,
GA.
Labovitz, S. (1970). The Assignment of Numbers to Rank Order Categories. The American Sociological Review,
35(3), 515-524.
Likert, R. (1932). A Technique for the Measurement of Attitudes. Archives of Psychology, 140, 5-52.
Mayer, L. S. (1971). A Note on Treating Ordinal Data as Interval Data. American Sociological Review, 36(3), 519-
520.
Miller, G. A. (1956). The magical number seven, plus or minus two: some limits on our capacity for processing
information. Psychological Review, 63(2), 81-97. doi: 10.1037/h0043158
Stacey, A. G. (2005). Reliability and Validity of the Item Means and Standard Deviations of Ordinal Level
Response Data. Management Dynamics, 14(3), 2-25.
Stacey, A. G. (2006). Estimating the Means and Standard Deviations of Rank Ordered Survey Items.
Management Dynamics, 15(3), 26-35.
Svensson, E. (2000). Comparison of the Quality of Assessments Using Continous and Discrete Ordinal Rating
Scales. Biometrical Journal, 42(4), 417-434.
Upton, G., & Cook, I. (2006). A Dictionary of Statistics. Oxford: Oxford University Press.
Yu, P. L. H. (2000). Bayesian analysis of order-statistics models for ranking data. Psychometrika, 65(3), 281-299.
390

Research Ethics for Sustainable Development
Anthony Stacey 1 and Julie Stacey 2
1
Graduate School of Business Administration, University of the Witwatersrand,
Johannesburg, South Africa
2
Centre for Sustainability in Mining and Industry, University of the
Witwatersrand, Johannesburg, South Africa
Anthony.Stacey@wits.ac.za
Julie.Stacey@wits.ac.za
Abstract: The challenges and crises that face organisations are frequently the result of unintended, unanticipated
or unforeseen consequences of well-intended decisions. In this paper the role of research ethics is analysed in as
far as it contributes to or militates against future decision-outcomes that compromise the development of
sustainable societies. The evolution of research ethics in the social and economic sciences will be illustrated by
citing historical examples of research that would be regarded as unethical by today’s standards. This will be
paralleled with the evolution of our understanding of the requirements for sustainable development. Some of the
major financial, social and environmental crises facing the world will be analysed. It will be demonstrated that
research and innovation played a seminal role in these developments, and that the current research ethics
paradigm would have been inadequate to prevent or address some of the unintended consequences of the
original research. The central thesis of this paper is that researchers are responsible and accountable for
applying their minds to the possible broader, unintended and unforeseen impacts of their research. Therefore a
more comprehensive view needs to be taken of the ethics of business and management research to include
procedural ethics, intrinsic ethics, and extrinsic ethics. The new paradigm also requires that the hierarchical
interdependencies of the five capitals of sustainable development be considered in research ethics. Natural
capital, human capital, social capital, manufactured capital, and financial capital are the five types of capital from
which we derive goods and services that we need to survive and/or to improve the quality of our lives. The
interactions and flow of stock between these capitals gives rise to twelve features of a sustainable society. It is
advocated that research ethics and protocols must change to take into account these twelve features. Ethical and
responsible researchers will consider not only the research process and the protection of human participants, but
also the application of their research in these broader features of sustainable societies. It may be argued that
individuals cannot be held responsible or accountable for that which is unintended. However, by applying their
minds using a more powerful ethical framework, researchers are more likely to anticipate and militate against
these potentially latent and often long term impacts of their research.
Keywords: extrinsic research ethics, five capitals; sustainable society; business decisions
1. Introduction
The concept of sustainable development (Brundtland, 1987) came into being because of the
enormous environmental, social and economic challenges manifesting in contemporary life. Due to its
very nature, sustainable development has pervaded all aspects of society – including business – in
recent decades. Managers of organisations have been challenged to apply their minds, modify
business practices and potentially amend their worldview based on this dynamic and maturing
understanding of sustainable development. It therefore behoves all academics and professionals in
business and management to consider their activities and the ethics of their research in the context of
this paradigm.
There are many theoretical perspectives from which ethics can be viewed, including philosophical,
legal, historical, moral, cultural, and human rights. However, research ethics will be addressed from a
broad practical and applied perspective in this paper. It will be shown using historical examples that
research ethics are dynamic and evolutionary. It can therefore be deduced that the ethical standards
against which contemporary research is being judged are unlikely to be the same as those which will
be applied by future researchers, who will apply profoundly different social, professional and
academic values. Further, the insufficiency of contemporary research ethics will be demonstrated by
showing that unanticipated negative consequences of particular historical research would not have
been mitigated by the application of contemporary research ethics.
The paper concludes by suggesting that an ethical protocol based on the twelve features of a
sustainable society will narrow the gap between current research ethics protocols and the standards
by which contemporary research will likely be judged by future generations.
391

2. Progression of research ethics
Anthony Stacey and Julie Stacey
A number of historical research projects have achieved notoriety on account of their methodologies or
unanticipated consequences. Some of these have been well-publicised, and have been dubbed “Mad
Science” by Schneider (2009).
The experiments involving “Little Albert” (Watson & Rayner, 1920) are controversial for a variety of
reasons. In these experiments, the reactions of a nine month old boy to a variety of animate and
inanimate objects were observed. Thereafter, the researchers conditioned the infant to fear a white rat
by associating the rat with a loud clanging noise, resulting in the boy crying at the sight of the white rat
(even when the noise was absent). The manner in which the experiments were carried out and the
analysis reported have been the subject of criticism by subsequent authors (Harris, 1979; Paul &
Blumenthal, 1989). Deliberately invoking fear in an infant in a laboratory setting, with or without
parental consent, would not be contemplated by contemporary social scientists.
The Tuskegee Syphilis Study by the US Public Health Service which began in the early 1930’s
involved withholding of medical treatment from a racially distinct group of syphilis infected individuals
(Thomas & Quinn, 1991). Such was the affront to the dignity of not only the participants, but also an
entire demographic group, that a presidential apology was offered by Bill Clinton on behalf of the US
government in 1997 (Harter, Stephens, & Japp, 2000). The study involved taking advantage of
vulnerable individuals and is highly unethical in a number of respects (Experiment-Resources.com,
2008).
The so-called “Milgram experiment” (Milgram, 1963) purported to be an experiment to determine
whether the administering of increasingly painful electric shocks by a volunteer research assistance
(the “teacher”) would improve the memory of the volunteer research subject (the “learner”). In fact, it
was the “teacher” that was the actual subject of the research, and the “learner” was receiving no
electric shock whatsoever and was merely an actor in collusion with the researchers. The purpose of
the research was to evaluate the subjects’ compliance with commands given to them by an authority
figure. The findings were that, in response to authority, about two thirds of the “teachers” administered
the maximum possible electric shock to the “learners” despite the extreme pain that the “learner”
appeared to be in. The problem was that the experiment was extremely stressful and traumatising for
the “teachers” and it is generally agreed that such an experiment would never receive ethical approval
today (McArthur, 2009).
Having reviewed some examples of historical research in the social sciences, it is evident that, while
these experiments may have been regarded as ethical at the time of the research, their
methodologies would have been rejected when judged by contemporary ethical standards. Research
ethics clearly evolve in order to remain consistent with societal values. Similarly, businesses have
“contracts” with society that evolve from frontier expectations through semi-formal contracts, to
formalised contracts such as legislation (Bonini, Mendonca, & Oppenheim, 2006). Therefore,
contemporary professional and academic researchers in business and management studies should
ask the question: Against what frontier ethical expectations could current business and management
research be judged by future generations?
3. Insufficiency of contemporary research ethics
The emergence of contemporary research ethics has not been a simple linear process (Isreal & Hay,
2006). Nevertheless, many of the generally accepted principles of research involving human
participants can be traced back to The Nuremberg Code (1949). Although this code, together with the
Helsinki Declaration (World Medical Association, 1964) are directed towards the ethics of medical
research involving human participants and material, they are the seminal works of contemporary
research ethics in the social sciences, including business and management.
The following principles are common to contemporary research ethics protocols and standards:
Informed consent, which implies the avoidance of covert or secret participant observation
(Bulmer, 1982).
Privacy of participants (confidentiality and anonymity).
Avoiding harm (including psychological effect) and doing good.
392

Cognisance of vulnerable groups.
Participants’ right to withdraw or terminate.
Restricted use of data.
Due care in the storage of data.
Avoidance of conflicts of interest.
Anthony Stacey and Julie Stacey
Reviewing some of social science’s most notorious research, it may be argued that the unanticipated
consequences of the research could have been avoided if the research had met contemporary ethical
standards. However, conforming to such a protocol would not necessarily have been a guarantee that
unanticipated consequences would not manifest.
In their research on the causes of AIDS, Duesberg and Rasnick (1998) argued that AIDS is
attributable to drug use, not HIV. They conclude that “the AIDS dilemma could be solved by banning
anti-HIV drugs, and by pointing out that drugs cause AIDS.” What has become known as the
Duesberg hypothesis had the support of a community of so-called AIDS-denialists, who were
influential in the formulation of South Africa’s post-apartheid HIV / AIDS policy. It is estimated that the
loss of more than 300 000 lives could have been averted were it not for this policy (Chigwedere,
Seage, Gruskin, Lee, & Essex, 2008; Nattrass, 2008). Despite the weight of research that has
subsequently criticised and refuted their hypothesis, Duesberg and Rasnick did not appear to have
breached contemporary ethical standards in the course of their research.
This narrow ethical focus is not limited to the social sciences. In the business world, prior to the global
financial crisis that occurred late in 2007; banks in the United States had adopted the originate-todistribute
model of granting credit, specifically mortgage loans. In terms of this model of banking,
banks (the originators) lent money to borrowers, and then aggregated the debt and sold it on to third
party lenders (Purnanandam, 2010). There are benefits of this model of banking to the borrowers, the
banks and the lenders such as enhancing the liquidity of the secondary syndicated loan market
(Berndt & Gupta, 2009). However, Brunnermeier (2009) shows that the financial crisis was partially
attributable to the originate-to-distribute banking model. Although there was nothing unethical from a
research perspective, the liquidity crunch and financial crisis were an unintended consequence of this
financial innovation (Brunnermeier, 2009). As another commentator observed “… what [the financial
industry] did was to innovate itself, and the rest of us, into a big, nasty mess” (Krugman, 2007).
It is clear from the summary by Aksin, Armony, and Mehrotra (2007) that there has been extensive
research into optimisation of capacity planning, queuing, and personnel scheduling for call centres.
Optimisation strategies have undoubtedly been adopted that had negative impacts on the customer
service representatives. At face value, the Sacrificial Human Resources Strategy (Wallace, Eagleson,
& Waldersee, 2000) could be considered callous in its disregard for the impact on individuals and
society. A parallel field of research exists that looks into the human resource issues in call centres
(Aksin, et al., 2007) in response to the impact that has not been studied in the traditional call centre
research.
Researchers may also face the possibility of their findings being used or applied in ways which they
had not intended. Mastrandrea and Schneider (2004) identified this possibility in the highly
controversial and politicised field of climate change when they observed that “… users of scientific
results (including policy-makers) will undoubtedly make their own assumptions about the probability of
different outcomes, possibly in ways that the original authors did not intend” (Mastrandrea &
Schneider, 2004, p. 571). Contemporary research ethics protocols are silent on any obligation on
researchers to consider the possibility of (mis)application, (mis)interpretation or (ab)use of their
research by others.
It is clear that, no matter how rigorously and meticulously researchers adhere to contemporary
research ethics protocols, these protocols do not necessarily militate against negative consequences.
Therefore, it is argued that current research protocols are insufficient to adequately protect society at
large from the totality of consequences of research, or to contribute with some degree of certainty
towards sustainable societies. It is neither feasible nor logical to expect that all potential risks can be
eliminated (i.e. reduced to a probability or consequence of zero); there remains nonetheless a gap
between contemporary research ethics and the protocols needed to promote and support sustainable
societies.
393

Anthony Stacey and Julie Stacey
4. Unanticipated consequences and unsustainability
The Five Capitals Framework of sustainable development (Porritt, 2007) has been developed as a
basis of understanding the entirety of sustainability using the concept of wealth or capital as it is
understood in economics, and is encapsulated in Table 1. This model is a constructive framework for
analysing the diverse consequences of research activities that to date have not been explicitly
addressed, mitigated or avoided by researchers.
Table 1: The five capitals framework (Porritt, 2007, p. 139)
Capital Definition
Natural Any stock or flow of energy and matter that yields valuable goods and services. It falls
into several categories: resources, some of which are renewable (timber, grain, fish and
water), while others are not (fossil fuels); sinks which absorb, neutralize or recycle
waste; and services, such as climate regulation. Natural capital is the basis not only of
production but of life itself.
Human Health, knowledge, skills and motivation (all of which are required for productive work),
as well as individual’s emotional and spiritual capacities, Enhancing human capital (for
instance through investment in education and training) is central to a flourishing
economy.
Social Structured, institutions, networks and relationships which enable individuals to maintain
and develop their human capital in partnership with others, and to be more productive
when working together than is isolation. It includes families, communities, businesses,
trade unions, voluntary organisations, legal / political systems and educational and
health bodies.
Manufactured Material goods – tools, machines, buildings and other forms of infrastructure – which
contribute to the production process but do not become embodied in output.
Financial Plays an important role in our economy by reflecting the productive power of the other
types of capital, and enabling them to be owned and traded. However, unlike the other
types of capital, it has no intrinsic value; whether in shares, bonds or banknotes, its
value is purely representative of natural, human, social or manufactured capital.
The unit of analysis in the AIDS research by Duesberg and Rasnick (1998) was the health and
behaviour of the individual, which falls within the definition of human capital. However, a substantial
consequence of the research was in the political and social spheres of social capital. The
implementation of the originate-to-distribute banking model was an innovation within the domain of
financial capital. The negative impact of the consequent financial crisis was a devaluation of not only
financial capital but also human and social capital. The financial crisis potentially affected natural
capital in as far as the resultant organisational budget cuts often affect the ‘non-core’ management
issues first (as environmental issues are often perceived in business). Typically financial models have
been developed for the management of personnel resourcing and scheduling at call centres, omitting
the profoundly negative human and social consequences. If the most important research question to
study is the longitudinal relationship between the aforementioned sacrificial call centre strategy and
market performance, as suggested by Wallace, et al. (2000), then human and social capital is
discounted. It is well understood that technological developments, even as far back in history as the
industrial revolution in the mid-1700’s, resulted in extraordinary growth in manufactured and financial
capital. Yet, the profound impact on the natural environment and the negative effects on society
represent extensive destruction of natural, social and human capital at rates that exceed their ability
to recover. The nuclear age ushered in similar developments in manufactured capital, but the military
applications of the technology, the production of extremely high risk waste and worker exposure to
radiation doses has come at a high price in natural, social and human capital – “nuclear power is a
Faustian bargain” (Williams & Cantelon, 1984, p. xi). These examples demonstrate that the negative
consequences of research activities can be the antithesis of sustainable development.
This analysis suggests that although typically the enhancement of capital as a result of research or
innovation occurs as intended within one of the five capitals, this may sometimes be achieved at the
cost of destruction of value in one or more of the remaining capitals. Schienke et al. (2009) proposed
that responsible conduct of scientific research recognises three distinct components of research
ethics: procedural ethics, intrinsic ethics and extrinsic ethics. As per their definition, extrinsic research
ethics are:
“Ethical issues extrinsic to the production of scientific research – that is, ethical issues in
how the outcomes of science research impact society, such as: policy-making; lawsuits;
changes in social norms; and education and entertainment.” (Schienke, et al., 2009, p.
322)
394

Anthony Stacey and Julie Stacey
This triad of ethical domains has been designated the Ethical Dimensions of Scientific Research
(Tuana, 2010). By introducing the concept of extrinsic ethics, Schienke et al. (2009) and Tuana (2010)
are effectively proposing that researchers should integrate and apply their minds to the capitals other
than that in which the research is being carried out, especially regarding the research outcomes. The
collaborative research involving Indigenous communities in Canada (Ball & Janyst, 2007) is an
example of the incorporation of social capital into the ethics of research. The principles of
collaboration and consultation are now embodied in the document which prescribes ethical conduct
for research involving humans in Canada (Canadian Institutes of Health Research, Natural Sciences
and Engineering Research Council of Canada, & Social Sciences and Humanities Research Council
of Canada, 2010).
5. A taxonomy and hierarchy of negative consequences of research
The central tenet of this paper is that researchers are responsible for considering all the outcomes
and consequences of their research, whether or not those consequences are favourable and whether
or not they are within the immediate objectives of the research. It can be argued that there are certain
unforeseeable consequences of research for which researchers cannot reasonably be expected to
accept responsibility. Conversely it is argued that researchers cannot merely abdicate responsibility
by expedient definitions of scope. Therefore a taxonomy of negative consequences of research is
helpful in articulating these consequences, allied to the ethical obligations of researchers.
Table 2 gives a proposed taxonomy of negative consequences of research which are considered to
be out of the research scope; the categories of consequences are ordered in descending order of
probability. The nondisclosure or omission from publication of negative, non-target effects or side
effects that are within the research scope (Holdrege, 2008) would be regarded as unethical in terms
of current standards; this situation is therefore not addressed in Table 2. It is proposed that
researchers’ ethical obligations vary according to the risk associated with the outcome, which is a
function of the consequence and probability of the event. In terms of this taxonomy, researchers have
some obligations for even the most problematic negative consequences – those that cannot be
foreseen.
Table 2: A proposed taxonomy and hierarchy of negative consequences of research
Categorisation Definition Probability Researchers’ obligations
Unaddressed Consequences are known to
the researchers but are not
examined because of the
predefined scope.
Unintended Consequences are known to
the researchers, but with low
enough probabilities that
associated risks can be
ignored.
Unexpected Consequences are known to
the researchers, but with no
anticipated risk because of
zero probability of occurrence.
Unforeseen Consequences are unknown to
the researchers, but could
reasonably have been
foreseen if researchers had
applied their minds.
Unforeseeable Consequences could not have
been reasonably identified by
researchers.
High Public disclosure and engagement
with stakeholders to achieve
consensus on research scope and
objectives.
Low Include in the scope of the research
measures to avoid negative
consequences, despite the very low
probability.
Zero Disclosure and consultation within
the professional and academic
community to corroborate zero
probability of occurrence.
Unknown Apply their minds to possible
negative consequences, and their
associated probabilities and impacts.
Undefined Remain open-minded to possible
negative consequences, and seek
input on these, no matter which
stakeholder(s) recognise(s) the risks.
Research is often mired in secrecy, and protected by copyright or patent laws, for reasons of
economic competitiveness and gain. The systemic nature and scale of sustainable development
challenges requires a worldview shift from protectionist to collaborative: hence the requirement for
broader consultation. That this might require new research methodologies is acknowledged.
Simultaneously, that “All people have the right to participate in and access information relating to the
decision-making processes that affect their lives and well-being” is equally well entrenched in social
and political doctrine through international law (United Nations Population Fund, 2012). This broader
consultation could even result in unforeseeable consequences being serendipitously identified.
395

Anthony Stacey and Julie Stacey
A similar taxonomy can be created for positive outcomes; ideally researchers would apply their minds
to maximising positive outcomes towards sustainable futures. The obligations proposed in Table 2 are
aligned with the typical hierarchy of risk treatment options as defined in ISO 31000:2009 (Purdy,
2010) which also apply equally to positive and negative outcomes. The preferential option is always to
eliminate or avoid negative outcomes. By virtue of their activities, researchers are ideally positioned to
identify and remove potential threats, and this should be their priority. It is worth noting here that the
rigour of the research process necessitates proper observation and evaluation of activities. It has be
argued (Hansson, 2011) that ethically problematic or risky activities should therefore only be carried
out within a well-defined research context. However, a statement of responsibilities of researchers is
worthless without a corresponding mechanism of accountability. The research ethics protocol is the
appropriate means of holding researchers to account for applying their minds to all outcomes of their
research.
6. A paradigm and protocol of research ethics for sustainable development
Tuana noted that “My point here is that this domain of Ethical Dimensions of Scientific Research is
ripe for research informed by philosophy of science perspectives” (2010, p. 481). It is being advocated
in this paper that the ethics of business and management research is ripe for transformation into the
context of sustainable development, due to the interdependent and constantly interacting nature of
the capitals on which organisational productivity depend. Organisations are after all the mechanism
through which people transform natural and human capital into social, manufactured and economic
wellbeing. The twelve features of a sustainable society (Forum for the Future, 2011) is a useful
framework within which consideration of potentially negative research outcomes can be applied by
researchers. It is noted that these statements are comprehensive, internally consistent and culturally
neutral. A protocol corresponding to this framework should govern and guide research and
professional innovation activities. Table 3 gives a proposed research ethics protocol using the five
capital framework based on the twelve features of sustainable societies. The proposed protocol is
consistent with sustainable development principles of fair access to resources across generations
(intergenerational equity), acknowledgement of the finite limits of the capitals, and the tension
between needs and wants.
Table 3: Proposed research ethics protocol based on the twelve features of sustainable societies
Construct Protocol
Natural capital Natural capital must be shared fairly between current and future
generations – intergenerational equity. Natural capital processes and
resources have finite capacities, and protracted timeframes may be
Harmful Effects of Extraction
of Substances from the Earth.
Harmful Effects of Artificial
Substances.
required to realise these capacities.
Describe how the research and its outcomes may influence the
environment's capacity to disperse, absorb, recycle or otherwise
neutralise the harmful effects (to humans and / or the environment) of
the extraction and use of substances taken from the earth.
Describe how the research and its outcomes may influence the
environment's capacity to disperse, absorb, recycle or otherwise
neutralise the harmful effects (to humans and / or the environment) of
the manufacture and use of artificial substances.
Ecological System Integrity Describe how the research and its outcomes may influence the capacity
and Biological Diversity of the environment to provide ecological system integrity, biological
diversity and productivity.
Human capital The needs of the many must enjoy priority over the wants of the few, for
both current and future generations. The limited capacity of specific
disadvantaged groups to engage must be addressed.
Standard of Health. Describe how the research and its outcomes may influence the standard
Relationships, Social
Participation, Development
and Learning.
Work, Personal Creativity, and
Recreation.
of health among individuals of all ages.
Describe how the research and its outcomes may influence individuals’
relationships and social participation, setting and achievement of
personal standards of their development and learning throughout their
life.
Describe how the research and its outcomes may influence individuals’
access to varied and satisfying opportunities for work, personal
creativity, and recreation.
Social capital Sustainable social capital requires at a minimum, the same
preconditions as sustainable natural and human capital.
Governance and Justice. Describe how the research and its outcomes may influence trust in and
access to systems of governance and justice.
396

Anthony Stacey and Julie Stacey
Construct Protocol
Positive Values and Sense of
Purpose.
Stewardship of Natural
Resources and Development
of People.
Describe how the research and its outcomes may influence shared
positive values and sense of purpose among communities and society at
large.
Describe how the research and its outcomes may influence the
structures and institutions of society to promote stewardship of natural
resources and development of people.
Safe, Supportive Living and Describe how the research and its outcomes may influence the capacity
Working Environments. for homes, communities or society at large to provide safe, supportive
living and working environments.
Manufactured capital Limits of current technology and human knowledge must be
acknowledged. Consideration must be given to needs versus wants, and
the natural, human and social capital expense of creating new
technologies, products and services.
Use of Natural Resources, and Describe how the research and its outcomes may influence
Human Innovation and Skills. infrastructure, technologies and processes to minimise the use of natural
resources and maximise the use of human innovation and skills.
Financial capital The financial cost of human, social, natural and manufactured impacts
for current and future generations that are external to the organisation
Accurate Representation of
the Value of Capital.
must be acknowledged and disclosed.
Describe how the research and its outcomes may influence the accuracy
of the valuation of natural, human, social and manufactured capital in
terms of financial capital.
Mindful that the imposition of a bureaucracy of ethical procedures may be perceived to hamper
academic research activity (Haggerty, 2004) the effect of the proposed protocol may rather be to
increase the scope of such activities. Without the broader considerations embodied in the proposed
protocol, the contribution of research to creating sustainable societies will be coincidental rather than
by design.
7. Conclusion
Sustainable development as a concept provides a lens for both upside opportunity and downside risk
research: the focus of this paper is the downside, or negative risk, given the scale of global
challenges in each of the capitals. It is not possible to foresee every possible outcome of research.
However, the current research ethic paradigm and associated protocols tend to encourage a narrow
focus based on the rights of human participants, bounded exclusively by the scope of the research
problem and methodology. The interconnected nature of the five capitals that support life and the
systems conditions established through these relationships requires that researchers apply their
minds to the broader features of sustainable societies when considering, at an ethical level, the
possible outcomes of research. At the same time, the boundary conditions of limits, needs versus
wants, and intergenerational equity must inform the ethical considerations of the research. The
proposed protocol is the step change that is required to shift the current narrow research ethic
paradigm to one that addresses the conditions for sustainable development. In order to move towards
research that contributes to sustainable development, business and management researchers and
professionals must apply a thorough understanding of systems thinking to their research in order to
anticipate the full suite of its possible outcomes. No single researcher can be expected to have a
complete understanding of all aspects of sustainable development; the proposed consultation,
dependent on the significance of the possible outcomes, addresses this limitation. Societal
heterogeneity (in many forms) further dictates that researchers engage publically on potential
outcomes to better gauge the current status of social expectations. The use of the proposed protocol
will not necessarily change the outcomes of the research; the broader considerations advocated in
this paper should encourage decision-making that is more likely to contribute to sustainable societies.
References
Aksin, Z., Armony, M., & Mehrotra, V. (2007). The Modern Call Center: A Multi-Disciplinary Perspective on
Operations Management Research. Production and Operations Management, 16(6), 665-688.
Ball, J., & Janyst, P. (2007). Enacting Research Ethics in Partnerships with Indigenous Communities in Canada:
“Do It in a Good Way”. Journal of Empirical Research on Human Research Ethics, 3(2), 33-51.
Berndt, A., & Gupta, A. (2009). Moral hazard and adverse selection in the originate-to-distribute model of bank
credit. Journal of Monetary Economics, 56(5), 725-743.
Bonini, S. M. J., Mendonca, L. T., & Oppenheim, J. M. (2006). When social issues become strategic. The
McKinsey Quarterly, 2006(2), 20-31.
397

Anthony Stacey and Julie Stacey
Brundtland, G. H. (1987). Our Common Future. Geneva: United Nations World Commission on Environment and
Development.
Brunnermeier, M. K. (2009). Deciphering the Liquidity and Credit Crunch 2007–2008. Journal of Economic
Perspectives, 23(1), 77-100.
Bulmer, M. (1982). Social Research Ethics: an examination of the merits of covert participant observation.
London: Macmillan Press Ltd.
Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, &
Social Sciences and Humanities Research Council of Canada. (2010). Tri-Council Policy Statement: Ethical
Conduct for Research Involving Humans. Retrieved from
http://www.pre.ethics.gc.ca/pdf/eng/tcps2/TCPS_2_FINAL_Web.pdf
Chigwedere, P., Seage, G. R., Gruskin, S., Lee, T.-H., & Essex, M. (2008). Estimating the Lost Benefits of
Antiretroviral Drug Use in South Africa. Journal of Acquired Immune Deficiency Syndromes, 49(4), 410-415.
Duesberg, P., & Rasnick, D. (1998). The AIDS Dilemma: drug diseases blamed on a passenger virus. Genetica,
104, 85-132.
Experiment-Resources.com. (2008). Tuskegee Syphilis Study (1932-72) Retrieved 2012-02-08, from
http://www.experiment-resources.com/tuskegee-syphilis-study.html
Forum for the Future. (2011). The Five Capitals, from http://www.forumforthefuture.org/project/fivecapitals/overview
Haggerty, K. D. (2004). Ethics Creep: Governing Social Science Research in the Name of Ethics. Qualitative
Sociology, 27(4), 391-413.
Hansson, S. O. (2011). Do we Need a Special Ethics for Research? Science and Engineering Ethics, 17, 21-29.
Harris, B. (1979). Whatever Happened to Little Albert? American Psychologist, 34(2), 151-160.
Harter, L. M., Stephens, R. J., & Japp, P. M. (2000). President Clinton's Apology for the Tuskegee Syphilis
Experiment: A Narrative of Remembrance, Redefinition, and Reconciliation. Howard Journal of
Communications 11(1), 19-34.
Holdrege, C. (2008). Understanding the Unintended Effects of Genetic Manipulation Retrieved 2012-02-13, from
http://www.natureinstitute.org/txt/ch/nontarget.php
Isreal, M., & Hay, I. (2006). Research Ethics for Social Scientists. London: SAGE Publications Ltd.
Krugman, P. (2007, 2007-12-03). Innovating Our Way to Financial Crisis. Retrieved from
http://www.nytimes.com/2007/12/03/opinion/03krugman.html
Mastrandrea, M. D., & Schneider, S. H. (2004). Probabilistic Integrated Assessment of "Dangerous" Climate
Change. Science, 304, 571-575.
McArthur, D. (2009). Good Ethics Can Sometimes Mean Better Science: Research Ethics and the Milgram
Experiments. Science and Engineering Ethics, 15(1), 69-75.
Milgram, S. (1963). Behavioral Study of Obedience. The Journal of Abnormal and Social Psychology, 67(4), 371-
378.
Nattrass, N. (2008). Aids and the Scientific Governance of Medicine in Post-Apartheid South Africa. African
Affairs, 107(427), 157-176.
Paul, D. B., & Blumenthal, A. L. (1989). On the trail of Little Albert. The Psychological Record, 39(4), 547-553.
Porritt, J. (2007). Capitalism as if the world matters. London: Earthscan.
Purdy, G. (2010). ISO 31000:2009—Setting a New Standard for Risk Management. Risk Analysis, 30(6), 881-
886.
Purnanandam, A. (2010). Originate-to-distribute Model and the Subprime Mortgage Crisis. The Review of
Financial Studies, 24(6), 1881-1915.
Schienke, E. W., Tuana, N., Brown, D. A., Kenneth J. Davis, Klaus Keller, James S. Shortle, . . . Baum, S. D.
(2009). The Role of the National Science Foundation Broader Impacts Criterion in Enhancing Research
Ethics Pedagogy. Social Epistemology: A Journal of Knowledge, Culture and Policy, 23(3/4), 317-336.
Schneider, R. U. (2009). Mad Science: 100 Amazing Experiments from the History of Science. London: Quercus.
The Nuremberg Code. (1949). Trials of War Criminals before the Nuremberg Military Tribunals under Control
Council Law No. 10 (Vol. 2, pp. 181-182). Washington, DC: US Government Printing Office.
Thomas, S. B., & Quinn, S. C. (1991). The Tuskegee Syphilis Study, 1932 to 1972: Implications for HIV
Education and AIDS Risk Education Programs in the Black Community. American Journal of Public Health,
81(1), 1498-1505.
Tuana, N. (2010). Leading with ethics, aiming for policy: new opportunities for philosophy of science. Synthese,
177(3), 471-492.
United Nations Population Fund. (2012). Human Rights Principles. Retrieved 2012-02-13, from
http://www.unfpa.org/rights/principles.htm
Wallace, C. M., Eagleson, G., & Waldersee, R. (2000). The Sacrificial HR Strategy in Call Centers. International
Journal of Service Industry Management, 11(2), 174-185.
Watson, J. B., & Rayner, R. (1920). Conditional emotional reactions. Journal of Experimental Psychology, 3, 1-
14.
Williams, R. C., & Cantelon, P. L. (1984). The American atom: a documentary history of nuclear policies from the
discovery of fission to the present, 1939-1984. Philadelphia: University of Pennsylvania Press.
World Medical Association. (1964). Ethical Principles for Medical Research Involving Human Subjects. Helsinki.
398

Eating our own Cooking: Toward a Design Science of
Research Methods
John Venable 1 and Richard Baskerville 2
1
School of Information Systems, Curtin University, Perth, Western Australia,
Australia
2
Department of Computer Information Systems, Georgia State University,
Atlanta, Georgia, USA
j.venable@curtin.edu.au
baskerville@acm.org
Abstract: This paper argues that Design Science is an appropriate paradigm for research into Research
Methods. Research Methods are designed and created by people to achieve a specific purpose –creating new
knowledge. Research methods vary in utility depending on how well they their purpose in the particular contexts
and contingencies for which they were developed. Applying a DSR perspective to research methods should yield
increased utility in the application of research methods, better guidance in applying them and greater confidence
in achieving the desired outcomes of applying them. Based on these premises, this paper analyses the logical
consequences of taking a Design Science perspective on research methods. First, it analyses the various
purposes of research methods to clarify the alternative and competing design goals of research methods.
Second, it analyses and characterises the types of design artefacts that comprise research methods. Third, it
considers issues of the evaluation of research methods. Fourth and finally, it considered the development of
design theories of research methods.
Keywords: research method, research design, design science research, evaluation, design theory, research
rigour
1. Introduction
There is a long history of discussion, debate, and publication regarding research methods (e.g. Nagel
1961; Campbell and Stanley 1963; Bunge 1967; Glaser and Strauss 1967; Miles and Huberman
1994). The ideas and methods used in research have evolved gradually, especially as the research
domains and purposes have evolved. The discipline of Information Systems (IS) is a comparatively
recent participant in this debate. However, methodological challenges inherent in the IS domain have
motivated progress toward research methods that better and more fully address the IS domain (e.g.
Mumford et al. 1985; Galliers and Land 1987; Benbasat 1991; Mason 1991).
The development and evolution of research methods has often been an ad hoc and non-rigorous
process, because scientific communities are social phenomena with their own social institutions and
mores (Berger and Luckmann 1966; Kuhn 1970; Latour 1987). New research methods and
guidelines are posited and developed, written up and communicated to the research community,
generally as research essays. But key aspects of rigour in their development may be lacking. Only
light consideration may be given to their intended goals and purposes; measures of effectiveness,
efficiency, or efficacy; evidence that the research method meets such goals or qualities; and
comparisons that provide evidence that the method better achieves its goals than existing methods.
This paper proposes that research methods would be better developed using rigorous design science
research (DSR) approaches. A DSR approach to the design and development of new or improved
research methods would provide a more rational and consistent basis for assuring both their utility
and rigour. The rigour from using a DSR approach would come primarily from (a) more precisely
stated design theories about research methods and (b) more rigorous evaluation of research
methods. Explicating such elements may even help make the peer-reviewing of the resulting research
more predictable and less capricious. A seminal paper on DSR by March and Smith (1995) provides
support for the use of Design Science Research for the development of research methodology.
“Natural science uses but does not produce methods. Design science creates the
methodological tools that natural scientists use. Research methodologies prescribe
appropriate ways to gather and analyze evidence to support (or refute) a posited theory.
They are human-created artifacts that have value insofar as they address this task.” (p.
258)
399

John Venable and Richard Baskerville
This paper is organised as follows. The next section reviews the relevant literature on DSR. Section
three applies a DSR lens to the goals and objectives of research methods, research methods as
designed artefacts, requirements for evaluation of research methods, and design theories in research
methods. Section four summarises the paper and identifies areas for further research.
2. Key aspects of design science research for research methodology
This section considers relevant literature from the field of Information Systems (IS) on Design Science
Research (DSR). Interestingly, among the work on DSR in IS, no one has attempted to define it. We
will define DSR as “Research in which a new solution technology is invented”. By solution technology,
we mean any kind of human designed artefact, whether product or process, tool, methodology,
technique or any other means for achieving some purpose. By invention we mean creation, design,
improvement, adaptation, not just ‘pure’ invention from scratch (which may not exist anyway).
In this section, we consider in turn research paradigms and goals, design artefacts, design theories,
and evaluation in DSR.
2.1 Research paradigms and goals
Orlikowski and Baroudi (1991) asserted that the then dominant paradigm of IS research was one
based on a positivist philosophy. They then argued that other philosophical positions were relevant,
including interpretive and critical philosophical positions. The interpretive position contrasted with the
positivist position fits the subjective-objective dimension in Burrell and Morgan’s (1979) paradigmatic
framework. The critical philosophical position roughly fits the radical change end of the regulationradical
change dimension of their paradigmatic framework. A fourth research paradigm is that of the
research essay, in which one makes an argument based on logic and reasoning. There is a long
tradition of this in philosophy. Galliers (1991) identified this as a research approach, but did not
distinguish it as a paradigm. He called it subjective/argumentative, but we do not see that it is
necessarily subjective. However, it may be subject to faulty reasoning. The different research
paradigm has implicitly been recognised by the Journal of the Association for Information System
editorial comment by Hirschheim (2008) giving guidelines for “conceptual papers”. Design Science
can be considered as a fifth research paradigm that is fundamentally different from other paradigms. It
seeks to create new ways and means to solve problems and make improvements, through the
invention, design, and development of new solution technologies. We therefore classify research into
five different paradigms: Positivist, Interpretive, Critical, Design Science, and Logico-Argumentative.
2.2 Design artefacts
March and Smith (1995) established the terminology of Design Science in IS, relating it to Herbert
Simon’s ideas of sciences of the artificial (Simon 1969; 1996) and proposed design science as a
research approach. They identified four kinds of design artefacts as the outputs of DSR: constructs,
models, methods, and instantiations. Constructs are concepts that are used to describe a problem
and specify its solution. They define models as “a set of propositions or statements expressing
relationships among constructs” (p. 256) that can be used to “represent situations as problem and
solution statements” (p. 256). They define methods as “a set of steps (an algorithm or guideline) used
to perform a task” (p. 257). Finally, they define instantiations as “the realization of an artifact in its
environment” (p. 258). Interestingly, the above artefacts do not include “design” per se. However, the
model artefact could be viewed as a design, in that it includes “solution statements” (p. 256). A design
can then be conceived of as different constructs (the design components) and statements about
relationships between them. Gregor and Jones (2007) distinguish two different kinds of technologies
that can be designed: product artefacts and process artefacts. A product artefact is a thing, such as a
tool, an object, or a system. It can be either physical (e.g. a computer) or abstract (e.g. a diagram
notation). A process artefact is similar to a method (March and Smith 1995).
2.3 Design theory
Design Theories are formalisations of the knowledge resulting from Design Science Research.
Researchers generally consider that all high quality research should generate, evaluate, or refine
theory. Walls et al (1992) argued that a proper design theory should have seven components as
shown in figure 1: meta-requirements (a general set of requirements that a generalised solution would
address), meta-design (a generalised design that could be adapted to a particular problematic
situation), a design method (for adapting the meta-design to the particular problematic situation),
400

John Venable and Richard Baskerville
kernel theories informing the meta-design (according to the meta requirements), kernel theories
informing the design method (according to the meta-requirements and meta-design), testable
hypotheses to test the meta-design and testable hypotheses to test the design method. They also
state that design theories are prescriptive in guiding designers as to what they should do.
Figure 1: Structure of a design theory (Walls et al, 1992)
Venable (2006b) asserted that kernel theories and testable hypotheses are not part of a design theory
per se and that a design method is unnecessary in a design theory. He advocated a simpler, nonprescriptive
form of theory (which he termed a “utility theory”) containing only the problem space to
which a design theory applies, a solution space describing a design, and a relationship between them
asserting that the design has utility of some kind (efficiency, effectiveness, etc.) with respect to the
problem space. Relationships between components of the problem space and between components
of the solution space should also be part of the design/utility theory. Gregor and Jones (2007)
developed an alternative form of design theory with eight parts: Purpose and scope, Constructs,
Principles of form and function, Artefact mutability, Testable propositions, Justificatory knowledge,
Principles of implementation, and an Expository instantiation (optional). Baskerville and Pries-Heje
(2010) reacted to the complexity of the formulations of ISDTs and design theories in Walls et al (1992)
and Gregor and Jones (2007). Their simplified formulation of “explanatory design theory” includes
only “general requirements” and “general solution” as well as an explanatory relationship between the
two of them (i.e. the general solution explains how to address the general requirements).
2.4 Evaluation
Evaluation is an essential activity in Design Science Research (Nunamaker et al. 1991; March and
Smith 1995; Hevner et al. 2004; Vaishnavi and Kuechler 2004; Venable 2006a; 2006b; Peffers et al.
2008). Evaluation is what makes DSR a science. Without evaluation, asserted design theories, utility
theories, or technological rules are just untested theories, conjectures, and hypotheses; there is no
evidence to back them up. We identify five evaluation issues. One issue in DSR concerns what the
evaluand (the artefact being evaluated) is: process or product; construct, model, method, or
instantiation; or design theory. A second issue concerns what aspects or qualities of the evaluand are
to be evaluated. Artefacts are evaluated for their fitness to purpose. Purposes are extremely varied,
but one can identify some key characteristics, such as Checkland’s five E’s: efficiency, effectiveness,
efficacy, (a)esthetics, and ethicality (Checkland and Scholes 1990), which are aspects of utility.
Venable (2006a) also notes three aspects of evaluation: (1) evaluation of efficiency, effectiveness and
efficacy (i.e. utility), (2) evaluation of the artefact in comparison to other solution technologies (i.e.
relative utility), and (3) evaluation for undesirable impacts (i.e. side effects). A third issue concerns the
rigour of the evaluation. There are two senses of rigour. The first sense is whether an improvement or
achievement of purpose can be attributed to the evaluand (i.e. its efficacy), or alternatively whether it
could be caused by some confounding variable. The second sense is whether the evaluation can
work within the myriad complicating aspects of real use, i.e. its effectiveness. The fourth issue is the
choice of evaluation method(s). Hevner et al (2004) suggest analytical, case study, experimental, field
study, and simulation. Venable (2006a) distinguishes between artificial evaluation and naturalistic
evaluation. Artificial evaluation supports controlling for confounding variables (to evaluate efficacy)
while naturalistic evaluation gives “the proof of the pudding” (evaluating effectiveness). Naturalistic
evaluation occurs when one has real users, using the real technology on a real problem or in a real
context (Sun and Kantor 2006). Artificial evaluation methods include various forms of simulation or
experiment while naturalistic evaluation includes case studies, surveys, field studies, and action
research. Pries-Heje et al (2008) identify different strategies for conducting DSR evaluations,
contrasting ex ante with ex post evaluation and artificial vs naturalistic evaluation, giving a 2x2 matrix
of strategies. Venable et al (2012) further expand on Pries-Heje et al (2008), proposing a framework
for helping design science researchers to select among different evaluation methods to better design
the evaluation part of a DSR project. A final issue is the potential for errors in evaluating a new
technology. Baskerville et al (2007) analyse and describe some key areas where evaluation can go
401

John Venable and Richard Baskerville
wrong or lead to unfaithful, inaccurate, or erroneous results, including both type 1 (false positive) and
type 2 (false negative) errors.
3. Analysis of research methods using a design science research lens
This section considers what lessons may be learned from applying Design Science Research to
research methods and how DSR might improve the rigour of research methods. We look at four main
areas in turn: goals (and constraints) of research methods, design artefacts in research methods, the
need for evaluation of research methods, and design theory in research methods.
3.1 Goals of research methods
Goals of research methods include both primary goals and practical goals that enable their effective
use. We discuss goals first, then implications from a DSR perspective.
3.1.1 Primary goals
The primary goals of research methods include both general goals and goals specific to each
research project. Rigour and relevance are general goals desired of all research and research
methods. Research methods must also be appropriate for answering the specific research question or
achieving the specific goal of the research. Rigour is concerned with the reliability of research.
Properly following a research method should ensure that the research findings are correct or that the
probability of the findings being incorrect is sufficiently low. Relevance is concerned with whether the
findings are useful – either to theory and further research or to practice. While following a research
method may not ensure relevance, some research methods may be more amenable to investigating
topics that are relevant. There are several kinds of research questions, including questions concerned
with theory testing (and extension), theory building, understanding and explanation, prediction
(cf.Gregor 2006), evaluation, invention of new technologies to solve problems, or achieving human
emancipation. Different research methods are more or less appropriate for answering different kinds
of research questions. There are also several different kinds of research field or domain, including
physical sciences, biological sciences, behavioural sciences, social sciences, and various forms of
applied technical sciences and socio-technical sciences. This list is very high level and is not
comprehensive. Different research methods will be more or less appropriate for research in these
different domains.
3.1.2 Practical goals
There are many practical goals for research methods that affect their ability to be applied and to
achieve the primary research goals given above, including ease of learning, ease of use, cost, time
required, facilities or other resources required, access to research subjects, ability to deal with more
people and research subjects, and depth of engagement with research subjects.
For novice researchers or researchers trying out a new methodology, ease of learning is an important
issue. Once learned, a research method that is easier to use will be preferred.
Research methods that have lower costs are preferable. Costs include time/labour, tools (e.g.
computers, software), and other resources (e.g. communications and physical space) needed to apply
the research method
Some research methods require less access to research subjects (people or organisations) than
others. Access is often difficult to obtain, so low requirements for access are preferable to high
requirements for access.
Some research methods are more suitable to dealing with research questions that concern larger
numbers (or breadth) of people while others support a greater depth of engagement with research
subjects.
Practical goals above are often in the form of constraints on available researcher time, expertise,
funding, access to research subjects, and others. The above list is probably not comprehensive.
402

John Venable and Richard Baskerville
3.1.3 Lessons from DSR
The goals described above represent meta-requirements for research methods. Research methods
represent meta-designs. Goals or meta-requirements form contingencies that are addressed with
differing levels of utility by different research methods.
A key issue is that the number of goals and constraints is fairly large and to some extent they conflict.
Evaluating different contingencies is complex and difficult as it is a combinatorial problem without hard
yes/no answers. However, the provision of a clear system of design theories with research goals as
the contingencies would aid in selecting an appropriate research method for a particular research
situation. The approach used in the Design Theory Nexus system described in Baskerville and Pries-
Heje (2008) for helping managers to select an appropriate change management methodology might
also be applied to a system to help selection of research methods.
3.2 Design artefacts in research methods
What kind of design artefact is a research method? A research method is clearly a process (method)
artefact. Research methods often (always?) make use of various tools, techniques, and frameworks
for organising and doing the research work. These are clearly product artefacts. Descriptions of how
to use a tool, technique, or framework are in turn process artefacts. Research methods clearly are a
class of March and Smith’s (1995) method design artefact.
From the Walls et al (1992) perspective, the research method itself seems to be a meta-design, which
is aimed at achieving the meta-requirements of research (see the goals discussion in section 3.1
above).
Venable (2006b) emphasised that it is important to clearly identify and precisely define the
components/constructs in the problem space and the solution space, in this case the goals and
requirements of the research method and the research method itself.
Importantly, tools and techniques may sometimes be used with other methods than the methods for
which they were developed. Any attempt to conduct DSR on research methods needs to consider the
possibility that research tools and techniques have uses both as part of and outside of any particular
research method.
3.3 The need for evaluation of research methods
A key aspect of DSR is evaluation of new technologies (including research methods).
3.3.1 Why evaluate research methods?
Currently, we lack rigorous evidence of the effectiveness, efficiency, and efficacy of research
methods. Some evidence clearly is provided by the evident progress of human knowledge. Other
evidence has been developed through rational argument in research essays (i.e. in the logicoargumentative
paradigm). However, without more rigorous evidence, how do we really know that
research methods are effective? We also lack evidence about whether research methods achieve
other desirable, practical goals – e.g. ease of learning, ease of use, and time and resource
requirements. Moreover, we especially lack evidence about the relative efficiency and effectiveness of
research methods, i.e. compared to each other. In order to make better judgments in the choice of
research methods, it would be very helpful to have better evidence than our currently ad hoc
assertions about them.
3.3.2 How could we evaluate research methods?
Like other designed artefacts, research methods should be evaluated for their utility in achieving their
goals, including primary goals of rigour and relevance, suitability to type of research questions and
goals, and suitability to type of research domain or topic, as well as secondary, practical goals.
Rigour assessment during the reviewing process is rather subjective and often simply confirms that
an established research method has been followed, assuming that the outcome will therefore be
correct, even though the research method itself may not have been rigorously evaluated.
403

John Venable and Richard Baskerville
However, perceived rigour is not clear evidence of the rigour of a research method. Convincing
evidence would come from the detection of errors in findings and determining error rates. Where
errors are found in findings, the research method itself is in some way responsible (after all, it didn’t
prevent the error), but additional causes are possible (e.g. misapplication of the method). Finding
errors would require triangulation of findings through other research, which may not be possible or
thought to be worth the research expenditure.
The rigour of some research methods, especially positivist ones, can be demonstrated through
mathematical analysis or proof.
To assess whether research methods not amenable to mathematical analysis achieve an acceptable
level of rigour, research conducted could be subjected to critical analysis and ratings of rigour. The
rigour ratings could then be compared and correlated across projects using different research
methods. Mixed- and multi-method approaches would complicate these analyses. It may also be
helpful to analyse failed research projects, where outcomes are not accepted for publication. One
might also consider whether a research method correlates with high or low quality journals.
While factors other than research method used are likely more important drivers of relevance,
nonetheless, one could attempt to assess or rate relevance (e.g. of research projects in the literature)
and correlate relevance ratings with the research method used.
One can also analyse whether a research method used correlates with types of research questions or
domains of research. To a great extent, rational argument can (and has been) used for this, but it may
be useful to identify anomalies in the extant research.
Assessment of the performance/utility of research methods in achieving goals in other, practical areas
may be more fruitful. One could, for example, survey users of various research methods about their
perceptions of ease of learning and ease of use of research methods, tools, techniques, etc. One
could evaluate resource consumption by gathering data on time required, resources consumed, costs,
etc. and correlating them with research methods.
Where a method is new and untried, evaluation is largely limited to analytical kinds of evaluation, e.g.
rational argument, surveys or focus groups with experts in research methodology, or possibly some
form of simulation. Empirical, naturalistic evaluation will likely be very limited at first, with limited trials,
probably using qualitative evaluation methods.
3.3.3 Pitfalls and difficulties of evaluation of research methods
Rigour is assessed in the reviewing process rather subjectively. Care must be taken to try to reduce
the variability by setting out very specific criteria, preferably objective criteria.
Relevance is more likely due to choice of research topic and domain than the research method.
However, some part of relevance (e.g. very new or contradictory findings) may be due to the research
method used. This needs careful assessment when evaluating a research method.
Research methods need to be applied properly or one isn’t actually evaluating the research method
(Baskerville et al. 2007). Misapplications of a research method may be attributed to the method rather
than to the method user. However, if research methods are prone to misapplication, then that is a
problem with the research method.
Different research projects may have different research contexts/situations, research questions,
resources, staff, which means that it may be completely inappropriate to compare different research
projects. Very importantly, some research projects and research method applications may present
unusual rather than typical circumstances.
3.4 Design theory in research methods
Design theories are formalisations of the knowledge obtained from Design Science Research,
including the design of research methods.
404

John Venable and Richard Baskerville
3.4.1 Why design theories of research methods?
Theory is the way that academics communicate with each other about research (Venable 2006b).
Clear, formalised design theories about research methods should reduce misinterpretation and allow
research method researchers to accurately research the same topics. Design theories about research
methods are important when developing new research methods as one needs to be able to clearly
articulate how a new research method is different from existing ones.
Clear statements of design theories may also be useful for practitioners – in this case the practitioners
are researchers. Even if one doesn’t research about research methods, research method users can
greatly benefit from clear and precise statements about them (their meta-design) and under what
circumstances they have utility or should be used (their meta-requirements).
We propose that formal design theories could and should be developed for every research method.
Doing so would aid the ongoing development and progress in research methods and in the long run
should increase our ability to generate new, valid knowledge.
3.4.2 What should be in a design theory for a research method?
Four different forms of design theories have been suggested (Walls et al. 1992; Venable 2006b;
Gregor and Jones 2007; Baskerville and Pries-Heje 2010). A design theory should use one of these
and at a minimum cover a core of the parts of the design theory.
A design theory for a research method at a minimum needs three things: (1) meta-requirements
(Walls et al. 1992) or equivalent, (2) a meta-design (Walls et al. 1992) or equivalent, and (3) a
relationship between them. Other aspects of design theories are optional (but may be useful).
All aspects of design theories need to be stated as precise, well-defined constructs. The metarequirements
or equivalent should include and clearly state the goals (and possibly the constraints)
for which the research method is suited (or possibly the level to which they are suited, such as low
cost). The primary goals that should be addressed include research rigour and relevance and the
level of each that can be achieved or expected, the type(s) of research question for which the method
is suited, and the domain(s) for which the research method is suited. Optional goals should also be
addressed by statements about the anticipated levels of ease of learning, ease of use, cost, time
required, facilities or other resources required, level of access to research subjects needed, ability to
deal with more people and research subjects, and depth of engagement with research subjects.
The meta-design or equivalent should include a detailed description of the research method’s
process, including steps, actions to take, tasks, decisions, and iterations, as well as tools and
techniques to use, and when and how to use them. Descriptions of tools and techniques may refer to
other design theories for detail about them.
It is also very useful to describe how to apply the methodology to create a detailed research design.
This could be a separate design method, included as part of the model of the solution space, or
provided separately as principles of implementation (Gregor and Jones 2007).
4. Conclusion
Research Methods are designed artefacts. The evolution and development of research methods
needs a more rational basis for assuring their utility and rigour. This paper proposes that DSR could
(and should) provide such a basis. The rigour would come primarily from (a) more precise and
complete statements of design theories of research methods and (b) more rigorous evaluation of
research methods. We have identified the primary and secondary (practical) goals of research
methods, which form the meta-requirements part of design theories about research methods and
serve as criteria for evaluation of the utility of research methods. We have also identified types of
design artefacts in research methods, which would define the research method meta-design in a
design theory. The configuration of constructs and artefacts that make up a research method are what
are evaluated for how well they fulfil the research method’s goals. We further identified relevant
evaluation methods and issues relating to research method evaluation. Applying a DSR approach to
research about research methods should enhance the quality of research methods and therefore the
quality of the research conducted using them.
405

John Venable and Richard Baskerville
What we have proposed in this paper is only a first step. Work is now needed to analyse existing
research methods to develop clear design theories about them. The basis for this already exists in the
literature; it simply needs critical reading and formalising in design theories. Consideration also is
needed re designing and carrying out appropriate evaluations of research methods, to provide
evidence of their effectiveness, efficiency, efficacy, etc. in achieving their disparate purposes. Once a
body of literature on design theories and evaluations of research methods is developed, there would
be a clearer basis for further research on research methods.
References
Baskerville, R., and J. Pries-Heje (2008) “The Design Theory Nexus”, MIS Quarterly, Vol 32 No. 4, pp. 731-755.
Baskerville, R., and J. Pries-Heje (2010) ”Explanatory Design Theory”, Business & Information Systems
Engineering, Vol 2010, No. 5, pp 271-282.
Baskerville, R., J. Pries-Heje, and J. R. Venable (2007) “Soft Design Science Research: Extending the
Boundaries of Evaluation in Design Science Research”, 2nd International Conference on Design Science
Research in Information Systems and Technology (DESRIST 2007). Pasadena, California, USA.
Benbasat, I., Ed. (1991) The Information Systems Research Challenge: Experimental Research Methods,
Harvard University Press, Boston.
Berger, P. and T. Luckmann (1966) The Social Construction of Reality: A Treatise in the Sociology of Knowledge,
Anchor Press, New York.
Bunge, M. (1967) Scientific Research I: The Search for System, Springer-Verlag, New York.
Burrell, G. and G. Morgan (1979). Sociological Paradigms and Organisational Analysis, Heinemann, London.
Campbell, D. and J. Stanley (1963) Experimental and Quasi-Experimental Designs for Research, Houghton
Mifflin, Boston, MA, USA.
Checkland, P. and J. Scholes (1990) Soft Systems Methodology in Action, J. Wiley, Chichester.
Galliers, R. and F. Land (1987) "Choosing appropriate information systems research methodologies."
Communications of The ACM, Vol 30, No. 11, pp 900-902.
Galliers, R. D. (1991) “Choosing Appropriate Information Systems Research Approaches: A Revised Taxonomy”,
IFIP 8.2 Working Conference on Information Systems Research: Contemporary Approaches and Emergent
Traditions, H.-E. Nissen, H. K. Klein and R. Hirschheim, Eds., North-Holland Elsevier, Copenhagen, pp 327-
345.
Glaser, B. G. and A. L. Strauss (1967) The Discovery of Grounded Theory: Strategies for Qualitative Research,
Aldine Publishers, Chicago.
Gregor, S. (2006) "The Nature of Theory in Information Systems", MIS Quarterly, Vol 30, No. 3, pp 611-642.
Gregor, S. and D. Jones (2007) "The Anatomy of a Design Theory", Journal of the Association for Information
Systems, Vol 8, No. 5, pp 312-335.
Hevner, A. R., S. T. March, J. Park, and S. Ram (2004) "Design Science in Information Systems Research", MIS
Quarterly, Vol 28, No. 1, pp 75-105.
Hirschheim, R. (2008) "Some Guidelines for the Critical Reviewing of Conceptual Papers", Journal of the
Association for Information Systems, Vol 9, No. 8, pp 432-441.
Kuhn, T. S. (1970) The Structure of Sientific Revolutions, Chicago, University of Chicago Press.
Latour, B. (1987) Science in Action, Harvard University Press, Boston.
March, S. T. and G. F. Smith (1995) "Design and natural science research on information technology", Decision
Support Systems, Vol 15, No. 4, pp 251-266.
Mason, R. (1991) “MIS experiments: A pragmatic perspective”, The Information Systems Research Challenge:
Experimental Research Methods, I. Benbasat Ed., Harvard University Press, Boston, pp 3-20.
Miles, M. B. and A. M. Huberman (1994) Qualitative Data Analysis: A Sourcebook of New Methods, Sage,
Newbury Park, CA, USA.
Mumford, E., R. Hirschheim, G. Fitzgerald, and A. T. Wood-Harper, Eds. (1985) Research Methods in
Information Systems, Elsevier Science Publishers, Amsterdam.
Nagel, E. (1961) The Structure of Science: Problems in Scientific Explanation, Routledge and Kegan Paul,
London.
Nunamaker, J. F. J., M. Chen, and T. D. M. Purdin (1991) "Systems Development in Information Systems
Research", Journal of Management Information Systems, Vol 7, No. 3, pp 89-106.
Orlikowski, W. J. and J. J. Baroudi (1991) "Studying Information Technology in Organizations: Research
Approaches and Assumptions", Information Systems Research, Vol 2, pp1-28.
Peffers, K., T. Tuunanen, M. A. Rothenberger, and S. Chatterjee (2008) "A Design Science Research
Methodology for Information Systems Research", Journal of Management Information Systems, Vol 24, No.
3, pp 45-77.
Pries-Heje, J., R. Baskerville, and J. R. Venable (2008) “Strategies for Design Science Research Evaluation”,
16th European Conference on Information Systems (ECIS 2008), Galway, Ireland.
Simon, H. A. (1969) The Sciences of the Artificial, MIT Press, Cambridge, MA, USA.
Simon, H. A. (1996) The Sciences of the Artificial, 3 rd edition. MIT Press, Cambridge, MA, USA.
Sun, Y. and P. B. Kantor (2006) "Cross-Evaluation: A new model for information system evaluation", Journal of
the American Society for Information Science and Technology, Vol 57, No. 5, pp 614-628.
Vaishnavi, V. and W. Kuechler (2004) "Design Science Research in Information Systems", [online], Association
for Information Systems, http://desrist.org/design-research-in-information-systems/.
406

John Venable and Richard Baskerville
Venable, J. R. (2006a) “A Framework for Design Science Research Activities”, Proceedings of the 2006
Information Resource Management Association Conference, Washington, DC, USA.
Venable, J. R. (2006b) "The Role of Theory and Theorising in Design Science Research", Proceedings of the
First International Conference on Design Science Research in Information Systems and Technology,
Claremont, CA, USA, 24-25 February, 2006, pp 1-18.
Venable, J. R., J. Pries-Heje, and R. Baskerville, “A Comprehensive Framework for Evaluation in Design Science
Research”, Proceedings of the 7th International Conference on Design Science Research in Information
Systems and Technology (DESRIST 2012), K. Peffers and M. Rothenberger, Eds., Springer, New York,
(forthcoming).
Walls, J. G., G. R. Widmeyer, and O.A El Sawy (1992) "Building an information system design theory for vigilant
EIS", Information Systems Research, Vol 3, No. 1, pp 36-59.
407

Constructing Homogeneous Likert-Type Summative Rating
Scales According To Classical Measurement Theory
Marianne Viljoen
School of Higher Education Studies, Faculty of Education University of the
Free State, Bloemfontein, South Africa
viljoenmc@ufs.ac.za
Abstract: The study of human behaviour requires the use of valid and reliable instruments in order to quantify,
analyse and interpret such behaviour. It is an accepted fact in research design that the reliability and validity of
quantitative research depends on the reliability and validity of the instruments applied (Kerlinger 1986; Maas
1998). Many researchers are hampered in their efforts to design quantitative research due to a lack of measuring
instruments. Knowing the requirements for constructing Likert-type summative rating scales will greatly increase
the reliability and internal validity of research projects. This paper offers the guideline for scale construction
according to the model of Classical Measurement Theory as presented by Nunnally in the 1970s. This method of
scale construction requires homogeneity of scale dimensions and applies item analysis as a measure of
standardisation and not factor analysis as is the case with heterogeneous scales. In executing quantitative
research, the statistical properties of scales should be scrutinised for their applicability. Researchers should take
care when selecting scales in quantitative research. and need experience in the methodological guidelines of
scale construction. The writing of an item pool, the pros and cons of different numbers of scale steps and types of
anchors are discussed with relevance to current literature. Determining the reliability and validity of a rating scale
is clearly outlined. All applicable references regarding the elements of scale construction as posited by literature
from the 1970s to the present (researching a time span of 40 years) have been consulted recounting the status of
the Nunnally principles as they stand today.
Keywords: classical measurement theory; likert-type; summative; rating scales, item analysis; homogeneous
scales
1. Introduction
All researchers strive to increase the reliability of rating scales. Measuring instruments are
fundamental to the implementation of solid quantitative research in the social sciences and business
fields. As scientists study human behaviour they increasingly need valid and reliable instruments in an
attempt to quantify, analyse and interpret such behaviour. Renowned authors, Kerlinger and Maas
unambiguously state that it is a clearly established fact that the reliability and validity of quantitative
research is tied in closely with the reliability and validity of the instruments applied (Kerlinger 1986;
Maas 1998). This fact presents a special problem to supervisors who need to assist prospective
students in designing quantitative research projects by finding valid and reliable measuring
instruments which measure human attributes of concern to planned studies. In addition it is necessary
that the construction characteristics of existing scales be examined for the appropriateness of use in
intended research and to assert their applicability. The basic premise in quantitative research design
is that a scale should only be used for measurement in a sample of which the hypothesised
population is similar to the population for which the original scale was designed; otherwise it should
be adapted or a more applicable one written. Therefore it becomes essential that researchers using
quantitative designs scrutinise all measuring instruments before they are applied to test groups and in
order to do so researchers need at least minimal proficiency in scale construction which is provided by
this article.
The range of standardised instruments available for measuring human behaviour in the study field of
business, health and social sciences can be broadly categorised into three main classes, namely the
measurement of achievement and aptitude, the measurement of personality traits and the
measurement of sentiments. Included under sentiments are the measurement of attitudes. All of the
above measurements, except the measurements of attitudes, require intensive expertise and the
knowledge of highly specialised statistical procedures. Consequently they are designed over long
periods of time.
Attitudinal rating scales however are easy to design and are widely applicable provided that the
principles of homogeneity of questions are adhered to. According to Kerlinger (1986 p. 494) “A rating
scale is a measuring instrument that requires the rater or observer to assign the rated object to
categories or continua that have numerals assigned to them.” Arnold, McCroskey & Prichard (Online)
agree that Likert-type attitude scales are “One of the most popular methods of measuring attitudes...”
408

Marianne Viljoen
The first rating scale was constructed by Likert in 1932 (Arnold, et al. Online ) and consequently
scales with scale steps are referred to as Likert-type scales. The attribute of homogeneity is a
necessary prerequisite of this form of scale construction. Homogeneity, as discussed furhteron,
implies that there should be a positive relationship between the sub dimensions of the scale.
Therefore a person answering positively on one dimension of the scale will be inclined to answer
positively on other dimensions as is the case in attitude scales. Gliem and Gliem (2003 p. 82)
describe the applicability of Likert-type scales: “Oftentimes information gathered in the social
sciences, marketing, medicine, and business, relative to attitudes, emotions, opinions, personalities
and description’s of people’s environment involves the use of Likert-type scales”.
Regarding the appropriateness of Likert-type summative rating scales to measure attitudes it is
necessary to state that the term attitude by no means inhibits the range of applicability of such scales
in research. Reflecting on the term attitude as defined by various resources, it becomes apparent that
the application of attitude scales to measure human behaviour are extremely varied. Coleman (2001
p. 63) defines an attitude as “...a more or less consistent pattern of, cognitive or conative, and
behavioural responses (or of feeling, thinking, and behaving) towards a psychological object,...”
Kerlinger (1986 p. 495) offers a similar definition wherein he states that an attitude “...is an organised
predisposition to think, feel, perceive, and behave toward a referent or cognitive object”. Applying this
construct to a research paradigm, it is clear that attitude scales may be used to measure attitudes
towards an array of different attitudinal objects in various fields such as the social and business
sciences, and education. Apart from cognitive objects such as management styles and consumer
behaviour, some innate personal qualities such as motivation, job satisfaction and time perspective
can also be measured with attitude rating scales.
This article will present the method for constructing and standardising homogeneous Likert-type
attitude rating scales by means of item analysis which was designed by Nunnally and which is still
highly relevant and applicable today. This method, endorsed by Nunnally in 1978 is referred to as
Classical Measurement Theory (Nunnally & Bernstein 1999) and the validity of all the steps in scale
construction will be argued against the recommendations of modern day researchers.
The guidelines that will be discussed are the following: Defining the attitudinal construct, writing of an
item pool, types of anchors, number of scale steps, piloting of the scale as well as reliability and
validity.
2. Conditions for constructing a likert-type summate attitude rating scale
Many researchers shy away from developing measuring instruments because of the sophisticated and
laborious task of standardising such instruments by means of factor analysis. Nunnally, however
clearly indicates that a scale need not be standardised by means of a factor analysis provided that the
scale is constructed from an item pool which is homogeneous with regard to the object being
measured. “ In the construction of most types of psychological measures, factor analysis of an item
pool should be considered only as a last resort, after efforts to hypothesise homogeneous scales
have led to naught” (1970 p 437).
The condition, under which a less experienced person may construct a summative rating scale for
measuring attitudes, is that the scale must consist of homogeneous items. Homogeneous items are
also called uni-dimensional items and such a test is standardised by doing an item analysis (La Trobe
& Acott, 2000) instead of a factor analysis which is more complicated. Gliem and Gliem (2003)
describe homogeneity by stating that the items in a scale should be connected by an underlying
quantitative dimension. If for instance you are writing a scale to measure the psycho social
background of individuals you could have sections that measure socio economic status, social
support and level of depression. Each of these topics would measure different dimensions of psycho
social background but according to the theory, these aspects all form part of the dimensions of
psycho social backgrounds and are related.
It is also necessary to consider that Classical Measurement Theory (CMT) differs from Item Response
Theory (IRT) which underpins latent trait scales. Polit and Beck (2008 p 476) state “Whereas the
items on a CMT scale are designed to be similar to each other to tap the underlying construct in a
comparable manner, items on a latent trait (IRT) Likert scale are carefully chosen and refined to tap
different degrees of the attribute being measured” Items written with IRT as basis lie on a continuum
indicating varying degrees of difficulty in choosing a rating. In lay terms this means that item number 5
409

Marianne Viljoen
in the scale should be emotionally more difficult to rate than item number 1. This method of scale
construction is also referred to as the Rasch model (Polit and Beck 2008) which is applied to measure
uni-dimensionality and item fit (Chachamovich, et al.)
3. Procedure for writing of a likert-type summative rating scale according to
classical measurement theory
The following summary contains a description of how to go about writing a rating scale. Each step will
be described in more detail further on.
Define the attitudinal object or construct, and if applicable, it’s dimensions which are to be
measured. Have a specific population in mind.
Write an item pool (begin with 40 item statements which will be reduced to 20)
Determine the number of scale steps
Choose the anchors
Decide on the length of the scale
Pilot the scale
Calculate reliability
Argue validity
3.1 Writing of an item pool
Summative rating scales for measuring verbalised attitudes consist of a number of item statements
(usually 20) from the content domain, with which respondents may agree or disagree on a five-ormore
point scale. The scores of a respondent are summed to give a total which then indicates his/her
stance on the measure. Neutral or extreme statements should not be used as this creates less
variance (Nunnally 1970). When scoring the negative statements, scores are inverted to give the
“true” score. The use of both positive and negative statements is an attempt to avoid response style
as a confounding variable.
Homogeneous attitude scales are developed from an underlying theory or hypothesis which is also
called the content domain of the variable. Therefore it is essential that before writing statements for
the item pool, a well-articulated definition of the variable under investigation should be given (Albaum
1997; Dawis, 1987). Abeles (1987) agrees that items should be based on preconceived structures.
Ambiguously understood constructs will certainly lead to invalid test scores (Polit & Beck 2008).
Collecting possible item statements can be done by employing qualitative techniques where
respondents have the opportunity to voice their feelings and experiences regarding the attitudinal
object. A team of professionals may work together in formulating statements. A technique which is
advised by Polit and Beck (2008 p 482) involve sending the item pool to a number of professionals in
the field and having them rate each item on its relevance to the construct being measured. Dawis
(1987) suggests that statements should be back-translated by experts in the field who know the
theory but were not involved in the development of the scale. In so doing it is possible to affirm the
relevance of the items to the theory. This also serves to establish coverage of the content domain
which determines validity
The mother-tongue of the respondents is an important consideration when formulating the
statements: When statements are written in the second language of the respondents, special
attention should be paid to the level of comprehension of the statements. Turner (1993) stresses this
aspect when stating that language limitations affect the accuracy of the responses on scale items and
inhibits the reflection of a true attitude. Dawis (1987) calls this aspect “readability” and emphasises
that often it is better to use lay language than more sophisticated academic language. Polit and Beck
(2008) recommends that statements be clear, short and should not contain jargon. Furthermore they
advise against double negatives and double barrelled item statements.
Another aspect of scale construction that may confound researchers is the aspect of positive and
negative scale item statements where reversing of score are employed when summating the scores
over all the items. Nunnally(1970) suggests that the scale should consist of equal numbers of
moderately positive and moderately negative statements. He posits that this may minimize response
410

Marianne Viljoen
bias or acquiescence as Polit and Beck call it (Polit & Beck 2008). There is however another
phenomenon to consider namely that second language speakers or inexperienced respondents could
become confused when confronted with both positive and negative items in the same scale. Polit and
Beck (2008) thus distinctly advise the use of only positively worded item statements and declare that
there is modern day evidence to support this theory.
3.2 Number of scale steps
The number of scale steps that is appropriate for reliable and valid research has been a bone of
contention among scale designers and researchers since 1932 when Likert wrote the first summative
rating scale. The original model contained only five steps;
Highly agree
Agree
Neither agree nor disagree
Disagree
Highly disagree
The debate surrounding the number of scale steps, centre around three issues; how the number of
scale steps affect reliability; whether parametric or non-parametric statistics may be applied to data
gathered by such a scale; and whether the number of scale steps should consist of an even or
uneven number.
Nunnally (1970 p 425) cites Guilford (1954) who said that according to psychometric theory it has
been demonstrated that “...the reliability of individual rating scales is a monotonically increasing
function of the number of steps”. This increase in reliability is said to level off when seven to eleven
scale steps are used. When implementing a scale with too many steps the researcher has to consider
the possibility that respondent fatigue may set in for a respondent having to gauge his perceptions,
feelings or thoughts about an object on so many levels.
With a higher number of scale steps, equidistance of scale steps can be assumed. And this translates
to higher reliability since it offers the respondents the opportunity to reflect true feelings, cognitions
and perceptions in their choice between a larger number of alternatives.
Another consideration when deciding on the number of scale steps to use in an attitude scale is that a
total score on a five point scale is considered to be an ordinal measurement which is analysed by
non-parametric statistical analysis. According to Turner, (1993) parametric statistics are more
powerful than non-parametric statistics but researchers do not agree on this point.
This argument is relevant only when considering the other measurements in the design and the group
size for applicability of parametric tests. Polit and Beck (2008) posit that variability is enhanced when
opting for “numerous response options”. Variability would indicate greater variance which has
statistical value in that the distributions of test scores would then approximate the normal curve and
facilitate the use of parametric techniques.
Bernstein (1996 p. 1) who describes himself as having much experience in the practical use of Likertscales
stresses the issue of the heuristic value of using Likert scales as interval measures. With an
interval scale one is able to “...quantify degrees of affect...” of test respondents.
In opposition to these statements comes the slant of modern day researchers who are adamant that
equidistance of adjacent points on a Likert scale cannot be claimed (Heiser 2007). According to them
a Likert scale with any number of scale points is merely an ordinal scale and should thus be analysed
with non-parametric statistics. However the debate continues. Heiser (2007) declares that the issue is
not whether intervals are equal but whether the steps are close enough to be treated as equal which
implies parametric statistics in analyses and drawing the right conclusions from hypotheses. To this
question the author argues that one ought to do both parametric and non-parametric statistical
analyses on such data.
The third issue, whether or not to use an uneven or even number of scale steps has also been
debated much. An odd number of steps make it possible for respondents to fall into the response
411

Marianne Viljoen
style of choosing the middle or neutral value. Regarding this Nunnally (1970) states that there is a
danger of uneven numbers of scale steps eliciting a response style when respondents have the option
to choose a “neither agree nor disagree” scale step. Often this is an easy way when respondents are
tired, disinterested or not willing to commit to an opinion. On the other hand he states that some
researchers believe that a “neutral option” makes subjects more “comfortable” in making ratings.
Modern researchers concur with this point of view (Polit & Beck, 2008) regarding the applicability of
scale steps with uneven numbers as more advantages.
In a research study done by Birkett (1986) 300 randomly selected respondents completed a Health
Locus of Control scale with two, six or fourteen response categories. According to the results reliability
were inclined to be highest when the questionnaire had six response categories A study done by
Preston and Colman (2000) tested scales that only differed in the number of response categories.
The indices of reliability, validity and discriminating power were significantly higher for scales with
more response options.
Notwithstanding these arguments, several modern authors advocate the use of only a five-point scale
(Huysaman 2001; Delport 2005; McMillan & Schumacher 2006).
3.3 Types of anchors
Rating scales usually have numeric as well as semantic anchors. There are a number of different
word pairs that may be used to rate attitudes. These are the scale steps i.e. ‘Completely agree’,
‘Mostly agree’ and so on. In actual fact the scale steps are numbers which are represented by word
pairs. Siegle (s.a.) gives a very useful summary of the most used word pairs:
Agreement: Strongly disagree, Mostly disagree Slightly disagree, Slightly agree, Mostly agree,
Strongly agree.
Frequency: Always, Usually, Seldom, Never.
Importance: Very important, Important, Moderately important, Of little importance, Unimportant.
Quality: Good, Acceptable, Poor
Likelihood: Definitely, Probably, Possibly, Very probably not.
Other options for anchoring that is supplied by Nunnally (1970) are percentages and adjective pairs.
Adjective pairs such as ‘Ineffective-Effective’, ‘Foolish-Wise’ and ‘Weak-Strong’ form a different type
of summative rating scale known as a semantic-differential scale which in essence has all the
characteristics of a summative rating scale and is easy to construct.
Regarding the types of word pairs that form anchors, care should be taken with the precise meaning
respondents attach to some words, especially if the scale is not written in their first language. The
exact difference in meaning between ‘Occasionally’ and ‘Rarely’ and ‘Somewhat’ and ‘Little’ is not
always clear to respondents. It is the author’s opinion that the anchor pairs ‘Agree/Disagree’ and the
range of scale steps that accompany them are more precise and easier to use because they cannot
be misunderstood. Likewise the adjective pairs of the semantic differential scale are concise and
easily understood.
3.4 Length of the scale
Nunnally (1970 ) advises that a scale include approximately 20 items. However Polit and Beck (2008)
state that variability is enhanced by including a larger number of items. They do not specify an exact
number. The concern should be to strike a balance between the number of steps that would ensure
optimal variance as apposed to a scale that creates response fatigue.
3.5 Pilot test the scale
The piloting of the scale is part of determining the reliability of the scale. Although the final scale
should have approximately 20 items, begin with 40 items:
412

Marianne Viljoen
Administer the scale to a group of subjects of which the size is at least five times the number of
items in the item pool. The more subjects used, the better.
Mark the test and reverse the scores if negative items are present.
Sum each item for all respondents to obtain its total scores.
Correlate each item’s total with the total score of the group using a Coefficient Alpha.
The item with the highest correlation coefficient becomes the gauge against which the other item
totals are measured. It is called the determining item.
Calculate the correlation of each item’s total score with that of the determining item’s total score.
Select the test items in the test with the highest correlation with that of the determining item.
Pilot test the shortened version of the test again and calculate the Cronbach’s Alpha statistic. It
should ideally be above 0.70 (Embi 2007 in Pallant 2002).
4. Validity
Being able to report on the reliability and validity of a scale is intrinsic to the whole process of scale
construction. Content validity is assumed when scale items are grounded in theory, provided that
scale items were taken from “...the universe of items related to the construct being measured” (Maas
1998 p. 11). This is possible when a scale is constructed by professionals who know the content
domain of the intended test. When researchers agree that the construct being measured is grounded
in theory, they may assume construct validity.
The Cronbach’s Alpha of the final scale as described above is also the reliability coefficient and
should be in the order of 0.7 There is an option to pilot the scale and directly and analyse the
Cronbach’s Alpha with deleted variables. This table indicates the effect of each item on the total alpha
coefficient by deleting it and reporting the alpha value without that item in the scale. The effect of the
item on the total reliability is then indicated. If the item affects the reliability negatively it may be
deleted. In this manner only scale items which are highly reliable are kept in the scale.
5. Confounding variables of Likert-type rating scales
The following confounding variables need to be controlled by the researcher;
5.1 Response styles
The most obvious of the measurement confounders in summative scales are response styles. There
are three typical response styles that may be biased. Firstly respondents who continually choose the
neutral option may be indicating fatigue, laziness or lack of cooperation. Conversely there may truly
be respondents with no opinion regarding the object being rated. To counter this bias, Nunnally
(1978) suggests using an even number of scale items.
The second response style bias may come in the form of respondents being overly positive regarding
the attitudinal object or the test administrator. This is similar to the confounding variable known as the
halo effect. Such may be the case when respondents favour the researcher and wish to impress or
gain favour from the researcher (Kerlinger 1986; Turner 1993). One of the ways in which to counter
this reaction is to keep the inventory anonymous. Of course anonymity prevents different scores of the
same respondent being recorded which in some cases defy the objective of the research.
Thirdly, the response style bias of being overly negative may be found among respondents who feel
negative towards the test situation or who are overly discontented in mood. In this case they will mark
the statements more negatively then they really feel.
5.2 Test-retest reliability and time interval
In assessing the test-retest reliability of a summative rating scale, the decision of the time interval
between first and second testing is a challenging aspect to consider. Huysamen (1976) warns against
the effects of test sensitization. It takes a certain time to elapse before subjects forget the questions
and answers they gave to the test. If they remember the questionnaire and give the same answers, a
false test-retest reliability is achieved.
413

Marianne Viljoen
In a study done by Inandi, Sahin and Guraksin (2001) a scale was developed to determine
perceptions about physicians. The retest was done with a 7-10 day interval on the same group of 220
respondents Clearly this time period is too short and allows for test sensitisation to confound the
Pearson correlation of 0.61 (p

Marianne Viljoen
The information supplied in this article will increase research capacity. It offers quantitative
researchers the opportunity to design and construct their own scales when standardised scales
cannot be found in literature.
Furthermore, for non-quantitative researchers who need to be consumers of quantitative research, the
knowledge gained from this article could be applied in assessing the reliability and validity of existing
scales. Data gathered by means of scales and variables thus quantified could be better evaluated. It
is thus another method to be used by the quantitative researcher to increase the reliability of selfconstructed
measuring instruments
References
Abeles, H. F. (1973) “ A Facet-factorial approach to the construction of rating scales to measure complex
behaviours”, Journal of Educational Measurement, Vol. 10, No. 2, pp 145–151.
Albaum, G. (1997) “The Likert scale revisited: an alternate version. (product preference testing)”, Journal of the
Market Research Society, Vol. 39 No. i2, pp 331-343.
Arnold, W. E., McCroskey, J. C. & Prichard, S. V. O. (s.a.) “The Likert–Type Scale, 2007”, [Online],
Babbie, E. and Mouton, J. (2001) The Practice of Social Research, University Press, Oxford.
Barkley, J. E. & Weaver, H. B. (1962) “Comparative Reliabilities and Ease of Construction of Thurstone and
Likert Attitude Scales”, The Journal of Social Psychology, Vol. 58, pp 109 – 120.
Bernstein, I. H. (1996) “Likert Scale Analysis”, [Online],
Birkett, N. J. (1986) “Selecting the Number of Response Categories for a Likert-Type Scale”,[Online],
Coleman, A.M. (2001) Dictionary of Psychology, University Press, Oxford.
Chachamovich, E., Fleck,M.P. Trentini, C.M. Laidlaw K and Power M.J. (2008), “Development and validation of
the Brazilian version of the Attitudes to Aging Questionnaire (AAQ): An example of merging classical
psychometric theory and the Rasch measurement model”, Health and Quality of Life Outcome, Vol. 6, No.5,
pp 1-10.
Dawis, R. V. (1987) “Scale Construction”. Journal of Counselling Psychology, Vol. 34 No. 4, pp 481 – 489.
Delport, C.S.L. (2005) “Quantitative Data Collection Methods” In De Vos, A.S., Strydom, H. Fouche, C.B. &
Delport, C.S.L. (Ed.) Research At Grass Roots: For The Social Sciences And Human Services Professions,
pp 165-196, Van Schaik, Pretoria.
Embi,R. (2007) Computor anxiety and computer self-efficacy among accounting educators at Universiti Teknologi
Mara (UITM), Malaysia, (Doctors dissertation), Malaysia State University.
Gliem, J. A. and Gliem, R. R. (2003) “Calculating, Interpreting, and Reporting Cronbach’s Alpha Reliability
Coefficient for Likert–Type Scales”, Midwest Research to Practice Conference in Adult, Continuing, and
Community Education, pp 82-88.
Heiser, D. A. (2007), “Ordinal, Nominal and Likert Scale Variables”, [Online], Ordinal Issues,
http://www.daheiser.info/excel/notes/NOTE%20J.pdf
Huysamen, G.K. (1976) Inferensiële Statistiek en Navorsingsontwerp, Academica, Pretoria.
Huysamen, G.K. (2001) Methodology for the social and behavioural sciences, University Press, Oxford.
Inandi,T.I.; Sahun, N. & Guraksin,A. (2002) “A Prelimanary Study for the Development of a Scale to Assess
Perceptions About Physicians”, Journal of Evaluation in Clinical Practice,Vol. 8 No.1, pp.71-75.
Kerlinger, F.N. (1986) Foundations of Behavioural Research,3 rd .ed.,Holt, Rinehart and Winston, INC., Orlando.
La Trobe, H.L. and Acott, T.G. (2000), "A modified NEP/DSP Environmental Attitude Scale”, The Journal of
environmental Education, Vol. 32, No.1, pp 12-20.
Maas, F. (1998) Research Methods and Statistics in Occupational Therapy, Department of Occupational
Therapy, The University of Queensland, Queensland.
McMillan, J.H. and Schumacher, S. (2006) Researching Education: Evidence Based Inquiry, 6 Th ed., Pearson,
New York.
Nunnally J. C. (1970) Introduction to Psychological measurement, Mcgraw-hill, New York.
Nunnally J. C. (1978) Psychometric Theory, Mcgraw-hill, New York.
Nunnally, J. C. and Bernstein, I.H. (1999) “ Psychometric Theory 3 rd ed.” Journal of Psycho Educational
Assessment, Vol. 17, pp 275-280.
Obermiller, C., and Spangenberg, E. R. (1998), “Development of a Scale to Measure Consumer Scepticism
Toward Advertising”, Journal of Consumer Psychology, Vol. 7, No. 2, pp 159–186.
Polit,D.F. and Beck,C.T. (2008), Nursing Research. Generating and Assessing Evidence for Nursing Practice, 8 th
ed., Wolters Kluwer, New York.
Preston, C. C. and Colman, A. M. (2000), “Optimal Number of Response Categories in Rating Scales: Reliability,
Validity, Discriminating Power, and Respondent Preferences”, Acta Psychologica, Vol. 104, No. 1, pp 1-15.
Siegle, D. (s.a.) “Beginning Steps in Developing an Attitude Instrument”, [Online]
.
Turner, J. (1993) “Using Likert Scales in L2 Research”, TESOL Quarterly, Vol. 27, No.4, pp 736–739.
415

Interpretative Research Paradigms: Points of Difference
Nevan Wright and Erwin Losekoot
Auckland University of Technology (AUT) Auckland, New Zealand
nevan.wright@aut.ac.nz
erwin.losekoot@aut.ac.nz
Abstract: This paper outlines the background to the debate surrounding positivisitic and phenomenological research
and the growing consensus of a mixed methods approach. It explains the increasing popularity of interpretative
methods but recognises growing concern about the lack of rigour by some researchers in using these
methods. In the move from positivism to interpretivism the two main interpretive approaches are symbolic interactionism
and hermeneutic phenomenolism, but these terms are not synonymous. It explains symbolic interactionism
and the fact that it is context specific and dependent on the environment in which the research is taking
place, recognising that humans are best understood in relation to their environment. Examples are given of currently
acceptable practices that were once prohibited and vice versa. It then outlines hermeneutic phenomenology’s
origins and why it is so popular with marketing academics and others. The tradition of ‘textual analysis’ now
needs to be expanded to include online and multi-media materials. This paper highlights the differences between
what may be reported happened and what actually did occur. Hermeneutics, it is argued, aims to find out what
happened, but this is always interpreted through the dasein of the researcher. Symbolic interactionism, on the
other hand, focuses not on what happened, but why. The authors warn, however, that one of the most popular
tools of such research, participant observation, may find itself compromised as the presence of the observer may
well influence the events they are observing. In conclusion, the authors recommend that phenomenological researchers
consider using a combination of hermeneutic and symbolic interactionism research in order to not only
understand what actually happened in a social environment, but also what it meant to those experiencing the
event. This cannot ever totally exclude the life world of the researcher from their interpretation of the events they
are studying, but it should at least allow a more accurate representation of what was observed.
Key words: interpretivism, symbolic interactionism, hermeneutic phenomenology, mixed methods
1. Introduction
As academic researchers have started to take a more nuanced view of research and moved from a
positivist perspective to a more intepretivist perspective, it is important to consider the two main interpretive
approaches of symbolic interactionism and hermeneutic phenomenology. After clarifying the
difference between quantitative and qualitative research, this paper considers the advantages of a
mixed methods approach and then outlines the differences between symbolic interactionism and
hermeneutic phenomenology. The paper concludes with a discussion of the implications of these
various approaches for researchers active in the business and management fields.
Phenomenological research, or qualitative research as it is popularly more known, can be regarded
as the opposite of positivistic research (often referred to as quantitative research). Words associated
with positivistic research are quantitative, objective, scientific, experimentalist, and traditional. Proponents
of the positivistic approach assert that as their results can be measured, before and after and
over time (longitudinal) that their research is objective and can be proved. Cresswell (1994) drawing
on a study of the literature of research methods provided a list of assumptions for each approach. A
summary of his findings is:
Quantitative Research
Quantitative research is objective and singular with the researcher being independent from the subject
being researched and is thus unbiased.
The researcher will use an impersonal voice, and by deduction will be able to determine cause and
effect.
Results will be accurate and reliable and will be able to be validated.
The inference being that qualitative research is none of these things: it is NOT objective, and unbiased,
the researcher will NOT be independent, and results will NOT be accurate or reliable. In the title
of a subsequent book Cresswell added the words mixed methods alongside qualitative, and quantitative
for research design, (Cresswell, 2003). In this book he is at pains to extol the virtues of a mixed
methods approach thus recognising the move in research thinking away from a polarised “quant or
416

Nevan Wright and Erwin Losekoot
qual” stance. This polarity between the two approaches was nicely summarised by Smith (1983)... “in
quantitative research facts act to constrain our beliefs: while in interpretative research beliefs determine
what should be counted as facts.” Thus research students in the past were encouraged to make
an ontological assumption, whereby they had to decide whether the world is objective and external to
the researcher, or socially constructed and only understood by examining the actions and perceptions
of the subjects.
In short positivistic research emphasises quantification in the collection and analysis of data, has a
deductive relationship between theory and research and has an objectivist conception of reality. In
addition, quantitative methods generally use standardised measures with predetermined response
categories (Patton, 2010). By comparison qualitative research tends to focus on words rather than
numbers and has an inductive, interpretive, and constructional approach (Husserl, 1936; 1965; Ramberg
& Gjesdal, 2009).
2. Mixed methods
There is some evidence that even in the new millennium some academics such as Bryman and Bell
(2007) see interpretativism as being antipathetic to positivism. Nonetheless the current thinking as
expressed by Gorard (2004) is that mixed method research can lead to “less waste of potentially useful
information” (p.7). In short many researchers now view qualitative and quantitative methods as
complementary rather than antipathetical (Jackson, 1991; 2000; Mingers & Gill, 1996; Thomas, 2003;
Creswell, 2003; and Jack & Raturi, 2006) and that a 'mixed method' enables richness of understanding
of phenomena through confirmation of result, extension of knowledge or by initiating new perspectives
about the subject of research (Bazeley, 2004). The mixed methods approach uses (conjointly)
both a positivistic and an interpretative approach, one feeding off the other. The use of both positivistic
and interpretative methods might be in a single approach design (SAD) or multi approach design
(MAD). With SAD only one analytic interest is pursued but both positivistic and interpretative is used,
for example qualitative research is added to a quantitative study to give added research quality. With
MAD there will be more than one analytic interest and again both methodologies will be used. For further
reading on Mixed Methods see the Journal of Mixed Methods Research (MMR). MMR is a quarterly,
interdisciplinary, international publication that focuses on empirical, methodological, and theoretical
articles about mixed methods research across the social, behavioural, health, and human sciences.
The journal gives its scope as including the delineation of where mixed methods research may
be most effective, illuminating design and procedure issues, and determining the logistics of conducting
mixed methods research.
A philosophical underpinning for a mixed methods approach to research is provided by Tashakkori
and Teddlie (1998) who see the importance of a focussed empirical attention on the research problem
in social science research and of using a pluralistic approach to derive knowledge about the research
issue. Cherryholmes (1992), Murphy (1990) and Cresswell (2003) believe that mixed methods is a
pragmatic approach in so far as there is no reliance on just one method of research but instead draws
liberally from both positivistic and interpretative approaches. In this way it is considered that researchers
have a freedom of choice of method and should look to a variety of approaches for collection
and analysis of data so as to find the best understanding of the research problem. Cresswell goes
onto warn that the researcher has to understand and be able to explain (the rationale) why it is necessary
to mix research methods.
It is relatively easy to envisage when mixed methods might appear to be appropriate. For example a
study into obesity might gather data in terms of percentage of population by sex, age, ethnicity, income,
and employment to identify a segment of the population as having a high proportion of obesity.
The next step could be to seek causes by comparison of diets, exercise etc. All this would be quantitative,
and it might be possible to conclude that people in sedentary work, with a high fat intake and
who do not regularly exercise are at risk of being obese. A positivistic study could stop at this point,
on the grounds that cause and effect have been established and of course can be measured. This
would be a very weak piece of research. Armed with the empirical quantitative data (without the statistics
it would be hard to know where and whom should be questioned/interviewed) a qualitative approach
might through questionnaires /interviews/focus groups try to establish why the obese have a
poor diet and why they are not exercising. The weaknesses in such approaches are that they have
simplified a complex social problem and this simplification will lead to fairly predictable conclusions
and platitudinous recommendations (eat less and exercise more).
417

Nevan Wright and Erwin Losekoot
“When we speak of obesity as a complex problem (and not simply a matter of calories in and calories
out), we allude to the complex interactions of a multitude of societal, psychological, and physiological
variables that do not allow a ready dissection as to cause and effect (are large portion sizes leading to
obesity or do obese people drive the demand for larger portions?)” (Sharma, 2010 p.1).
Thus a mixed method approach where qualitative research is built on positivistic data can lead to a
simplistic conclusion. Likewise at the outset of any research, the framing of the research question will
limit the scope and will lead the researcher down a very narrow and straight path although the richness
and beauty of the research journey could well be along byways or even perhaps in a totally different
direction. In general where a researcher in a mixed methods approach employs quantitative or
empirical methods the starting point will be on what is observable and available with “researchers focusing
primarily on those areas and questions that are amenable to the adherence of empirical methods
of enquiry” (Laverty, 2003 p21). With the use of hypotheses and propositions the direction of the
journey is even more restricted and it becomes a very black and white affair (null hypothesis, or alternative
hypothesis with limited scope for a middle ground). Thus at the end of the research journey the
researcher, although ostensibly a mixed method approach has been used, will have achieved a very
limited result which will add little to their own, let alone the overall, body of knowledge.
3. Interpretative research
The growing popularity of interpretative research methodologies has led to concern that there is not
sufficient understanding of the rigour necessary to ethically utilize them, (Laverty, 2003; Maggs-
Rapport, 2001). Novice researchers often shy away from the statistical approach of positivistic research
as they are frightened of “doing the maths” and see qualitative as being, for them, more easy
to manage. This paper considers the difficulties associated with interpretative research, in particular
with symbolic interactionism and hermeneutic-phenomenolism. Bryman and Bell (2007) claim that
symbolic interactionism and the hermeneutic-phenomenological tradition have a common antipathy
for positivism and both share an interpretative stance. The word antipathy is unfortunate for, as shown
with the above introduction to mixed methods, interpretative and positivistic approaches can work together
with one supporting the other. However, it is agreed that any interpretative approach taken in
isolation will have common ground in being diametrically removed from positivism. Further it is contended
that although both symbolic interactionism and the hermeneutic-phenomenological paradigms
have similar approaches there are significant differences in direction and differences in difficulty of
application.
4. Symbolic interactionism
Blumer (1986) coined the label symbolic interactionism to delineate thinking originated by Dewey
(1925), Mead (1934) and Weber (1920). Mead and Dewey are regarded as influential founders of the
pragmatist movement in sociology. Mead, a philosopher and social activist (for example he was an
active supporter of the suffrage movement), believed that the true test for any theory was if it was
useful in solving complex social problems (Griffin, 2006). George Henry Mead was the son of a Congregational
Minister who was also a Professor of Theology. Over time Mead drifted from his strong
Christian beliefs to become in his words a ‘non believer’ but he never lost his crusading spirit. Mead, a
Harvard graduate, was a great friend of John Dewey. When Dewey gained a Chair at Chicago University
he was influential in Mead being given an appointment at the University. Although Mead is regarded
as an important American philosopher in sociology he never published a monograph, however
he was a prolific writer of papers. His students published ‘Mind Self, and Society from a standpoint of
a Social Behavorist’ in 1934, plus three other volumes, from a compilation of papers, lecture notes
and other unpublished papers of Mead. Mead claimed that people talking together is a significant, if
not the most important, human and humanizing activity that they can engage in. He said talking was a
gesture, but for a gesture to have significance it must have the same meaning for the initiator and the
receiver. He gives the example of a dog barking and that another dog in the vicinity might react by
barking or running away. Mead said dogs do not understand the meaning of the bark, they are merely
responding to a gesture whereas if a person calls out ‘don’t walk’ to someone on a busy street the
meaning of the gesture ‘don’t walk’ is understood by the recipient. According to Mead, “Gestures become
significant symbols when they implicitly arouse in the individual making them, the same responses
which they explicitly arouse or are supposed to arouse, in other individuals.” (p. 47)
Blumer’s seminal summary of this perspective is that people act toward things based on the meaning
those things have for them; and these meanings are derived from social interaction, and conditioned
by their environment. He said that the derived meanings, which are very real to those holding the
418

Nevan Wright and Erwin Losekoot
meanings will be modified through interpretation by an observer. As stated by Dewey (1925, 1934),
human beings are best understood in relation to their environment.
Two key strands of Mead's work, and critical for symbolic interactionism, are the philosophy of pragmatism
and social behaviourism. Mead was concerned with the stimuli of gestures and social objects
with rich meanings rather than bare physical objects which psychological behaviourists considered
stimuli. Mead’s pragmatism as identified and summarised by Ritzer (2008) has four main tenets:
1. For the pragmatists true reality does not exist "out there" in the world, it "is actively created as we
act in and toward the world”.
2. People remember and base their knowledge of the world on what has been useful to them and are
likely to alter what no longer "works".
3. People define the social and physical "objects" they encounter in the world according to their use
for them.
4. If we want to understand why people act (the actors), we must base that understanding on what the
actor actually does.
Three of these ideas are critical to symbolic interactionism:
1. The focus on the interaction between the actor and the world.
2. View both the actor and the world as dynamic processes and not static structures.
3. The ability of the actor to interpret the social world.
Thus, to Mead and other symbolic interactionists, consciousness is not separated from action and
interaction, but is an integral part of both. (Ritzer, 2008 pp347 – 350).
To summarise; Mead postulated that people interact with things (events or situations) based on the
meaning those things have for them; and these meanings are derived and evolve from social interaction
and are modified through interpretation. He attempted to provide an understanding of the internal
meanings or essences of a person’s experience in their life world by description rather than seeking
for explanations of causes (Van Maanen, 1983; Van Manen, 1997).
Hermeneutic-Phenomenology The word hermeneutics is derived from ancient Greece (Hermes, the
messenger). The origin of hermeneutics was in the interpretation of ancient texts, originally scriptural
(exegis) and later the study of ancient and classic cultures. From medieval times hermeneutics included
the study of law and the interpretation of judgements in the context of when and where the
judgement was made with an attempt to take into account social and cultural mores of the times. In
contemporary management research, marketing academics in particular are comfortable with hermeneutic
phenomenology as a research methodology and the term is used for qualitative studies in
which interviews with one or a few people are analyzed and interpreted.
Philosophers whose inspiration is more ontological, such as Heidegger, emphasize the uncovering of
Being from the perspective of the experiencing human being, and how the world is revealed to this
experiencing entity within a realm of things whereas the pragmatist school as epitomized by Mead
concentrate on the development of the self and the objectivity of the world within the social realm, "the
individual mind can exist only in relation to other minds with shared meanings" (Mead, 1934 p 5).
Heidegger's philosophical hermeneutics shifted the focus from interpretation to existential understanding,
which was treated more as a direct, non-mediated, way of being in the world than simply as a
way of knowing (Heidegger, 1927). For example, Heidegger called for a "special hermeneutic of empathy"
to dissolve the classic philosophic issue of "other minds" by putting the issue in the context of
the being-with of human relatedness. Heidegger used the word texts to cover written and spoken expression
and suggested it is a tautology that the written or spoken word cannot be studied using positivistic
numerical methods. In the 21 st century ‘texts’ has expanded to include all forms of multi-media
including the people who produce them. As texts are expressions of the experience of the author, in
the Heidegger tradition interpretation of a text will reveal something about the social context in which it
419

Nevan Wright and Erwin Losekoot
was formed, and more significantly, provide the reader with a means to share the experiences of the
author. The reciprocity between text and context is part of what Heidegger called the hermeneutic
circle (Weber, 1920; Heidegger 1927; Agosta, 2010). Gadamer, a celebrated student of Heidegger,
goes further to assert that methodical contemplation and reflection is the opposite of experience on its
own and that truth comes from understanding and mastering our experience. Gadamer claims experience
is not static but is always changing with hints of further changes. He sees the growth of individual
comprehension as being important. With continued improved, and hopefully enlightened, comprehension
prejudice is a non fixed reflection of our growing comprehension. There are obvious examples
of changes in prejudice over the last 50 years (e.g. legalisation of same sex marriages).
Gadamer sees that being alien to a particular tradition is a condition of understanding and he further
asserts that we can never step outside of our tradition; all we can do is try to understand it. This further
elaborates the continuous nature of the hermeneutic circle (Gadamer 1960; Agosta, 2010)
Heidegger’s hermeneutics is not just a matter of understanding linguistic communication. Nor is it
about providing a methodological basis for research. As far as Heidegger is concerned, hermeneutics
is ontology; it is about the most fundamental conditions of man's being in the world. The hermeneutics
of “facticity”, as he called it, is primarily what philosophy is all about (Heidegger, 1927).
This reflects back on Heidegger's definition of terms such as understanding, interpretation, and assertion.
Understanding, in Heidegger's account, is neither a method of reading nor the outcome of a
carefully conducted procedure of critical reflection. It is not something we consciously do or fail to do,
but something we are. Understanding is a mode of being, and as such it is characteristic of human
being, of Dasein. We have a pragmatic basic intuitive understanding of the world as we see it. This
understanding of our life world is limited by the manner in which we, without consciously thinking and
without theoretical considerations, orient ourselves in the world. Heidegger argues, we do not understand
the world by gathering a collection of neutral facts by which we may reach a set of universal
propositions, laws, or judgments that, to a greater or lesser extent, corresponds to the world as it is,
ergo life world is only our conception of the world. Through the synthesizing activity of understanding,
the world is disclosed as a totality of meaning, a space in which Dasein is at home. Dasein is distinguished
by its self-interpretatory endeavors. Dasein is a being whose being is the issue. Fundamentally
Dasein is embedded in the world and therefore it is not possible to understand ourselves or others
without knowing the world, and the world cannot be understood if Dasein is ignored (Heidegger
1927, Gadamer 1960, Agosta 2010).
5. Implications for business and management research
Contemporary use of the hermeneutic approach has broadened to include business research and is
in particular well accepted as a research methodology by marketing academics. The approach is to
interpret text including oral and written (now expanded to include multi media) and to attempt to relate
text to context. Relating text to context requires a further degree of interpretation which will be limited
by the observer’s understanding of the context. Hermeneutics is looking for what has been reported
as having occurred and to refine what has been reported to what might have really occurred in the
context of the occurrence. Thus there are several levels of interpretation; the interpretation of the
meaning of what has been said, and this interpretation is from the observer’s interpretation of what
the originator of the text was meaning to communicate limited by the observer’s interpretation of the
originator’s life world bounded by the originator’s own life world. Hermeneutics is not looking for the
reason why but is looking for what. The weakness in looking for what happened is that the observer,
no matter how hard they attempt to be unbiased, is limited by their own life world (bounded rationality)
and by their ability to understand the life world of those being observed, and the difficulty they will
have of putting aside and ignoring own experiences and prejudices which invariably will have led to
preconceptions resulting in subconscious bias. One should also bear in mind that the ultimate goal for
any research is to publish findings, and those that read the findings will themselves be interpreting
from their own life world and additionally will have their own preconceptions and biases. The researcher,
in publishing for this audience, will also be mindful of those who will be judging the efficacy
of their report and will be attempting to interpret what will satisfy the judges. This is of particular concern
in the examination of a PhD thesis, where the supervisor will steer the student in a particular direction
based on their own understanding, and who will approach or nominate examiners who they
interpret will be of a similar persuasion to themselves!
Symbolic interaction as an approach to research is used by sociologists in the form of participant observation
to study social interaction and individual actions. By looking for symbols and meanings the
420

Nevan Wright and Erwin Losekoot
objective is more than just relating what has happened to determining causes of phenomenon. Simplistically
this might be seen as Hermeneutics taken one step further with Hermeneutics being used to
interpret what has happened and Symbolic Interaction to interpret the reason why.
Symbols includes the meaning of words. For example the word gay today has a very different meaning
from that of just fifty years ago when a popular colloquialism of the day for a heterosexual unmarried
male was ‘a bachelor gay’ which in turn was derived from the song “A Bachelor Gay” which featured
in the 1 st World War operetta “Maid of the Mountains”. In interpretation of symbols it is important
to establish the context of the time when the symbol was being used. This requires two conditions;
knowing the meaning ascribed to the symbol at the time it was used and knowing the context in which
it was used. When the time the symbol was used and observed is not historical but here and now the
context will be apparent but the meaning might be taken for granted. Take for example a large and
obviously valuable diamond engagement ring. Seen from different perspectives the recipient might
see that it symbolise a life-long engagement and that the size of the diamond symbolises that her betrothed
loves her dearly, for her jealous friend the size of the ring is a flashy symbol of ostentation,
and for the suitor it might symbolise financial extravagance. When interpreting the symbolic meaning
of words written or spoken in the here and now a literal approach might appear to be safe. But, the
problem with literal interpretation is that no matter how careful the writer is in trying to convey a precise
meaning such as may be found in a legal contract drafted by a firm of solicitors to cover any contingency,
the intended precise meaning can be argued by an opposing member of the same legal fraternity.
In short every symbol will have several meanings and it is possible, indeed most likely, for
every symbol to be interpreted by different people in different ways.
The majority of interpretative research involves participant observation. Participant observation is intended
to allow the researcher to access symbols and meanings. It can be argued that close contact
and immersion in the everyday activities of the participants is necessary for understanding the meaning
of actions and for defining situations and context. However as seen with Elton Mayo and the Hawthorn
experiments, the interaction of Mayo constructed the situation and influenced the behaviour of
participants. In any interaction research with close contact, interactions cannot remain completely liberated
of value commitments. In most cases researchers make use of their values in choosing what to
study and their own values are predetermined by their life world. Life world boundaries will always be
present no matter how hard they seek to be objective in how they conduct the research. Findings of
the research, if relying purely on qualitative methods, will be limited by what is deduced from interpretation
of symbols, and by interpretation of context. Bear in mind also that as found by Gadamer comprehension
is not static, and the hermeneutic circle involves a continuous re-evaluation of what we
think we have comprehended. A combination of the Hermeneutic approach and Symbolic Interaction,
will provide a prima facie understanding of what happened and the context of what happened, and will
indicate causes for what happened. Knowing the causes should lead to recommendations as to what
actions would be better in future. The reservation remains however that any conclusion or recommendation
will be based on interpretation of texts and symbols and the researcher’s own life world. If
any action is taken to carry through a recommendation, The action taken will also be limited to an interpretation
of the meaning of the recommendation. Interpretation of the recommendation by the actor
will invariably be limited by the actor’s own life world.
This paper set out to consider some of the different options available to researchers within interpretivism.
In particular, it has considered the differences between symbolic interactionism and hermeneutic
phenomenology. We suggest that a more nuanced understanding of the situations they are studying
will come from a combination of both methods.
References
Agosta, L (2010) Empathy in the context of philosophy UK: Palgrave and MacMillan
Bazeley , P (2004) Issues in mixing qualitative and quantitative approaches to research In B.R. Gadner and L.
Richards (Eds), Applying qualitative methods to marketing management research pp 141-56 UK: Palgrave
MacMillan
Blumer, H. (1986) George Herbert Mead and Human Conduct Walnut Creek, CA: Alta Mira Press
Bryman, A and Bell, E. (2007) Business Research Methodology Oxford: Oxford University Press
Cherryholmes, C.C (1992) Notes on pragmatism and scientific realism Educational researcher 21 pp13-17
Cresswell, J.W (2003) Research Design: Qualitative, Quantitative and Mixed Method Approaches Thousand
Oaks: Sage
Cresswell, J.W (1994) Research Design: Qualitative and Quantitative Approaches Thousand Oaks: Sage
Dewey, J (1925) Experience and Nature New York: Dover
421

Nevan Wright and Erwin Losekoot
Dewey, J (1934) Art as Experience New York: Dover
Gadamer, Han-Georg (1960) Truth and Method (Wahreit und Methode) London: Sheen and Ward
Gorad, G (2004) Combining methods in educational and social research Berkshire: Open University Press
Griffin, E.A (2006) A First Look at Communication Theory Boston: Mc Graw-Hill
Heidegger, M (1927/1962) Sein und Zeit Berlin: Max Niemeyer Verlag Being and Time London: SCM Press
Husserl, E (1936) Crisis of European Sciences and Transcendental Phenomenology
Husserl, E (1965) Phenomenology and the Crisis of Philosophy New York: Harper and Row/Torchbook
Jack ,E.P., and Raturi, A.S (2006) Lessons learned from methodological triangulation in management research
Management Research News 29(6), pp 345-357
Jackson, M.C (1991) Systems Methodology for Management Sciences New York: Plenum
Jackson, M.C (2000) System approaches to management New York: Kluwer/Plenum
Laverty, S.M (2003) Hermeneutic Phenomenology and Phenomenology: A Comparison of Historical and Methodological
Considerations International Journal of Qualitative Methods 2 (3) pp21-35
Maggs-Rapport, F (2001) Best Research Practice:In pursuit of methodological rigor Journal of Advanced Nursing
35(3), pp373-383
Mead, G.H. (1934) Mind, Self and Society from the perception of a Social Behaviorist, edited, with an Introduction,
by Charles W. Morris (Chicago: University of Chicago Press, 1934)
Mingers, J.C. and Gill, A (1996) Multimethodology: Towards a framework of mixing methodologies Omega 25 pp
448-509
Murphy, J.P. (1990) Pragmatism from Pierce to Davidson Boulder: Westview Press
Patton, M.Q. (2010) Development Evaluation: Applying Complexity Concepts to Enhance Innovation and Use
New York: Guildford Press
Ramberg, B and Gjesdal, K, (2009) Hermeneutics, The Stanford Encyclopedia of Philosophy (Summer 2009 Edition),Edward
N. Zalta (ed.)
Ritzer, G (2008) Sociological Theory New York: McGraw-Hill
Sharma, A.M (2010) Obesity is a complex problem which may not have simple solutions , (online)
www.KevMcD.Com accessed 16 Jan 2011.
Smith, J.K (1983) Quantitative and Qualitative Research: An attempt to Classify the Issue Educational Research,
March pp 6 -13
Tashakkori, A and Teddlie, C (1998) Mixed Methodology: Combining qualitative and quantitative approaches
Applied Social Research Methods Vol.46
Thomas, R.M. (2003) Blending qualitative and quantitative research methods in thesis and dissertations Thousand
Oaks: Sage
Van Maanen, J (1983) Qualitative Methodology London: Sage
Van Manen, M (1997) Researching Lived Experience: Human science for an action sensitive pedagogy (2 nd Ed.),
London, Canada :The Althouse Press
Weber, M. (1920) Economy and Society ( Wirtschaft und Gesellschaft, Grundriss der verstehenden Soziologie )
Translated J. Wincklelmann (Ed) Tubingen: J.C.B Mohr (Paul Siebeck) 1956
422

Application of Cluster Analysis and Discriminant Analysis
in Market Segmentation and Prediction
Ruth Yeung 1 and Wallace Yee 2
1
Institute for Tourism Studies, Macao, China
2
Faculty of Business Administration, University of Macau, Macao, China
ruth@ift.edu.mo
wallacey@umac.mo
Abstract: Cluster analysis is commonly used for classifying subjects, but the analytical technique often receives
skepticisms of the way of measuring of similarity and the number of clusters. Despite applying discriminant
analysis can improve target segmentation accuracy; this analytical technique is less adopted. Hence, the
underlying purpose of this paper is to demonstrate how cluster analysis in conjunction with discriminant analysis
can be applied in a multifaceted business field in tourism research as targeting an optimal market segments is
crucial to organization success. With the growth of cross border shopping between the Pearl River Delta Regions
in Greater China after the launch of “one year multiple endorsements” of the Individual Visit Scheme by the
Chinese government, many popular international brands start to set up shops in Hong Kong and Macau to
capture the emerging market. As such, this study adopted hierarchical cluster analysis, followed by K means
cluster analysis to classify cross border shoppers into mutually exclusive groups based on their motivation and
attitudes in the context. Cross tabulation analysis was then conducted to test if there is any association between
the product/service purchased and the cluster membership of respondents. Finally, discriminant analysis was
employed to assess the adequacy of classification, and to determine which variables are the best predictors of
group membership so that the variables can be used to predict new cases of group membership in the context.
To achieve the research purpose, quantitative research design was adopted and data was collected using
intercept method with convenience sampling technique. A total of 194 respondents who normally reside in
Mainland China were recruited, and their motivation and attitudes on cross-border shopping in Hong Kong were
measured. Results of the cluster analysis suggest that there exist three distinct groups of cross-border shoppers
based on the motivational and attitudinal criteria. The attitudes towards product price and quality, agglomeration
of comprehensive retails, together with age, marital status, education, occupation, types of goods purchased, and
frequency of visit are significantly different among the three groups. The cross tabulation analysis reveals that
there was an association between cluster membership of respondents and purchase of high involvement
products such as photographic equipment, and certain food products. Using discriminant analysis, the group
membership of tourists was predicted based on their experience such as comfortable shopping environment,
salesperson’s product knowledge, easy to cross-border and convenient opening hours; and the product attributes
such as product price and quality, and comprehensive agglomeration of retails. This research provides a
quantitative basis for formulating marketing plan and helps the industry to reduce the uncertainty in the decision
making process and increase the probability of success through a systematic and objective analysis. The
practical information can be translatable into operational scheme in terms of targeting, and cross-border shopping
can be promoted in a strategic approach.
Keywords: customers classification, group membership, cross border shopping, motivation, attitudes
1. Introduction
Market segmentation has become a frequently used strategic marketing tool as it helps to break the
entire market into a series of smaller markets according to consumer characteristics. To be effective,
market segmentation cannot be based on a single factor; so, homogenous clusters should be formed
with a combination of variables. Cluster analysis serves this purpose well for classifying a sample of
subjects / objects on the basis of a set of measured variables into a number of different groups with
similar subjects / objects placing in the same group (Cornish, 2007). The purpose of cluster analysis is
indeed to classify entities into mutually exclusive and collectively exhaustive groups with homogeneity
within clusters but heterogeneity among clusters. So, cluster analysis is treated as a common
technique for market segmentation which can be used to identify groups of entities such as people,
markets or organizations; in looking for a better understanding of purchase behaviors by identifying
homogeneous groups of consumers; serves as an alternative to factor analysis and discriminant
analysis; utilized to test market selection or conducted as a general data reduction tool (Punj and
Stewart, 1983). Though cluster analysis is commonly used for classifying subjects, the analytical
technique often receives skepticisms of the way of measuring of similarity and the number of clusters.
To overcome this discrepancy, discriminant analysis can be employed to improve the accuracy of
target segmentation for discriminant analysis is a statistical technique to correctly categorizing
observations or entities into homogeneous groups. It can serve the descriptive purpose to assess the
423

Ruth Yeung and Wallace Yee
adequacy of classification or the predictive one to assign entities to one of the groups of entities.
However, this analytical technique is less likely to be adopted. The possible reason for this, to the
best of our knowledge, most literature appears to focus more on the theoretical foundation of these
two statistical analyses with litter guidance on how to carrying out the technique for marketing
research as well as how these two techniques could complement each other in market segmentation
and prediction.
Hence, this paper aims to demonstrate how cluster analysis in conjunction with discriminant analysis
can be applied in a multifaceted business field in tourism research as targeting an optimal market
segments is crucial to organization success. This study uses Mainland Chinese cross border
shoppers across the Pearl River Delta regions as the context. The first reason for choosing this
context is due to the enormous market potential of Chinese population and the rapid economic growth
has stimulated Chinese outbound travel and spending. Secondly, cross border shopping becomes a
trend under the “one year multiple endorsements” of the Individual Visit Scheme which allows many
Mainland Chinese cross border to Hong Kong or Macau for shopping and consumption as these cities
top their first-choice destinations list. With over 1.3 billion population, it can be understood that
heterogeneity would exist among people in Mainland China. With the growth of cross border shopping
in the Greater Pearl River Delta Regions, many popular international brands start to set up shops in
Hong Kong and Macau to capture the emerging market. A better understanding of the characteristics
and shopping habits of cross border shoppers can help to predict and target potential consumers for
future business.
2. Characteristics of cross border shoppers
Different international market environments and the challenge of moving goods across border make
cross border shopping more complex than domestic shopping (Piron, 2001). The extant literature
states that tourists’ shopping behaviour tends to differ from usual habit when they shop at home town
(Stansfield, 1971). Hence, demographic characteristics that generally used as the basis for
segmenting the domestic market may not be sufficient to categorize cross border segments.
Moreover, using macroeconomic and political criteria for segmentation may overlook the withincountry
heterogeneity of cross border shoppers (Hofstede, et al., 1999).
Miller (2001) suggests that there exists a link between attitudinal criteria and consumer motivations for
shopping across border. A number of researchers comment that the main motivation for cross border
shopping builds on the economic advantage of product price, perceived product quality, variety of
product selections, store facilities, and service provided by salesperson in the cross border region
(e.g.Evans et al., 1992; Blakney and Sekely, 1994; Wang, 2004; Lau, Sin and Chan, 2005; Ghaddar
and Brown, 2005; Asplund et al., 2007; Yeung and Yee, 2011). Other studies have identified a range
of attributes which include attitudinal criteria, such as attractiveness of the shopping areas,
convenience accessibility, product assortment and service provided by salesperson, and
socioeconomic characteristics, such as age, income, education, and occupation in market
segmentation of out-shoppers (e.g. Papadopoulos, 1980; Chatterjee, 1991; Piron, 2002). The set of
criteria have been tested in the study of market segments of domestic shoppers against cross border
shoppers in Europe (Dmitrovic and Vida, 2007).
Moreover, the types of goods purchased by tourists varied (Timothy and Butler, 1995; Di Matteo and
Di Matteo, 1996; Wang, 2004, Yang, 2006) and a relationship between behaviour and the
environmental setting for shopping tourism is found due to differences in goods, a wide range of
specialized and traditional goods, such as tea, dairy products, cloths, leather goods, electronic goods,
jewelry and so forth, and the distinguishing feature of the shopping area as well as differences in
opening hours. These environmental setting provides a unique leisure and shopping experience for
tourists (Jansen-Verbeke, 1991; Lord et al., 2002). The freedom to cross the border, contrast between
home and neighbouring markets, as well as the awareness of purchase opportunities also contribute
to the willingness to shop cross border (Lord, Putrevu and Shi, 2008; Pico, 2009; Leal, Lopez-laborda
and Rodrigo, 2010).
3. Development of segmentation model
The research adapted a quantitative approach using cluster analysis to segment the tourists
according to their motivation and attitudes towards cross border shopping and subsequently used
discriminant analysis to predict visitors’ group membership based on their motivation and attitudes.
424

3.1 Questionnaire design
Ruth Yeung and Wallace Yee
As cluster analysis has no mechanism for differentiating between relevant and irrelevant variables, the
choice of variables must be underpinned by conceptual considerations (Cornish, 2007). Hence,
measures chosen for this study should represent tourists’ motivation to shop across border that is
relevant to the environmental factors exhibited in the Pearl River Delta regions. Focusing on opinion
on cross border shopping in Hong Kong from people residing in Mainland China, a questionnaire with
a set of ten variables using 7-points scales were developed following a review of extant literature (e.g.
Papadopoulos, 1980; Jansen-Verbeke, 1991; Evans et al., 1992; Wang, 2004; Yang, 2006; Dmitrovic
and Vida, 2007; Asplund et al., 2007; Dmitrovic and Vida, 2007; Lord, Putrevu and Shi, 2008)
(Appendix).
3.2 Data collection
In order to achieve the research aim, an intercept survey was carried out in various tourist spots at
different time of the day and different days of the week to minimize the over-representation of the
personal characteristics of respondents (Casley and Kumar, 1992). The survey was conducted by
trained interviewers who fully understood the issues relating to the survey. Respondents were
requested to answer a self-administered questionnaire which was pre-tested to assure its
functionality. A total of 194 usable questionnaires were collected, of which, 80 were male, 128 from 21
to 40 years of age, 109 were single, 72 were white-collar, 45 were students and 129 with an annual
income below HKD8,000. Majority of respondents (74.7%) shopped across the border to Hong Kong
from Mainland China for less than once per two month in average in the past year.
3.3 Identify the number of clusters - hierarchical cluster analysis
Hierarchical cluster analysis was first performed to identify the number of segments that were
relatively homogeneous in the importance they attached to the ten motivational/attitudinal variables.
The technique used an algorithm that starts with each case in a separate cluster and combines
clusters until only one is left. Proximities procedure was used to generate similarity measures. The
between groups linkage out of the alternatives was selected so that an observation was joined to a
cluster if it has a similarity with at least one of the members of that cluster. Base on the coefficients
column (indicating the distance between the two cases or clusters joined at each stage) of the
agglomeration schedule which displays the cases or clusters combined at each stage, the number of
cluster was set if there exists a sudden jump in the distance coefficient when read down the column.
As shown in Table 1, the sudden jump occurred at stage 191 indicating that it was the optimal
stopping point for merging clusters and suggesting three clusters.
Table 1: Agglomeration schedule
Stage Cluster First
Stage Cluster Combined
Coefficient
Appears
Cluster 1 Cluster 2 s Cluster 1 Cluster 2 Next Stage
1 31 35 .000 0 0 13
2 16 34 .000 0 0 3
3
.
.
.
11 16 .000 0 2 28
187 1 72 60.570 186 176 190
188 155 192 61.000 0 180 193
189 55 166 64.667 185 182 190
190 1 55 68.989 187 189 191
191 1 117 77.534 190 0 192
192 1 136 80.158 191 0 193
193 1 155 93.567 192 188 0
3.4 Identify the cluster membership - K-mean cluster analysis
K-means cluster analysis was then conducted to identify the cluster membership by setting the
number of clusters to three which was obtained from the hierarchical cluster analysis. By specifying
the number of clusters equals three, the Euclidean distances between Final Cluster Centers shows
425

Ruth Yeung and Wallace Yee
that the distance between cluster 1 and cluster 2, cluster 1 and cluster 3, and cluster 2 and cluster 3
were 3.843, 3.883 and 4.361 respectively (minimum F statistic = 15.05; d.f. = 2, 194, p-value < 0.001)
indicating that the clusters were significantly different from each other (Kaufman and Rousseeuw,
1990; Gong and Richman, 1995). The result reveals that the group size of the three segments were
55 respondents (27.9% of total), 80 (40.6%) and 62 (31.5%) for Group 1, Group 2 and Group 3
respectively.
3.5 Comparison of motivation and attitudes among the clusters – ANOVA analysis
The mean of total and individual groups, as well as standard deviations for the independent variables
were calculated. The result of one-sample t-test with test value of 4 shows that respondents have
motivation and positive attitudes towards cross border shopping except “there are bargain hunting
opportunities in Hong Kong” with a mean score of 3.54 (t-value = 4.893, p-value < 0.001) (Table 2).
Table 2: Mean of motivation and attitudes towards cross border shopping
Total Group 1 Group 2 Group 3
Mean S. D. t-value* p-value* Mean S. D. Mean S. D. Mean S. D.
Price 4.55 1.712 12.619 .000 5.04 1.440 3.20 1.441 5.59 1.214
Quality 5.51 1.256 27.779 .000 6.37 .714 4.71 1.177 5.45 1.202
Variety 4.25 1.667 10.425 .000 5.34 1.601 3.68 1.118 3.66 1.671
Service 4.98 1.388 19.860 .000 5.94 1.057 4.91 1.040 3.95 1.330
Comfortable 5.01 1.238 22.611 .000 5.94 .967 4.86 .862 4.12 1.171
Opening hours 4.78 1.375 18.066 .000 5.48 1.223 4.83 1.124 3.93 1.362
Bargain
hunting
3.54 1.541 4.893 .000 4.09 1.545 3.51 1.491 2.95 1.382
Custom
procedure
4.92 1.545 17.287 .000 5.82 1.278 5.07 1.276 3.69 1.314
Trend 4.29 1.560 11.504 .000 4.75 1.700 4.10 1.395 3.98 1.481
Agglomeration 5.14 1.335 22.373 .000 5.99 .945 4.55 1.243 4.88 1.352
*Test value = 4
A one-way analysis of variance (ANOVA) was employed to compare visitors’ motivation and attitudes
towards cross border shopping among the three segments. As shown in Table 3, price is the most
important factor attributable to different clusters (mean square = 170.13), followed by service, variety,
comfortable etc. Similarly, large F ratios indicate variables that are important for separating clusters (F
ratio of Price = 143.957, d.f. = 2, 191). However, the K-means clustering algorithm is designed to
minimize within-cluster variability; the F statistics cannot be interpreted as in a traditional ANOVA. So,
the significance value reported is not a reliable estimate of the probability associated with the
hypothesis of no effect for a particular variable. In other words, the F tests should be used just for
descriptive purposes because the clusters have been chosen to maximize the differences among
cases in different clusters only.
Table 3: Comparison of attitudes towards cross border shopping among the three clusters
Cluster Error
Mean
Mean
F p-value
Square df Square df
Price 170.130 2 1.182 191 143.957 .000
Service 68.129 2 1.234 191 55.219 .000
Variety 61.879 2 2.159 191 28.661 .000
Comfortable 54.352 2 .980 191 55.433 .000
Quality 44.846 2 1.125 191 39.876 .000
Opening hours 42.894 2 1.461 191 29.352 .000
Agglomeration 39.682 2 1.385 191 28.645 .000
Custom procedure 19.944 2 2.203 191 9.053 .000
Bargain hunting 18.231 2 2.208 191 8.257 .000
Trend 9.614 2 2.359 191 4.075 .018
426

Ruth Yeung and Wallace Yee
3.6 Association between products purchased and cluster membership - cross
tabulation analysis
Cross tabulation analysis was conducted to test if there is any association between the
product/service purchased and the cluster membership of the three segments. Cramer's V which is a
measure of association based on chi-square was calculated. The results obtained from the cross
tabulation analysis reveal that there was an association between purchase of Photographic Goods,
Audio-Visual Equipment, Electronic Products, Food products/Dairy products, and tea and cluster
membership of respondents (Nominal by Nominal Cramer's V = 0.017, 0.017, 0.046, 0.041 and 0.025
respectively). Each of the three segments was then labeled as leisure buyers (Group 1), planned
buyers (Group 2) and infrequent buyers (Group 3) according to their purchase pattern.
The characteristics and the purchase pattern of the group members were compared and summarized
in Table 4.
Table 4: Characteristics and purchase pattern of the three segments
Group Motivation and
attitude
1
leisure
buyers
2
planned
buyers
3
infrequent
buyers
usually scored high in
every aspect of
motivation and
attitudes except “low
product price in Hong
Kong is low”.
scored in-between the
other groups except
lowest in “Product
price in Hong Kong is
low”, “Quality” and
“Agglomeration”.
scored low except
highest in “Product
price in Hong Kong is
low”; “Quality” and
“Agglomeration” inbetween
Demographic Visit
Frequency
relatively young < 30
years (68.6%), mostly
single (65.7%) , an
undergraduate degree
(55.2%), Income
$8,000-15,000 (23.9%)
relatively older > 40
years (23.2%), married
/with children (29%),
income < $8,000 (71%)
laborer (28.9%).
mostly young adults
between 21 and 40
(67.2%), highly
educated with
postgraduate degree
(17.2%), relatively high
income level >$15,000
(13.9%)
* Above the average percentage
4. Development of prediction model
once a
week
twice per
months
< once per
two months
Product purchase Group
Size
leather goods (8%*)
photographic goods
(11%*) audio-visual
equipment (8%*)
jewelry/watch (11%)
personal care
products (6%*) food
and/or dairy
products (12%*),
tea (4%*)
Books/Magazines
(22.4%*)
Medicine/Chinese
Herbal Medicines
(31%*)
Beauty services /
Spa (13.8%*)
67
samples
(34.5%)
69
samples
(35.6%)
58
samples
(29.9%)
4.1 Association between the three segments and the discriminant functions -
discriminant analysis
For the discriminant analysis, the group membership, i.e. leisure buyers (Group 1), planned buyers
(Group 2) and infrequent buyers (Group 3) was treated as dependent variable and the observed
characteristics – motivational/attitudinal variables were treated as predictor variables. Prior to carrying
out the analysis, the sampling distribution of any linear combination of the independent variables was
checked for normality and absence of multicollinearity and singularity. Discriminant analysis was then
performed in order to build a predictive model of group membership based on the observed
characteristics of cross border shopping. The procedure generates a set of discriminant functions
based on linear combinations of the predictor variables that provide the best discrimination between
the segments (Tabacknick and Fidell, 2007).
427

4.2 Canonical discriminant functions
Ruth Yeung and Wallace Yee
The two significant Canonical discriminant functions, namely “cross border shopping experience” and
“product offering” obtained were significantly differed from each other at 5% level (p-value < 0.001).
Furthermore, the Canonical variables “cross border shopping experience” and “product offering”
accounted for 64.8% and 35.2% of the between group (explained) variance respectively.
4.3 Test of group mean difference – Wilks' lambda
The difference between the mean of the functions were tested using Wilks' lambda, the proportion of
the total variance in the discriminant scores not explained by differences among the groups. Wilks'
lambda obtained for the functions was 0.143 (p-value < 0.001) indicating the group means were
different.
4.4 Prediction of group membership for new cases
The correlation (loading) between the variables and the discriminant function was shown in the
structure matrix which indicates the usefulness of each variable in the discriminant function (Table 5).
For each variable, an asterisk marks its largest absolute correlation with one of the canonical
functions. By convention, loading in excess of 0.33 (i.e. 10% of variance) is considered as eligible. As
shown in Table 5, “The shopping environment in Hong Kong is comfortable” has the strongest
correlation (0.490) with Function 1“cross border shopping experience”, while “Product price in Hong
Kong is low” has the strongest correlation (0.662) with Function 2 “product offering”.
To predict the membership of new cases of cross border shopper, their motivational and attitudinal
scores were fed into the three classification functions (Table 5) to obtain the classification scores
using Equation 1 shown below.
Classification score of Group i = Constant for Group i + ∑Coefficients of Xij * Sij (Equation 1)
Where i ranges from 1 to 3,
j ranges from 1 to 10,
Xij denotes Variable Xj (i.e. Comfortable, Service,…, Agglomeration) of Group i, and
Sij denotes the score of the respective Variable Xj of Group i
Group membership of the new cross border shopper was then assigned based on the highest
classification score calculated.
Table 5: Structure matrix and classification function coefficients
Structure matrix
Classification Function Coefficients
Function1 Function2 Group 1 Group 2 Group 3
CROSS BORDER
SHOPPING
EXPERIENCE
Comfortable .490(*) .148 2.964 2.752 1.399
Service .473(*) .080 2.534 2.357 1.213
Custom procedure .453(*) -.044 5.113 4.356 3.218
Opening hours .342(*) .005 1.672 1.610 1.046
Variety .300(*) .290 3.461 2.537 2.320
Bargain hunting .209(*) .033 2.754 2.285 1.973
Trend
PRODUCT
OFFERING
.132(*) .095 1.393 1.320 1.119
Price -.138 .662(*) 2.448 1.427 2.949
Quality .216 .539(*) 2.999 1.830 3.311
Agglomeration .238 .364(*) 1.880 1.157 1.909
Constant -76.431 -49.115 -45.661
*The largest absolute correlation with one of the canonical functions and ordered by absolute size of
correlation within function.
428

5. Discussion and conclusion
Ruth Yeung and Wallace Yee
This study achieves the aim to demonstrate how cluster analysis in conjunction with discriminant
analysis can be applied. Using cross border shopping as illustration, the study of 194 Chinese
conducted cross border shopping to Hong Kong exhibits that cluster and discriminant analyses using
the ten relative important shopper attitudes towards cross border shopping as clustering variables can
be effectively applied to the segmentation of cross border tourism markets. The step by step
procedure presents how the three distinct homogeneous segments were identified, with each
segment possessing a unique group profile on the basis of the two discriminant functions, namely
“cross border shopping experience” and “product offering”.
Results of cluster analysis show that the attitudes towards product price and quality, agglomeration of
comprehensive retails, together with age, marital status, education, occupation, types of goods
purchased and frequency of visit are significantly different among the three groups. The cross
tabulation analysis reveals that there was an association between cluster membership of respondents
and purchase of high involvement products such as photographic equipment, and certain food
products. Group 1 tends to enjoy the cross border shopping experience and views it as a leisure
activity. Group 2 has indifferent view on both cross border shopping experience and the product
offerings while Group 3 is likely to be concerned about the product offerings and considers shopping
for essential goods and views it as a laborious activity. The group membership of tourists was
predicted based on their experience such as comfortable shopping environment, salesperson’s
product knowledge, easy to cross-border and convenient opening hours; and the product offering
such as product price and quality, and comprehensive agglomeration of retails.
The main advantage of complementary use of cluster analysis and discriminant analysis is that a
predictive model of group membership could be developed through a set of discriminant functions
which were derived by linking the cluster group membership with the motivational /attitudinal
variables. These motivational / attitudinal profiles were meaningful and distinctive enough to offer the
cross border shopping tourism marketer actionable and practical information which can be
translatable into operational scheme in terms of targeting and promoting cross border shopping in a
strategic approach.
Though with small sample size and non-randomly recruited sample, the study testifies that both
cluster and discriminant analysis are invaluable and effective analytical techniques to profiling cross
border shoppers segments in such a way that the importance of tourism destination shopping
environment could be attributed to specific motivational / attitudinal orientation. The structure matrix
and classification function coefficients provide an algorithm to assigning membership for new cases of
cross border shopper. The application of the two analytical techniques can be extended to other study
contexts and sectors.
6. Appendix 1
Variable Explanation
Price Product price in Hong Kong is low
Quality Product bought from Hong Kong with high quality
Variety Products / services in Hong Kong are not available in home town
Service Hong Kong sales people provide good services
Comfortable The shopping environment in Hong Kong is comfortable
Opening hours Opening hours of the shops at Hong Kong are convenient
Bargain hunting There are bargain hunting opportunities in Hong Kong
Custom procedure Border crossing procedures to Hong Kong is simple
Trend Follow the current trend in travelling to Hong Kong for cross border
shopping
Agglomeration Agglomeration of comprehensive retails in Hong Kong
References
Asplund, M., Friberg, R. and Wilander, F. (2007), Demand and distance: Evidence on cross border shopping,
Journal of Public Economics, Vol. 91 pp.141–157.
Blakney, V. and Sekely, W. (1994), Retail attributes: influence on shopping mode behaviour, Journal of
Managerial, Vol. 6 No. 1, pp. 101–118.
429

Ruth Yeung and Wallace Yee
Chatterjee, A. (1991), “Cross border shopping: searching for a solution”, Canadian Business Review, Vol. 18 No.
4, pp. 26-9.
Cornish, R. (2007), Statistics: Cluster Analysis, Mathematics Learning Support Centre.
Di Matteo, L. & Di Matteo, R. (1996), An analysis of Canadian cross border travel, Annals of Tourism Research,
Vol. 23 No. 1, pp. 103-122.
Dmitrovic, T. and Vida, I. (2007), An examination of cross border shopping behaviour in South-East Europe,
European Journal of Marketing, Vol. 41 No. 3, pp. 382-95.
Evans, W., Lane, H. W., O'Grady, S. and Hildebrand, T. (1992), Border crossings: doing business in the US,
Scarborough, Ont: Prentice-Hall, Canada.
Fitzgerald, J., Quinn, T. P., Whelan, B. J. and Williams, J. A. (1988), An Analysis of Cross border Shopping, The
Economic and Social Research Institute, Dublin.
Ghaddar, S. and Brown, C. (2005), The cross border Mexican shopper: a profile, Research Review, Vol. 12 No.
2, pp.46–50.
Gong, X. and Richman, M. B. (1995), On the Application of Cluster Analysis to Growing Season Precipitation
Data in North America East of the Rockies. Journal of Climate, Vol. 8, No. 4, pp. 897-931.
Helsen, K., Jedidi, K. and Desarbo, W.S. (1993), A new approach to country segmentation utilizing multinational
diffusion patterns, Journal of Marketing, Vol. 57 No. 4, pp. 1-17.
HKTB, (2011), 2010 Total tourism spending reaches all-time high of over hk$200 billion - Tourism Expenditure
Associated to Inbound Tourism, Hong Kong Tourism Board, 3 April, 2011 available at
http://tw.partnernet.hktb.com/pnweb/jsp/doc/listDoc.jsp?doc_id=137260 (accessed on 6/1/11)
Hofstede, F., Steenkamp, J. E. M. and Wedel, M. (1999), International market segmentation based on consumerproduct
relations, Journal of Marketing Research, Vol. 36 No. 1, pp. 1-17.
Jansen-Verbeke, M. (1991), Leisure shopping: a magic concept for the tourism industry. Tourism Management,
12, 9–14.
Kaufman, L. and Rousseeuw, P. J. (1990), Finding groups in data: an introduction to cluster analysis, Wiley
Online Library.
Lau, H., Leo, Y., Sin, L. Y. and Chan, K. K. (2005), Chinese Cross Border Shopping: An Empirical Study, Journal
of Hospitality & Tourism Research, Vol. 29, pp. 110-133.
Leal, A., Julio López-Laborda, J. and Rodrigo, F. (2010), Cross Border Shopping: A Survey, Int Adv Econ Res,
Vol. 16, pp. 135–148.
Lord, K. R., Putrevu, S. and Shi. Y. Z. (2008), Cultural influences on cross border vacationing, Journal of
Business Research, Vol. 61, pp. 183–190.
Mares, N. (1990), Impacts of the single European market on the spatial structure of the Netherlands and the
relation between the Netherlands and the Federal Republic of Germany, Institut f .ur Raumplanung
Universit.at, Dortmund.
Miller, N.J. (2001), Contributions of social capital theory in predicting rural community in-shopping behaviour, The
Journal of Socio-Economics, Vol. 30, pp. 475-93.
Papadopoulos, N.G. and Heslop, L.A. (1993), Product Country Images: Impact and Their Role in International
Marketing, International Business Press, New York, NY.
Papadopoulos, N.G. (1980), Consumer out-shopping research: review and extension, Journal of Retailing, Vol.
56 Winter, pp. 41-58.
Picó, M. B. (2009), Shoppers without borders: cross border shopping is big business around the world, Retailing
Today, December, 2009, pp. 15-16.
Piron, F. (2001), “International retail leakages: Singaporeans outshopping in Malaysia”, Singapore Management
Review, Vol. 23 No. 1, pp. 35-58.
Piron, F. (2002), International outshopping and ethnocentrism, European Journal of Marketing, Vol. 36 No 1/2,
pp. 189-210.
Sethi, V. and King, W. R. (1994), Development of measures to assess the extent to which an information
technology application provides competitive advantage, Management Science, Vol. 40 No. 12, pp. 1601-27.
Stansfield, C. A. (1971), The Nature of Seafront Development and Social Status of Seaside Resorts, Society and
Leisure, Vol. 4, pp.117-141.
Tabachnick B G & Fidell L S (2007), Using Multivariate Statistics, Harper & Row, Publishers, Inc, New York.
Timothy, D. J., & Butler, R. W. (1995), Cross border shopping: a North American perspective, Annals of Tourism
Research, Vol. 22 No. 1, pp. 16–34.
Wang, D. (2004), Hong Kongers’ cross border consumption and shopping in Shenzhen: patterns and motivations,
Journal of Retailing and Consumer Services, Vol. 11, pp. 149–159.
Yang, C. (2006), The Pearl River Delta and Hong Kong: an evolving cross-boundary region under ‘‘one country,
two systems’’, Habitat International, Vol. 30, pp. 61–86.
Yeung, R. M. W. and Yee, W. M. S. (2011), “Cross border shopping: stimulant factors for crossing from China to
Macao”. Proceedings of the Academy of Marketing Conference 2011: Marketing Field Forever, 5-7 July,
Academy of Marketing, Liverpool, UK.
430

PhD
Research
Papers
431

432

Application of Delphi Expert Panel in Joint Venture
Projects in Malaysian Construction Industry
Hamimah Adnan, Azizan Supardi, Zul Zakiyuddin Ahmad Rashid, and Norazian
Mohamad Yusuwan
Universiti Teknologi MARA, Shah Alam, Malaysia
mimad856@gmail.com
zzan659@gmail.com
zakiphd@gmail.com
azianyusuwan@gmail.com
Abstract: This paper discusses the application of the Delphi Technique to validate the findings from previously
administered questionnaire surveys and in-depth interviews in joint venture (JV) projects in Malaysia. The aim of
the Delphi technique is to achieve consensus amongst the selected expert panellists in Malaysia on the primary
success factors in JV projects. Essentially, it concerns using selected expert panels to assist in gathering data
and information to achieve research objectives by designing a progressive series of questions to which the
selected panel of experts responds. The rationale for using Delphi, its benefits, limitations and recommendations
for its use are provided. The literature review identified twenty-one factors as critical to the success of joint
venture projects. A questionnaire survey and forty three in-depth interviews were conducted to obtain detailed
information from the contractors. The Delphi Panel confirmed that inter-partner trust, mutual understanding,
criteria for partner selection, agreement of contract, compatibility of objectives, conflict and commitment are the
critical success factors while fourteen factors were identified as Failure Reduction Criteria. The application of
Delphi Techniques helped to identify the primary factors which will be useful in driving the successful application
of joint venture arrangements in Malaysian construction projects.
Keywords: Delphi expert panel, joint venture projects, Malaysia
1. Introduction
A joint venture is a procedure for handling specific phenomena such as access to new markets,
competition, government policy, or economies of scale. Careful analysis of the economic, political,
social and cultural environments within which the venture will be implemented and managed should
assess the feasibility and the desirability of a JV. A planned approach to JV necessitates a thorough
and careful evaluation of these aspects by both partners to ensure successful implementation.
Construction JV’s in Malaysia are become increasingly popular both with multinational constructions
and local governments to achieve specific objectives. The contractors need to know the critical factors
if they want their JVs to be successful. The aim of this paper is to report how the Delphi Technique
was used to validate and confirm a list of factors that was considered critical from the literature
review, questionnaire survey and interviews on the critical success factors on construction JV projects
in Malaysia. Overall, it aims to reach a consensus on the most critical success factors, confirmed by
expert panellists, as the final findings. The effectiveness of the Delphi method will be evaluated and
the difficulties in conducting a Delphi survey will be discussed.
2. Problem statement
Changes in Malaysian regulations that allow Bumiputera Contractor Class ‘A’ to qualify in construction
projects are influenced by the growth in international cooperative behaviour between companies.
Since the 1990’s, the presence of foreign contractors in Malaysian construction projects has been
growing. Similarly, Bumiputera Contractors are keen to participate in joint ventures because of the
nation’s political stability, economic growth, and relatively low cost of labour and resources. The
environment, conducive to IJVs, provides opportunities for foreign firms to participate in joint ventures
projects with Malaysian firms and Bumiputera Contractors. The Malaysian government is encouraging
and supporting local contractors to participate in regional and global markets based on their expertise
and experience in construction of buildings, infrastructure projects, highways, power generation, ports
and airports.
Foreign contractors have been operating in Malaysia long before the signing of the GATS accord in
January 1995. During that time, foreign firms were admitted mostly in a non discriminate manner, in
the absence of any rigid trade related sanctions on their 100% foreign-owned firms that were allowed
to operate freely in the Malaysian domestic market. With the implementation of GATS, market access
restrictions promulgated by Malaysian regulators have opened up opportunities for foreign firms to
433

Hamimah Adnan et al.
operate through joint ventures with Malaysian individuals or institutions, as local as well as foreign
contractors.
The Malaysian market is becoming smaller for the growing number of companies involved in
construction inasmuch that efforts must be made to strategise, consolidate and prepare for exit and
replacement. Foreign firms are often required to bid on large infrastructure projects in JV with local
partners as JV bids require at least 30% Bumiputera (indigenous Malay) participation. Malaysia’s
open-door policy to foreign participation is evidenced by the amount of Malaysian payments for
contract and professional services.
Problems and difficulties in JVs should be identified because these can cause instability and problems
during the implementation and operation of foreign JV projects. Besides, due to the inherent
challenges in establishing trust and partnering success factors, strategies for competitiveness in
international construction have to be adopted. Additionally, natural barriers such as language, culture
and local standards can act as obstacles in JV projects. Crossing cultures can lead to culture shock
which may impact the organization, work and co-ordination of projects. Joint venture projects are
subject to various challenges: technological, financial, legal, equity control, profit margin, size
compatibility, socio-cultural and political. Hence, a planned approach by both partners encompassing
these aspects is a requisite to ensure successful implementation of joint ventures.
Various behavioural, cultural and administrative contexts make the management of a joint venture
project a demanding task in terms of time, effort and management. Hence, such ventures need
careful identification and evaluation of potential problems and challenges. This study attempts to
identify the problems and challenges for the successful implementation of JV construction projects
involving the indigenous Bumiputera Class ‘A’ Contractors in Selangor as well as local and foreign
contractors.
3. Overview of construction joint ventures
The formation of joint ventures in Malaysia has increased at an unexpected rate over the last two
decades. Within the context of developing nations, joint ventures have been predominantly preferred
by multinationals over other types of strategic alliances (Bower and Smith 2004). However, a critical
review carried out by Bygballe et al (2010) highlighted that the actual practice of partnering
contradicts the original intention of the CII, which is “a long-term commitment by two or more
organizations for the purpose of achieving specific business objectives by maximising the
effectiveness of each participant’s resources”. Besides, there has been relatively little empirical
evidence on the success factors associated with construction joint ventures in the Malaysian
construction industry (Adnan and Morledge 2003).
Moreover, the 1998 economic crisis adversely affected the performance of Bumiputera Contractors.
Although there has been an increase in the number of Bumiputera Contractors business ventures in
the modern and dynamic sectors of the economy, a large proportion was concentrated in the
construction sector and the lower-end service activities such as gas distribution and petrol kiosk
operations. The Malaysian infrastructure construction and engineering industry has delivered
impressive results with many projects being delivered through project-based joint ventures between
‘local’ and ‘foreign’ construction contractors (Walker and Johannes 2001)
Mohammad and Torrance (2004) defined joint ventures as two or more parties that jointly execute a
commercial enterprise which may take the form of a consortium, partnership or joint venture
corporation in which the outcomes of success or loss is jointly held by the joint venture. Gale and Luo
(2003) stated that joint venture (JV) project based companies exist both as a mode of foreign
investment and as means of technology transfer in Multinational Corporations.
Kumarasmawy (2000) stated that the increasing magnitude, complexities and risks associated with
major construction projects have brought together organizations with diverse strengths and
weaknesses to form joint ventures to collectively bid and execute projects. International joint ventures
(IJVs) in construction often face a highly complex and dynamic environment as, in most instances,
they are formed to build large scale engineering projects. One of the primary requirements of actors in
such systems is to reduce the environmental complexity (Girmscheid and Brockman 2000)
434

Hamimah Adnan et al.
Jeffrey (2005) mentioned the determinants of firms’ use of explicit call options to acquire equity in their
international joint ventures (IJVs). Such options are an important contractual element of IJVs because
they allow a firm to secure a claim on future expansion opportunities and to safeguard itself against
various exchange hazards. He draws on real options and transaction cost arguments, respectively, to
develop hypotheses on the circumstances under which firms use such options.
Ravinder, (2006) revealed that for IJVs between firms from diverse cultures, partner related factors
are of importance for their successful management. Since Singapore and China share a common
Confucian culture, it was expected that a Singaporean firm's approach to joint venture formation in
India will differ significantly from its approach to joint ventures in China. The results, however, suggest
that the partner selection process follows a different logic in Confucian societies, and Singaporean
firms were as successful in forming IJVs in India as in China.
Turner (2009) distinguished success criteria as “the dependent variables by which we will judge the
successful outcome of the project”, or how to realize the predefined project objectives in IJVs. It
should consider the performance achieved through the various management strategies of IJVs.
Ozorhon et al. (2010) highlighted that performance of IJV management was based on their
management control at different levels such as strategic control at board of directors level, operational
control at general management level, as well as organizational control imposed by the partners in
forming the venture’s organizational structure, processes, and operating routines. The role of the
CEO/GM is highly critical at the first level, the effective management of the IJV, as this is the focal
point for the maintenance of relationships with each parent (Petrovic et al. 2006). Subsequently,
operational control at general management level should focus on proper staffing and technical
requirements of the IJV (Petrovic and Kakabadse 2003). The last level focuses on control by the
ventures.
4. Success factor in the joint ventures
Adnan (2005) stated that agreement of contract terms is critical for the successful implementation of
joint venture projects. Good JV agreement drafted with clear terms that are easily understood by all
partners as well as the working staff is an essential success factor and can prevent conflicts in future
joint venture operations. It has been found to be an important factor in preventing conflicts in large
projects.
Gale and Luo (2004) suggest that technology transfer is the key objective of entering into JVs and
that the JV will be regarded as successful if the parent firm learns from its partner about technology
and management know-how. Adnan et. al (2003) reported that commitment is important in a long term
JV project to ensure the provision of the resources and capabilities necessary for its success. She
added that the JV’s partner knowledge of joint venturing is a key determinant of success because
having gone through the JV formation, the partners are aware of issues and what is required to make
the project a success. Partner experience increases the chances of joint venture success. Financial
stability in JV is an important consideration because it impacts the degree of project success. It
determines whether projects are funded, the type of funding raised, and the cost of funds. The
project’s feasibility is a preliminary opportunity assessment that determines whether the project in
question has potential for success. It involves the screening of a JV partner and brings together
several considerations in deciding to proceed to the business.
Ogunlana (2005) stressed that competence is a prerequisite for the success of construction projects.
The component includes utilization of up-to date technology, proper emphasis on past experience,
technology, multidisciplinary/component project team, and awarding bids to the right contractor.
Effective communication has been gaining increasing importance in today’s information age. It is a
central factor in leading and integrating people and making decisions to create a successful project.
As international JV involves parties from different cultures, effective communication is essential for
ensuring that project teams have shared understanding and values.
Adnan (2005) suggests that the top 5 - ranked critical success factors in Malaysian construction joint
ventures are agreement of contract, commitment, cooperation, management control and inter-partner
trust.
435

5. Objective of the research
Hamimah Adnan et al.
The objective of the research is to identify the primary factors which support the successful
application of joint venture arrangements in construction projects in Malaysia. The basic research
question concerns the identification of these factors by the adoption of a robust and replicable
methodology and the ranking, weighting and criticality of these factors. The results of this study are
expected to provide useful guidelines for forming and operating effective and efficient JVs both in
Malaysia and in other similar economies.
6. Literature on Delphi background
The name Delphi was taken from the Greek Oracle’s skills of interpretation and foresight. The Delphi
was developed in the 1950’s by the Rand Corporation for the US Air Force involving the use of expert
opinion (Robinson 1991). The procedure was designed to obtain the most reliable consensus of
opinion of a group of experts by a series of intensive questionnaires interspersed with controlled
feedback, with the results of each round being fed into the next round (Linstone and Turoff 1975). It
involves the selection of procedures for suitable experts, development of appropriate questions to be
put to them and analysis of the answers given by them (Outhred 2001). The intended outcome is that,
by the final round, the experts will have reached a consensus of opinion on the issues put before
them.
Delphi is an iterative forecasting procedure characterized by three features (Dickey and Watts, 1978):
anonymity; iteration with controlled feedback; and statistical response. Panel members remain
unknown to one another and respond to a series of questionnaires. The iterative nature of the
procedure influences them to modify their assessments and projects them beyond their own
subjective opinions. It can represent the best forecast available from the consensus of experts
(Corotis et.al. 1981). The process is continued until a consensus is reached on the various issues
under consideration, or until it becomes evident that no further consensus can be developed. The
Delphi generally runs from two to seven rounds of questioning, at most. The major difficulties of
Delphi, however, lie in maintaining the high level of response and in reaching and implementing a
consensus (Robinson 1991).
7. Research methodology and the application of Delphi technique for the
research study
Following a thorough literature research, 21 factors critical to the success of construction JV projects
were identified. These factors were then assembled into a questionnaire survey that was distributed to
1630 local and 70 foreign contractors in Malaysia. A response rate of 341 (20%) was obtained. The
purpose of the questionnaire survey was to discover which companies had experience of JV projects,
the extent of that experience and their views of key factors associated with joint ventures.
The 2 nd part of the research involved in-depth interviews conducted in Malaysia which involved 43
participants from various foreign and local companies. It involved Chief Executive Officers and
General Managers to provide detailed information about their company structure, management and
experience of any JV projects undertaken in Malaysia. Results from the interviews confirmed the
postal survey results of the 12 ranked CSF by at least 90 per cent.
To confirm that the CSFs discovered by this process are reliable, a Delphi Technique which involved
expert panels in Malaysia was conducted to validate/confirm the final findings. The triangulation
approach was adopted for this study which enabled the integration of the quantitative results
(questionnaire surveys) with the results from the qualitative method (interviews). This robust research
process will result in higher quality, reliable data.
8. Implementation of Delphi study
In this study, 22 experts were selected based on recommendations from the contractors involved in
the in-depth interviews conducted in Malaysia. Emails were sent to the experts in Malaysia to obtain
agreement. The 19 expert panels comprise Chief Executive Officers, Executive Directors, Senior
General Managers with vast experience in various local and overseas JVs and 3 academics from
Malaysian universities who are involved in JV projects.
436

8.1 Delphi round
Hamimah Adnan et al.
The questionnaire comprising 21 factors critical to success of construction JV projects were sent out
in early May 2003 with a four week return period, followed by email phone calls to encourage
participation. This generated a response rate of 86% with 19 experts (out of 22) returning the
questions. The experts were asked to rate each statement on a 4-point Likert type rating scale in
terms of their criticality. They were encouraged to add additional comments at the end of the
questionnaire. Cabanis (2002) suggested that “an expert may be defined as someone with special
skills or knowledge evidenced by leadership in professional organizations, holding office in
professional organizations, presenting at national conventions, published in recognized journals.
8.2 Delphi round 2
Round two of the study provided the expert panel with 21 statements of contents with some
amendments made from the expert comments during the first round. They were given the numbers of
the response of each factor based on the scale of criticality to achieve consensus on the statements
added by the panel during the first round. The experts were directed to review their ratings again in
terms of their criticality. The second questionnaire was sent out in early July 2003, generating a
response rate of 17 or 77%. Most of the experts had reconsidered and made adjustments to their
score.
8.3 Results and analysis
The responses were then compiled and analyzed using the above procedure and formula. The
questions were analyzed by providing numerical scores on each of the scale of criticality with 100
points for Extremely Critical (4), 50 points for Critical (3), 10 points for (Less Critical) and 0 points for
(Not Critical). These scores were then transformed to importance indices to determine the relative
ranking of the attributes. A Relative Importance Index (RIX) was then developed to convert the E-
Scores into a decimal figure using the following formula from Kometa et.al (1994):
∑ W
A x N
Where W = weighting given to each factor in the evidence scale, A = the highest weight applied, N =
total number in sample.
Consensus was achieved on 7 CSFs, which include inter-partner trust, criteria for partner selection,
mutual understanding, agreement of contract, compatibility of objective and management control.
Mitchell (1991) points out that consensus can simply mean a ‘group opinion, general agreement or
group solidarity in sentiment and belief’. The panel selected these CSFs as being highly important for
the implementation of construction JVs in Malaysia.
These factors scored 0.80 or more on the RIX scale and were considered as extremely critical
success factors wherein if included, such factors may increase the chance of success but if left out,
will increase the chance of failure (Morledge and Owen 1999). The factors that score less than 0.80
on the RIX scales were not considered as critical. Fourteen factors were designated as Failure
Reduction Criteria (FRC) which if included in the project, might reduce the chance of failure but would
not increase the chance of success (Morledge and Owen 1999). In addition, to the RIX scale, the
percentage of respondents scoring 4 (extremely critical) with 80% and more was used to rank the
most critical success factors. If the RIX were equal, the distribution of the percentage of each scale
will be taken into account. These were confirmed too by the scores of 50% to 74% from results in
Round One.
Table 1: Perceptions of the finalized project success factors from round 1 and round 2
Rank Round 1 RIX Round 2 RIX
1 Criteria for Partner Selection 0.87 Inter- Partner Trust 1.00
2 Mutual Understanding 0.77 Mutual Understanding 0.96
3 Inter- Partner Trust 0.74 Criteria for Partner Selection 0.94
4 Agreement of Contract 0.75 Agreement of Contract 0.91
5 Compatibility of Objectives Compatibility of Objectives 0.88
6 Conflict Commitment 0.85
7 Commitment Conflict 0.82
437

Hamimah Adnan et al.
Rank Round 1 RIX Round 2 RIX
8 Partner Experience Communications 0.61
9 Cultural Understanding Cultural Understanding 0.55
10 Coordination Profit 0.54
11 Profit Partner Experience 0.54
12 Communications Financial Stability 0.53
13 Management Control Cooperation 0.53
14 Cooperation Equity Control 0.53
15 Equity Control Management Control 0.48
16 Organizational Structure Coordination 0.46
17 Financial Stability Effective HRM 0.43
18 Effective HRM Motivation 0.30
19 Motivation Organizational Structure 0.24
20 Knowledge Transfer Knowledge Transfer 0.17
21 Size Compatibility of Partner Firms Size Compatibility of Partner Firms 0.12
There was increased convergence of opinion in the agreed factors during the second round. Although
a three-round sequence seems to be the most popular and recommended method, there were two
cases where two rounds were found to be satisfactory (Olsen 1982). It was judged from the response
of the experts that the marginal gain from the third round may be small compared to the effort
required to perform it, because of the declining participation and subsequently, the Delphi study was
stopped at the second round. This concurs with Dalkey (1969) that answers are most accurate in
Round Two and became less accurate in subsequent rounds. The Delphi final results confirmed the
questionnaire surveys and in-depth interviews which produced similar results with 85% of the
respondents agreeing on the seven CSFs and the fourteen FRCs.
9. Difficulties in conducting the Delphi techniques
It is important that panel members treat the work seriously and devote the time necessary to provide
thoughtful and reasoned responses to questions. Secondly, the wording of the questions and the
presentation format of the survey are extremely important (Robinson 1991).While it is vital to keep the
whole panel responding to each round of Delphi, the successive rounds of the technique were time
consuming. Despite a response rate of 77%, this was considered relatively high and acceptable for
the purposes of this research.
10. Conclusion
The Delphi method was preferred to validate the final findings as it involved panellists who are experts
in their fields to provide reliable data on JV projects in Malaysia. These experts could also confirm the
results of the study. The Delphi Expert Panel resulted in the identification of seven critical success
factors as being extremely important to the construction JV in Malaysia which require a significant
level of application in order to facilitate additional success. Fourteen factors were identified as FRCs
without which a project is at significant risk of failure. The results of this study are expected to provide
useful guidelines for forming and operating effective and efficient JVs both in Malaysia and in other
similar economies.
References
Adnan, H. (2005) “Critical success factors for construction joint venture projects in Malaysia”. Paper read at
International conference on Multi-National joint venture for construction works, Penang, Malaysia, March.
Adnan, H. and Morledge, R. (2003) “Joint venture projects in Malaysian Construction Industry: Factor critical to
success”, Paper read at nineteenth of Arcom, University of Brighton, September, Vol 2, pp 765-774.
Adnan, H., Morledge, R., and Chong, Y. H. (2011) “Success Criteria International Joint Ventures: Malasyian
Contractors in Middle East”, African Journal of Business Management, July.
Aibinu, A. and Jagboro, G. O. (2002) “The Effect of the construction Delays on Project Delivery in Negerian
Construction Industry”, International Journal of Project Management, Vol 20, pp 593-599.
Bower, D. A. and Smith, N. J. (2004) “Types of collaboration between foreign contractors and Malaysian
partners”, Journal Construction Engineering and Management, Vol 132, No. 10, pp 1069-1082.
Bygballe L. E., Jahre, M. and Sward, A. (2010) “Partnering relationships in construction: A literature review”,
Journal of Purchasing & Supply Management, Vol 16, pp 239-253.
Cabanis, K. (2002) “Computer-Related Technology Use by Counselors in the New Millenium – A Delphi Study”,
[online], Journal of Technology in Counselling, Vol 2-2,
http://jtc.cplstate.edu/Vol2.2/cabanis/cabanis.htm.
Corotis, R., Fox, R and Harris (1981) “Delphi Methods: theory and design load application”, Journal of the
Structural Division, Vol 107, No. 6, pp 1095-105.
Dalkey, N. (1969) “Analysis from a Group Opinion Study”, Futures, Vol. 1, No. 6, pp 541-555.
438

Hamimah Adnan et al.
Delios, A., Inkpen, A. C. and Ross, J. (2004) “Escalation in international strategic alliances”, Management
International Review, Vol 44, pp 457-479.
Dickey, J. and Watts, T. (1978) Analytic Techniques in Urban and Regional Planning, McGraw–Hill, New
York.
Gale, A. and Luo, J. (2004) “Factors affecting construction joint ventures in China”, International Journal of
Project Management, Vol 22, pp 33-42.
Girmscheid and Brockman (2000)
Kumarasawamy, M., Palaneeswaran, E. and Humpreys, P. (2000) “Selection matters in construction supply chain
optimization”, International Journal of Physical Distribution of Logistic Management, Vol 30, No. 708,
pp 661-680.
Linstone, H. and Turoff, M. (1975) The Delphi Method: Techniques and Application, Addison, Wesley,
Reading, pp 3-12.
Mohammad, M. and Torrance. J. V. (2005) “Critical success factors in Malaysian construction joint ventures”,
Paper read at International conference on Multi-National joint venture for construction works, Penang,
Malaysia, March.
Morledge, R. and Owen, K. (1999) “Developing a Methodological Approach to the Identification of Factors Critical
to the Success in Privatised Infrastructure Projects in the UK”, Paper read at Harmony and Profit,
ChiangMai, Thailand.
Ogunlana, (2005)
Olsen, S. A. (1982) Group planning and problem solving methods in engineering management, John Wiley
and Sons, New York, Chapter 4.
Outhred, G. P. (2001) “The Delphi Method: A Demonstration of its Use for specific Research Types”, Paper read
at the RICS Foundation, Construction & Building Research Conference, Melbourne, Australia, Vol1,
September.
Ozorhon, B., Arditi, D., Dikmen, I. and Birgonul, M. T. (2010) “Performance of International Joint Ventures in
Construction”, Journal of Management in Engineering, Vol 26, No. 4, pp 209-222.
Petrovic, J. and Kakabadse, N. K. (2003) “Strategic staffing of international joint ventures (IJVs): an integrative
perspective for future research”, Management Decision, Vol. 41, No. 4, pp 394-406.
Petrovic, J., Kakabadse, A. and Kakabadse, N. K. (2006) “International joint venture (IJV) directors' contribution
to board effectiveness: Learning from the literature”, Management Decision, Vol. 44, No. 3, pp 346-366.
Reuer, J. J. (2005) “Real options in International Joint Ventures”, Journal of Management, Vol 31, No. 3, pp
403-423.
Robinson, J. B. L (1991) “Delphi Technology for economic impact assessment”, Journal of Transportation
Engineering, Vol 117, No. 3.
Turner, J. R. (2009) The Handbook of Project Based Management, MrGraw-Hill, New York.
Walker, D. H. T. and Johannes, D. S. (2001) “Organizational learning intentions and joint ventures in Hong Kong
infrastructure projects: A pilot study”, Paper read at ARCOM Seventeenth Annual Conference, Salford, UK.
Zutshi, R. K. (2006). “Confucian value system and its impact on Joint Venture Formation”, International Journal
of Technology Management, Vol 33, No. 2/3, pp 160-182.
439

Action Learning and the Environmental Factors in
Developing Engineering Expertise
Matthew Bell
The University of Bolton, UK
Matthewbell5627@yahoo.co.uk
Abstract: This paper discusses the use of Action Learning to develop specialist engineering capabilities within a
large aerospace organization. Action learning and learning by using both theoretical and experiential practice has
been defined as being more effective at transferring knowledge than pure classroom theoretical presentations.
Action learning was used to transfer the knowledge from a small number of engineering specialists to a larger
group of new recruits and trainees. This paper concludes on the environmental and positional action learning
factors required to develop practical competency in a group of trainees using the knowledge and experiences
from a small group of experts.
Keywords: action learning, engineering capabilities, knowledge transfer, experiential learning
1. Action learning
Action learning is an experiential paradigm to Action Research, which is specific to developing the
people within the organisation (Coghlan and Brannick, 2005). The engagement of action and learning
attributes a higher level of thinking and problem solving ability (Warren et al. 2008). A higher level of
thinking is an essential element for the capability of RCM analysts (Levitt, 2003; Rezendes, 2008).
Organisations cannot develop capability unless their rate of learning (L) is equal to or greater than the
rate of change being experienced (C) (Coghlan and Brannick, 2005; Pedler, 2008; Revans, 1998).
Learning in action has two elements; programmed knowledge (P) and questioning insights (Q)
(Coghlan and Brannick, 2005; Pedler, 2008; Revans, 1998). Organisational learning is developed
from a combination of a transference of programmed knowledge and questioning action (L=P+Q)
(Coghlan and Brannick, 2005; Pedler, 2008; Revans, 1998).
Action learning is the extension of the Action Research learning cycle, as an individual development
mechanism (Coghlan and Brannick, 2005)). Action learning is a learning science, whereby the quality
of the group interaction emphasises engaged learning through direct action (Coghlan and Brannick,
2005). Action learning empowers individuals within an organisation to be critically subjective and
responsible through their actions (Coghlan and Brannick, 2005; Pedler, 2008). The best methods for
transferring tacit, expert knowledge and developing capability are by using a mixture of group
facilitated and self-Action Learning sets (McGill and Brockbank, 2004). Organisational capabilities are
developed from an emergence of both practical and cognitive capabilities (Bitar and Hafsi, 2007).
Action and interaction are the fundamental enablers of the development of micro (cognitive within the
individuals) and macro (organisational) capabilities (Bitar and Hafsi, 2007). Capabilities are fractal
components of knowledge in action (Bitar and Hafsi, 2007). There is no organisational capability
without the action of potential knowledge within the organisation (Bitar and Hafsi, 2007). Capabilities
work by marshalling local level knowledge and issues into CoPs, to develop macro level
organisational competitive advantage (Bitar and Hifsi, 2007). Bryson and O’Neil (2003) emphasise
this in stating that work organisations are the providers, guardians and developers of human
capability. Human capability development is acknowledged using action, empowerment and
knowledge sharing (Bryson and O’Neil, 2003). The enhancing institutions within society and
organisations are implemented with an acknowledgement of human capability development (Bryson
and O’Neil, 2003).
Kolb (1984) proposes an adaptation of the Action Research cycle against learning by action. This
cyclical process starts with the development of theoretical knowledge and the transformation of
theoretical knowledge into practical knowledge using experiential learning methods (Honey and
Mumford, 2000). This cycle of Action Learning can be seen in Figure 35, which shows the cyclical
double-loop learning process of putting theory into practice and reflecting on practical experiences.
Bluckert (2009) uses a psychological frame of reference to define the environment required for selfregulated
experiential development, where he uses a Gestalts method of cyclical development. The
correlation between both Revans (1998) and Bluckert (2009) is the cyclical development of practical
ability and theoretical knowledge arising from an engaged learning process. Essential factors for
440

Matthew Bell
successful Action Learning are homeostasis self-regulation of external environment and internal
threats, and positioning by the facilitator with a clear objective strategy (Revans, 1998; Bluckert,
2009). Bluckert (2009) identifies that operational and performance orientated environments are
typically task focused, whereas to initiate participatory learning and development the individual
behaviours, beliefs and values should be targeted first. This engagement should be positive and lead
to an increased self-awareness of the participants, whereby both their reflexive stances, experiences
and abilities can be explored (Bluckert, 2009). The positioning of the participants into this reflexive
Action Learning process is the route to a heighten awareness and sustained self-regulated
development (Bluckert, 2009). It is essential that the facilitator leads this participatory learning process
as the orchestrator, this includes dealing with conflicting beliefs and motivations within the individuals,
which is often seen as difficult and overlooked (Bluckert, 2009). The development of positive
experiences and self-awareness is the key differentiator to Action Learning and it brings about a
paradox of change, whereby the individual and group's reflexive stances form the environment and
objective reality. This objective setting of the learning environment using experiences and selfawareness,
are of foremost importance to this process to develop group cohesion and participation
(Bluckert, 2009). The Gestalts oriented learning process that is associated with participatory
behaviour change in-order to emancipate self-development can be seen in Figure 2.
Source: Kolb, 1984; Honey and Mumford, 2000
Figure 1: Experiential learning
Revans (1998) states that this is where the role of the Action Learning facilitator is essential to set the
scene, objectives and environment to enable group participation and questioning in establishing
individual self-awareness. The facilitator is also responsible for being aware of the grand narratives
and controlling the systemic ecosystem, although the participants may not be fully aware of these
external meta-factors (Bluckert, 2009). Without a control of the grand systemic factors it can be very
difficult for the participants to be set within an engaged learning environment (Bluckert, 2009; Revans,
1998). This can create restrictions for the participants in applying their theoretical knowledge and can
lead to negative experiences that can close down group participation and cause individual anxieties
and positional threats (Bluckert, 2009). Therefore, it is essential for the facilitator to direct the group
and individuals towards a cohesive learning environment of self-awareness, positive motivation and
commitment (Bluckert, 2009; Revans, 1998). The outcomes of an engaged group Action Learning
process that is behaviourally focused is one whereby, the individuals have a greater capacity to think
from an individual basis in both theoretical and practical stances (Bluckert, 2099; Revans, 1998).
Other positive outcomes can be associated with a greater level of confidence, well-being and
responsibility taken by the participants, which can be used to reflexively influence new theoretical
developments from practical experiences (Bluckert, 2009).
This research is based around participatory Action Research to develop an organisational RCM
capability. Action learning will be required as both formal and in-formal processes to transfer RCM
Subject Matter Expert knowledge across the organisation. The transference of RCM Subject Matter
Expert knowledge is a key enabler towards establishing an organisational capability. Formal RCM
training will be required for all members of the capability. Action learning will not be a specific
methodological approach in this research, it will be central to the role of the participant researcher and
the RCM Subject Matter Expert’s. The RCM Subject Matter Expert’s and the participant researcher
441

Matthew Bell
have responsibilities to continually mentor and coach the RCM analysts. A formal Action Learning
course will be applied as a sub-action cycle to transfer knowledge from the RCM Subject Matter
Expert’s to the new students.
Figure 2: Gestalt’s cycle of experience (cited in Bluckert, 2009)
2. Methodology - action learning and licensing RCM engineers
Multi-disciplined Action Learning was delivered to develop RCM quality against the corporate and
platform programmed knowledge (P) using Socratic questioning (Q) to engage critical thinking and
experiential learning (L), where L=P+Q (Coughlan and Brannick, 2005; Pedler, 2008; Revans, 2008).
Engineers were interviewed as external trainees or internal retraining transferee’s, from redundant
projects as a result of the economic environment. The recruitment and training framework was
developed using best practice from the UK SPEC Engineering Council (EC) Chartered Engineering
(CEng) accreditation route. Competencies, soft skills, experience and knowledge criteria’s were
generalised across the RCM discipline to create homogenous RCM roles. Typically RCM training
consisted of a 3 day facilitation course, which was considered not to fully develop the RCM analysts
for project based analysis on-aircraft. The Action Learning course engaged engineers with a total of
three weeks on-aircraft being taught and mirroring RCM analysis. Each trainee had to pass a final onaircraft
assessment that required on-aircraft analysis, documentation and approval. Each of the RCM
processes required multi-disciplinary learning which involved additional taught modules on aircraft
structure, corrosion, fatigue, aerodynamics, hydraulics, pneumatics, governance, and organisational
strategy. On completion of the RCM practioner’s course the RCM analysts were appointed a RCM
Subject Matter Expert lead on project and licensed for a period of two years to conduct RCM analysis.
The licensing was dependant on RCM Subject Matter Expert quality assurance, refresher training and
platform familiarisation.
Entry barriers were created into the RCM discipline by implementing a licensing and CPD programme
which enabled professional recognition. This enabled RCM governance and capability control.
Additional training was provided for RCM Subject Matter Expert’s in mentoring and coaching skills and
industry regulations. RCM overview training was provided to external disciplines such as Reliability,
Supply Chain and Programme Management teams. This enabled greater integration and a standard
communicative language between the end to end business process and CoPs.
Project requirements were reviewed by the RCM Subject Matter Experts on a quarterly basis where
project audits and cross-platform working practices were established. UK Military RCM resource is a
rare skill set not only within the organisation, but also within the external industrial community. Many
of the projects have had serious project failures, with possible product safety issues as the skill sets
employed were not competent to carry out RCM analysis or RCM Subject Matter Expert roles. Once
the governance framework was implemented and each of the RCM analysts and Subject Matter
Experts were validated, projects had to be reworked. RCM capability growth continued to support
internal project demand into Land, Naval and overseas projects, most of which were new trainees.
Preferred suppliers and approved contractors were also drafted into the capability to manage nonrecurring
workloads. Training was carried out in-accordance-with the training needs department, who
identified external University courses for teaching and mentoring. As a result a quality culture was
sustained and a high performance team ethic was created objectively aimed at assuring product
safety and maximising economic through life maintenance savings where possible. The sustainability
of quality is highly dependent on RCM Subject Matter Expert’s within the discipline. Therefore the
capability of the RCM resource is fundamental to the long term quality of both the RCM programme
and aircraft operation.
442

Matthew Bell
The Action Learning process used a combination of Kolb’s (1998) learning cycle and Honey and
Mumfords (2000) typology of learning styles. This was used to create a safe environment for the
students and an increased awareness of their participation within the learning process to facilitate
experiential dialogue within the group (Rensaa, 2009; Tan, 2006). This method has been identified as
being essential in producing a self-efficacy in learning complex theoretical data by practioners (Rynes
et al. 2001).
3. Conclusions
Positive outcomes were achieved in the complex delivery of the Fast Jet RCM programme. RCM
Subject Matter Experts were empowered to support each heterogeneous project requirement.
Engineering governance assessments were put in place in-accordance-with the Engineering council
UK-SPEC regulations. This served to validate the RCM Subject Matter Experts and provide a
continual peer-to-peer quality audit programme. A training academy was developed in-accordancewith
a Training Needs analysis (TNA) and accredited RCM Subject Matter Expert trainers. This
training programme was developed using an Action Learning course structure for engaged learning.
The objective was to develop the tacit skills and knowledge across the capability. RCM students
needed to be practically capable of conducting RCM on their projects. RCM managers needed to
understand the governance, regulations and management guidelines in developing quality and
performance on their RCM programmes. RCM Subject Matter Experts needed to develop strong
leadership, mentoring and coaching skills in ensuring that quality, performance and continual
development were instilled as a quality culture. Training was provided by a strict interview process,
looking primarily for the “soft” skills and work attitudes that constructed an emphasis on product
quality. These candidates were placed on external RCM familiarisation courses, which were
traditionally the only method of RCM practioner training. This was used to provide a general level of
RCM pre-understanding to the trainee. The RCM Subject Matter Expert took on additional mentoring
roles across all projects. Their aim was to develop shortfalls in the project teams’ knowledge, by
continually engaging with the new recruits, existing analysts and their managers. The training and
development programme was essential in engaging with the individual and subjective viewpoints
across the projects. This engaged training programme developed a much greater degree of RCM
awareness and knowledge. This greater awareness helped to develop a longer term quality culture.
This qualitative finding was substantiated by the quantitative analysis, as analysts ranked there RCM
knowledge as either the same or less after the first training courses. The increased level of awareness
gave the trainee a better idea of what gaps they had in their existing knowledge, which was a
foundation to develop new tacit knowledge upon. Alvesson and Deetz (2000) state in the quote below,
that this level of awareness must also be taught, as it is a trained capacity that requires an
understanding.
“Attention is a trained capacity if you did not know what you were looking for you would
not know when you had found it.” (Alvesson and Deetz 2000, p.41).
The continual conceptualisation and experimentation of exogenous learning was used in both the
Action Learning course, and by embedding RCM Subject Matter Experts onto each project with new
mentoring responsibilities. Serendipity and implicit learning were used as reflective processes for the
students (subjects) within the RCM capability (environment-situation). This formed the foundation for
the Action Learning course assessments, the on-project RCM QA and mentoring by the Subject
Matter Experts. This helped to articulate exogenous and experimental learning factors within the
Action Learning environment as shown in Figure 3 (Baumard, 2001). The pragmatism of the
candidates was crucial in engaging with the learning and development programme in emancipating a
transfer of knowledge. The quantitative analysis supports these qualitative findings, with 92 percent of
the students delivering RCM to the correct quality at the end of the taught training courses. The
effects of this on project identified both an increase in schedule adherence and deliverable quality.
Deliverable quality increased from 24 percent to 82 percent. On-time deliverables increased from 28
percent to 64 percent. And as a result rework reduced from 48 percent to 7 percent with ahead of
schedule deliveries increasing from 2 percent to 12 percent across all of the projects. This supports
the qualitative analysis findings for a high requirement in RCM capability quality. The quality is
developed around skilled resource, tacit knowledge, the QMS and governance.
443

Matthew Bell
Source: Baumard, 2001
Figure 3: Knowledge articulation
4. Tacit knowledge development
An Action Learning programme for engaging theoretical and practical competency was used to
develop specialist engineering capabilities within a large aerospace organization. There are
contributions from the teaching method and taught course. As a result of the Action Learning
programme created in this research, large aerospace organization now have a RCM training
capability to develop their own specialist engineers. This has had a great effect in the engagement,
direction and motivation of teams. There are contributions around using behavioural change and
Action Learning to engage students with a common organisational strategy. This leverage of
technology, organisational aims, behavioural change and Action Learning promoted a wider
awareness of both theoretical and practical competency (Fulmer et al. 2009). However, coaching and
mentoring of students was essential to ensure long term sustainability (Pollitt, 2009). Due to the
impact of the SDSR there has been limited support for the on-going mentoring and coaching, which is
leading to negative outcomes across the teams. The development of practical knowledge from
heterogeneous projects towards a homogenous objective is highly suited to Action Learning
(Guzman, 2009).The future of the Fast Jet Maintenance Programme Optimisation capabilities are
highly reliant on the Action Learning programme developed in this research.
5. Educational engagement using action learning
The main contributory factors to RCM programme failures were identified as a lack of understanding
and education (Bloom, 2006; Levitt, 2003; Moubray, 1997). This research has identified theoretical
contributions in the use of Action learning to develop both awareness and competency within the
RCM discipline. This provides a theoretical contribution in the method for developing RCM knowledge
across an organisation. There are also theoretical contributions to the existing knowledge base on the
use of Action Learning. There is already a substantial amount of literature and knowledge in the
theory and use of Action Learning. This research adds to this theoretical knowledge from the positive
outcomes in the use of Action Learning programmes to develop specialist capabilities over relatively
short timescales.
References
Alvesson, M., Deetz, S. (2000) Doing Critical Management Research. London: SAGE Publications.
Baumard, P. (2001) Tacit Knowledge in Organisations. Paris: SAGE.
Bitar, J., Hafsi, T. (2007) Strategizing through the capability lens: sources and outcomes of integration.
Management Decision [Online] 45(3), pp. 403-419. Available from [Accessed 08
September 2010].
Bloom, N. B., (2006) Reliability Centred Maintenance, Implementation made simple. 1st Ed. New York, NY:
McGraw-hill.
Bluckert, P., (2009) Psychological Dimensions of Executive Coaching. 1st Ed. New York, NY: McGraw-hill.
Bryson, J., O Neil, P. (2003) A Workplace View of Drivers and Barriers to Developing Human Capability. New
Zeeland Journal of Employment Relations [Online] 34(1), pp. 62-76. Available from
[Accessed 08 September 2010].
Coghlan, D., Brannick, T. (2005) Doing Action Research in your own organisation. 2nd Ed. London: SAGE.
444

Matthew Bell
Fulmer, R. M., Stumpf, S. A., Bleak, J. (2009) The Strategic development of high potential leaders. Strategy &
Leadership [Online]. 37(3), pp. 17-22. Available from [Accessed 16 March
2011].
Guzman, G. (2009) What is practical knowledge? Journal of Knowledge Management [Online[. 13(4), pp.86-98.
Available from [Accessed 16 March 2011].
Honey, P., Mumford, A. (2000) The Learning Styles Helper‟s Guide. 1st Ed. Berkshire: Peter Honey Learning.
Kolb, D. A. (1984) Experiential Learning: Experience as the Source of Learning and Development. 1st Ed. New
Jersey NJ: Prentice-Hall Inc.
Levitt, J. (2003) Complete guide to Preventive and Predictive Maintenance. 1st Ed. New York, NY: Industrial
Press.
McGill, I., Brockbank, A. (2004) The Action Learning Handbook: Powerful Techniques for Education.1st Ed.
Oxon: RoutledgeFalmer.
Moubry, J. (2001) Is Streamlined RCM Worth the Risk? [Online]. Available from
[Accessed 18 January 2010].
Moubry, J. (1997) Reliability Centred Maintenance. 2nd Ed. Oxford: Butterworth-Heinemann.
Pedlar, M. (2008) Action Learning for Managers. 1st Ed. Hampshire: Gower Publications ltd.
Pollitt, D. (2009) Responsive engineering boosts profitability and performance through its people: Coaching,
Action Learning and workshops help to transform the business. Human resource Management International
Digest [Online]. 17(1), pp.15-18. Available from [Accessed 16 March 2011].
Rensaa, R. J. (2009) Some Experiences with Integrating Student Teachers and Engineering Students in the
Same Class. PRIMUS: Problems, Resources, and Issues in Mathematics Undergraduate Studies [Online].
19(5), pp. 473-490. Available from [Accessed 16 March 2011].
Rezendes, M. (2009) Reliability Centred Maintenance, The cognitive psychology of maintenance development.
International RCM Conference, Daytona, FL.
Revans, W. R. (1998) ABC of Action Learning. 1st Ed. London: Lemos & Crane.
Rynes, S. L., Bartunek, J. M., Daft, R. L. (2001) Across the great divide: Knowledge creation and transfer
between Practioners and Academics. Academy of Management Journal [Online]. 44(2), pp.340-356.
Available from [Accessed 16 March 2011].
Tan, O. S. (2006) Development of a thinking programme for engineering students. Innovations in Education and
Teaching International [Online]. 43(3), pp. 245-259. Available from [Accessed 16
March 2011].
Warren, S., Doorn, D., Green, J. (2008) Changes in Vision: Teachers Engaging in Action Research. The
Educational Forum [Online] 72(3), pp. 260-270. Available from [Accessed 08
September 2010].
445

An Insight Into the Promise and Problems of Combining
Life History and Grounded Theory Research
Bronwyn Betts
Anglia Ruskin University, Cambridge, UK
b.betts673@btinternet.com
Abstract: This paper describes the research carried out in partial fulfilment of the Degree of Doctor of Education.
The study was qualitative in nature with a phenomenological interpretive paradigm dominating the philosophical
approach. The research methods adopted combined Life Story and Grounded Theory. As far as I have been able
to determine there are very few, if any studies which have applied this approach specifically to my area of
research. My study investigated the influence life history has on attitude to lifelong learning. Twenty five
respondents were interviewed in face-to-face informal interviews. The main aim was to elicit the respondent’s
subjective interpretation of the interaction between school, family, work and learning within their lives. I, (the
researcher) was then able to identify when they occurred and what or who made them particularly meaningful.
Within this paper I describe how initial decisions were made regarding the substantive area to research. My
sampling technique and method for collecting the data is discussed and a worked example is given of how the
data was analysed. It is intended that this paper will give an insight into the challenge of combining these much
debated methods of research. The empirical data lead to some interesting findings which educators and policy
makers will find helpful in order to strengthen the school, college and workplace interface.
Keywords: life history, grounded theory, methodology, family, school, learning
1. The raison d’être of the research
At the inception of my doctoral studies I was facilitating the reflective practice of a group of students
studying for a postgraduate degree in human resource management. An early observation made as a
result of reading their reflective notes supported a perspective which pointed to the strong influence
that experiences in the early years had had on their current (positive) attitude to learning. Therefore
the initial motivation for the study was to seek further understanding of an otherwise little understood
area, namely the influence life history can have on attitude to learning in later life.
1.1 The Theoretical and Methodological Framework
The study was a qualitative study with a phenomenological interpretive paradigm dominating the
philosophical approach adopted. The research methods were chosen because of their close
association with this paradigm and were used to explore the dynamics of the relationship between life
history and attitude to learning. As suggested by Baker et al., (1992) cited in Wimping and Gass
(2000) the researcher’s own philosophy and beliefs will form the bedrock on which the choice of
research method is based. Additionally it is imperative that the chosen method or methods are
congruent with the research aims. In this study my own philosophical beliefs, the experiences of my
students, the phenomenon under investigation and the research aim all directed the methods used for
data collection and analysis.
2. The chosen methods
2.1 Life history
The history method was chosen to gather data as it focuses on the ways individuals account for and
theorise about their actions in the social world over time (Musson, 2004). In other words it is basically
retrospective. It focuses on the stories people tell about their lives so far. Goodson and Sikes (2002)
suggest that it helps achieve a better understanding of the past in order to open up possibilities for the
future and this was key in my decision to adopt the life history method for data collection The in-depth
interview was favoured as the tool for the collection for the data as it gave me access to the
respondents’ accounts of how life history may have influenced their current attitude to learning. It was
also chosen as it “fitted” with the epistemological standpoint adopted for the data analysis which was
Grounded theory. (Glaser and Strauss. 1967).
2.2 Grounded theory
Although grounded theory is a method that can be used with any kind of data collection (Glaser and
Strauss (1967), Glaser, (1978) and (Charmaz (2007:28) suggest that intensive qualitative interviewing
446

Bronwyn Betts
fits grounded theory methods particularly well. Both grounded theory methods and intensive
interviewing are open-ended yet directed, shaped yet emergent, and paced yet unrestricted.
Grounded theory is a general, inductive and interpretive research method which was originated in
1967 by Barney Glaser and Anselm Strauss (Glaser and Strauss. 1967). Its method for analysing
data was chosen, as already alluded to, because it fitted with the philosophical base of this research,
that is to say I did not set out to verify a theory, but to develop a theory which sought to explain how
past experiences can influence attitude to learning. Grounded theory emphasises theory
development. It focuses on unravelling the elements of experience then, as a result of studying these
elements and their interrelationships a theory is developed which enables the researcher to
understand the nature and meaning of an experience for a particular group of people in particular
settings (Glaser and Strauss 1967)
3. The research process
Figure 1 provides an overview of the research process. Although the steps taken are presented in
linear form, in reality these are often cyclical and interdependent processes.
Open -
coding
Axia
coding l
Step 1 - The identification of
substantive area to research
Step 2 – Data collection
Step 3 -Data analysis
Step 4 - Final literature
comparison
and analysis
On-going
memoing
Identification
of the sample
Life history
interviews
Constant
comparison
Ongoing
Literature
Review
Figure 1: The research process
All methodological decisions were perceived to meet key requirements for doctoral research, that is to
say they fitted well with the research aims and philosophical underpinning. Identifying the substantive
area to research (step 1) was relatively straightforward. Notwithstanding the interest which my
student’s reflections had generated, my own personal experiences had also contributed to my choice
of direction that my research would take. My own personal experiences of school were not positive
and consequently it has always been a great puzzlement that throughout my adult life I have been
motivated to engage in learning. This is all despite the fact that much of the literature suggests that
people’s willingness to engage in learning, particularly as adults, depends to considerable extent on
whether they draw on positive experiences from this period of initial education and early years. Thus
the study began with the identification of a substantive area which I was interested in researching,
rather than a narrowly focussed research question (Corbin and Strauss, 1998)
447

Bronwyn Betts
4. The role of the literature in grounded theory research
Whilst stage 1 did not cause me too many problems step 2, data collection, was more problematic.
When the literature should be consulted is a fundamental issue in grounded theory work. For Glaser
and Strauss (1967), Glaser, (1978) and Strauss, (1987) the defining components of grounded theory
practice include conducting the literature review after the collection and analysis of empirical data.
This presented me with something of a dilemma, as on the one hand I was anxious to critique earlier
studies to make comparisons with my findings from the interviews, on the other I did not want to
import preconceived ideas and impose them on my work (Charmaz 2007)
In the end, my decision was influenced by two writers. Firstly, McGhee et al. (2007), who suggests
that the literature in grounded theory research is simply another source of data which should be
theoretically sampled along with other emergent data. Theoretical sampling according to Charmaz
(2007:96) is about:
“Seeking pertinent data to develop your emerging theory
The main purpose of theoretical sampling is to elaborate and refine the
Categories, constituting your theory”
My second influence was Onion (2004) who claims that the literature can be used as a form of
primary as well as secondary data. Thus, as can be seen from fig. 1 I made the decision early in the
research to delay the first real move into the literature until the process of open coding had taken
place and then use it to theoretically sample.
5. The sample
Grounded theory studies are characterised by theoretical sampling and this gave the study direction.
However in order for this process to start there needs to be an initial sample for some data to be
collected and analysed. Baker et al (1992) maintain that the researcher using the grounded theory
approach initiates the sampling process by interviewing significant individuals who have the
knowledge and experience that the researcher requires, have the ability to reflect and articulate, and
have the time. Thus, the sample chosen for the study was selected because the participants could
provide relevant sources of data, they were relatively diverse in background but all shared a positive
attitude to learning. Additionally the size of the sample was quite small which reflected the view of
Goodson and Sikes (2001) who purport that research samples for life history research are usually
quite small. This relevance was determined by the requirements for generating and delimiting the
theoretical codes (Hutchinson, (1993) cited in Cutcliffe, (2003). Within this initial sample. I was
concerned with generating as many categories as possible. Once these events or happenings had
been categorised, I then sampled the literature with the aim to purposefully gather data related to
these categories
5.1 Research participants
All respondents were either engaged in promoting learning and development within their organisation,
or full-time students having recent past work experience and intending to re-join such organisations
upon graduation. All students were interviewed in face-to-face informal interviews. The respondents
were all interviewed in places convenient to them, either their places of work or at the college where
they were attending their course.
Table 2 shows the gender, age, mode of study and the type of employing organisations of the
participants
6. The In-depth interview
As can be seen from fig 2 the respondents were quite diverse in terms of age and background. The
interviews began by me asking the respondents to “offer a sequential account of their life” (Mann,
1992:273), but more specifically throughout the narrative to focus on significant experiences of
learning. The main aim was to elicit the respondent’s subjective interpretation of the interaction
between school, family work and learning within their lives. I was satisfied that most of the criteria for
a good interview were met. Briggs (1986) suggests that good interviews are those in which the
subjects are at ease and talk freely.
448

Bronwyn Betts
Permission to tape the interviews was requested and given in all cases. Although I took transcripts of
the interview I was keen to ensure that my body language communicated personal interest and
attention.
Table 2: Gender and age of participants
Gender Number
Female 18
Male 7
Age Range Number
20-30 10
30-40 8
40-50 7
Mode of Study
Mode of Study Number
Full-time 10
Part-time
Employing Organisation
15
Employing Organisation Number of Students
Full-time students with recent past employment history 10
NHS 3
Private Training Organisation 2
Constabulary 2
Insurance Co. 1
Supermarket (a) 1
LEPRA 1
Supermarket (b) 1
Holiday Camp 1
Fashion Retailer 1
Construction Company 1
Learning Skills Council 1
Bogdan and Biklen (2002) advise that, when asking the respondent about the past, the interviewer
suggests that he or she think back to that time and try to relive it. This proved to be a valuable
strategy in the interviews when encouraging the respondents to contextualise their narrative, as did
their reassurance that the interviewer need not fear silence. Silence, as I discovered can enable
subjects to get their thought together. Another helpful lesson I learnt was not to interrupt and change
the direction of the narrative.
6.1 Ethical considerations
Social science, like life, is shot through with personal, political and ethical dilemmas (Plummer,
206:277) and the research I was undertaking was no exception to this. The implication for informants
in life history research was a particular concern for me. Goodson & Sikes (2001) suggest it is not
always possible to predict the sort of harm that informants may experience as a consequence of their
involvements. I gave an enormous amount of thought and attention to what I was doing and to the
potential problems which may arise. The British Educational Research Association (BERA, 2004)
considers that all education-related research should be conducted within an ethic of respect for the
person. The participants in my research were active subjects and I as a researcher (and their tutor)
had a responsibility to them. I was aware that the life history approach to gathering data entailed
reflecting on things that had happened in their lives, both good and bad. I therefore had a
responsibility to avoid any embarrassment and intimidation, or to expose confidentialities (Stake
2000)
It was emphasised that participation was voluntary and they would not be prejudiced if they refused to
participate. They were assured of their right to withdraw their consent at any stage. I was a senior
member of staff in the college where the research was conducted at the time and, as their tutor knew
them reasonably well. This raised ethical issues about personal and professional responsibility. For in
effect I was “doing research in my own back yard” (Goodson and Sikes 2001) with this in the forefront
of my mind I aimed to establish mutuality, minimise communication barriers and allay discomfort
(Seiber, 1993)
449

7. Data analysis phase
Bronwyn Betts
As already alluded to detailed transcripts were made following each interview and the data examined
several times in order to develop explanations of the phenomenon being studied (Douglas, 2003:46)
Memoing, or notes to myself, were made during and after the interviews. These were basically
reflections on what I felt my relationship with the informant was like, any doubts about the quality of
some the data, and any remark that related to a theory or concept which I could explore in the
literature (Strauss and Corbin 1998). I also noted any issue I wanted to pursue further with the next
contact or to re-interview at some point (Miles and Huberman, 1994)
7.1 Initial coding
The data in the transcripts and to some extent the memos was analysed phrase by phrase. This
involved taking each phrase, examining and labelling. The labels were referred to as categories,
concepts and properties. This process of initial coding (or often referred to as open coding) requires
application of what is referred to as the “comparative method” (Panddit, 1996) that is to say broken
down and questions asked. For example, I asked ‘what did this mean?’, ‘what is being referred
to?’’Who was involved?’, ‘When did this happen?’ The data is then compared with data from the other
interviews, grouped together and given a conceptual label Fig 3 below is a worked example. The first
two respondents mentioned the influence of family in the early years, so this was tentatively coded as
“family influence. The categories helped me make sense of what the participant had said. Whilst two
participants mentioned the family influence, it was different depending on the context, characteristic,
behaviour and impact of the family. Properties were then examined, for example the type of behaviour
in the family, the extent of that behaviour and so on. The data from the coding was then used to
theoretically sample the relevant literature in order to elaborate and refine the categories constituting
a theory
CONCEPT KEY CATEGORIES PROPERTIES
Family influence Context
Characteristics
Behaviour
Impact
Figure 3: Initial coding
7.2 Axial coding
Frequency
Extent
Behaviour type
Duration
The next stage in the process of analysis is axial coding. This process allowed me to examine the
data more closely and identify relationships between the categories and properties identified in open
coding. This is done in order to develop core or major codes. . Axial coding utilises a coding paradigm
which involves the researcher looking at the concepts, the context again and /intervening conditions
involved. Consequences are then looked at in order to bring out the central idea or core category. So
for example after family influence had been analysed under the key categories and properties the
process of axial coding allowed me to think about the material in new ways and examine the data
more closely under a different coding paradigm. This enabled me to build up a dense texture of
relationships around the axes of the category being focussed upon. Hence Strauss and Corbin’s
(1998) definition of the process ‘axial coding’.
The emergent themes from axial coding as with the themes which emerged from the open coding
procedure were then explored in greater detail against selected literature. This led me to identify
relationships between life experiences and attitudes to learning in later life.
8. Findings
The above process yielded a number of interesting findings and themes. The study presented
evidence and supports previous findings that underline the important influence family and school have
on developing a positive attitude to learning (see for example Pugh, (1999); Pollard and Bourne
(2002), Nut-brown and Clough, (2006) and Pugh and Duffy,( 2007). It adds to the vast volume of
literature in this area, insofar as whilst it may be the case that adults who are the best learners in
terms of achievement come from homes which value learning (Csikszentmihalyi, 1993,) my findings
450

Bronwyn Betts
suggest that adults who have had bad experiences of learning within the home and school can and do
still go on to develop a positive attitude to learning in later life.
One respondent had had a particularly negative experience at school and was told “she would never
do anything with her life and was definitely not grammar school material. She had remembered these
words and they had, she claimed, made her more determined to succeed. She went on to obtain a 2:1
degree, a post graduate diploma and aspire to a very senior position in her organisation. Similar
stories were told by many of the respondents. They were clearly driven by these negative
experiences.
Another central theme which was drawn out is that individual perceptions of learning can be a key
influence. The interviews bring out the differing perceptions and experiences of learning in schools
and college as opposed to learning in the workplace. The research supported Pollard and Bourne’s
(2002) assertion that schools focus too much on curriculum and teaching whilst paying insufficient
attention to social factors. The findings from my research support the viewpoint, which underlines the
importance of making learning in schools more relevant and individualised.
Finally the influence and importance of context and learning was a recurring theme in the study. The
respondents perceived the workplace environment as informal, supportive and relevant in terms of
meeting their learning needs. Unlike school and to a certain extent colleges, workplaces are
developing the whole person, not just a single facet of the individual (Compton et al., 2006). My
conclusions suggested that my research has shown the disjuncture between different phases of
learning (school, college and the workplace.)
9. Some final thoughts
The aim of this paper was to provide an insight into some of the issues of adopting grounded theory
and life history methodologies.
One of the biggest challenges of using grounded theory was adapting it to meet my own research
needs. It takes a degree of confidence to take the general principles and practices of grounded theory
and vary them according to your own research needs.
Grounded theory offers a set of guidelines rather than rigorous rules to be followed and as a novice
researcher I could have adopted a more structured practically orientated approach. However with
grounded theory the data itself shapes the processes and product and for me it was important that I
was not totally relying on pre-conceived ideas and had the flexibility to pursue emerging themes.
Life history as a methodology presented its own challenges. The researcher should never
underestimate the importance of thoroughly familiarising themselves with this methodology and the
associated ethical issues, some of which have been discussed within this paper. The implications for
the respondents and the researcher role in particular require special consideration and attention. As
reported earlier in this paper it is not always possible to predict the sort of harm that informants may
experience as a consequence of their involvement. Goodson and Sikes (2001)
Combining the two methodologies has not always been straightforward. However these two
approaches have given me valuable experience and insight into conducting phenomenological
research. It can be summed up by Moustakas, (1994:50)
“In a phenomenological investigation the researcher has a personal interest in whatever
she or he seeks to know, the researcher is intimately connected with the phenomenon,
the puzzlement is autobiographical, making memory and history essential dimensions of
discovery, in the present and extensions into the future”.
References
Baker C., West J. & Stern P. (1992). Method slurring: the grounded theory/phenomenology example, Journal of
Advanced Nursing, 17, 1355-1360.
BERA (2004). Guidelines for Ethical Research, The British Education Research Association. Available on-line at
www.bera.ac.uk. London
Bogdan, R.C. & Biklen, S.K. (1992). Qualitative Research for Education: An Introduction to Theory and Methods,
2 nd edn., Allyn & Bacon,
451

Bronwyn Betts
Briggs, C. (1986). Learning How to Ask: A Sociolinguistic Appraisal of the Role of the Interview in Social Science
Research, Cambridge University Press, Cambridge, Massachusetts.
Charmaz, K. (2007). Constructing Grounded Theory: a Practical Guide Through Qualitative Analysis, Sage,
London.
Csikszentmihalyi, I. (1993). “Family influences on the development of giftedness”. In: Bock, G.R. & Acknill, K.
(eds.) (1993). The Origins and Development of High Ability, CIBA Foundation Symposium 178, Chichester,
UK.
Corbin, J. & Strauss, A. (1998). Basics of Qualitative Research: Techniques and Procedures for Developing
Grounded Theory, Sage, London.
Compton, J.I., Cox, E. & Santon Laanan, F. (2006). “Adult learners in transition”, New Directions for
StudentServices, 114, 208. Also available on-line at www.interscience.wiley.com/DO1.101002/ss.208.
Cutliffe, R. (2003). “Methodological issues in grounded theory”, Journal of Advanced Nursing, 31(6), 176-194.
Douglas, D. (2003). “Grounded theories of management: a methodological review”, Management Research
News, 26(5), 44-52.
Glaser, B. & Strauss, A. (1967). Discovery of Grounded Theory, Aldine, Chicago.
Glaser, B.G. (1978). Theoretical Sensitivity, Sociology Press, Mill Valley, California.
Goodson, I. & Sikes, P. (2001). Life History Research in Education Settings: Learning From Lives, Open
University Press, Bucks.
McGhee, G., Glenn, R., Marland, J. & Atkinson, J. (2007). “Grounded theory research: literature reviewing and
reflexivity”, Research Methodology, January
Mann, S.J. (1992). “Telling a life story: issues for research”, Management Education and Development, 23(3),
271-280.
Miles, M.B. & Huberman, A.M. (1994). Qualitative Data Analysis, 2 nd edn., Sage, London.
Moustakas, C. (1994). Phenomenological Research Methods, Thousand Oaks/Sage, London.
Musson, G (2004) Life histories, In Cassell C. & Symon G. (eds) (2004). Essential Guide to Qualitative Methods
in Organizational Research, Sage, London pp-34-44
Nutbrown, C. & Clough, P. (2006). Inclusion in the Early Years: Critical Analysis and Enabling Narratives, Sage,
London.
Onion, P. (2004). “Grounded theory applications”, Reviewing Knowledge Management Literature, 1, 18.
Pandit, N.R. (1996). “The creation of theory: a recent application of the grounded theory method”, The
Qualitative Report, 2(4), December, 1-21, also available on-line at http://www.nova.edu/sss/QR/QR2 -
4/pandit.html.
Plummer, K. (2006). Documents of Life: an Invitation to a Critical Humanism, Sage, London.
Pollard, A. & Bourne, J. (eds.) (2002). Teaching and Learning in the Primary School, Open University, London.
Pugh, G. (1999). Young Children and Their Families. A Community Response, Falmer, London.
Pugh, G. & Duffy, B. (2007). Contemporary Issues in the Early Years, Sage, London.
Seiber, J.E. (1993). “The ethics and politics of sensitive research”. In: Renzetti, C.M. & Lee, R.M. (eds.) (1993).
Researching Sensitive Topics, Sage, Newbury Park, pp.626-633.
Strauss, A. (1987). Qualitative Analysis for Social Scientists, Cambridge University Press, Cambridge.
Strauss, A. & Corbin, J. (1998). Basics of Qualitative Research: Grounded Theory Procedures and Techniques,
2 nd edn., Sage, London.
Sikes, P. & Gale, K. (2006). “Narrative approaches to education research”, available on-line at
www.edu.plymouth.ac.uk (accessed 03 April 2008).
Stake, P. (2000). Case Studies, In: Denscombe, N.K. & Lincoln, Y.S. (eds) (2003) Handbook of Qualitative
Research, Sage, Thousand Oaks,pp.435-454
452

The Doctoral Level Practitioner: The Development of new
Knowledge Through an ‘All Channels Open Project’ for the
Teaching of Biosciences in National and International
Nurse Education
John Lorimer Campbell 2 , Robert Campbell 1 and Gill Green 1
1
University of Bolton, Bolton, UK
2
University of Cumbria, Fusehill Street Campus, Carlisle, UK
john.campbell@cumbria.ac.uk
R.H.Campbell@bolton.ac.uk
g.green@bolton.ac.uk
Abstract: Over the past 20 years one researcher has developed a range of educational materials relating to
nurse education. These use the widest methods of dissemination facilitated by the technology available at any
time, utilising various channels of communication in order to increase the number of people who can access the
materials. It is therefore an ‘all channels open project’ (ACOP), including journal articles, traditional text books, e
books, DVDs, podcasts with downloadable and streamed materials. Two hard copy text books have been
written, use of digital publishing has allowed thousands of copies to be produced and internationally distributed at
low cost. Profits from sales in richer countries have paid for free, or heavily subsidised, distribution in poorer
countries. This has allowed traditional publishers to be bypassed, facilitating free access to materials in many
poorer countries. Novel approaches in this project include the utilisation of technology by the development of
modalities of communication to suit a particular channel. Concepts discussed include; the scholar-practitioner,
work based problem solving and the development of new knowledge in the context of the emerging professional
doctorate. Development of new knowledge in the professional context is inter-related with the concept of ‘Mode 2’
methodologies, progressing work from Gibbons (1994).
Keywords: professional doctorate, scholar practitioner, professional researcher, context driven problem solving,
lifelong development, health care impacts
1. The scholar–practitioner
Stokes (2005) points out that the stereotype of the full time, young, undergraduate student, living on a
university campus now represents a minority of the total student population. Kuipers (2011) further
suggests that the situation is similar at postgraduate level with adult professionals continuing a pattern
of lifelong learning. Indeed Kuipers suggests this is required in the modern professional environment
where knowledge is the new economic currency and the rapid development of new knowledge in a
discipline requires continuous learning. In a similar vein, Dickerson et al. (1990) argue that
traditionally the focus of educational research has been on students of 25 years of age or younger.
However, more recent work highlights the ongoing importance of education in the lives of older people
(Manheimer et al. 1995; Woodley & Wilson, 2002). Shapiro (2001) concludes scholar–practitioners
are those who apply their scholarship to practice in order to develop and create new knowledge to
inform their work and the work of others.
This point is crucial to the PhD by professional practice philosophy. This researcher (and paper
author) also contends he has developed and created new knowledge to inform his work and the work
of others. He further claims this new knowledge constitutes an original contribution to knowledge and
these educational and research outcomes have led to improved practice in the nursing and nurse
education communities. In this way the new knowledge developed has had a real impact on the
wellbeing of potentially many thousands of patients.
2. The professional researcher at doctoral level
Previous professional learning and experience mean the professional is very likely to bring a range of
transferable skills into the research domain. Watts (2009) point out that these are likely to include
project management and organisational skills. Other advantages she identifies include probable high
levels of motivation and commitment to their subject areas. This background skill set is likely to equip
the professional researcher with many of the skills required to negotiate the range of complex
academic tasks of doctoral study.
453

John Lorimer Campbell, Robert Campbell and Gill Green
While the insightful professional is well aware he or she does not know all of the answers, they are
prepared to focus directed effort onto areas where knowledge is limited, uncertain or contradictory. In
order to enhance the ability of the professional researcher to attain this objective they will recognise
the need to also acquire many of the traditional skills of the academic researcher. These include the
development of enhanced critical thinking skills that question assumptions and established positions,
together with the ability to communicate these in high-level academic writing. This duel skill set,
(professional and academic) can be seen in the United Kingdom’s Council for Graduate Education
(UKCGE 2002) statement which describe a professional doctorate as
‘a programme of advanced study and research which, whilst satisfying the University
criteria for the award of a doctorate, is designed to meet the specific needs of a
professional group external to the University, and which develops the capability of
individuals to work within a professional context’.
In other words the professional doctorate is academically equal to a traditional academic PhD, but in
addition meets a professional ‘real world’ need, while also enhancing the abilities of the individual
professional researcher.
3. Doctoral degree characteristics in relation to professional doctorates
The Quality Assurance Agency for higher education (QAA 2011) maintains an active document
covering doctoral degree characteristics. Some of these will now be considered in the context of
professional doctorates in general and the ACOP in particular.
‘Institutions, in determining the nature of the courses which they provide, and the level of
entry to these, need to pay particular regard to the employment opportunities that follow
post graduation study, as part of assuring the quality and standards of the provision'.
(page 5).
The current researcher has already been employed in professional nurse education for many years. It
could therefore be argued that this employability criterion has already been met. In terms of entry
requirements a good BSc and an MSc are already held in addition to several professional and
educational qualifications.
‘All UK doctorates, however, continue to require the main focus of the candidate's work to
be their contribution to knowledge in their discipline or field, through original research, or
the original application of existing knowledge or understanding’. (page 7).
ACOP has developed new knowledge in terms of educational delivery. The application of subject
matter to a particular channel of communication has also required novel approaches in an attempt to
optimise the delivery process to the particular medium. The development of multimedia vehicles of
communication constitutes an original application of knowledge. This is also pragmatically
demonstrated in the numbers of learners who use the materials.
‘In professional and practice-based doctorates the research may be undertaken in the
workplace and may have a direct effect on improving the professional practice of
individuals and their host organisation’. (page 7).
All of the ACOP developments have taken place in the workplace, with ongoing feedback from
learners and peers. Many registered nurses have graduated under this workplace activity, thereby
improving the individual student’s levels of practice. The development of quality resources has aided
the reputation of the host institution and also helped other educationalist to develop their practice.
‘Often, professional and practice-based doctorates are the choice of doctoral degree for
mid-career professionals’ ….. doctoral ‘degrees can provide an opportunity for individuals
to situate professional knowledge developed over time in a theoretical academic
framework’. (page 14).
After 22 years of nurse educational experience, the present researcher can certainly claim to be ‘mid
career’. This has allowed time for the accumulation of a range of professional knowledge and ‘know
how’ which is amenable to contextualisation in a theoretical framework.
454

John Lorimer Campbell, Robert Campbell and Gill Green
‘Research projects in these (professional practice) degrees are normally located within
the candidate's profession. In practice-based or practitioner doctorates the candidate's
output involves practice-related materials’. (page 15).
Again, all of the work for the current professional doctorate has been conducted within the
professional context. The ACOP involves the production of verifiable, practice related educational
materials.
‘Professional doctorates are normally rooted in an academic discipline as well as in a
profession’ (page 15)
The content used in ACOP draws on a number of academic disciplines as well as education. Material
comes from academic aspects of nursing, psychology, medicine, therapeutics, biology and health
related studies. All of these academic disciplines are focused through a professional lens to provide
functionally useful educational materials.
4. The professional doctorate
Concerns have been expressed that a professional doctorate may not meet the same exacting
academic standards of the traditional PhD route. Fore example Pearson et al. (1997) questioned
equivalence in the area of nursing while Maxwell and Shanahan (2001) raised the issue with regard to
education. Kirkman et al. (2007) suggest that as the research and thesis component is shorter than in
a traditional PhD that the same depth of argument cannot be sustained. The clear implicate of these
claims is that the professional practice level of argument is less than that contained in the more
traditional PhD thesis. One important aspect of adjudication in answering these concerns relates to
whether or not a professional doctorate demonstrates a novel contribution to a field by the generation
of new knowledge, an area addressed below.
The traditional PhD has a long history of requiring the graduate to make an important and original
contribution to an academic field. However, while not undermining the essential nature of this
criterion, a survey based investigation by Nyquist (2002) and a longitudinal study by Wulff et al.
(2004) have indicated that this traditional, academic focus may not prepare a doctoral graduate who is
well equipped to transpose their learning into the rapidly changing praxis of the modern work based
environment. In contrast, the professional doctoral graduate should be capable generating creative
methods of investigation into problems and of developing modalities of investigation and problem
solving. While this practical problem solving is important, any research methods used at doctoral level
and beyond should also be rigorous, demonstrating reliability, validity and be able to withstand
vigorous peer review.
Most ‘real world’ problems are complicated and nuanced. This means they may well not be amenable
to investigation using a stereotyped and established research methodology. Rather they often require
the development of a bespoke modality of investigation, orientated to an analysis of the problem and
the generation of specific recommendations to deal with the issue. It therefore follows that the
professional researcher needs a knowledge of traditional methods of research based investigation,
but must in addition be able to adapt these to meet a particular need and also to generate new
investigative techniques a particular problem might well demand. Indeed Fulton et al. (2012) arguably
departs further from the traditional methodologies of research based investigation. Rather they put the
focus on identification and analysis of the experience of the individual doctoral candidates. This is
because the professional doctoral candidate will bring a previously established portfolio of work to
their doctoral course. Their overarching framework is not tied to a particular established methodology,
but rather is linked to whatever method will best answer the research’s work based problem. Despite
this somewhat bespoke approach it is vital to ensure a coherence and logical epistemological
consistency throughout the thesis.
As the professional practice doctoral graduate is already (by definition) a field expert, this promotes
the likelihood of synergistic developments between the professional knowledge and the new research
based ways of thinking and working. This means the professional practice PhD graduate is in a
unique position, possessing years of professional experience and a research training. This allows for
the probability of ongoing significant development of the professional field in the years of practice
before and after doctoral graduation. Another result of this synergy is that conducting research
becomes a normal part of their professional repertoire. This means that a research mentality
essentially becomes a new norm, allowing for enhanced and ongoing professional problem solving. In
455

John Lorimer Campbell, Robert Campbell and Gill Green
the case of the current researcher, it is hoped that this will facilitate the ongoing and novel
development of methods for improving the undergraduate and postgraduate education of health care
professionals worldwide, with a cascading beneficial effect on numerous patients.
5. Development of new knowledge
Kirkman et al. (2007), in similar thinking to Lester (2004), states that a professional practice PhD is a
form of taught doctorate, often involving close association with a particular discipline such as
psychology, business administration or education. However, Fulton et al. (2012) have a much more
eclectic view of the professional doctorate pointing out that it is an umbrella term, covering many titles
and modes of delivery. Presenting a study focused on work-based activity, involving development of
an area of practice, Fulton et al. greatly expand on the meaning and implication of a professional
doctorate. Arguing from the work of Gibbons et al. (1994), Fulton el al. describe the epistemological
basis for the professional doctorate as ‘Mode 2 knowledge acquisition’. Likewise, Lester (2004) also
advocates that ‘Mode 1 knowledge acquisition’ underpins the traditional route PhD, while Mode 2 new
knowledge underpins the professional doctorate.
Gibbons et al. (1994) claimed to have identified a new modality for the development of knowledge.
This new way of generating knowledge is real world context-driven, focused on problem solving and is
typically interdisciplinary. Knowledge is generated in the ‘context of application’. Gibbons described
this new modality as ‘Mode 2 knowledge production’. This Mode 2 knowledge is generated in practice,
involves reflexivity, and has the potential to impact on practice. The converse of this modality is ‘Mode
1 knowledge acquisition’, which uses the traditional methods of academic based research to acquire
new knowledge. Mode 1 knowledge acquisition is academic, investigator-initiated and usually focused
within a single discipline.
The Mode 2, interdisciplinary approach is consistent with the more generic, broad based focus to
doctoral studies advocated by Doncaster (2000) and Bond and Tennant (2006). The generic
approach attracts practitioners from a wide range of backgrounds, as opposed to a single discipline
focus. The philosophy behind this generic professional doctorate relates to the development of
professional practice, in the work based, real-world context. Doncaster (2000) suggests this will
generate a ‘broad based capacity as opposed to a discipline based experience’.
There is no claim that Mode 1 or Mode 2 knowledge acquisition modalities are inferior or superior to
each other. Rather they can both be seen as methodologies of new knowledge development. Indeed,
they should be seen as complementary forms of knowledge acquisition. They are both valid if
assessed by the pragmatic criteria of the existence of the new knowledge the Modes have generated.
While the current research has been developed within a specific discipline related area, it has
principally used a Gibbons’ Mode 2 approach to knowledge acquisition. This is consistent with Yam
(2005) who considers that nurse education is more akin to a Mode 2 approach, based on the
assertion that nursing and nurse education are both practical disciplines.
By adopting rigorous approaches which can then be applied to practice, the ACOP has the potential
to transcend professional and geographical boundaries. As a result it can be argued that it has
potential interdisciplinary implications. For example it is entirely possible for other academics and
professional specialists to use the ACOP approach to disseminate knowledge in any discipline. If
others follow this example, high quality educational and training materials could be supplied to an
international student base, even if those students were unable to pay. This could result in the
empowerment of individuals and communities all over the world, who would then be able to develop
expertise in fields as diverse as economics and finance to medicine, engineering or agriculture. The
direct route, from professional to learner, bypassing publishers has the potential to transform
international professional education and practice. This is especially true with e based materials, where
a whole course could be supplied, at low cost, on a laptop or e reader.
This thinking is completely consistent with the summary of the professional doctorate provided by
Maxwell (2003) as a vehicle drawing together the state of the art in professional practice, interrelated
with relevant academic theory which is then applied to the solution of work-based problems. As a
result of this problem solving process there will be a change the doctoral student’s own workplace, as
results are applied in practice.
456

John Lorimer Campbell, Robert Campbell and Gill Green
This does however, give rise to difficulties in assessment of levels attained. Mode 1 knowledge is
academically based with a long tradition, meaning that there are traditional established criteria for
quality control. However, as Mode 2 knowledge is concerned with application in context, novel
methods of quality control become necessary. Validation of doctoral levels also requires a grappling
with these problems.
A possible criterion for evaluation of doctoral level studies is that the study has attained a level of
interrogative and conceptual thinking which, when applied to practice, has the ability to facilitate
developments or even transformations in professional practice. If these criteria are accepted, it would
provide a pragmatic quantification of ‘interrogative and conceptual thinking’. This would mean that the
quality of the thinking is evaluated by the practical outworking in the practice context.
This is consistent with work by Fealy (1997) working in the field of nurse education. He developed a
classification of how theory could be applied to practice based on four components. Firstly was the
traditional application of applying theory to inform practice. This is commonly applied at
undergraduate level, where it is necessary to provide literature based rationales for claims or
interventions.
Secondly, he pointed out that theory can be extrapolated from practice or tacit knowledge. This form
of ‘reverse engineering’ certainly applies to the ACOP, where resources have been pragmatically
developed in practice, often applying tacit knowledge. Having developed the materials, and probably
more importantly the approach or problem solving methodology, new theory has been developed from
this practical base.
Thirdly, Fealy (1997) discussed what he referred to as the moral imperative; this involves an ability to
identify moral elements that inform a course of action. The ACOP would make claims to be a moral
course of action as it provides educational materials, which have the potential to benefit the learners
in positive directions. It does this by providing accessible educational materials to improve the practice
of individual health care workers and through them to improve the lives of their patients. This has the
potential to empower learners and their patients.
Fourthly, is critical theory; this involves the ability to see the wider politico-social implications and then
use this perception to plan a particular approach. The potential wider implications of the ACOP
approach have already been discussed. However, politically, if the approach were to be widely
adopted, it would certainly have the potential to upset current vested interest and generate a broad
based, hopefully democratic, empowerment. These changes could be considered to be political.
6. Mapping the characteristics of mode 2 new knowledge production
Ongoing work by Nowotny, Scott and Gibbons, (2002) developing the original Gibbons et al. (1994)
work, went as far as suggesting that that the production of knowledge and the processes of research
were being radically transformed by the incorporation of Mode 2 acquisition. They continued to
support the Mode 2 concept which was ‘socially distributed, application-oriented, trans-disciplinary,
and subject to multiple accountabilities’. Conversely they critiqued the traditional Mode 1 paradigm as
‘characterized by the hegemony of theoretical or, at any rate, experimental science; by an internallydriven
taxonomy of disciplines; and by the autonomy of scientists and their host institutions, the
universities’. Such a radical shift (or at least addition) in the modality of new knowledge acquisition
clearly cannot be accepted at face value, but should be vigorously critiqued in order to evaluate the
validity of the claims. These components of Mode 2 will now be inter-related with the current
professional doctorate under the headings clarified by Nowotny, Scott and Gibbons, (2002).
6.1 Mode 2 knowledge is generated within a context of application
The work discussed in the current professional doctorate was not generated from pure science then
‘applied’ or ‘transferred’ into the practical environment. Rather the research and problem solving took
place in the environment in which the problems arose. Methodologies were developed in situ, and
applied in the same context. Problems in people’s health were identified, as were difficulties and
obstacles in the education of those who strove to reduce the burden of ill health. The current
researcher then developed strategies to help the helpers, i.e. to educate those directly involved in
professional health care duties.
457

John Lorimer Campbell, Robert Campbell and Gill Green
6.2 Mode 2 uses a trans-disciplinary approach
This relates to the utilization of a spectrum of theoretical concepts and practical methodologies to
solve problems. These do not need to be derived from previously existing thinking, and so leave room
for creativity. This allows for the application of expertise embodied in the individual researcher or in
research teams. The approach might well be highly eclectic. Conversely, in Mode 1 modalities, the
research methodology is encoded in the particular tradition of the research method being employed.
This aspect of Mode 2 is illustrated in the methods of explanation used by the current researcher.
These ‘techniques of explanation’ relate to material communicated in diagrams and models as well as
words. The way a concept is explained is written, essentially during one session, by the researcher.
However, the experience to communicate a complex concept in understandable terms has been
accrued over many years of clinical and teaching experience. It is not possible, and is even simplistic
to try and reduce this work based innovation to a single approach. Rather each explanation
represents a synthesis of clinical experience, theoretical knowledge and educational ‘know how’. New
approaches are generated, but they emerge from the multiple experiences of the researcher.
6.3 Diversity of the sites at which knowledge is produced, and in the types of
knowledge produced
The first component has always been present to some extent in Mode 1, in that research communities
have always existed. While these communities have often been international in nature, use of
technological advances allows for international interaction which is unconstrained and instantaneous.
This effect undermines the traditional Mode 1 hierarchies. While it could be argued that this
unconstrained communication allows a potentially chaotic free-for-all, it could also be suggested it
allows a more meritocratic order to emerge. Good ideas will be preserved and poor ones will atrophy.
Ideas are therefore subject to a form of natural selection, with only the fittest surviving and
reproducing.
As a result of improved communications Nowotny, Scott and Gibbons, (2002) suggest that these open
frontiers have allowed several new kinds of ‘knowledge’ organizations, including think-tanks,
management consultants, and activist groups, to join the research community. In this same way the
current researcher has established an international ‘community’ particularly through the use of online
interactions such as YouTube. International critique is now instantaneous, informing new
developments and inspiring ‘budding’ of the methods used in a wide variety of secondary contexts.
6.4 Mode 2 knowledge is highly reflexive
This means that the research process is not characterized as an ‘objective’ investigation of previously
existing phenomena. Rather the development of new ideas occurs automatically in response to
altered circumstances. Research becomes a dialogue, an ongoing ‘conversation’ between research
and ‘subjects’, usually in a real world context. The current researcher has interacted with his client
groups directly and via electronic communications. In addition he has interacted with his client group’s
client group. In other words he interacts with learners and the people they treat in their own
professional context. This allows for two levels of interaction, both of which influence the future
modalities and direction if his research.
This reflexive interaction has implications for accountability as new knowledge will have predictable
and non-predictable outcomes. Consequences, like applications, can no longer be considered to be
external to the research process. Rather, consequences will be felt within the problem solving
environment, which will in turn also influence topic choice and the ongoing and adapting research
process.
6.5 Novel forms of quality control apply to mode 2 generated new knowledge
Clearly, with any new knowledge there must be a process of validity verification. This is essential to
ascertain if some claimed new knowledge has an objective basis. It is necessary to determine if there
is internal and external validity. Internal validity will consider how consistent components of the new
knowledge are, to see if they are mutually supportive or contradictory. External validity will determine
if the new knowledge is consistent or inconsistent with already established knowledge.
458

John Lorimer Campbell, Robert Campbell and Gill Green
For new knowledge to be accepted into the ‘cannon’ of knowledge there must be a process of
assessment, discrimination, prioritization, and selectivity. These criteria will then inform the
acceptance of the new knowledge, which will in turn influence dissemination of the knowledge and
inform policy-makers and funding agencies.
One difficulty in the quality control of new Mode 2 generated knowledge is that there may be a lack of
an identified hierarchy. Scientific ‘peers’ can become difficult to identify. For example, the current
research draws feedback from an international community of students as well as peers. This is
consistent with Mode 2 thinking, which assumes the research process is joined by more interested
parties such as orchestraters, brokers, disseminators, and users.
A further difficultly in quality control relates to research questions being framed in a wide context. This
means that traditional reductionist forms of quality control can be difficult to apply. Far from being a
disadvantage, the current researcher views the broad dissemination of quality control to be strength of
the ACOP. Wide dissemination of materials allows for wide spread generation of feedback. It also
opens up the opportunity for a new criterion of quality control assessment. This is the pragmatic
assessment of how useful clinicians find the materials in supporting their own professional practice. In
this context, the new knowledge generated by the current researcher is ‘true’ if it can be applied to the
real world of human health and used to help people.
7. Communities of practice provide empirical testing of new developments
Newbury (2002) found that doctoral learning processes were frequently of a solitary nature, with
students often working in relative isolation. Early work by Wenger (1997) had suggested that
‘communities of practice’ (CoP) are based on a working team or group, involved in shared learning.
Shacham and Od-Cohen (2009) added that a CoP share a concern or a passion for what they do, and
learn how to improve performance via shared activity. Such communities can avoid doctoral student
isolation while facilitating external support by participating in collective doctoral research. Burkitt et al.
(2001) considered learning in CoPs to be a key element of work, and highlighted the importance of
social engagement in learning and in the process of new knowledge creation. For Burket el al.
learning was seen as an ‘embedded, contextualised performance likely to include immersion in the
practice world’. As illustrated by the ACOP, the current researcher has also been embedded in this
professional ‘practice world’. This has allowed time for significant interaction with those communities,
and the constant feedback this provides. As a result, the new materials and modalities of educational
delivery he has developed have been empirically tested within these professional communities. The
feedback received from these communities of practice constitutes an ongoing empiricism amounting
to a research method. This has allowed for the ongoing evaluation and modification of the materials
he has developed in the ‘laboratory’ of the real worlds of clinical and educational practice. His
situation is therefore the converse of that of isolation seen as a potential risk by Newbury (2002).
References
Burkitt, I. Husband, C. McKenzie, J. Torn, A. Crow, S. (2001). Nurse education and communities of practice.
London: ENB.
Boud, D., Tennant, M. (2006). Putting doctoral education to work: challenges to academic practice. Higher
Education Research and Development. 25(3)293–306.
Dickerson, B.E., Myers, D.R., Seelback, W.C., Johnson-Dietz, S. (1990). A 21st century challenge to higher
education: Integrating the older person into academia. In R.H. Sherron D.B. Lumsden (Eds.), Introduction to
educational gerontology (pp. 297–328). New York: Hemisphere Publishing.
Doncaster, K. (2000). The Middlesex University professional doctorate: a case study. Continuing Professional
Development. 3,1–6.
Fealy, G. (1997). The theory practice relationship in nursing: an exploration of contemporary discourse. Journal
of Advanced Nursing. 1061–1069.
Fulton, J., Kuit, J., Sanders, G., Smith, P. (2012). The role of the Professional Doctorate in developing
professional Practice. Journal of Nursing Management. 20, 130–139.
Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., Trow, M. (1994). The New Production of
Knowledge: The Dynamics of Science and Research in Contemporary Societies. Sage. London.
Kirkman, S., Thompson, D.R., Watson, R., Stewart, S. (2007). Are all doctorates equal or are some ‘‘more equal
than others’’? An examination of which ones should be offered by schools of nursing. Nurse Education in
Practice. 7, 61–66.
Kuipers, J.L. (2011). PhD and EdD Degrees for Mid-Career Professionals: New directions for adult and
continuing education. no. 129. Wiley Periodicals, Inc. Published online in Wiley Online Library
(wileyonlinelibrary.com). DOI: 10.1002/ace.401.
Lester, S. (2004). Conceptualising the practitioner doctorate. Studies in Higher Education. 29(6)757-70.
459

John Lorimer Campbell, Robert Campbell and Gill Green
Manheimer, R.J., Snodgrass, D.D., Moskow-McKenzie, D. (1995). Transforming older adult education: An
emerging paradigm from a nationwide study. Educational Gerontology. 21, 613–632.
Maxwell, T. (2003) From first to second generation professional doctorate. Studies in Higher Education. 28
(3)279–291.
Maxwell, T.M., Shanahan, P. (2001). Professional doctoral education in Australia and New Zealand: reviewing
the scene. In: Green, B., Maxwell, T.W., Shanahan, P. (Eds.), Doctoral Education for Professional Practice:
The Next Generation?, vol 2. Kardoorair Press, pp. 17–37.
Newbury, D. (2002). Doctoral education in design, the process of research degree study, and the ‘trained
researcher’. Art, Design and Communication in Higher Education. 1(3)149–159.
Nowotny, H., Scott, P., Bibbons, M. (2002). ‘Mode 2’ Revisited: The New Production of Knowledge. Minerva. 41:
179–194.
Nyquist, J.D. (2002). The Ph.D.: A tapestry of change for the 21st century. Change: The Magazine of Higher
Learning. 34, 12–20.
Pearson, A., Borbasi, S., Gott, M. (1997). Doctoral education in nursing for practitioner knowledge and for
academic knowledge. Image-the Journal of Nursing Scholarship. 29 (4), 365–368.
Quality Assurance Agency for Higher Education (2011) Doctoral degree characteristics. QAA Publications.
Available from www.qaa.ac.uk
Shacham, M., Od-Cohen, Y. (2009). Rethinking PhD learning incorporating communities of practice. Innovations
in Education and Teaching International. 46(3)279–292.
Shapiro, S. A. (2001). “From Andogogy to Collaborative Critical Pedagogy: Learning for Academic, Personal, and
Social Empowerment in a Distance Learning PhD Program.” Journal of Transformative Education. 1(2)150–
166.
Stokes, P. J. (2005). “Hidden in Plain Sight: Adult Learners Forge a New Tradition in Higher Education.”
Washington, D.C.: Secretary of Education’s Commission on the Future of Higher Education. 2005.
UK Council for Graduate Education, (2012). Professional doctorates. Available from:
.
Watson, R., Thompson, DR., Amella, E. (2011). Doctorates and nurses. Contemporary Nurse. 38(1–2):151–159.
Watts, JH. (2009). Points of Departure, From professional to PhD student: challenges of status transition.
Teaching in Higher Education. 14(6) 687-691.
Wenger, E. (1997). Communities of practice: Learning, meaning, and identity. Cambridge, UK: Cambridge
University Press.
Woodley, A., Wilson, J. (2002). British higher education and its older adults. Higher Education. 44, 329–347.
Wulff, D.H., Austin, A.E., Nyquist, J.D., Sprague, J. (2004). The development of graduate scholars as teaching
scholars: A four-year longitudinal study. In D.H. Wulff & A.E. Austin (Eds), Paths to the professoriate:
Strategies for enriching the preparation of future faculty (pp. 46–73). San Francisco: Jossey-Bass.
Yam, B.M.C. (2005). Professional doctorate and professional nursing practice. Nurse Education Today. 25, 564-
572.
460

Can Post Modernism Contribute to Saving the World?
Christine Gilligan
Sheffield Hallam University, Sheffield UK
c.k.gilligan@shu.ac.uk
Abstract: As human beings, our future sustainability could be under threat if we continue to ‘trash the planet’.
Sustainable development, suggested as a way of addressing the environmental, economic and social problems
ahead, has fundamental implications for the governance of modern society. In a complex system, such as the
Earth, linear analysis and rational planning are no longer considered appropriate, and new ways of understanding
change are needed. Post modernism offers an alternative approach and a different way of perceiving the world
and its problems. It is however, criticised as almost useless because its plurality of competing dialogues is said to
undermine the basis for meaningful change. Revisionary post modernism may provide a way forward. This paper
examines a theoretical approach to exploring discourses around sustainable development in non environmental,
voluntary organisations using revisionary post modernism, based on a belief that in complex systems small
changes can bring about major change and that voluntary organisations, working at a local level, can bring about
change in their communities. It highlights how a post modern approach could contribute to securing a more
sustainable future, by helping us understand the importance of local narratives in co-creating the future.
Keywords: sustainable development, revisionary post modernism, complex systems, change, voluntary sector
1. Introduction
There is increasing recognition that socio-cultural wellbeing, economic wellbeing and environmental
wellbeing cannot be achieved independently of each other (Capra 1997, Porritt 2005). Climate
change in particular, poses serious threats to economic development (Stern 2006), and Mckibben
(2007) suggests that as economic growth bumps against physical limits it may be impossible to
continue to expand the economy. Furthermore it is generally the poor, already disadvantaged in terms
of wealth distribution, who are likely to suffer disproportionately from the environmental problems.
(Hawken 1993)
Sustainable development has been suggested as a way forward. (Porritt 2005) It is about redefining
how we live in the world, a process rather than an outcome, and requires changes in behaviour
across all sectors of society: government, business and individuals. Rationalism and linear analysis
are no longer considered adequate to deal with the challenges presented by sustainable
development, because they ignore the complexity of the situation, leading to the belief that there are
simplistic solutions. (Voss et al. 2006, Smyth 2006) Post modernism, defined as a rejection of the
meta-narrative of modernity and rationalism (Hassard 1993), offers an alternative approach more
appropriate to the complex, multidimensional dynamic world we inhabit. Unlike rationalism, which
shuts down alternative voices, post modernism opens up possibilities, and provides a different way of
theorising, structuring and organising that could generate new understandings and offer emancipatory
potential by acknowledging the importance of disparate localised narratives in the co-creation of
reality. (Cilliers 1998, Hearn and Parkin 1993).
Post modernism has been criticised because it offers a plurality of competing representational
dialogues that undermine the basis of meaningful change, (Burrell 1993:82) but Sterling (2003)
suggests revisionary post modernism overcomes this problem. Revisionary post modernism is a third
order, ecological worldview that recognises our fundamental embeddedness in wider natural systems,
and unlike deconstructive post modernism, it acknowledges the concept of the ‘whole’, as an
interdependent pattern of agents interacting with each other and with their environment. The ‘whole’ is
not an objective reality but a co-constructed temporary accommodation. In this respect revisionary
post modernism resembles complex systems theory, in which a system is defined as a multiplicity of
interacting, locally determined discourses. (Cilliers 1998) All agents in the system co- create the
system. Agents can be human and non human and relationships between agents determine the
nature of reality. This redefines the relationship between humans and the natural environment and
recognises reality as a dynamic co-construction, constantly open to revision.
This paper examines how revisionary post modernism can contribute to the understanding of a
complex problem like sustainable development by exploring discourses in the voluntary sector. The
voluntary sector was chosen because this sector works with those most likely to be aversely affected
by unsustainable development. Furthermore, the UK government believes that the voluntary sector
461

Christine Gilligan
has the potential to bring about the changes needed to create a more sustainable society. ‘Voluntary
and other non-profit organisations can mobilise millions of people in the fight against climate change
to help create and safeguard a better future,’ (Cabinet Office 2007). EAC (2007b) identified urban,
non environmental voluntary organisations as the least likely to engage in this agenda. The
methodology involved engaging in participatory conversations with a variety of stakeholders (13
stakeholders in 10 organisations, including: managers, workers, volunteers, trustees and service
users) to understand their perspectives around the need for behaviour changes to support sustainable
development. These were conversations rather than interviews, because a conversation suggests a
two way process, where the researcher acts as part of the process of knowledge creation, rather than
as an independent observer.
The aim of the research was about generating a different type of knowledge, an understanding of the
relationships between the agents, human and non human, in the system in an attempt to explore how
relationships in a dynamic system influence the system.
Firstly, the complex problem that is sustainable development is outlined, followed by a discussion
around the relationship between human beings and the natural environment, including the theory that
the Earth is a complex adaptive system, in which human beings are a part. Revisionary post
modernism as a way of researching a complex system is discussed in the context of sustainable
development, before briefly considering the role of the voluntary sector in bringing about change. The
conclusion discusses how in a complex system like the Earth, revisionary post modernism could offer
alternative understandings that may contribute to the creation of a more sustainable world.
2. Sustainable development - a complex problem
Sustainable development, first outlined in 1987 as a concept for social modernisation on a global
scale, focuses on the triple bottom line of social equity, environmental quality and economic
prosperity. (Porritt 2005, Bruntdland 1987) There are many definitions, which include phrases such as
vision, value change, moral development, or social re organisation (Gladwin et al 1995), and the
many interpretations (Lozano 2008) and the abundance of related terms, such as, environmental,
green, eco friendly, climate change and global warming, have confused the concept, diffused the
challenges and allowed particular interest groups to evade responsibility. (Gladwin et al 1995, Voss et
al 2006:3). The scale of the problem also means it can be seen as ‘too big an issue to deal with’,
(EAC 2007b:ii) and the complexity renders the possibility of a simplistic, universal solution unlikely.
Jacobs (1991) suggests that although sustainable development is a contestable concept and the
exact meaning can never be agreed, it does not mean that it has no value. It presents an opportunity
to introduce new ideas into debates around how we wish to live and the quality of nature we wish to
preserve, as a fundamental aspect of sustainable development centres on the relationship between
humans and nature. (Voss et al. 2006, Borland 2009) The anthropocentric world view, which sees
nature as a resource for humans to exploit, has lead to the problems of unsustainable development,
(Giradot et al 2001, Borland 2009) and according to Sterling (2003) we need a new way of looking at
the world, a cultural shift or third order change, which acknowledges that humans are just one species
amongst many and can not stand outside nature. The natural environment is part of the system on
which we depend, (Gladwin et al 1995:896, Voss et al 2006) and this realisation presents
fundamental implications for the governance of modern society, moving away from traditional, linear,
reductionist approaches.
Sustainable development can be seen as a guide for human behaviour, as opposed to a formula for
management practices, and rather than a quest for a desired end state it is about the capacity of
society to learn about the conditions of its future existence. (Blewitt 2010) Focussing on the process
of change rather than the end goal, it provides opportunities for groups to reshape the urban
environment and make it more equitable for disadvantaged groups. (Kiel and Desfor 2003)
Sustainable development is meaningful because it brings difference into a common dialogue, and
mutual agreement as the basis of collective action.
The problem therefore, is not confusion around the concept of sustainable development, rather it is
the current discourses around linear analysis, problem solving and the need for certainty that need to
be re examined as we face choices about the future. We will have to learn to live with diversity,
ambiguity and lack of control, where the only certainty is that all decisions will be made on the basis of
uncertain knowledge in an ever changing dynamic world.
462

Christine Gilligan
3. The earth as a complex adaptive system
‘The sheer scope of global activities combined with the interconnectedness and the
diversity of the world’s population and societies create an inter connected highly complex
system where what is done in one part of planet Earth affects what happens in other
parts.’ (Waddock 2007:546)
Sterling (2003) and Blewitt (2010) talk about a mismatch between the systemic world we inhabit and
the fragmented way we think about it and believe we need to encourage a more holistic view of our
relationship with the Earth. ‘The unhealthiness of our world today is in direct proportion to our inability
to see it as a whole’. (Senge 1990:68) Moving away from seeing nature as a passive resource to be
deployed, we must recognise it as a dynamic partner in our existence, an equal participant in our
networks. Latour (2004) seeks to explain social order through the networks of connections between
agents: humans, technology and objects (Couldry 2004) and for Latour, objects can only be defined in
relation to other objects. They become what they are and what they mean through social and
ecological relations, thus both nature and science are seen as human constructs and open to
interpretation. Replacing the split between nature and society with a dynamic community,
incorporating humans and non humans, challenges the dominant Western rational view based on the
separation of mind and matter, domination and control (Blewitt 2010). Other cultures have long
recognised our interdependence, the importance of living in harmony with nature, (Peate 2005) and
their responsibility, not only for themselves but for the world around them. We are in a co-evolutionary
relationship where each affects the other, and the natural world channels human interpretations of
both culture and nature as much as human interpretation of culture and nature are channelled by
socio historic pressures. (Argyros 1991)
Lovelock reflected this idea of interdependency in his Gaia theory (Lovelock 2000), suggesting that
the Earth is a self regulating, complex, dynamic web of interactions, of which humans are a part. In
such a system each part affects the other parts and each depends on the whole. A system cannot be
understood by analysing its parts, (Hatch 1997) and subsystem interdependence produces features
and characteristics that are unique to the system as a whole. Borland (2009) suggests that the current
problems have arisen because business models work from a linear framework and once we
understand that we are in a relationship with the natural environment our way of interacting with it will
change. Action to protect the natural systems on Earth will be understood as also being about
ensuring human sustainability.
Decisions are a product of the social contexts within which an individual is located (Blakie 2000) and
an understanding of the wider context that influences people’s behaviour towards the environment is
therefore an important step forward. In a complex system we are all co-creators of the future and
solutions will ‘depend on the development of shared understandings of the problem and an ability to
reframe system dynamics so that short term individual interest and long term sustainability and
development become more balanced and integrated.’ (Morgan 2006:271) An iterative, participatory
approach, that acknowledges the different views and opinions of people with different values, will help
us to understand the world differently and accept that in a dynamic environment, solutions can only be
local and specific to that time.
4. Revisionary post modernism
Post modernism is a basic ontological and epistemological attitude towards our perception of reality
that presents a philosophical break with positivism. It refutes generalisable meta-narratives and does
not attempt to impose a rationalist logic on events. (Johnson and Duberley 2000). Whereas
modernism relies on notions of expertise, truth, and objectivity, and attempts to predict the future
based on linear analysis, post modernism recognises plurality of perspective, relativity and a
multiplicity of truths none of which has more validity than any other. Objective truth is replaced by
narratives based on particular perspectives and this allows us to deconstruct frameworks, understand
reality in a different way, and challenge the validity of taken-for-granted ideas or meta-narratives that
depend on the exclusion of other narratives for their legitimacy. The underlying principle behind post
modernism is that knowledge is socially constructed and reality is changing and indeterminate, thus
rendering prediction difficult or impossible.
Revisionary post modernism, like traditional post modernism, recognises the transience of our lives in
a continuously changing environment.
463

Christine Gilligan
‘An uncertain future is a key reality of the human condition and forecasting the whole as
opposed to the parts of civilisational development is far beyond our abilities.’ (Smil
1993:32)
Unlike deconstructive post modernism however, revisionary post modernism, acknowledges the
whole, as an interrelated pattern of interacting agents in a system, rather than a collection of disparate
narratives, and relativity, often seen as a weakness of post modernism, becomes a strength, because
ambiguities and dissonances create tension that releases associations and new ideas. (Berg 1989)
Revisionary post modernism counteracts the dissociative alienation that has been a feature of
Western culture for centuries, and acknowledges the fundamental embeddedness of human beings in
their natural environment. (Sterling 2003) It provides a way of understanding the interaction between
natural systems and human activity and its co -evolutionary ontology, suggests that our every act is
involved in creating the world. By recognising the contribution of multiple perspectives to the whole
and by eroding the boundaries between things, i.e. the known from the observed, it removes the
distinction between reality and its representation. Reality as an independently existing reference point
is erased and language is no longer seen as representing reality but as creating it. To understand our
world therefore, we have to explore the different narratives that create it.
Revisionary post modernism has similarities with complexity. Complexity also recognises a multiplicity
of locally determined discourses and challenges the notion of predictability. (Cilliers 2006) In a
complex system, the behaviour of a system is characterised by multiple local discourses interrelating
in an open network, and information from the environment has a direct, non determinate influence on
the system, causing changes in the system but not determining the nature of these changes. Local
agents, in relationship with each other, can plan their own actions but they cannot plan the actions of
others rendering predictability impossible. Small changes can have large effects (Cilliers 2006,
McMillan 2004) but the change is not totally random. Trends or directions can be discerned and this
process is referred to as emergence (Stacey 2007). Development and evolution are not the result of
consensus, but of acted and expressed differences. (Berg 1989) Consensus can only be a local
phenomenon limited in time and space, a temporary accommodation because a system needs
dynamism to function and if consensus is the goal, the system would freeze. A healthy system is
always in flux. Prigogine (1997:71) suggests the adaptability of self organising systems to the
prevailing environment makes them extraordinarily flexible and robust.
Post modernism is often criticised for its inability to make value judgements, but from a revisionary
post modern perspective, society is an emergent property of social systems, and the relevance of
activities is determined by the effectiveness of their entry into the network. (Berg 1989) The distinction
between right and wrong, true and false is blurred and the evaluation of reality becomes a matter of
choice or acceptance. Credibility is more important than truth and propositions are judged according
to their usefulness or the support they gain (Cillliers 1998). A revisionary post modern research
approach to a complex system will allow dissenting and different voices to contribute to the future
without privileging them. The relevance of the differing discourses will be dynamically determined
through competition and cooperation in terms of the history and the changing needs and goals of the
system.
In a complex system where instability and constant change make prediction difficult or impossible,
instead of trying to control change, we must acknowledge it as a constant feature of our lives (Stacey
2007) and McMillan (2004) suggests complexity has major implications for how we manage change.
Development is a result of the interactions in the system and for a system to operate effectively it
needs a multiplicity of interactions. (Stacey 2007) A revisionary post modern approach to researching
sustainable development will examine a variety of differing discourses to explore how change can
emerge as a result of the interactions.
5. Revisionary post modernism: application to sustainable development
Luhman (1985) outlines features of modern society that resemble a complex system:
Large numbers of elements (human and non human), interacting dynamically
Many connections in many different capacities
Non linear interactions in which the same piece of information has different effects on different
individuals
464

Christine Gilligan
Short range interactions determined locally with no meta-level control of information
Within this system, sustainable development can be seen as a complex problem that will require an
approach to problem solving that differs from the traditional linear, rational approach. Complex
problems arise ‘from environments characterised by turbulence and uncertainty.’ They are ‘valueladen,
open-ended, multidimensional, ambiguous and unstable.’ Klein (2004:4) and ‘they resist being
tamed, bounded or managed by classical problem solving approaches.’
All agents in the system are responsible for co- creating the future through interactive networks and
no single agent is any more important than any other. If sustainable development requires us to
address questions about the kind of world we want to live in, we have to think about how we
understand the world and our role in creating the future. If we view the Earth as a commodity to be
bought and sold our responses will be different than if we recognise the Earth as an interactive
system in which we are an intrinsic part. In terms of sustainability our long range future will be
dependent on us all understanding that a system that destroys its environment destroys itself and that
we have a role to play in sustaining the environment.
From a complexity perspective it is the actions of the multiple, independent actors at local level that
create the changes in the system. Creativity and innovation emerge, not from the intention of
individual agents, but from interactions. (McMillan 2004) This suggest a need for dialogue or rather
many dialogues, exploring our relationship with each other and with the natural environment and post
modern research is about exploring narratives. (Johnson and Duberley 2000) Empirical research from
a post modern perspective is about gaining an understanding of a situation at a particular point in
time, recognising that there are many possible interpretations. It challenges the content and form of
dominant models of knowledge to produce new forms of knowledge and gives voice to those not
represented in the dominant discourses. (McAuley et al 2007:250.) We have all contributed to the
creation of the current way of life and to find a way forward the voices of voluntary sector stakeholders
at a local level, are as important as those of governments. My research attempted to put into practice
a revisionary post modern approach, working with a group of stakeholders from non environmental,
voluntary organisations in an urban environment, to explore their understanding of sustainable
development. It aimed to develop inclusive narratives around sustainable development that could
contribute to local action and encourage emergence by exploring different understandings of our
relationship with nature and with each other in a dynamic interplay.
6. The voluntary sector and sustainable development
Although most organisations are quite small, Hale (2008) feels the third sector holds the key to
mobilising public concern around sustainability. The capacity of the sector as an agent of change is
widely recognised (Tandon and Mohanty 2002, Schwabenland 2006) and it can bring people together,
explore differences and ‘create opportunities for conversations about the world: how it is; how it could
be; how it should be.’ (Etherington (2008:3) The sector can bring new issues and different
perspectives into public policy debates, provoke public debate, give voice to the disenfranchised and
its proximity to citizens and communities makes it more trusted than business or government. (Buchs
et al 2011) The government acknowledges the sector’s ability to influence sustainability. (HM
Government 2007) ‘The thousands of organisations that make up the third sector are powerful forces
for change in our society – and it’s a force we need on our side in the fight against climate change.
We know that climate change will hit the world’s poorest and most vulnerable people first, both here
and abroad.’
Although the UK has one of the more active and visible movements on climate change, and through
organisations such as Greenpeace, FOE, WWF, the voluntary sector has been at the forefront of
awareness raising, campaigning and providing information, there is a dearth of research evidence
about organisational change in the sector specifically in relation to sustainability (EAC 2007a). Many
non environmental voluntary organisations don’t understand the need for change or see it as relevant
to their organisation, and there appears to be a relatively poor understanding of sustainability at a
local level, even though the ethos of care, linked to social justice, suggests sustainable development
is an area the sector should be considering because of its impact on the poor. (Baring Foundation
2007) A further complication is that the term sustainability in the voluntary sector is commonly taken
to mean organisational or financial sustainability and related to the need to secure funds. (Big Lottery
2006)
465

Christine Gilligan
Therefore, although there is a belief that voluntary organisations are important agents of change,
unless the actors at local level, voluntary organisations and their stakeholders, understand that they
are part of the problem and part of the solution, it is unlikely that change will happen. My research
used a participatory approach with a small group of stakeholders from non environmental voluntary
organisations in an urban environment to explore their understanding of sustainable development. In
an attempt to develop inclusive narratives around sustainable development that could stimulate local
action, the aim was not to provide definitive outcomes or solutions, but to generate a different type of
knowledge, an understanding of the relationships between the agents, human and non human, in the
system to explore how relationships in a dynamic system could influence the system.
This approach could help individuals in organisations recognise their role in the creation of society
and to recognise the natural environment as a legitimate stakeholder, rather than an externality
outside of their responsibility. (Banerjee 2008) This new understanding may contribute to a wider
acceptance by organisations of the need to consider sustainable development in their operations and
help them explore different ways of working at a local level that encourages emergence of new ideas.
7. Methodology
The research took place in a major UK city whose strategy was to make the city ‘an attractive and
sustainable low carbon city’ based on a vision of environmental excellence. A city location was
chosen because research identified that urban, non environmental sector organisations were the least
likely to change.
Conversations were arranged with a variety of stakeholders: managers, employees, trustees,
volunteers and service users, a total of 13 people in 10 organisations. The approach attempted to
invoke co-participation, where the interviewer was a co-creator of the knowledge. This reflects a
complex system in which all agents interact to co-create the knowledge and the researcher cannot
stand outside this process as a rational objective observer. A conversational approach was chosen
over interviews because ordinary conversations with no clearly defined objectives can lead to the
formation and transformation of ideas and concepts as one theme triggers another in an apparently
random way. (Kuhn and Woog 2005) However, there were weaknesses in my approach. It was an
unequal, unbalanced relationship and the artificiality of the situation and the power dynamics may
have meant that the participants did not see it as a conversation and merely attempted to answer the
questions as per an interview (Gubrium and Holstein 2003). In an attempt to overcome this I made an
effort to be open about my views and opinions and encouraged participants to discuss their own
interests and ideas in the hope that new insights and understanding would emerge spontaneously as
a result of the interaction. As a social researcher it was not my aim to design specific emergences but
to support and enable conditions that may inspire movement in certain directions. The aim of the
conversations was to develop a qualitative, contextual relationship that would help participants
understand their own role in world and their relationship with the natural environment through
exploring the concept of sustainable development.
As well as recognising my role in co-creating the knowledge generated from the interactions, I have to
recognise my role in interpreting the data collected. Just like reading is a creative process, where the
reader brings their own understandings to the process, interpreting the words of others operates in
the same way. Post modern researchers should be humble about their findings, and recognise their
role in the construction of these findings through a process of reflexivity. (McAuley et al 2007) This is
especially important in the co-creative process of revisionary post modernism.
8. Findings
All conversations resulted in an ‘emergence’, an unplanned outcome. For example, one manager
remembered that he had intended to do a travel audit but had not done so. Another, who at the
beginning of the conversation intimated that sustainable development was not an issue for his
organisation to consider because they were too small to make any difference, realised in the course
of the conversation that ‘if you look at the Third Sector collectively it’s big. If you look at it individually
it’s small - that’s probably the dilemma.’ He began to understand the bigger picture and at the end of
the conversation he said he would discuss sustainable development at the next board meeting,
adding, ‘It’s (sustainable development) probably one of the most important issues of our time.’
Although these can not be documented as examples of changes in behaviour they highlight how, just
by having a conversation, agents can understand a situation differently. According to McMillan
(2004:74) this reflection can encourage new mental models of the world which, in a complex system,
466

Christine Gilligan
can contribute to change. ‘Transformation is achieved by a continuous process whereby changes in
individuals and groups that arise from individual and group learning experiences change the culture
and behaviours.’ A weakness of my research was that I did not build in opportunities to revisit the
organisations to see what changes, if any, had occurred since my initial visit.
9. Conclusion
This paper has explored how a revisionary post modern approach could provide a new way of
exploring sustainable development. Voluntary organisations were chosen as the subject because of
the belief that the voluntary sector has the potential to act as a catalyst of change, and as a values-
based sector, would be supportive of changes that may benefit their service users.
Sustainable development is a complex, political, social and economic concept that requires
negotiation around vision and value change, and changes in the way we organise our lives that will
not be realised unless we better understand the need for the changes. Sterling suggests that complex
issues in the modern world, especially those around the natural environment, can only be understood
and effectively addressed by holistic, flexible, integrative approaches which help us understand things
systemically. (Sterling 2003, Capra 1997) Western civilisation is not the result of any kind of
calculated long term planning. Individuals do not form an intention to change civilisation and then
gradually realise this intention through rational purposive measures. Society is created by the
interplay of intention and actions of many people to produce a pattern of development in a particular
direction. (Elias 2000) This co-evolution, a core aspect of complexity, suggests that human beings,
through interaction, both make and are made by the environment and are thus actors in their own
evolutionary history, and in a complex system local agents acting in their own interest can bring about
large scale change. The key elements in the system are relationships as new ways of working
emerge through a process of discussion and negotiation where people and organisations come
together to create understanding.
I tried to highlight how a post modernism approach could facilitate research in a complex system.
Ontologically, both post modernism and complexity regard matter and consciousness not as separate
but as complementary aspects of life, engaged in a continual process of reconstruction and from this
perspective, research that helps us better understand the dynamic complexity in which we co-exist
may offer the potential for appropriate change. Any vision for the future about how we want to live will
be the outcome of a collective discourse. (McMillan 2004) Human dynamics bring about real change
and it is through interactions with others that new ideas and new ways of seeing the world emerge as
we move away from linear thinking. Berg (1989) suggests that post modernism allows us to remove
the human agent as the rationalising power and helps us learn to live with interdependence. Post
modern research may, therefore, set us free, not to do as we like, but to restore to agents the fullness
of moral choice and responsibility whilst simultaneously depriving us of the comfort of the universal
guidance that modernism promised.
References
Argyros A.J. (1991) ‘A Blessed Rage for Order: Deconstruction, Evolution and Chaos’, University of Michigan
Press, Ann Arbor, quoted in Cilliers op.cit. pp135-6
Banerjee S.B. (2008) ‘Corporate Social Responsibility: the Good, the Bad and the Ugly,’ Critical Sociology, 34(1)
51-79
Baring Foundation (2007) www.baringfoundatin.org.uk/specialinitiatives.htm accessed 15/12/10
Berg Per Olaf (1989) ‘Postmodern Management? From facts to fiction in theory and practice’, Scandinavian
Journal of Management, Vol 5, No 3 pp 201-217
Big Lottery Fund, Sustainable Development Web Resource
www.biglotteryfun.porg.uk/sustainable_development_web_resource.pdf accessed 12/10/06
Blakie N. (2000) ‘Designing Social Research’ Polity, Cambridge
Blewitt J. (2010) ‘Understanding Sustainable Development’, Earthscan, London
Borland H. (2009) ‘Conceptualising global strategic sustainability and corporate transformational change’,
International Marketing Review, Vol 26 Nos 4/5 pp554-572 www.emeraldinsight.com/0265-1335.htm
accessed 6/7/10
Brundtland Commission: The World Commission on Environment and Development (1987) ‘Our Common
Future’, Oxford University Press, Oxford
Buchs M., Smith G., Edwards R., (2011) ‘Low Carbon Practices: a third sector research agenda’, TSRC May
2011 working paper 59
Burrell G. (1993) ‘Eco and the Bunnymen’ in Hassard and Parkin eds. op.cit pp71- 82
467

Christine Gilligan
Cabinet Office, Office of the Third Sector, ‘Overview and Third Sector Review’, November 2007
www.cabinetoffice.gov.uk/third_sector accessed 28/06/08 (Environment Secretary at the launch of the Third
Sector Declaration on Climate Change for Third Sector Organisations)
Capra F. (1997) ‘The Web of Life: a new scientific understanding of living systems’, Anchor Books, New York
Cilliers P. (1998) ‘Complexity and Postmodernism: understanding complex systems’, Routledge, London
Couldry N. (2004) ‘Actor Network Theory and the Media: do they connect and on what terms?’, in Hepp et al
(eds) (2004) http://www.andredeak.com.br/pdf/Couldry_ActorNetworkTheoryMedia.pdf
EACa,CAG consultants, ‘Every Action Counts Evaluation Baseline’ April 2007
EACb, CAG consultants, Susan Parham, ‘A background paper on organisational change in community and
voluntary sector organisations’ November 2007
Elias N. (1939 and 2000) ‘The Civilising Process’, Blackwell, Oxford in Stacey R.D. (2007:245-9)
Etherington S. Chief Executive of NCVO, speech to Sheffield Hallam University’s Centre for Voluntary Sector
Research, 23/10/08 ‘Does a strong and independent third sector mean a strong and healthy civil society?’
Giradot N.J., Miller J., Xiaogan Liu (eds.) (2001) 'Daoism and ecology', Harvard Centre for the Study of World
Religions, Harvard University Press, Cambridge, Massachusetts pp361
Gladwin Thomas N. Kennelly James J. and Krause Tara Dhelomith, (Oct 1995) ‘Shifting Paradigms for
Sustainable Development: Implications for Management Theory and Research’, The Academy of
Management Review, Vol 20, No 4, pp.874-907
Gubrium J.F. and Holstein J.A. eds (2003) ‘Post Modern Interviewing’, Sage, Thousand Oaks CA.
Hale S (2008) ‘The New Politics of Climate Change - why we are failing and how we will succeed’, Green
Alliance, http://www.green-alliance.org.uk/grea_p.aspx?id=3400
Hassard J. and Parker M. eds. (1993) ‘Postmodernism and Organizations’, Sage, London
Hassard J. (1993) ‘Postmodernism and Organizational Analysis: an Overview’ in ‘Postmodernism and
Organisation’, op. cit. pp 1-23
Hatch M.J. (1997) ‘Organization Theory’, Oxford University Press, Oxford
Hawken P. (1993) ‘The Ecology of Commerce’, Harper Business, New York
Hearn J. and Parkin W. (1993) ‘Organisations, Multiple Oppressions and Post Modernism’, in Hassard and
Parker eds. op.cit. pp148-162
HM Government, (2007) Environment Secretary, Third Sector Declaration on Climate Change for Sector Third
Organisations, (http://www.defra.gov.uk/sustainable/government/advice/community/index.htm accessed
14/11/09
Jacobs M. (1991) ‘The Green Economy: Environment, Sustainable Development and the Politics of the Future’,
Pluto Press, London
Johnson P. and Duberley J. (2000) ‘Understanding Management Research’ Sage, London
Keil R. and Desfor G. (2003) Ecological modernization in Los Angeles and Toronto, Local Environment, 8, 27-44
in Krueger and Gibbs (2007)
Klein J (2004) Interdisciplinarity and complexity: an evolving relationship E:CO 6:1-2:2-10
Kuhn L and Woog R (2005) ‘Vortical Postmodern Ethnography: Introducing a Complexity Approach to Systemic
Social Theorising’, Systems Research and Behavioural Science, 22 p139-150
Latour B. (2004) ‘Politics of Nature: How to Bring the Sciences into Democracy’, Harvard University Press
Cambridge MA
Lovelock J (2000) ‘Gaia: A new Look at Life on Earth’, Oxford University Press, Oxford
Lozano R. (2008) ‘Envisioning sustainability three-dimensionally’, Journal of Cleaner Production, vol. 16 1838-
1846 (www.elsevier.com/locate/jclepro) accessed 4/7/10
Luhman N. (1985) ‘A Sociological Theory of Law’, Routlege and Kegan Paul, London, quoted in Cilliers op.cit pp
138-139
McAuley J. Duberley J. Johnson P. (2007) ‘Organization Theory Challenges and Perspectives’, Prentice Hall,
London
Mckibben B. (2007) ‘Deep Economy’, Holt, New York
McMillan E. (2004) ‘Complexity, Organisations and Change’, Routledge, London
Morgan G. (2006) ‘Images of Organization’ Sage, California
Peate F.D. (2005) ‘Blackfoot Physics’ Weiser Books, Boston
Porritt J. (2005) ‘Capitalism as if the World Matters’, Earthscan, London
Prigogine I. (1996) ‘The End of Certainty: Time, Chaos and the New Laws of Nature’, Free Press, New York, in
Hudson (2000:552)
Schwabenland C. (2006) ‘Stories, Visions and Values in Voluntary Organisations’, Ashgate, Aldershot
Senge P.M. (1990) ‘The Fifth Discipline’, Random House, London
Smil V. (1993) ‘Global Ecology, Environmental Change and Social Flexibility’ Routledge, London
Smyth J.C. (2006) ‘Environment and education: a view of a changing scene’ Environmental Education Research,
vol. 12 nos. 3-4, July-September, pp 247-264
Stacey R.D. (2007) ‘Strategic Management and Organisational Dynamics: The Challenge of Complexity’ Fifth
Edition, Prentice Hall, London
Sterling S. (2003) ‘Whole Systems Thinking as a Basis for Paradigm Change in Education: Exploration in the
Context of Sustainability’, unpublished PhD, University of Bath
Stern Sir Nicholas, (Oct 30 2006) 'The Economics of Climate Change', Executive Summary p ii Independent
Review, commissioned by the Chancellor of the Exchequer www.hm-
468

Christine Gilligan
treasury.gov.uk/independent_reviews/stern_review_economics_climate_change/ sternreview_index.cfm -
14k –
Tandon R. and Mohanty R. (2002) ‘Civil Society and Governance’, PRIA, New Delhi (in Schwabenland 2006:11)
Voss Jan-Peter, Bauknecht D. and Kemp R. (eds) (2006) ‘Reflexive Governance for Sustainable Development’,
Edward Elgar Cheltenham UK
Waddock S. (2007) ‘Leadership Integrity in a Fractured Knowledge World’, Academy of Management Learning
and Education, Vol 6 No 4 pp543-557
469

Modeling Organisational Trust: Addressing Challenges of
Measurement Invariance
Carvell McLeary¹ and Paula Cruise²
¹University of the West Indies, Mona, Jamaica
²University of Cambridge, UK
carvell.mcleary02@mymona.uwi.edu.
pc378@cam.ac.uk
Abstract: The traditional conceptual model of organisational trust emphasizes trust propensity, organisational
ability, organisational benevolence and organisational integrity as core determinants of trust between employees
and organisations (Mayer, Davis & Schoorman, 1995). However while being developed to be as parsimonious
and generalizable as possible, there are two primary limitations of the model. Firstly, by ignoring specific
contextual variables such as shared value of collective effort, respect for authority, identification, humility and
closeness (Tan & Chee, 2005; Tan, Wasti & Eser, 2007), the invariance of the model across a range of
organisational contexts and structures remains unknown. Secondly, given the transient and social nature of the
construct of trust, validity evidence of the model’s application to unique cultural settings is limited. The current
research proposes a global structure of trust as further evidence of the impact of the social and cultural context
on the perceived relationship between employees and organisations. Using a sample size of 652 employees from
6 companies based in Jamaica, structural equation modeling was used to confirm the more robust model and
address issues surrounding measurement invariance and applicability. Theoretical, methodological and practical
implications of measuring organisational trust are discussed.
Keywords: organisational trust; measurement invariance; cross-cultural organisational research; structural
equation modeling
1. Introduction
Trust is a major factor in the growth of employees’ organisational commitment, the attainment of
employees’ performance goals, and an organisation’s ability to achieve its objectives (Gilbert, & Li-
Ping Tang, 1998). When trust levels are high, organisations are in a better position to take risks and
use their resources optimally, thus facilitating all activities generally (Yilmaz & Atalay, 2009). However,
despite the important role of trust in organisational functioning, there is a lack of agreement among
researchers about the definition and determinants of trust (Mayer, Davis & Schoorman, 1999) and
difficulties measuring the manifestations of trust in a range of organisational and cultural contexts
(Ferrin & Gillespie, 2010; Schoorman, Mayer & Davis, 2007). Here, we argue that the traditional
definition of trust precludes our ability to measure it effectively and propose the inclusion of additional
components to improve validity and measurement invariance.
2. Theoretical and conceptual orientations of trust
The traditional model of trust proposes that organisational ability, benevolence and integrity combined
with an individual’s propensity to trust determine the trust relationship (Mayer, Davis, Schoorman,
1995; Rousseau, Sitkin, Burt, & Camerer, 1998). In this model organisational trust is conceptualized
as a decision or behavioral intention, and defined as one’s willingess to be vulnerable to the actions of
another based on expectations that the other will perform a particular action important to the trustor,
regardless of the ability to monitor or control that other party. Whilst this is the most frequently cited
model of trust, the creators ignored specific contextual variables that would be relevant to capture a
more valid trust domain (Schoorman, Mayer & Davis, 2007). In focusing on parsimony, they conflated
several important determinants of trust within integrity, such as respect and justice. As justification,
the authors argued that for restricted domains, the salience of respect and justice variables would not
be adequately observed and treated.
In recognizing the limitation of the original model, some authors have added a fourth dimension -
predictability or consistency (Crossley & Colatat, 2008; Six & Sorge, 2008). This too however, is
subsumed within the integrity factor of the original model. As a consequence of this conflation and
parsimony, the determination of organisational phenomena and components of trust across various
contexts is challenging (Wasti & Tan, 2007; Katigbak, Church, Guanzon-Lapeña, Carlota & del Pilar,
2002; Rousseau & Fried, 2001). Furthermore, this conceptual shortcoming has also contributed to
researchers undervaluing the emotional influence involved in trust formation. While the traditional
model is based on cognitive evaluations, affective interactions between individuals influence trust
470

Carvell McLeary and Paula Cruise
development (Williams, 2001) and emotions provide us with feedback on the evolution of trust
situations that we are involved in (Jones & George, 1998). We are therefore proposing that the
inclusion of affective variables relating to organisational justice and organisational respect will be
more effective determinants of the trust construct. Figure 1 depicts the traditional model of
organisational trust as utilized by Rawlins (2008).
Organisational
Competence
Organisational
Goodwill
Organisational
Integrity
Propensity to
Trust of
Employees
Organisational
Trust
Organisational
Outcomes
Figure 1: Rawlins (2008) traditional model of organisational trust comprising organisational
competence, goodwill and integrity (CGI) and employee’ trust propensity
Unlike the cognitive slant of the traditional model, Tschannen-Moran and Hoy (1999) emphasized the
social psychological components of trust and defined organisational trust as one party’s willingness to
be vulnerable and proposed benevolence, reliability, competence, honesty and openness as core
determinants of trust. Although the social psychological perspective regards trust as an intention,
expectation or disposition, research evidence on the validity of these components is lacking (Mayer,
Davis & Schoorman, 2007; Rousseau, Sitkin, Ronald & Camerer, 1998). Similarly, Schockley-
Zalabak, Ellis and Cesaria (2000) proposed trust as a social psychological construct influenced by
organisational competence, openness and honesty, concern for employees, reliability, and
identification. Here, the authors emphasised an organisation’s culture and communication structure as
key facilitators of a trust relationship. However, there is some inconsistency in their conception and
subsequent measurement of trust, as while the definition regards the organisation as the trustor,
items in the scale identify the employees as the trustor. Despite this the scale has generated strong
reliability and validity evidence ranging from 0.85 to 0.90 and .075 to .091 respectively, and has been
demonstrated to be equivalent across several nations, including USA and Europe.
Two domains that have not been given sufficient attention in the organisational trust debate are
justice and respect. Organisational justice (OJ) refers to an employee’s perception of fairness in the
workplace and relates to the distribution of resources, fairness in decision-making and interactional
treatment (Colquitt, Greenberg, & Zapata-Phelan, 2005). Research has demonstrated that
employees’ perception of justice is a predictor of supervisory trust (Ambrose & Schminke, 2003;
Cohen-Charash & Spector, 2001; Cropanzano, Rupp, Mohler, & Schminke, 2001); absenteeism
(Colquitt, Noe & Jackson, 2002), and organisational citizenship behaviour (Colquitt, Conlon, Wesson,
Porter & Ng, 2001). Justice perceptions by workers are also related to organisational trust as research
indicate that employees assess their organisation through their daily interaction, and thereafter make
trustworthy judgements (Pillai, Williams & Tan, 2001; Aryee, Budhwar & Chen, 2002). However,
despite promising evidence, there is still a dearth of research on the role of justice in organisational
trust discourse.
Like organisational justice, organisational respect plays a crucial role to employee perceptions of
fairness and development of trust (Mishra & Spreitzer, 1998). In social identity research,
organisational respect is defined as an individual’s evaluation of their standing or relative status within
an organisation, it is based on an individual’s perception of whether an organisation values them
(Fuller, Hester, Barnett, Frey, Relyea & Beu, 2006). Laschinger and Finegan (2005) proposed that
when employees are empowered at work and treated with respect they will trust their management.
Using structural equation modeling, the researchers found that interactional justice had a direct effect
on respect and organisational trust. Additionally, respect had a direct effect on managerial trust, which
then resulted in trust having a direct effect on job satisfaction. Importantly, too, is that while justice
471

Carvell McLeary and Paula Cruise
had an impact on respect, both variables had direct impact on managerial trust. The structural paths
in their model were all significant.
In light of such evidence, we modified the CGI model of organisational trust, by substituting integrity
with organisational justice and organisational respect. We argue that trust is facilitated when an
employee is trusting and the organisation is just, respectful, competent, has integrity, and
demonstrates goodwill. Figure 2 outlines the proposed Five Factor model of trust.
Organisational
Competence
Organisational
Goodwill
Organisational
Respect
Organisational Justice
Propensity to Trust of
Employees
Organisational
Trust
Organisational
Outcomes:
Figure 2: The proposed five-factor model of organisational trust comprising organisational
competence, goodwill, integrity, respect, justice and employee’ trust propensity
3. Methodological challenges in trust measurement
The diversity of trust measures stem in part from the multi-disciplinary interest in the construct. To
date, trust has been conceptualized with three primary loci: belief, decision and action. Dietz and
Hartog (2006) found that most measures are only tapping into beliefs, while intention to act is a
greater predictor of behaviour. They proposed that measures of trust should be comprehensive and
tap into all forms of the construct and the items should be consistently worded to reflect the referent
under consideration. Inconsistent wording has been cited as a prevalent violation which challenges
survey participants to determine the referent that they are responding to.
Methodological problems within trust research are varied. Firstly, researchers have utilised trust
measures that are incongruent with their proposed definition. For example, studies have provided
definitions of trust that emphasized expectations and willingess to be vulnerable, yet the measure only
included one factor (Dietz & Hartog, 2006; Gillespie, 2003; Lewicki, Tomlinson & Gillespie, 2006). If
measures are not operationalized in a consistent and accurate manner, the problem of interpreting
findings from and across studies will continue. Secondly, there is little replication of trust measures in
subsequent studies of the topic. In a review of 119 trust measures, only 11 had been used more than
once. This fragmentation is caused by the various trust referents studied, and the modification of
scale items to match these referents (Lewicki, Tomlinson & Gillespie, 2006; McEvily & Tororielo,
2005). It is therefore imperative that researchers use relevant and validated measures to reduce this
problem. Finally, many studies do not provide validity evidence of their measure to demonstrate how
well the construct is operationalised. McEvily and Tororielo’s (2011) review of trust measures found
only five instruments with good construct validity evidence. It is strongly recommended that
researchers be consistent in their use of validated measures or conduct tests of validation in their
studies in order to generate more definitive findings on the measurement and determination of
organisational trust.
Another challenge to trust research concerns measurement invariance (MI). MI refers to the
consistency of measurement across some specified group demarcation such as nationality, ethnicity,
gender, time or any number of other factors. The focus of MI is to evaluate the lack of variance
between the measurements used across groups, thereby ensuring that the same attribute must relate
to the same set of observations in the same way across groups (Ellis, Aguirre-Uretta, Sun & Marakas,
2008). Examination of MI is particularly beneficial to the measurement of organisational trust given the
sensitivity of the concept, social components associated with its formation and the effect of variables
472

Carvell McLeary and Paula Cruise
extrinsic to the individual on its maintenance. Also, for validity and confidence to be established
organisational trust measures have to be equivalent across groups. We must ascertain the extent to
which respondents are interpreting items similarly, items are loaded similarly across groups; and
identified differences across groups reflect actual differences in the trait being measured (Drasgow,
1984).
Finally replication of trust measures across organisational studies is low as researchers create and
utilize different measures. This is further worsened by researchers modifying existing scales or
borrowing components of scales without establishing subsequent effects on psychometric properties.
As a consequence comparison of findings across studies is difficult. The traditional measure of trust,
while being well utilized within organisational studies, has been found to be lacking in measurement
invariance. To date, only the integrity determinant was found to have invariance across samples from
USA, Singapore and Turkey, whilst items measuring ability and benevolence were interpreted
differently across the three cultures. (McEvily & Tortoriello, 2011).
4. The present study
The present study had two key aims: 1) to establish the construct validity of a proposed five factor
model of organisational trust that consisted of organisational justice, respect, goodwill, competence,
integrity and employee trust propensity; and 2) to establish the measurement invariance of the
proposed five factor model of trust across three organisational sectors – Finance,
Telecommunications and the Public Services.
5. Method
Participants
Six hundred and fifty two employees participated in the study. Participants were drawn from six
organisations across Jamaica. One hundred and ninety males and 463 (70.9%) females participated.
Participants ranged in age from 18-69 years (M = 34.42; SD = 9.49) and had an average of 7.73
years (SD = 6.43) service to the profession. Approximately 41% of participants were educated to
degree level. Respondents represented a range of roles including non-management (52.2%);
supervisory (23.6%); managerial (13.9%) senior managerial (4.4%) and specialist (5.8%).
Measures
Employees’ degree of Organisational Trust was measured with the 13-item Organisational Trust
Measure – OTM (Rawlins, 2008). The OTM was developed to evaluate the level of organisational
trust within public relations and communications organisations and utilized the conceptual framework
of Hon and Grunig (1999). The measure includes the components of competence, goodwill and
integrity. Statements use the organisation as the referent and are written in the first person.
Employees indicate on a seven-point Likert scale (1 = strongly disagree to 7 = strongly agree) the
extent to which each statement reflects their perception. Examples are: “I am willing to let the
organisation make decision for people like me” and “This organisation does not mislead people like
me”. Item 2 of the overall trust measure was reverse coded (I think it is important to watch this
organisation closely so that it does not take advantage of people like me). As with most scales
measuring trust, the OTM is not replicated. Rawlins (2008) found the scale to have alpha reliability
ranging from 0.79 to 0.93. Tests of validity were not conducted.
Employees’ perception of Organisational Respect was measured with the 8-item organisational
respect scale, the short form of the perceived organisational support scale (Eisenberger, Cummings,
Armeli & Lynch, 1997). Using the short form is not problematic as the original scale is unidimensional
and reliable with alpha of 0.97 (Rhoades and Eisenberger, 2002). Scale items are written as
statements in regards to the organisation on a seven point Likert scale (1 = strongly disagree to 7 =
strongly agree). Eisenberger, Cummings, Armeli and Lynch (1997) found alpha reliability of 0.9, while
Ghani and Hussin (2009) found an alpha of 0.897.
Employees’ perception of Overall Justice was assessed with Ambrose and Schminke’s (2009)
Perceived Overall Justice scale (POJ). The statements of the 6-item scale are written in the first
person and in reference to the organisation. Items are measured on a seven point Likert scale (1 =
strongly disagree to 7 = strongly agree). Examples are: “For the most part, this organisation treats
employees fairly” and “Overall I am treated fairly by my organisation”. Two items are reverse coded.
473

Carvell McLeary and Paula Cruise
Psychometric properties indicate good construct validity (CMIN=1480.02, df =293; CFI=0.96;
RMSEA=0.11) with alpha reliability of 0.93 (Ambrose & Schminke, 2009).
Employees’ Trust Propensity was measured with the 5 item instrument of Huff and Kelly (2003). This
instrument was evaluated across seven nations and had Cronbach alphas of 0.78 at the aggregate
level, with 0.73,China; 0.70, Hong Kong; 0.79, Japan; 0.70, Korea; 0.72, Malaysia; 0.80,Taiwan;
0.72,United States of America. The statements are written in reference to the first person where
respondents select the option which best reflects their views. An example is “I believe that people
usually keep their promises”.
Procedure and Analytical Strategy
Questionnaires were created and hosted by www.surveymonkey.com. Questionnaires were password
protected, ensuring security and confidentiality of participant information. Invitation letters were sent to
employees via the company’s intranet system with the aid of respective Directors of Human
Resources. Participation was voluntary with confidentiality of responses assured. Paper based
formats were made available upon request of which only four were distributed. Questionnaires were
generally completed in 15 minutes. The global goodness of fit of the CGI (four factors) model and
proposed five factor model were assessed through confirmatory factor analysis (CFA), using the root
mean square error of approximation - RMSEA, (Steiger, 1990) and comparative fit index - CFI,
(Bentler, 1990). Levels of RMSEA less than .05 suggest a good fit, values between .05 and .08 is
regarded as a moderate fit, while values above .08 are viewed as a poor fit (Browne & Cudeck, 1993).
CFI values that are greater than .90 are regarded as being of good fit. Measurement invariance of the
five factor model across different industries was measured using the Multi-Group Confirmatory Factor
Analysis (MGCFA) application in AMOS 16.0. We assessed configural, metric, and scalar invariance
across organisational contexts (industries) and across organisational structures (size). Whilst there is
much debate within the methodological literature concerning the efficacy of partial measurement
invariance, we conducted full measurement invariance to ensure that the evaluation was robust and
defensible. Factorial ANOVAs were then conducted to determine the effect of gender, industry and
job role on organisational trust.
6. Results
Construct Validity for Four Factor and Five Factor Models
Prior to CFA and MGCFA tests, the data were examined for normality and outliers and then
descriptive statistics conducted on the measures. Table 1 outlines the means, standard deviations,
reliability alphas and intercorrelations of the OTM, Organisational Respect and POJ scales
administered to the 652 employees. There was no kurtosis (an indication of peakness or flatness
relative to a normal distribution), as assessment showed all values to be between -1.132 and 2.9.
Values greater than 7 are regarded as kurtotic (West, Finch & Curan, 1995). Also there was no
evidence of serious multivariate outliers. While there was evidence of intercorrelation, which is
consistent with other studies concerning the determinants of organisational trust, given the large
sample size there is no expected effect on regression weights (Colquitt, Scott & LePine, 2007).
Table 1: The means, standard deviations, reliability alphas and intercorrelations of the OTM,
organisational respect and POJ scales
Measure Mean
s
SD Alpha
Reliabilit
y
1 2 3 4 5 6
1 Propensity to Trust 4.15 1.07 0.756 1 .215 .299 .299 .291 .246
2 Organisational 5.49 1.12 0.841 .215 1.00 .616 .533 .486 .506
Competence
0
3 Organisational Integrity 4.79 1.29 0.895 .299 .616 1.00
0
.815 .717 .760
4 Organisational
Goodwill
4.25 1.45 0.924 .299 .533 .815 1.000 .777 .709
5 Organisational 4.48 1.16 0.887 .291 .486 .717 .777 1.00 .789
Respect
0
6 Organisational Justice 4.76 1.22 0.914 .246 .506 .760 .709 .789 1.00
0
Note: n =652; Correlations are all significant at the .01 level; p

Carvell McLeary and Paula Cruise
We conducted CFAs to evaluate the goodness of fit of the traditional four factor model. This model
was then compared with the proposed five factor model, which substitutes organisational integrity for
organisational respect and organisational justice. The results in Table 2 demonstrate that the
proposed five factor model yields a better fit to the data than the original CGI (four factors) model. The
goodness of fit indices confirms that the five factor model is a valid measure of organisational trust.
Table 2: Comparative goodness of fit Indices for the traditional four-factor and the proposed fivefactor
model
Item CMIN DF CFI RMSEA
Four factor model 272.83 59 0.96 0.075
(competence, goodwill,
integrity and
trust propensity)
Five factor model 1003.59 265 0.931 0.065
(competence, goodwill
trust propensity,
organisational justice
and organisational respect)
Measurement Invariance of the Five Factor Model of Organisational Trust
Using AMOS 16, an evaluation of Measurement Invariance (MI) on the five factor measure was
conducted using Multi-Group Confirmatory Factor Analysis (MGCFA). The six organisations were
grouped across industries of Transportation and Telecommunications; Banking and Financial
services; and Public and Governmental services; and also grouped according to employee size (small
1100). Table 3 outlines the distribution of organisations across
industry and employee size.
Table 3: Distribution of employees across organisations and industry sectors
Company Employee Sample Industry Sample Size
Size Size in Industry
A 186 65 Transportation & Telecoms.
B 182 41 Transportation & Telecoms. 106
C 1450 356 Financial
D 15 10 Financial 366
E 118 33 Public
F 1006 147 Public 180
Table 4 shows the Chi square (CMIN), CFI and RMSEA data for the evaluation across industries. The
baseline model or unconstrained model (configural invariance), measurement weights model (metric
invariance), and measurement intercepts model (scalar invariance) are shown. Results for
measurement invariance across organisational size are presented in Table 5.
Table 4: Multi-group confirmatory factor analysis of five factor model demonstrating measurement
invariance across industry
Item Unconstrained Measurement Measurement Weights Intercepts
CMIN 1767.362; df = 795 1824.256; df = 835 2053.350; df = 885
CFI .91 .908 .892
(RMSEA) .043 .043 .045
475

Carvell McLeary and Paula Cruise
Table 5: Multi-group confirmatory factor analysis of five factor model demonstrating measurement
invariance across organisational sizes
Item Unconstrained Measurement weights Measurement intercepts
CMIN 1760.302 df=795 1814.069, df=835 2043.198, df=885
CFI .911 .910 .893
(RMSEA) .043 .042 .045
Examination of modification indices revealed no evidence of mis-specification within the outputs.
Based on the ΔCFI cut off criteria of .01(Cheung & Rensvold, 2002), the measure has configural and
metric equivalence, but lacks scalar invariance. This means that employees across the three
industries and organisations sizes conceptualize the constructs in the same way, and also respond to
the items in the same way. However, since for the test of scalar invariance, the ΔCFI is > .01, we
cannot compare latent means across industries or organisation sizes.
Employee variables as determinants of organisational trust
The effect of gender, industry and job role on organisational trust was conducted using betweensubjects
ANOVA. There was a significant main effect of industry on employees’ organisational trust
levels; F (2,610) = 3.022; p=.049; partial eta squared = .010. The Bonferoni post hoc test revealed that
organisational trust was significantly higher for employees in the public service than the transportation
and telecommunications sectors, and was significantly higher for employees in the public services in
relation to the financial services (p = .006 and p = .023 respectively). The level of organisational trust
in the public service and the financial services was not significantly different.
There was no significant effect of gender on organisational trust levels; F (1,610) = .032; p = .858,
partial eta squared = .000, and no significant effect of job role on organisational trust levels (F (4,610)
= 1.818; p = .124, partial eta squared = .012). There was also no significant effect of organisational
size on organisational trust levels; F (2,610) = .047; p = .954; partial eta squared = .000. Linear
Regressions found, job tenure to be a significant predictor of organisational trust levels (β=0.16; p =
.003) suggesting that as tenure in the organisation increases, employees’ degree of trust in the
organisation increases. There was also a significant effect of employees’ role tenure with
organisational trust levels (β = -.196; p ≤ .000) as employees’ tenure within a job increased, their
degree of trust in the organisation decreased.
7. Discussion
A key objective of the current study was to establish the construct validity of a five factor model of
organisational trust across six Jamaican companies, in comparison with the traditional four factors
model of competence, goodwill, integrity and trust propensity. The four factor model being similar to
the seminal model of ability, benevolence, integrity, and trust propensity (Mayer et al, 1995). Analysis
revealed the five factor model is valid with good goodness of fit indices. Findings indicate that the
measure also has configural and metric invariance across the three industries of Transportation and
Telecommunications services, Financial services, and Public services, and also across organisational
size (small, medium and large). This means that employees across the three contexts and structures
interpreted the scale items similarly, and the instrument measures the same attribute across the three
industries. Noteworthy too is the finding that organisational trust is impacted by the type of industry an
employee works in and also the length of time an employee works in their job and within the
organisation.
This study makes theoretical contributions and has implications for future research. Firstly, it improves
our understanding of the determinants of organisational trust in the Jamaican workplace. To the best
of the researchers’ knowledge this is the first study that has been conducted in this regard. The
evidence demonstrate that an organisation’s ability, benevolence, respect, judiciousness, and
employees’ trust propensity are determinants of organisational trust. The implication for Jamaican
managers for building trust in their organisation is evident as emphasis should be placed on all factors
to improve organisational trust.
476

Carvell McLeary and Paula Cruise
Whilst it has been shown that the five factor measure has a degree of measurement invariance, this
was achieved with only three industries, and organisational sizes. This potentially has implications for
large scale studies looking at trust between and within organisations and by extension other
organisational social factors such as satisfaction, motivation and commitment. Future researchers
should therefore not only expect variability in the structures of these pheneomena across key
organisational demographics, but measure them a priori. We therefore recommend that future
research examines the construct of trust in other industries such as information technology,
manufacturing, distribution and media. Given the social nature of trust, there is much benefit to
research in examining the construct of trust within organizations experiencing change as well as the
longevity of the construct over time. Also, researchers in cultural studies will benefit from this work, as
a pre-requisite to conducting cross-group or cross-cultural studies, is to establish measurement
invariance (Vanderberg & Lance, 2000). Of importance is the inclusion of the measurement
invariance method within this study of organsational trust as it is a critical imperative for any
meaningful comparisons across groups. While this method is not a regular feature within
organisational studies, we risk incomparability of findings without it. Additionally, MI should be
included as a specification of a good measure, as is validity, and reliability, when constructing a scale
(Steinmetz, Schmidt, Booh, Wieczorek &Schwartz, 2004).
One limitation of the current research was the evidence of multivariate kurtosis. This is known to be a
problem in structural equation modeling analysis and the Satorra-Bentler procedure is regarded as
the most reliable test statistic for conducting scaling correction, when these distribution assumptions
are violated (Byrne, 2010). It is therefore recommended that future research examine the construct
validity of the five factor model using this methodological approach.
Whilst the measurement of sensitive topics like organisational trust can be quite challenging within
organisations, it is arguably more challenging within restricted domains. We encountered barriers to
access at the levels of the executive management, the human resources management, and that of
the employees. Executive management feared knowing how they were perceived; human resources
management feared repercussions from executive management; whilst employees feared retaliation
by management. This reduced the number of participating companies from twenty to six. Beyond the
confidential guarantees associated with research, we had to field numerous telephone calls and
assure potential respondents and management on the confidential use of the information and that of
the research process. Future researchers should be cognizant of these issues as they conduct trust –
based organisational studies.
In sum, the following research examined the construct validity of the traditional model of
organisational trust and tested the measurement invariance of an amended model across
organisational industry and size. Findings reveal the construct of trust is more effectively measured
when components of organisational respect and organsiational justice are identified. The five-factor
model of trust is also a valid measure of determining trust levels across organisational and employee
individual variables.
References
Aguirre-Urreta, M., Wenying, S. & Marakas, G. (2008). Establishing the need for measurement invariance in
information systems research: A step-by-step example using technology acceptance research. Decision
Sciences Institute Annual Meeting. Retrieved on January 10, 2012 from
http://www.decisionsciences.org/Proceedings/DSI2008/docs/446-6332.pdf
Ambrose, M. L., & Schminke, M. (2003). Organisation structure as a moderator of the relationship between
procedural justice, interactional justice, perceived organisational support, and supervisory trust. Journal of
Applied Psychology, 88 (2), 295-305.
Ambrose, M. L., & Schminke, M. (2009). The role of overall justice judgments in organisational justice research: a
test of mediation. Journal of Applied Psychology, 94(2), 491-500.
Aryee, S., Budhwar, P., & Chen, Z. X. (2002). Trust as a mediator of the relationship between organisational
justice and work outcomes: Test of a social exchange model. Journal of Organisational Behavior, 23, 267-
285.
Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 238–246, 1990.
Brown, M.W. & Cudeck, R. (1993). Alternative ways of assessing model fit. In Bollen, K.A., Long, J.S., editors.
Testing structural equation models. Newbury Park, CA: Sage; pp. 136–162
Byrne, B. (2010). Structural Equation Modeling With AMOS: Basic Concepts, Applications, and Programming,
New York, Routledge, Taylor Francis Group
477

Carvell McLeary and Paula Cruise
Chou, C. P., Bentler, P.M., & Satorra, A. (1991). Scaled test statistics and robust standard errors for non-normal
data in covariance structure analysis: A Monte Carlo study. British Journal of Mathematical and Statistical
Psychology, 44, 347-357.
Cohen-Charash, Y., & Spector, P.E. (2001). The role of justice in organisationa: A meta-analysis. Organisational
Behavior and Human Decision Processes, 86, 278-321.
Colquitt, J. A., Conlon, D. E., Wesson, M. J., Porter, C. O. L. H., & Ng, K. Y. (2001). Justice at the millennium: A
meta-analytic review of 25 years of organisational justice research. Journal of Applied Psychology, 86, 424-
445
Colquitt, A. J., Noe, A. R., Jackson, L. C. (2002). Justice in teams: antecedents and consequences of procedural
justice climate. Personnel Psychology, 55(10), pp. 83-109
Colquitt, J. A., Greenberg, J., & Zapata-Phelan, C. P. (2005), “What is organisational justice? A historical
overview,” in J. Greenberg & J. A. Colquitt (Eds.), Handbook of Organisational Justice, Mahwah, NJ,
Erlbaum (pp. 3-56).
Colquitt, J. A., Scott, B. A., & LePine, J. A. (2007). Trust, trustworthiness, and trust propensity: A meta-analytic
test of their unique relationships with risk taking and job performance. Journal of Applied Psychology, 92(4),
909-927
Cropanzano, R., Rupp, D. E., Mohler, C. J., & Schminke, M. (2001). Three roads to organisational justice. In J.
Ferris (Ed.), Research in personnel and human resource management (vol. 20, pp. 1-113). New York, NY:
JAI
Crossley, C.D., & Colatat, M.C. (2008). In the Eyes of the Beholder: Developing a Measure of Leadership
Trustworthiness. Paper submitted to the Annual Conference of Academy of Management, Anaheim, CA
DeConick, J. B. (2010). The effect of organisational justice, perceived organisational support,and perceived
supervisor support on marketing employees’ level of trust. Journal of Business Research, 63, 1349-1355.
Dietz, G., & Hartog, D. N. (2006). Measuring trust inside organisations. Personnel Review, 35, 557-588.
Drasgow, F. (1984). Scrutinizing psychological tests: Measurement invariance and equivalent relations with
external variables are the central issues. Psychological Bulletin, 95, 134-135.
Eisenberger, R., Huntington, R., Hutchison, S., & Sowa, D. (1986). Perceived organisational support. Journal of
Applied Psychology, 71 (3), 500-507.
Eisenberger, R., Cummings, J., Armeli, S., Lynch, P. (1997). Perceived organisational support, discretionary
treatment, and job satisfaction. Journal of Applied Psychology, 82(5), 812-820.
Ferrin, D. L., & Gillespie, N. (2010). Trust differences across national-societal cultures: Much to do, or much ado
about nothing? In M. Saunders, D. Skinner, G. Dietz, N. Gillespie & R. J. Lewicki (Eds.), Trust Across
Cultures: Theory and Practice. Cambridge, UK: Cambridge University Press
Fiske, S. (2004). Social beings, core social motive approach to social psychology. New York: Wiley
Fuller, J., Hester, K., Barnett, T., Frey, L., Relyea, C., & Beu, D. (2006). Perceived external prestige and internal
respect: New insights into the organisational identification process. Human Relations, 59, 815-846.
Ghani A. N. & Hussin, T. A. (2009). Antecedents of Perceived Organisational Support. Canadian Social Science,
5(6), 121-130
Gilbert, J. & Li-Ping Tang, T. (1998). An examination of organisational trust antecedents. Public Personnel
Management, 27(3), 321-338.
Hon, L. C., & Grunig, J. (1999). Guidelines for Measuring Relationships in Public Relations. Gainesville, FL: The
Institute for Public Relations. Retrieved on January 7, 2012 from
http://www.instituteforpr.com/pdf/1999_guide_measure_relationships.pdf
Horn, J. L. & McArdle, J. J. (1992). A practical and theoretical guide to measurement invariance in aging
research. Experimental Aging Research, 18(3), 117-144.
Hubbell, A., & Chory-Assad, R. (2005). Motivating factors: Perceptions of justice and their relationship with
managerial and organisational trust. Communication Studies, 56, 47-70
Jones, G. R., & George, J. M. (1998). The experience and evolution of trust: Implications for cooperation and
teamwork. Academy of Management Review, 23: 531–546.
Karriker, J.H. & Williams, M.L. (2009). Organisational justice and organisational citizenship behavior: A mediated
multifoci model. Journal of Management 35: 112..
Katigbak, M. S., Church, A. T., Guanzon-Lapeña, M. A., Carlota, A. J., & del Pilar, G. H.
(2002). Are indigenous personality dimensions culture-specific? Philippine inventories and the five-factor model.
Journal of Personality and Social Psychology, 82, 89-101
Kumar, N., Scheer, L.K. & Steenkamp, J.E.M.(1995a). The effects of supplier fairness
on vulnerable resellers. Journal of Marketing Research, 54-65.
Laschinger, H., & Finegan, J. (2005). Using empowerment to build trust and respect in
the workplace: A strategy for addressing the nursing shortage. Nursing Economics,23, 6-14.
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995). An integrative model of organisational trust. Academy of
Management Review, 20(3), 709-734.
Mishra, A.K., & Spreitzer, G.M. (1998). Explaining how survivors respond to downsizing: The roles of trust,
empowerment, justice, and work redesign. Academy of Management Review, 23, 567-588.
McEvily, B., & Tortoriello, M. (2011). Measuring trust in organisational research: Review and recommendations.
Journal of Trust Research, 1(1), 23-63. doi:10.1080/21515581.2011.552424
Norwegian Social Science Data Services. (2012). The comparability of attitude measurements. The European
Social Survey Education Net. Retrieved on January 4, 2012 from
http://essedunet.nsd.uib.no/cms/topics/immigration/1/
478

Carvell McLeary and Paula Cruise
Pillai, R., Williams, E. S. & Tan, J. J. (2001). Are the scales tipped in favor of procedural or distributive justice? An
investigation of the U.S., India, Germany and Hong Kong. The International Journal of Conflict
Management, 12 (4), 312-332.
Ramarajan, L. & Barsade, S. (2006). What makes the Job tough: the influence of organisational respect on
burnout in the human services. Retrieved on January 4, 2012 from http://www.zuckersystems.com/1327.pdf
Rawlins, B. (2008). Measuring the relationship between organisational transparency and employee trust. Public
Relations Journal. 2 (2), 1-21
Rousseau, D. M.., Sitkin, B. S., Ronald, S. B. & Camerer, C. (1998). Not so different after all: A cross-discipline
view of trust. Academy of Management Review, 23(3), 393-404.
Rousseau, D. M., & Fried, Y. (2001). Location, location, location: Contextualizing organisational research. Journal
of Organisational Behavior, 22, 1-13.
Shockley-Zalabak, P., Ellis, K., & Cesaria, R. (2000, August). IABC Research Foundation
unveils new study on trust. Communication World, 7-9.
Six, F., & Sorge, A. (2008). Creating a high-trust organisation: An exploration into organisational policies that
stimulate interpersonal trust building. Journal of Management Studies, 45, 857-884.
Schoorman, F. D., Mayer, R. C., & Davis, J. H. (2007). Editor ’ s forum an integrative model of organisational
trust: past , present , and future the university of akron. Academy of Management Review, 32(2), 344-354.
Skinner, D, Saunders, M. N. K. & Thornhill, A (2002), Human resource management in a changing world.
Strategic Change, 11 (7), 341-345.
Steenkamp, Jan Benedict and H. Baumgartner (1998), Assessing measurement invariance in cross-national
research. Journal of Consumer Research, 25, (June), 78-90.
Steiger, J.H. (1990). Structural model evaluation and modification. Multivariate Behavioral Research. 25, 214-12.
Steinmetz, H., Schmidt, P., Booh, A.T., Wieczorek, S. & Schwartz, S. (2004). Testing measurement invariance
using multigroup CFA: Differences between educational groups in human values measurement. 24th
conference of the Society for Multivariate Analysis in the Behavioral Sciences (SMABS), July 2004 Jena/
Germany.
Tan, H.H., Chee, D. (2005). Understanding interpersonal trust in a Confucian-influenced Society: an exploratory
study. International Journal of Cross Cultural Management: 5, 197-212
Tan, H. H., Wasti, S. A., & Eser, S. (2007). Location, location, location: antecedents of trust across hierarchical
and geographical positions’. Research Collection, Lee Kong Chian School of Business. Paper 1020.
Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature:
Suggestions, practices, and recommendations for organisational research. Organisational Research
Methods, 3(1), 4-70.
Wasti, S., Tan, H., Brower, H., & Onder, C. (2007). Cross-cultural measurement of supervisor trustworthiness: An
assessment of measurement invariance across three cultures. The Leadership Quarterly, 18(5), 477-489.
West, S. G., Finch, J. F. & Curran, P. J. (1995) Structural equation models with non-normal variables: Problems
and remedies. In Hoyle, R. (ed.), Structural Equation Modeling: Concepts, Issues, and Applications. Sage,
Thousand Oaks, CA, pp. 56–75.
Williams, M. (2001). In whom we trust: Group membership as an affective context for trust development.
Academy of Management Review, 26: 377–396.
Wong, Y., Ngo, H. & Wong, C. (2004). Perceived organisational justice, trust, and OCB: A study of Chinese
workers in joint ventures and state-owned enterprises. Journal of World Business. 41(4), 344-355
Yilmaz, A., & Atalay, C. (2009). A theoretical analysis on the concept of trust in organisational life. European
Journal of Social Sciences, 8 (2), 341-352, 2009
479

Health and Safety of a Project Design: Professional
Designers and Malaysian Legal Perspective
Zul Zakiyuddin Ahmad Rashid, Hamimah Adnan, Azizan Supardi, and Norazian
Mohamad Yusuwan
Universiti Teknologi MARA, Shah Alam, Malaysia
zakiphd@gmail.com
mimad856@gmail.com
zzan659@gmail.com
azianyusuwan@gmail.com
Abstract: This paper is meant to look at the professional designers understanding and the role of Malaysian law
in design risk management, particularly on matters related to health and safety. Health and safety of a project is
an important area to be properly managed by the professional designers. The designers have to consider the
aspect of health and safety in the element of their design. Failure to do so will expose the professional designers
to possible legal actions. In addition to this, the manner the Malaysian law correspond to ensure the designers
adhere to the basic risk management practice, particularly on health and safety of the project will also be looked
into. In achieving the above, mix method approach was adopted. The general perceptions of the professional
designers were sought during questionnaire survey, which form the quantitative approach. Once the general
perception of the matter has been gathered, in-depth interview involving more experience respondents were
conducted. The in-depth interview formed the quantitative approach in this research. The findings were then
evaluated by experts of the matter to complete the triangulation approach. It was found out that majority of the
professional designers were unable to fully grasp the health and safety requirement in their design. In addition,
legal requirements for health and safety to be practiced by the designers in Malaysia are in need to be further
improved.
Keywords: health and safety, professional designers, legal, Malaysia
1. Introduction
The construction industry is statistically one of the most hazardous industries in many countries
(Gangolells, Casals, Forcada, Roca and Fuertes, 2010). The construction industry has suffered
through a decade of poor design and construction performance (CFMA, 2006; Simonson, 2006;
Flores and Chase, 2005). In the construction process, there are many unpredictable factors, so the
construction project is a high-risk process. Accordingly, the acceptance of an obligation in a building
contract is associated with the acceptance of commensurate risks, namely the risk of being unable to
fulfill the obligation because of one’s own inadequacy, incapacity, inadvertence or error, or because of
interference from outside sources or supervening events . (Robinson, Lavers, Tan and Chan, 1996). In
particular, designers have been criticised for not being accountable to deliver designs on time,
minimising construction change orders, and not giving contractors adequate design directives (CFMA,
2006; Chang, 2002; Rubin, 2004). Some construction industry participants have identified the
professional designers as a major source of risk and inefficiency in the design-bid-build process
resulting in the current poor delivery of construction services (Chang and Chiu, 2005; HC&O Editor,
2004).
One of the elements which involve the designing process includes matters related to health and
safety. Issues related to health and safety involves many parties, including the contractors and its
workers, building user and third party. In relation to health and safety, the damages may include
injuries, financial loss and legal actions taken against the party in default.
2. Research objectives
The objectives of this paper are:
To identify risks related to health and safety, particularly in relation to professional designers.
To analyse the manner Malaysian law correspond to the need of managing such risk.
3. Research methods
In achieving the above objectives, mix research methods were adopted, which involves questionnaire
survey, in-depth interview and expert evaluation via expert interview. The questionnaire survey is
meant to look at the general perception of professional designers on matters related to health and
480

Zul Zakiyuddin Ahmad Rashid et al.
safety. With the data, further in-depth interview were adopted, with the intention of soliciting more
detail data on the issue from experienced respondents. The findings were then evaluated by experts
in construction and law, which will act as an endorsement to the findings. In addition, the experts’
evaluation will also signify the criticality of the findings.
4. Malaysian legal provisions on health and safety
Matters related to health and safety in Malaysia can be traced to 3 main branches of law, namely the
law of contract, the law of tort and statutory provisions. Under the law of contract, the responsibilities
of the designers on health and safety can be outlined by the contractual terms. It is stipulated under
the contract that professional designers have to ensure reasonable skill and care, as decided in
Bolam v Friern Hospital Management Committee [1957] 1 WLR 582, which is applicable in the
Malaysian court. This is the most essential rule in the performance of the duties and obligations under
the contract. The law implies such duty, in the absence of any contractual terms on the standard of
care to be exercised by the designer. Under the common law, the duty of care of professional
designers is known as the duty of care for professionals, which include matters related to health and
safety. In such circumstances, a person who holds himself as a designer, whether he possesses the
qualifications and knowledge or not, impliedly warrants that he is reasonably competent to carry out
the duty.
Under the law of tort, the professional designers owe a duty of care when it is foreseeable by any
reasonable man that a failure act on part of the designer in his duty will affect others. For instance, it
is foreseeable by any reasonable man that failure in design will affect the health and safety of users of
the building. This principle is stated in the landmark case of Donoghue v Stevenson [1932] AC 562.
Accordingly, it must be noted that the duty of care owed under the law of tort is owed by the designers
to anyone, provided that the above conditions are met. In contrast to contractual duty, duty under the
law of tort can extend to any third party.
With regards to the statutory provisions available in Malaysia on health and safety, the main reference
is the Occupational, Safety and Health Act 1994: Guidelines for Public Safety and Health at
Construction Sites. The Occupational, Safety and Health Act 1994 is basically related to the
supervisory duty of a designer. Accordingly, in the normal context of an architect engagement, he will
be responsible in supervising the works as the employer’s agent.
In addition to the Occupational, Safety and Health Act 1994, the Department of Occupational Safety
and Health (DOSH) and other government agencies have regulations that lay down the legal
requirements to ensure the safety and health of not only the workers at the place of work but also the
public as well. The Guidelines For Public Safety And Health At Construction Sites applies to all place
of work in building operation and work of engineering construction activity in Malaysia covered by the
Occupational Safety And Health Act 1994 (Act 514), the Factories And Machinery Act 1967 (Act 139),
and all the regulations made there under. It is designed to serve as a handy reference and to be read
together with the above mentioned legislations and other industry codes of practice.
While there are laws regulating health and safety in construction, it is essential to look at the
designers’ perception on the matter. Accordingly, few research methods were applied to gather the
data. From the data, it is possible to chart the understanding of the designers on health and safety
issue, and whether there is a need for the existing laws to be further improved.
5. Questionnaire survey: Professional designer’s perception on effectiveness
of the existing contract in outlining their duties, which include matters
related to health and safety
The questionnaire survey was deployed to 1000 architects and 1000 engineers registered with
respective professional bodies, namely Pertubuhan Akitek Malaysia (PAM) and Institute of Engineers
Malaysia (IEM). The purpose of the questionnaire survey is to gather general perception of the
respondents on risks related to professional designers in Malaysia. Based on the replies, further indepth
data collection was carried out.
During the questionnaire survey, the respondents were asked to evaluate the standard form of
contracts, Memorandum of Engagement/Agreement and Codes of Professional Conduct in the sense
of its effectiveness in outlining the designers’ responsibility. With regards to the standard form of
481

Zul Zakiyuddin Ahmad Rashid et al.
contract, the contracts referred were PAM 1998 and IEM Conditions of Contract for Works Mainly of
Civil Engineering Construction.
5.1 Analysis
From Table 1, generally the respondents gave split opinion on the effectiveness of the contracts in
outlining the responsibilities. While many evaluated the contracts as good, quite a number ranked the
contracts as average. By taking into consideration the replies that evaluated the contract as
average/poor, it is submitted that there is a room for improvement, in relation to the contract
document. With reference to this, it is important to understand the reason why some of the
respondents regarded that the contract as average/poor in outlining the responsibility of a designer.
Such understanding can be gathered from subsequent research method, namely the in-depth
interview involving more experienced designers. Subsequently, further improvement to the contract
can take place based on the findings.
Table 1: Sufficiency of the contract in outlining the designer’s responsibility
Total
Respondents
Sufficiency in outlining the
designers’ responsibility Good
114 Standard Form of Building
Contract
114 Memorandum of
Engagement/Agreement
114 Codes of Professional Conduct
Percentage (%)
Average
Percentage (%)
Poor
Percentage (%)
Missing
Percentage (%)
85 75 18 16 8 7 3 3
71 62 34 30 6 5 3 3
76 67 29 25 6 5 3 3
6. In-depth interview on design related risks
The interview is meant to get in-depth data from experienced respondents. The general perceptions
of the respondents were collected through set of questionnaire survey. From the exploratory nature of
questionnaire survey, the researcher needs to gather further data to really comprehend the main
issue as stipulated in the research questions.
The interview session involved more experienced respondents. The respondents of the interview
were selected among architects and engineers with practicing experience of 10 years and more. The
selection was done based on replies gathered from questionnaire survey, where respondents of the
questionnaire survey were asked to indicate their working experience, as well as their willingness to
take part in subsequent in-depth data collection, namely the interview. In addition, references were
also made to respective professional bodies in identifying the suitable respondents.
With regards to the experience criteria of the respondents in this interview, it was established by the
American Institute of Architects (AIA) that a senior architect/designer is defined as licensed architect
or non-registered graduate with more than 10 years of experience. He has a design or technical focus
and is responsible for significant project activities. Accordingly, within this definition, only respondent
with experienced of 10 years and more were selected. It is submitted that experienced respondents
from architectural and engineering field will gives clearer and in-depth picture of the problems related
to professional designers in Malaysia.
Respondents of the interview were asked to explain risks related to professional designers and design
works. With reference to this particular interview question, the respondents gave various aspects of
design related risks.
On risks related to design professionals, the respondents gave various risks elements in relation to
their works. According to Respondent E15, poor design and negligent supervision were quoted as risk
factors. Respondent E15 further elaborate that poor design can be in the sense that the engineer
prepares a design without properly considering the load bearing of the land. He said, “it is important
482

Zul Zakiyuddin Ahmad Rashid et al.
for architect to properly calculate the load bearing of land in his design. Fail to do so will result in poor
design being prepared.”
Respondent E14 also agrees that negligent supervision as a risk factor, in addition he added that
constant interference of the client with design works and architect failure to keep proper
documentations of his works also caused risks in design. He said that the client should know the
structure of the contract and duties of each party. He further stated that “...the client attitude in
interfering with the design might be due to the attitude of trying to save the project cost. However, this
attitude might force the designer to come up with lower design quality.” Respondent E12 also stated
that risk related to engineers and design works include client interference with design works, but
added more by quoting insufficient information given by the client during brief as another risk element.
This was agreed by Respondent E13, on which he said, “…insufficient documentations of the
architect works and not enough brief by the client are the risks related to engineers and design
works.”
According to Respondent E7, failures to fulfil the statutory requirements and clients’ unprofessional
attitude with the design works are the risks related to engineers and design works. With reference to
respondent architect, similar risks were cited during the interview sessions. According to Respondent
A1, risks related to professional designers in their duties include duty to supervise carried out
negligently and insufficient documentation of the designer’s works. He stated that it is important for an
architect to properly record their works. Proper records of the works will enable the architect to
execute their works according to the work plan, and as a way to safeguard their interest. Apart from
this insufficient documentation of the designers’ works, insufficient brief by the client was also quoted
as possible risk elements. Respondent A2 stated that insufficient brief causes delay in the completion
of the design, as the designer need to revert back to the client for further details. If tenders were
invited based on the incomplete design due to insufficient brief, it is most likely the project cost will
exceed. Two architect respondents, Respondent A9 and Respondent A10, elaborated extensively on
risks related to professional designers and design works. Their lists were inclusive of all other risk
elements as stated by other respondents.
Certificate of Completion and Compliance (CCC) is relatively new requirement in Malaysia.
Accordingly, professional designers can be appointed as “Principal Submitting Person” under the
requirement. Principal Submitting Person is responsible for the certification of the completion and
compliance of the project.
With reference to this, 6 engineer respondents and 6 architect respondents stated that this
requirement as an added responsibility, in addition to their existing duties and liabilities. According to
Respondent E7, the relatively new CCC requirements, which centralised the responsibility over
certification of compliance and completion of a project on a submitting person, poised more risks to
the professional designers, as architect or engineer could be appointed as the submitting person. He
stated that “Engineers is already handful with existing duties to be performed. The introduction of
CCC poised more risks to the profession, as it centralised the responsibility of project certification on
the engineer.”
As a whole, risks in relation to professional designers and design works as perceived by the
respondents are illustrated in Table 2. The table also stated number of replies and overall replies
percentages. Missing column in the table refers to number of respondents who did not consider the
elements stated as risk related to design.
6.1 Analysis
The respondents were asked about risks related to professional designers and design works. From
the replies, risk elements stated by the respondents can be linked to negligence of the parties related
to the project as well as risk related to the traditional procurement route and legal provisions.
Negligence supervision, poor design, insufficient documentation on the designer’s works, interference
by the client, insufficient brief by the client, failure to comply with statutory requirement and low
understanding on part of the contractor can be categorized as negligent act by the parties involved in
the project.
483

Zul Zakiyuddin Ahmad Rashid et al.
Table 2: Risks related to professional designers and design works
Design Related Risk
Total Respondents
Negligent supervision 30 17 57 13 43 1
Replies
Poor design 30 15 50 15 50 2
Insufficient documentation of the
designer’s works
Percentage (%)
Missing
Percentage (%)
Rank
30 14 47 16 53 3
Client interference with design work 30 13 43 17 57 4
CCC requirements as added
responsibility
30 12 40 18 60 5
Insufficient brief by the client 30 11 37 19 63 6
Risk in contract 30 10 33 20 67 7
Failure to comply with statutory
requirement
Low understanding on part of the
contractor
30 9 30 21 70 8
30 8 27 22 73 9
Looking at the replies, general observation of the percentage indicates insufficient understanding on
part of the respondents on risks elements related to design. For instance, based on the ranking of the
replies, the highest percentage on risk element in design works was merely fifty seven percent (57%),
namely on risk related to negligent supervision. With reference to poor design as an element of risk, it
was quoted by fifty percent (50%) of the respondents, while the rest did not see it as a risk. It is
submitted that the designers have to fully understand the risks involved before proper risk
management can be exercised. It has to be noted that, none of the respondents during semistructured,
in-depth interview quoted health and safety as part of designer’s duty as stipulated by the
contract.
Emanating from the above, the next critical area to be considered is on the need to introduce external
provisions on health and issue matter, as an external requirement for risk management practice.
During the semi-structured, in-depth interview, reference was made to the United Kingdom
Construction (Design and Management) Regulations 1994 as illustration on external provision
regulating design risk management on health and safety. On this point, merely thirty one percent
(31%) of the semi-structured, in-depth interview respondents agreed on the importance of external
requirement in providing a better outline on risk management.
7. Experts’ evaluation on risks related to health and safety
The objective of having a second set of interview involving expert is to get the important findings
gathered from previous data collections reviewed by experts with knowledge in construction and law.
Expert interview will complete the triangulation approach adopted in this research, which will make the
data reliable and robust.
It must be noted that the questions asked in the questionnaire survey and semi-structured, in-depth
interview transcend the construction field. For instance, the respondents were asked about legal
matters and risk management. Such area might be unfamiliar to them, especially on law related to
design and design risk management. Occasionally, there were situations where the mix opinions were
484

Zul Zakiyuddin Ahmad Rashid et al.
gathered. Mix of opinion indicates the existence of misunderstanding among the respondents. As
such, review by experts is important as it comes from persons knowledgeable in construction and law.
In addition, the expert opinions will put perspective to the law and design risk management, as well as
acted as endorsement to the replies by the respondents.
Apart from reviewing the findings, the expert will also evaluate the criticality of the issues identified
from the previous data collection. With the criticality of the findings being placed accordingly by the
expert, the issues can be properly evaluated based on the expert evaluations. According to Cabanis
(2002), an expert may be defined as someone with special skills or knowledge evidence of leadership
in professional organizations, holding office in a professional organization, presenter at national
conventions, published in recognised journals, etc.
Within the definition of the law, an expert witness is regarded as a specialist who by virtue of special
knowledge, skill, training or by experience is qualified to provide testimony to aid the factfinder in
matters that exceed the common knowledge of ordinary people. An expert can only be as effective as
academic education, professional training, years of experience and charisma will allow. (Rasmussen
and Leauanae, 2004).
Referring to the above definitions, as far as this research is concerns, experts with extensive
knowledge and experience in construction and law, endorsed with relevant credentials and
recognised by the industry was selected as an expert respondent for this research. The identification
of expert was made during the in-depth interview, where the respondents were asked to name
individuals in the industry appeared to them as having extensive knowledge in law and construction.
From the replies, a list of expert was created. In addition to this, suitable expert was also identified by
referring to the professional bodies such as Pertubuhan Arkitek Malaysia (PAM) and the Institution of
Engineers Malaysia (IEM). The researcher also attended seminars and conferences in identifying the
right expert for this research. Among conferences attended were 3rd International Conference on Built
Environment in Developing Countries held in Universiti Sains Malaysia, One-Day Intensive Seminar
on Malaysian Standard Form of Construction Contract (PAM 98) & Comparison with PAM 2006 at JW
Marriott Hotel organised by Asia Pacific Diligence Sdn Bhd, One-Day Seminar on PAM Contract 2006
by The Chartered Institute of Building Malaysia (CIOB) and various talks and forums organised by
Society of Construction Law-Kuala Lumpur and Selangor.
The experts were asked to evaluate the findings of questionnaire survey and in-depth interview,
particularly on health and safety matter. Accordingly, health and safety is under the purview of
professional designers’ duty and liability. For instance, the designer must ensure that his design will
not affect the health and safety of those involved with it, be it the client, contractor, worker and third
party users. During the in-depth interview, the respondents did not quote this particular duty as being
sufficiently illustrated by the contract. Expert opinions were sought in determining the criticality of this
point to be properly understood and executed.
7.1 Expert respondent number 1
The expert indicated that it is critical for the professional designer to take in account the matter of
health and safety in his work. Health and safety is under the purview of professional designers’ duty
and liability. For instance, the designer must ensure that his design will not affect the health and
safety of those involved with it, be it the client, contractor, worker and third party users.
7.2 Expert respondent number 2
It was also his expert evaluation that health and safety is under the purview of professional designers’
duty and liability, as such to be considered as critical in term of the designer’s standard of care. For
instance, the designer must ensure that his design will not affect the health and safety of those
involved with it, be it the client, contractor, worker and third party users.
485

Zul Zakiyuddin Ahmad Rashid et al.
Table 3: Performance of designer’s duty: health and safety
Expert 1
Expert 2
Critical Less Critical Not Critical
Expert Evidence
√
√
7.3 Analysis
Convincing Practical
Evidence
Weaker Practical
Evidence
Expert Evidence
The experts agreed that matters related to health and safety is part of the designers’ duties. In
contrast to the findings of the in-depth interview, the experts submitted that it is critical for the
designers to understand this point and ensures that their design does not breach the health and
safety issues. Comparing the criticality of the matter as viewed by the experts and findings gathered
from the in-depth interview, it signifies crucial issue on the understanding of in-depth interview
respondents in health and safety requirement; as importance in this point was not highly regarded by
the in-depth interview respondents.
Convincing Practical
Evidence
Weaker Practical
Evidence
Expert Evidence
Convincing Practical
Evidence
8. Expert Interview: Performance of designer’s duty: legal provisions on
health and safety
In relation to the above, the experts were asked to give their opinion on state of existing legal
provisions available in Malaysia, dealing with health and safety issue.
8.1 Expert respondent number 1
Another expert evaluation was given by the respondent on the lacking of legal provisions on health
and safety. It is critical for this matter to be addressed properly. Accordingly, the statutory provisions
on health and safety is still lacking in Malaysia, thus creating misconception and misunderstanding
with regard to the duties and liabilities of professional designers towards this area. UK’s Construction
(Design and Management) Regulations 1994 is a good example for Malaysia to follow, especially in
defining the scope of responsibilities of designers with reference to health and safety and design
works.
8.2 Expert respondent number 2
Similarly, the respondent gave his expert evaluation that the following statement to be considered as
critical, with reference to the designer’s standard of care. The statement reads:
The statutory provisions on health and safety is still lacking in Malaysia, thus creating misconception
and misunderstanding with regard to the duties and liabilities of professional designers towards this
area. UK’s CDM Act is a good example for Malaysia to follow, especially in defining the scope of
responsibilities of designers with reference to health and safety and design works.
8.3 Analysis
The expert respondents submitted that the Malaysian legal provisions on health and safety issue is
lacking. They viewed this matter as critical, based on expert evidence. Accordingly, it is important for
this matter to be improved.
486
Weaker Practical
Evidence

Zul Zakiyuddin Ahmad Rashid et al.
When the point was presented before expert respondents, both experts confirmed the need for
external provision to be introduced, particularly on health and safety issue. Accordingly, the experts’
evaluation signifies the need for the law to properly structure the matter. Consensus of the experts
shows that there is a need for external provision to be introduced. This is an indication for the matter
to be further improved.
Table 4: Performance of designer’s duty: Legal provisions on health and safety
Critical Less Critical Not Critical
Expert 1
Expert 2
Expert Evidence
√
√
Convincing Practical
Evidence
Weaker Practical
Evidence
Expert Evidence
Convincing Practical
Evidence
Weaker Practical
Evidence
9. Improvement on health and safety matters through law
On matters related to design risks and health and safety, the appropriate branches of laws are law of
contract, law of tort and statutory provisions. Improvement to the laws is needed in order to improve
matters on health and safety. For instance, illustration on health and safety can be properly illustrated
in the building contract, in contrast to the existing contractual provisions available in the industry.
Health and safety is also regarded as part of the duties stipulated to be performed by the designers.
As such, the designers have to properly observe it, for failure will be considered as breach of
reasonable skill and care under the law of tort. In addition to the above, the statute can provide clear
provisions on matters related to health and safety, as measures taken by the English legislature
through the introduction of Construction (Design and Management) Regulations 1994. On this matter,
apart from similar legislation to the same effect, our very own Occupational, Safety and Health Act
1994 can be further improved to accommodate better design risk management on health and safety
matters.
10. Conclusion
Professional designers’ works is shrouded with various risks, which include risks related to health
safety aspect of the project. Accordingly, risk occurrence can bring catastrophic effect in various
ways, such as financial disaster, damages to property and personal, even fatality. Health and safety is
one major part of risks that involved professional designers in the course of their works. It is critical for
the designer to fully understand the issue on health and safety as part of design risks and manage it
properly. Form the finding of the research, it can be concluded that existing laws on health and safety
is insufficient. It is submitted that professional designers understanding on health and safety matter,
as well as the contractual and legal provisions available can be further improved.
11. Recommendation
Considering the effect of risks related to design works, it is important to have it properly managed.
Professional designers need to better understand the risk and the importance of risk management.
The legislators can take into consideration the list of risks in improving the laws meant to cater for
better risk management practice by the industry. This is possible as the principle of risk management
is in line with the objective of the law itself.
In general, the law relevant to risks related to professional designers and design works can be
classified as follows:
487
Expert Evidence
Convincing Practical
Evidence
Weaker Practical
Evidence

Law of Contract
Law of Tort
Statutory provision
Zul Zakiyuddin Ahmad Rashid et al.
It is submitted that the framework of Malaysian law is sufficient to provide a good risk management
structure. This is because the nature of the law is dynamic and adaptive, which suits the nature of
construction industry. It must be noted that such approach is in line with the purpose and objective of
the law, namely to address the public safety, security, clarity, flexibility, transparency and adaptability.
This is achievable by refraining the people from doing something or binding them to commit certain
act. Use of risk and risk management knowledge can effectively serve these purposes in meaningful
ways (Raquib, 2002).
The dynamic nature of the law will suit to the role required in risk management. It is suggested that
the law being actively utilized as a tool to correspond to risk management needs. It is achievable by
adapting risk management approaches in the laws related to professional designers. For instance, the
court can be proactive by setting precedent in deciding cases related to design risk management.
Lacuna in the existing legal provisions can be further improved. The lawmakers can be proactive by
incorporating risk management policy in statutory regulations. Example on this can be taken from the
initiative taken by the British legislatures via the passing of Construction (Design and Management)
Regulations 1994, which was later repealed by the Construction (Design and Management)
Regulations 2007. It is submitted that the CDM Regulations are meant to improve the overall
management and co-ordination of health, safety and welfare throughout all stages of a construction
project, with the purpose of reducing the number of serious and fatal injuries. Various aspects of the
legal provisions can be improved, as discussed in the previous chapters, which include matters
related to coordination of the designing schemes and professional indemnity insurance.
References
Bolam v Friern Hospital Management Committee [1957] 1 WLR 582
Cabanis, K. (2002). “Computer-related Technology Use by Counselors in the New Millenium: A Delphi Study”,
Journal of Technology in Counseling, Vol.2_2.
CFMA’s (2006). Construction Industry Annual Financial Survey, (18th ed.), Moss-Adams, LLP, Seattle, WA.
Chang, A.S. (2002). “Reasons for cost and schedule increase for engineering design projects”, Journal of
Management in Engineering, Vol. 18 No. 1, pp. 29-36.
Chang, A.S. and Chiu, S. (2005). “Nature of Engineering Consulting Projects”, Journal of Management in
Engineering, Vol. 21 No. 4, pp. 179-88.
Construction (Design and Management) Regulations 1994.
Construction (Design and Management) Regulations 2007
Derk G. Rasmussen and Joseph L. Leauanae (2004). Expert Witness Qualifications and Selection, Journal of
Financial Crime, Vol. 12, No 2.
Donoghue v Stevenson [1932] AC 562
Factories And Machinery Act 1967
Flores, V. and Chase, G. (2005). “Project Controls from The Front End”, Cost Engineering, Vol. 47, No. 4, pp. 22-
4.
HC&O Editor (2004), “Survey Reveals Construction Challenges and Trends”, HC&O, Vol. 3 No. 1.
MA Raquib (2002). Analyzing The Concept of Risk and Risk Management to the Formulation of Laws and
Regulations and Establishment of a Legal Framework, International Conference on Law and Commerce
2002, International Islamic University, Kuala Lumpur.
Marta Gangolells, Miquel Casals, Nuria Forcada, Xavier Roca, Alba Fuertes (2010). Mitigating Construction
Safety Risks Using Prevention Through Design, Journal of Safety Research 41
Nigel M. Robinson, Anthony P. Lavers, George KH Tan and Raymond Chan (1996). Construction Law in
Singapore and Malaysia ( 2 nd Ed.), Butterworths Asia, Singapore, p. 185.
Occupational, Safety and Health Act 1994
Rubin, D. (2004), “Audit of Federal Highway Job Overruns Worries Engineers”, available at:
http://construction.com/NewsCenter/Headlines/ENR/20040524g.asp.
Simonson, K. (2006), Quick Facts, Chief Economist Report, Association of General Contractors, Houston, TX.
The Guidelines For Public Safety And Health At Construction Sites
488

Promoting Innovation in Data Poor Public and Private
Business Areas Through Systems Analysis
Chiang Ren 1 , Gillian Green 2 and Robert Wood 3
1
University of Bolton, UK and Kepler Research Inc., USA
2
University of Bolton Business School, UK
3
University of Bolton Faculty of Well-Being & Social Sciences, UK
ChiangRen@KeplerResearch.com
gmg1@bolton.ac.uk
Bob.Wood@mbs.ac.uk
Abstract: Despite continuous advances in systems and complex systems research, many public and private
sector business areas still exhibit latent levels of system dynamics that are poorly studied. This is in part due to a
lack of data in those areas for supporting and validating research. A proposed way to approach data poor
problems is for systems analysts to cross disciplinary boundaries and deeply observe from within the
environments of the problems. Then, they can identify inconsistencies and build predictive models that integrate
incomplete knowledge. The process of observation requires patience, and many years were needed to clearly
identify and study six data poor problems from multiple disciplines to demonstrate this unique path in
interdisciplinary research. This paper presents a methodology for approaching data poor problems and discusses
published works in disaster preparedness, defense planning, cellular biology, systems engineering, acquisition of
IT systems, and public administration that utilized this methodology. To summarize the specific data poor
problem analysis experiences, we were able to use systems analysis to provide the business community with an
understanding of how to respond to linked crises events despite a lack of historical data on such events in a
published paper for disaster preparedness. In a paper for defense planning, systems analysis explained how
capabilities and agility were being lost within the U.S. defense enterprise. In a paper for cellular biology, systems
analysis showed a new theoretical path for investigating cell specialization. In a paper for systems engineering,
complex systems analysis presented engineers with new opportunities to support scientific research. In a paper
for the acquisition of IT systems, the understanding of complex systems introduced a new framework for initiating
IT programs. And in a paper for public administration, complex systems analysis presented agent-based
modeling constructs for the study of disruptive personnel in the government workforce.
Keywords: systems analysis, interdisciplinary, research methods, data poor, complexity, innovation
1. Introduction
The application of analytical methods to improve the overall design of mechanical systems, optimize
the processes of organizational systems, and understand the behaviors of natural systems can be
traced back to the days of World War II (Blanchard, 2010). Some methods further extend back to the
start of the industrial revolution. With the dawn of the information age, computer-based methods are
increasingly being applied to study the dynamics of complex systems with self-organizing
characteristics, poorly defined boundaries, and/or many constituent elements (Bar-yam, 2003).
Despite decades of advancement in systems analysis and related fields such as systems engineering,
operations research, industrial engineering, management science and decision science, this paper
proposes that many public and private sector business areas still exhibit latent levels of system
dynamics that are poorly studied. Greater insight into such system dynamics can lead to innovations
in business practices, new research paths, and expanded opportunities.
We further propose that the reason for the lack of investigation into some business areas is due to
inadequate data for validating system and complex system models. Instead of waiting for data to self
manifest in these “data poor” areas, systems and complex systems analysis can be applied to initially
build predictive constructs that integrate incomplete knowledge. These imperfect higher order models
or frameworks for models can then direct researchers toward new ways of unearthing data, achieving
innovation, and growing opportunities. To support this proposal, a simple but effective methodology is
presented in the context of interdisciplinary research (Moran, 2010), and six peer reviewed papers
published in scientific and technical journals that addressed data poor problems through this
methodology are discussed. The refinement of our methodology through the assessment of the
specific published works occurred as a collaborative research effort at the University of Bolton’s
Business School. Our path of investigation has been in a different direction than the research of those
scholars advocating tighter coupling between theories and specific data sets (Eisenhardt, 1989). Yet,
the results do show a unique path in the domain of interdisciplinary research.
489

Chiang Ren, Gillian Green and Robert Wood
2. Methodology for approaching data poor problems
The central approach is to focus systems and complex systems analysis on a unique class of
problems and opportunities with poor supporting data. The objective is to integrate available
knowledge in new ways to gain insight on latent system dynamics, possible drivers, and potential
consequences. Today, systems thinking is recognized as a transdisciplinary approach with the
potential to bring holistic understanding across multiple disciplines (Meadows, 2008). Researchers
from multiple fields routinely form teams and work together on projects using a systems framework
where levels of the problem are either addressed by different experts (multidisciplinary research) or by
mixed groups of experts trying to integrate schools of knowledge (interdisciplinary research) (Borner,
2010). At the initiation point of our path of investigation, however, the crossing of disciplinary
boundaries was still a sensitive endeavor, often drawing criticism from researchers uncomfortable with
the introduction of alien methodologies and techniques. Thus, as recognized by other researchers,
the expanded application of systems approaches had to be very problem focused (Chubin, 1986).
By the 1990s, disciplinary boundaries were recognized as inhibiting the integration and generation of
knowledge, and a growing number of researchers were advocating the breaking down of the archaic
models in which disciplines were organized and knowledge was constrained (Klein, 1996). However,
studies of co-authorships and surveys among researchers were still showing that collaboration varied
great by the disciplines (Qin. 1997). While other interdisciplinary researchers increased efforts in
knocking down disciplinary boundaries and creating new hybrid disciplines (Lattuca, 2001), our
research strategy over the years has been to remain a trespasser into disciplines and research
clusters with very set attitudes, cognitive styles, and methods. While many researchers were seeking
disciplines with a wealth of data for integration, we continued to engage data poor areas by following
three simple but important steps:
Step 1: Observation of the world to find the problems and opportunities not yet investigated by
other scholars
Step 2: Definition of these problems through the identification of inconsistencies
Step 3: Formulation of predictive system and complex system constructs to achieve greater but
not necessarily precise insight
The significance of these steps and how they can create a coherence of research across different
problems and business areas is further discussed. The controversy in these steps is the inherent
willingness to challenge disciplinary views without regard for academic reputation, career impact, and
near-term validation of success. Scholars studying interdisciplinary activities have noted that
interdisciplinary research does not have to involve large collaborative teams or the breakdown of
disciplines through either unifying or transformative knowledge (Salter, 1996). However, studies of
have also shown that young scientists often turn away from interdisciplinary research to avoid risks
when on an academic tenure track (Rhoten, 2004). Our methodology suggests that disciplines can
still be impacted by intruders at points where other researchers have not gained interest.
Implementing this methodology will, however, require substantial commitment.
2.1 Observation of the world
To find problems in the real world environment without historical and interconnective data or with nonuseful
data gathered based on established schools of thought, the analyst must get involved in the
environment and search from within the business activities. This involves not just examining
disciplinary knowledge but also the entire way in which knowledge is generated, priorities are set, and
activities occur. Observations from outside the environment may not reveal all the forces and
behaviors at work. For example, studies have shown that the definition and coherent practice of
interdisciplinary research is still very lacking in the health care and health policy business areas
(Abolelela, 2006). So, our willingness to observe and study the forces and behaviors in that
environment has led to published works on a data poor problem and on a problem with overwhelming
data – both to be discussed later.
Once in the environment, the analyst might see that some significant activities occur at too low a
frequency or too isolated a locality to be statistically measured. Other activities may give clues
regarding the emerging pattern of future events. To make sense of these activities, the observer must
realize that the human mind can at times still gather and process information in ways that data mining
tools and techniques cannot. With evermore powerful tools for finding patterns in data, we often forget
490

Chiang Ren, Gillian Green and Robert Wood
that human cognition remains a uniquely powerful instrument. The analyst must therefore trust his /
her direct perceptions and perhaps probe the environment with his / her actions to stir-up more
dynamic behaviors.
As interdisciplinary research and working seamlessly across knowledge boundaries on data rich
problem areas became more promoted in the scientific communities, interest in data poor problems
may have actually been hindered. Since the 1990s, the U.S. National Institute of Health, Defense
Advance Research Agency, National Science Foundation, and Office of Naval Research have all
claimed to fund interdisciplinary research. However, such funding have followed the dominant
priorities of university groups, created new centers of gravity in research that resisted further
innovation, ignored complementary sciences, and created new mazes of activities for those
concerned with career risks (Metzger, 1999). Therefore, the ability to conduct unconstrained
innovation is still limited by funding, attitudes, and career risks. Even today, scientists and analysts
wishing to observe the world, cross into disciplines, and attack data poor problems will have to
contend with these limitations.
2.2 Identification of inconsistencies
Sometimes, a potential problem area may have to be directly observed and probed for years before
behavioral inconsistencies, such as hidden patterns and/or anomalies, can be seen. Once there is
awareness, the problem or opportunity needs to be characterized and better defined for systems and
complex systems analysis. For example, one set of characterization could be:
Awareness of possible future system behaviors and interactions (data to emerge)
Awareness of current system behaviors and interactions that cannot be seen outside the
reference frame (data not gathered)
Awareness of known mysteries in system behaviors and interactions that can be newly explained
(data not yet gatherable)
Awareness of how the system can be affected in new ways (influencing with limited data)
Awareness of how the system can be reorganized in new ways (controlling with limited data)
Awareness of how the system is evading awareness (correcting with limited data)
The above characterization happens to align with each of the six published papers to be presented
and they helped to establish the scope and objective for each analysis endeavor. One can also
characterize a problem or opportunity by the nature of incomplete data and reason for why it has been
inadequately studied.
This decision process for determining opportunities for interdisciplinary work fits within the boarder
concept of establishing interdisciplinary teams and initiating active learning approaches advocated by
other researchers (Repko, 2011). Interdisciplinary researchers understand that they must analyze
problems from the perspective of different disciplines, integrate insights, resolve conflicts, find
common ground, and produce interdisciplinary understanding. For data poor problems, this
understanding will often have to be asymmetric in nature where the intruding systems researcher
takes on the greater burden of insight integration, adapting system discussions to the style and
taxonomy the problem environment, and presenting new findings.
2.3 Formulation of predictive system or complex system construct
As the objective is to gain system insight and not necessarily to advance analysis techniques, highly
complex systems can sometimes be analyzed through traditional deductive and inductive analysis if
the techniques are focused on addressing the specific problem characteristics. If a complex system
without supporting data can be conceptually collapsed to isolate higher order dynamics, then
deductively formed hierarchies, matrices, and databases might be able to look past the complexity
into core behaviors driving future consequences. If a complex system can in whole or in part be
conceptually bounded, then inductively formed functional diagrams, computer models, and theories
might be able to look past the complexity into core behaviors. The integration of modeling techniques
and empirical understanding across disciplines has been recognized by other researchers as
requiring time and patience (Blythe, 2010). They have also advocated the initial use of simple models
with focus on patterns instead of specific data points. The move to more complex models can then
occur iteratively as insight is gained.
491

Chiang Ren, Gillian Green and Robert Wood
Sometimes complexity cannot be ignored from the start. When the problem lies within the complexity
and uncertainty of elements and relationships, explorative / agent-based methods might be effective.
The application of agent-based models against data poor problems and opportunities is specifically
discussed in our paper for systems engineering (Ren, 2003) and the application of deductive and
inductive methods to support agent-based modeling is demonstrated in our paper for public
administration (Ren, 2012). While agent-based modeling capability for complex systems analysis has
been advancing in the research communities, its application in some areas such as within the
government has remained limited. As a result, there are still many opportunities for complex systems
analysis to push past barriers due to poor data.
Scientific communities, such as anthropology, have argued for the application of different research
paths. However, the ideal path for many remained experimental, reductionist, and narrowly
disciplinary (Abel, 1998). Even with the emphasis on interdisciplinary, integrative, analytical, and
comparative research, such results have not always been fully embraced. Therefore, getting scientific
communities to consider the theoretical results of studying data poor problems can be overwhelming.
This is in part why we have categorized our investigation as promoting innovation in business areas
instead of scientific areas. Focusing on business innovation reflects an attempt to step away from
disciplinary debates and argue for theoretical considerations based on the practical urgency of
problems.
2.4 The way forward
Our methodology does not align perfectly with the interdisciplinary objective of harvesting,
capitalizing, and leveraging multiple expertises. Also, our results do not necessarily touch upon all the
principles of interdisciplinarity as proposed by some evaluators (Klein, 2008). These principles
include: 1) variability of goals, 2) variability of criteria and indicators, 3) leveraging integration, 4)
interaction of social and cognitive factors in collaboration, 5) management, leadership, and coaching,
6) iteration in a comprehensive and transparent system, and 7) effectiveness and impact. Yet, we
believe that the methodology contributes to the broader understanding of opportunities in
interdisciplinarity.
Through multiple case studies, the breath and growing importance of interdisciplinary and
transdisciplinary research plus knowledge application are being documented (Frodemen, 2012). As
many other interdisciplinary researchers are trying to reshape academic communities, influence public
policy, and shift society overall, we propose the more modest objective of addressing the problems
left behind. Unfortunately, this could be a lonely path as hiring practices, promotion opportunities,
tenure considerations, and funding will most likely continue to favor traditional disciplinary centric
paths of research (Committee on Facilitating Interdisciplinary Research, 2005). Even when there is
funding for interdisciplinary studies, such funds will most like be for researching problems with enough
validating data.
3. Demonstrating methodology for data poor problems
The following short presentations will demonstrate that the proposed path in studying data poor
problems can be taken and that research results in multiple disciplines can find acceptance by
different peer groups. Please reference the actual publications for detailed results.
3.1 Data poor problem in disaster preparedness
To demonstrate that systems analysis can be applied to gain awareness of possible future situations
that are not indicated by current data, we observed from within the business area of U.S. Homeland
Security around the period from 1999 to 2000. The observations showed that U.S. crisis planning and
disaster management efforts tend to be either against specific isolated crisis events or for building
general capability against all hazards. There was no material on how to plan against multiple crises
occurring in a coupled manner, and there was a lack of historical data to support analysis. In
response, a paper was published to introduce the worldwide crisis planning community to the benefits
of predictive systems analysis (Ren, 2000a). As there was limited historical data, the analysis focused
instead on types of linkage between crisis events to isolate macro-dynamic characteristics and
important consequences.
In the publication, a conceptually simple model of crises is summarized where passive crisis response
such as reinforcing structures and positioning population centers can reduce the severity of crisis.
492

Chiang Ren, Gillian Green and Robert Wood
However, it is active response involving the mobilization of society through the machines and
command / control / communications / intelligence (C3I) systems of the modern age that can reshape
the recovery profile to restore societal integrity. Using this model of crisis, there are four types of
linkage if a second or third crisis occurs before recovery from the first crisis is complete.
Linkage Type 1 (societal coupling) is when a second crisis occurs within the same society before
recovery from the first crisis is complete.
Linkage Type 2 (spatial coupling) is when there is an overlap between the impact region of the
first and second crisis.
Linkage Type 3 (operational coupling) is when two crises impact the operational processes of
society in a manner that amplify effects.
Linkage Type 4 (resource coupling) is when crises drain common resources such as medical
supplies, building material, and rescue personnel, or cut-off common resources such as food,
water, and fuel.
For linked crises scenarios new complementary strategies, such as those below, should be
considered.
Controlling the slope of the recovery curve from the first crises to minimize total damage or
optimize resources when confronting the second crisis
Shaping the impact areas of some crises to decouple spatial overlaps or to leverage overlaps to
protect other regions
Adjusting operations in the least obstructive manner to decouple the effects of crises
Identifying all potential shortages, stockpiling resources, and increasing distribution capability
during linked crises events
In the years that followed publication, the importance of planning against linked crises was
demonstrated in events such as: 1) the second tower being hit by an airplane on Sept 11, 2001, 2) the
failure of levies in New Orleans after Hurricane Katrina, and 3) the nuclear problems that followed the
tsunami in Japan.
3.2 Data poor problem in defense planning
To demonstrate that systems analysis can be applied to gain awareness of current system behaviors
that cannot be measured outside a business reference frame, we observed the changing dynamics of
the U.S. defense planning community after the end of the Cold War and the first Persian Gulf War.
This observation concluded that, under the new priorities of reducing defense spending and planning
against more complex threats, the organic capabilities of acquisition professionals in mid-level
organizations were being eroded. These professionals applied experientially gained knowledge and
ad-hoc analysis capabilities to agilely connect strategies and developmental activities. In response, a
paper was published to show how forces at the leadership level and at the program execution level in
the United States were weakening the capabilities and effectiveness of mid-level connective entities in
the 1990s (Ren, 2002). As the defense environment became more volatile and unpredictable in the
post September 11, 2001 world, the paper predicted that U.S. defense systems acquisition and
utilization strategies may become disconnected from mission needs.
The U.S. defense community with its millions of stakeholders is clearly a complex system. However,
some parts of the system which are causing dynamic limitations can be isolated and conceptually
simplified to enable inductive analysis. While systems analysis has been extensively used by the U.S.
defense community in developing specific weapon systems, the community has not been so ready to
examine its own organizational behaviors in such a manner. First hand observations of mid-level
organizations revealed the following essential qualities.
Institutional Memory: Actual inheritance of experience passed down through mentorship
Interdisciplinary Capabilities: Key personnel’s ability to understand the interplay of diverse
disciplines from technology management to policy
Culture that Promotes Problem Identification and Impromptu Analysis: Willingness for people to
go beyond official coordination and integration responsibilities to informally address problems
493

Chiang Ren, Gillian Green and Robert Wood
Inductive analysis on the connectivity and forces acting at both ends of mid-level organizations
revealed a polarization of activities that reduced their ability to recommend adjustments. Instead of
leveraging the connectivity of the Information Age to improve coordination processes, mid-level
professionals often used computer-based tools to increase the complexity and bureaucracy of basic
functions to preserve jobs. As misalignments between the two ends of the system grew and the ability
of mid-level organizations to promote system agility and adaptiveness diminished, what naturally
occurred was a self-formed recursive process where activities were further increased at both ends.
As the acquisition system was reduced after the Cold War, the system started to confront a more
complex acquisition environment. This environment amplified the consequence of losing capabilities
within the mid-level organizations.
3.3 Data poor problem in cellular biology
To demonstrate that systems analysis can be used to provide new explanations for mysteries in
system behaviors, we turned to the field of cellular biology. The success of the human genome project
at the end of the 20th Century led to a deeper understanding of how cells should grow relative to one
another in the body based on DNA guidance and how DNA is organized to form genes that are
expressed by transcription factor proteins to control cellular functions. The commonality of genes
within each cell of an organism affirms the idea that the genes are objective maps and that a cellular
reference frame across the body is required for each cell to correctly used the map for growth and
specialization. The mystery in system behavior, which we can all observe within our own bodies
everyday, is how such a reference frame that relates cells to one another works as there is little to no
data to aid research. Therefore, cellular biologists have largely stayed away from studying this
mystery, preferring instead to devote energy towards the mounds of data coming from human
genome research. In response, a paper was published to show the biological research community
how systems analysis can propose a new course of research (Ren, 2000b).
Based on how DNA must be applied, we can conclude that the cellular reference frame should have
an awareness of cell position relative to other cells and an awareness of time as in generations of cell
growth. Awareness of position is important because the cells must organize into body shapes and
specialize to support component functions. Awareness of generation is important because the growth
of cells changes base on age from embryonic development through adulthood. In order for cells to
have a localized mechanism for positional and temporal awareness, there must be markers in each
cell that differentiate its position and generation from other surrounding cells. Then, intercellular biochemical
communication of marker information can coordinate growth and specialization. As the
markers have to be advanced with each cell division, there are only a few mathematical options.
If we let N0 = marker number of the parent cell and N1 plus N2 = the marker numbers of the children
cells, then the types of advancement schemes fitting our system requirements include:
N1 = N0 + N0 and N2 = N0 + N0 + 1
Leading to the progression of (1 | 2, 3| 4, 5, 6, 7| 8, 9, 10, 11, 12, 13, 14, 15| …) as N1 and N2 become
N0 for the next generation.
N1 = N0 + N0 - 1 and N2 = N0 + N0
Leading to the progression of (1| 1, 2| 1, 2, 3, 4| 1, 2, 3, 4, 5, 6, 7, 8| …) as N1 and N2 become N0 for
the next generation.
The generational and position marker numbers, when aligned with the DNA map, could tell cells when
to specialize, what shape to grow into, and how to repair injuries. Taking deductive analysis a step
further to discover where to search for supporting data, the paper further concluded that the proposed
marker information are most likely stored in the ribosomes and their constituent rRNAs. Scientifically,
the structure and functionality of the ribosome is still not fully understood as the use of x-ray diffraction
techniques, cross-linking immunoelectron microscopes, and low-angle neutron scatter methods have
only provide partial results.
3.4 Data poor problems for systems engineering
To demonstrate how complex systems can be affected in new ways without the support of adequate
data, a proposal was presented to the systems engineering community for the development of new
494

Chiang Ren, Gillian Green and Robert Wood
tools to conduct agile complex systems analysis. This proposal was based on the recognition that
some data poor areas cannot be properly addressed by the simplification of system concepts to
enable deductive and inductive analysis. Nevertheless, the urgency of problems in many disciplines
may require actions to be taken against unknown system behaviors. A paper was published to explain
how current world problems often do not have the right level of supporting data and thus can benefit
from tools that allow analysts to: 1) rapidly build ad-hoc agent-based models, 2) test complex system
dynamics across a range of potential scenarios, and 3) flexibly adjust the agent constructs and
simulation conditions to improve insight (Ren, 2003).
The proposed complex systems engineering tools and techniques should be based on the following
principles to cope with disconnects in data and leverage the explorative methodology in agent-based
models.
Design Principle 1: Lean Characteristics for Agile Operations – tools and techniques must oppose
the nature of complexity in promoting evermore elaborate models.
Design Principle 2: Features for Rapid Adaptation and Simulation – tools and techniques must
allow for rapid adjustment to agent and environmental constructs to enable modification of initial
imperfect simulations.
Design Principle 3: Synergistic Operations with User Cognition – tools and techniques must
leverage cognitive theories to help users mentally go beyond systematic reasoning to see the
deeper nature of complexity.
Design Principle 4: Visualization Centric Presentations – tools and techniques must help users
see interlocked parametric behaviors such as links, configurations, changing relations, and
relative motion.
Using different visualization schemes, the tools and techniques can not only model complex systems
but also search across disconnected data to identify voids and potential data groups. By exploring
what could be in the voids and probing data groups through predictive system constructs, system
engineers can help scientists figure out new ways to search for connective data.
3.5 Data poor problem in the acquisition of IT systems
To demonstrate how a complex system can be reorganized in new ways, observations revealed that
the overall acquisition process for U.S. defense IT systems possessed complexities that have not
been fully mastered by government program managers. These complexities, which are often driven by
the pace of technology advancement and the lack of coherent data to gain performance understand,
presented an opportunity for innovation through systems analysis. In response, a paper was
published to present a systematic way to better initiate defense IT programs for avoiding cost,
schedule, and performance problems in the acquisition lifecycle (Ren, 2010).
The paper first showed that current acquisition processes do not fully address the complexity caused
by a large number of stakeholders, requirement sources, integratable technologies, development
opportunities, and/or mission priorities. As a result, what constitutes a single IT system is not always
clear. To accurately define the system concept, a reference frame must be established for mapping
the total system. Then, the acquisition community can structure user requirements and concepts into
a manageable development approach where the integration of too many IT components or scaling to
too broad a user base can be avoided. A reference frame is proposed in three dimensions.
Dimension 1 (Mapping the Magnitude Level of the Concept): Level 1 (Subroutines), Level 2
(Modules), Level 3 (Applications), Level 4 (Environments)
Dimension 2 (Mapping the Utility Scope of the Concept): Scope 1 (Unit Support), Scope 2
(Organizational Support), Scope 3 (Enterprise Support), Scope 4 (Societal Support)
Dimension 2 (Mapping the Delivery Sources for the Concept): Source 1 (Equipment Specific),
Source 2 (Local Distributed), Source 3 (Network Based), Source 4 (Cloud Based)
Once user requirements and concepts of needs are mapped to the reference frame, the mappings
can be studied and reorganized to establish the best initiating concepts for systems acquisition. This
reorganization can be by the following methods.
Method 1: Forming multiple systems or reduced systems based on decomposition of the mapped
concept.
495

Chiang Ren, Gillian Green and Robert Wood
Method 2: Forming a single system or reduced set of systems based on integration of multiple
mapped concepts.
Method 3: Establishing co-development activities between multiple mapped concepts.
Method 4: Forming new systems based on reorganization or redefinition of activities for mapped
concepts.
The value of controlling complexity at the initiation point of IT systems acquisition within the U.S.
Department of Defense implies that the acquisition process should start earlier with a pre-concept
phase. However, the idea of a soft start may be difficult to embrace for a community driven by precise
paths, set timelines, and clear decisions.
3.6 Data poor problem in public administration
To demonstrate how organic systems that are evading awareness through self-adaptation can be
corrected, observations showed that the U.S. government workforce is not always meeting
transformational objectives. In response, a paper was published to explain how localized interactions
by select individuals in the government can yield proliferated impact and hidden consequences for the
total organization (Ren, 2012). When there is insufficient data on disruptive individuals seeking to
falsely change other’s perception of their utility, authority, and criticality, predictive agent-based
models of the organizational environment with select actors might reveal how the effectiveness the
total organization will be impacted.
To support agent-based modeling, this paper took the first step of identifying all the perception
altering approaches associated with a generic government worker seeking to game his/her situation
for false gains.
Perception Altering Approaches Based on Positional Relations:
Join internal events of high importance regardless of one’s role
Join external events of perceived high prestige
Establish many relations regardless of functional importance
Insert oneself into hierarchy regardless of defined role
Establish ambiguous chains of command and shared power
Attach oneself at lowest level of hierarchy
Perception Altering Approaches Based on Motion:
Maintain commitment to clear but stove-piped path
Involve many others in path through coordination and collaboration
Follow a complex path of activities that is very hard to understand
Follow a fast path with no defined direction
Intensely engage others on multiple issues
Evade engagements with others on key issues: in large organizations
Position to watch and align with winners in conflicts:
Even before a specific organizational analysis is conducted, the nature of impact by agents
(government workers) can be described.
Types of Disruptive Impact:
Underperformance of critical functions
Confusion in the management process
Validation of ability to abandon responsibilities
Communication barriers
Wrong conflict results
Competing or altered processes
496

Chiang Ren, Gillian Green and Robert Wood
Types of Degrading Impact:
Proliferation of behaviors
Distrust among coworkers
Cynicism about the merit process
Wasted effort by others
Unnecessary consumption of resources
Current public administration resource commitments are very aligned with established academic
schools of thought and survey results. Therefore, the tendency is to automatically increase training
and incentives as well as to restate goals when a performance issue is suspected. To break past
these disciplinary barriers, systems analysis has shown that there are others ways to address
performance issues which focus on root causes.
4. Conclusions
With so much data being generated in our modern world and so much computing power available to
analyze data, it is easy to forget that there are still real world behaviors that are not revealed by
existing data and cannot be well measured by current disciplinary means. The identification of data
poor problems requires patience and real world engagement by the researcher. The solution
approaches that stretch across disciplinary boundaries could be imperfect to start. However, the
payoff may be new insights on mysteries that have plagued other researchers for years and new
courses of action that will innovate all manners of business.
This paper showed how different data poor problems have been approached using our proposed
methodology and how the methodology fits within the advancing course of interdisciplinary research.
The methodology can also be used to provide new ways to interpret overwhelming data associated
with problems. Therefore, systems analysts are encouraged to lead innovation efforts when the lack
of data has hindered other researchers and when the overwhelming amount of data has trapped
researchers in constraining schools of thought.
References
Abel T. (1998) “Complex Adaptive Systems, Evolutionism, and Ecology within Anthropology: Interdisciplinary
Research for Understanding Cultural and Ecological Dynamics”. Georgia Journal of Ecological
Anthropology, Vol. 2, pp. 6 – 29.
Abolelela S., Larson E., Bakken S., Carrasquillo O, Formicola A., Glied S.A., Haas J., and Gebbie K.M. (2006)
“Defining Interdisciplinary Research: Conclusions from a Critical Review of the Literature”. Health Services
Research, Vol. 42, Issue 1, pp. 329 – 346.
Bar-yam Y. (2003) Dynamics of Complex Systems. New York, NY, Westview Press.
Blanchard B.S. and Fabrycky W.J. (2010) Systems Engineering and Analysis 5 th Edition. Upper Saddle River, NJ,
Prentice Hall International Series in Industrial and Systems Engineering.
Blythe R.A. and Croft (2010) “Can a Science – Humanities Collaboration Be Successful”. Adaptive Behavior, Vol.
18, No. 1, pp. 12-20.
Borner K., Contractor N., Falk-Krzesinski H.J., Fiore S.M., Hall K.L., Keyton J., Spring B., Stokols D., Trochim W.,
and Uzzi B. (2010) “A Multi-Level Systems Perspective for the Science of Team Science”. Science
Translational Medicine, Vol. 2, Issue 49, pp. 49cm24.
Chubin, D.E. (1986) Interdisciplinary Analysis and Research: Theory and Practice of Problem-Focused Research
and Development. Mt. Airy, MD, Lomond.
Committee on Facilitating Interdisciplinary Research, National Academy of Sciences, National Academy of
Engineering, Institute of Medicine (2005) Facilitating Interdisciplinary Research. Washington D.C., National
Academies Press.
Eisenhardt K.M. (1989) “Building Theories from Case Study Research”. The Academy of Management Review,
Vol. 14, No. 4, pp. 532-550.
Frodemen R., Klein J.T., and Mitcham C. (2012) The Oxford Handbook of Interdisciplinarity. Oxford, U.K., Oxford
University Press.
Klein J.T. (1996) Crossing Boundaries: Knowledge, Disciplinarities, and Interdisciplinarities. Charlottesville, VA,
University of Virginia Press.
Klein J.T. (2008) “Evaluation of Interdisciplinary and Transdisciplinary Research”. American Journal of Preventive
Medicine, Vol. 35, Issue 2, pp. 116-123.
Lattuca L.R. (2001) Creating Interdisciplinarity: Interdisciplinary Research and Teaching Among College and
University Faculty. Nashville, TN, Vanderbilt University Press.
Meadows D.H. (2008) Thinking in Systems: A Primer. White River Junction, VT, Chelsea Green Publishing.
497

Chiang Ren, Gillian Green and Robert Wood
Metzger N. and Zare R.N. (1999) “Interdisciplinary Research: From Belief to Reality”. Science, Vol. 283, No.
5402, pp. 642-643.
Moran J. (2010) Interdisciplinarity 2 nd Edition. New, NY, Routledge.
Ren C.H. (2000a) “Understanding and Managing the Dynamics of Linked Crisis Events”. Disaster Prevention and
Management, Vol. 9 Num. 1.
Ren C.H. (2000b) “Additional Factors Governing Specialized Cell Dynamics Based on Deductive Systems
Analysis”. Theory in Biosciences, Vol. 119-2.
Ren C.H. (2002) “The Dynamic Limitations of the U.S. Defense Acquisition Process”. Defense & Security
Analysis, Vol. 18 Num. 2.
Ren C.H. (2003) “Expanding the Role of Systems Engineers in the Age of Escalating Complexity”. Systems
Engineering, Vol. 6 Num. 2.
Ren C.H. , Busch S., and Prebble M. (October 2010) “Improving the Initiation of Acquisition Activities for
Automated Information Systems”. Defense Acquisition Review Journal, Defense Acquisition University,
Vol.17 Num. 4.
Ren C.H. (2012) “Micro-dynamic Disturbances in the Government Workforce as a Cause of Organizational Poor
Performance”. The Innovation Journal, Accepted for Publications in Vol. 17 Num. 3.
Repko A.F. (2011) Interdisciplinary Research: Process and Theory 2 nd Edition. Thousand Oaks, CA, Sage
Publications.
Rhoten D. and Parker A. (2004) “Risks and Rewards of an Interdisciplinary Research Path”. Science, Vol. 306,
No. 5704, p. 2046.
Qin J., Lancaster F., and Allen B. (1997) “Types and Levels of Collaboration in Interdisciplinary Research in the
Sciences”. Journal of the American Society for Information Science, Vol. 48, Issue 10, pp. 893-916.
Salter L. and Hearn A.M.V. (1996) Outside the Lines: Issues in Interdisciplinary Research. Montreal, Quebec,
McGill-Queen’s University Press.
498

Appreciative Leadership: Dreaming the Underland-to-
Wonderland Dream for Managing a Higher Education
Institution
Emmie Smit and Rita Niemann
Faculty of Education, University of the Free State, Bloemfontein, South Africa
smitej@ufs.ac.za
niemannrs@ufs.ac.za
Abstract: A vision is seen as a valued and strategic tool for a leader, but a dreaming leader might be frowned
upon. However, Appreciative Inquiry (AI) and Appreciative Leadership employ ‘dream’ as one of the crucial
elements in this qualitative research method. This conceptual paper differentiates between predominantly
conscious-mind or preconceived dreaming (where the vision is restrained and directed by components of strategy
formation) and predominantly subconscious-mind or free-range dreaming (where the vision unveils a future
unlimited by conscious constraints). An alternative concept of vision that might assist leaders to drive creative
transformation, instead of merely adjusting and upgrading existing strategies and practices is conceptualised.
Theoretical support is supplied by the Alice in wonderland chronicle (2010) as well as research by Harvard’s
Medical and Business Schools. The paper forms part of a bigger research project within a post-critical paradigm,
whereby the reflective and theoretical epistemology of leadership include metaphors of Alice in Wonderland.
These metaphors are juxtaposed with the University of the Free State—where an internationally awarded
transformation took place after a debacle that was covered by various mayor international news agencies.
Keywords: University of the Free State, Alice in Wonderland, appreciative Inquiry, visionary dream, vision
statement, appreciative leadership
1. Introduction
Visions and dreams are often seen as mere metaphors, ideas, emotions, and insights that are part of
spiritual guidance, but in the management sector dreams are regarded as a self-determined
destination, a conscious developed vision of long-term goals or a preconceived future that is
determined by the available recourses. Some dreams are constructs originating predominantly from
the conscious mind and others, predominantly from the unconscious mind. These differences might
be two opposing concepts or two halves of the same concept. The connection between AI and the
dream will be dealt with later in detail.
2. Whose message is a dream?
Ancient cultures distinguished between three types of dreams: the first are divine messages or
revelations; the second are those that reflect the physical and psycological state; and the third are the
visionary dreams that subconsiously perceive forthcoming events. A semantic study on the origins of
the word ‘dream’ in various early languages indicates that it originated from the first and third types of
dream or tesha-/zashai, and therefore ‘dream’ means ‘divine guidance’ (Byrd, 2011). According to this
perception, dreams are interior realities, subjective to the divine creators thereof and objective to the
receivers thereof.
However, leadership books dynamically define a vision as a dream that is consciously conceptualised,
and to some extent as ‘a clearly stated verbal picture of what you will be like when you are living,
functioning, and performing at your best – the more concrete and specific, the better. To be powerful,
vision statements should run well ahead of what you can now do, as they describe what you’re
stretching to become. They are more powerful if written in the present tense: ‘I am being/doing that
remarkable thing!’ (Schwah & Spady, 2010, bl. 44). This perception of a dream, as an exterior reality,
subjective to the team/committee that created it, and objective to the receiver/s of it, echoes the
widespread paradigm of the manegement sector (Lehrer, 2009). Even Peter Drucker’s ‘planned
abandonment’ practice echoed the need to abandon or reject the continuation of existing tactics,
strategies, or techniques. Even though consciously known practices contributed to past successes, by
continually establishing innovative and previously non-existent means to arrive at the destiny prevents
stagnation (Drucker, 2008).
The introductory scenes of Tim Burton’s Alice in Wonderland contain two types of visionary dream of
the future. Charles Kingsley, successful entrepreneur, dreamed a conscious-mind dream, while his
499

Emmie Smit and Rita Niemann
daughter, Alice, dreamed a subconscious-mind dream. However, both were taken seriously by the
people who experienced them and belittled by those who did not experience them.
FADE IN:
1 EXT. LONDON – 1855 – NIGHT 1
Alice
(Script of movie directed by Tim Burton, 2010)
by
Linda Woolverton
Based on
Alice's Adventures In Wonderland
Through The Looking Glass
by Lewis Carroll
Warm light shines from the study of the gracious Kingsley home. A man, silhouetted in the window,
speaks with ardour.
2 INT. THE STUDY – NIGHT – CONT. 2
CHARLES KINGSLEY has just described his new venture to his friends, including LORD ASCOT.
LORD ASCOT: Charles, you have finally lost your senses.
A COLLEAGUE: This venture is impossible.
CHARLES KINGSLEY: For some. Gentlemen, the only way to achieve the impossible is to believe it
is possible.
A COLLEAGUE: That kind of thinking could ruin you.
CHARLES KINGSLEY: I’m willing to take that chance. Imagine trading posts in Rangoon, Bangkok,
Jakarta...
(He stops. His nine-year-old daughter ALICE stands at the door in her nightgown, clearly frightened.
He goes to her.)
CHARLES KINGSLEY: The nightmare again?
(She nods. He takes her hand and turns to his guests.)
CHARLES KINGSLEY: I won’t be long.
3 INT. YOUNG ALICE’S BEDROOM – LATER 3
(He sits on her bed, listening with utmost seriousness.)
ALICE: I’m falling down a dark hole, then I see strange creatures...
CHARLES KINGSLEY: What kind of creatures?
ALICE: Well, there’s a dodo bird, a rabbit in a waistcoat, a smiling cat...
CHARLES KINGSLEY: I didn’t know cats could smile.
ALICE: Neither did I. Oh, and there’s a blue caterpillar.
CHARLES KINGSLEY: Blue caterpillar. Hmm.
ALICE: Do you think I’ve gone round the bend?
(He feels her forehead as if for a fever.)
CHARLES KINGSLEY: I’m afraid so. You’re mad. Bonkers. Off your head. But I’ll tell you a secret . . .
all the best people are.
(She smiles and leans against him.)
500

Emmie Smit and Rita Niemann
CHARLES KINGSLEY: It’s only a dream, Alice. Nothing can harm you there. But if you get too
frightened, you can always wake up. Like this.
(He pinches her. She screams and pinches him back.)
3. Dreams and visions in strategic leadership and their linkage to appreciative
inquiry
The fundamental components of basic strategic leadership concepts include an understanding of the
individual or institutional vision, mission, values and strategies. The vision sketches the dream of
‘what the future might be’, ‘what if’ or ‘what is the ultimate long term ideal’. The mission defines the
fundamental purpose of the individual/institution, while the values are the shared belief (culture and
priorities) of the stakeholders. The strategy, again, is the detailed description of the direction and of
how the individual or institution will reach the envisioned destination. The available resources:
expertise, time, and finances, form the basis of the strategy; although it should not exclude ad hoc
resources, and definitely not influence the visioning. Strategy bridges the gap between the current –
where we are – and the future in terms of where we want to be (Nickols, 2011).
The components of strategy formation in the strategic leadership process include aspects such as
objectives, goals, position, status strengths, potential, limitations, opportunities and economic trends
(Niemann & Kotze, 2006). Unfortunately, the vision is often limited by these components. In the
second phase of AI’s 4D-cycle, Bill George encourages leaders to dream freely about ‘what if’ and
‘what might be’ and not to consider the components of strategic planning. Strategic planning
components first appear in the third phase of the AI-cycle and in the fourth phase of strategy
formation process (Figure 1).
Figure 1: The vision is often dictated by the components of strategy formation
Bill George (George, 2009), former chairman and CEO of Medtronic and Professor of Management
Practice at Harvard Business School, urges leaders to dream dreams of unexplored destinies:
‘Let’s find a better way to do things! We have to have a vision about what it could be, not what it is‘
(George, 2009). As an Appreciative Leadership practitioner, Bill George values the dream as part of
the Appreciative Approach’s 4D cycle. In 1980, as a doctoral student under the academic guidance of
Suresh Srivastava (Deschamps, 2008), David Cooperrider conceptualised the transformative
dynamics of endorsing and focussing on envisioning ‘what works’ rather than on ‘what does not work’
(Cooperrider & Whiteney, 2008). Cooperrider then developed Appreciative Inquiry – a value-adding
research approach, instead of the traditional deficit based approach (Cooperrider D. W., 2003). AI
encourages ‘what if’ – dreaming as the core of the developmental process of the vision (Figure 2).
Figure 2: Cooperrider's appreciative 4D-cycle
501

Emmie Smit and Rita Niemann
The value-adding approach of Appreciative Inquiry is in opposition to a problem-solving approach, as
indicated in table below.
To dream is to envision ‘What might be’. Together people build a vision of a future they want. They
respond to their sense of what the world is calling them to become. They imagine that the best of
‘what is’ forms the foundation for the way things are to be in the future. Questions in this phase
include ‘What does our positive core indicate that we could be?’; ‘What are our most exciting
possibilities?’; and ‘What is the world calling us to become?’ (Whitney & Ludema, 2006).
The 7 Pillars of Visionary Leadership (Rock & Cox, 1997) identified seven foundations on which
creative dreaming leaders can build:
Visioning: I envision where I am going;
Mapping: I know how to get there, I have mapped indicators;
Journeying: I am willing to leave my old points of view behind and start on a shared future;
Learning: I am open to change: to move from design to destiny;
Mentoring: I am open to share with others: to direct them to a future;
Leading: I will set a visible example of venturing with experience; and
Valuing: I will act with honour and transparency.
These foundations incorporate the belief that visionary leadership is transformative. Each of the
components includes aspects of dreams/visions that we need to activate in order to transform our
hearts and minds. All are connected to seeing forthcoming events spiritually. These components
constitute the vision to have the perspective to see; the insight to inspire leadership; the images to
visualize change; and the spirit to guide the transformation.
The reality in the world of leadership is that although dreams of visions are considered a must-have,
and extreme amounts of money and time are invested in their development, the general culture of
mediocrity makes it difficult to detect the dynamic, extraordinary and creative character of divinity in
corporate and institutional dreams (Krippner, Bogzaran, & de Carvalho, 2002), just as in ancient
times. However, painstakingly developed visions and dreams usually have a passive role in corporate
and institutional leadership realities (Richter, n.d.) by not being involved in visible reaction or active
participation in leadership processes, even though visions are seen as one of the most important tools
of successful leaders.
4. Efficacy variance of institutional dreams and visions
Countless dreams and visions of institutional leaders lack efficacy as guidance towards results with
divinely inspired quality, sustainability and dynamics. However, some dreams are seen as powerful
assistance, offering guidance towards innovative and vibrant ideas that materialise into transformative
authenticities. Even though scholars and researchers of dreams often admit that humans know very
little about dreams, new knowledge was established recently (Barrett, 2001; Kaufman, 2010;
Hartmann, 1995) that supports ancient beliefs about divine intervention in the human subconscious.
In this regard, Jung (Jung, 1945) was convinced that dreams reflected the richness and complexity of
the unconscious or subconscious, but also that conscious issues are only a small part of the
unconscious. Jung suggested that the dreamer should imagine the image as vividly as possible and
connect to it an explanation of the logic of the dream. He believed that dreams convey ineluctable
truths, philosophical prophecies, daydreams, memories, plans, irrational experiences and even
502

Emmie Smit and Rita Niemann
spiritual visions. Jung argued that the un/subconscious experience is as important as the conscious
one.
Deirdre Barrett (2001) clinical psychologist at the Harvard Medical School documented her dream
incubation study involving prominent artists and scientists who draw inspiration from their dreams as
well as some of her students who came up with novel solutions in their dreams when they focused on
a challenging issue before going to sleep. She described her approach as follows:
My emphasis has been on its relation to creativity and objective problem-solving, dream
incubation, and the role of dream in PTSD [Post-traumatic stress disorder].
Theoretically, I'm an evolutionary psychologist; I believe that dreams are essentially
thinking in a different biochemical state and that they can be extremely helpful because
of focusing on our life issues from a very different perspective. (p. 1).
Other studies also confirm the innovative and directive properties of sub-conscious dreaming.
‘Dreams are the royal road to the unconscious’, according to Sigmund Freud’s well-known statement.
(Hartmann, 1995). In studies, trauma suffers find that dreaming contextualises, guides and organises
emotion. In addition, dreams make more connections more widely, more broadly, more laterally than
does the conscious mind. Dreams are often, but not always, a metaphorical explanation of challenges
in the dreamer’s life. Although the basic nature of differs between individuals – from vivid to trancelike,
from emotional to aloof – dreams assist in developing the nervous system. Hartmann (1995) suggests
that dream’s connection-making ability is one of the means used by humans to devise and develop
new inventions: from equipment to approaches. He refers to the example of how the idea of a sewing
machine needle with the hole at the tip came in a dream to Elias Howe.
Other studies also concluded that the connection-making ability of dreams makes humans smarter,
boosts memory, creativity, and enhances our ability to be strategic. Recent psychiatric research by
the Universities of California and Harvard indicated that people’s ability to combine ideas in new ways
increased after REM [rapid eye movement] sleep, the phase of sleep in which dreaming happens.
The ability to connect seemingly unrelated concepts, or imagining, originates with past experiences,
but continues with predictions or expectations of future experiences. The same areas in the brain that
is responsible for memory seems to be responsible for imagining future events such as long-term
visions in visionary leadership (Kaufman, 2010).
5. The sub-conscious mind, Alice and leadership
When comparing Alice’s dream with her father’s dream, the difference in the vibrancy and
extraordinary is obvious. Charles Kingsley’s dream contains aspects that are known of, but not
experienced. The conscious and cognitive are obvious, even though Lord Ascot describes Charles’
dreams as mad, impossible and financially suicidal. Charles’ dream to develop trading posts in
Rangoon, Bangkok and Jakarta might still be accepted as long as it stays an interior reality, but to
turn the dream into exterior reality was not even considered. On the contrary Alice Kingsley’s dream
contains characters, spaces and events from the subconscious, the impossible that definitely does not
or could not exist, not even as an interior reality: an extinct bird, a clothed and talking rabbit, a smiling
cat that could appear and disappear, and a hookah-smoking caterpillar all living in Underland - which
you enter through a rabbit-hole (Carroll, 1864/1969).
Both of these dreams, interior or sub-conscious realities, are unified with their exterior or conscious
realities. Charles’ dreams to expand commercial ventures during a time of transformation in the East
were, in retrospect, not only realistic, but quite logical – to the extent that Queen Victoria was declared
Empress of India fifteen years later. The unification of Alice’s interior and exterior realities resulted in
countless books, research papers, movies, live performances and other cultural experiences and one
of the biggest money-makers ever to have accumulative sustainability (Breton 1924/1971) (Breton,
1971). Here realism confronts the unification of interior and exterior realities, because they are seen
as the exact opposites of each other. Surrealism, however, views the unification of interior and
exterior realities as two parts that form a complete reality after unification. The distinguished Professor
of Mathematics and a specialist in logic at Oxford, and creator of the Alice in Wonderland legacy,
Charles Lutwidge Dodgson (alias Lewis Carroll), can be regarded as a surrealist precursor. This
honorary acknowledgement is remarkable since he enabled the literary nonsense of the unification of
Alice’s interior and exterior realities to come into being half a century before the surrealist movement
emerged (Breton, 1971). When higher educational leaders dare to dream the impossible dream, even
503

Emmie Smit and Rita Niemann
if it seems to be nonsense, this might initiate an accepted and important movement or principle in the
future.
FADE IN:
University of the Free State’s
(script of transformation directed by the current UFS Rector, Jonathan Jansen, 2009/10)
by
Emmie Smit
Based on
Kovsies Adventures In Underland
Through The Eyes of Social Justice
by the International Media
1 EXT. BLOEMFONTEIN, SOUTH AFRICA – 22 February 2008 – EARLY MORNING
The sun rises over a calm Kovsie campus. All signs of vandalism – the red graffiti messages burned
car tyres, wrecked streetlamps and ruined traffic booms – that bore witness to extreme student
frustration were removed within hours. The ‘Hush, little baby, don't say a word’ tune plays softly.
2 INT. A TELEVISION SCREEN – 28 February 2008 – NIGHT
A CNN news report shows footage of white students at the University of the Free State tricked black
residence workers into eating stew containing urine.
The scene changes to an interview with Helen Zille, the leader of South Africa’s main opposition
party, the Democratic Alliance, who asked the South African Human Rights Commission to conduct
an investigation into racial tensions at the university.
HELLEN ZILLE (the Premier of the Western Cape, leader of South Africa's opposition Democratic
Alliance political party, and a former Mayor of Cape Town:
The abhorrent footage of students abusing university workers is a fundamental infringement on the
victims' constitutional right to have their dignity respected and protected. This incident is symptomatic
of racial tensions that have been simmering at the campus for some time over the issue of residence
integration.
The scene changes to an interview with Frederick Fourie, the Rector of the University of the Free
State in front of the UFS Main Building.
FREDERICK FOURIE (former Rector of the UFS):
The fact that it is openly linked to the integration process in UFS residences is also most disturbing.
The university is going through a difficult time with its efforts to racially integrate its residences and to
create a new residence culture based on diversity, respect, human dignity and human rights.
3 CNN WEBSITE – 27 May 2008 - 1247 GMT
JOHANNESBURG, South Africa (CNN) -- A South African university said Tuesday it would close the
dormitory where white students tricked black workers into tasting stew laced with urine, an incident
that sparked protests when revealed earlier this year.
‘The decision by the executive board of the University of the Free State to close the Reitz Residence
was unanimous. The Reitz video reopened racial wounds, and is deeply regretted. It was an isolated
504

Emmie Smit and Rita Niemann
manifestation of resistance to the impact of ongoing transformation initiatives at the university,’ the
school's acting administrator, Teuns Verschoor, said in a written statement.
‘The video and other acts of public violence and vandalism on the campus have undermined the
efforts of the university to foster diversity in student and staff life and create an inclusive institutional
culture on the campus,’ the statement said.
‘The university will transform itself over time into a beacon of hope, combating racism and other forms
of discrimination in South Africa and elsewhere in the world,’ Verschoor said.
3 UFS PRESS RELEASE – 08 SEPTEMBER 2008 21:24:47
Bloemfontein – Prof. Frederik C.v.N. Fourie stepping down.
It is with sadness that I hereby announce my intention to step down as Rector and Vice-chancellor of
the University of the Free State (UFS) in the 4th quarter of this year . . . this flows primarily from the
exhausting times that I have experienced during the past nine years, first as Vice-Rector (since 1999)
and then as Rector (since 2003), in managing and implementing several complex strategic projects.
The challenges and complexities of continuous change management at a higher education institution,
and specifically the demands of further dynamic development and transformation at the UFS, demand
enormous amounts of emotional energy and drive . . . especially, the political divisions and tensions in
the UFS Council and the broader university community during the past year have been extremely
draining.
I think it is time for new and fresh leadership, especially in the light of the transformation challenges of
the UFS. I have thus decided to step down in the interest of transformation and the further dynamic
development of the UFS.
4 WIKIPEDIA WEBPAGE HTTP://EN.WIKIPEDIA.ORG/WIKI/UNIVERSITY_OF_THE_FREE_STATE
The then new Vice-Chancellor, Jonathan Jansen, was appointed and he has subsequently initiated a
process for campus-wide racial integration among students, which included inviting the four students
to continue with their studies at the university.
In the midst of extreme racial and/or cultural tensions, the University of the Free State featured daily in
international news reports and footage. International television crews and squads of police officers in
action gear became a regular sight on this almost rural campus. The integration process in UFS
residences failed and the executive board of the University of the Free State closed the Reitz
residence. The reactions of the students on the creation of ‘an inclusive institutional culture on the
campus,’ were condemned by the management. The Rector, Frederik C.v.N. Fourie stepped down
and a new Vice-Chancellor, Jonathan Jansen, was appointed.
In 2010 the UFS was awarded the World Universities Forum Award for Best Practice in Higher
Education, which praised amongst other aspects the racial integration and harmonisation of the
student community. On receiving her honorary doctorate from the university, Oprah Winfrey called the
transformation of the university as ‘nothing short of a miracle’ when referring to the incident and
subsequent racial integration.
During the graduation ceremony, Oprah offered these words of advice: ‘Focus on what you can give,
how you can serve, then you will be the best that you can be. And you will live the glory of the dream
that that-which-I-know-to-be-God holds for all of us.’ Oprah ended her address with the words, ‘God
bless South Africa.’
505

Emmie Smit and Rita Niemann
Figure 3: The leadership for change programme – one of the dreams of Jonathan Jansen – is aimed
at exposing 150 first-year students annually to diverse cultures and to enable them to learn
leadership skills. The goal of the programme is to build a new class of UFS students who
become leaders during their years of study and commit to building a non-racial community
during and beyond their years at university. This group travelled to the Vrije University of
Amsterdam
6. Conclusion
This paper discussed different levels and types of dreaming: the interior or sub-conscious and the
exterior or conscious, and therefore dreams have a variation of multi-dimensional impacts; and that
although dreams/visions are a basic concept in leadership and strategic planning, not all individuals
and/or institutions are expecting, and therefore not experiencing, the extreme creative and farreaching
directive value of dreams equally.
Most often dreams or visions are allowed to exist no further than in a vision-statement, and these
visions are allowed only to enter the vision-statement once they fit into measureable and/or
comprehendible strategies. Very rarely, leaders dare to give visions the freedom to lead towards
unexplored possibilities.
In the traditional concept, the dream or vision is influenced and limited by components of the strategy
formation process and forms therefore, in reality, the first basis of the strategic planning process. In
the new concept the dream forms the basis of the planning process.
Figure 4: Concept 1 and 2
This research explored the scientific value of subconscious dreaming and found that medical research
proves the benefits thereof to conscious state, providing novel solutions to challenges;
contextualising, guiding and organising emotions; connecting concepts laterally; explaining challenges
and opportunities more clearly; developing the nervous system and enhancing memory, intelligence,
and the ability to be strategic. This paper thereby encourages leaders to pursue free-range dreaming;
and then only after full completion of the dreaming process, should goals and measurability be
welcomed into the long-term planning process. This research does not include short- and medium-
506

Emmie Smit and Rita Niemann
term planning, because this is eventually developed from within the long-term planning. Additional
further research is necessary before the Concept 2-type of free range dreaming can be appropriated
in higher education with AI as a research method and Appreciated Leadership.
References
Barrett, D. (2001). The committee of sleep: How artists, scientists, and athletes use their dreams for creative
problem solving—and how you can too. New York: Crown Books/Random House.
Breton, A. (1971). First surrealist manifesto (Le Manifeste du surréalisme, 1924). In P. Waldberg, Surrealism (pp.
66-75). New York: McGraw-Hill.
Byrd, A. M. (2011). Deriving dreams from the divine: Hittite tesha-/zashai. Los Angeles: University of California.
Carroll, L. (1969). Alice in Wonderland. Chicago: Childrens Press. (Original work published 1864).
Cooperrider, D. L., & Whiteney, D. A. (2008). Appreciative inquiry handbook. San Francisco: Berrett-Koehler.
Cooperrider, D. W. (2003). Appreciative inquiry handbook. Bedford Heights, OH: Lakeshore Publishers.
Deschamps, J. P. (2008). Innovation leaders: How senior executives stimulate, steer and sustain innovation. San
Francisco: Jossey-Bass.
Drucker, P. (2008). The essential Drucker– The best of sixty years of Peter Drucker's essential writings on
management. Key West: Harper Business.
George, B. (2009). 7 Lessons for leaders in crisis. Retrieved from
http://www.josseybass.com/WileyCDA/Section/id-131153.html: Jossey-Bass.
Hartmann, E. (1995). Making connections in a safe place: Is dreaming psychotherapy? Dreaming, 5, 213-228.
Jung, C. (1945). On the vision of dreams. Seattle: Scriptor Press.
Kaufman, R. (2010). Dreams make you smarter, more creative, studies suggest. National Geographic
News,Retrieved from http://news.nationalgeographic.com/news/2010/08/100813-sleep-dreams-smarterhealth-science-naps-napping-rem/.
Lehrer, J. (2009). How we decide. New York: Houghton Miffin Harcourt.
Mail & Guardian (2011, December 02). Jansen an inspiration to all, including Oprah. Mail & Guardian Online,
Retrieved from www.mg.co.za.
News, B. (2003). What does your sleep position reveal about your personality? Retrieved from
http://news.bbc.co.uk.
Nickols, F. (2011, July 24). Strategy: Definitions and meaning. Distance Consulting LLC, Retrieved from
http://www.nickols.us/strategy_definition.htm. http://www.nickols.us/strategy_definition.htm.
Niemann, R., & Kotze, T. (2006). The relationship between leadership practices and oraganisational culture: an
education management perspective. South African Journal of Education, 26 (4), 609-624.
Richter, M. (n.d.). How to create a leadership vision driven by intrinsic motivation. In Thiagi Inc., Practical guides
(pp. 341-351). Bloomington: Thiagi Inc.
Rock, M. E., & Cox, M. (1997). The 7 pillars of visionary leadership: Aligning your organization for enduring
success. Toronto: Dryden.
Schwah, C., & Spady, W. (2010). Leading in the age of empowerment. Plymouth: Rowman & Littlefield
Education.
The American Heritage Dictionary of the English Language (2000). ( 4th ed.). Dream. Retrieved from
http://www.thefreedictionary.com/dream.
Whitney, D. Appreciative leadership: Focus on what works to drive winning performance.
Whitney, D., & Ludema, J. (2006). Appreciative Leadership development program, Participant workbook. Chapel
Hill: Corporation for Positive Change.
Whitney, D., & Trosten-Bloom, A. (2003). The power of Appreciative Inquiry: A practical guide to positive change.
San Francisco: Berret-Koehler Publishers.
507

Developing Methodology for Subcontractors’ Security of
Payment Under Malaysian Construction Industry
Azizan Supardi, Hamimah Adnan, Zul Zakiyuddin Ahmad Rashid, and Norazian
Mohamad Yusuwan
Universiti Teknologi MARA, Shah Alam, Malaysia
zzan659@gmail.com
mimad856@gmail.com
zakiphd@gmail.com
azianyusuwan@gmail.com
Abstract: In Malaysia, if subcontractors are to benefit from payment provisions in the proposed Construction
Industry Payment and Adjudication Act (CIPA Act), the relevant small-sized sub-contractors need to enhance
knowledge on the ‘Security of Payment’ regime. This on-going research attempts to introduce balance and
appropriate guidelines to the sub-contractors in claiming for payment and in ensuring that the main contractors
make prompt payments. The paper discusses the methodology used in developing this attempted framework.
Firstly, analysis of the content and judgements of Malaysian law cases which are published in the Malayan Law
Journal to date is conducted to identify the real problems and legal issues disputed by the sub-contractor relating
to payments. A questionnaire survey was sent to various sub-contractors throughout Malaysia to determine the
sub-contractors’ level of knowledge on the proposed CIPA Act which was followed by semi-structured interviews
with selected sub-contractors to analyse the various avenues to improve the payment process in the construction
industry. The findings will be analysed in developing the framework which is further justified by the intended use
of a qualitative focus group analysis. This is to validate the proposed framework to be used by the subcontractors
to prepare their claims so they may fully benefit from the security of payment legislation. Most of the
small-sized sub-contractors have limited knowledge on the proposed ‘Security of Payment Legislation’ or the
CIPA Act and for as long as the CIPA Act remains a proposal, the sub-contractors have to bear with the current
and problematic payment mechanisms in the standard forms of construction contracts.
Keywords: Malaysia, security of payment, construction industry, subcontractor, methodology
1. Introduction
In any country’s progress development, an essential function is acted by the construction industry
which is both growth-initiating and growth-dependent (Abdullah 2004). Building and infrastructure
works are established by the industry which contribute to the large economic increase required for
social economic development. Disposable incomes which generate extra construction activities will be
raised through economic development (Sin 2006).
In most engineering and construction contracts, it is the contractor’s responsibility to deliver the
works, e.g. construction, installation, material supply, etc. This represents his piece of the bargain. In
reciprocation, the employer should maintain part of this bargain by furnishing the contractor’s
requirements, which in most cases, is in the shape of money (Singh 2002; Singh 2006).
While payment is believed to be the life-blood of the construction industry, payment defaults,
particularly delayed and non-payment, is acknowledged as a perennial problem in the industry
(Ameer Ali 2006). May and Siddiqi (2006) state that well-timed run of money is required for the
achievement of a construction project, commencing from the owner to the contractor downwards to
the subcontractors, sub-subcontractors, suppliers, and vendors. They stress, however, in the
subcontract, contractors frequently try to transfer the risk of the owner’s non-payment to
subcontractors. This is done by including contingent payment provisions, for instance pay-when-paid
or pay-if-paid clauses.
There has been little research in the Malaysian context, on the security of payment regime. Instead,
past studies focused on the likely introduction of a Malaysian Construction Industry Payment and
Adjudication Act (Ameer Ali 2006) and attention to the Malaysian Government on the choices for its
requirements (Constable 2006). Given the lack of empirical studies, this study has the potential to
offer better theoretical explanations and thus, a better understanding of the usefulness of the security
of payment regime (Che Munaaim 2009).
Abidin (2007) recognized payment as the nature of construction disputes and that misleading
payment methods according to conditions of the standard contract and non-payment of certified sums
508

Azizan Supardi, Hamimah Adnan et al.
are connected to such disputes. Che Munaaim (2006) concluded that in the Malaysian construction
industry, non-payment or delayed payments have been experienced by numerous local contractors
and when it comes to government clients, the state of affairs is highly ineffective. In fact, huge
amounts of money are involved in these transactions, a factor which may affect the image and
performance of the construction industry.
Git (2007) points to various challenges that oppose enforcement of the adjudicator’s verdicts offered
to the distressed parties. Mohd Nazir (2006) specified the client-related key factors of delayed
payments while Rosli (2007) revealed that because of the non-standard form, the domestic
subcontractor faced problems with payment phrases, termination clauses, variation and also
arbitration. However, the most common problem concerns payment phrases. Sin (2006) shows the
main concern in the construction industry to be that of payment while Yin (2007) suggests that
according to the main conditions, the employer is not the repudiator of the contract in cases of nonpayment.
Uher and Brand (2008) conclude that the Act has been positive from the subcontractors’ viewpoint as
those making lesser payment claims, have been winning at adjudications under the Act.
Nevertheless, subcontractors have been revealed to have little understanding and operational
knowledge of the adjudication procedure. Thus increasing these small-sized subcontractors’
awareness of the Act is essential.
The unexpected winding up of key contractors in a number of developed countries, over the years,
has served as a wake-up call for industry players because of the consequent cash flow problems
arising from non-payment. Those facing non-payment issues desire quick resolutions to decrease the
financial implications to all concerned (CIDB 2007). Some research questions that arise are:
What are the provisions made for the sub-contractor in relation to payment?
What issues arise in the legal cases on payments to sub-contractor?
What is the sub-contractors’ perspective on the benefits of security of payment in claiming for
prompt payment?
What are the avenues taken to improve payment problems that are suited to the Malaysian
Construction Industry?
Is there any need for improvement in the security of payment for sub-contractors?
How are improvements on security of payment justified?
The proposed research attempts to introduce a framework on ways for the sub-contractor to
successfully claim for payment under security of payment legislation. Firstly, the research identifies
the legal aspects of current payment provisions in the sub-contractors’ forms of agreement, and
determines the legal issues relating to payment for sub-contractors. By determining the subcontractors’
perspective on security of payment in the proposed Construction Industry Payment and
Adjudication Act (CIPA Act), and by determining the ways available to improve the security of
payment for the sub-contractor, the aim of producing a framework on Security of Payment for subcontract
works will be subsequently achieved.
2. Background to the study
2.1 Principal methods of payment to sub-contractors
There are three principal methods of payment to the subcontractor for work executed. These include
‘payment upon certification, ‘direct payment’, and ‘contingent payment’. Even though the main
contractor disburses the required payments to the sub-contractor in the first and third methods, the
second method is only an employer and sub-contractor deal. Where payment of the amount due is
concerned, the main contractor is a sheer third party or ‘bystander’ in the second method (Singh
2006).
2.1.1 Payment upon certification
Under this payment regime, the receipt of the interim payment certificate by the main contractor
signifies the sub-contractor's right to payment, and following that, the distinct 'window-period' for
payment. Whether the main contractor has received the supposed amount from the employer or the
509

Azizan Supardi, Hamimah Adnan et al.
period to honour the sub-contractor is longer than the grace period specified to him is irrelevant. The
clock starts ticking against him once he receives the pertinent certificate, in view to his obligation to
pay (Singh 2006).
2.1.2 Direct payment from the employer
Under this payment regime, payments are paid directly to the sub-contractor concerned by the
employer, even though the payments payable to the sub-contractor are incorporated in the Interim
and/or Final Certificates to the main contractor. This is due to the fact that such payments are unpaid,
as in the conventional way, through the main contractor. However, the main contractor is disbursed
only the pertinent profit for the supposed sub-contractor (Singh 2006).
2.1.3 Contingent payment or conditional payment
‘Contingent payment’ is the third ordinary method available for paying sub-contractors, which
encompasses numerous labels such as ‘pay if paid’ clauses, ‘pay when paid’ clauses, and ‘back-toback’
clauses” (Singh 2006).
May and Siddiqi (2006) state that risks may not be efficiently transferred to the degree projected by
the contractor via generally drafted contingent-payment clauses. Most courts differ in their willingness
to put into effect contingent payment provisions, and such enforcements depend on the exact
phrasing of the clauses.
Therefore, the amounts afforded to the contingent payment clauses under dissimilar jurisdictions do
not demonstrate uniformity. Furthermore, in some instances, the circumstances are so unclear that no
common main beliefs can be identified. In view of the vagueness in the procedures, it is maybe
helpful to scrutinise the related law cases and/or authoritative pronouncements to shed some light on
this subject and give direction to practitioners (Singh 2006).
2.2 Proposed Malaysian construction industry payment and adjudication act (CIPA
Act)
In 2003 and 2004, chaired by the Malaysian Minister of Works, captains of the Malaysian construction
industry jointly were gathered by the Construction Industry Development Board Malaysia (CIDB).
Consequently, a Malaysian ‘Latham Report equivalent’ identifying 10 areas of main concerns was
delivered and working groups produced. Technology, human resources, health and safety,
industrialised building systems, productivity and quality, and payment were amongst the areas of
main concerns recognized (Ameer Ali 2007). Following a meeting on 24th June, 2003, the initial
outline of the consolidated master plan was concluded and named the ‘Malaysian CIMP Framework
2005 – 2015’ (CIDB 2007).
Consequently, the manuscript was further improved by CIDB into a 10-year Master Plan, spanning
2006 to 2015 (CIDB 2007). Under strategic thrust 2 of 7 (Strengthen the construction industry’s
image), and by working jointly with the industry, the Construction Industry Payment and Adjudication
Act was built (CIDB, 2007).
The contents of the Construction Industry Payment and Adjudication Act (CIPA Act) (Ameer Ali 2006)
are as below:
A method for regular payment where there is no condition for payment means in a construction
contract
Prohibition of ‘pay-when-paid’ and ‘pay-if-paid’ clauses in construction contracts
The rights for suspension of works by a party who has not been paid
The provision of quick dispute resolution methods called adjudication for disputes relating to a
construction contract
The provision of remedies for the recuperation and security of payment under a construction
contract
Since its inception in 2006, the proposed Construction Industry Payment and Adjudication Act (CIPA
Act) is still under development and the draft can be seen on the CIDB website. The Act is expected to
be enacted before the Master Plan ceases in 2015.
510

Azizan Supardi, Hamimah Adnan et al.
2.3 Avenues to improve payment problems
In the construction industry, there are a variety of avenues that are offered to improve payment
problems. A few of these options have been included in the construction contracts in other developed
countries. The best solutions which are suited to serve the Malaysian construction industry should be
implemented (Sin 2006). In summary, these avenues include:
2.3.1 Payment of stipulated interest
In referring to Clause 42.9(b) of the CIDB 2000 Form of Contract, the contractor’s claim for interest is
one probable remedy for breach of contract by the employer in not paying or not paying on time. This
affords relief to the contractor. By doing thus, the employer will not definitely commit a breach of
contract, however, will efficiently allows the employer to suspend payment tract (Oon 2000; Sin 2006).
2.3.2 Suspension of work or going slow
Clause 30.7 of the Agreement and Conditions of PAM Contract 2006 (With Quantities) and Clause
42.10 of the CIDB Standard Form of Contract for Building Works 2000 Edition provide for suspension
of work. In general, there are no rights for suspension of work in common law (Fong 2005; Ameer Ali
2006) for non-payment.
2.3.3 Eradication of “pay when paid”
The standard forms of construction contract do not provide for such a remedy other than the CIDB
Standard Form of Contract for Building Works 2000 Edition under Option Module C Clause C3(c). If
the sub contract is subjected to a “”pay when paid” form, suspension is ineffective, except in incidents
when the contractor has absconded with money paid by the employer which is somewhat common
(Fong, 2005).
2.3.4 Adjudication
Adjudication is provided in the Agreement and Conditions of PAM Contract 2006 (With Quantities)
under Clause 34.0. In the United Kingdom, the adjudication procedure does not make the claimant a
secured creditor following judgment. A summary judgment must be applied to the court by the
successful claimant and a decision is made after that (Fong, 2005).
2.3.5 Liens
No construction contract elsewhere provides clauses on liens except for the United States of America
and Canada which addresses it by way of mechanic lien statutes. It is also nonexistent in Malaysia.
“Any attempt to give security for payment to a contractor, subcontractor or supplier through a lien
(Fong 2005) or charging order method may not be in the best interest of the public or other parties –
specifically the purchasers” (Ameer Ali 2006).
2.3.6 Trust
The trust concept is not alien in Malaysia in respect of retention of monies. It is provided in Clause
30(6) (a) of the Agreement and Conditions of PAM Contract 2006 (With Quantities) and Clause
42.3(c) (i) of the CIDB Standard Form of Contract for Building Works 2000 Edition. Nevertheless, the
trust is conditional and permits allowable deductions to be set off by the contractor or the employer
(Fong 2005).
2.3.7 Payment bonds
Clause 42.1(e) of the CIDB Standard Form of Contract for Building Works 2000 Edition provide for
payment bonds. Supardi, Yaakob and Adnan (2007; 2009; 2011) state that after discussing
interpretations of applications of injunction relief in performance bonds, as in payment bonds, it is
noted that careful choice of words should be adopted by the constructor of a performance bond to
achieve clear understanding of its conditionality and make it undisputable.
Therefore, Supardi, Adnan, and Yaakob (2009) further stated that to ensure undisputed
interpretations of the words in performance bonds, as in payment bonds, the performance bond itself
should be either purely conditional or unconditional 'on-demand' bond. The best examples for this are
511

Azizan Supardi, Hamimah Adnan et al.
in the cases of Esso Petroleum Malaysia Inc v Kago Petroleum Sdn Bhd [1995] 1 MLJ 149 and IE
Contractors Ltd v Lloyd's bank plc and Rafidain Bank [1990] 2 Lloyd's Rep 296 (Supardi 2007).
2.3.8 Direct payment from principal
Direct Payment is provided in the P.W.D. Form 203A (Rev. 2007) Standard Form of Contract to be
used where Bills of Quantities form part of the contract under clause 60.1.
2.3.9 Contractor’s project account
The prospect of creating a contractor’s project account has been a ‘creative’ suggestion by REHDA.
However, this has yet to be explored in detail (Ameer Ali 2006).
3. Research methodology
To ensure reliability, the research methodology employed for this research consists of the following
strategies:
3.1 Phase 1: Literature review
Researchers typically look at the literature, prior to scheduling the particulars of a study, to discover
what has been printed concerning a subject they are attracted to investigate. Attention is given to both
the view of experts in the field and other research studies (Mahmud 2009).
The literature review of this study is conducted in two parts. The first is to identify the current legal
aspect of payment provisions in the sub-contractors’ forms of agreement. This is done by exploring
the current and past research on the subject-matter through books, articles, internet, standard forms
of contract, acts, etc.
Secondly, the literature review also attempts to identify the disputed legal issues in payments to subcontractors
by analyzing judgments made in law cases. The method used is legal research through
qualitative case studies. This will help identify the variables that can be used in the research survey.
3.2 Phase 2: Evaluation
Based on the literature review, the assessment of the various avenues used to overcome payment
problems in the construction industry conducted in this phase addresses two questions. Firstly, it will
determine the sub-contractors’ perspectives on security of payment in the proposed Construction
Industry Payment and Adjudication Act (CIPA Act) via a questionnaire survey that will be sent out to
various sub-contractors. The survey is piloted first for validity and reliability.
Legal or law studies frequently refer to the study of legal rules, principles, concepts, theories,
doctrines, decided cases, legal institutions, legal problems, issues or questions or a combination of
some or all of them (Yaqin 2007).
Table 1: Research methodology
Research
Methodology
Content
Analysis
Law Cases
Analysis
Questionnaires Interviews
Focus Group
Validation
Objective 1
Objective 2
Objective 3
Objective 4
Objective 5
Quantitative investigation involves choosing subjects, data collection techniques, procedures for
gathering data, and data analysis techniques (Singh, Fook and Sidhu 2006).
Triangulation is conducted through qualitative in-depth semi-structured interviews which involve the
senior manager of the sub-contractor’s firm. They are sought to clarify and elaborate on their
responses in the questionnaire survey as well as to determine ways available to improve the security
of payment for the sub-contractor.
512

Azizan Supardi, Hamimah Adnan et al.
The qualitative investigation conducted examines the quality of relationships, activities, situations, or
materials (Singh, Fook and Sidhu 2006).
3.3 Phase 3: Production
This remains the main aim of this research, which is to produce a framework on Security of Payment
for sub-contract works. The introduction of a mock-analysis might be suited for discussing acceptance
of the guidelines.
Focus Group validation will be held to complete triangulation (questionnaire survey and in-depth semistructured
interviews) on acceptance of the frameworks produced.
An essential part of undergoing a post-graduate study course is writing a thesis, a paper or report
resulting from an original study submitted by a candidate for a degree. As the candidate starts the
study course, thesis writing begins at the earliest (Abdul Talib and Ariffin 2007).
4. Findings and discussion
Analysis from the judgment made in law cases is done to identify the real problems and legal issues
disputed by the sub-contractor relating to payment. In a search based on the word ‘Payment’, 6,717
cases from judgments dating as far back as the Malayan Law Journal have been identified for further
analysis. First screening of the downloaded cases was conducted using the words ‘Sub-contract’ or
‘Subcontract’ and returned 273 cases. From the second reading and screening of the 273 cases, the
court held 186 cases involving the construction industry. A further third screening was done of the 186
cases to identify cases in which the judges discussed the relationship between payment and the subcontract
works. Subsequently, 126 cases were identified. However, only 23 case that indicated issues
of payment to the sub-contractors as their main judgment, will be further analyzed. (Supardi, Adnan
and Mohammad 2011b)
The above cases highlight the issues and judgments on matters related to the payment mechanism of
sub-contract works. The issues are listed below (Supardi, Adnan and Mohammad 2011b):
Agreement on the terms of payment
Counter claims
Delays
Direct payments
Final payments and certification
Injunctions
Insolvency or winding up
‘Pay-when-paid' clauses
Performance bonds
Progressive payments and certification
From 186 cases dealing with payments to sub-contractors, only 12 selected cases were analysed that
thoroughly addressed the issue of ‘payment upon certification’ as the main issue. Currently, based on
the analysis of law cases, there cannot be any set-off or cross-claims when an amount is due under
an interim certificate of payment as an interim certificate is not conclusive and binding on the
employer as an expression of satisfaction with the quality of work and materials supplied. Only the
final certificate is conclusive and expresses the architect’s satisfaction with the works carried out, with
the payment certificate issued by the architect informing the main contractor to pay the sub-contractor
upon receipt. (Supardi and Adnan 2011b)
From 186 cases dealing with payments to sub-contractors, it was found that only 5 of the cases
analysed thoroughly addressed the issue of “direct payment” in their judgments. Based on the
analysis of law cases, it is currently not legitimate to use anything which the parties said or did after
the construction of a contract and that the meaning of the terms needs to be clearly understood for
e.g. a winding up petition is not a legitimate means of seeking to enforce payment of a debt. When
there is a self declaratory statement in a formal document, it strongly implies that the representation
513

Azizan Supardi, Hamimah Adnan et al.
was made; and the onus is on the plaintiffs to show that the defendants had been guilty of dishonest
fraud. (Supardi and Adnan 2011a)
From 186 cases dealing with payments to subcontractors, only 3 cases that thoroughly addressed the
issue of contingent payment were analysed. Based on the analysis, currently, the parties have to bear
with whatever is stated in their sub-contract according to the form of payment, be it in real money, in
particular form or historically done (Supardi and Adnan, 2011c).
Response to statements on the following are sought: problems with regard to existing payments and
dispute resolution mechanisms, the adequacy of payment provisions in the proposed Construction
Industry Payment and Adjudication Act (CIPA Act), the adequacy of adjudication provisions in the
proposed Construction Industry Payment and Adjudication Act (CIPA Act) and the use of certain
provisions on payment remedies for sub-contractors. The 5-level Likert scale was used and the
questions include consideration of the respondent’s lack of knowledge on the proposed Construction
Industry Payment and Adjudication Act (CIPA Act), i.e. whether they are unsure of or do not
understand the Act. (Supardi, Adnan and Mohammad 2011a)
A questionnaire survey was sent to 1,500 sub-contractors throughout Malaysia. 100 respondents
replied which constitutes 6.67% and the researcher is still trying to get more replies. It was found that
the adequacy of the security of payment framework for particular small-sized sub-contractors needs to
be produced. The accepted security of payment framework, though, needs to be easily
comprehensible to be effective in delivering knowledge. Additionally, analysis of alternative avenues
to resolve payment problems in the construction industry which have been incorporated in the
construction statutes of other developed countries as well as the proposed Construction Industry
Payment and Adjudication Act (CIPA Act), needs to be conducted. (Supardi, Adnan and Mohammad
2011a)
Qualitative, in-depth semi-structured interview surveys for the first 26 out of the identified 78 subcontractors
throughout the country were preliminary analyzed. Under the CIPA Act, most of the
subcontractors prefer the use of ‘direct payment’ as the main remedy to their payment problems
whereas in the statutes of other countries, subcontractors prefer ‘payment of stipulated interest’.
5. Conclusion
In the current standard forms of construction contracts, payment structures to the sub-contractors are
divided into three: payment upon certification, direct payment and contingent payment. For as long as
the Malaysian CIPA Act remains a proposal, the sub-contractors have to bear with the current,
problematic structure of payment mechanisms (Supardi, Adnan and Mohammad 2010).
References
Abdul Talib, S. and Ariffin, J. (2007) Thesis Writing and Preparation for Viva-Voce, University Publication
Centre (UPENA) Universiti Teknologi MARA, Shah Alam.
Abdullah, F. (2004) Construction Industry and Economic Development: The Malaysian Scene, Universiti
Teknologi Malaysia, Skudai.
Abidin, A. (2007) “The Profile of Construction Disputes”, Unpublished Masters Dissertations, Universiti Teknologi
Malaysia.
Ameer Ali, N. A. N. (2006) “A "Construction Industry Payment and Adjudicationn Act": Reducing Payment-Default
and Increasing Dispute Resolution Efficiency in Construction”, Masters Builders, 3 rd Quarter, pp 4-16.
Ameer Ali, N. A. N. (2007) “One step at a time”, Construction Journal, Vol 6, pp 18-21.
Che Munaaim, M. E. (2006) “Payment Woes Among Malaysian Contractors”, Unpublished Masters Dissertations,
Uniiversiti Teknologi Malaysia.
Che Munaaim, M. E. (2009) “The security of payment regime in Malaysia: will it be effective?” Paper read at
Construction and Building Research Conference of the Royal Institution of Chartered Surveyors, University
of Cape Town, pp 1626-1639.
CIDB (2007) Construction Industry Master Plan Malaysia 2006 - 2015, Construction Industry Development
Board, Kuala Lumpur.
Constable, A. (2006) “Construction Industry Payment and Adjudication Legislation: The Choice That Lies Ahead
for Malaysia”, International Construction Law Review, pp 78-88.
Fong, L. C. (2005) “The Malaysian Construction Industry - The Present Dilemmas of Unpaid Contractors”, Master
Builders, 4 th Quarter, pp 80-82.
Git, T. S. (2007) “Challenges to the Adjudicator's Decision”, Unpublished Masters Dissertations, Universiti
Teknologi Malaysia.
514

Azizan Supardi, Hamimah Adnan et al.
Mahmud, Z. (2009) Handbook of Research Methodology: A Simplified Version, University Publication Centre
(UPENA) Universiti Teknologi MARA, Shah Alam.
May, A. L. and Siddiqi, K. (2006) “Contingent-Payment Provision Puzzle - Safeguarding against an Unintended
Outcome”, Journal of Architectural Engineering, December, pp 158-162.
Mohd Nazir, N. Z. (2006) “Late Payment Problems Among Contractors in Malaysia”, Unpublished Masters
Dissertations, Universiti Teknologi Malaysia.
Oon, C. K. (2000) “Standard Construction Contracts in Malaysia - Issues and Challenges”. Paper read at Half
Day Seminar on Contract Formation, Documentation, Issues and Challenges, Kuala Lumpur.
Rosli, N. M. (2007). “The Use of Standard Form of Domestic Subcontract in Malaysian Construction Industry”,
Unpublished Masters Dissertations, Universiti Teknologi Malaysia.
Sin, T. A. S. (2006) “Payment Issues - The Present Dilemmas of Malaysian Construction Industry”, Unpublished
Masters Dissertations, Universiti Teknologi Malaysia.
Singh, H. (2002) Engineering and Construction Contracts Management: Law and Principles, LexisNexis,
Kuala Lumpur.
Singh, H. (2006) “Construction of Contingent Payment Clauses: Is There Light at the End of the Tunnel?”, The
Malayan Law Journal Article, Vol 3, p 9
Singh, P., Fook, C. Y. and Sidhu, G. K. (2006) A Comprehensive Guide to Writing a Research Proposal,
Venton Professional, Batu Caves.
Supardi, A. (2007) “Performance Bond: Conditional or Unconditional”, Unpublished Masters Dissertations,
Universiti Teknologi Malaysia.
Supardi, A. and Adnan, H. (2011a) “Security of Payment in Malaysian Construction Industry: Issues on Sub-
Contract’s Direct Payment”, Paper read at 3rd International Conference on Mechanical and Electrical
Technology, Dalian, China, August, pp 407-412.
Supardi, A. and Adnan, H. (2011b) “Security of Payment in Malaysian Construction Industry: Sub-Contract
‘Payment upon Certification’ Cases”, Paper read at 2nd International Conference on Construction
and Project Management, Singapore, September, pp 220-224.
Supardi, A. and Adnan, H. (2011c) “Security of Payment in Malaysian Construction Industry: Eradication of
Sub-Contract’s Contingent Payment”, Paper read at International Conference on Energy,
Environment, Entrepreneurship, Innovation, Lanzarote, Canary Islands, Spain, May, pp 116-121
Supardi, A., Adnan, H, and Mohammad, M. F. (2010) “Sub-Contractors’ Readiness on the Malaysian Security of
Payment Legislation in Construction Industry”, Paper read at 2010 International Conference on Construction
and Project Management, Chengdu, China, pp 248-252.
Supardi, A., Adnan, H. & Yaakob, J. (2009) “Legal Analysis on Malaysian Construction Contract: Conditional
versus Unconditional Performance Bond”, Journal of Politics and Law, Vol 2, No. 3, pp 25-34.
Supardi, A., Adnan, H. and Mohammad, M. F. (2011a) “The Adequacy of Malaysian Security of Payment
Legislation for Sub-Contractors in Construction Industry”, Paper read at Sixth International Conference
on Construction in the 21st Century, Kuala Lumpur, Malaysia, July, pp 677-684.
Supardi, A., Adnan, H. and Mohammad, M. F. (2011b) “Security of Payment Regime in Construction Industry:
Are Malaysian Sub-Contractors Ready?”, The Built & Human Environment Review, Vol 4, No. 1, pp 122-
137.
Supardi, A., Adnan, H. and Yaakob, J. (2011) “Legal Comparison between Conditional and Unconditional on
Performance Bond in Malaysian Construction Contract”, International Surveying Research Journal, Vol
1, No. 1, pp 45-55.
Supardi, A., Adnan, H., and Yaakob, J. (2007) “The Legal Conditionality of Performance Bond in Malaysian
Construction Contract”, Built Environment Journal, Vol 4, No. 2, pp. 11-22.
Supardi, A., Yaakob, J. and Adnan, H. (2009). “Performance Bond: Conditional or Unconditional”, Paper read at
the 2nd Construction Industry Research Achievement International Conference, Legend Hotel, Kuala
Lumpur.
Uher, T. E. and Brand, M. C. (2008) “Impact of the 'Security of Payment' Act in New South Wales on clients,
contractors and subcontractors”. Paper read at CIB International Conference on Building Education and
Research, Heritance Kandalama, Sri Lanka, pp 1274-1286.
Yaqin, A. (2007) Legal Research and Writing, LexisNexis, Kelana Jaya.
Yin, L. S. (2007) “A Repudiatory Breach in the Construction Industry – Non-Payment”, Unpublished Masters
Dissertations, Universiti Teknologi Malaysia.
515

Researching Implementation of Quality Assurance Policy
in Malaysian Private Higher Education
Siew Fun Tang 1 and Sufean Hussin 2
1 Taylor’s Business School, Taylor’s University, Malaysia
2 Department of Educational Management, Planning and Policy, Faculty of
Education, University of Malaya, Malaysia
siewfun.tang@taylors.edu.my
drsufean@um.edu.my
Abstract: This paper presents the methodology undertaken by a Ph.D. researcher to investigate the
implementation of quality assurance (QA) policy in Malaysian private higher education. The methodology was
intended to bring out accurate data and provide useful sources for the induction of emerging insights relating to
the problem identified, which is how the QA system was institutionalised and implemented in two selected
Malaysian private higher education institutions. Multiple forms of data were collected using interviews,
observation, and questionnaires. Qualitative interviews using semi-structured questions were conducted with
government officials, representatives from the association of private higher education, administrators, lecturers,
and students of private higher education institutions. In addition, documents and activities relating to QA policy
implementation were examined. Quantitatively, responses on structured questionnaires were obtained from the
students on their perceptions of quality in private higher education, and QA policy implementation in Malaysian
private higher education institutions was analysed. This paper is written in a systematic manner, whereby the
rationale for engaging a multiple-case study will be presented first, followed by an overview of the research
design where it describes the overall planning of the study and indicates how the present research was
operationalised, that is, how a general set of research objectives could be translated into a practical and
researchable topic. Details on selection of participants, choice of data gathering methods, and measures to
ensure validity and reliability will ensue. Four procedures were adhered to in this study to enhance the validity of
findings. They were triangulation, member checks, peer examination, and audit trail. This paper focusses on the
description of how the research was effectively executed using multiple-case studies rather than the research
outcome. Finally, this paper discusses the issues of originality and generalisability of the research, anticipated
biases, and assumptions made in this study.
Keywords: multiple-case study; qualitative and quantitative data; quality assurance; policy implementation;
Malaysian private higher education
1. Background to research and significance of study
According to Hill and Hupe (2002), an implementation perspective cannot be precluded in the
changing state-society interface towards more cooperative and negotiated networks. They found that,
given these new systemic features, translation of policies into practice becomes more challenging and
is a legitimate concern. Policy implementation is of much interest to the research world. The large
number of reports on implementation issues produced by governments and various international
agencies each year attests to their endemic nature (Saetren, 2005). With the rapid expansion of
private higher education and emerging of related quality issues, research into the implementation
process will provide quality policy developers better understanding of the concerns and practices of
private higher education institutions and their stakeholders in designing future policies that hopefully
contribute further to the nation’s development of higher education. In addition, developments towards
universal governing structures (e.g., the European Union and UNESCO), international agreements,
and resolutions on quality assurance (QA) policy issue developments give rise to daunting
implementation challenges that advance more research in the area of quality policy implementation.
While most literature is biased towards failure cases, this research focusses on exploring the process
of policy implementation for attainment of positive ultimate outcomes rather than policy performance,
which is generally referred to as policy impact. While the study of impact searches for the
consequences of a policy decision, the study of implementation examines those factors that contribute
to the realisation or non-realisation of policy objectives (Van Meter and Van Horn, 1975). The classical
model of policy implementation process proposed by Van Meter and Van Horn (1975) identifies the
factors influencing the implementation process and the model provides a suitable theoretical
framework for the inquiry into this study.
The QA in Malaysian private higher education has evolved for more than a decade. Before 1996,
there was no formal accreditation system governing privately owned institutions. In 1996, the passage
of several education-related pieces of legislation, in particular the Lembaga Akreditasi Negara (LAN,
516

Siew Fun Tang and Sufean Hussin
the national accreditation council) Act of 1996, significantly revamps the regulations governing private
institutions of higher education, including the change in the registration and approval system. LAN
was authorised by the government to accredit programmes offered by the private institutions of higher
education, at the certificate, diploma, and degree levels. In 2007, a single QA system for both
Malaysian public and private higher education was established following the introduction of the
Malaysian Quality Framework (MQF), and the qualifying responsibility was taken over by the Malaysia
Qualifications Agency (MQA). The implementation of MQF is ongoing, and this study is timely
because an ongoing assessment of an existing implementation is always beneficial as feedback can
be given for further improvement.
Research involving the private sector is always difficult due to the nature of business. In general,
private higher education is subjected to a trade-off between quality and quantity as the providers are
profit-making institutions. Geiger (1986) argues that the constraints of limited finance and resources
force private higher education institutions to become primarily teaching institutions. However, as
competition among private higher education institutions in recruiting students, especially international
students, becomes more intense, there has been a change in the way private higher education
institutions are managed. Due to their credibility in producing quality graduates, they are now
regarded as quality education providers. Hence, some of these private universities become reputable
and applauded for accountability and quality teaching and learning. They are now more open to share
their “success stories,” and this study intends to explore this challenging market segment of higher
education. Furthermore, the amount of research done in the area of QA in private higher education is
not much, thus the findings from this study will bring significance to the body of literature. Two recent
studies were from Zainudin and Mohd Azuhari (2005), who focussed on assessments, and Sirat et al.
(2006), who studied enhancing quality of faculty.
The purpose of this study was to ask private higher education stakeholders on their views about QA
implementation issues and to investigate their institutionalised operations and procedures in ensuring
higher education quality. This study was designed to answer the following research questions:
What are the perceptions of higher education stakeholders on the quality of higher education?
How does the government regulate QA in private higher education institutions?
What are the motivations for implementing QA policy in the two private case universities?
How do the two case universities implement QA policy?
What are the challenges faced by the two case universities in implementing QA policy?
What are the impacts of QA process on the development of the two case universities?
This paper delves into the research methodology used that was intended to bring out accurate data
and provide useful sources for the induction of emerging insights in answering these research
questions rather than answering the questions themselves. The outcomes of this study are presented
in other papers.
2. Multiple-case study research
A case study research strategy is employed in situations where an in-depth understanding of a
phenomenon is needed (Creswell, 2007; Merriam, 2001; Patton, 1990). The aim is usually to solicit or
reveal meaning from the participants as the main players immersed in the phenomenon regarding the
process they have undergone or are currently undergoing (Merriam, 2001; Newman and Benz, 1998).
Since the focus of this study is to explore and discover the process of institutionalisation and
implementation of QA, deep understanding and meaning from the perspective of the participants have
to be solicited. This has rendered the use of case studies as a suitable research methodology. In
earlier studies on case study methodology, Stake (1981) professes that the knowledge learned from
case studies is different from other research knowledge in four important ways. First, a case study
gives more concrete research knowledge, which resonates with a researcher’s own experience
because it is more vivid, concrete, and sensory than abstract. Second, it is more in context because
the researcher’s experience is rooted in context, as is knowledge in case studies, and this knowledge
is distinguishable from the abstract and formal knowledge from other research designs. Third, the
knowledge is developed further by reader interpretation as readers bring to a case study their own
experience and understanding and add new data to the old data, thus leading to generalisation, which
was considered by Stake as “part of the knowledge produced by case studies” (1981: 36). Last, in
“generalising” as described above, readers have some population in mind. Hence, the knowledge
517

Siew Fun Tang and Sufean Hussin
obtained from a case study is based more on reference populations determined by the reader. Thus,
unlike traditional research, the reader participates in extending generalisation to reference populations
(Stake, 1981: 35-36). Outcome of a qualitative exploratory case study cannot be generalised to the
larger population (Merriam, 2001; Stake, 1995; Yin, 1989). However, this study bounded a system of
private higher education institution staff and students, whereby insights relevant to the masses of
university staff and students may be discovered, further studies can be generated, and policy can be
affected. In contemplating all these issues, it was decided that although there are constraints to the
generalisation of findings, a qualitative case study was the best research design to adopt for the
research questions proposed in this study. Following from Stake, Merriam (2001) also agreed that
although the methods employed in a study are important, the uniqueness of a case study lies in the
questions asked and their relationship to the end product.
The present study has both the “what” and “how” questions, which are frequently asked for case study
and exploratory study. Since there has been no in-depth research on the implementation of QA in
private higher education institutions, especially on the institutionalisation of the system in supporting
national educational goals, the researcher has no inkling of what to expect. Thus, the present study
has been designed to be a multiple-case study that is exploratory in nature and involves two
institutions of higher education. It is also the intention of the researcher to develop an in-depth
understanding of the implementation of QA policy in private higher education by collecting multiple
forms of data. According to Creswell (2008), providing this in-depth understanding requires only a few
cases to be studied. The works of Creswell and Plano Clark, 2007, Greene et al., 1989, Morgan,
1998, and Tashakkori and Teddlie, 1998, reveal various approaches to using both qualitative and
quantitative data that have advanced in the literature. The basic rationale for the use of both
qualitative and quantitative data is for data triangulation. One data form gives force to counterbalance
the weakness of the other form. In the case of this study, quantitative scores on an instrument from
many participants will offset the weaknesses of qualitative response from a few people. Conversely,
the in-depth findings from the interviews of a few people offer strength to quantitative data that could
not provide adequate detailed information about the context in which participants provide information.
Both data sets are analysed separately, and an interpretation will be made to reveal whether the
results support or contradict each other. In this way, the direct comparison of the two data sets
provides a “triangulation” of data sources (Creswell, 2008).
In this study, qualitative interviews using semi-structured questions were conducted with government
officials, representatives from the association of private higher education, administrators, lecturers,
and students of private higher education institutions, and documents and activities relating to QA
system implementation were examined. Quantitatively, responses on structured questionnaires were
obtained from the students on their perception of QA and its implementation in Malaysian private
higher education institutions. Triangulated data could show convergence, inconsistency, and
complementary results. It is not simply collecting two sets of quantitative and qualitative data
respectively, but it involves merging, linking, integrating, or embedding the two different sets of data
(Creswell, 2008).
In justifying the use of both qualitative and quantitative approaches, Creswell defends that “the
quantitative provides the opportunity to gather data from a large number of people and generalized
results, whereas qualitative permits an in-depth exploration of a few individuals” (2008: 562).
According to Creswell (2003), most researchers recognise that limitations exist in all methods, and
biases inherent in any single method could be neutralised. These considerations point to an
exploratory qualitative multiple-case study, using the qualitative and quantitative approach, being the
most suitable methodology for this study.
3. Addressing the research questions
3.1 Participants of the study
When inquiring qualitatively, “the intent is not to generalize to a population, but to develop an in-depth
exploration of a central phenomenon. Thus to best understand this phenomenon, the qualitative
researcher purposefully or intentionally selects individuals and sites” (Creswell, 2008: 213). Merriam
(2001) recommends that several “cases” be selected based on relevant criteria in multi-case or
comparative case studies. She suggests one criterion using as much variation as possible so that a
maximum variation sampling strategy can be employed. The aim of this research is to develop an indepth
understanding of the implementation of QA policy in private higher education by collecting
518

Siew Fun Tang and Sufean Hussin
multiple forms of data. For this study, participants and institutions that could best understand the
central phenomenon of implementing QA policy and provide rich information to the study were
selected. Since private universities are the larger institutions (higher number of students per
institution) compared to the many small private colleges, two private local universities (research sites)
were chosen, thus making it a multiple-case study with purposeful sampling applied to both individuals
and sites. The distinctive characteristics between the two selected universities include the age of
establishment, diversity of courses, and course delivery of these private higher education institutions.
One university was recently upgraded to university status and still offers twinning or franchised
programmes from foreign universities, while the other offers mainly home-grown programmes. In the
former case, the foreign university partners may have great influence on the adoption of quality
culture in the institution.
Participants in this study were not selected based on their representativeness of colleagues in their
area of expertise but on their worthiness of study. Platt (1999) points out that unanticipated findings
are more likely to be uncovered for case studies that are chosen in virtue of their own right rather than
as an instance, to him, that the real strength of the study lies in this decision. In that sense, the
chosen participants must be worthy of study. At each institution, participants consisted of staff
members from various organisational levels, students, and alumni who were interviewed with regard
to current QA practices. They were directly involved in the implementation of quality systems and their
views and experience contributed to the data collected for this study. In addition to the above
participants, one participant each from the Ministry of Higher Education, in particular from MQA and
Malaysian Associations of Private Higher Education (MAPCU), and two participants from the
employers were included in this study to provide insights leading to answers for some of the research
questions in this study.
3.2 Data collection
Merriam (2001: 69) sees data as “nothing more than ordinary bits and pieces of information found in
the environment.” The interest and perspective of the investigator determines whether the pieces of
information become research data. In other words, the techniques for collecting data and what is
considered data are determined by “the researcher’s theoretical orientation, by the problem and the
purpose of the study, and by the sample selected” (Merriam, 2001: 70). According to her, qualitative
data are data conveyed through words, whereas data presented in number form are quantitative.
Wolcott (1992) describes collecting data as asking, watching, and reviewing. Data are not readily
available for us to collect. “Collecting data always involves selecting data, and the techniques of data
collection…will affect what finally constitutes ‘data’ for the purpose of research” (Dey, 1993: 15). In
this study, data were collected in multiple forms using interviews, observation, and questionnaires.
The details of each are described in the following section for the purpose of keeping an audit trail of
the study to enhance validity of the findings.
Interview
A qualitative interview is an intentional interaction in which the researchers ask one or more
participants using general open-ended questions and record the information obtained from them
(Creswell, 2008). The data are then transcribed by the researcher for analysis. Gay et al. (2006)
distinguish interviews by their degree of structure and formality. They classify structured interviews as
interviews with a set of pre-set questions and unstructured interviews as interviews with questions
prompted by the flow of the interview. Semi-structured interviews are a combination of both structured
and unstructured interviews.
Semi-structured interviews were one of the main data collection methods employed in this study. They
were used to collect data from a range of people presenting a broad range of ideas. The target
interviewees were the stakeholders of private higher education, namely the institution’s staff members
from various organisational levels, students, alumni, employers, MQA officers, and MAPCU
representatives. Interviews with participants were carried out on a one-to-one basis to explore what
was in the participant’s mind about the discussion topic and to elicit information from them. As Patton
(1990: 196) explains, “We interview people to find out from them those things we cannot directly
observe…We cannot observe feelings, thoughts, and intentions, We cannot observe behaviours that
took place at some previous point in time. We cannot observe situations that preclude the presence of
an observer. We cannot observe how people have organized the world and the meanings they attach
519

Siew Fun Tang and Sufean Hussin
to what goes on in the world. We have to ask people questions about those things. The purpose of
interviewing, then, is to allow us to enter into the other person’s perspective.”
Questionnaire
Gay et al. (2006: 420) defines a questionnaire as “a written collection of self-report questions to be
answered by a selected group of participants.” Event and feelings concerning an event are best
captured as they happen. Hence, rich information is gathered when the individual participant writes
down his or her responses, including views and feelings on the form provided. Using questionnaires,
large amounts of data are collected in a relatively short time. Since most respondents to
questionnaires would not disclose their names, it would be difficult to have a follow-up interview with
the particular respondent. However, the responses from the questionnaires could act as a guide for
the researcher to clarify interpretations or tentative research results. This had definitely helped in
escalating the face-to-face interview, which is usually time-consuming. While Gay et al. (2006)
recommend the use of structured items with a variety of possible responses, Creswell (2008)
suggests a combination of closed- and open-ended questions. He argues that the advantage of this
type of questioning is that the “predetermined closed-ended responses can net useful information to
support theories and concepts in the literature” (2008: 228). The open-ended questions can add rich
information while exploring the responses to the closed-ended questions.
Two sets of questionnaires were developed for this study. The developments of these two sets of
questionnaires were preceded by an extensive review of the literature, guided by the research
questions, and also expert advice provided by scholars in the areas of education management,
planning, and policy. The first set consisted of a mixture of closed-ended and open-ended questions
focussing on the implementation of a QA system, in particular the institutionalised system adopted by
private higher education institutions. The intended respondents were administrators of the institutions
and were directly involved in the implementation process. The second set of questionnaires, intended
for students, also had a mixture of closed-ended and open-ended questions to investigate the general
perception of quality in education and its related issues. The responses from these two sets of
questionnaires complemented the interview results, thus providing a more complete picture of QA
implementation at the research sites.
Observation
Observation provides another primary source of qualitative data. Creswell (2008: 221) defines
observation as the “process of gathering open-ended, firsthand information by observing people and
places at a research site.” That means the observation happens in the natural field setting. Very often,
when collecting data in the field, observation intertwines with informal interviews and conversations
(Merriam, 2001). One advantage of observation is that, as an outsider, the observer (in this study, the
observer was the researcher herself) was able to see things that may lead to further understanding of
the context but may have become routine and mundane to the participants themselves. It is true that
no one can observe everything, and Patton (1990) suggests a few elements to note during
observation. They are the physical setting; the participants; the activities and interactions; the
conversation; subtle factors (less obvious but important) such as informal activities and nonverbal
communication; and the observer’s behaviour. According to Neuman (2006), field notes are important
and may include maps, diagrams, photographs, video tapes, memos, objects from the field, brief
notes taken in the field, and detailed notes written away from the field.
In this study, the researcher took on the role of observer as participant. Her observer’s activities were
known to private higher education institutions, but they had limited contact and her main role was to
gather information, record her findings, and jot down thoughts she was having about what was going
on. She was intrigued about details that reveal “what’s going on here” through careful listening and
watching. Through the use of reflexive writing her comments on all these formed an important part of
the field notes because something of significance might be revealed.
4. Validation and reliability of findings
4.1 Validation of findings
Throughout the data collection and analysis processes, the trustworthiness of the findings and
interpretations must be maintained. Creswell defines validity of findings as “that the researcher
520

Siew Fun Tang and Sufean Hussin
determines the accuracy or credibility of the findings through strategies such as member checking or
triangulation” (2008: 266). Maxwell (2005) finds that there isn’t any “objective truth,” or in other words,
there is nothing “out there” that can act as a standard for an account to be compared with. Thus, the
interpretation of the researcher on the finding is subjective. However, by engaging in self-reflexivity,
the researcher constantly investigates herself while investigating others (Berg and Smith, 1998).
The researcher identified two specific validity threats in this study. First, she did not deny the
subjectivity of the researcher in selecting data and was fully aware of the tendency for the researcher
to pick the data that fit her existing theoretical framework or data that were prevalent. Second, the
researcher was also aware of the possibility of influencing the participants during data collection,
especially during interviews, because the researcher herself was part of the understudy setting. In
qualitative research, eliminating these threats is impossible, but it was more important for the
researcher to understand how she was influencing what the participant said, and how this could affect
the validity of the inferences she drew from the interviews. Four procedures were adhered to in this
study to enhance the validity of findings. They were triangulation, member checks, peer examination,
and audit trail.
4.2 Reliability of findings
Reliability is linked to validity. According to Newman and Benz (1998), a valid research design or
research instrument is also reliable, though it may not be true vice versa. In general, reliability is
regarded as a measure of the extent to which research findings can be replicated (Merriam, 2001;
Newman and Benz, 1998). Merriam (2001) rightly points out that a replication of a qualitative study
would not yield similar results because it considers the human behaviour factor. She argues that if
defining reliability from this traditional sense, reliability is regarded as a misfit in qualitative research.
Merriam then summarises that if we refer to reliability as describing consistency and dependability,
the question now becomes “whether the results are consistent with the data collected” (Merriam,
2001: 206). In this study, this definition is used, therefore, if the study is repeated, a different set of
findings is expected, but if the results are consistent with the new data set and are dependable, then it
is reliable.
In this multiple-case study, the findings of the research depend on the cases being studied and none
of the results being discredited. In fact, the findings build up the body of knowledge in the area of QA
implementation in higher education, particularly in the private sector. To enhance reliability of this
study in terms of consistency and dependability, the methods recommended by Merriam were
followed. She suggests looking at the investigator’s position (explaining the assumptions and theory
behind the study), triangulation, and audit trail, all of which are covered in the process of ensuring
validity of the study.
5. Conclusion and discussion
This paper has presented a systematic inquiry into a worthwhile problem. It was intended to bring out
accurate data, providing useful sources for the induction of emerging insights relating to the problem
identified, which was how selected private universities institutionalised the QA system and
implemented it. Through the methodology discussed, results were developed to help answer
questions and gain a deeper understanding of the implementation of QA policy in private higher
education institutions. Outcome of a qualitative exploratory case study cannot be generalised to the
larger population (Merriam, 2001; Stake, 1995; Yin, 1989). Stake (1981) professes that the knowledge
learned from case studies gives more concrete research knowledge that resonates with a
researcher’s own experience, and the knowledge is developed further by reader interpretation as
readers bring to a case study their own experience and understanding and add new data to old data.
In this study, the researcher assumes a self-reflexive role, which Mishler (1990) considers theoretical.
The researcher sees validity embracing subjectivity and is “trustworthy enough to be relied upon for
their own work” at the same time. Readers have some population in mind, and the knowledge
obtained from a case study is based more on reference populations determined by the reader. Thus,
unlike traditional research, the reader participates in extending “generalisation” to reference
populations (Stake, 1981: 35-36).
In this study, in order to enhance external validity, rich and thick descriptions are provided by the
researcher in her final thesis so that readers are able to determine how closely their situations match
this research situation and hence, whether the findings can be transferred.
521

References
Siew Fun Tang and Sufean Hussin
Berg, D. & Smith, K. (eds.) (1998) The Self in Social Inquiry: Researching Methods. Newbury Park, CA, Sage.
Creswell, J. W. (2003) Qualitative, Quantitative, and Mixed Methods Approaches, 2 nd edition. Thousand Oaks,
CA, Sage.
Creswell, J. W. (2007) Qualitative Inquiry and Research Design: Choosing among Five Traditions, 2 nd edition.
Thousand Oaks, CA, Sage.
Creswell, J. W. (2008) Educational Research – Planning, Conducting, and Evaluating Quantitative and
Qualitative Research, 3 rd edition. Upper Saddle Creek, NJ, Pearson Education Inc.
Creswell, J. W. & Plano Clark, V. L. (2007) Designing and Conducting Mixed Methods Research. Thousand
Oaks, CA, Sage.
Dey, I. (1993) Qualitative Data Analysis. London, Routledge.
Gay, L. R., Mills, G. E. & Airasian, P. (2006) Educational Research: Competencies for Analysis and Applications,
8 th edition. Upper Saddle Creek, NJ, Pearson Prentice Hall.
Geiger, R. L. (1986) Public and Private Sectors in Higher Education: Structure, Function, and Change in Eight
Countries. Ann Arbor, University of Michigan Press.
Greene, J. C., Caracelli, V. J. & Graham, W. F. (1989) Toward a conceptual framework for mixed-method
evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255-274.
Hill, M. & Hupe, P. (2002) Implementing Public Policy: Governance in Theory and in Practice. London, Sage.
Maxwell, J. A. (2005) Qualitative Research Design: An Interactive Approach, 2 nd edition. Thousand Oaks, CA,
Sage.
Merriam, S. B. (2001) Qualitative Research and Case Study Applications in Education, 2 nd edition. San
Francisco, Jossey-Bass.
Mishler, E. (1990) Validation in inquiry-guided research: The role of exemplars in narrative studies. Harvard
Educational Review, 60(4), 415-442.
Morgan, D. L. (1998) Practical strategies for combining qualitative and quantitative methods: Applications to
health research. Qualitative Health Research, 3, 362-376.
Neuman, W. L. (2006) Social Research Methods – Qualitative and Quantitative Approaches. 6 th edition, Boston,
Pearson Education, Inc.
Newman, I. & Benz, C. R. (1998) Qualitative-Quantitative Research Methodology – Exploring the Interactive
Continuum. Carbondale, IL, Southern Illinois University Press.
Patton, M. Q. (1990) Qualitative Evaluation Methods, 2 nd edition. Thousand Oaks, CA, Sage.
Platt, J. (1999) What can case studies do? In: Byman, A. & Burgess, R. G. (eds.) Qualitative research. London,
Sage, pp. 160-179.
Saetren, H. (2005) Facts and myths about research on public policy implementation: Out-of-fashion, allegedly
dead, but still very much alive and relevant. Policy Studies Journal, 33(4), 559-582.
Sirat, M. (series ed.), Jantan, M., Jantan, C., Chiang, C. H., Shahnon, S. & Sibly, S. (eds.) (2006) Enhancing
Quality of Faculty in Private Higher Education Institutions in Malaysia. Penang, National Higher Education
Research Institute (IPPTN).
Stake, R. E. (1981) Case study methodology: An epistemological advocacy. In: Welsh, W. W. (ed.) Case study
methodology in educational evaluation – conference proceedings. 1981 Minnesota Evaluation Conference,
Minneapolis, Minnesota Research and Evaluation Center.
Stake, R. E. (1995) The Art of Case Study Research. London, Sage.
Tashakkori, A. & Teddlie, C. (1998) Mixed Methodology: Combining Qualitative and Quantitative Approaches.
Thousand Oaks, CA, Sage.
Van Meter, D. S. & Van Horn, C. E. (1975) Policy implementation process: A conceptual framework.
Administration & Society, 6(4), 445-488.
Yin, R. K. (1989) Case Study Research: Design and Methods, revised edition. Thousand Oaks, CA, Sage.
Zainudin, H. A. & Mohd Azuhari, C. M. (2005) Quality assurance in higher education: Serqual model as the
alternative measuring instrument for service quality performance in Malaysian higher education. Conference
Proceedings. National Conference on Tertiary Education (NCTE), Pusat Penerbitan Universiti (UPENA),
Universiti Teknologi MARA.
522

Methodology for Investigating Rationale for Unsuccessful
EOT Claim in Malaysian Construction Industry
Norazian Mohamad Yusuwan, Hamimah Adnan, Azizan Supardi and Zul
Zakiyuddin Ahmad Rashid
Universiti Teknologi MARA, Shah Alam, Malaysia
azianyusuwan@gmail.com
mimad856@gmail.com
zzan659@gmail.com
zakiphd@gmail.com
Abstract: This paper presents a proposed research methodology to be adopted in on-going PhD research which
aims to investigate the rationale behind the unsuccessful Extension of Time (EOT) claims in the Malaysian
construction industry. At the end of the research, a framework for quality EOT claims will be produced. To
achieve the aims and objectives of this research, both quantitative and qualitative methods will be applied. A
quantitative research method in the form of a questionnaire survey will be conducted to investigate reasons for
unsuccessful EOT claims from both the contractor’s and architect’s point of view. The next stage of data
collection will involve in-depth qualitative investigation using face-to-face interviews meant to get in-depth
information from the respondents about their experience and regarding the rationale behind unsuccessful EOT
claims. To complete the triangulation approach, expert group validation will be conducted to validate the findings
from the research (through quantitative and qualitative methods) and also the expert group’s acceptance of the
framework produced. This paper specifically explores and describes the research methodology proposed in
undertaking the research. Review of prior research, books and publications will be conducted in determining the
most effective research methodology in executing this research.
Keywords: extension of time (EOT), construction industry, projects, construction delay, research method
1. Introduction
Every construction contract will clearly stipulate the date or period for the contractor to complete their
works. Specifying the date of completion facilitates claims for damages by the Employer for any
delays created by the contractor in performing their works (Gibson 2008). If the contractor fails to
complete the project within the stipulated period or within reasonable time, and the delays are proven
to be caused by the contractor, the employer will be entitled for liquidated and ascertained damages
(LAD). The LAD stipulates the employer’s entitlement to the damages for contractor’s breach of
obligation to complete construction by the agreed date (Gibson 2008). However, if the delays are
beyond the control of the contractor, an extended completion period will be granted to the contractor
to provide ample time to complete the works.
The definition for extension of time by The Society of Construction Law (SCL) as contained in the
Protocol for Determining Extension of Time and Compensation for Delay and Disruption (SCL 2002),
is “the additional time granted to the contractor to provide an extended contractual time period or date
by which work is to be, or should be completed and to relieve it from liability for damages for delay
(usually liquidated damages)”. An extension of time will be granted to the contractor if he/she has
successfully proven that they are not responsible for the delays. The contractor must submit a
completed application of Extension of Time (EOT) claim to the employer which requires all the
relevant facts and documents related to the delays including a thorough analysis of the delayed
events. In preparing the application for EOT, the claimant (contractor) must, in the first place,
determine the contract provision under which he/she is entitled to make such claims. A vital
prerequisite in establishing entitlement for EOT is for the claimant to produce documentation to
absolve them of responsibility for the delays and that other parties are accountable (it can be either
the employer or the neutral events i.e. exceptionally inclement weather etc.).
2. Literature review: Extension of time
Preparing and assessing of EOT claims is not as easy as it sounds. Alkass, et al. (1995) suggest that
the preparation of EOT claims is tedious as huge efforts in preparing these claims involve the
meticulous digging through piles of project documentation to sort and ascertain pertinent delays
encountered during the project. Hence, the process is usually difficult and expensive due to lack of
available documentation and often a service of claim consultant is employed to assist the preparation
of EOT claims.
523

Norazian Mohamad Yusuwan et al.
Among the sources of information that are useful in establishing delayed events and their impacts as
identified by Braimah and Ndekugri (2006) include contract documents, project schedules, progress
reports, letters, minutes of meetings, material receipts, supervision and inspection reports, resource
usage and costs, daily reports and extra work order. Good record management is essential to ensure
that the project data are well-kept and updated throughout the project cycle. As emphasised by
Kartam (1999) keeping all the project data and information in an accurate and well organized manner
throughout the life cycle of the project is a key requirement in preparing, analyzing and resolving
claims. Yet, it is common practice that the management and record of project data and information is
undertaken on adhoc basis making it difficult to dig up all the project files to locate the relevant facts
and data for a particular claim.
In order to ease the delay assessment, the claimant (contractor) should provide adequate related
information of the delayed event. Among the problems in substantiating the delayed claim is the
detailed investigation of the project involving vast numbers of documents and persons to be reviewed
and interviewed which can be a time-consuming and resource-hungry process (Carmichael and
Murray 2006). Alkass, et al.(1995) in his study identified that 70 per cent of the efforts in a claim is
spent on searching and organizing information. Thus, it is essential for the construction professional
to understand the documents and supporting data and information needed to support claims.
Carmichael and Murray (2006) stressed that an effective delay analysis must consider all delaying
events and is dependent on the ‘facts’ of the case which refers to project’s factual records, i.e.
variations, correspondence, minutes of meetings, progress reports. Additionally, if this process is not
managed effectively, claims can be at best, time consuming to assess in terms of validity and at
worst, deceivingly inaccurate. Gibson (2003), as cited in Carmichael and Murray (2006), states:
“Too often do we see extension of time and delay claims submissions containing several
lever arch files of these facts with no specific linkage to the alleged events that caused
the delay. There may also be a bundle of computer printouts indicating the claimed
effects but the causal link is not clearly defined. The referee is expected to find it and
often it’s like looking for a needle in a haystack!”
According to Kumaraswamy and Yogeswaran (2003) the claimant and the assessor often spend
considerable time on substantiating and assessing the EOT claim. Complete documentation of EOT
claims should be prepared to facilitate the consultant in assessing them and to avoid delays in EOT
assessment that would lead to conflict amongst the construction team members. Kumaraswamy and
Yogeswaran (2003), identified that the key factors in delayed assessment of delay claims are
associated with lack of details and clarity in substantiation, and delays in submissions of details by the
claimant (contractor). As stated by Yogeswaran, Kumaraswamy, and Miller (1997), “...most
contractual regimes...do not provide details of the principles governing the assessment of claims for
extension of time and this is left to the professionals involved in each project”. This leads to
unsuccessful EOT claims by the contractor, hampered by lack of guidance in substantiating such
claims. Kumaraswamy and Yogeswaran (2003) emphasise the need to spell out in “Particular or
General Specifications” the type of detailed particulars that are required by the assessor in assessing
EOT claims. This will simultaneously ease the contractor’s task in preparing the EOT application and
speed up the process of claim assessment. Complete submission and proper documentation will also
help to prevent conflict and disputes amongst the construction team members. Therefore, there is a
need to produce a framework for a quality EOT claim that can be used as a reference for both the
contractor and the consultant in preparing and assessing claims respectively.
3. Problem statement
According to Gibson (2008), EOT submissions are common sources of construction disputes. Among
the related disputes arising from EOT claims are eligibility of a delayed event for claiming EOT; choice
of method for evaluating EOT; and quantification of the EOT (Danuri et al. 2006). On the other hand,
Palaneswaran and Kumaraswamy (2008) revealed that, disputes on time extensions often involve the
eligibility of time extension claims i.e. the permissible delayed events for justifying a project time
extension; the choice of methods for evaluating the delay; and the permissible period of time
extension. Hence, this research aims to investigate the major disputed issues in relation to EOT
application specifically in the Malaysian construction industry.
Most standard forms of contract provide clauses for extension of time (EOT) due to excusable
delayed events in construction projects. In Malaysia, the Pertubuhan Arkitek Malaysia (PAM)
524

Norazian Mohamad Yusuwan et al.
Standard forms of contract 2006, Public Works Department (PWD) Standard forms of contract 203 A
(2010) and CIDB 2000 forms are the most common types of standard forms of contract used in the
construction industry. PAM forms are the most popular standards forms of contract for the private
sector while PWD forms are widely used in government projects. The latest version of the PWD form
is PWD 203 (2010) while the revised version for PAM is PAM 2006. CIDB 2000 is the only existing
edition of the CIDB form. All standard forms of contract provide clauses for EOT due to excusable
delays. Provision for EOT in PAM 2006 is explained in Clause 23 while in PWD 203 A (2010) it is
covered under Clause 43 and for CIDB 2000, delays and extension of time are explained under
Clause 24.
All local Standard forms of contract list out the excusable delayed events that entitle the contactor for
an extension of time. However, unsuccessful EOT claims frequently occur in the Malaysian
construction industry. This research will study the rationale for unsuccessful EOT claims, at the end of
which, a recommended framework will be produced to help both the claimant (contractor) and the
assessor in substantiating and assessing the EOT claim.
4. Proposed research methodology
To ensure the achievement of the aims and objectives of this research, a triangulation research
approach will be implemented, employing the quantitative, qualitative and expert validation methods.
At the first stage of the research, a thorough review of related previous research is conducted to
identify the major disputed issues in EOT. A law case analysis will be conducted to support the
literature review. The analysis of law cases will be limited to construction industry cases in Malaysia
and Commonwealth countries.
The next stage of data collection involves quantitative methods in the form of a questionnaire survey.
The selection of the architect as the respondent is based on the limitation of this study to focus on
construction projects using PAM 2006 forms of contact which require the architect to act as the
assessor of the EOT claims. The next stage of data collection will involve an in-depth qualitative
investigation utilising face-to-face interviews to get in-depth information from the respondents about
their experience in the construction industry. To complete the triangulation research approach, an
expert group validation will be held to validate the findings from the research and also to obtain the
expert group’s endorsement of the framework produced.
5. Conclusion
Review of literature suggests the triangulation research approach is suited for carrying out this ongoing
PhD research. According to Sarantakos (1997), the triangulation approach will enable
researchers to obtain a variety of information on the same issue and use the strengths of each
method to overcome the deficiencies of the other. At the same time, it will help the researcher to
achieve a higher degree of validity and reliability and to overcome the deficiencies of single method
studies (Sarantakos 1997). Similarly, as contended by Fellows and Liu (1997) (as cited in Rashid
2011), employing more than one research method, such as combining qualitative and quantitative
approaches will lessen or eradicate the disadvantage of each individual method and will reap the
advantages of each. The data from this research is reliable since it has been obtained from robust
methods.
References
Alkass, S., Mazerolle, M., Tribaldos, E. and Harris, F. (1995) “Computer Aided Construction Delay Analysis and
Claims Preparation”, Construction Management and Economics, Vol 13, pp 335-352.
Braimah, N. and Ndekugri, I. (2006) “The Link Between Planning and Programming Practice an the Analysis of
Delay and Disruption Claims”, AACE International Transaction.
Carmichael, S. and Murray, M. (2006) “Record keeping for contemporaneous delay analysis: a model for
effective event management”, Construction Management and Economics, Vol 24, pp 1007-1018.
Danuri, M. S. M., Othman, M., Abdul-Rahman, H. and Lim, C. C. (2006) “Application and Assessment of
Extension of Time Claim: Findings of Case Studies Conducted in Malaysia”, Journal of Design and the
Built Environment, Vol 2, pp 15-29.
Gibson, R.( 2008) Construction Delays Extension of Time and Prolongation Claims, Taylor & Francis.
Kartam, S. (1999) “Generic Methodology for Analysing Delay Claims”, Journal of Construction Engineering
and Management, Vol 125, pp 409-419.
Kumaraswamy, M. M. and Yogeswaran, K. (2003) “Substantiation and Assessment of Claims for Extensions of
Time”, International Journal of Project Management, Vol 21, pp 27-38.
525

Norazian Mohamad Yusuwan et al.
Palaneswaran, E. and Kumaraswamy, M. M. (2008) “An integrated decision support system for dealing with time
extensions entittlements”, Automation in Construction, Vol 17, pp 425-438.
Rashid, Z. Z. A. (2011) Malaysian Legal Provisions and Design Risks in Construction Works, Shah Alam,
Universiti Teknologi MARA.
Sarantakos, S. (1997) Social Research, Palgrave.
SCL (2002) Protocol for determining extensions of time and compensation for delay and disruption,
Society of Construction Law (SCL).
Yogeswaran, K., Kumaraswamy, M. M. and Miller, D. R. A. (1997) “Claims for Extension of Time in Civil
Engineering Projects”, Construction Management and Economics, Vol 16, pp 283-293.
526

The Study of Strategy-Making Process in UK Universities:
An Interpretative Approach
Lilia Zerguit
Sheffield Business School, Sheffield Hallam University, Sheffield, UK
l.zerguit@shu.ac.uk
Abstract: This paper examines the proposed methodology to be adopted by the author, to study the strategymaking
process within the context of a sample of UK universities. The research aims to build an understanding of
the process universities follow in developing their strategies. It can be argued that strategy takes different forms
within an organization (Mintzberg 1998); but is this the case in UK universities? How can researchers untangle
the complexity of strategy-making in different organizational contexts? This paper does not explore the outcomes
of this research; neither does it examine the appropriateness of the methodology. Its purpose is to explain the
need to take into consideration the researcher’s own philosophical position, assumptions and methodological
commitments while designing a research strategy. As a result, the discussion tackles the complex issues of
maintaining a consistent ontological and epistemological stance throughout the design of a PhD research
methodological approach. Researchers in the field of strategic management have mainly adopted a positivist
approach to the study of strategy. Their view of “reality exists objectively”, therefore illustrates “a relatively
accurate representation of the underlying phenomenon” being explored. However, as the strategy- making
process is not always formerly recorded or deliberately realised, the positivist approach would disregard the
social and cultural dimensions of strategy development. Adopting an interpretivist approach therefore seems to
be more appropriate for this type of study. The phenomenological base of this approach relates to the idea that
the “person” and the “world” are related through their lived experiences of that world. In order for the researcher
to better understand the strategy-making process, it is therefore more appropriate to adopt an approach that uses
the voice of participants to constitute a picture of their lived experience of reality. First, an ontological reality is
constructed - inter-subjectively through the “meanings” and “understandings” developed socially and
experientially. This underpins the view that the relevant reality (the strategy-making process) exists within the
perceptions of the person experiencing it (social actors). Secondly, the epistemological underpinnings of the
approach assume that researchers cannot separate themselves from what they know and how they perceive
things. The author addresses this dilemma by becoming engaged in the interpretivist reality. In this case, the preexisting
knowledge of the author in the subject area and the personal involvement in academia has potential to
establish rapport with the social “actors” and penetrate their cognitive processes by extrapolating and analysing
their trains of thought. This paper provides other researchers with insights on how to conceptualise and design a
qualitative study firmly grounded within their philosophical and methodological positions.
Keywords: strategy-making process, interpretivist approach, ontology, epistemology, abduction, retroduction
1. Setting the scene: the research background
The aim of the paper is to explain the philosophical assumptions that lie behind the methodology
proposed to carry out a PhD investigation into the strategy-making processes of a sample of UK
universities. The main motivation is to extend the understanding of the strategy-making process to a
higher education context. Whilst the setting differs from a commercial business in many ways, it might
be possible to learn lessons from the higher education context on the way strategy is crafted. It might
be that the processes do differ; in which case, by discovering why this is so, it may help to understand
and explain the differences.
The starting point for the identification of the approach was to explore the different papers published
to date and document a picture of how researchers approached their study of strategic management.
The rational behind this, is the identification of the best approach that would make a contribution to
knowledge in the field of strategy-making process. It is very difficult to predict the contribution of the
research before understanding the approach to be adopted to carry out the investigation. According to
Adcroft and Willis (2008) research contribution to knowledge is classified into four categories:
“developing theory; stretching theory; reflecting or reviewing theory; and other”. The “other”
classification, relates to studies conducted to provide a profile of companies or biographies without
any clear contribution to knowledge. Adcroft and Willis (2008) expand further their argument and
relate each contribution to a research paradigm adopted by researchers in the field of strategic
management. Strategy research taking a positivist approach accounts for 55% of all studies, with a
focus on discovering facts and generating or testing fundamental laws. The non-positivist research,
on the other hand, has a contextual rather than theoretical contribution to the field. This is consistent
with the aim of the author; to extend the application of theory to different contexts.
527

Lilia Zerguit
Through an exploration of strategy-making process literature, it is clear that the main focus of
researchers was prescribing norms and models of the strategy-making process. Central to their
explanation is the development of theory from practice, which consists of rules or recommendations.
Arguably, researchers paid more attention to their research contribution and getting published rather
than on the research methodology that might yield more of an accurate picture of the strategy-making
process. Several authors in the field of strategy (Bracker 1980; Huff and Reger 1987; McKiernan
1996) have reflected on the evidence derived from past research conducted in the field of strategic
management. The purpose of the research was classified into two groups: prescriptive and
descriptive. The researchers aim was either to explain how things should be done or how things are
done in the strategy-making process. Drawing from Meehl (1967), Edwards and Barry (2010) argued
that management researchers should increase the rigor of their methodological commitment in order
to decrease the risk of their theoretical contribution to knowledge being scrutinised and de-valued.
Researchers in the field of strategy process have focused their attention to the idea that strategy
content has little to no effect on the strategy-making process (Quinn 1980; Pettigrew 1985; Mintzberg
et al 1989). They argued that it is more valuable to understand the internal politics, organisational
culture and management cognition while conducting strategy process research. A new research trend
emerged lately focusing on the management team practices of strategy (Whittington 1996;
Jarzabkowski 2005). In the authors PhD research, process is conceived as being set in a context of
interaction between social actors (Chakravarthy and Doz 1992; Van de Ven 1992) and strategy as
being an “abstract conception” that social actors construct socially either intentionally or
unintentionally, within one or more minds. This construct (strategy) might or might not be formally
recorded or even articulated. Different definitions of process are scattered around the literature of
psychology, organisational behavior, sociology, operations and other fields. But for the purpose of this
research, it is regarded as a set of inter-acts and responses by which individuals construct their own
reality (Weick and Roberts 1993). Therefore, the construct is subjective in nature; how can its creation
be explored and explained? The purpose of this paper is therefore to explain how an appropriate
research approach and strategy was developed to answer the following research questions:
Q1: What is the strategy-making process adopted by UK universities?
Q2: What perceptions do people (social actors) have on the way strategy is developed in their
institution?
Q3: Why do universities adopt these approaches (socially constructed realities)?
Q4: Why might the identified regularities (social patterns) be deterministic in the strategy-making
process of higher education institutions?
While the research has a descriptive foundation, its main emphasis will be on understanding the way
people perceive the strategy-making process and explaining the regularities in the accounts of the
social actors.
2. Dominant approach to the study of strategic management
It is crucial to understand the epistemological and methodological assumptions of the current
dominant research approach adopted in strategy-process research. Since the purpose of this
research is the conceptualisation of “social reality”, the research strategy and approach must consider
this fully.
As explained earlier, researchers in the field of strategic management have adopted differing research
approaches. One strand of research seeks to review the strategic management field using
Historicism. Marieta and Perlman (2000) argued that the assumptions underlying this approach reflect
their importance. They argue, “Regularities based on chronologies and observations do exist and that
historical evidence, although interpretative, is meaningful”. Their contribution to knowledge is mainly a
discussion around the current state of the field of strategic management and suggestions about its
future direction. Another strand of research focuses on the interplay of language, meaning, text and
talk in the strategy-making process (Knights and Morgan 1995; Suddaby and Greenwood 2005). Their
approach is either positivist or constructivist in nature. For example, a positivist will consider text
material as “isomorphic to a set of factors and that the reader can infer the meaning of text without
interaction with the speaker” (Lacity and Janson 1994). Whereas a conventionalist or constructivist
528

Lilia Zerguit
believes that “there is no pure data as all data is mediated by ones own reasoning”. Therefore, they
seek to describe the cultural influences on the speaker rather than identifying and explaining the
random variations in the text.
However, quantitatively operationalised research designs dominate the strategic management field
(Molina-Azorin, 2010). Its aim is mainly to gain an understanding of the causal relationship between
variables such as environment and structure; environment and performance; structure and
performance; and so on. Does this approach fit with the purpose of this research? The next section
will explore the philosophical and methodological commitments of positivism.
2.1 Mainstream positivist approaches to the study of strategic management
2.1.1 The aims of positivist research in strategic management:
As stated earlier, the positivist research approach aims at developing theory that is generalisable and
regarded as true knowledge. These assumptions are based on Popper’s (1959) argument that “the
output of academic research should primarily make a theoretical contribution to the field of study”. He
considers theory as a vehicle that helps us “rationalise”, “explain” and “master” the world. In this
context, human behavior is reduced to the product of the external environment and the human
interactions are a set of causal relations. Donaldson (1997) highlights that positivists regard strategy
as an analytical technique for improving corporate success rather than a social phenomenon. He
states: “a fully positivist approach would not presume to call the approach strategic management but
would rather call it corporate development. It would seek to ascertain the laws that cover corporate
development that is the laws that explain changes in corporate size… Attention would be paid to
material factors as explanatory variables…” (1997: 87). The aim therefore, of positivist research is to
explain events, and in doing so, provide a set of laws about behavior.
2.1.2 Ontological and epistemological assumptions
From a positivistic perspective, the researcher assumes a dualist ontology; the subject and the object
are two separate and independent entities. Sandberg (2005) argues that dualism considers the
research objects as divided into two main entities: “a subject in itself and an object in itself”. For
instance, in the strategic management field, a study of strategy process is typically explained by
considering the structure of the organization and the external environment as two distinct entities. As
such, it is possible for an observer to understand the true nature of a phenomenon because it is
“independent of what individuals perceive” (Hudson and Ozanne, 1988). Therefore, reality exists
objectively. Researchers have always assumed that their findings about the strategy process are
accurate representations of the way things happen in the organization; i.e. “what you see is what is
there” (Bhaskar 1978). Doolin (1966) argues that this is possible because the researcher will take a
“passive, neutral” role. Therefore the ontological assumption that the “nature of being can be
understood in an external and objective manner” is the cornerstone of positivist strategic management
research. For instance, strategy researchers such as Mintzberg studied “strategy as a pattern or a
sequence of events” in different settings such as a university, a film company, a supermarket chain,
among others. His argument that strategy could be observed empirically across an extended period of
time and space reflects the shallow realist ontology (Blaikie 2000) that reality is independent of us.
Epistemologically, a positivist researcher stipulates that there is an objective reality beyond human
consciousness and that the meanings we experience are inherent to reality itself. It is assumed that
researchers will come close to the true picture of reality by systematic observations of variables and
careful monitoring. The assumptions underlying this approach are: “phenomena is a set of facts; facts
could be observed objectively; and knowledge is an accurate representation of the phenomena”.
Schwandt (2003) argued that these assumptions are “representationalist and foundationalist” because
they claim that there is a final and an ultimate criterion that determine knowledge and represents the
external world.
2.1.3 Methodological commitment
Positivist strategy research is based on empirical social science methods that consider the knowledge
produced as “valid, reliable and generalisable”. Cooper (1997) suggests that management research is
usually carried out using quantitative methods through the testing of hypothesis. The methods usually
adopted are surveys and experiments (Doolin, 1996). These allow the researcher to conceptualise
529

Lilia Zerguit
and explain deterministically human behavior. This methodological monism is usually a result of the
use of “erklaren” (Outhwaite 1975) to produce a generalisable nomothetic knowledge.
2.1.4 What next?
The preceding philosophical and methodological assumptions indicate that there are substantial
obstacles to the application of positivism in subjective social construct research. The research
strategies adopted are either inductive or deductive. The epistemological assumptions of the former
are based on the principle of empiricism. The researcher would apply objectivist procedures in their
observations and personal opinions are excluded from the process. The latter is based around
Popper’s (1959) ideas of falsification. The main argument is that a good theory is falsifiable “because
it makes definite claims about the world”. As stated earlier, the field of strategic management is very
complex and there is no agreed way of explaining the process of strategy-making. It is argued that the
strategy-making process differs between organizations (Miller and Friesen 1977; Mintzberg et al 1998;
Hart 1991). Therefore, developing hypothesis based on the current theories is not appropriate and
would narrow the scope of the research. In addition, the assumptions around “natural laws” conflicts
with the basis of the research that strategy (as a social construct) could exist in the mind of an
individual or a “collective fiat”. Hence the arguments against the adoption of a positivist approach
seem persuasive.
The following section will discuss the proposed research design to untangle the mysteries of strategymaking
in UK universities. The research approach has elements of abduction and retroduction
research strategies to overcome the weaknesses of induction and deduction discussed above (Peirce
1903; Blaikie 2000)
3. The proposed “interpretivist” approach:
3.1 The epistemological and ontological assumptions
The author will adopt the ‘interpretivist tradition’ to the research design (Burrell and Morgan 1979;
Blaikie 2000; Cunliffe 2011). Accordingly, the human world is never a world in itself; it is always an
experienced world and relates to conscious subjects. Heidegger (1927/1981) contributes to the
argument further suggesting that reality is also “mediated” by a specific culture and language in which
the subjects are situated. Table (1) illustrates the different extensions of the meaning of reality by the
interpretivist tradition. These assumptions reinforce the authors’ position in the study of the strategymaking
process. The emphasis is on the “social view” of organisational activities, in that the different
perceptions of the “social actors” will in turn describe the occurrence of the social phenomena. The
idealist ontology that reality is constructed inter-subjectively through the meanings and
understandings developed socially and experientially is therefore central to the design of this research
approach. The author is interested in understanding the participants’ multiple interpretations and
reflections. The most significant aspect to consider at this stage is the ability of the researcher to
construct the knowledge of the social actors realities and interpret their actions. The first order
concepts generated from the data collected will only reflect the participants’ reflections and views.
Table 1: meaning of reality in the “interpretivist” tradition
Husserl (1900/1971)
Meaning of “reality”
“Subjects” are related to reality through their lived experience of the reality
Heidegger (1927/1981) Reality is mediated through our lived experience and mediated through the
specific culture, language, history...etc
Derrida (1972) Meaning of “reality” is never fixed but it is fundamentally undefined
Harbermas (1972) The description of reality is colored by the “taken-for-granted” ideologies
Bruner (1996) Reality is furnished by established “cultural conventions”
Richardson (1995) Knowledge is shaped by “images”
Wittgenstein (1922) Language shapes reality
Pictures portray reality
Keat and Urry (1982) “Theory-neutral” observations do not exist
Kant (1781) Knowledge contains elements of “ourselves”.
The external world is constructed by our mind working on our “sensory
inputs”.
The “mental forms” are accessible to us and organise what we see
530

Lilia Zerguit
The challenge posed for the researcher at this stage is the selection of appropriate subjects for the
research. If, it is assumed that strategy is a socially constructed reality, it follows that it will be
impossible to locate the people involved in the process within universities. In fact, Hardy et. al. (1983)
concluded that universities “are inundated with strategies…”. It is very difficult to understand the way
they “evolve, their trajectory and the way they change over time”. Therefore, how can we rely on the
perceptions of participants? It is very important to remember however, that the participants would
have experienced (lived) the strategy at some point in time. Their perception of their experience will
reveal some underlying concepts that could be meaningful for the research. Therefore, selecting
participants is not considered to be a major concern. In addition, the heterogeneous nature of the
participants has the potential to add richness to the data generated from the different participant
perceptions.
3.2 Methodological commitment
The author believes that the research strategy adopted to collect the data should be built around the
philosophical assumptions of the researcher, but also that the findings should be an accurate
interpretation of the perceptions of the participants. Thus allowing for theory development. The theory
will foster an understanding of reality rather than explaining reality. According to Charmaz (2006),
“interpretive theories allow for indeterminacy rather than seeking causality, and give priority to
showing patterns and connections rather than linear reasoning”. Therefore, the findings will assume
that multiple realities and truths are provisional. In addition, that the knowledge gained will be derived
from particular experiences, hierarchical positions and subject to the language used and accounts of
the social actors’. According to Blaikie (2008) “such search begins by describing these accounts, and
then deriving the categories and concepts that could be the basis of understanding”. This creates the
link between the idealist ontology and the subjectivist/constructivist epistemology of the author.
3.3 Research strategy: A double logics model “abduction and retroduction”
The proposed research approach outlined below (Figure 1) is adapted from Blaikie’s (2000) abductive
and retroductive strategies to qualitative interpretative research. This section will explain its adoption
to the strategy-making process research and its linkages with the research questions and purpose.
Stage 1: Discovery
Immersion Withdrawal
Stage 2: Production of 2 nd
order concepts
Immersion Withdrawal
Stage 3: Generating
Explanatory accounts
Figure 1: The double logics model of abduction and retroduction
From the above discussion, it follows that positivist assumptions do not fit within strategy-making
process research. It seems that induction and deduction are not appropriate research strategies
because they are both based on realist ontology (the former shallow realist and the latter cautious
realist – Blaikie 2000). The first logic, abduction aims at providing understanding rather than an
explanation of phenomena and also aims at providing reasons rather than causes of social reality.
The importance of this logic is based around constructing meanings from social actors accounts in the
context of everyday activities. According to Blaikie (2000) and Peirce (1903) abduction incorporates
what “induction and deduction ignore – the meanings and interpretations, the motives and intentions
that social actors use in their everyday lives”. As a consequence, the social world is perceived from
“the inside” with no imposition of an “outsider” view from the researcher. The researcher plays a major
role in discovering why people do what they do in the strategy-making process and making sense of
their tacit knowledge. According to Giddens (1976) tacit knowledge is not formally articulated but
social actors usually use it and refer to it.
531

Lilia Zerguit
Different from what positivists, such as Glasser and Straus (1976) believe, the author assumes that
theories and data are not created and discovered. The opposite, we construct grounded theories
through our lived experiences, interactions and research practices. The “picture” drawn from the data
collected will represent the richness of the experience of the participants only. The grounded theories
of the researcher and the participants’ perceptions are therefore “constructions of reality”. The
relevance of the method to the research relates to the philosophical assumptions that the object
(strategy-making process) and subject (participant) are related. As stated by Charmaz (1999) and
Glasser and Straus (1976), the researcher is both active (immersed in the data collection) and
reflexive (withdrawn from the social context / setting) about what the participants bring to the scene
and how they see it. The rationale behind the reflexive practice of the researcher is essential to the
abductive logic.
The researcher would
then introduce the second logic of retroduction into the explanation of the social
activity.
Retroduction involves working backwards from data, in order to generate a possible
explanation of the actors’ perceptions and their application to a given context. Therefore, the aim is to
apply the knowledge of the researcher around strategy-making process to the concepts generated by
the abductive logic. In this instance, the working definition of process and strategy-making could be a
starting point for retroduction. Again this will involve the researcher being immersed in the retroduction
process, in that it is essential that conclusions made from the abductive logic are accurate. Followed
by withdrawal from the context specific stage for reflexive consideration. This stage will provide an
understanding of the regularities and their relevance to strategy-making process within UK
universities.
Therefore, mapping
these strategies against the purpose of the research is illustrated in table (2). The
different
logics (research strategies) are used at different stages of the research to answer the
research questions posed earlier. It is clear that the purpose of each question has epistemological
and ontological underpinnings that are consistent with the researchers’ own assumptions about
reality. Therefore, to avoid contradiction between the idealist and realist ontology of retroduction, the
researcher will adopt the constructionist version of retroduction (Harre 1974). This is more concerned
with cognitive mechanisms. The process follows, placing the subjects (participants) in context and
describing strategy-making process in relation to their lives. It follows therefore, that the researcher
will gain an understanding beyond description, and will be able to explain the participant perceptions
within a set context.
Table 2: Making sense of the research strategies and research questions (adapted from Blaikie 2000)
Research questions Type of research
Purpose Research strategy
question
Q1 What… Exploratio n into the strategy Abductive logic
making process
Q2 What… Description of social actors Abductive logic
perceptions
Q3 Why… Generating understanding Abductive logic
Q4 Why… Explaining or rendering
Retroductive logic
The following section will disc uss the stages of the research strategy
to b e adopted within the
interpretative paradigm to answer the above questions.
3.4 Summary of the research strategy:
The scope of the paper is limited to the research
design based around the assumptions of the
researcher, the research strategies and the paradigm
underpinning the approach. The diagram (1)
above illustrates the different stages of the research strategy and the level of involvement of the
researcher in the different stages of the research. It is important to keep in mind the reflexive nature of
the researcher and the need for capturing social perceptions as they arise. The principle behind the
cycle is the ability of the researcher to adapt and adjust to the social reality, and then distance oneself
from the collected data to generate an abstract conception.
The first stage entails the discovery of how actors perceive and typify the strategy-making process.
There
is a greater need to understand the setting and the context of the participants, to engage in
their social reality. The researcher will start by collecting “extant text” from institutional documents and
external bodies reports to gain a clearer idea about the strategies or strategic decision of the
institutions over an extended period of time. Exploring the purposes and objectives of the records
532

Lilia Zerguit
enables them to be placed into perspective and helps in understanding the setting of the participants.
This allows for establishing rapport with participants and gaining their trust. Then, the researcher will
engage in an intensive interviewing process. The purpose of the interviews seeks to explore the
experience of the participants in relation to the strategy-making process of their own institution, their
actions and perceptions. The main purpose of this stage is to discover how the social actors
(participants) view and understand the way strategy develops/forms in their own institution. The
researcher’s role is to elicit participants’ interpretation of their experiences. An open-ended, semistructured
approach to questions is more appropriate for this type of research. The ability to capture
the perceptions and views of the participants are the key to generating rich data. The researchers role
then is to interpret their meaning in the context of the research question. This allows the researcher to
keep a close eye on the perceptions and views of the participants. The researcher has a hermeneutic
task to be “immersed in the way of life of the actors”. It is crucial to fully understand the setting of the
subjects. This will help the researcher to build rapport with the social actors by relating to their preexisting
personal involvement in the topic (Blaikie 2000; Chamberlain 2006).
The data collection and analysis will take place simultaneously. It is argued that new themes and
concepts
could emerge during the data collection stage (Charmaz 1999). Consequently, the
researcher will consider them as new themes or trails of enquiry worth investigating further. Therefore,
the researcher will play an active role of collecting and analysing the data at the same time. The
double-headed arrows represent the going forward and backward in the data collection/analysis.
Therefore, grounded theory logic comes into play. The researcher aims at reporting the accounts in
language that stays as close to the original participant perceptions. These accounts are called “firstorder
concepts”.
The second stage entails the generation of “second-order concepts”. The researcher should produce
a technical account of the lay concepts, meanings and motives of the participants’ perceptions. At this
stage, the researcher is removed (or withdrawal) from the social reality in order to generate an
understanding of the way strategy is perceived to be created/formed in the institution. This assumes
the application of an interpretive act in collecting the data but utilising a neutral tone of analytical
discourse. The participants are supposed to be able to recognise themselves in the discourse. To
avoid the contamination of the accounts, the author might adopt “member checking” process
(Douglas, 1971). This entails actors’ checking the accounts and the representations generated by the
researcher.
The third stage
is based on the retroductive logic; meaning-making is implicit moving between theory
and
practice, each informing one another. This will reflect the constructivist approach, focusing on
subjectively experienced events and making-sense of the accounts in specific contexts. This final
stage will supplement the understanding generated from the lay accounts with the researcher’s
opinion through immersion and the importation of existing theoretical ideas. The explanation of
accounts necessitates the researchers withdrawal from the social reality to interpret their meaning.
As explained above the best methods for data collection consists of: retrospective data collection, in
depth
semi-structured interviews and periods of participants’ observation. It is argued that retroduction
is the process of using one theory as a model to devise another theory (Blaikie 2008). The author
therefore selected the double logic strategy in order to allow for theory development from the
qualitative accounts.
4. Limitations and challenges
The complexity of the devised methodological
approach could be questioned for the scope of the
current PhD. The researcher might have to stop after generating understanding of the lay concept
without explaining their meaning. The third stage could be conducted after the period of the PhD as a
research project involving different comparative case studies. Another issue that may arise from the
research is the difficulty to retain the integrity of the phenomena. The author aims at researching
further the field of research strategies and their relevance to make sure the participants perceptions
are well represented and not contaminated. At this stage, the researcher is new to the field of
management research. The process of devising this approach is very enriching in relation to
broadening the knowledge of the underlying assumptions about research.
533

References
Lilia Zerguit
Adcroft, A and Willis, R. (2008) “A Snapshot of Strategy Research: 2002-2006”, Journal of Management
History, Vol. 14, No. 4
Bhaskar, R. (1978) A Realist Theory of Science, Leeds Books, London
Blaikie, N. (2000) Approaches to Social Research, Polity Press, Second Edition, UK
Blaikie, N. (2008) Designing Social Research, Polity Press, Second Edition, UK
Bourdieu, (2004) The Logic of Practice, Polity Press, UK
Bracker, J. 1980) “A Historical Development of Strategic Concept”, The Academy of Management Review, Vol
15, No. 2
Bruner, J. (1996) The Culture of Education, Harvard University Press, Cambridge
Burrell, G. and Morgan, G. (1979) Sociological Paradigms and Organisational Analysis, Heinemann, London
Chakravarthy, B.S. and Doz Y. (1992) “Strategy Process Research: Focusing on Corporate Self-Renewal”,
Strategic Management Journal, Vol. 13
Chamberlain, G. P. (2006) “Researching Strategy Formation Process: An Abductive Methodology”, Quality and
Quantity, Vol. 40
Charmaz, K. (1999) Constructing Grounded Theory: A Practical Guide Through Qualitative Analysis, Sage
Publications Ltd
Charmaz, K. (2006) “Measuring Pursuits, Marking Self: Meaning Construction in Chronic Illness”, International
Journal of Qualitative Studies on Health and Well-Being, Vol. 1
Cooper, M. (1997) “On Objectivity in Qualitative Research”, College Composition and Communication, Vol.
48, No 4
Cunliffe, A. L. (2011) “Crafting Qualitative Research: Morgan and Smircich 30 Years on”, Organisational
Research Methods, Vol. 14, No. 4
Derrida, J. (1973) Speech and Phenomena, Northwestern University Press, Evanston
Donaldson, L. (1997) “A Positivist Alternative to the Structure Action Approach”, Organisation Studies, Vol. 18,
No. 1
Doolin, B. (1996) “Alternative Views of Case Research in Information Systems”, Australian Journal of
Information Systems, Vol. 3, No. 2
Douglas, J. D. (1971) Understanding Everyday Life, Rutledge and Kegan Paul, London
Edwards, J. R. and Barry, J. W. (2010) “The Presence of Something or the Absence of Nothing: increasing
theoretical precision in management research”, Organisational Research methods, Vol. 13
Giddens, A. (1976) New Rules of Sociological Method, Hutchinson
Glasser, B.G. and Strauss, A.L. (1967) The Discovery of Grounded Theory: Strategies for Qualitative
Research, Aldine, Chicago
Habermas, J.(1972) Knowledge and Human Interest, Heinemann Educational Books, London
Hardy, C., Langley, A., Mintzberg, H. and Rose, J. (1983) “Strategy Formation in the University Setting”, Review
of Higher Education, Vol. 6
Harre, R. (1974) “Blue Print for a New Science”, P. 240-9, in Armstead, N. Restructuring Social Psychology,
Penguin
Hart, S (1991) “Intentionality and Autonomy in Strategy Making Process: Modes, Archetypes, and Firm
Performance”, Advances in Strategic Management, Vol. 7
Heidegger, M. (1927/1981) Being and Time, Vol. 1-2, Doxa, Sweden
Hudson, L. and Ozanne, J. (1988), “Alternative Ways for Seeking Knowledge: a practice perspective”, European
Management Journal, Vol. 24. No. 5
Huff, A. S. and Reger, R. K. (1987) “A Review of Strategic Process Research”, Journal of Management, Vol. 13,
No. 2
Husserl, E. (1900/1971) Logical Investigations, Routeldge and Kegan Paul, Vol. 2
Jarbkowski, . (2005)
Kant, I. (1781) Critique of Pure Reason
Keat, R. and Urry, J. (1982) Social Theory as Science, Routeldge and Kegan Paul, London
Knights, D. and Morgan, G, (1995) “Strategy Under the Microscope: Strategic Management and IT in Financial
Service”, Journal of Management Studies, Vol. 32
Lacity, M. and Janson, M. A. (1994) “Understanding Qualitative Data: a framework of text analysis methods”,
Journal of Management Information Systems, Vol. 11
Marieta, M. and Perlman, M. (2000), “The uses of authority in economics: shared intellectual frameworks as the
foundation of personal persuasion”, American journal of economics and sociology, Vol. 59, N. 2
McKiernan, P. (1996), The Historical Revolution of Strategic Management, Aldershot, Vol. 2
Meehl, P. E. (1967) “Theory Testing in Psychology and Physics: a methodological paradox”, Philosophy of
Science, Vol 34
Mintzberg, H., Ahlstrand and Lampel, J. (1998), Strategy Safari: the complete guide through the wilds of
strategic management, Prentice Hall, First Edition
Miller D. and Friesen, P. H. (1978) “Archetypes of Strategy Formulation”, Management Science, Vol. 24, No. 9
Molina-Azorin, J. (2010) “Mixed methods Research in strategic management: Impact and Applications”,
Organisational Research Methods, Vol. 1
Outhwaite, W. (1975) Understanding Social Life: The method called Verstehen, Allen and Unwin
Pettigrew, A. (1985) The Awakening Giant: Continuity and Change in ICI, Basil Blackwell, Oxford
534

Lilia Zerguit
Pierce, C. S. (1903) “Pragmatism as a Logic of Abduction” (p. 226), in The Essential Pierce: Selected
Philosophical Writings, Vol. 2, Indiana
Popper, K. (1959) The Logic of Scientific Discovery, Hutchinson, London
Quinn, J. B. (1980) Strategies for Change: Logical Incrementalism, Irwin
Richardson, L. (1995) “Poetics, Dramatics, and Transgressive Validity: The Case of the Skipped Line”,
Sociological Quarterly, Vol. 4
Sandberg, J. (2005) “How Do We Justify Knowledge Produced within Interpretive Approaches?”, Organisational
research methods, Vol. 8
Schwantd, T. A. (2003) “Tree Epistemological Stances for Qualitative Inquiry: Interpretivism, Hermeneutics and
Social Constuctionism”, P. 292-331. In Denzin, N. K and Lincoln, Y. S. (2008) The Landscape of
Qualitative Research: Theories and Issues, Sage
Splitter, V. and Siedl, D. (2011) “Does Practice-Based Research on Strategy Lead to Practically Relevant
Knowledge? Implications of Bourdieusian perspective”, Journal of Applied Behavioural Science, Vol. 47,
No. 1
Suddaby, R. and Greenwood, R. (2005) “Rhetorical Strategies of Legitimacy”, Administrative Quarterly, Vol. 50
Van de Ven, A. H. (1992) “Suggestions For Studying Strategy Process: A research note”, Strategic
Management Journal, Vol. 13
Weick, K. E. and Roberts, K. H. (1993) “Collective Mind in Organization: Heedful interrelating on Flight Decks”,
Administrative Science Quarterly, Vol. 38
Whittington, R. (1996) “Strategy as Practice”, Long Range Planning, Vol. 5, No. 29
Wittgenstein, L. (1922) Tracatus Logico-Philosophocus, Routeldge and Kegan Paul, London
535

536

Work
in
Progress
Papers
537

538

Focus Groups as Evaluation: Exploring Issues Connected
With “Insider” Research
Deborah Anderson
Kingston University, Kingston upon Thames, UK
D.Anderson@kingston.ac.uk
Abstract: This paper discusses the use of focus groups in evaluating a specific learning and teaching initiative
amongst a group of students. Of particular interest is the issue of “insider research” i.e., teaching staff conducting
research amongst students. “Insider research” raises questions for both the interviewer and the respondents.
From the interviewer’s perspective, insider research can raise concerns over taken for granted understanding
and the potential loss of crucial information. Being too familiar with a situation may prevent us from recognising
important patterns of practice. On the other hand, access to respondents will generally be more straightforward,
rapport may be easier to establish and being an insider may force a researcher to surface any assumptions in a
way that an “outsider” may not. The paper concludes in favour of insider research as long as the interview
process includes continual reflection and detachment.
Keywords: focus groups, research amongst students, learning and teaching, insider research
1. Introduction and the research context
Assessment in business education often involves the development of a literature review. Whilst
guidance on search, retrieval, writing and referencing is widely available, there is less explicit help
with the synthesis of theoretical ideas and concepts required for critical review. With this in mind, we
developed a learning and teaching initiative: “Experts and Novices” in which students are asked to
research a specified topic, then act as an “expert”, by explaining their reading to other “novices” and
in doing so begin to identify themes and contradictions in a body of literature. This exercise would be
completed several weeks before the hand-in date for the assessment, giving time for reflection and
further thought.
Whilst initial results in the form of an improved quality of literature reviews was observed, we were
keen to develop a deeper understanding of the students’ views of this initiative. Structured
questionnaires were not appropriate as we wanted to explore the students’ perceptions in their
language, and not our own understandings.
A qualitative approach was chosen as the best way to investigate detailed perceptions and opinions
of the students who had taken part in the “experts and novices” sessions. Focus groups had the
advantage of generating data quickly, which was important as we wanted to talk to students soon
after the process and before they became engrossed in their next assignments. “Certainly openended
interviews and focus groups can give you data much more quickly than observation and/or
recording-although more slowly than text” (Silverman, 2001, p160). Robson (2002) also suggests that
focus groups provide an opportunity to efficiently generate substantial amounts of data. In addition,
“Participants are empowered and able to make comments in their own words, while being stimulated
by thoughts and comments of others in the group” (Robson, 2002 p. 285).
Groups were facilitated by a member of the teaching team, therefore as a tutor known to all of the
participants and as an “insider” we had to be alert to the possibility of students not discussing as
honestly and openly as they would have done with a complete outsider. On the other hand, as the
facilitator did have a strong understanding of the processes involved in the experts and novices
sessions, it could be argued that they may have brought something to the research that an outsider
would not have been able to do. For example, an in-depth knowledge of the literature reviews
discussed may have enabled a fuller probing than someone with no understanding.
In order to ensure that the primary research questions were addressed, a research guide was
developed with some “prompts” in case discussion dried up. We acknowledged that as qualitative
researchers regard knowledge as situated, then it is likely that interviews or discussion will draw on
social experiences or processes rather than abstract questions. With this in mind, wherever possible,
participants were asked about their own personal experiences of writing the literature review.
539

2. Insider research
Deborah Anderson
By conducting qualitative research in this way, the research team were faced with several challenges.
In qualitative research, the researcher is often the primary instrument for data collection and analysis.
Data is mediated through this instrument (as opposed to say a questionnaire) and the researcher can
adapt techniques and tools (such as the interview guide) to changing circumstances (Merriam, 1998;
Lloyd-Jones, 2003). In this research, as well as being the research instrument, the teaching team
were also in the position of “insiders” as individuals known to the participants. This raises several
issues which will now be discussed.
Insider research issues are well documented in the nursing literature (Bonner and Tolhurst, 2002;
Coghlan and Casey, 2001; Hart, 1996) and often aim to defend qualitative research against the
arguably more objective scientific research which dominates medical research. Asselin (2003) defines
insider research as when the researcher “conducts studies with populations, communities, and
identity groups of which they are also members” (Asselin, 2003, p100).
Insider research issues arise for both the interviewer and the respondents. From the interviewer’s
perspective, insider research can raise concerns over taken for granted understanding which may
cause the researcher to overlook important data (Asselin, 2003; Bonner and Tolhurst, 2002). By
assuming we already know and understand the subject under discussion, we may not probe in
sufficient depth and therefore lose crucial information. Being too familiar with a situation may prevent
us from recognising important patterns of practice. As someone very close to the subject of the
research, there is also a danger of reflecting on one’s own experiences rather than those of the
participants. However, as Eraut (2004) notes in relation to studies of workplace learning,
“performance therefore cannot be well understood by disengaged novices, trainers or researchers”
(Eraut, 2004, p.259). Furthermore, Eraut (2004) also notes that relationships of high mutual trust that
take time to develop can greatly facilitate data collection.
Ethical issues were also a potential problem. For example, what if we uncovered unprofessional
practice? What if we were asked to keep remarks “off the record”? Would it be possible to ignore such
remarks or would they stay with us and influence our thinking even if not transcribed? None of these
issues in fact materialised, despite us being alert to the possibility.
Whilst there are problems associated with insider research, it is also useful to identify some of the
advantages. For example, access to respondents will generally be more straightforward, rapport may
be easier to establish (Bonner and Tolhurst, 2002) and most of the structures and procedures under
discussion will already be understood, requiring less explanation. Bonner and Tolhurst (2002) also
claim that insider research can explore the process rather than the outcome of practice. As our
research was concerned with understanding processes, this is particularly relevant. It is also
important to note that these issues are not only present in insider research. Bassey (1999) notes that
all research, including data collection, interpretation and dissemination reflects a “partisanship”
(Bassey, 1999 p.90) deriving from the social identity and values of the researcher. Indeed the notion
of the researcher as a neutral spectator in the social world has been widely dismissed (Smith and
Hodkinson, 2002).
To overcome some of the issues, detachment, continual reflection and attempts to put aside previous
knowledge in an interview were vital. Continually checking and reflecting on respondents’ viewpoints
can help avoid making assumptions during data collection (Bonner and Tolhurst, 2002). As part of our
data collection plan, we asked respondents for an opportunity to return to them after the first interview
to probe further into areas where we may have allowed assumptions to obscure the true picture.
Despite the issues raised, we would argue that being an insider forces a researcher to surface these
assumptions in a way that an “outsider” may not. An outsider could assume that they are objective
and yet they inevitably will have some prior knowledge which will affect their interpretations. Indeed, in
educational research, the quality of interpretations is an important part of the research, a stance which
inevitably draws as much upon prior understanding as it does on the actual research data. It is
unlikely then, that even the most committed outsider will not bring some prior knowledge to the
analysis and interpretation of findings (Flyvbjerg, 2001). In addition, from a practical point of view, an
outsider may not have the opportunity to re-interview participants, even if assumptions were
recognised. Overall, despite concerns about insider research, we would argue that acknowledgement
540

Deborah Anderson
of the position enabled us to deal with the issues in a way that an outsider may have overlooked, and
ultimately contributed to academic rigour in the study.
Whilst confidentiality between participants could not be assured, any quotations used in the
subsequent write-up were not attributable to individuals. We also provided a full explanation to
participants including why they were invited to be interviewed. In this way, we hoped to avoid
deception about the nature of the study (Creswell, 1998).
Mason (2002) asks to what extent interviewing achieves ethical goals. She suggests that interviewees
may be given more freedom and control in a semi-structured interview than a more structured
approach. In this study, it certainly felt that a semi-structured interview enabled participants to broach
issues not necessarily anticipated by the interview team, thereby generating a fuller and fairer
representation of their perspectives.
3. Conclusion
In conclusion, we would argue that as long as ethical issues are considered, and the interview
process includes continual reflection and detachment, then the benefits of being an “insider” when
researching amongst students outweigh the disadvantages.
References
Asselin, M.E. (2003) Insider research: issues to consider when doing qualitative research in your own setting.
Journal for Nurses in Staff Development. Vol. 19, No. 2, pp 99-103.
Bassey, M. (1999) Case study research in educational settings. Open University Press, Buckingham.
Bonner, A. and Tolhurst, G. (2002) Insider-outsider perspectives of participant observation. Nurse Researcher.
Vol. 9, No. 4, pp 7-19.
Coghlan, D. and Casey, M. (2001) Action research from the inside: issues and challenges in doing action
research in your own hospital. Journal of Advanced Nursing. Vol. 35, No. 5, pp 674-682.
Creswell, J.W. (1998) Qualitative Enquiry and Research Design. Sage Publications.
Eraut, M. (2004) Informal learning in the workplace. Studies in Continuing Education. Vol. 26, No. 2, pp 247-273.
Flyvbjerg, B. (2001) Making social science matter: why social inquiry fails and how it can succeed again.
Cambridge, Cambridge University Press.
Hart, E. (1996) Action Research as a professionalizing strategy: issues and dilemmas. Journal of Advanced
Nursing. Vol. 23, pp 454-461.
Lloyd-Jones, G. (2003) Design and control issues in qualitative case study research. International Journal of
Qualitative Methods. Vol. 2, No. 2, pp 1-18.
Mason, J. (2002) Qualitative Researching. 2 nd Edition. Sage Publications, London.
Merriam, S.B. (1998) Qualitative research and case study applications in education. Wiley, San Francisco.
Smith, J.K. and Hodkinson, P. (2002) Fussing about the nature of educational research: the neo-realists versus
the relativists. British Educational Research Journal. Vol. 28, No. 2, pp 291-296.
Robson, C. (2002) Real World Research. Second edition. Oxford, Blackwell.
Silverman, D. (2001) Interpreting Qualitative Data. Methods for analysing talk, text and interaction. Second
edition. Sage.
541

The Drivers Behind the Establishment of a Dissertation
Supervision Process in the French Grande Ecole System
Marie Ashwin 1 and Alan Hirst 2
1 Ecole de Management de Normandie, France
2 London Southbank University, UK
mashwin@em-normandie.fr
hirsta@lsbu.ac.uk
Abstract: Over recent years the School in question has been working towards international accreditation to
maintain its status within the Grande Ecole system in France. The Grade de Master requires that a school
prepares students for the working world, that is has well qualified full-time academic staff, and that it provides the
opportunities for study overseas. The focus upon obtaining internationally recognized accreditation in recent
years has resulted in the improvement of a number of systems within the School. The most recent initiative is the
drive to improve academic research skills and the introduction of a system for the allocation of supervisors for
final year students undertaking a dissertation. The initial results following the harmonization of the research
methods syllabus across 15 programmes and the development of a centralized allocation of supervision are
promising. Students are demonstrating an understanding of the differences between professional and academic
literature and the need to identify an appropriate methodology for their research projects. Review of the initiative
at the end of the academic year will enable the steering committee to make necessary modifications for the
future.
Keywords: research methods, research skills, supervision, supervisor training
1. Introduction
As has been already identified in previous papers presented at the ECRM over the past four years,
the educational philosophy within France differs greatly from that in what is referred to as the Anglo-
Saxon environment. In addition to the university sector there are approximately 250 grande ecoles,
which were originally founded by local Chambers of Commerce, offering programmes to meet the
needs of the job market. The focus of these institutions belonging to the Conférence des Grandes
Ecoles is very much on practical application of learning in the business context over a number of
years of post-baccalaureate study. There are different levels of entry, some of which require students
to compete in a national selection process, and schools offer a range of different diplomas.
The School in question has adopted the teaching and learning approach of the Ecole des Hautes
Etudes Commerciales (HEC Entrepreneur), which emphasizes the regular engagement of students in
live field projects for companies. The projects are focused upon elements of the syllabus in each year.
HEC was founded in 1881 by the Chambre de Commerce et d'Industrie de Paris and has been
ranked the No 1 European business school for the six years up to and including 2010 by the Financial
Times.
The field projects last for a period of four or five weeks, during which time students undertake
research addressing a particular management concern within their host organization. The students
are asked, for example, to undertake a market study in the second semester of their second year, to
complement the Marketing modules they have studied in their first semester. Results are transmitted
in a written report for the project sponsor and the School, and an oral exposé to a panel of judges
made up of lecturers and professionals. They are required during this presentation to deliver an
overview of the project and identify their findings, conclusions and recommendations for the host
organization. Each student also receives feedback from their sponsor/mentor which evaluates their
individual role and team work during the project. The benefits of these experiences are plentiful. In
addition to identifying and reinforcing the links between what they study in School and the work
environment, these projects enable students to test their knowledge and develop competences for
their future professional lives.
Students are also required to undertake a period of work experience at the end of each academic
year, ranging from one to six months as they progress through the five-year programme. Each
placement is assessed in two ways: the production of a ‘Work Placement Report’ gives the
background to their placement as well as outlining the projects they were involved with during their
time in the company. They are also required to make a presentation to a member of faculty on their
542

Marie Ashwin and Alan Hirst
return to the School at the beginning of the following academic year in which they summarize their
experiences. The written element for these placements does not, however, require them to include a
section identifying the academic underpinning from the literature as would be the case when
completing a dissertation.
These projects throughout their time at the School should theoretically help develop their skills and
enable them to undertake meaningful research and produce appropriate documentation to transmit
their findings.
It is a requirement for all students that they undertake a piece of research in their final year of study,
linked to the specialism they are studying, often with a focus upon their future professional plans. Until
the current academic year this project could culminate in one of a number of options. Students had
the choice to prepare a dissertation (mémoire), a business plan dossier or a work placement report.
The new requirements are that all students are to prepare a dissertation which meets specific
academic criteria in terms of theoretical and empirical content.
This paper investigates the steps that have been taken in the past six months to harmonize the
teaching of research methods and the supervision of research projects across 15 specialist
programmes on four campuses. It must be highlighted that two of these programmes are run under
the auspices of a British University where the students have a choice of activities for their final
research project. The students can choose whether to work on a Dissertation or an Integrated
Business Project. The guides for these two activities have been taken as a starting point for the
discussion of how the School might match the expectations of the Grande Ecole system, and hence
the students, and the need to meet criteria for a masters-level research project.
The authors have a background in the UK Higher Education sector where independence is
encouraged from the first year of undergraduate studies. Activities both within and outside the formal
classroom situation are aimed at helping students develop their evaluative and decision-making
abilities with an emphasis upon the importance of making informed decisions. The importance of
independent study is something which does not always fit easily within the French education system,
where students are conditioned to receiving information from tutors without question. This paper
evaluates the developments of the curriculum and learning outcomes within the School and their
implications within this specific educational and cultural environment. It also raises issues that the
authors feel confident that others have faced or are currently facing, and as such looks forward to
opening debate on these experiences.
2. Drivers and quality measurement of the programme
Unlike many institutions the School has a single award, a five-year programme leading to a Masters
diploma. The School is one of 42 which has been awarded the ‘Grade de Master’ which is validated
by the French Ministry of Education. In addition to the requirements for high levels of professional and
academic competence found in other master’s level programmes, this accreditation emphasizes
several additional elements deemed essential in contemporary business education. There is the
requirement that the programme prepares students for entry into the workplace, evidenced by the
subjects studied and annual work placements. Secondly, there is an emphasis upon the quality and
role of full-time faculty in the delivery of the programme. Finally, the School must clearly demonstrate
the international perspective of its offer, with students spending a significant period of time overseas
in a partner institution.
To add to this prestigious French status, for a number of years the School has been working towards
gaining international recognition for its education programme. Its commitment to this objective has
been rewarded by the recent success of its application for EPAS accreditation. Furthermore, having
worked for several years with an AACSB mentor, the audit is anticipated in the 2012/2013 academic
year.
One of the issues discussed during earlier meetings with the AACSB mentor was that of the
dissertation process. This paper charts the discussions which ensued and decisions made by the
School management to address the concerns. It identifies what has already been achieved to date
and the future plans to support staff and students undertaking a masters’ level dissertation.
543

3. Cultural and environmental issues
Marie Ashwin and Alan Hirst
Using the work of Hofstede (2003) it can be seen that the three dimensions of Individualism, Power
Distance and Uncertainty Avoidance support the hierarchical structures which can be found within
French organizations. In this environment power is held at the top of the hierarchy. Decisions are
made at the top, after discussions have taken place throughout the organization. Traditionally in this
culture ideas for change coming from outside the system are rejected, and the concept of being
different causes discomfort and is found to be threatening to anyone within the system. The Grande
Ecole system was set up to develop the skills and competences of those who would go on to lead
French industry. There is a hierarchy within the system with institutions such as the Ecole
Polytechnique and ENA seen as the elite. Within the Financial Times European Business Schools
ranking, HEC Paris maintained its No1 position in 2010 for the sixth consecutive year. Traditionally
admission to such an institution was gained by success in the national entrance competition, the
concours. This success recognized the individual’s ability for critical thinking, evaluation and
abstraction, qualities considered to be the keys to the development of the specific competences
required in whatever position is offered to the graduate. (Barsoux and Laurence, 2009) It is the
School’s role to then offer opportunities to develop these qualities. Curriculum content and interaction
with the professional world are regularly reviewed and revised to ensure that current practice is
reflected in what is delivered.
Integrating the professional practice with academic development is something that the Anglo-Saxon
system has developed over many years. Students expect to have to investigate a concept for
themselves and come to class prepared to debate their findings and develop their knowledge and
understanding. This concept is somewhat at odds with the predominant philosophy within the French
Education system where students expect to be given all that they need by the lecturer during their
class. In 1972 Friere criticized such an educational environment which restricts the act of thinking to
the teacher. In what he identified as the ‘banking concept of education’ (p.46) he outlined an approach
to teaching which sees the teacher as the person who knows everything, and whose role it is to
impart their knowledge to the student.
4. The learning experience
Whilst discussing dissertations Todd et al (2004) identify that whilst the scope and nature of student
projects can be very different there are a number of shared characteristics. In the first place it is the
role of the student to choose the topic and focus the project with appropriate aims and objectives. The
second characteristic is that it is a student-directed activity, albeit with support and guidance from a
tutor or supervisor. The projects typically require a substantial research component, requiring
students to collect primary data and/or to analyze secondary data. It is anticipated that the students
will be more engaged with the topic that when involved in assignments which require the production of
a presentation, an essay or a report. The result is the expectation that students will produce more ‘indepth’
work than when engaged in other assessment activities.
The benefits of undertaking an undergraduate dissertation highlighted by many authors are felt to be
pertinent to this situation, albeit that the students are undertaking master’s level research.
Involvement in the dissertation process can develop a ‘deep learning’ approach (Ramsden, 1992) to
the subjects studied. Linking to the future professional demands, Stefani et al (2000) highlight that
such activities help develop independent and effective self-directed learning skills, which in turn build
confidence in one’s ability to make decisions.
If a programme of study is to be successful in developing independent learning, there needs to be a
long-term commitment to embed it within the curriculum and developing independence within the
learning experience (Hughes, 2002).
Hand (2001) stated that as a method action research was flexible enough to be used across a wide
spectrum of research areas. It has been identified as suitable for classroom activity (Bassey, 1998)
because of the inherent involvement of the individuals who follow sequential steps during a given
project. McNiff (1988) questions whether this approach is in fact linked to research, or simply to good
teaching practice.
544

5. Previous situation
Marie Ashwin and Alan Hirst
At the end of the 2010/2011 academic year the Director of Research was the catalyst for a series of
workshops to evaluate current practice in terms of the teaching of research methods and the
supervision of research projects within the School. Two significant agreements resulted: the first was
the decision to harmonize the teaching of research methods across the 15 programmes, and the
second was to develop and implement a process for the allocation and support of supervisors. A
steering committee consisting of the Dean of Faculty, the Director of Programmes, the Director of
Research and the Dissertation Coordinator was set up to design and implement an appropriate
system.
The final years are taught in either French, or English, or a combination of the two languages.
Students can choose the language they wish to use to complete their research project as currently the
dissertation is conditional for the award of the diploma and as such is not limited to the language of
the programme being followed. There were more than one dozen tutors involved in the delivery of
research methods across the School, each of whom had developed their own syllabus with differing
content and forms of evaluation. In addition there was no single set of documentation within the
School to provide guidelines on the process of undertaking a research project, and the production of
the final document, as the students had a choice of three outputs for the module entitled
‘Mémoire/Dissertation.’ The result was that supervisors found themselves supporting students who
had not only chosen to prepare different documents for evaluation, but who had received differing
input on research design and methodology for their projects.
An additional pressure upon the system was the increase in student numbers in the final year, which
had resulted in many tutors having to supervise for the first time.
It was recognized that this situation needed to be addressed if the School wished to improve the
situation before its final audit for AACBS accreditation.
6. Developments implemented in 2011/2012
One of the first actions taken by the Dean of Faculty and the Director of Research in July 2011 was to
reduce the size of the teaching team for research methods and instigate a series of workshops to
develop a standard syllabus with a single assessment vehicle.
The next was to identify a dissertation coordinator whose task it was to design and implement a
supervisor allocation system for 52 staff and over 400 students. In addition the coordinator was
charged with the task of producing a set of bilingual documents for use across all programmes. The
timescale for completion of this second element was restricted as the documentation needed to be
approved and delivered to students at the beginning of the 2011/2012 academic year.
The final stage of the development was the running of training sessions for all tutors who would be
supervising, whether they had done so previously or not. These sessions had two objectives, the first
of which was to introduce all staff to the new approach being spearheaded by the Director of
Research. The second was to outline the requirements of a research project leading to a dissertation,
for tutors who had only previously supervised students who produced business plans or work
placement reports.
7. Reflection on results to date
This section will reflect on the progress to date and identify the positive and negative effects of the
step change that has been initiated across the School. It will address three elements: the
harmonization of the teaching of research methods; the allocation of supervisors; and the
development of a single set of bilingual documentation for use across the School.
7.1 Harmonization of the research methods (RM) modules
Evaluation of the first initiative, the harmonization of the teaching programme for research methods,
reveals that there has been significant progress in this area. Across the programmes the evaluation of
the research methods module was the preparation of a poster which presented the student’s critical
review of ten academic articles on a subject of their choice. There were similarities and differences in
545

Marie Ashwin and Alan Hirst
the work demanded by the different tutors, but all students were required to demonstrate their ability
to review and synthesize what they had found in the literature.
Feedback from the School’s librarian during the period of preparation for the assignment revealed that
many students now demonstrated that they understood the difference between academic literature
and other professional and more general sources of information. This in itself was not proof that the
initiative had worked so staff waited with baited breath as the submission date approached.
After the work had been received and marked, colleagues discussed their results informally and drew
a collective sigh of relief to find that, although the grades were not exceptional, on the whole there
was evidence of students’ understanding of the difference between professional and academic
sources. With this knowledge they were able to identify appropriate academic literature and evaluate
the contents as required in the assignment.
There was evidence that several of the tutors had had discussions with their students about whether
or not professional sources might be used when preparing a research project. The tutors emphasized
the need to focus upon academic literature for the foundation of the research, whilst supporting the
use of professional sources to build the environmental framework for the project.
7.2 Allocation system for supervision
The second phase of feedback for this initiative has only just started, and is inextricably linked to the
second initiative as students began working with their supervisors on their dissertations. In previous
years Programme Directors organized the supervision for their students on an ad-hoc basis. The new
system centralized this allocation with overall approval of the pairings of student/supervisor being with
the Dean of Faculty. Working with over 400 students and 52 staff members, on 15 programmes
across four campuses, the task was not expected to be easy.
The initial allocation took longer than anticipated as it proved difficult to develop a complete and
correct list of the students who would be doing a dissertation. This was partly due to the fact that
students changed their minds within the first few weeks of the academic year and switched
programmes. Also, students on exchange programmes from partner institutions did not necessarily
have the same requirements. Some needed to do a dissertation whilst at the School, whilst others did
not. Other students were only staying in the School for one semester so would not need a supervisor.
These details were not immediately available to the coordinator, and a great deal of time was spent
collecting information and identifying the exceptions, before the allocation could begin.
It was found that a few students and staff members had already begun discussing possible research
projects but, without exception, it was only once the initial allocations had been published that the
information was provided to the coordinator. Within the parameters of the system in place some
modifications were possible, but each one took time and needed to be negotiated and approved by
the Dean of Faculty. The final allocation documents were issued at the beginning of February 2012, a
good four months after the initial lists had been produced.
7.3 Documentation
Working with existing School documentation, and previous handbooks developed in other institutions,
the coordinator was able to produce a single set of documents for all programmes. Evaluation
feedback sheets with assessment criteria for each stage of the process were developed and
approved by the Director of Programmes, the Director of Research and the Dean of Faculty and
included in the appendices of the Dissertation Handbook. With versions in French and English, this
handbook, the ’Guide de mémoire’ for those studying in French, was sent to all staff and students in
October 2011. Instructions were sent out to ignore all previous handbooks and documents and only to
work with the updated edition.
Other staff members have been providing documentation to their students on specific areas, so a
request has been made that all these be lodged centrally to ensure that they do not contradict the
official handbook and, if this is the case, to make them available for others to access.
546

8. Conclusion
Marie Ashwin and Alan Hirst
The harmonization of the research methods modules in the final year has had positive outcomes. The
first is that students have demonstrated their understanding of the differences between academic and
professional sources. Secondly, the undertaking of a similar assessment for the research methods
module across all the programmes means that the initial preparation for the dissertation has been
more consistent than in previous years. Supervisors working with students from more than one
programme are reporting fewer differences in levels of awareness and understanding of the research
process. The work submitted to date shows the students are investigating different methodological
approaches and identifying appropriate options for their individual projects.
The initial stages of the supervision allocation process were implemented, in general without any
great difficulty. Minor difficulties were reported and resolved with the Dean of Faculty. Supervisors are
being supported with workshops run by the Dean of Research, and the publication of the new
documentation which is being used across all programmes.
The strategy of the School with its focus upon the future is driving the change process with a relatively
short time scale to ensure it is ready for the accreditation audit. It has been possible to make what can
be considered as significant advances within the School in a relatively short time. The coordinator has
had to ensure that the changes suggested and implemented have taken into account both the existing
situation and the requirements of the accreditation process, not forgetting the tensions within the
context of French culture, where debate and consultation are the norm.
Within the constraints of the organizational and educational system the results are considered very
promising, and work has already begun on improving the system for the next academic year. This
positive outcome highlights the need to address the impact of cultural differences within educational
systems when working to implement any change.
Questions to be addressed in the future include the development of supervision skills as well as the
understanding of responsibilities in the student/supervisor relationship.
The authors are looking forward to a lively discussion of current practice, strategies tried and
resolutions found for these and similar issues from like-minded colleagues.
References
AACSB- http://www.aacsb.edu/ (Accessed on 10 th January 2012)
Barsoux, J. and Laurence, P. (2009) The Making of a French Manager, Harvard Business Review, 69:4, 58-67
Bassey, M. (1998) Action research for improving educational performance. In R; Halsall (ed) Teacher
Research and School Improvement, pp 93-108. Buckingham: Open University Press
Financial Times - http://rankings.ft.com/businessschoolrankings/european-business-school-rankings-2010;
(Accessed 6 th February 2012)
Friere, P. (1972), Cultural action for freedom Penguin education. Harmondsworth, Penguin
Hand, L. (2001) Action research – a way forward for accounting educators. Paper presented at the BAA SIG
in Accounting Education annual Conference, University of Glamorgan, 2-4 July.
Hofstede, G. (2003) Culture’s Consequences: comparing values, behaviors, institutions and organizations
across nations 2 nd Ed, SAGE, Los Angeles
Hughes, P. (2002) Developing independent learning skills, paper presented at the 2nd Annual Skills
Conference ‘Implementing skills development in higher education: reviewing the territory’, University
of Hertfordshire. Available online at : http://www.herts.ac.uk/envstrat/
HILP/conferences/2nd/PAPERS/Hughes%20.doc; (Accessed 6 th February 2012)
McNiff, J. (1988) Action Research: Principles and Practice, Basingstoke, Macmillan
Ramsden, P. (1992) Learning to teach in higher education (London, Routledge).
Todd, M., Bannister, P. & Clegg, S. (2004): Independent inquiry and the undergraduate dissertation: perceptions
and experiences of final year social science students, Assessment & Evaluation in Higher Education, 29:3,
335-355
547

Community Participants Wellbeing
Karen George 1 , Petia Sice 1 , Robert Young 2 , Safwat Mansi 1 , and Jeremy Ellman 1
1 CEIS School, Northumbria University, Newcastle upon Tyne, UK
2 School of Design, Northumbria University, Newcastle upon Tyne, UK
karen.george@northumbria.ac.uk
petia.sice@northumbria.ac.uk
robert.young@northumbria.ac.uk
safwat.mansi@northumbria.ac.uk
jeremy.ellman@northumbria.ac.uk
Abstract: The third sector provides valuable services for local communities but is struggling due to reduced
funding. More effective community engagement is called for by UK Government to devolve power and enable
local improvements. Devolved power is often gained through local community associations who are tasked to
manage community assets, calling for highly skilled community participants. They are under tremendous stress,
which may have a detrimental impact on individuals’ wellbeing. Community associations often struggle for
capable community participants as they compete with the well-known giants of the voluntary sector. When the
public think about participating in community activities there are a series of local social interactions that take
place, culminating in a tipping point, when they decide to participate, or not. This process is complex with varying
sources of information linking into decision making, which, when coupled with the needs of community
associations, necessitates careful management to ensure the wellbeing of both. Prior evidence of complex
networks with active communicating social agents show emergent properties. Exploitation of these local social
interactions and emergent properties for enhanced community participation should have a powerful influence.
The research entails a mix of narratives, semi-structured interviews and workshops to innovate appropriate online
mediation to effectively communicate to the public, encouraging virile tipping points. Involvement from the public,
community and third sectors gives this research credibility with stakeholders. The outcome will be a frame of
reference for effective community participation through ICT, applying both complexity and design perspectives to
recognise the role of local interactions that enhance wellbeing. The community sector can exploit this research by
offering well-fitting roles for career development of young and unemployed people. This could bring in additional
funding and provide more skilled community participants.
Keywords: community participation, information and communication technologies (ICT), complexity, design,
local interactions and appreciative inquiry
1. Introduction
Local Authorities are tasked by the Quirk Review to re-examine their community assets (Communities
and Local Government. 2007). Community associations are requested to manage the assets of their
leased (often peppercorn rent) buildings from Local Authorities. A new appropriately skilled breed of
community participants is needed for this role. Hudson (2011) states the additional pressure causes
stress and suggests filling the skills gap by attracting the right skilled people from outside the local
community.
To compete for community participants, associations need to develop their use of ICT. The
'Communities in Control' report raises the need for Government to find more effective ways to
communicate and engage with the community and to develop power to enable improvements
(Communities and Local Government. 2008). Wheatly and Frieze (2006) propose; collaborative
partnerships of local government, private and community sectors ensure well connected powerful
social interactions that influence at a global level.
Present systems for community participation tend to be under developed; assumptions are made
overlooking the wellbeing of stakeholders. To ensure the best fit, relevant information needs to be
gained to create a beneficial wellbeing partnership.
2. Complexity and design approach
The complexity perspective stems from developments in modern science such as chaos and quantum
theory, the study of consciousness and psychology, pointing towards a new paradigm of oneness,
where everything is connected. Energy pervades these connections, including community spirit and
the energy that flows within community groups. Positive energy supports wellbeing and creativity
(Hawkins, 2002:13).
548

Karen George et al.
Design theory focuses on the interactions and processes that enable individuals and communities to
engage in the creation of new activities or infrastructures (Schuler and Namioka, 1993). The research
focuses on the value of local social interactions to community participants, the process, employees
and management committee members of community associations, knowledge management and
society. It considers innovation opportunities to clarify complexity, scoping and mapping the objects,
services and processes to collaboratively formulate a new way forward. Faust (2009) recommends
the ingredients for positive design through utilising active involvement with narratives of rich
discussion and discourse focused on moving forwards with decision paradigms to enhance and give
ownership throughout the design process thus developing a sustainable solution for change.
Small, Sice, and Venus’s (2008) complexity perspective offers an approach representing the learning
and innovation dynamics of social systems as an emergent rather than a managed process and offers
tools for the dynamic analysis of systems behaviour. However, rigorous models are yet to be
developed as the impact of the insights of complexity theory into human affairs and the use and role
of ICT have not been fully appreciated (Stacey, 2010:211-3). Communities are social systems where
people commit to be there for each other and participate not only for their own needs but for the
needs of others. It is important to consider communities as an emergent phenomenon and take a
more generative perspective in terms of seeking to understand how they function from within. It is
equally important to consider what makes people contribute and be there for each other in an area of
common interest and wellbeing.
Design challenges simplistic ICT systems to provide unique services for individuals to enhance their
experience and wellbeing. Using existing community participants to design through social innovation
encourages engagement (Jason, 2004). Design can operate at a holistic and ecological level to
promote understanding of the nature of the community context as well as at the level of detailed
resolution. It uses sophisticated systems of visualisation and representation to encourage coownership
and responsibility for the new forms of service.
Complexity and design theories work in partnership embracing life phenomena to communicate and
control in life like emergent patterns.
3. Research
Hodges (2009) assessed four community participation websites in America for trial roles. The results
showed limitation in achieving the required benefit from ICT with failure to set clear dates for
opportunities, search the websites without registering or matching without timely searching pages.
Absent from the research is how online communication can benefit community participation as it
emerges through local interactions (Gilchrist, 2009). What are the critical tipping points that motivate
people to participate and how can these be exploited through the use of ICT? This research
examines the social epidemics that occur through tipping points for community participation (Gladwell,
2000:255). Pryor and Bright (2011:51) discuss a similarity of patterns of local interactions when
making decisions regarding career development. These social epidemics could be innovatively
exploited to enhance community participation.
“What are the motivational tipping points in community participation and how can they be utilised to
maximise active engagement through ICT?” The research aims to develop a framework for
conceptualising an online communication mediation system for wellbeing community participation. It
links to both the current UK ‘Empowering Local Communities’ policy agenda and to practical ways of
developing community participation; in keeping with the aspirations of the Big Society (Cabinet Office.
2011).
To make sense the research objectives are to:
Evaluate the best practice application of complexity and design insights into effective community
participation;
Combine the best practices to develop an approach to the study of community participation;
Utilise the approach and conduct research with the participating North East community to identify
the motivational factors of community participation;
Design and evaluate a framework with guidelines for beneficiaries for conceptualisation of an
online communication mediation system for effective community participation
549

Karen George et al.
Figure 1, shows how the research is enabled through an appreciative inquiry, focusing on positive
potential, utilising people’s imagination and innovation (Cooperrider and Whitney, 2005:16).
Destiny
How to empower, learn and
adjust/improvise?
Sustaining
Figure1: Appreciative inquiry 4-D cycle
Discovering
The best of Community
Participation
Appreciating
Affirmative
Topic
Choice
Design
What should be the
ideal?
Co-constructing
Dream
What might the world
be calling for with
Community Participation
Envisioning Results
To aid discovery, narrative data is collected from umbrella organisations to appreciate the best of
community participation, assess their needs and gain their reflective and reflexive views. The
evaluated findings direct the structured conversation for the semi-structured interviews with new
community participants and existing community associations. They rate certain feelings, emotions and
needs whilst developing individual action plans for positive improvement. This insight of the quality of
interactions is evaluated. This information is used within workshops for reflection to visualise/dream
and narrate their ideas about effective community participation and improved ICT. The correlations
from the semi-structured interviews are compared with information from the workshops to develop a
comparative list of revised motivational factors versus community participation characteristics
interpreted and visualised to design the proposition framework. This is fed back for peer review. The
stakeholders' destiny is to utilise developed ICT. Appreciative inquiries empathise with community
participation as detailed in Table 1 (Adapted from Cooperider and Whitney, 2005:30).
Table 1: Appreciative inquiry process creating appreciative community particpation
4-D Cycle Focus Community Participants Creating Appreciative
Community Associations
Discovery Mobilise a systemic inquiry into wellbeing Engage in appreciative Inquiry into the call and
community participation
interviews
capacities to create
Reflect on interview encompassing approach
highlights
Uncover multiple and
Aligning strengths for diverse stakeholder
competitive advantage
strengths
Share best practice Aligning strengths for
enhancing effectiveness and
efficiency
collective potential
Dream Envision wellbeing community Images of a better world Share collected dreams,
participation's greatest potential for Strategic vision of narrative and best practice
positive influence and effect community participation Create and present
serving society dramatic enactments
Design Craft propositions in which wellbeing Articulating community Crafting clear purpose and
community participation is boldly alive in participation values wellbeing community
all strategies, processes, systems, Crafting provocative participation principles
decisions and collaborations propositions and wellbeing Crafting a charter of
community participation relationships, roles and
principles
responsibilities
Design ideal system for
community participation
Craft design for integration
Destiny Invite action inspired by the discovery, Acting to realise the dream Publicly ask for support
dream and design days.
in alignment with the with new ICT product
principles by developing Continuous network
ICT.
innovation in alignment
with the vision and values
550

4. Conclusion
Karen George et al.
Appreciative inquiries promote empowerment and wellbeing feelings as stakeholders are engaged
and appreciated. They are coherent with the cyclic link between complexity and design. This hinges
upon local interactions which stimulate people to feel very positive and pulled towards visions of
happiness/wellbeing and positive action, thereby enriching the interactions.
The anticipated contribution to knowledge of this research is the development of an effective, efficient,
framework tailored to promote wellbeing community participation, with local to global relevance and
impact. Also, the framework will incorporate improved matching, reduced stress and disharmony for
community participants and be used to support career change/development through wellbeing
community participation.
References
Cabinet Office. 2011. Big Society - Overview [Online]. Available: http://www.cabinetoffice.gov.uk/content/bigsociety-overview
[28 March 2011].
Communities and Local Government. 2007. Making Assets Work: The Quirk Review of Community Management
and Ownership of Public Assets, Norwich: HMSO.
Communities and Local Government. 2008. Communities in Control: Real People, Real Power, Norwich: TSO.
Cooperrider, D. L. and Whitney, D. K. 2005. Appreciative Inquiry: a Positive Revolution in Change, San
Francisco, CA, Berrett-Koehler.
Faust, J. 2009. Positive Design. Journal of the American Society for Information Science and Technology
[Online], 60. Available: http://dx.doi.org/10.1002/asi.21130 [15 July 2011].
Gilchrist, A. 2009. The Well-Connected Community: A Networking Approach to Community Development. 2nd
ed. Bristol: The Policy Press.
Gladwell, M. 2000. The Tipping Point: How Little Things can Make a Big Difference, London, Abacus.
Hawkins, D. R. 2002. Power vs. force: The Hidden Determinants of Human Behavior, Carlsbad, Calif.; London,
Hay House.
Hodges, J. 2009 'A Virtual Matchmaker for Volunteers' The Wall Street Journal. [Online]. Available:
http://online.wsj.com/article/SB123439604394574921.html#printMode [7 Nov 2011].
Hudson, S. 2011. Community Organisation Boards are 'Often Short of Skills' [Online]. Available:
http://www.thirdsector.co.uk/channels/Governance/Article/1063067/Community-organisation-boards-oftenshort-skills/
[5 April 2011].
Jason, L. 2004. Participatory Community Research: Theories and Methods in Action, Washington, DC; London,
American Psychological Association.
Pryor, R. and Bright, J. 2011. The Chaos Theory of Careers, London: Routledge.
Schlosser, T., Dunning, D. and Fetchenhauer, D. 2011. What a Feeling: The Role of Immediate and Anticipated
Emotions in Risky Decisions. Journal of Behavioral Decision Making [Online]. Available:
http://dx.doi.org/10.1002/bdm.757 [18 Nov 2011].
Schuler, D. and Namioka, A. (eds.) 1993. Participatory Design Principles and Practices, New Jersey: Lawrence
Erlbaum Associates.
Small, A., Sice, P. and Venus, T. 2008. A Framework for Promoting Learning in IS Design and Implementation
Learning Organization [Online]. Available: http://hdl.handle.net/10145/77383. [7 May 2011].
Stacey, R. D. 2010. Complexity and Organizational Reality: Uncertainty and the Need to Rethink Management
After the Collapse of Investment Capitalism, London: Routledge.
Wheatly, M. and Frieze, D. 2006. 'Using Emergence to Take Social Innovation to Scale', [Online]. Available:
http://www.margaretwheatley.com/articles/emergence.html [19 Mar 2011].
551

Enhancing Students Academic Skills: The Case of the
Bucharest Academy of Economic Studies
Irina Purcarea
Academy of Economic Studies, Bucharest, Romania
irina.purcarea@gmail.com
Abstract. The area of students’ academic skills is crucial in what concerns ensuring academic success. Students
should be encouraged to become effective learners and universities play a key role in providing a comprehensive
academic skills support. Throughout time, various studies have pointed out a number of strategies meant to help
students improve their academic skills. At the moment, Romanian universities should gain a thorough
understanding of the educational process, especially from the perspective of the academic skills Romanian
students acquire within the learning process as well as identifying ways in which to encourage Romanian
students to take more responsability for their learning. My current project focuses on the academic skills of the
students of the Academy of Economic Studies, namely students at the Faculty of Commerce and students at the
Faculty of Business Administration in foreign languages (English section), both faculties being part of the
Bucharest Academy of Economic Studies. The research intends to gain an insight into the academic skills that
students are using in their learning process as well as providing the opportunity to make a comparative analysis
between the academic skills employed by students in the Faculty of Commerce and students in the Faculty of
Business Administration in foreign languages (English section). In order to meet the research objectives set, a
questionnaire was designed to address four key academic skills, namely independent learning skills, critical
thinking skills, research skills and exam skills. The questionnaire consists of both closed-ended questions and
open questions. The closed-ended questions relate to aspects specific to each of the four types of academic
skills whereas the open questions allow students to provide input in what concerns the methods and tools to be
used in order to further develop their academic skills. The findings will help our University gain an understanding
of the academic skills students benefit from and specific areas for improvement as well as identifying specific
methods/tools that can be applied to further develop their academic skills.
Keywords: academic skills, students, research
1. Introduction
Many researches seek to find how learners go about learning something, what makes learners
successful at learning something, and why some people are more effective at learning than others
(Chien Kuo Lee, 2010). Other researches have addressed the issue of academic performance, with
an emphasis on a specific group of learners – the so-called „underachievers”, looking at the causes
for underachievement as well as the specific approaches to address this aspect (Ogbonnia Chukwu-
Etu, 2009).
Strategies have a prominent role in the cognitive view of learning because they represent the dynamic
mechanisms underlying thinking and learning processes (Abhakorn, J., 2008). Understanding and
identifying the various strategies employed in the learning process constitutes one step forth in
ensuring academic success.
The paper looks at the issue of enhancing students academic skills, with a focus on the Academy of
Economic Studies in Bucharest. The first section of the paper relates to several examples of „success
stories” in terms of providing support to developing students academic skills. Then I provide a short
presentation of the Academy of Economic Studies in Bucharest, with an emphasis on the Faculty of
Business Administration (teaching in foreign languages), which is the main focus of the research. In
the last section of the paper I describe the research methodology used in conducting this study as
well as the expected research findings.
2. University support to developing students academic skills – success
stories
Nowadays, universities around the world acknowledge the importance of developing students
academic skills and offer various services for enhancing students academic skills.
The University of Melbourne provides support in terms of enhancing students academic skills in the
following areas:
Language skills in what concerns the features of language use in academic English; the objective
is to help students’ writing to be clear, accurate, and academic
552

Irina Purcarea
Speaking and presenting, focusing on effective strategies in the areas of pronunciation,
presentations and participation
Reading and research, will provide students with strategies to enhance their reading and
researching skills, with an emphasis on the following areas: literature search, reading strategies
and note-taking
Study skills, focusing on effective strategies in the areas of active learning, exams and time and
task management; students are being prepared for major assessments such as exams, and
organizing their time and tasks
Writing skills, providing guidelines to writing essays, other writing tasks as well as drafting written
work
The Academic Skills Unit within the university works closely with the faculty student centre to provide
students with a variety of skills development options such as workshops and short courses, resources
and individual tutorials.The university is also running individual tutorials with an academic skills
adivser that will help the student develop effective study skills, communication skills and academic
language skills and writing.
The University of Exeter provides, in terms of student academic skills development, two services,
namely skills e-resources and student support skills advisers. In what concerns the skills e-resources,
the main objective is to develop a suite of online generic academic skills resources, directed towards
helping students develop strategies for reading, getting the most out of lectures and essay writing.
Another important area for providing support represents employability and graduate development,
helping students in choosing their career, employability, creating a CV, choosing postgraduate study,
searching for jobs, volunteering opportunities, and support for postgraduate research students.
3. Enhancing academic skills at the Academy of Economic Studies in
Bucharest
The Bucharest Academy of Economic Studies, an institution established by Royal Decree on April 6,
1913, comprises 10 faculties, has got over 49.000 students and course attendants; 35500 -
graduation cycle, 9400 - master programmes, 2500 - PhD enrolled, over 1600 in academic schools
and post-graduation courses and 2000 didactic staff and technical and administrative personnel.
3.1 The faculty of business administration (teaching in foreign languages)
The Faculty of Business Administration (in Foreign Languages), one of the newest Faculties of the
Academy of Economic Studies, has approximately 1900 students from Romania and 44 other
countries. The Faculty's mission is to form specialists in business administration, able to meet both
the development needs of multinational companies operating in Romania and the requirements of
Romanian companies interested in expanding at international level.
The Faculty of Business Administration (in Foreign Languages) has always been interested in both
meeting and exceeding students’ expectations due to its student-centered teaching approach.
In this context, receiving feedback from students as well as identifying other possibilities in order to
meet and/or exceed their expectations (such as conducting research projects with the aim to identify
and implement new methods/teaching practices that are able to enhance students academic skills)
have always played a key role in our strategic “quest” for excellence in teaching and research.
3.2 Research methodology
In order to gain a thorough understanding of the academic skills acquired by our students as well as
their own suggestions or recommendations for support offered in what concerns enhancing their
academic skills, a research will be conducted on a sample of 200 students studying at the Faculty of
Commerce and the Faculty of Business Administration (teaching in foreign languages). The sample of
students includes 100 students from the Faculty of Business Administration (teaching in foreign
languages), English section and 100 students from the Faculty of Commerce.
The research will be conducted using two research methods. In the first stage, we will use a selfadministered
questionnaire. The questionnaire designed includes a set of statements that students
have to rank on a scale from 1 to 5 according to the degree to which they agree with that specific
553

Irina Purcarea
statement which reflects their own learning behavior. The questionnaire focuses on four types of
academic skills, namely independent learning skills, research skills, critical thinking skills and exam
skills. A number of 12 statements are provided and 4 open questions. Besides ranking each
statement given, students are required to provide specific suggestions/recommendations in what
concerns ways to help students develop these specific types of academic skills. The second stage
involves running several focus groups with students from both faculties that will focus on the
suggestions provided by students in the questionnaires with respect to identifying new methods/tools
for developing students’ academic skills.
The research is meant to underline the current orientations in what concerns the process of acquiring
academic skills as well as providing directions for enhancing the learning skills acquired by students.
The objective is therefore twofold: first to outline issues pertaining to the current learning pattern and
secondly, to bring in suggestions or initiatives that will contribute to shaping new learning patterns.
Purposes:
Independent learning skills:
Are students able to explore events, issues from different perspectives?
Are students able to analyze and evaluate information, assessing its relevance and value?
Are students able to achieve a balance between learning, employment and other responsibilities?
Are students able to use prioritization of tasks in structuring their own learning, conducting
additional research and reading to support their understanding?
Research skills
Are students familiar with the research methodology, from formulating a research problem to
acknowledging the results of the research?
Are students familiar with both quantitative and qualitative research methods?
Critical thinking skills
Which are the teaching methods used at the highest extent at the moment to develop students’
critical skills?
Which teaching methods are most effective in enhancing students’ critical thinking?
Exam skills
Are tutors providing support in terms of developing revision strategies for students?
Are students able to use various revision techniques in preparing for exams?
Are students able to cope with exam stress?
Are students able to organize the subject material?
Do students receive all important information regarding exams from the tutors/via faculty website?
3.3 Research findings
In what concerns the independent learning skills acquired, the research seeks to find out whether
students are able to explore issues from different perspectives, analyse and evaluate information in
terms of its relevance and value, if they are able to achieve a balance between learning, employment
and other responsibilities and if they use prioritization of tasks in structuring the learning, conducting
additional research and reading to support their understanding. Related to students research skills,
the findings would reveal whether the respondents are familiar with the research methodology, from
formulating a research problem to acknowledging the results of the research and if they are familiar
with both quantitative and qualitative research processes. In relation to students’ critical thinking skills,
the focus is on the current teaching practices used by tutors in enhancing students’ critical thinking as
well as those teaching practices that, according to students’ opinions, contribute to the largest extent
to enhancing students’ critical thinking. Another important component of the research is centered on
students’ exam skills, in terms of ability to use various revision techniques in preparing for exams,
ability to cope with exam stress, to organize the subject material as well as the tutor’s role in
developing revision strategies for students.
554

4. Conclusion
Irina Purcarea
Ensuring academic success represents a very important target set nowadays by universities all over
the world. Students are now acknowledging the importance of maximizing their academic experience
and an important step in this regard consists of acquiring an understanding of their own learning
experience.
The current project seeks to gain an understanding of the academic skills students from the Academy
of Economic Studies benefit from as well as specific areas that need further initiatives in order to
provide support to developing students’ academic skills. It sets to create a framework for assessing
the current learning pattern as a starting point for identifying and then implementing new teaching
methods/tools that can be applied to further develop students’ academic skills. We intend to use the
results of the research not only to improve our teaching practices but also to put forward concrete
proposals in support of further developing students’ academic skills. As the research involves
students from two faculties, it will also offer us the opportunity to outline and compare similar/different
learning patterns.
References
Abhakorn,J. (2008) The implications of learner strategies for second or foreign language teaching, ARECLS,
Vol.5, 186-204.Available at: http://research.ncl.ac.uk/ARECLS/volume_5/abhakorn_vol5.pdf
Chukwu-Etu,O. (2009) Underachieving learners: Can They Learn At All?, ARECLS,Vol.6, 84-102. Available at:
http://research.ncl.ac.uk/ARECLS/vol6_documents/ogbonnia_vol6.pdf
Kuo Lee,C. (2010) An overview of language learning strategies, ARECLS, Vol.7, 132-152, Newcastle University.
Available at: http://research.ncl.ac.uk/ARECLS/vol7_documents/lee_vol7.pdf
The Bucharest Academy of Economic Studies. [Online] Available at: www.ase.ro [Accessed 10 J University of
Melbourne, Services for Students Academic Skills. [Online] Available at:
http://cms.unimelb.edu.au/studentservices/asu/home [Accessed 10 January 2012]
University of Exeter, Academic Services. [Online] Available at: http://as.exeter.ac.uk/divisions/ee/skills/ [Accessed
10 January 2012]
anuary 2012]
555