New Zealand Association of Private Education Providers - Tertiary ...

Quality Management Systems And Their Impact
On A Tertiary Institution
Developing excellence by collegial growth, customer satisfaction and a value driven
quality culture
Bruce Knox, Executive Director, Bible College of New Zealand Inc., Auckland
Abstract
This workshop is aimed to provide each manager or supervisor within a tertiary
institution with the skills and confidence needed to most effectively develop a quality
management system. The workshop itself shares the journey and experience of one
provider in developing its custom-made quality management overview. The session
will be interactive and will explain the four stages of quality management
development.
Why quality management
How to develop a quality management system (the involvement of staff)
How to monitor quality management.
How to validate or audit quality management.
Involvement in the workshop will provide the participant with sample pages and
templates, which could be used or modified within any institution. By the end of the
workshop the participant will have gained the confidence to proceed with the
customised development of a quality management system that will reflect the needs
of their particular section, department, or institution.
Overview
This workshop answers the following questions and statements put forward. It will
provide an interactive experience specifically focused for those at the start of the
Quality journey. It assumes a provider does not already have a quality system in
place and will provide a step by step process develop such a system by addressing A
to D below, from one provider’s point of view.
A) Why is quality management important
Who in the organisation "does quality"
(See sample page 1 available to workshop participants)
B) How do you develop a QMS system using staff-based workshops
How do you achieve organisational ownership as a key outcome
(See sample pages 2 – 8 available to workshop participants)
C) How do you monitor a QMS by using reporting cycles
(See sample pages 9 – 21 available to workshop participants)
D) How does QMS reporting lead to peer review and external audits
What was BCNZ's experience in preparing for peer review
What suggestions in preparing for audit can be made
(See sample pages 22 – 34 available to workshop participants)

Why Quality Management
Fad, Common Sense or A Key to the Future
The reason why BCNZ as a tertiary theological provider got involved with the quality
journey was because of the external changes in the educational landscape.
Fundamentally, BCNZ found the definition of Quality meaning, "Doing it right the
first time" to be extremely relevant. In the new environment of tertiary education it
was imperative for a provider to be at the forefront of quality delivery.
It was the objective of Bible College of New Zealand Inc. (BCNZ) to be a quality
provider of theological education thus meeting the learning needs of each
stakeholder grouping. Since BCNZ’s inception in 1922, there had been a significant
increase in the number of providers in the New Zealand Theological Sector. BCNZ
was keen to retain its excellent reputation, and therefore, constantly recognised the
importance of the philosophy (belief) that the provider who serves the students’
learning aspirations best will grow and prosper. Accordingly, BCNZ embarked on a
journey committed to developing quality systems.
BCNZ knew that the term "quality" had been used and abused since it first appeared
in the tertiary education lexicon during the last decade. While some viewed the
implementation of quality systems as a fad and inappropriate in tertiary education,
the journey undertaken at BCNZ came from the belief that an effective quality system
was not only a key to the future of Bible College of New Zealand, but was really just
good old fashioned common sense. It was the philosophy that BCNZ's founders
Kemp, Rolls and Laidlaw believed in and were totally committed to.
For our organisation to be quality assured, we agreed that each and every person at
all levels within the organisation must commit to doing the job right the first time. We
accepted we could not afford the cost of rework. In the educational sector this
related to the area of student dissatisfaction, where "inferior workmanship" adversely
affects the quality of classroom learning, student outcomes, and organisational
systems. If we were to be quality assured, that required our people to be focused on
quality delivery at every point.
We realized each person in the organisation must know the students personally,
must know what BCNZ's common goals are and what are the special needs for each
individual student. Staff must be clear on the generic policies and procedures,
especially those that affect their area of responsibility. Individuals must have specific
quality performance goals, which they consistently own, meet or exceed.
BCNZ accepted that if each person in the organisation commits to "practising
quality", then collectively, as an organization, this quality would be reflected within the
sector. BCNZ's commitment to quality was firstly for the sake of the students, and
secondly for our organisation’s survival. We knew that if we did not deliver quality in
our culture, and systems, then students within the modern era of NZ's tertiary scene
would simply vote with their feet.

Hence, in today's highly competitive and global marketplace, quality has become the
backbone of a successful organisation. Consumers (students) at all levels of buyingpower
are smarter and more demanding. Quality literature tells us, for those who
wish to maintain leadership status there is always room for products or services that
take quality to new heights. BCNZ had to face this challenge. We had observed that
while brand loyalty still measured strongly, today's successes are a reflection of true
customer satisfaction. This, we knew, would be no different in the tertiary sector, and
in particular amongst the 134 teaching sites at each level between 1988 and 1999.
Choice was a reality in the tertiary theological marketplace. Quality performance was
a key ingredient in a student's decision-making arsenal. We accepted that a provider
not only had to be able to say they were quality assured, they must be able to prove
it.
What is a Quality Management System
Quality, we discovered, was a derivative of the entire organisation’s commitment to
developing a product, course, or service and an educational programme or course
was no exception to this rule. So quality is dependent on the totality of an
organisation’s efforts, and the involvement and input of everyone. This was
monitored through comprehensive documentation where standards were checked,
and procedures created so that performance was evaluated. The documentation, its
implementation, and periodic evaluation is a Quality Management System that
provided us with the necessary benchmarks for quality judgements to take place.
How is compliance and effectiveness measured
Once the quality management system was developed and implemented we realised
that the next step was to put in place a system of measurement. This was to report
whether or not we were achieving what we had set out as our quality indicators. A
polished QMS was great but meaningless unless it was used as a benchmark
against which to record performance. BCNZ identified that there are several tiers
available regarding the assessment of compliance and effectiveness. This
commenced at self-evaluation, followed by informal reporting, moving to formal
reporting, then peer review, and finally an external audit. The literature told us that
an analysis was essential before selecting any method of certification, as each
method provided a different degree of objectivity. BCNZ chose a hierarchy of
approaches. The final tier was an external audit. This meant that by bringing in a
third party that held no interest in the organisation, the evaluation would have more
credibility in the eyes of the students and stakeholders. We found that a seasoned
third party (NZQA) guided us through the process, and issued a certificate of
compliance (registration and programme approval) once the external audit had been
performed and accepted. In hindsight we found it was of utmost importance to select
a QMS process that mirrored our special character, integrity, motivation and
commitment to quality. The importance of this factor was to prevent a mismatch of
systems.
The aim of BCNZ's QMS was to ensure the effective and consistent delivery of good
quality biblical and theological education, and to continuously improve the quality of
theological reflection. This was achieved by enabling our educational delivery and
associated systems to be subject to self-evaluation, informal and formal reporting,

peer review and external audit. As a result we could benchmark our current
performance. This allowed for future measurements to have a baseline and hence
result in continuous improvements.
In conclusion QMS at BCNZ was used for the following reasons;
A comprehensive guide to quality in theological education and reflection.
A guide to the quality requirements and perspectives of funding agencies (MOE)
and their systems.
A self-appraisal system for organisational development.
A framework for gathering information relating to purposes such as accountability
and marketing.
A benchmark for improving consistency of performance, in order to achieve
ongoing registration and accreditation with NZQA.
A training framework for both experienced and new staff, with respect to BCNZ's
polices and procedures.
A tool for internal and external quality audit, hence ongoing creditability.
To achieve the goals stated above, BCNZ's QMS set out to identify quality features
or characteristics of our management systems, education and training services
across several standards as detailed below. These standards provided an overview
of what was involved in the system as a whole.
These standards were:
strategic management
premises and equipment
quality management
communication and administration
marketing management
financial management
staffing
guidance services
staff development
programme design
equal opportunities
programme delivery
health and safety
assessment for registration
research
relationship and alliance developments.
The system as a whole was developed against these standards. It allowed for key
performance indicators to be crafted in a comparative way that reflected both how

BCNZ performed and the quality benchmark of our organisation. This allowed for
total organisational ownership of the project and its implementation.
How to develop a QMS in detail
When engaging in the process of quality management development BCNZ
recognised three possible approaches.
Top-down, dictatorial implementation
Pseudo consultation
Genuine consultation and staff ownership
Our understanding of the literature and our experience had told us that only one
approach would genuinely work. Educational reforms that are implemented with a
top-down mentality or pseudo consultation, we believed, were doomed to failure at
the worse, or insipid modification at the best. Our organisation was therefore
committed to a process of genuine consultation. The first step to achieve this was in
a staff workshop. The consultation workshop considered BCNZ's quality process
achievements to date, obstacles to future implementation and major lessons learned
from other providers. In this, BCNZ sought to understand the QMS process
implementation, and its impact upon all its staff.
The undergirding principle BCNZ had adopted was that the person doing the job was
in the best place to know whether or not the job was done well, and thus was also the
best judge of a quality indicator. The process of a staff workshop was adopted, and
small group discussion started from ground zero and built the quality management
system. The processes that the staff workshop undertook are detailed in the
attached sample pages 2 - 8. The workshop was, if anything, the most critical part of
the exercise. Over time the initial key performance indicators (which tended to be at
the level of micro-management) were modified to reflect issues of macromanagement.
This was borne out through practice. Experience meant the workshop
drafts were then subjected to continued revision. This was supported by revisiting
the groups that did the initial work. Our experience of this process was that if
somebody wrote a key performance indicator, they were committed to living and
working towards it. The development of a quality management system, we believed,
must be a coherent, jointly-owned exercise across all participants. This venture, for
our organisation, was the hallmark as to why the QMS was a key success.
While the workshop was primarily a practical hands-on development experience
focusing on KPI implementation, it also highlighted a number of conceptual issues
about BCNZ in general, such as the need to overtly re-define the organisation’s
goals, vision and purpose as an educational provider and ministry developer. These
issues were moved off the QMS floor to the appropriate task force for response.
The workshop showed that, while nothing was new, these core concepts from quality
systems formed a basis for the future development of our tertiary theological ministry
education into the new millennium.
Planning a Quality Management System (The process)

The three-day workshop, referred to above, adopted the following simple statements
and guidelines. Quality comes from doing the right thing first time and from finding
solutions to the problems instead of simply identifying them, or laying blame. When
faced with the task of writing a QMS, BCNZ had observed many organisations had
made the comments that the process is;
Too bureaucratic
Too time-consuming,
Too costly.
BCNZ simplified the process by defining a QMS system in 3 simple steps;
Say what you do
Do what you say
Prove that you do it
Hence a working definition that was adopted within the institution was
A Quality Management System is an outline of the process and procedures used to
assure students and stakeholders of the quality of courses and its delivery.
BCNZ agreed that the QMS should be a living document (ie. an ongoing process or
standards) which was user-friendly, easy to read and 'owned' by everyone in the
organisation, otherwise it would simply sit on the shelf and collect dust. One useful
technique we adopted was to plan the critical steps over an expected time-frame,
and regularly review the performance against the plan. The understanding and
culture of QMS was to become a simple system such that outside scrutineers could
be reassured that BCNZ was what we said we were. Universal Tertiary Tuition
Allowance (UTTA) funding demanded such reassurance. NZQA used it as the basis
of the re-accreditation. The system of checks and balances adopted throughout the
organisation ensured quality was maintained. The processes of peer review and
external audit covered all aspects of College life to ensure organisational creditability.
BCNZ preferred a simple grid approach (see sample pages) that moved from the
general to the specific to achieve this. The workshop followed a step-by-step
approach that provided a clear, simple strategy, (KPI-reporting-review-audit) for all to
follow. In this exercise, as in any discipline, BCNZ discovered that QMS had its own
specific language. This included the language of elements, standards, outcomes,
evidence, performance indicators, audit and reporting. BCNZ adopted, as
appropriate, terminology for the exercise as shown on sample page 10-11.
Implementing A Quality System at Bible College of New Zealand
Following the principles outlined above, coupled with the "KISS and SMART"
approach, BCNZ's QMS was designed. The QMS's broad framework was
developed. The outcomes allowed the introduction of a quality system into BCNZ's
overall structures. Implementation happened at department and site level. Deans
and Heads of Department systematically put the KPI system in place. This was
relatively easy, as it was these staff members who had written the system, and what
was being implemented was good practice already in place. But the process didn't
end there. Arising out of the workshops were ongoing discussions concerning the
processes of self-assessment, reporting, and peer review, leading to effective
external audit.

An idea is only as good as its execution.

How to monitor a Quality Management System
The workshop highlighted the importance of monitoring the system, and identified the
challenge to create a meaningful validation mechanism for this purpose. Since our
organisation was multi-site (14) and multi-level (Certificate to PhD), the issue was to
create a reporting framework that served our organisation. Having implemented the
system of quality, it was important to make its success evident. The greatest danger
that we foresaw within a quality management system was, that having been written, it
could then gather dust in the corner, and not make an impact on the day-to-day
operational management of our organisation. We wanted the key performance
indicators of quality to be at the forefront of the minds at each operational delivery
point. Consequently, we set in place a process of reporting to our National Board.
The reporting simply reflected achievement or otherwise. Such a cyclical reporting
process achieved the goal of keeping the quality management system at the point of
being a high profile, up-front, issue. The result of the reporting gave the necessary
reassurance to the National Board that each delivery centre was doing what it had
said it would do. The documentation used within this reporting cycle can be found on
sample pages 9 - 21.
Quality Management leads to Peer Review (validation)
The final stage of the quality management system within the Tertiary Sector was that
of validating that what we said we would do, we actually did. This went beyond the
reporting to the National Board, moving on to that of a face-to-face peer review
followed by external audit. The individual deans of each Learning Centre requested
the concept of an annual peer review as opposed to a National Office inspection.
Our experience has been that within the peer review process the level of
thoroughness and collegial support was greatly magnified. The process consisted of
one dean from a Learning Centre visiting another dean from another Learning
Centre, each undertaking a comprehensive review of their processes. The key
review questions were:
Where is the evidence that you have achieved each KPI
Is what you're doing making a difference
The aim of the peer review was to encourage effectiveness as opposed to mere
compliance, that is, to validate a job well done, and to identify good practice
indicators.
This process provided immense professional benefit to both parties involved, and
professional growth and responsibility was significant. Furthermore, the issue of
quality was enhanced with the continuous attention to improvement.
Peer review (Summary)
BCNZ found a peer review had the following advantages:
Conference of equals
Understanding of each other’s process
Commitment to mutual excellence
Sharing of good practices
Opportunity for professional growth both ways.

The following key facts were given to each Dean prior to peer review:
You are not an inspector – you are in the same boat
You want growth and excellence
Reviews run parallel to and validate our QMS Report
Relationships are a key
Build the auditee up with encouragement and affirmation.
External Audit: (NZQA)
The final step within the process of validation was that of the external audit team
(NZQA) visiting our organization on a triennial (every 3 years) basis. Such a visit was
simply to validate the internal quality systems.
The following steps were the basic process for an NZQA external audit.
1. Notification of date
2. Provider completion of the NZQA self-evaluation workbook. (Linked to BCNZ
systems).
3. Scoping for the audit.
4. On-site verification.
5. Draft report to the organization followed by the final report.
Self-Evaluation (point 2 above)
BCNZ accepted that the credibility of our self-evaluation (arising from the process of
KPI-reporting-review) was largely determined by how closely the evidence gathered
matched the organisational objectives being assessed. The self-evaluation
programme, we found, was significant as a tool for review, audit, longer-term
planning, monitoring, and the evaluation of effectiveness of both the systems and the
programmes.
With NZQA forming judgments and opinions about BCNZ, the self-evaluation
provided the BCNZ and NZQA with a base from which to verify that;
there was a planned and systematic approach to achieving the organisation’s
goals and objectives
cultural and other expectations of clients (including learners) were taken into
account
information was obtained, reported and presented in a manner appropriate to
clients (including learners); and
the organization was substantially achieving its goals and objectives.
Verification of Evidence (point 4 above)
The NZQA audit team focused on verifying the organization’s evidence of compliance
with stated requirements, and evidence of effectiveness against the indicators of
good practice. The "uniqueness" of our organisation and its diversity of stakeholder
organisations were taken into account when considering the adequacy and
appropriateness of evidence.

Verification and sufficiency of evidence was substantiated using a number of
techniques
examination of documents and records
interviews and observation
sampling
corroborative evidence from different sources.
The Final Report was NZQA's decision-making about our organisation’s status for
recognition (that is, registration and/or accreditation). An action plan for improvement
was developed.
Issues arising from the QMS journey for BCNZ as a provider.
The QMS development required total organisational ownership, and hence absorbed
significant time and money to achieve its development. On-going ownership is a
process of recommitment and involvement so that the QMS doesn't slip into the
realm of the QMS manager to implement. The question of who owns quality had to
be constantly addressed so that the corporate commitment was not lost. Reporting
Cycles have needed to be modified and re-written in the light of experience as the
QMS evolved. This has meant putting in place a process of follow-up in order to
engender meaning, purpose and significance to the reporting and review process.
The peer review has now been moved to a 3-year cycle, covering section by section
on rotation. This approach has proved costly in terms of time and travel, but overall
was considered worth it.
The external audit exercise worked to BCNZ’s benefit, with the auditors showing
warm affirmation and adopting a low-key manner. We found the audit days to be
relaxed and mutually beneficial. The evidence was easily available and the paper
trail arising from our internal processes of implementation, reporting and review was
most beneficial in achieving a positive outcome. The validation of BCNZ by students,
staff, and stakeholders was imperative, with the stakeholder interviews being a key.
We found the initial timeframes of 1 day per site too short for an effective audit, the
process became too rushed. This resulted in no positives being given in the
summing up report. However, this factor was addressed in external audit Round
Two. Time was allowed for engagement, investigation, dialogue and creative
feedback covering the ‘good’, ‘bad’ and ‘ugly’.
In conclusion, the process arising from the staff workshops has proved significant in
the overall development of a quality culture.
BCNZ's journey in the development of a QMS has involved 4 years of toil but it was a
task we would gladly repeat in order to achieve the results internally and externally
for the organization, as we can now be proud of the Quality our organisation can
present in the Theological Sector.