Response to Assessment of Generic Skills Discussion Paper Thank ...

Response to Assessment of Generic Skills Discussion Paper
Thank you for the opportunity to provide an institutional response to the Assessment of Generic
Skills discussion paper. QUT has chosen to selectively respond to several questions posed within the
document.
What factors should guide the design of performance measurement instruments to assess generic
skills?
For several years, QUT has argued to the Commonwealth Government that the assessment of
institutional excellence within performance schemes, and any accompanying allocations of reward
funding, should not be executed with a level of precision that exceeds the ability of the metrics to
differentiate performance given validity, error and other limitations. This principle is paramount
regardless of the measures ultimately chosen because performance measurement is inherently a
flawed process. All direct and indirect performance metrics have limitations that prevent conclusive
comparison of outcomes and full understanding of the multiple causal factors of excellence. Error,
artefact, reliability, validity, utility, context and social value are constant challenges.
National higher education performance assessment in recent years has largely depended on selfreport
survey feedback and government datasets as sources of evidence. QUT feels that a national
performance system will be more robust if it encourages multiple sources of performance evidence
and does not leave itself vulnerable to a weakness in any one methodology or approach. It is in this
context that the potential exists for the Collegiate Learning Assessment (CLA), or a similar direct
assessment of generic skills, to provide a new and different approach to assessment of generic skills
provisioning, and to proffer insight into any institutional value-add to the generic competencies of
students at the end of their degrees.
The sector should not rush the implementation, but rather recognise that this is a relatively new
paradigm in performance assessment within Australian higher education that may have limited
discriminatory value if poorly executed. It is anticipated that it will take a couple of years for the
process to ‘bed down’ and for any initial model to be trialled and refined, even given the opportunity
for Australia to draw upon the experiences of other countries utilising the CLA and similar
approaches. The Commonwealth Government should plan accordingly and with consideration of
AHELO experiences.
QUT has provided further feedback on appropriate principles for development of performance
measurements in response to the Development of Performance Measurement Instruments in Higher
Education Discussion Paper.
What level of student participation is desirable and for what purposes? What design features or
incentives would encourage student participation?
A direct assessment instrument needs to be authentic and deep to be a reasonable test of selected
generic skills. The CLA appears to take this seriously, but the length and sophistication of the
instrument necessitates a significant investment in time by students despite little intrinsic
motivation or relative advantage. Valuing of the CLA by employers may, in time, help increase the
value of individual CLA outcomes to students.

A comment on discipline specific assessments
QUT would like to highlight a project that may offer leverage to any attempts to include disciplinespecific
assessments in the development of a broader assessment of generic skills. The Sector-wide
model for assuring final year subject and program achievement standards through inter-university
moderation project led by Kerri-Lee Krause and Geoff Scott involves eight universities and aims to
produce resources capable of supporting inter-institutional moderation for assuring final-year
subject and program achievement standards. As stated in the project application:
This project will yield a validated, robust approach for assuring subject achievement standards
through inter-university moderation in common final year subjects across disciplines. It will also trial
approaches for moderating and assuring program achievement standards, building on ALTC
discipline standards. Project resources will assist universities to implement sustainable, selfregulatory
moderation processes for monitoring subject and program standards.
Is ‘value-add’ an appropriate measure of generic skills?
The capacity for a national system to delimit any institutional value-add towards the generic skills of
students is challenging at best, arguably spurious if a worst-case scenario occurs whereby the
approach is implemented without adequate modelling and fine-tuning. QUT considers that valueadded
skill assessment has the potential to be a ‘reasonable’ measure provided several issues,
including the following five, are considered:
Firstly, the benefit of using ATAR/entry-rank-type entry scores as a proxy for baseline
competencies is questionable. These scores are a summation of prior academic performance
across a diversity of disciplines. The relevance of generic versus specific skills in achieving the
academic success in this single score is unknown and not readily determined.
Secondly, adult learning does not only occur in educational institutions. Individuals continue
to develop generic competencies during their degree – partially through the educational
institution, but also as part of general learning from daily experiences. Therefore, challenges
in separating value-add attributable to university experience from value-add attributable to
personal growth are inherent in the model.
Thirdly, the role of previous institutions and the learner in establishing lifelong, self-directed
learning capability further muddies the ability to attribute value-add to a current institution.
How much of the value-add is attributable to lifelong learning skills developed prior to
university studies (e.g. in high school) or through self-development?
Fourthly, the value-add model appears founded upon very simplistic interpretations of
generic skills and competencies development. The concepts of adult learning and education
have progressed far beyond simple models involving quanta of knowledge transferred via
instruction as held sway in the middle of last century. A depth and breadth of literature
concerning adult education, lifelong learning and deeper learning is essential context for a
thoughtful implementation. While the CLA and other similar approaches are situated within
literature acknowledging more recent constructions of learning, the usage of the CLA within
any performance measurement system would arguably result in a misalignment between
these richer underpinnings and usage to quantify systemic value-add.
Fifthly, the CLA is designed to measure critical thinking, analytic reasoning, problem solving
and written communication skills. If the purpose of the value-add assessment is to
determine the extent to which individuals have extended generic skills competencies that
are valued for societal and workplace contribution, then the applicability of these skills to
workplace success needs to be demonstrated. Validating the design with industry is
essential.
2

Despite shortcomings, QUT considers that the CLA is a more reasonable and direct test of a select
number of generic skills than the old Graduate Skills Test. The approach of the former involving
respondents solving a complex problem drawing from a range of artefacts is considered more
informative than the multiple choice approach of the latter (that has formed part of AHELO). The
CLA is better suited to development and improvement purposes. Arguably, prior efforts to calibrate
the CLA instrument will help support meaningfully comparison of cohorts to provide insights for
improvement.
Ultimately, directly measuring generic skills does not mean that the sum of outputted skills minus
inputted skills is a direct measure of added value. Rather, it is simply a variation in scores from
different points in time that needs to be interpreted against a range of confounding factors. As a
concluding comment, QUT would like to include the following quote by Educational Testing Service
President and CEO Kurt Landgraf as a summation of the role of value-add assessment in an overall
performance model:
Results from value-added models should not serve as the primary basis for making consequential
decisions. Other measures must be included in any fair and valid teacher evaluation system. (Banta
& Pike as cited in Klein et al., 2007)
QUT believes that the system will benefit from ongoing robust discussion within the sector and a
willingness to review the success of the measure in future years.
Klein, S. Benjamin, R., Shavelson, R., & Bolus, R. (2007). The Collegiate Learning Assessment: Facts
and fantasies. Evaluation Review, 31(5), 415–439.
Dr Sam Nielsen
Director, Reporting and Analysis
Queensland University of Technology
GPO Box 2434, Brisbane, Qld 4001
Email s.nielsen@qut.edu.au
Tel 07-3138 5085
3