Migrating digital records
1URFZdw
1URFZdw
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong><br />
A guideline for Queensland public authorities<br />
Queensland State Archives<br />
June 2012<br />
Security classification: Public
Department of Science, Information Technology and Innovation<br />
Document details<br />
Security Classification<br />
PUBLIC<br />
Date of review of security classification June 2012<br />
Authority<br />
Queensland State Archives<br />
Author<br />
Queensland State Archives<br />
Document Status<br />
Final<br />
Version Version 1.0<br />
Contact for enquiries<br />
All enquiries regarding this document should be directed in the first instance to:<br />
Digital Archives Unit<br />
Queensland State Archives<br />
07 3131 7777<br />
info@archives.qld.gov.au<br />
Copyright<br />
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: a guideline for Queensland public authorities<br />
Copyright © The State of Queensland (Department of Science, Information Technology, Innovation<br />
and the Arts) 2012<br />
Licence<br />
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: a guideline for Queensland public authorities by Queensland State<br />
Archives is licensed under a Creative Commons Attribution 3.0 Australia Licence. To view a copy<br />
of this licence, please visit http://creativecommons.org/licenses/by/3.0/au/.<br />
Information security<br />
This document has been security classified using the Queensland Government Information<br />
Security Classification Framework (QGISCF) as PUBLIC and will be managed according to the<br />
requirements of the QGISCF.<br />
Page 2 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Executive Summary<br />
This guideline provides advice to public authorities on the key recordkeeping issues to be<br />
considered when planning for and carrying out data migration projects where the business system<br />
contains public <strong>records</strong>. Moving public <strong>records</strong> from one hardware/software configuration to<br />
another, or from one generation of computer technology to a subsequent generation, requires<br />
explicit planning to ensure that the public <strong>records</strong> maintain their authenticity and integrity in<br />
accordance with the Public Records Act 2002. Digital source <strong>records</strong> that have been migrated<br />
cannot be disposed of unless they have met certain criteria as outlined under the General<br />
Retention and Disposal Schedule for Digital Source Records (QDAN678), and this guideline<br />
provides guidance to Queensland public authorities about the migration of <strong>digital</strong> <strong>records</strong> and the<br />
lawful disposal of the source <strong>records</strong>.<br />
The guideline assumes:<br />
<br />
<br />
familiarity with the requirements of the Public Records Act 2002 and a knowledge of the<br />
various supporting standards and guidelines, and<br />
that <strong>records</strong> management expertise is incorporated into the overall migration project<br />
planning and implementation.<br />
The guideline is organised into three parts:<br />
1. Introduction – outlines the context of migrating <strong>digital</strong> <strong>records</strong> as it is applied in the<br />
guideline.<br />
2. Fundamental issues when migrating public <strong>records</strong> – examines the challenge of evolving<br />
technology, drivers for migration, risks, obligations and the treatment of source <strong>records</strong><br />
after migration.<br />
3. Recordkeeping considerations for migration projects – outlines key recordkeeping<br />
considerations to be incorporated into the overall migration project, including various<br />
checklist tools included as appendices.<br />
This document forms part of a wider framework of <strong>digital</strong> continuity advice under development by<br />
Queensland State Archives to support public authorities in meeting their obligations in managing<br />
their <strong>digital</strong> public <strong>records</strong>.<br />
Page 3 of 57
Department of Science, Information Technology and Innovation<br />
Table of contents<br />
Executive Summary ...................................................................................................................... 3<br />
1. Introduction ............................................................................................................................ 5<br />
1.1 Purpose ............................................................................................................................ 5<br />
1.2 Audience .......................................................................................................................... 5<br />
1.3 Scope ............................................................................................................................... 5<br />
1.4 Authority .......................................................................................................................... 6<br />
1.5 Relationship to other Queensland State Archives’ publications ................................. 6<br />
1.6 Definitions ....................................................................................................................... 7<br />
1.7 Acknowledgements ......................................................................................................... 9<br />
2. Important considerations for migrating public <strong>records</strong> .................................................... 10<br />
2.1 Drivers for migration ..................................................................................................... 10<br />
2.2 Risks .............................................................................................................................. 11<br />
2.3 Obligations .................................................................................................................... 11<br />
2.4 Treatment of source <strong>records</strong> after migration .............................................................. 12<br />
3. Recordkeeping considerations for migration projects ...................................................... 14<br />
3.1 Migration project initiation ........................................................................................... 15<br />
3.2 Determining what is to be migrated ............................................................................. 17<br />
3.3 Determining where and how to migrate ....................................................................... 23<br />
3.4 Ensuring quality ............................................................................................................ 27<br />
3.5 Performing migration .................................................................................................... 30<br />
3.6 Post migration ............................................................................................................... 32<br />
Appendices ................................................................................................................................. 37<br />
Appendix A – IS40 Principle 7 and the migration of <strong>digital</strong> <strong>records</strong> ................................... 38<br />
Appendix B – Records manager’s project stages flowchart ............................................... 40<br />
Appendix C – Records manager’s migration project checklist ........................................... 42<br />
Appendix D – Recordkeeping controls in the target system ............................................... 53<br />
Appendix E – Testing and validation checklist ................................................................... 55<br />
Appendix F – Lessons learnt – hints and tips ...................................................................... 56<br />
Page 4 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
1. Introduction<br />
1.1 Purpose<br />
The purpose of this guideline is to assist public authorities to meet their recordkeeping obligations<br />
under the Public Records Act 2002 when they are planning and implementing data migration<br />
projects involving business systems that contain <strong>digital</strong> public <strong>records</strong>. The document also forms<br />
part of a wider framework of <strong>digital</strong> continuity advice developed by Queensland State Archives, to<br />
support public authorities in <strong>digital</strong> recordkeeping obligations. Specifically the guideline provides<br />
advice on:<br />
<br />
<br />
key recordkeeping considerations where the migrated version of the record will act as the<br />
official public record, and<br />
the correct disposal of the <strong>digital</strong> source <strong>records</strong> in accordance with the General Retention<br />
and Disposal Schedule for Digital Source Records (QDAN678).<br />
1.2 Audience<br />
The intended target audience for this guideline is information and <strong>records</strong> management staff<br />
responsible for advising on a public authority’s recordkeeping compliance and who will be working<br />
with:<br />
<br />
<br />
project team members, business analysts, business system owners and ICT colleagues,<br />
and<br />
other key stakeholders responsible for data and business system migration projects or<br />
activities.<br />
1.3 Scope<br />
This guideline applies to all Queensland public authorities, as defined in Schedule 2 of the Public<br />
Records Act 2002.<br />
The guideline provides support for the migration of <strong>digital</strong> public <strong>records</strong>, where the migration of the<br />
<strong>digital</strong> public <strong>records</strong> is from one hardware/software configuration to another or from one<br />
generation of information technology to a subsequent generation, for ongoing business and<br />
legislative purposes.<br />
This guideline:<br />
<br />
<br />
<br />
does not seek to provide all the technical advice on carrying out a data migration project<br />
does not replace the existing project management methodology adopted by the public<br />
authority, and<br />
does not cover the digitisation of <strong>records</strong> from a physical hardcopy format to a <strong>digital</strong> format<br />
such as when paper <strong>records</strong> are digitised. 1<br />
1<br />
For guidance about digitisation or microfilming of <strong>records</strong>, please see the advice on Digitisation and disposal<br />
Page 5 of 57
Department of Science, Information Technology and Innovation<br />
1.4 Authority<br />
The State Archivist has issued this guideline in accordance with section 25(1)(f) of the Public<br />
Records Act 2002, which enables the State Archivist to make policy, standards, and guidelines<br />
about the making, keeping, preserving, managing and disposing of public <strong>records</strong>.<br />
1.5 Relationship to other Queensland State Archives’ publications<br />
This guideline is specifically supported by:<br />
<br />
<br />
the QSA publication Metadata for <strong>digital</strong> continuity: a companion guideline to the<br />
Queensland recordkeeping metadata standard and the<br />
General Retention and Disposal Schedule for Digital Source Records (QDAN678).<br />
Metadata for <strong>digital</strong><br />
continuity<br />
supports<br />
<strong>Migrating</strong> <strong>digital</strong><br />
<strong>records</strong><br />
expands on<br />
Queensland<br />
Recordkeeping<br />
Metadata Standard<br />
(QRKMS)<br />
Guidelines and<br />
functional<br />
requirements for<br />
<strong>records</strong> in<br />
business systems<br />
supports<br />
supports<br />
General Retention<br />
and Disposal<br />
Schedule for Digital<br />
Source Records<br />
Figure 1 – Migration, metadata and retention and disposal connections<br />
It is also recommended that recordkeeping and information management staff have an<br />
understanding of the following documentation:<br />
<br />
<br />
<br />
Information standard 40: recordkeeping (IS40)<br />
Information standard 31: retention and disposal of public <strong>records</strong> (IS31)<br />
Related public <strong>records</strong> briefs:<br />
o<br />
o<br />
ICA: Guidelines and functional requirements for <strong>records</strong> in business systems<br />
Advice on the Destruction of Public Records<br />
o<br />
Decommissioning Business Systems<br />
It is assumed that the recordkeeping considerations detailed in this guideline will be incorporated<br />
into the public authority’s own project management methodology, for example the Queensland<br />
Government Project Management Methodology 2 .<br />
2<br />
Accessible at Queensland Government Project Management Methodologies<br />
Page 6 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
1.6 Definitions<br />
Records and information management specific terms are defined in Queensland State Archives’<br />
Glossary of Archival and Recordkeeping Terms available on Queensland State Archives’ website.<br />
Some specific terms as they are used in the context of migration processes in this guideline are<br />
described below.<br />
Migration<br />
There are many different definitions and understandings of the term ‘migration’ across disciplines.<br />
Within the context of this guideline, the term migration is used to encompass the process of<br />
preserving the authenticity of <strong>records</strong> when relocating <strong>digital</strong> <strong>records</strong> from one hardware/software<br />
configuration to another, or from one generation of computer technology to a subsequent<br />
generation. 3<br />
Source <strong>records</strong> and migrated <strong>records</strong><br />
‘Source record’ and ‘migrated record’ are terms used in this document. For clarity, ‘source record’<br />
refers to the record being migrated, and the ‘migrated record’ is the record resulting from the<br />
migration, as illustrated in Figure 2 below.<br />
Old<br />
home<br />
Relocation<br />
to<br />
New home<br />
Source<br />
system<br />
Target<br />
system<br />
Source<br />
<strong>records</strong><br />
Migration<br />
Migrated<br />
<strong>records</strong><br />
Figure 2 – Migration of source <strong>records</strong> to a new target location<br />
3<br />
Referred to by R. Harvey (2005), Preserving Digital Materials, K. G. Saur, Munich, page 148. Originally used in<br />
Preserving Digital Information: Report of the Task Force on Archiving of Digital Information, 1996. Available at:<br />
http://www.clir.org/pubs/reports/pub63watersgarrett.pdf<br />
Page 7 of 57
Department of Science, Information Technology and Innovation<br />
Data cleansing<br />
Data cleansing involves the removal of out-of-date, duplicated, inaccurate or redundant information<br />
and is a recommended practice prior to undertaking migration activities in order to minimise the<br />
relocation of unnecessary or inaccurate information. In the context of this document, data<br />
cleansing processes are applied to the review of the data/<strong>records</strong>/metadata in the source system in<br />
order to make decisions about what data/<strong>records</strong>/metadata will be migrated into the target system.<br />
It should be noted that data cleansing activities are likely to trigger the need for an authorised<br />
disposal of time-expired <strong>records</strong>. This means that any public <strong>records</strong> identified for deletion during<br />
data cleansing in the source system prior to the migration activity will need to be disposed of in<br />
accordance with an approved Retention and Disposal Schedule.<br />
Data mapping<br />
Data mapping refers to the process of establishing connections or relationships between the data<br />
elements of the source and target systems that will enable the migration of the <strong>records</strong> and their<br />
associated metadata, and contribute to the maintenance of the authenticity and integrity of the<br />
<strong>records</strong>.<br />
Deletion vs destruction vs disposal<br />
In this guideline, these three terms have distinct meanings and are described below:<br />
<br />
<br />
<br />
Deletion in this guideline is a generalised term referring to the erasing of <strong>digital</strong> information<br />
contained in a business system.<br />
Destruction refers to the process of eliminating or deleting <strong>records</strong> that do not have continuing<br />
value, beyond the point of any possible reconstruction.<br />
Disposal of <strong>records</strong> as described in the Public Records Act (2002) includes the actions of<br />
destroying, damaging, abandoning, transferring, selling, donating or giving away a record, or<br />
part of a record. In the context of this guideline, the term disposal has been used to refer to:<br />
o<br />
o<br />
the deletion of the source record, following migration, or<br />
the deletion of eligible time-expired <strong>records</strong> prior to migration, undertaken in<br />
accordance with the terms stated in an approved Retention and Disposal Schedule.<br />
Digital continuity<br />
Digital continuity used in the context of this guideline is the ability to ensure that despite<br />
organisational, business and technological change, the access, readability and the authenticity of<br />
<strong>digital</strong> <strong>records</strong> is maintained for as long as they are required to be kept for business, legal and/or<br />
archival requirements. Digital continuity involves more than preserving <strong>digital</strong> <strong>records</strong>, and requires<br />
a range of robust <strong>records</strong> and information management policies and practices.<br />
Queensland State Archives is responsible for developing a <strong>digital</strong> continuity strategy and tools for<br />
state government and this guideline is one of the tools developed to support the continuity of <strong>digital</strong><br />
public <strong>records</strong>.<br />
Page 8 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Encryption and decryption<br />
Encryption refers to a process where security coding is applied to data that converts it into a<br />
secured format for protected transmission across networks. To read the data the receiver must<br />
have the decryption key coding to restore the data to its original form.<br />
1.7 Acknowledgements<br />
A range of advice was referenced in the development of this guideline. Queensland State Archives<br />
in particular acknowledges State Records New South Wales’ migration advice in their Managing<br />
Digital Records Guideline.<br />
Queensland State Archives would like to thank the many public authorities that contributed to the<br />
development of the guideline and their representatives who also commented on the exposure draft.<br />
Page 9 of 57
Department of Science, Information Technology and Innovation<br />
2. Important considerations for migrating public <strong>records</strong><br />
Two important recordkeeping values need to be upheld when undertaking migration projects. They<br />
are:<br />
<br />
<br />
maintaining the authenticity, useability, integrity and reliability of public <strong>records</strong> for as long<br />
as they are required to be kept for business, legal and / or archival purposes, and<br />
ensuring the disposal of <strong>records</strong> is authorised and in accordance with an approved<br />
Retention and Disposal Schedule.<br />
These values have broad reaching implications for <strong>records</strong> migration projects, and the<br />
recordkeeping and information professional not only needs to have a grounding knowledge of<br />
these values but also needs to be able to translate them and have input into applying these values<br />
to the overall migration project.<br />
Before dealing with specific recordkeeping considerations in Section 3, this section will specify the:<br />
<br />
<br />
<br />
<br />
<br />
drivers for migration<br />
potential risks to <strong>records</strong><br />
recordkeeping obligations, and the<br />
treatment of source <strong>records</strong> after migration<br />
and will assist in setting the scene for a recordkeeping and information management<br />
professional’s input into a <strong>records</strong> migration project.<br />
2.1 Drivers for migration<br />
Digital <strong>records</strong> are dependent on technology for their access and use. Over a period of time, the<br />
migration of the <strong>records</strong> from one hardware/software configuration to another may be required for<br />
the continuing access, useability and ongoing management of the public <strong>records</strong> in accordance<br />
with business, legal and archival obligations.<br />
The need to migrate public <strong>records</strong> may be triggered by a range of drivers, for example:<br />
<br />
<br />
<br />
<br />
obsolescence and technological changes that require the upgrading or decommissioning of<br />
business systems<br />
changing business needs that lead to the adoption of a new business system<br />
system and storage rationalisation, e.g. as a result of the implementation of the Application<br />
Rationalisation Methodology (ARM), and / or<br />
machinery of government changes that require the transfer of functions or activities<br />
recorded in business systems from one public authority to another entity.<br />
Page 10 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Upgrading or decommissioning<br />
business systems<br />
Changes to business needs<br />
Machinery of<br />
government changes<br />
Bulk <strong>records</strong><br />
migration<br />
Figure 3 – Examples of drivers for migration<br />
System and storage<br />
rationalisation<br />
2.2 Risks<br />
The processes for the relocation of public <strong>records</strong> from one hardware/software configuration to<br />
new technological environment/s may threaten the legal authenticity and integrity of <strong>digital</strong> public<br />
<strong>records</strong>. To mitigate risks to the public <strong>records</strong>, migrations must be controlled by authorised and<br />
planned policies and procedures that incorporate recordkeeping considerations within the public<br />
authority’s overall project management methodology.<br />
Specifically the threats and risks to <strong>records</strong> during migration include:<br />
<br />
<br />
<br />
failure to maintain the authenticity and integrity of the <strong>records</strong> (refer to Information Standard<br />
40 for details)<br />
loss of readability and accessibility, particularly for <strong>records</strong> that are required to be kept for<br />
the long-term or in perpetuity, and<br />
non-compliance with legal obligations for the maintenance of the public <strong>records</strong>, for<br />
example through inadvertent loss or unauthorised deletion through failure to identify and<br />
manage the existence of <strong>digital</strong> public <strong>records</strong> when decommissioning a system.<br />
It is important that risk mitigation assessments and strategies relevant to recordkeeping be<br />
incorporated into planning and that ownership of risk is delegated, managed and documented. Risk<br />
mitigation strategies may include:<br />
<br />
<br />
<br />
identifying and appraising the nature of the <strong>digital</strong> <strong>records</strong> to be migrated in accordance<br />
with approved Retention and Disposal Schedules, to assist in determining the potential risk<br />
value<br />
determining and documenting the capacity of the target system to capture the required<br />
recordkeeping metadata to sustain the authenticity and integrity of the migrated <strong>records</strong><br />
adopting robust quality assurance planning and processes.<br />
Section 3 of this guideline will highlight the key recordkeeping considerations that need to be<br />
integrated into risk mitigation planning.<br />
2.3 Obligations<br />
The Queensland Government Recordkeeping Policy Framework sets out the recordkeeping<br />
obligations of public authorities. Public authorities must meet all legislative and regulatory<br />
requirements relating to the management of <strong>records</strong> during migration projects and activities.<br />
This includes compliance with:<br />
Page 11 of 57
Department of Science, Information Technology and Innovation<br />
<br />
<br />
Public Records Act 2002 – this Act is concerned with ensuring the public <strong>records</strong> of<br />
Queensland are made, managed, kept and, if appropriate, preserved in a useable form for<br />
the benefit of present and future generations. It mandates a number of obligations for<br />
Queensland public authorities which include making and keeping full and accurate <strong>records</strong><br />
of its activities, ensuring the safe custody and preservation of <strong>records</strong> in its possession, and<br />
who must authorise disposal.<br />
Information Standard 40: Recordkeeping – this standard outlines seven mandatory<br />
principles for compliance by public authorities. These principles are summarised in Table 1.<br />
Appendix A provides further guidance on the application of Principle 7 in the <strong>digital</strong> <strong>records</strong><br />
migration context.<br />
Principle<br />
Principle 1<br />
Principle 2<br />
Principle 3<br />
Principle 4<br />
Principle 5<br />
Principle 6<br />
Principle 7<br />
Requirement<br />
Public authority recordkeeping must be compliant and accountable<br />
Recordkeeping must be monitored and audited for compliance<br />
Recordkeeping activity must be assigned and implemented<br />
Recordkeeping must be managed<br />
Recordkeeping systems must be reliable and secure<br />
Recordkeeping must be systematic and comprehensive<br />
Full and accurate <strong>records</strong> must be made and kept for as long as they are<br />
required for business, legislative, accountability and cultural purposes.<br />
Table 1 – Mandatory principles of Information Standard 40: Recordkeeping<br />
<br />
Information Standard 31: Retention and Disposal of Public Records - the Public Records<br />
Act 2002 prohibits the disposal of public <strong>records</strong> without the permission of the State<br />
Archivist. The primary purpose of this Information Standard is to assist public authorities to<br />
lawfully retain and dispose of public <strong>records</strong>.<br />
This standard, among other obligations, requires that public authorities must:<br />
o<br />
o<br />
o<br />
o<br />
dispose of public <strong>records</strong> in accordance with a Retention and Disposal Schedule<br />
approved by the State Archivist that is current at the time of disposal<br />
ensure all disposal is endorsed by the Chief Executive or an authorised delegate<br />
ensure the method of destruction of public <strong>records</strong> is appropriate to the sensitivity of<br />
the <strong>records</strong> and conforms with local environment regulations<br />
document the disposal of public <strong>records</strong>.<br />
2.4 Treatment of source <strong>records</strong> after migration<br />
Decisions about the treatment of the source <strong>records</strong> after their successful migration need to be<br />
incorporated into the migration project plan. Deletion of source <strong>records</strong> needs to be in accordance<br />
with the General Retention and Disposal Schedule for Digital Source Records (QDAN678), which<br />
addresses the treatment of the remaining source <strong>records</strong> after they have been migrated and<br />
enables lawful disposal.<br />
Section 3.6.1 and 3.6.2 of this guideline provide further advice about the application of retention<br />
and disposal obligations within the <strong>records</strong> migration project stages.<br />
Page 12 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Key points:<br />
The processes of moving <strong>records</strong> from one system to a new technological environment can<br />
threaten the authenticity, integrity, reliability and useability of <strong>digital</strong> <strong>records</strong>.<br />
To mitigate risks, migrations must be controlled by authorised and planned policies, processes<br />
and procedures that are designed to incorporate key recordkeeping standards.<br />
Source <strong>records</strong> are not automatically eligible for deletion. Certain obligations outlined in this<br />
guideline and the General Retention and Disposal Schedule for Digital Source Records<br />
(QDAN678) need to have been met before deletion can be authorised to be undertaken.<br />
Page 13 of 57
Department of Science, Information Technology and Innovation<br />
3. Recordkeeping considerations for migration projects<br />
This section outlines the recordkeeping considerations that need to be included in planning<br />
migration projects. The structure of this section follows a generic migration project methodology,<br />
however, it is acknowledged that public authorities will not necessarily all use the same<br />
methodology for migration projects. A public authority’s approach can be based on this document<br />
but will need to be adapted to the specific project management methodology adopted by the public<br />
authority.<br />
Table 2 below summarises the considerations that are detailed further in this section. A more<br />
readily useable version of the information in this table has been prepared as a flowchart, which is<br />
available at Appendix B. Additionally, the flowchart stages have been expanded into a checklist for<br />
<strong>records</strong> managers which provides, in essence, a summary of the material in the rest of this section.<br />
The checklist is available at Appendix C.<br />
Records<br />
migration<br />
stages<br />
1<br />
Migration<br />
project<br />
initiation<br />
2<br />
Determining<br />
what is to be<br />
migrated<br />
3<br />
Determining<br />
where &<br />
how to<br />
migrate<br />
4<br />
Ensuring<br />
quality<br />
5<br />
Performing<br />
migration<br />
6<br />
Post migration<br />
Description<br />
Recordkeeping considerations<br />
The initial stage<br />
of high-level<br />
scheduling and<br />
resource<br />
planning for the<br />
project.<br />
Project<br />
planning<br />
Roles and<br />
responsibilitie<br />
s<br />
Stakeholder<br />
communicatio<br />
ns<br />
The stage of the<br />
project when the<br />
scope of the<br />
migration is<br />
being<br />
established.<br />
Identifying the<br />
<strong>records</strong><br />
Establishing<br />
retention<br />
obligations<br />
Determining<br />
recordkeeping<br />
metadata<br />
Understanding<br />
characteristics<br />
of the <strong>records</strong><br />
The stage<br />
when<br />
decisions are<br />
being explored<br />
and made<br />
about the<br />
approach,<br />
method and<br />
target<br />
system(s) for<br />
the migration.<br />
Verifying<br />
target<br />
system/s<br />
recordkeepin<br />
g capabilities<br />
Making file /<br />
record format<br />
decisions<br />
Establishing<br />
migration<br />
approach<br />
The stage<br />
when quality<br />
assurance and<br />
testing and<br />
validation<br />
processes are<br />
identified,<br />
planned and<br />
undertaken.<br />
Preparing<br />
<strong>records</strong> for<br />
migration<br />
Testing and<br />
validation<br />
The stage<br />
concerned with<br />
the actual<br />
undertaking of<br />
the migration to<br />
extract/export,<br />
transfer and<br />
load/import the<br />
<strong>records</strong> into the<br />
target system(s).<br />
Testing roll<br />
back strategy<br />
Recordkeeping<br />
during cut over<br />
period<br />
Migration of<br />
<strong>records</strong><br />
Quality<br />
assuring the<br />
results of the<br />
migration<br />
The final<br />
recordkeeping<br />
considerations<br />
prior to project<br />
closure.<br />
Verifying<br />
conditions for<br />
disposal of<br />
source <strong>records</strong><br />
Disposal of<br />
<strong>digital</strong> ‘source<br />
<strong>records</strong>’<br />
Capture<br />
mandatory<br />
recordkeeping<br />
documentation<br />
Stakeholder<br />
relations and<br />
continuous<br />
improvement<br />
Table 2 – Project stages for recordkeeping considerations<br />
Page 14 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
3.1 Migration project initiation<br />
Migration<br />
project<br />
initiation<br />
Determining<br />
what is to be<br />
migrated<br />
Determining<br />
where & how<br />
to migrate<br />
Ensuring<br />
quality<br />
Performing<br />
migration<br />
Post<br />
migration<br />
3.1.1 Project planning<br />
A migration project that is systematically planned, initiated and carried out to well-defined<br />
procedures will help protect <strong>records</strong> and is required to ensure their evidential integrity.<br />
This guideline assumes that appropriate processes and documentation will be established to<br />
initiate a migration project by the broader organisational project team. This would include project<br />
planning documentation which clearly describes the drivers, approaches, legislative and policy<br />
requirements and expected outcomes for the project. It also assumes that appropriate change<br />
management activities are planned. It therefore does not address these planning issues in detail<br />
and assumes that the <strong>records</strong> manager will engage with the broader project.<br />
During the planning process it is important to ensure that:<br />
<br />
<br />
<br />
there is recorded acknowledgement within plans of the need to meet a public authority’s<br />
recordkeeping obligations throughout the project<br />
risk mitigation planning is incorporated that is commensurate with the level of value<br />
associated with the <strong>records</strong>, and<br />
there is appropriate scheduling of time and resources for the recordkeeping activities<br />
associated with the migration of <strong>digital</strong> public <strong>records</strong>.<br />
Key point:<br />
Ensure recordkeeping obligations, standards and activities are well understood by the project<br />
team and are sufficiently incorporated into the planning and scheduling for the project.<br />
3.1.2 Roles and responsibilities<br />
As outlined in section 2.2 migrations can be a high risk activity for <strong>records</strong>. Those responsible for<br />
the management of <strong>records</strong> must be involved in all <strong>records</strong> migration projects. This includes<br />
projects to migrate <strong>records</strong> stored in any business system, not just for projects that involve<br />
dedicated recordkeeping systems such as electronic document and <strong>records</strong> management systems<br />
(eDRMS). 4<br />
When migration projects involving <strong>records</strong> are established within a public authority, and roles and<br />
responsibilities are planned, <strong>records</strong> and information managers are well placed to contribute<br />
expertise to the project through:<br />
4<br />
In this guideline business systems refer to automated systems that have a primary purpose of facilitating business<br />
transactions (e.g. finance, licensing, client-relationship systems, etc.) and that do not necessarily contain <strong>records</strong><br />
management functionality. This is distinct from electronic document and <strong>records</strong> management systems (eDRMS) that are<br />
designed with <strong>records</strong> management functionality to manage the <strong>records</strong> it captures throughout their lifecycle.<br />
Page 15 of 57
Department of Science, Information Technology and Innovation<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
assisting with the identification and appraisal of the <strong>records</strong> within business systems, how<br />
long the <strong>records</strong> must be legally retained and which <strong>records</strong> are eligible for lawful disposal<br />
determining how the <strong>records</strong> are being managed within the existing system and their<br />
current and ongoing recordkeeping compliance needs<br />
facilitating the process for the authorised disposal of <strong>records</strong><br />
supporting quality assurance of migrated <strong>records</strong> in collaboration with business area<br />
experts and others<br />
assisting in the data mapping process between the source and target systems to ensure<br />
<strong>records</strong>, including their metadata, remain full and accurate <strong>records</strong> of business and that<br />
their authenticity can be verified<br />
advising on the recordkeeping controls needed in the design and configuration of the target<br />
system<br />
developing and delivering training and awareness materials and sessions related to <strong>records</strong><br />
management, and<br />
advising on and promoting continuous improvement in recordkeeping functionality within<br />
new systems.<br />
3.1.3 Stakeholder communication planning<br />
As mentioned above, this guideline assumes that appropriate change management activities are<br />
planned as part of the overall project management plan. However it is important that those<br />
responsible for <strong>records</strong> and information management engage with the overall communications<br />
strategy and the proposed timings for the migration of the <strong>records</strong>. This is needed in order to<br />
understand and plan for the potential impact on recordkeeping requirements, and on the business<br />
and its stakeholders, during the time of the <strong>records</strong> changeover period between the source system<br />
and the target system. (Please note: Appendix F provides some real-life insights into<br />
recordkeeping considerations during this time. Refer to F.3.)<br />
There will be a need to ensure that appropriate training of staff is undertaken. This may require<br />
targeted promotion and the development of new procedures relating to the management of the<br />
<strong>records</strong> in the newly migrated environment.<br />
Key points:<br />
Those with responsibility for <strong>records</strong> management should be engaged and consulted in migration<br />
projects to ensure that recordkeeping considerations are integrated into the overall project<br />
management planning.<br />
The key objective of <strong>records</strong>/information managers being involved is to support the achievement<br />
of maintaining the authenticity and evidential integrity of the public <strong>records</strong> throughout the<br />
migration process and to assist with retention and disposal compliance.<br />
Page 16 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
3.2 Determining what is to be migrated<br />
Migration<br />
project<br />
initiation<br />
Determining<br />
what is to be<br />
migrated<br />
Determining<br />
where & how<br />
to migrate<br />
Ensuring<br />
quality<br />
Performing<br />
migration<br />
Post<br />
migration<br />
A key decision point in a migration project is determining what information will be migrated. To<br />
determine this, there are some fundamental details about the <strong>records</strong> themselves that must first be<br />
established. Understanding these details will help to inform decisions about what data to migrate<br />
and what data does not need to be migrated. The processes for determining this are described<br />
below.<br />
3.2.1 Identifying the <strong>records</strong><br />
Records are the consequence or product of a business activity. Digital <strong>records</strong> are not simply data<br />
but comprise a complex set of related information which includes the:<br />
<br />
<br />
<br />
content of the record<br />
contextual metadata, for example, information about who created the record, how it has<br />
been managed, relationships to other <strong>records</strong>, etc, and the<br />
structure that enables the record to be accessible and meaningful, e.g. database table<br />
headings.<br />
Knowing what the <strong>records</strong> are within a system is critical to protecting and maintaining their<br />
completeness, integrity and authenticity. It is important to clearly scope and document them prior to<br />
undertaking a migration.<br />
Depending on the nature of the migration project, the <strong>records</strong> to be migrated may be easily<br />
identified and can be clearly described. However, identifying <strong>records</strong> in business systems can often<br />
be challenging as they are generally complex and are not always packaged as neatly discrete<br />
objects.<br />
The process of determining what <strong>records</strong> are contained in business systems involves undertaking<br />
an analysis of the business processes associated with the system’s use, in order to identify the<br />
information that <strong>records</strong> the evidence of business activity and transactions. For further advice on<br />
this process see Part 2.3 of the Guidelines and Functional Requirements for Records in Business<br />
Systems. 5<br />
This process of analysis will also highlight ephemera within the system, including any duplicated<br />
copies of information. Ephemera are items of short-term value that are not required to be kept as<br />
<strong>records</strong> and therefore may be able to be disposed of, if not required for other business or legal<br />
purposes, prior to the migration.<br />
5<br />
The use of this guideline has been endorsed by the State Archivist and is accessible from Queensland State Archives’<br />
website. Also refer to the public <strong>records</strong> brief Managing <strong>records</strong> in <strong>digital</strong> environments using ISO 16175:<br />
Information and documentation - Principles and functional requirements for <strong>records</strong> in electronic office environments.<br />
Please note that in 2008 the International Council on Archives (ICA) published three modules of its Principles and<br />
Functional Requirements for Records in Electronic Environment (ICA-Req). These modules have since been adopted by<br />
the International Standards Organisation and published as ISO 16175.<br />
Page 17 of 57
Department of Science, Information Technology and Innovation<br />
Key point:<br />
Know what <strong>records</strong> there are and describe them to ensure the <strong>records</strong> can be successfully<br />
maintained throughout and following the migration process.<br />
3.2.2 Establishing retention obligations<br />
The retention periods for <strong>records</strong> are documented in authorised Retention and Disposal<br />
Schedules. Once the <strong>records</strong> have been identified, an authorised schedule can be consulted and<br />
applied to determine how long they must be kept. Some <strong>records</strong> may have passed their retention<br />
period and can be deleted, which will reduce the volume of <strong>records</strong> requiring migration.<br />
IMPORTANT: The deletion of <strong>records</strong> must be in accordance with the authorised disposal<br />
requirements. It is also important to consider other aspects that may apply to the <strong>records</strong> prior to<br />
deletion, such as whether a disposal freeze is in place or another business or legislative reason<br />
overrides the Retention and Disposal Schedule/s. For example, are the <strong>records</strong> in question<br />
potentially needed in evidence in an existing judicial proceeding, including any reasonably<br />
possible judicial proceeding?<br />
When deletion is to be undertaken in the source system prior to migration, all disposal actions must<br />
be documented in accordance with requirements under Principle 2 of Information standard 31:<br />
retention and disposal of public <strong>records</strong>. For further information about disposing of <strong>records</strong>, see<br />
section 4 of the Guideline for the implementation of retention and disposal schedules.<br />
The disposal of public <strong>records</strong> must be authorised by the State Archivist under section 26 of the<br />
Public Records Act 2002. Such authorisation is usually given in a Retention and Disposal<br />
Schedule. In the event the identified <strong>records</strong> are not covered by an authorised Retention and<br />
Disposal Schedule, <strong>records</strong> cannot be deleted prior to migration.<br />
Key points:<br />
Identify the lawful retention period for the <strong>records</strong> through consultation with authorised Retention<br />
and Disposal Schedule/s and consider any other legal obligations e.g. <strong>records</strong> subject to litigation,<br />
discovery process.<br />
After establishing the lawful retention period, if there are no longer any legal, business or<br />
legislative requirements to retain the <strong>records</strong>, undertake an authorised process to dispose of any<br />
<strong>records</strong> that have met their minimum retention period.<br />
If there is no authorised Retention and Disposal Schedule covering the migrated <strong>records</strong>,<br />
disposing of them is unlawful.<br />
3.2.3 Determining recordkeeping metadata<br />
Recordkeeping metadata is a fundamental tool for creating and maintaining ‘full and accurate’<br />
<strong>records</strong> and is a key means by which the integrity and authenticity of a record can be proven.<br />
A key feature that differentiates recordkeeping metadata from other types of metadata is that it is<br />
not a static profile of a document or other resource. Recordkeeping metadata initially defines a<br />
record at the point of capture, but is also dynamic and accrues through time, to provide<br />
information on how a record has been used or managed. This characteristic of recordkeeping<br />
metadata is essential for preserving the authenticity of <strong>records</strong>.<br />
Page 18 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Safeguarding metadata relationships<br />
Effective migration of <strong>records</strong> requires that the associated metadata including the connections<br />
between a record and its metadata are persistently maintained. This requires an understanding of<br />
the <strong>records</strong>’ existing metadata. Documenting this metadata will help to highlight all the necessary<br />
data that must be migrated, particularly metadata which <strong>records</strong> the information about relationships<br />
and connections between data.<br />
Relationships requiring safeguarding may be:<br />
<br />
<br />
<br />
<br />
<br />
structural – that is within the document, such as a document containing a linked<br />
spreadsheet or images<br />
between <strong>records</strong> – e.g. between <strong>records</strong> documenting related aspects of business<br />
between <strong>records</strong> and/or ‘containers’ – e.g. documents aggregated to files/folders<br />
between <strong>records</strong> and other entities – e.g. between <strong>records</strong> and creating agents<br />
between <strong>records</strong> and control tools such as Business Classification Schemes, Retention and<br />
Disposal Schedules, access and security controls and mandates.<br />
Some of this information may be outside the system in question, for example, the Business<br />
Classification Scheme, however processes will need to be established to ensure the relationships<br />
are successfully maintained with the migrated <strong>records</strong>.<br />
Once the metadata and <strong>records</strong> are understood and documented, the process of mapping the<br />
metadata from source to destination system is essential to ensure that all the necessary metadata<br />
is carried over to the target system and remains useable and linked to the <strong>records</strong> it describes.<br />
This will contribute significantly to the maintenance of the <strong>records</strong>’ context and authenticity.<br />
Metadata and mapping analysis is an opportunity to address any insufficiencies and/or apply<br />
improved metadata capture that ensures that the minimum recordkeeping metadata requirements<br />
are met.<br />
In some migrations the date of the migration can become the default date for all recordkeeping<br />
actions, including date of creation of <strong>records</strong>. This severely impacts recordkeeping processes such<br />
as disposal and undermines the authenticity, useability and integrity of the <strong>records</strong>. Checks should<br />
therefore be in place to ensure dates that trigger recordkeeping actions in the source system<br />
remain accurate and unchanged post migration.<br />
The Queensland recordkeeping metadata standard and guideline (QRKMS) can be used as a tool<br />
to help identify the metadata that requires migration, and the links that must be maintained.<br />
Page 19 of 57
Department of Science, Information Technology and Innovation<br />
For example<br />
The table below contains columns to identify the QRKMS required metadata, the relevant field in<br />
the current system and the target field in the new system. The comments column can be used to<br />
note any system limitations or issues that require resolution before migration.<br />
QRKMS element Current system Target system Comments<br />
Identifier<br />
Transaction<br />
reference<br />
Transaction ID<br />
Check if ID format is supported by<br />
new system, or whether new<br />
format is required<br />
Date created Transaction date Date Must ensure dates carry across<br />
as is from old system and are not<br />
updated<br />
A metadata implementation matrix is available that can be adapted for this purpose.<br />
Capture metadata about migration<br />
When migration of <strong>digital</strong> <strong>records</strong> occurs, it is a key event in the lifecycle of the <strong>records</strong>. Metadata<br />
must be captured to document the process of the migration of the record. The mandatory ‘Record<br />
Event History’ element within the Queensland Recordkeeping Metadata Standard requires<br />
documenting the preservation, retrieval, disposal, control, access or use related activities<br />
performed on a record. This metadata provides a visible and auditable trail of <strong>records</strong> management<br />
actions and decisions, and can be used to record the migration activities on the record.<br />
Other metadata considerations<br />
In addition to specific recordkeeping metadata mentioned above, the system may hold other<br />
metadata that is crucial to the meaning of the <strong>records</strong>. For example, a database containing<br />
demographic data will need information that explains that a “1” in the gender column means the<br />
individual is a man, while a “2” indicates a woman. It is important that all metadata dependencies<br />
are identified when preparing for migration.<br />
It is acknowledged that the export of metadata from systems can be technically challenging. It is<br />
important that the time and effort required to undertake this step is not underestimated. Where it is<br />
not possible to export particular metadata, the effects on the functionality and authenticity of the<br />
<strong>records</strong> must be carefully analysed and documented.<br />
For further advice about recording metadata to manage <strong>digital</strong> <strong>records</strong> over time, see the Metadata<br />
for <strong>digital</strong> continuity: a companion guideline to the Queensland recordkeeping metadata standard.<br />
Key points:<br />
Analyse and document the recordkeeping metadata that exists about the <strong>records</strong>.<br />
Plan for the mandatory capture of metadata about the migration activities to be undertaken on the<br />
record in the line with Queensland Recordkeeping Metadata Standard.<br />
Take the opportunity to improve the capture of metadata about the <strong>records</strong>, if appropriate, by<br />
influencing the design of the target system.<br />
Page 20 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
3.2.4 Understanding characteristics of the <strong>records</strong><br />
Continued access to a record requires preservation of its content and other information and<br />
characteristics if the record is to remain authentic, meaningful and useable.<br />
All migrations carry risk related to the loss of these characteristics, information and content. One of<br />
the aims in any migration project is to minimise the amount of loss. An ideal approach may be to<br />
preserve all the <strong>records</strong>’ characteristics where possible, in order to maximise future access and<br />
use opportunities. However it may not be technically or economically feasible to achieve this, nor<br />
may it be necessary. For example, a new format may not technically be able to support all features<br />
of the preceding format but retains sufficient characteristics for business purposes.<br />
Defining the characteristics of the <strong>records</strong> needs to be driven by business requirements in<br />
consideration of obligatory recordkeeping requirements. This process includes consultation with<br />
the business stakeholders and considering the access and use requirements that current and<br />
potential future users of the <strong>records</strong> may have. Characteristics that enable these requirements can<br />
then be identified and migration decisions made to ensure their maintenance.<br />
Examples<br />
<br />
<br />
If a report contains a map where colours are used to signify different agricultural areas, these<br />
colours have meaning and the report could not be interpreted accurately if these colours were<br />
lost through migration to another format. For this report, the colours are a critical characteristic<br />
and any migration performed on this report must ensure that they are maintained.<br />
An experiment in the United States sought to determine whether complex three-dimensional,<br />
geometric CAD (Computer Aided Design) <strong>records</strong> of high tolerance machined piece parts<br />
could be converted from their native CAD environment to an open source archival format to<br />
facilitate their long-term preservation. The experiment showed that, at the time of the study, it<br />
was not possible to successfully convert all aspects of the <strong>records</strong> to the non-proprietary<br />
format. The open format could not adequately represent the fine accuracy and measurement<br />
levels (down to a millionth of an inch) that were necessary to sustain the accuracy of the<br />
engineering drawing. Because the exacting engineering requirements that were a core<br />
characteristic of the <strong>records</strong> could not be reproduced, a different conversion option was<br />
required. 6<br />
Characteristics may encompass:<br />
<br />
<br />
<br />
Content – information content within the record, e.g. text, still and moving images, audio,<br />
and other intellectual productions.<br />
Context - information that describes the environment in which the content was created and<br />
managed, e.g. creator name, date of creation, file format.<br />
Structure - information that describes the extrinsic or intrinsic relationship between content,<br />
as required to reconstruct the performance of the record, e.g. e-mails and their<br />
attachments.<br />
6<br />
This case study is discussed in L Duranti and R Preston, International research on permanent authentic <strong>records</strong> in<br />
electronic systems (InterPARES) 2: Experiential, interactive and dynamic <strong>records</strong>, 2008, pp. 34-35, Accessible from<br />
http://www.interpares.org/display_file.cfm?doc=ip2_book_complete.pdf<br />
Page 21 of 57
Department of Science, Information Technology and Innovation<br />
<br />
<br />
Appearance/rendering - any information that contributes to the re-creation of the<br />
performance of the record, e.g. font type, colour and size, bit depth.<br />
Behaviour - properties that indicate the method in which content interacts with other stimuli,<br />
e.g. hyperlinks. 7<br />
For example, <strong>records</strong> in business systems may possess the following characteristics: 8<br />
Content the actual content of the database tables<br />
queries used so the required content can be retrieved and represented<br />
Context<br />
metadata identifying:<br />
the name of the creator and creating organisation<br />
the business process(es) that produced the record<br />
date of creation of the record<br />
specified relationships to other <strong>records</strong><br />
original and current file formats<br />
name and version of the query languages that are used<br />
history of the recordkeeping actions performed on the record<br />
Structure the physical structure of the database. This includes:<br />
<br />
<br />
- the tables<br />
- relationships between the tables, including the constraints<br />
- the views<br />
- field attributes<br />
the structural composition of the data as presented onscreen<br />
the logical structure of the database can be preserved in an entity<br />
relationship diagram or an XML schema<br />
Appearance/ rendering the onscreen representation<br />
Behaviour the behaviour of the user application. This may be preserved in the form<br />
of descriptions of system-supported functions, as well as screenshots of<br />
the displays used for entering and amending data, generating reports,<br />
etc. It may also be preserved in system documentation and user<br />
manuals.<br />
Table 3 – Characteristics of <strong>records</strong> in business systems<br />
Where it is not possible to maintain all of the characteristics, a risk analysis of the characteristics<br />
which will and will not be preserved should be undertaken and justified, and the decisions made,<br />
approved and documented. At a minimum, characteristics should be prioritised for preservation by<br />
their capacity to ensure the authenticity, accessibility, usability and meaningfulness of the record<br />
as evidence of business activity.<br />
7<br />
InSPECT Project (Dec 2009) Investigating the significant properties of electronic content over time: Final report.<br />
Accessible from http://www.significantproperties.org.uk/inspect-finalreport.pdf<br />
8<br />
This example has been taken from the Digital Preservation Testbed Project led by the National Archives of the<br />
Netherlands. It should be noted that these characteristics are generic and are based on a specific file format type. There<br />
will be other specific characteristics unique to an agency’s business that will need to be identified and managed for<br />
databases.<br />
Page 22 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Key points:<br />
Assess, document and prioritise the characteristics of <strong>records</strong>.<br />
Consider the characteristics from different business perspectives and usage requirements.<br />
Ensure decisions about acceptable loss and/or change of record characteristics between the<br />
source and target systems are documented, and the consequences for useability, reliability, and<br />
authenticity of the migrated <strong>records</strong> are understood and accepted.<br />
3.3 Determining where and how to migrate<br />
Migration<br />
project<br />
initiation<br />
Determining<br />
what is to be<br />
migrated<br />
Determining<br />
where & how<br />
to migrate<br />
Ensuring<br />
quality<br />
Performing<br />
migration<br />
Post<br />
migration<br />
Following the determination of what requires migration to the target system, the following may<br />
require consideration:<br />
<br />
<br />
<br />
the target system’s recordkeeping options and capabilities<br />
the format into which the <strong>records</strong> will be migrated, and<br />
the overall approach, method and tools required to achieve the migration.<br />
3.3.1 Verifying target system/s recordkeeping capabilities<br />
Options for migration of inactive <strong>records</strong><br />
A key consideration in determining where the <strong>records</strong> will go and how they will be treated is<br />
whether the <strong>records</strong> are required for ongoing business purposes (active <strong>records</strong>) or can be less<br />
immediately accessible (inactive <strong>records</strong>).<br />
This guideline assumes that all active <strong>records</strong> are migrated to an online target system and that<br />
different migration options can be applied to the inactive <strong>records</strong>. The table below contains some<br />
examples with the related issues and outcomes:<br />
Source system Migration options Issues or outcomes<br />
Active <strong>records</strong><br />
Migrate to the new<br />
online target system<br />
<br />
<br />
Maintain business continuity<br />
Preservation monitoring is less of an overhead as<br />
<strong>records</strong> are in the current online environment<br />
Inactive <strong>records</strong><br />
1. Convert and<br />
move to a data<br />
store, repository<br />
or removable<br />
media<br />
<br />
<br />
May be most efficient and sustainable way of<br />
managing inactive <strong>records</strong> that must be retained for<br />
longer period<br />
Requires a conversion process on the <strong>records</strong> into an<br />
accessible format that is not dependent on the<br />
originating software<br />
Page 23 of 57
Department of Science, Information Technology and Innovation<br />
Source system Migration options Issues or outcomes<br />
2. Retain in the<br />
existing and now<br />
off-line system<br />
<br />
<br />
<br />
<br />
<br />
<br />
Business system functionality will not remain<br />
Requires active monitoring of the hardware, media and<br />
file formats for potential obsolescence<br />
Requires active monitoring and management in<br />
compliance with recordkeeping requirements under<br />
IS40 and IS31<br />
Higher risk of loss of accessibility due to technology<br />
obsolescence<br />
Inability to decommission the existing system which<br />
carries associated ongoing costs<br />
Requires ongoing maintenance and monitoring of the<br />
obsolete business system from the hardware, software<br />
and file format points of view<br />
<br />
Requires active monitoring and management in<br />
compliance with recordkeeping requirements under<br />
IS40 and IS31<br />
Table 4 – Inactive <strong>records</strong> migration options<br />
The chosen approach will be influenced by a range of business, recordkeeping and technical<br />
issues. An important initial step to undertake is to understand how the <strong>records</strong> are controlled in the<br />
current environment, for example whether recordkeeping controls are built into the system itself, or<br />
whether the <strong>records</strong> are exported and managed through a separate recordkeeping system.<br />
Ability of target system to meet recordkeeping obligations<br />
Whichever target system is selected, it is important to ensure there are adequate recordkeeping<br />
controls to manage the <strong>records</strong> in line with Information Standards 31 and 40. Refer to Appendix D<br />
for a checklist on the key obligations. The Guidelines and functional requirements for <strong>records</strong> in<br />
business systems provides additional guidance.<br />
Retention periods influencing target system decisions<br />
The retention and disposal obligations attached to the <strong>records</strong> may influence decisions about the<br />
migration approach. For inactive <strong>records</strong> that only need to be kept for relatively short periods of<br />
time beyond the planned migration, transfer into a target online business system and subsequent<br />
disposal of them once their retention period is reached may be the most appropriate choice, as<br />
migrating or storing them elsewhere may not be as cost-effective.<br />
Retention in the old system may also be appropriate, provided there is a commitment to sentence<br />
and dispose of these <strong>records</strong> when they are lawfully able to be disposed and to keeping the legacy<br />
system alive so the <strong>records</strong> remain findable and actionable until they can be disposed of.<br />
However leaving <strong>records</strong> in legacy systems for extended periods without active management to<br />
ensure they remain reliable, useable and authentic will not meet a public authority’s obligations<br />
under the Public Records Act 2002, nor necessarily comply with other legislation such as the Right<br />
to Information Act 2009.<br />
Page 24 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
3.3.2 Making file / record format decisions<br />
Repeated migrations can compromise the integrity, reliability, useability and authenticity of <strong>records</strong>,<br />
particularly those that are required to be retained for long periods of time. To minimise the number<br />
of migrations, conversion of those <strong>records</strong> that are no longer needed for active business purposes<br />
and have lengthy retention periods into an open file format which is not dependent on the<br />
originating software is recommended. 9 Open formats are stable, longer-term formats that are<br />
widely used. They are usually non-proprietary, platform-independent formats with freely available<br />
specifications. 10 The QSA Public Records Brief Advice on choosing file formats for the<br />
management of <strong>digital</strong> <strong>records</strong> contains more detailed information about choosing file formats.<br />
Some examples of open standard formats include:<br />
PDF/A<br />
ODF<br />
HTML<br />
XHTML<br />
JPEG<br />
TIFF<br />
PNG<br />
Portable document format/ archive<br />
OpenDocument Format<br />
Hypertext Markup Language<br />
Extensible Hypertext Markup Language<br />
Joint Photographic Experts Group<br />
Tagged Image File Format<br />
Portable Network Graphics<br />
FLAC<br />
Free Lossless Audio Codec<br />
Table 5 – Examples of open standard formats<br />
This approach still requires an active and ongoing commitment to monitoring and managing these<br />
migrated <strong>records</strong> to ensure their ongoing access.<br />
Appendix F provides an example of a record format conversion decision made by a public authority<br />
when it was required to decommission a business system.<br />
Key points:<br />
Ensure that the target system/s have recordkeeping capabilities and/or will meet the<br />
responsibilities for retention management.<br />
Plan to minimise the need for record migrations by determining a target system/s and record<br />
format(s) that suit the <strong>records</strong>’ retention periods and record types.<br />
Understanding the retention requirements, conversion formats and their associated stability also<br />
involves an active and ongoing commitment to their monitoring and maintenance.<br />
9<br />
It is acknowledged conversion is not the only longer-term preservation approach. There are other techniques such as<br />
emulation that can be applied.<br />
10<br />
State Records New South Wales (2009) Managing Digital Records Guideline: Effectively Manage the Migration of your<br />
Digital Records.<br />
Page 25 of 57
Department of Science, Information Technology and Innovation<br />
3.3.3 Establishing a migration approach<br />
There may be a number of approaches required to successfully migrate <strong>records</strong>. 11 After<br />
determining what is to be migrated as described in section 3.2 and considering the points raised<br />
above, a documented and approved plan needs to be devised which comprehensively outlines the<br />
approaches to be taken for all of the information in the business system.<br />
The plan should describe:<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
both the source and target business system<br />
formats for the information/<strong>records</strong><br />
the tools that will be used to achieve the migration<br />
details of any manual intervention activities<br />
security requirements for the information<br />
business rules and quality assurance procedures, and<br />
roll back strategy for mitigating risks to the <strong>records</strong> if affected by an unforeseen migration<br />
event.<br />
The migration approaches should be appropriately selected based on defined requirements,<br />
informed by the analysis of the <strong>records</strong> and recordkeeping obligations.<br />
Technological tools - Migrations may be facilitated by technological tools for extracting the data<br />
from the existing system, transforming the data and loading the data into the target system. For<br />
example the conversion of <strong>records</strong> to stable open standard formats can be achieved through a<br />
range of freely available tools such as the National Archives of Australia’s Xena. 12<br />
Manual intervention - Migration processes are likely to involve some manual intervention<br />
supported by appropriate business rules and quality assurance procedures. For example, if the<br />
mapping of information from the existing system to the target system proved complex, manual<br />
data entry of some of the information into the new system may be an appropriate approach when<br />
it is supported by appropriate checking processes to ensure the accuracy and completeness of<br />
the entered data.<br />
Key points:<br />
Document and secure approval for the plans that describe what will happen to all of the<br />
information within the system including the corresponding security classifications.<br />
Select tools which can best facilitate the maintenance of full and accurate <strong>records</strong> throughout the<br />
migration process.<br />
11<br />
Refer to Appendix F for examples of approaches taken by some public authorities.<br />
12<br />
Xena is free and open source software developed by the National Archives of Australia to converts <strong>records</strong> to an XMLbased<br />
archival data format. Xena is an acronym meaning Xml Electronic Normalising for Archives. See<br />
http://xena.sourceforge.net/ for further information.<br />
Page 26 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Ensure migration approaches, including any transforming of the data, loading and any reliance<br />
on manual extraction, is supported by clear business rules and quality checking processes.<br />
Ensure the approach includes a tested roll back strategy, as part of risk mitigation planning.<br />
3.4 Ensuring quality<br />
Migration<br />
project<br />
initiation<br />
Determining<br />
what is to be<br />
migrated<br />
Determining<br />
where & how<br />
to migrate<br />
Ensuring<br />
quality<br />
Performing<br />
migration<br />
Post<br />
migration<br />
For a record to retain its evidentiary value, it must be possible to demonstrate that migration has<br />
resulted in a full and accurate reproduction of the source record which has not been altered in an<br />
accidental or unauthorised way since it was made. The General Records Disposal Schedule for<br />
Digital Source Records (QDAN 678) requires that <strong>digital</strong> source <strong>records</strong> are retained until quality<br />
assurance has been undertaken and the results accepted. Trusted systems and quality assurance<br />
processes are therefore a critical component of migration activities.<br />
3.4.1 Preparing <strong>records</strong> for migration<br />
Prior to migration, it is important to take the opportunity to ensure that all <strong>records</strong> and metadata are<br />
as accurate and up-to-date as possible. This will improve the efficiency and quality of the<br />
information migrated. A migration project provides the opportunity to “cleanse” and maximise the<br />
access to data, for example through:<br />
<br />
<br />
<br />
Reviewing the location metadata about <strong>records</strong> when migrating between recordkeeping<br />
systems, for example to ensure people who have left the organisation are not recorded as<br />
holding <strong>records</strong>.<br />
Ensuring all <strong>records</strong> to be migrated are in the source system. If there is a high likelihood<br />
that there are <strong>records</strong> which should be migrated stored on shared directories, email, or<br />
personal and any other systems outside the identified system to be migrated, awareness<br />
activities can be scheduled to encourage transfer of the <strong>records</strong> into the source system<br />
prior to the migration.<br />
Ensuring any <strong>records</strong> stored in encrypted forms are decrypted as appropriate, as<br />
maintaining unnecessary encryption poses significant risks to the continuing useability and<br />
readability of public <strong>records</strong> over time.<br />
As migration can be a resource-intensive activity, it may also be appropriate to minimise its impact<br />
where relevant, by deleting any <strong>records</strong> that have met their minimum lawful retention period prior to<br />
the performance of the migration activity, as outlined in section 3.2.2.<br />
Any data cleansing activity undertaken should be documented and decisions to dispose of <strong>records</strong><br />
prior to migration endorsed and documented as required by IS31, to help ensure the authenticity of<br />
the <strong>records</strong> and to provide justification for decisions to dispose of existing <strong>records</strong> prior to<br />
migration.<br />
Page 27 of 57
Department of Science, Information Technology and Innovation<br />
Key points:<br />
Strive to minimise the overheads associated with migration processes by ensuring <strong>records</strong> and<br />
metadata are accurate and up-to-date and undertaking sentencing projects prior to migration to<br />
dispose of <strong>records</strong> that have met their minimum retention periods.<br />
Document any disposal decisions arising from data cleansing activities prior to migration, and<br />
ensure the documentation is retained as required by the General Retention and Disposal<br />
Schedule (QDAN249).<br />
3.4.2 Testing and validation<br />
To have assurance that migrated <strong>records</strong> retain their full and accurate attributes, testing and<br />
validation that includes recordkeeping considerations must be undertaken, both pre-migration in a<br />
test environment and post-migration in the live system. This will help to minimise and mitigate any<br />
unanticipated and unacceptable results at the ‘go live’ stage of the new system.<br />
A thorough and documented testing and validation process that is undertaken successfully also<br />
provides valuable assurances to a third party as to the authenticity, reliability, integrity and<br />
useability of the <strong>records</strong>. It is also required before source <strong>records</strong> can be disposed of lawfully in<br />
accordance with the General Retention and Disposal Schedule for Digital Source Records<br />
(QDAN678).<br />
The recordkeeping and information professional’s role and responsibilities through the testing and<br />
validation stages will need to be defined and incorporated into the overall project management<br />
planning.<br />
Appropriate tests to be undertaken to ensure the quality of the <strong>records</strong> will need to be devised in<br />
collaboration with the wider project team. A knowledge and understanding of the source and target<br />
states of the <strong>records</strong>, as set out in sections 3.2 and 3.3, will assist in this process. In addition, a<br />
guide to the key recordkeeping related testing and validation checks that should be considered is<br />
provided in Appendix E.<br />
The process will involve working with the project team, to establish and document baseline<br />
information about the <strong>records</strong> themselves and establish recordkeeping benchmark standards and<br />
allowances. Testing procedures and processes should include steps to follow when validation or<br />
testing falls short of the standards set. For example, where more than an agreed percentage of the<br />
total <strong>records</strong> and the required recordkeeping metadata examined are found to be defective, the<br />
project migration phase pauses and a review of technical migration elements are undertaken and<br />
issues resolved before resuming the migration project.<br />
The level of risk to the business if the authenticity of a particular class or group were not effectively<br />
maintained influences the testing nature and scope. For example, testing approaches could<br />
include: scripted and manual testing of migrated information within a business process context;<br />
sampling a broad range, percentage or block of migrated <strong>records</strong> for validation; analysis and<br />
confirmation that migration mapping outcomes have been achieved; log and error-message<br />
checking, etc.<br />
Page 28 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Under the General Retention and Disposal Schedule (QDAN249) there are a number of record<br />
classes that require long term retention with conditional criteria that applies. This would influence<br />
the nature and scope of testing and validation required. The level of validation required should<br />
be considered in the context that includes the possibility of any future activities that could bring<br />
the authenticity of the migrated <strong>records</strong> into question. This is in addition to the ability to be able<br />
to recall and access them in a useable format.<br />
The following are two examples of types of <strong>records</strong> that would need to have a risk analysis<br />
applied to determine the scope and level of testing and validation required:<br />
<br />
<br />
a workers’ compensation incident report concerning an individual that did not involve an<br />
immediate claim that would need to be held on the individual’s personnel file and retained<br />
for 70 years from date of birth of the individual or seven years from date of separation, or<br />
resignation, whichever is later;<br />
<strong>records</strong> relating to the removal and disposal of hazardous materials (including asbestos<br />
and radioactive material) from buildings or structures which require retention for 70 years<br />
after removal of the hazardous materials.<br />
In both of the above cases, if there was a thorough and documented testing and validation<br />
process this would assist in providing evidence to justify the authenticity, reliability and integrity<br />
of the <strong>records</strong> to a third party if this were ever required into the future.<br />
The extent of testing and validation should therefore be commensurate with the risks associated<br />
with the <strong>records</strong> retention requirements and the level of on-going importance required to justify<br />
their authenticity for the life of the <strong>records</strong>.<br />
IMPORTANT: The time and resources needed to undertake testing and validation is often<br />
underestimated. It is important that the <strong>records</strong> manager factors some flexibility into planning for<br />
this time-consuming and resource intensive activity to ensure that it can be undertaken adequately.<br />
Pre-migration testing is strongly recommended to be undertaken within a test environment and<br />
allows for a revised strategy to be devised in the event that adverse effects are observed. For<br />
example, the test may result in an undesirable outcome in its connection with other integrated<br />
software systems and these unanticipated results may be unacceptable to the business and put a<br />
halt to the migration until a solution is devised as part of the overall migration planning. If this were<br />
to occur once the new system had gone live, it could have serious adverse effects on the<br />
stakeholders’ confidence and business operations.<br />
Best practice recommends, and project management methodology calls for, pre and post migration<br />
testing and validation results to be documented and signed off by the appropriate project delegate<br />
at both the pre and post migration phases. As mentioned previously, this documentation will<br />
provide some assurance that the right practices were undertaken and results anticipated were<br />
validated and/or remediation was implemented.<br />
Key points:<br />
A thorough and documented testing and validation process must be undertaken and signed-off so<br />
that assurances can be made to a third party that the authenticity, reliability, integrity and<br />
useability of the <strong>records</strong> have been maintained.<br />
In collaboration with the broader project team members, contribute to the inclusion of<br />
Page 29 of 57
Department of Science, Information Technology and Innovation<br />
recordkeeping baseline standards and the target results/outcomes.<br />
Ensure the extent, range and types of tests are commensurate with the risks and value<br />
associated with the <strong>records</strong>.<br />
Be detailed in your checking for any unanticipated results or consequences from the testing or<br />
migration including any unacceptable affects on integrated systems.<br />
Be aware that time and resources are often underestimated for testing and validation processes.<br />
Ensure that planning includes flexibility of time and resources for this important element.<br />
3.5 Performing migration<br />
Migration<br />
project<br />
initiation<br />
Determining<br />
what is to be<br />
migrated<br />
Determining<br />
where &how<br />
to migrate<br />
Ensuring<br />
quality<br />
Performing<br />
migration<br />
Post<br />
migration<br />
3.5.1 Testing roll back strategy<br />
Prior to migration, it is important that a roll back strategy has been established and thoroughly<br />
tested. Sometimes, despite comprehensive pre-migration efforts and particularly with complex<br />
migrations, unexpected outcomes can occur in migration exercises. To mitigate the risks to the<br />
<strong>records</strong>, due to unexpected and unacceptable outcomes, having a tested roll back option means<br />
that <strong>records</strong> remain protected and business processes can be resumed with the old system until<br />
another migration attempt is made.<br />
3.5.2 Recordkeeping during cut over period<br />
Business arrangements will need to be made to maintain recordkeeping continuity and compliance<br />
during the period when the migration activities are performed up to when the target system<br />
becomes live and available for use. For example certain business transactions may need to be<br />
suspended or recorded manually until the new system is available, with any updates made or<br />
repeated in the new system once it is live.<br />
3.5.3 <strong>Migrating</strong> <strong>records</strong><br />
The process of migration will involve some element of extraction/export, transformation and<br />
import/loading of data into the target system. As outlined in section 3.2.3, metadata should be<br />
captured to record the actions carried out on the data as it progresses through this extraction,<br />
transformation and loading process.<br />
The process of migration in collaboration with members of the broader project team should be<br />
undertaken with the minimum of intervention and time delay as possible. This helps to demonstrate<br />
an unbroken chain of custody and can help to prove the authenticity and reliability of the migrated<br />
<strong>records</strong>, particularly in the event of legal proceedings. The process should be designed to prevent<br />
irreversible activity so in the event that problems arise the migration can be rolled back as<br />
necessary.<br />
To further strengthen the evidential integrity which may be required to be demonstrated in the<br />
event of legal proceedings, documentation should include the actions taken to validate that the<br />
particular equipment and systems used in the migration are in good operating order.<br />
Page 30 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Once <strong>records</strong> have been successfully migrated and all testing/validation processes accepted it is<br />
important that stakeholder expectations are clearly communicated to the business. For example, it<br />
should be clear that the source record is no longer the official record and business rules and<br />
processes should be established to ensure that the source <strong>records</strong> cannot be altered inadvertently<br />
or continue to be used for documenting business activity. Further guidance about the management<br />
of source <strong>records</strong> post-migration is detailed in sections 3.6.1 and 3.6.2.<br />
Key points:<br />
Ensure that a roll back strategy is included in planning and has been tested prior to undertaking<br />
migration activities.<br />
Record information about the process of migration that includes recordkeeping continuity<br />
arrangements during the cut over period.<br />
In collaboration with the broader project team, keep time delays and interventions to a minimum<br />
when moving <strong>records</strong> from the existing system to the target system to strengthen evidential<br />
integrity of the <strong>records</strong> and reduce data updates or repeated actions needed in the new system.<br />
Ensure business rules and processes are put in place to protect the authenticity of the <strong>records</strong>.<br />
3.5.4 Quality assuring the results of the migration<br />
It is necessary to validate the results of the complete migration of the <strong>records</strong> into the live<br />
environment. This is similar in its process to preliminary testing and validation undertaken in the<br />
test environment, prior to go-live, as described in section 3.4.2.<br />
Once again the results should be assessed, documented and a report compiled and signed off by<br />
the assigned project team delegate prior to the completion of the project. This report provides<br />
essential context to <strong>records</strong> that have undergone migration and is a key event in the lifecycle of<br />
<strong>digital</strong> <strong>records</strong>. This documentation should indicate, at a minimum, that the migrated <strong>records</strong> were<br />
inspected, that metadata and characteristics were compared to the original <strong>records</strong> and were intact<br />
in their entirety, that the <strong>records</strong> were properly migrated, and that all necessary control procedures<br />
were applied during the process.<br />
Only after this report attesting to the successful migration of the <strong>records</strong> has been signed-off,<br />
should the disposal authorisation for the <strong>digital</strong> source <strong>records</strong> be activated. Public authorities are<br />
encouraged to maintain this documentation for the life of the record, in order to help demonstrate<br />
the authenticity of the migrated <strong>records</strong>.<br />
Key points:<br />
Testing and validation of recordkeeping functionality post-migration is similar to the process<br />
undertaken pre-migration and requires the same thoroughness.<br />
Only after the post-migration testing and validation report has been signed-off, should the<br />
disposal authorisation for the <strong>digital</strong> source <strong>records</strong> be activated.<br />
Page 31 of 57
Department of Science, Information Technology and Innovation<br />
3.6 Post migration<br />
Migration<br />
project<br />
initiation<br />
Determining<br />
what is to be<br />
migrated<br />
Determining<br />
where & how<br />
to migrate<br />
Ensuring<br />
quality<br />
Performing<br />
migration<br />
Post<br />
migration<br />
3.6.1 Verifying conditions for disposal of source <strong>records</strong><br />
Migration should not immediately cause the destruction of the source <strong>records</strong> that are the subject<br />
of, or input to, those processes. The source <strong>records</strong> generally continue to exist in tandem with the<br />
output of the migration process. An informed business decision therefore needs to be made about<br />
when it is appropriate to dispose of the source <strong>records</strong>.<br />
The General Retention and Disposal Schedule for Digital Source Records (QDAN678) sets out the<br />
minimum conditions for the retention of source <strong>records</strong>. However the period of time for which the<br />
source <strong>records</strong> might require further retention beyond the minimum, should be established by<br />
individual public authorities through a documented risk assessment as necessary.<br />
When conducting the risk assessment, public authorities must consider any other legal or business<br />
continuity issues that may influence the duration for the further retention of the source <strong>records</strong>.<br />
These issues could include, but are not necessarily restricted to, the following:<br />
Verifying conditions for disposal - risk assessment considerations<br />
<br />
<br />
<br />
<br />
<br />
business purpose, litigation potential and value of the <strong>records</strong> and the risks associated with<br />
this, including the potential business, financial and legal implications of any loss of<br />
trustworthiness or access<br />
level of assurance that ‘full and accurate’ <strong>records</strong> have been achieved through the<br />
migration, including the maintenance of the completeness and authenticity of <strong>records</strong><br />
level of assurance that the migrated record is being well managed in a robust recordkeeping<br />
environment<br />
thoroughness of the migration processes, including quality assurance processes and the<br />
size and complexity of the migration<br />
extent of usage and level of integration into the operational environment of the <strong>records</strong>.<br />
(Please note: Records that are likely to be immediately used after migration may require a<br />
shorter retention period as use will quickly highlight any problems. Those <strong>records</strong> that are<br />
unlikely to be used in the immediate period following migration, may need to be considered for a<br />
longer retention that fits into a quality evaluation and validation plan before disposal.)<br />
Table 6 – Risk assessment considerations prior to disposal<br />
During the established period of retention, source <strong>records</strong> must be stored and protected to ensure<br />
that they remain accountable, well-managed <strong>records</strong> of business that can be used again as<br />
appropriate source <strong>records</strong>, should there be a need to roll back and re-initiate migration if issues<br />
arise.<br />
Page 32 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Also during this retention period, isolation of the original source <strong>records</strong> from further access by<br />
general stakeholders is highly recommended to eliminate the possibility of their inadvertent use.<br />
It is recommended that the authorised deletion or disposal of source <strong>records</strong> be performed in a<br />
timely way as part of good business practice in the context of the closure of the migration project.<br />
Key points:<br />
Source <strong>records</strong> cannot be deleted until the conditions outlined in the General Retention and<br />
Disposal Schedule for Digital Source Records (QDAN678) are met.<br />
Undertake a risk assessment to validate the need to retain source <strong>records</strong> for a longer period than<br />
specified in the General Retention and Disposal Schedule for Digital Source Records (QDAN678).<br />
Make sure processes are in place to ensure retained source <strong>records</strong> cannot be confused with or<br />
used for working business transactions.<br />
Deletion of source <strong>records</strong> at the end of the authorised retention period should be scheduled as<br />
part of the project plan closure.<br />
3.6.2 Mandatory requirements before disposing of <strong>digital</strong> ‘source <strong>records</strong>’<br />
The disposal of public <strong>records</strong> including <strong>digital</strong> source <strong>records</strong> requires the authorisation of the<br />
State Archivist under section 26 of the Public Records Act 2002. Authorisation for the disposal of<br />
<strong>digital</strong> source <strong>records</strong> following a successful <strong>records</strong> migration activity is given through the General<br />
Retention and Disposal Schedule for Digital Source Records (QDAN678).<br />
Before <strong>digital</strong> source <strong>records</strong> are deleted under the General Retention and Disposal Schedule for<br />
Digital Source Records (QDAN678), public authorities must ensure that:<br />
<br />
<br />
<br />
agreed quality assurance procedures as described in sections 3.4 and 3.5 have been<br />
conducted and any remediation need has been satisfied,<br />
the verification of the conditions for disposal readiness as outlined in 3.6.1 have been met,<br />
and<br />
their eligibility for disposal signed off by the Chief Executive Officer or their authorised<br />
delegate in line with IS31.<br />
Once the above conditions have been satisfied, the source <strong>records</strong> can be deleted under the<br />
General Retention and Disposal Schedule for Digital Source Records (QDAN678). The disposal of<br />
source <strong>records</strong> must be documented. 13 For disposal of the source <strong>records</strong> in a migration project,<br />
document at system level information about the disposal of the <strong>digital</strong> source <strong>records</strong>.<br />
Approval/endorsement and sign-off of the disposal of the source <strong>records</strong> can, for example be<br />
retained as an administrative record in the public authority’s EDRMS. Note that the General<br />
Retention and Disposal Schedule (QDAN249) requires that master disposal documentation be<br />
retained permanently (reference 9.4.2).<br />
13<br />
See: http://qgcio.qld.gov.au/qgcio/architectureandstandards/informationstandards/current/Pages/IS31-<br />
Implementationadvice.aspx<br />
Page 33 of 57
Department of Science, Information Technology and Innovation<br />
Public authorities should ensure that the source <strong>records</strong> are securely deleted once the approved<br />
retention period has expired so that they cannot be retrieved or recreated. While some business<br />
systems have recordkeeping functionality built-in that includes workflow for an authorised<br />
destruction process, there are many business systems that do not have this specific utility and<br />
secure deletion involves a separate process and documentation. To cover a public authority’s<br />
specific circumstances, it is recommended that the secure and irretrievable deletion process be<br />
incorporated into the overall project planning and be devised in consultation with the broader team<br />
for the project.<br />
As detailed in section 3.6.1, performing a migration results in the existence of two versions of the<br />
same record – the ‘source’ record and the new ‘migrated’ record (Refer to Figure 3). Maintaining<br />
two <strong>records</strong> is not cost effective and may cause confusion as to which record is the official record<br />
of the business. Digital source <strong>records</strong> should therefore be deleted once quality assurance and risk<br />
assessment procedures have been completed in line with the General Retention and Disposal<br />
Schedule for Digital Source Records (QDAN678). At a minimum they should be isolated and made<br />
inaccessible for general use.<br />
Key points:<br />
Source <strong>records</strong> that have been subject to migration to a new target system can only be destroyed<br />
in accordance with the General Retention and Disposal Schedule for Digital Source Records<br />
(QDAN678).<br />
Quality assurance procedures and validation must be completed and signed off by the authorised<br />
delegate prior to deletion.<br />
Disposal must be endorsed by the Chief Executive Officer or their authorised delegate and<br />
information about the source <strong>records</strong> disposal retained as required by IS31 and the General<br />
Retention and Disposal Schedule (QDAN249).<br />
Source <strong>records</strong> must be securely destroyed so that they cannot be retrieved or recreated.<br />
Capturing mandatory recordkeeping documentation<br />
An essential part of all project management methodologies is the documentation and sign-offs on<br />
the finalisation of the process to confirm that the project was completed successfully. In the case of<br />
the migration of public <strong>records</strong> the sign-offs need to include sign-off on the recordkeeping<br />
considerations.<br />
This is important because if migrated <strong>records</strong> were ever called into question during legal<br />
proceedings, it may be necessary to certify that the migration processes were reliable and<br />
produced accurate and authentic <strong>records</strong>. Generally, the greater the risk associated with the need<br />
to rely on the <strong>records</strong> in court or in business transactions, the greater the need to make and keep<br />
adequate documentation that attests to the accuracy and authenticity of the <strong>records</strong>.<br />
A range of documentation will have been produced throughout the project in accordance with the<br />
public authority’s project management methodology. As a guide for <strong>records</strong> and information<br />
managers, documentation of the aspects of the migration project outlined in the Table below is<br />
important for recordkeeping purposes to provide assurance about the integrity of the <strong>records</strong>’<br />
treatment throughout the migration process and the results achieved.<br />
Page 34 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Guide to recordkeeping documentation:<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
descriptions of the identified <strong>records</strong> and their characteristics, and the metadata being<br />
migrated along with reports associated with decisions made<br />
any data cleansing performed, including <strong>records</strong> of decisions and the processes<br />
undertaken<br />
the formal migration plans and processes<br />
the dates/times of the migration activities and the people involved, including their roles and<br />
positions<br />
system configurations, including metadata definitions and mappings<br />
risk assessments and mitigation planning and activities<br />
testing and quality assurance plans and results, pre and post migration<br />
any reports that compare original system functionality to target system functionality<br />
all sign-offs/approvals and all decisions, including any decisions not to migrate certain data<br />
any variations to plans around the recordkeeping considerations<br />
any necessary variation in <strong>records</strong> structure, metadata, format or content that will or has<br />
resulted from the migration<br />
the disposal of the source <strong>records</strong> including approved assurances that the minimum<br />
conditions were met and the appropriate further period of retention had elapsed.<br />
Table 7 – Guide to documentation requirements<br />
Key points:<br />
Projects should not be closed until there are appropriate assurances that the <strong>records</strong> have been<br />
successfully migrated and the original source <strong>records</strong> have been attended to in accordance with<br />
the authorised plan.<br />
Documenting the recordkeeping considerations and results applicable to the <strong>records</strong> migration<br />
process is important as this helps to demonstrate the integrity and authenticity of the <strong>records</strong> in<br />
the future.<br />
3.6.4 Stakeholder relations and continuous improvement<br />
While the migration project may have been completed and signed off by the business, the work of<br />
those responsible for <strong>records</strong> and information management continues. The following pointers<br />
highlight some of those areas for consideration.<br />
Staff training<br />
Although this is not dealt with in detail in this guideline, an important aspect arising from migration<br />
projects for a <strong>records</strong> and information manager to consider is the need to ensure that appropriate<br />
Page 35 of 57
Department of Science, Information Technology and Innovation<br />
training of staff is undertaken. This may require targeted promotion and the development of new<br />
procedures relating to the management of the <strong>records</strong> in the new migrated environment.<br />
Ongoing monitoring<br />
To ensure that the migrated <strong>records</strong> continue to be maintained as expected, and as a mechanism<br />
for identifying opportunities for recordkeeping improvements, ongoing monitoring of system usage<br />
and the system’s treatment of the <strong>records</strong> is required.<br />
Lessons learnt and continuous improvement opportunities<br />
There will be many lessons learned from the migration project that can be applied to future projects<br />
to improve the process and outcomes of <strong>records</strong> migration projects.<br />
By documenting the learnings for potential future reference and having <strong>records</strong> management<br />
expertise involved in business system procurement and implementation projects, ongoing<br />
recordkeeping considerations can be included and ongoing business benefits can be achieved.<br />
Key points:<br />
Migration projects are an opportunity to improve recordkeeping practices.<br />
Recordkeeping training material may need updating.<br />
Monitor ongoing usage of the system to ensure <strong>records</strong> continue to be managed in accordance<br />
with requirements.<br />
Page 36 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Appendices<br />
Attached supporting materials for the <strong>records</strong> and information manager<br />
To further assist recordkeeping and information professionals in their engagement with<br />
recordkeeping considerations during a migration project, various supporting documents are<br />
appended here as quick reference materials. These are referred to in the text where appropriate<br />
and include:<br />
Appendix A<br />
Appendix B<br />
Appendix C<br />
Appendix D<br />
Appendix E<br />
Appendix F<br />
IS40 Principle 7 and the migration of <strong>digital</strong> <strong>records</strong> – a guide<br />
Records manager’s project stages flow chart<br />
Records manager’s migration project check list<br />
Recordkeeping controls in the target system – a summary<br />
Testing and validation check list<br />
Lessons learnt – migration hints and tips for recordkeeping professionals<br />
Page 37 of 57
Department of Science, Information Technology and Innovation<br />
Appendix A – IS40 Principle 7 and the migration of <strong>digital</strong> <strong>records</strong><br />
The following is a guide to the application of IS40 Principle 7 to the migration of <strong>digital</strong> <strong>records</strong>.<br />
To be a full and accurate record a public record must, amongst other requirements, be:<br />
Attribute Standard meaning Digital migration context<br />
Complete<br />
Meaningful<br />
Accurate<br />
Authentic<br />
Inviolate<br />
Complete <strong>records</strong> comprise contextual<br />
and structural data as well as content.<br />
Contextual data is information about the<br />
creation and use of the record, including<br />
the business processes that give rise to<br />
it. Structural data includes formal<br />
internal structures of the information<br />
object and the structural relations<br />
between <strong>records</strong>. Content is the<br />
information contained within the<br />
information object.<br />
Meaningful <strong>records</strong> may be understood<br />
in the context of the processes and<br />
business for which they are created and<br />
in which they are used.<br />
An accurate record reflects what was<br />
communicated, decided or done (or not<br />
done). That is, the record’s content,<br />
context and structure can be trusted as<br />
a true and accurate representation of<br />
the transactions, activities or facts that<br />
they document and can be depended<br />
upon in the course of subsequent use.<br />
An authentic record is a record that can<br />
be proven and trusted to be what it<br />
purports to be and to have been<br />
created, used, transmitted or held by the<br />
person to whom these actions have<br />
been attributed. The maintenance of<br />
authentic <strong>records</strong> is facilitated by wellplanned,<br />
comprehensive and successful<br />
migration processes and metadata<br />
capture.<br />
Inviolate <strong>records</strong> are time-bound (that is<br />
they are fixed at a point in time) and<br />
A complete migrated record is an<br />
accurate replica of the source record,<br />
including all the record’s content,<br />
context and structure i.e. metadata.<br />
For <strong>digital</strong> <strong>records</strong> to be meaningful,<br />
metadata that <strong>records</strong> the history of the<br />
recorded information’s use is required to<br />
ensure it is understood in a factual<br />
context.<br />
An accurate migrated record is one that<br />
remains a reliable representation of the<br />
business activity after the migration.<br />
An authentic migrated record is one<br />
where it can be demonstrated,<br />
particularly through the capture of<br />
metadata about the migration process,<br />
that it maintains its trustworthiness as<br />
being what it purports to be. The<br />
requirement for authenticity means that<br />
any migration must be managed and<br />
documented with care so that the<br />
reasons for the process are explicit, the<br />
method used to validate the quality of<br />
the record following the process is clear,<br />
and the metadata detailing who did what<br />
and when is recorded and maintained.<br />
An inviolate migrated record is one<br />
where the content data acknowledges<br />
Page 38 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Attribute Standard meaning Digital migration context<br />
complete. To be inviolate, a record must<br />
be securely maintained to prevent<br />
alteration and unauthorised removal.<br />
Where alteration or changes must<br />
occur, the inviolate nature of the <strong>records</strong><br />
can be protected through the capture of<br />
metadata about the alterations.<br />
its time-bound elements (fixed at the<br />
point in time and evidence of business<br />
activity it is recording) and where it can<br />
be demonstrated that no unauthorised<br />
changes have been made to it. If<br />
changes are authorised, these changes<br />
and the authorisation are recorded in<br />
metadata.<br />
Accessible<br />
Useable<br />
Accessible <strong>records</strong> are <strong>records</strong> that can<br />
be identified, located and viewed as<br />
required.<br />
Useable <strong>records</strong> are those that may be<br />
viewed and are functional and reuseable.<br />
An accessible migrated record is one<br />
that is captured in a format, media and<br />
system in which the current<br />
technologies will enable timely retrieval<br />
and readable access of that migrated<br />
record<br />
A useable migrated record is one that<br />
can be used with the appropriate level<br />
of functionality required to provide a<br />
factual and reliable interpretation of the<br />
source record. For example, if the<br />
hyperlinks do not successfully work after<br />
migration, and these links have been<br />
determined as being an essential<br />
characteristic of the record, then the<br />
record is not “useable”.<br />
Page 39 of 57
Department of Science, Information Technology and Innovation<br />
Appendix B – Records manager’s project stages flowchart<br />
Project planning<br />
1. Migration project<br />
initiation<br />
Roles and<br />
responsibilities<br />
Stakeholder<br />
communications<br />
Identifying the<br />
<strong>records</strong><br />
2. Determining what<br />
is to be migrated<br />
Establishing<br />
retention<br />
obligations<br />
Determining<br />
recordkeeping<br />
metadata<br />
Understanding the<br />
characteristics of<br />
the <strong>records</strong><br />
3. Determining<br />
where and how to<br />
migrate<br />
Verifying target<br />
system/s<br />
recordkeeping<br />
capabilities<br />
Making file/record<br />
format decisions<br />
Establishing<br />
migration<br />
approach<br />
Page 40 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Preparing <strong>records</strong><br />
for migration<br />
4. Ensuring quality<br />
Testing and<br />
validation<br />
Testing roll back<br />
strategy<br />
5. Performing<br />
migration<br />
Recordkeeping<br />
during cut over<br />
period<br />
Migration of<br />
<strong>records</strong><br />
Quality assuring<br />
the results of the<br />
migration<br />
6. Post migration<br />
Verifying<br />
conditions for<br />
disposal of source<br />
<strong>records</strong><br />
Disposal of <strong>digital</strong><br />
‘source <strong>records</strong>’<br />
Capture of<br />
mandatory<br />
recordkeeping<br />
documentation<br />
Stakeholder<br />
relations and<br />
continuous<br />
improvement<br />
Page 41 of 57
Department of Science, Information Technology and Innovation<br />
Appendix C – Records manager’s migration project checklist<br />
Stages for <strong>records</strong><br />
manager involvement<br />
Key recordkeeping considerations<br />
Guideline<br />
section<br />
reference<br />
Supporting<br />
guidance available on QSA<br />
Website<br />
1. Migration project<br />
initiation<br />
The initial stage of highlevel<br />
scheduling and<br />
resource planning for the<br />
project.<br />
Project planning ensure that recordkeeping obligations and<br />
activities are well understood by the project<br />
team and are incorporated into the project<br />
brief and other project initiation<br />
documentation e.g. project stage plans.<br />
<br />
recordkeeping matters for consideration<br />
include:<br />
3.1.1<br />
o<br />
o<br />
risk mitigation planning is commensurate<br />
with the level of value associated with<br />
the <strong>records</strong> e.g. vital <strong>records</strong>, permanent<br />
value <strong>records</strong>, etc<br />
appropriate scheduling of the associated<br />
<strong>records</strong> related activities is incorporated<br />
into the project stage plan (as listed in<br />
the rest of this check list).<br />
Roles and responsibilities Records management role may include:<br />
3.1.2<br />
<br />
<br />
<br />
the appraisal / identification of <strong>records</strong>, how<br />
long the <strong>records</strong> must be retained and which<br />
<strong>records</strong> are eligible for lawful disposal<br />
determining whether the system is currently<br />
managing the <strong>records</strong> or whether this is<br />
through another recordkeeping system or<br />
combination of systems<br />
contribute to the mapping process between<br />
the existing and target system to ensure<br />
Page 42 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Stages for <strong>records</strong><br />
manager involvement<br />
Key recordkeeping considerations<br />
recordkeeping elements are included,<br />
including the relevant metadata to maintain<br />
the authenticity of the <strong>records</strong> and that they<br />
will be ‘full and accurate’ in their<br />
characteristics<br />
Guideline<br />
section<br />
reference<br />
Supporting<br />
guidance available on QSA<br />
Website<br />
<br />
contribute to the design or configuration of<br />
the target system to ensure that adequate<br />
recordkeeping controls are established<br />
<br />
facilitate the process for the authorised<br />
disposal of eligible <strong>records</strong><br />
<br />
participate in testing and validation<br />
processes<br />
<br />
support the process of the quality assurance<br />
of migrated <strong>records</strong><br />
<br />
provide training and awareness in <strong>records</strong><br />
management procedures<br />
<br />
advocate for continuous improvement of<br />
recordkeeping functionality within business<br />
system/s<br />
<br />
Stakeholder<br />
communications<br />
<br />
engage with the overall project’s<br />
communications strategy<br />
3.1.3<br />
<br />
ensure that recordkeeping considerations<br />
are integrated into planning e.g. in<br />
communications and system training<br />
Page 43 of 57
Department of Science, Information Technology and Innovation<br />
Stages for <strong>records</strong><br />
manager involvement<br />
2. Determining what is<br />
to be migrated<br />
The stage of the project<br />
when the scope of the<br />
migration is being<br />
established.<br />
Key recordkeeping considerations<br />
Identifying the <strong>records</strong> undertake an analysis of the business<br />
processes<br />
<br />
Establishing retention<br />
obligations<br />
<br />
<br />
<br />
identify information that <strong>records</strong> evidence of<br />
business transactions (the <strong>records</strong>)<br />
disposal of public <strong>records</strong> must be<br />
authorised and given under a Retention and<br />
Disposal Schedule that has been approved<br />
by the State Archivist for use by the public<br />
authority.<br />
determine and document the lawful retention<br />
obligations that apply to the <strong>records</strong> in<br />
accordance with the authorised Retention<br />
and Disposal Schedule/s applicable to the<br />
public authority.<br />
Guideline<br />
section<br />
reference<br />
Supporting<br />
guidance available on QSA<br />
Website<br />
3.2.1 Part 2.3 Guidelines and<br />
Functional Requirements for<br />
Records in Business Systems<br />
3.2.2 Guideline for the implementation<br />
of retention and disposal<br />
schedules<br />
Information standard 31:<br />
retention and disposal of public<br />
<strong>records</strong><br />
General Retention and Disposal<br />
Schedule (QDAN249)<br />
<br />
<br />
<br />
determine which <strong>records</strong> have passed their<br />
retention period and that are eligible for<br />
authorised disposal.<br />
after establishing the lawful retention period,<br />
and if time and resources allow and there<br />
are no other legal, business or legislative<br />
requirements to retain the <strong>records</strong>,<br />
undertake an authorised process to dispose<br />
of any <strong>records</strong> that have met their minimum<br />
retention period.<br />
if disposal of eligible <strong>records</strong> has occurred<br />
prior to migration, capture metadata about<br />
the <strong>records</strong> that have been disposed in<br />
accordance with IS31. The documentation<br />
about the disposed public <strong>records</strong> must be<br />
Page 44 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Stages for <strong>records</strong><br />
manager involvement<br />
Key recordkeeping considerations<br />
retained permanently.<br />
Guideline<br />
section<br />
reference<br />
Supporting<br />
guidance available on QSA<br />
Website<br />
<br />
Determining recordkeeping<br />
metadata<br />
<br />
<br />
<br />
analyse and document the recordkeeping<br />
metadata that exists about the <strong>records</strong><br />
plan for the capture of metadata about the<br />
migration activities to be undertaken on the<br />
<strong>records</strong><br />
if possible and appropriate, improve the<br />
capture of metadata about the <strong>records</strong> by<br />
contributing to the design and configuration<br />
decisions about the target system.<br />
3.2.3 Queensland Recordkeeping<br />
Metadata Standard and<br />
Guideline (QRKMS)<br />
Metadata for <strong>digital</strong> continuity: a<br />
companion guideline to the<br />
Queensland recordkeeping<br />
metadata standard<br />
<br />
Understanding the<br />
characteristics of the<br />
<strong>records</strong><br />
<br />
analyse and document the characteristics of<br />
the <strong>records</strong> including: content, context,<br />
structure, appearance/rendering,<br />
behavioural properties (e.g. interactions with<br />
other software/information, hyperlinks, etc).<br />
3.2.4<br />
Appendix A<br />
Appendix<br />
F.2<br />
<br />
consider the characteristics from different<br />
business perspectives and usage<br />
requirements.<br />
<br />
advise on the priority of the characteristics<br />
that need to be carried over with the<br />
migration.<br />
<br />
at a minimum, those characteristics that<br />
contribute to the authenticity, access,<br />
usability and meaningfulness of the <strong>records</strong><br />
should be preserved.<br />
<br />
undertake a risk analysis of the<br />
characteristics which will or will not be<br />
Page 45 of 57
Department of Science, Information Technology and Innovation<br />
Stages for <strong>records</strong><br />
manager involvement<br />
Key recordkeeping considerations<br />
carried over, and justify and document<br />
decisions about acceptable change and/or<br />
loss.<br />
Guideline<br />
section<br />
reference<br />
Supporting<br />
guidance available on QSA<br />
Website<br />
3. Determining where<br />
and how to migrate<br />
The stage when decisions<br />
are being explored and<br />
made about the approach,<br />
method for the migration<br />
to the target system(s).<br />
<br />
Verifying target system/s<br />
recordkeeping capabilities<br />
<br />
<br />
in collaboration with the relevant project<br />
team members, explore the migration<br />
options for active and inactive <strong>records</strong> and<br />
ascertain how recordkeeping compliance<br />
can be maintained for each.<br />
gain an understanding of the ability of target<br />
system/s to meet recordkeeping obligations.<br />
3.3.1 Information Standard 31:<br />
Retention and Disposal of Public<br />
Records<br />
Information Standard 40:<br />
Recordkeeping<br />
<br />
note that retention periods that apply to the<br />
<strong>records</strong> may have an influence on target<br />
system decisions.<br />
<br />
Making file / record format<br />
decisions<br />
<br />
minimise the need for record migrations by<br />
determining record format(s) that take into<br />
account the <strong>records</strong>’ retention periods and<br />
the record types<br />
3.3.2<br />
Appendix<br />
F.1<br />
Storage for <strong>digital</strong> <strong>records</strong><br />
<br />
Establishing migration<br />
approach<br />
Contribute to the:<br />
<br />
documentation and approval of the plans<br />
that describe what will happen to all the<br />
information within the system including the<br />
corresponding security classifications<br />
3.3.3<br />
Appendix<br />
F.3<br />
<br />
<br />
selection of tools which can best facilitate<br />
the maintenance of full and accurate <strong>records</strong><br />
throughout the migration process<br />
development of the business rules and<br />
quality checking processes to support the<br />
Page 46 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Stages for <strong>records</strong><br />
manager involvement<br />
Key recordkeeping considerations<br />
chosen migration approach/es, including any<br />
transforming of the data, loading and any<br />
reliance on manual extraction.<br />
Guideline<br />
section<br />
reference<br />
Supporting<br />
guidance available on QSA<br />
Website<br />
4. Ensuring quality<br />
The stage when quality<br />
assurance, including<br />
testing and validation<br />
processes, are identified,<br />
planned and undertaken.<br />
<br />
Preparing <strong>records</strong> for<br />
migration<br />
Contribute to the minimisation of the overheads<br />
associated with migration processes by:<br />
<br />
<br />
ensuring <strong>records</strong> and metadata are accurate<br />
and up-to-date<br />
undertaking sentencing projects prior to<br />
migration to dispose of <strong>records</strong> that have<br />
met their retention periods<br />
3.4.1<br />
<br />
documenting any data cleansing and<br />
disposal activities in accordance with the<br />
requirements of IS31.<br />
Testing and validation Contribute to the:<br />
3.4.2<br />
<br />
<br />
<br />
<br />
development of testing that incorporates the<br />
maintenance of the authenticity of the full<br />
and accurate <strong>records</strong><br />
extent, range and types of testing that is<br />
commensurate with the risks and value<br />
associated with the <strong>records</strong><br />
documenting of the testing and validation<br />
plans, including baseline information and<br />
target results / outcomes<br />
engagement of appropriate recordkeeping<br />
skilled people within the migration stage<br />
Appendix E<br />
Page 47 of 57
Department of Science, Information Technology and Innovation<br />
Stages for <strong>records</strong><br />
manager involvement<br />
Key recordkeeping considerations<br />
<br />
sign-off process by ensuring that where<br />
appropriate recordkeeping elements are<br />
signed-off by an authorised delegate in<br />
addition to any other authorised sign-offs<br />
Guideline<br />
section<br />
reference<br />
Supporting<br />
guidance available on QSA<br />
Website<br />
<br />
<br />
provision of advice on adequate time and<br />
resources to be allocated to the testing and<br />
validation of recordkeeping elements<br />
including the recommendation of conducting<br />
testing on copies of <strong>records</strong> within a test<br />
environment prior to migration<br />
documentation of the recordkeeping<br />
elements of the testing and validation<br />
processes and outcomes.<br />
5. Performing migration<br />
The stage concerned with<br />
the actual undertaking of<br />
the migration to extract /<br />
export, transfer and load /<br />
import the <strong>records</strong> into the<br />
target system(s).<br />
Testing roll back strategy check that roll back strategy has been tested<br />
prior to migration exercise<br />
<br />
Recordkeeping during cut<br />
over period<br />
<br />
prior to performing the migration, ensure that<br />
procedures are in place to protect<br />
recordkeeping elements during the period of<br />
cut over to the new system<br />
3.5.1<br />
3.5.2<br />
Appendix F<br />
<br />
as some system transactions may have<br />
been suspended or recorded manually<br />
during this period, factor in time and<br />
resources to perform the system updates<br />
Migration of <strong>records</strong> Contribute to:<br />
3.5.3<br />
<br />
<br />
documentation of the recordkeeping<br />
elements of the migration process<br />
engagement of appropriate recordkeeping<br />
Page 48 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Stages for <strong>records</strong><br />
manager involvement<br />
Key recordkeeping considerations<br />
skilled people within the migration stage<br />
Guideline<br />
section<br />
reference<br />
Supporting<br />
guidance available on QSA<br />
Website<br />
<br />
<br />
development of recordkeeping business<br />
rules and the establishment of subsequent<br />
processes to protect the authenticity and<br />
integrity of the <strong>records</strong><br />
supporting the minimisation of time delays<br />
and interventions.<br />
<br />
Quality assuring the results<br />
of the migration<br />
Similar to the testing and validation stage above,<br />
contribute to the:<br />
3.5.4<br />
<br />
<br />
<br />
<br />
<br />
<br />
development of testing that incorporates the<br />
maintenance of the authenticity of the full<br />
and accurate <strong>records</strong><br />
extent, range and types of testing that is<br />
commensurate with the risks and value<br />
associated with the <strong>records</strong><br />
documenting of the testing and validation<br />
plans, including baseline information and<br />
target results / outcomes<br />
engagement of appropriate recordkeeping<br />
skilled people within the migration stage<br />
sign-off process by ensuring that where<br />
appropriate recordkeeping elements are<br />
signed-off by an authorised delegate in<br />
addition to any other authorised sign-offs<br />
provision of advice on adequate time and<br />
resources to be allocated to the testing and<br />
validation of recordkeeping elements<br />
Page 49 of 57
Department of Science, Information Technology and Innovation<br />
Stages for <strong>records</strong><br />
manager involvement<br />
Key recordkeeping considerations<br />
including the recommendation of conducting<br />
testing on copies of <strong>records</strong> within a test<br />
environment prior to migration<br />
Guideline<br />
section<br />
reference<br />
Supporting<br />
guidance available on QSA<br />
Website<br />
<br />
documentation of the recordkeeping<br />
elements of the testing and validation<br />
processes and outcomes.<br />
6. Post migration<br />
The final recordkeeping<br />
considerations prior to<br />
project closure<br />
<br />
Verifying conditions for<br />
disposal of source <strong>records</strong><br />
Contribute to ensuring that recordkeeping<br />
considerations are incorporated into:<br />
<br />
processes that eliminate any business user<br />
confusion, access or potential inadvertent<br />
use of the now redundant source <strong>records</strong><br />
3.6.1 General Retention and Disposal<br />
Schedule for Digital Source<br />
Records (QDAN678)<br />
<br />
review and update the risk level attached to<br />
the <strong>records</strong> that have been subject to<br />
migration, based on the success of the<br />
validation process, and confirm the lawful<br />
period of retention for the original source<br />
<strong>records</strong> left behind in the old home location,<br />
and<br />
<br />
once conditions for retention have been met,<br />
source <strong>records</strong> can be deleted in<br />
accordance with the General Retention and<br />
Disposal Schedule for Digital Source<br />
Records<br />
<br />
Disposal of <strong>digital</strong> ‘source<br />
<strong>records</strong>’<br />
<br />
source <strong>records</strong> can only be destroyed in<br />
accordance with the General Retention and<br />
Disposal Schedule for Digital Source<br />
Records<br />
3.6.2 General Retention and Disposal<br />
Schedule for Digital Source<br />
Records (QDAN678)<br />
<br />
quality assurance procedures must be<br />
Page 50 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Stages for <strong>records</strong><br />
manager involvement<br />
Key recordkeeping considerations<br />
completed and signed off by the appropriate<br />
delegate<br />
Guideline<br />
section<br />
reference<br />
Supporting<br />
guidance available on QSA<br />
Website<br />
<br />
disposal of the source <strong>records</strong> must first be<br />
endorsed by the Chief Executive Office or<br />
their authorised delegate<br />
<br />
disposal must be documented and<br />
documentation must capture the minimum<br />
metadata requirements as outlined under<br />
IS31<br />
<br />
once these conditions have been satisfied,<br />
source <strong>records</strong> must be securely destroyed<br />
so that they cannot be retrieved or recreated<br />
<br />
Capture mandatory<br />
recordkeeping<br />
documentation<br />
<br />
the greater the risk associated with the need<br />
to rely on the <strong>records</strong> in court or in business<br />
transactions, the greater the need to make<br />
and keep adequate documentation that<br />
attests to the accuracy and authenticity of<br />
the <strong>records</strong><br />
3.6.3 Information Standard 31:<br />
Retention and Disposal of Public<br />
Records<br />
<br />
the disposal of source <strong>records</strong> post migration<br />
must be documented in line with IS31<br />
<br />
projects should not be closed until there are<br />
appropriate assurances that the<br />
recordkeeping considerations have been<br />
satisfied<br />
<br />
documenting the migration projects<br />
recordkeeping considerations and the<br />
associated results will help to demonstrate<br />
the integrity and authenticity of the <strong>records</strong><br />
Page 51 of 57
Department of Science, Information Technology and Innovation<br />
Stages for <strong>records</strong><br />
manager involvement<br />
Key recordkeeping considerations<br />
Guideline<br />
section<br />
reference<br />
Supporting<br />
guidance available on QSA<br />
Website<br />
<br />
Stakeholder relations and<br />
continuous improvement<br />
<br />
follow-up with your stakeholders to confirm<br />
system results and gauge user adaptation<br />
3.6.4<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
ensure that training and awareness<br />
materials are up-to-date<br />
ensure that communications and awareness<br />
sessions are completed<br />
document the recordkeeping related lessons<br />
learned from the migration project<br />
monitor ongoing usage and management of<br />
the system and the treatment of the <strong>records</strong><br />
by the new system and its business users<br />
contribute to continuing education of staff in<br />
recordkeeping matters<br />
continue to promote opportunities for<br />
business process improvement in<br />
recordkeeping matters<br />
maintain relationship/s with system owner<br />
and business users and ensure that<br />
recordkeeping responsibilities continue to be<br />
assigned<br />
Page 52 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Appendix D – Recordkeeping controls in the target system<br />
Key recordkeeping considerations for the target system<br />
can protect the<br />
integrity,<br />
authenticity and<br />
context of the<br />
<strong>records</strong><br />
is reliable and<br />
secure<br />
This includes the maintenance of recordkeeping metadata connections.<br />
<strong>Migrating</strong> <strong>records</strong> without effective recordkeeping controls, for example<br />
migrating <strong>records</strong> onto a shared drive, may not capture adequately the<br />
contextual metadata (e.g. author, version history, true date of creation,<br />
etc) required to maintain the authenticity and integrity of the <strong>records</strong>.<br />
Under Information Standard 40, all systems used to create or maintain<br />
<strong>records</strong> must work reliably and be secure to ensure that <strong>records</strong> are<br />
credible and authoritative. Public authorities are therefore required to<br />
implement systems which are secure from unauthorised access,<br />
damage and misuse. This requires supporting processes and tools that<br />
prevent the corruption or loss of the <strong>digital</strong> <strong>records</strong>, e.g. backup/restore<br />
procedures, disaster recovery procedures, sampling of media and <strong>digital</strong><br />
<strong>records</strong> to detect deterioration and corruption, and the refreshing of<br />
media.<br />
An ability to ensure the reliability and security of the <strong>records</strong> will also<br />
influence choices made about the storage medium of <strong>records</strong>. For<br />
example, storage of <strong>records</strong> on RAID (Redundant Array of Independent<br />
Disks) servers may be considered more secure and reliable than<br />
storage of <strong>records</strong> on removable media such as CDs, or DVDs.<br />
can enable access<br />
to the <strong>records</strong><br />
Under section 14 of the Public Records Act 2002, public authorities<br />
controlling <strong>records</strong> that can only be produced or made available with the<br />
use of particular equipment or information technology must take all<br />
reasonable action to ensure the information remains able to be<br />
produced or made available.<br />
At a minimum, because of technology obsolescence, the selected<br />
system should have the ability to easily export <strong>records</strong> so that at a future<br />
point in time when the system’s lifespan has been reached and<br />
decommissioning occurs, the <strong>records</strong> are able to be transferred again to<br />
a successor for continued access if required.<br />
Some approaches will enable easier and wider access to <strong>records</strong> than<br />
others. For example, <strong>records</strong> in offline systems that become legacy<br />
systems are arguably less accessible than other online <strong>records</strong> or<br />
<strong>records</strong> stored in accessible file formats.<br />
To be able to access a record requires that first the <strong>records</strong> are<br />
structured and stored in a way that they can be searched for and are<br />
discoverable. The ability of the system to facilitate discoverability would<br />
factor in the selection of a target system, as a system that does not<br />
enable discoverability prevents access to the <strong>records</strong>. For example,<br />
converting <strong>records</strong> and storing them in a data store that does not<br />
provide adequate searching facilities will not enable access to the<br />
<strong>records</strong>.<br />
Page 53 of 57
Department of Science, Information Technology and Innovation<br />
Key recordkeeping considerations for the target system<br />
The lifespan of the system and associated technologies, software<br />
licensing arrangements or any other constraints which may limit access,<br />
should also be considered when selecting an appropriate target system.<br />
can preserve the<br />
characteristics of<br />
the <strong>records</strong><br />
The characteristics of the <strong>records</strong> must be maintained throughout a<br />
migration process, as mentioned in section 3.2.4. The ability to<br />
maximise the extent of the characteristics that can be preserved may<br />
influence the choice of file format or system. Should a particular file<br />
format not be able to maintain, for example, the appearance of the<br />
content of the record, and the appearance is critical to the meaning of<br />
the record, it may not be an appropriate format to choose.<br />
Page 54 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
Appendix E – Testing and validation checklist 14<br />
Details<br />
Guideline<br />
reference if<br />
applicable<br />
□ Check accuracy of the content and metadata of the <strong>records</strong>. For example:<br />
expected fields are populated with data, e.g. mandatory value 3.2.3<br />
<br />
<br />
field values are in keeping with expected lengths and valid date ranges<br />
values are populated with expected data types (e.g. numeric,<br />
alphanumeric, date, etc.)<br />
Appendix F.2<br />
<br />
metadata values remain accurate, particularly those such as the date of<br />
creation and dates which trigger disposal countdowns<br />
accurate metadata about the migration process itself is captured 3.2.4<br />
□ Check for completeness – that all <strong>records</strong> identified for migration, have been<br />
migrated and that the completeness of the <strong>records</strong> themselves is verified.<br />
For example:<br />
<br />
numbers of <strong>records</strong> remains the same<br />
Appendix A<br />
linkages/relationships between metadata and <strong>records</strong> are maintained Appendix F.2<br />
<br />
results of standard reports and searches are repeatable, consistent, and<br />
accurate<br />
□ Check for maintenance of the integrity and authenticity of the <strong>records</strong>. For<br />
example:<br />
<br />
all agreed characteristics of the <strong>records</strong> as previously identified for<br />
migration during the planning process are maintained<br />
3.2.4<br />
<br />
any errors or corruption during transmission are flagged, e.g. as identified<br />
through checksums – an IT computational process that can highlight<br />
errors that might have occurred during the migration process.<br />
□ Check for meaningfulness and the access and usability of the <strong>records</strong>. For<br />
example:<br />
Appendix A<br />
<br />
real time checks of a sample of the <strong>records</strong> to ensure they remain<br />
readable and understandable<br />
□ Check the reliability of the system/s including functionality, processes, and<br />
integrated system/s and peripherals. For example:<br />
<br />
after roll-out the system is stable and operating within acceptable error<br />
log margins that have been established in collaboration with IT<br />
colleagues<br />
14<br />
A guide only. The individual example items in this checklist should be adapted and expanded to suit the public<br />
authority’s specific migration project and developed in collaboration with IT colleagues.<br />
Page 55 of 57
Department of Science, Information Technology and Innovation<br />
Appendix F – Lessons learnt – hints and tips<br />
The following hints and tips have been provided by <strong>records</strong> and information managers from public<br />
authorities that have been involved with <strong>records</strong> migration projects.<br />
F.1 File format decisions<br />
When migrating documents from an old system that was marked for decommissioning, single page<br />
tiff files were identified that were bound together by the proprietary software. Once migrated, these<br />
documents would have separated into single pages causing many access issues and creating<br />
thousands more documents than were necessary. The public authority was able to copy them,<br />
optical character recognition (OCR) them, convert them to single (multi-page) PDF documents<br />
prior to placing them into the new eDRMS, along with the treatment history of these <strong>records</strong> to<br />
ensure their authenticity could be verified.<br />
Overall there were significant benefits to the business in undertaking this exercise. These included:<br />
considerable improvements to the searchability and accessibility of these <strong>records</strong>, and<br />
maintenance of the <strong>digital</strong> continuity of the <strong>records</strong>.<br />
F.2 Complexities of determining what is to be migrated and what is to be tested and<br />
validated<br />
Identifying the full characteristics of the <strong>records</strong> within the business system and their<br />
dependencies and relationships with data elements can be complex. Records managers need<br />
to work with their IT colleagues to ensure that the <strong>records</strong> to be migrated will retain their<br />
authenticity and remain full and accurate. Decisions made need to be justified and<br />
documented to ensure that the integrity of the <strong>records</strong> can be substantiated.<br />
For example:<br />
Data elements in a database record may be:<br />
mandatory or not<br />
constrained to specific data types (e.g. date, text or number) and character lengths<br />
limited to a set of allowed values, and/or<br />
default values.<br />
There may also be complex editing rules. For example, if element A has a value of X then<br />
element B may be restricted to a particular set of values, etc.<br />
These attributes should be documented and included in validation testing.<br />
Sets of allowed values are often rationalised or updated in migration processes. These changes<br />
and the justifications for these changes should be documented in the migration process.<br />
Relationships between <strong>records</strong> can be constrained. They may be one to one, one to many,<br />
many to many or reflexive. For example, an employee may be the manager or supervisor of<br />
another employee. These constraints may be important in the design of the migration<br />
processes and the validation testing of the new system.<br />
Page 56 of 57
<strong>Migrating</strong> <strong>digital</strong> <strong>records</strong>: A guideline for Queensland public authorities<br />
F.3 Approaches to migrating <strong>digital</strong> <strong>records</strong> and the recordkeeping implications<br />
Within the planning phases for a migration project, a roll back strategy is highly recommended<br />
in order to mitigate any risks that may occur if unacceptable deficiencies, despite pre-migration<br />
testing, come to light after the live/production migration.<br />
The roll back strategy should itself be tested and validated before final migration takes place,<br />
and should include recordkeeping functionality testing.<br />
In addition, the following migration approaches require careful consideration and action planning by<br />
the <strong>records</strong> manager in collaboration with their IT colleagues:<br />
<br />
<br />
<br />
<br />
<br />
A cut over strategy could be planned. Ideally use of the source system should cease and<br />
the source system be frozen before migration commences. Business transactions may be<br />
suspended or recorded manually until the data is migrated and the new system is up and<br />
running.<br />
If the system is critical to the functioning of the business, a snapshot may need to be taken.<br />
Data additions, changes or deletions or functions performed must be logged so the changes<br />
can be repeated in the new system.<br />
Certain non critical functions in the source system might be suspended during the cut<br />
over period to simplify this process. For example, only new <strong>records</strong> may be allowed to be<br />
entered.<br />
Changes to existing <strong>records</strong> may be manually recorded and performed in the new<br />
system once the initial migration is validated and signed off.<br />
New <strong>records</strong> could be migrated to the new system in a second step to synchronise the<br />
two systems once the new system is signed off as functional and the source system frozen.<br />
Page 57 of 57