OAIS/METS Report (PDF) - UK Data Archive

More documents

Recommendations

Info

Ingest: TNA ASSESSMENT OF UKDA AND TNA COMPLIANCE WITH OAIS AND METS STANDARDS TNA receives submissions on TNA branded storage media (CD-ROMs), each of which carries a serial number. After producing a security copy, the received submission is loaded into a pre-accession quarantine area, which exists on an isolated network, where it is virus checked and checked by the records management and digital preservation department to ensure that the record is complete. The record will remain in quarantine for four weeks. No alterations to the record are made at this stage; the original bit-stream is regarded as an essential piece of evidence supporting the presumption of authenticity. However, record metadata may be updated to reflect the fact that it has been received and that the checksums of the digital objects match the transfer manifest. TNA cannot control the file formats in which data are delivered to them but it does control the quality of the associated metadata. Transfers without valid TNA metadata, represented as an XML file or a spreadsheet, which must include pre-assigned TNA catalogue references, are not accepted. TNA Ingest validation phase checks that: ■ the record is complete and valid; ■ all objects meet the descriptions and checksums in the transfer manifest; 33 ■ TNA Records metadata are present and complete to the level required at ingest; ■ all objects are certified as virus free; ■ objects have been delivered on approved media. Descriptive information (cataloguing) metadata are manually checked and amended if required. Once cataloguing is complete, the record is handed to the Digital Preservation department. The record is copied from the preaccession server with its enhanced metadata and placed onto a loading client on another isolated network. The loading client once again validates and verifies that the data object it has received is correct, then will initiate an upload to the storage function (the Digital Archive system). The server performs additional validation checks and virus scans and, if the entire record is loaded successfully, a message will be returned signalling success. The record will appear in the archive under the catalogue references assigned in the metadata. It is not possible to overwrite material held by a current catalogue reference. All loaded material must have a unique reference which did not exist previously in the archive. The Web Archive issues reports listing those web sites that it has crawled successfully and are catalogued in PROCAT under the reference ZWEB. With the exception of the built-in reporting function at present, TNA ingest workflow matches the OAIS recommendation. The submission validation tools in some cases may be hand-crafted queries and scripts that extract the information required for validation. Most of the co-ordinate updates, data management and the archival storage functions are represented by a single application: the Digital Archive system. Similar to UKDA, TNA creates dissemination versions of the SIPs at the Ingest stage and stores them in a separate archive system. Initial migration is limited to the production of presentation surrogates (e.g., for easier delivery within a web browser without requiring proprietary software to view the record).
ASSESSMENT OF UKDA AND TNA COMPLIANCE WITH OAIS AND METS STANDARDS Ingest: Conclusions A key conclusion is that, for both organisations, the pre-ingest functions are considered to be essential for efficient and effective archiving. The OAIS reference model would serve the community better if it included this function rather than relying on the existence of a separate model. The strongest argument for this is that preingest activities are not without cost and they ensure quality, understandability and accessibility of information packages. It is a stated aim of this report that it will be of value to others who might want to undergo the process of validating their work against the model or to use the model in conjunction with the setting up of a digital archive. As it stands, the model does not, in the view of either the UKDA or TNA, incorporate the essential functions associated with pre-ingest activity and, as such, is leaving open the potential for problems later in the process. 34 Both institutions testing for OAIS compliance are institutions that have been functioning as archives for a long time and, therefore, have a detailed set of policies and operating procedural manuals that cover their work with depositors. Due to the nature of the information producer community and the nature of digital objects submitted to the archives, there are slight differences in the way the submissions are negotiated, received and processed by either institution. The main groups of functions identified in the OAIS and Producer-Archive Interface standards are all demonstrably present but, again, with some discrepancies in the order and level of detail of services. By far the most time-consuming stages in the ingest process are the negotiation phase with the depositor, the quality assurance (with in-depth validation of data and metadata), generate AIP (and DIP, in the case of these institutions), and generate descriptive information. One possible reason for the expenditure of time at the ingest stage is that both organisations have very clearly defined dissemination functions. The skilled researchers who form the majority of the UKDA user community expect meticulously detailed information about a DIP and thus the UKDA imposes stringent internal definitions on what is understood as ‘independently understandable’. From our reading of the reference model, it appears that OAIS assumes that DIPs are created on demand from AIPS. This is not the case for either the UKDA or TNA. For a short period in its history, the UKDA took this approach and preserved the DIP as a separate file, created on demand after the AIP. It soon became evident that this approach has serious drawbacks. This is because many of the UKDA AIPs are created by ephemeral research teams, making it difficult, as time passes, to contact members of these teams to resolve substantive and detailed queries identified during the creation of the DIP. It is also worth noting that in the past there was an assumption on the part of both the depositor and the UKDA that neither a DIP nor an AIP had to be completely understandable. This was evidenced in early versions of the Depositor Licence and the user agreement in which the user was required to notify the archive of errors in the data and, in turn, the archive was to inform the depositors of such notifications. In the case of TNA, digital records are frequently between five and ten years old by the time they are selected for archiving, and will normally be in old data formats. This makes it extremely important to validate that the records are in a technically usable state. By creating the DIP as part of the ingest process, it is possible to eliminate many such errors in co-operation with the depositor and thereby improve the understanding of the AIP. This is because the process of creating the DIP involves processes not essential to the production of the AIP but which are essential to ensuring that the AIP is independently understandable in the future. Essentially, the creator of the DIP is the first user of the AIP and only by using the AIP can one identify missing information at the detailed level required to meet the organisations’ high standard for the definition of independently understandable.
Page 1 and 2: C O N S ervation Planning Data Mana
Page 3 and 4: ASSESSMENT OF UKDA AND TNA COMPLIAN
Page 33: ASSESSMENT OF UKDA AND TNA COMPLIAN
Page 85 and 86:
ASSESSMENT OF UKDA AND TNA COMPLIAN
Page 87 and 88:
Page 89 and 90:
Page 91 and 92:
Page 93 and 94:
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
Page 101 and 102:
Page 103 and 104:
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
108 ASSESSMENT OF UKDA AND TNA COMP
show all

OAIS/METS Report (PDF) - UK Data Archive

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?