User Manual - Web Curator Tool - SourceForge

More documents

Recommendations

Info

Essential terminologyImportant terms used with the Web Curator Tool include:Web Curator Tool or WCT — a tool for managing the selectiveweb harvesting process.Target — a portion of the web you want to harvest, such as awebsite or a set of web pages. Target information includescrawler configuration details and a schedule of harvest dates.Target Instance — a single harvest of a Target that isscheduled to occur (or which has already occurred) at a specificdate and time.harvest or crawl — the process of exploring the internet andretrieving specific web pages.harvest result — the files that are retrieved during a harvest.seed or seed url — a starting URL for a harvest, usually theroot address of a website. Most harvests start with a seed andinclude all pages “below” that seed.harvest authorisation — formal approval for you to harvestweb material. You normally need permission to harvest thewebsite, and also to store it and make it accessible.permission record — a specific record of a harvestauthorisation, including the authorising agencies, the datesduring which permissions apply and any restrictions onharvesting or access.authorising agency — a person or organisation who authorisesa harvest; often a web site owner or copyright holder.Impact of the toolThe Web Curator Tool is used at the National Library of New Zealand,and has had these impacts since it was introduced into the existingselective web archiving programme:Harvesting has become the responsibility of librarians andsubject experts. These users control the software handling thetechnical details of web harvesting through their web browsers,and are much less reliant on technical support people.Many harvest activities previously performed manually are nowautomated, such as scheduling harvests, regulating bandwidth,generating preservation metadata.The institution’s ability to harvest websites for archivalpurposes has been improved, and a more efficient and effectiveworkflow is in place. The new workflow ensures material issafely managed from before it is harvested until the time itenters a digital archive.Web Curator Tool User Manual Version 1.6 Page 6 of 80
The harvested material is captured in ARC format which hasstrong storage and archiving characteristics.The system epitomises best practice through its use of auditing,permission management, and preservation metadata.How Does it Work?The Web Curator Tool has the following major componentsThe Control CentreThe Control Centre includes an access-controlled web interfacewhere users control the tool.It has a database of selected websites, with associatedpermission records and other settings, and maintains a harvestqueue of scheduled harvests.Harvest AgentsWhen the Control Centre determines that a harvest is ready tostart, it delegates it to one of its associated harvest agents.The harvest agent is responsible for crawling the website usingthe Heritrix web harvester, and downloading the required webcontent in accordance with the harvester settings and anybandwidth restrictions.Each installation can have more than one harvest agent,depending on the level of harvesting the organizationundertakes.Digital Asset StoreWhen a harvest agent completes a harvest, the results arestored on the digital asset store.The Control Centre provides a set of quality review tools thatallow users to assess the harvest results stored in the digitalasset store.Successful harvests can then be submitted to a digital archivefor long-term preservation.Web Curator Tool User Manual Version 1.6 Page 7 of 80
Page 1 and 2: User ManualVersion 1.6September 201
Page 3 and 4: Quality review with the browse tool
Page 5: System OverviewBackgroundMore and m
Page 9 and 10: Harvest AuthorisationsIntroductionW
Page 11 and 12: http://*.alphabetsoup.com/* —incl
Page 13 and 14: Harvest authorisation search pageTh
Page 15 and 16: Permissions — details of the auth
Page 17 and 18: Figure 7. Permissions tab11 Note th
Page 19 and 20: How to send and/or print a permissi
Page 21 and 22: icon from the Actions column. This
Page 23 and 24: TargetsIntroductionIn the Web Curat
Page 25 and 26: NominationAfter a Target has been c
Page 27 and 28: one character. So, for example, a s
Page 29 and 30: 5 Select a permission record (or re
Page 31 and 32: Access14 Click the Access tab.The A
Page 33 and 34: Users with permission to Approve Ta
Page 35 and 36: aborted — the harvest was manuall
Page 37 and 38: Target instance search pageYou mana
Page 39 and 40: The queue view is actually just a p
Page 41 and 42: Figure 17. View/Edit Target Instanc
Page 43 and 44: Target Instance Quality ReviewIntro
Page 45 and 46: Review this Harvest — Open a view
Page 47 and 48: Quality review with the prune toolT
Page 49 and 50: The display returns to the Harvest
Page 51 and 52: • If the URL is not in the crawl.
Page 53 and 54: 5. If the URL was not retrieved bec
Page 55 and 56: entered the URL incorrectly in the
Page 57 and 58:
g2_subView=core.UserLoginwill be ex
Page 59 and 60:
User Agent Switcher for FirefoxThe
Page 61 and 62:
Group search pageYou manage Groups
Page 63 and 64:
Enter general information about the
Page 65 and 66:
Figure 23. Access TabSave the compl
Page 67 and 68:
The In TrayIntroductionThe In Tray
Page 69 and 70:
Type Trigger RecipientTargetInstanc
Page 71 and 72:
User, Roles and AgenciesIntroductio
Page 73 and 74:
ReportsIntroductionThe Reports scre
Page 75 and 76:
The output includes the counts of a
Page 77 and 78:
Permission Request TemplatesTo be c
Page 79 and 80:
Target InstancesTargetsAuthorisatio
show all

User Manual - Web Curator Tool - SourceForge

Create successful ePaper yourself

Delete template?

Save as template?