Transfer of Data from Terabyte Disk to BMRC Archives

More documents

Recommendations

Info

$LATEX Macros for Typewriter Font Characters$

Finally, ‘auto-nxt-run.config’:% tail $HOME/auto-nxt-run.config#MailTo="l.hanson@bom.gov.au"export MailToRunPath="/bm/gkeep/lih/bmrc/afm/p03"export RunPathRemDFlist="remoteDirFileList.txt"export RemDFlistYou will need to change the ‘MailTo’, and ‘RunPath’ entries. The ‘RunPath’ item is the ‘gale’directory you have set up to hold the scripts and data lists, and from which you will run the scriptswhich automate the various processes of this work.5. Create, or take a copy of the ‘listalldirs.txt’ file (used by scripts like ‘list-size.pl’), whichis an ‘ls -lR’ format listing of all of the files available on the PCMDI data portal.4 Transfer Data From Terabyte Disk To ‘gale’Data is transferred from the TB disk (connected on the Linux PC) over to a staging area on ‘gale’(rather than directly accross to the long term archive storage system) because some files (i.e., huge dailydata files) need to be further processed to extract a sub-set of the data they contain (to reduce the finalamounts of storage required). The steps required in this data transfer process are:1. Partition the data list file (e.g., ‘data?.txt’), if required, into a series of more manageable part listfiles:% ~lih/bin/df-part.sh data1.txtThis creates a series of files named like ‘data1-p1.txt’, ‘data1-p2.txt’, ‘data1-p3.txt’, etc.Or if the data list file is deemed small enough to handle in one chunk, then make a copy of it withan appropriate part number, for example:% cp data1.txt data1-p1.txt2. Use the ‘list-size.pl’ script to determine the total size of the netCDF files in the data list file(or part data list file), for example:% list-size.pl data1-p1.txtlist-size: List number of netCDF files: 55list-size: List total netCDF file size: 49394698640 => ( 46 GB)list-size: File sizes were obtained by: SymLinks: 55list-size: Symbolic Links: file count: 553. Start the transfer of data from the terabyte disk to ‘gale’ by running the ‘nxt-run’ script with theappropriate arguments. For example, to commence the transfer of the files listed in the part file‘data1-p1.txt’, you would run the command:% nxt-run 1 1This process can take a considerable amount of time. For example, when connected to a USB-1port, the transfer of approximately 46 gigabytes of data took just over seventeen (17) hours, hence,the effective data transfer rate was about 0.8 megabytes per second.Soon after we started, we replaced the USB port with a FireWire (i.e., IEEE 1394) interface, and thisconsiderably reduced the data transfer times, producing data transfer rates of about 11.5 megabytesper second, so a data transfer of 46 GB now takes approximately sixty eight (68) minutes.4
5 Consistency Checks Of The Transferred DataCheck that the files transferred from the terabyte disk to the ‘/bm/gwork/lih/bmrc/afm/p03’ directoryon ‘gale’ are all there, and are of the correct size.1. Collect a list of the files that now reside in the destination directory on ‘gale’:% cd /bm/gwork/lih/bmrc/afm/p03% ls -lR > gw-data1-p1.list% cd /bm/gkeep/lih/bmrc/afm/p03% cp /bm/gwork/lih/bmrc/afm/p03/gw-data1-p1.list .Other users will probably need to substitute their own directory path names and file names in placeof the ones shown here.2. Run the ‘list-size.pl’ script to display the size of the files listed in the data part list file:% ~/lih/bin/list-size.pl data1-p1.txtlist-size: List number of netCDF files: 55list-size: List total netCDF file size: 49394698640list-size: File sizes were obtained by: Reference: 553. Run the ‘list-size.pl’ script to display the size of the files listed in the ‘ls -lR’ list file of thedata now residing in the ‘gwork’ area:% ~/lih/bin/list-size.pl gw-data1-p1.listlist-size: List number of netCDF files: 55list-size: List total netCDF file size: 49394698640list-size: File sizes were obtained by: List: 55It should be apparent that the set of data has been transferred to ‘gale’ successfully, with all files presentand of the correct size. If the two total netCDF file size items are different, then you should check furtherto discover what has changed, and perhaps re-run ‘nxt-run’ on the same data file part list again.The words ‘Reference’, and ‘List’ following the phrase ‘File sizes were obtained by’ in the outputof the ‘list-size.pl’ command show where the individual file size numbers were obtained to calculatethe ‘List total netCDF file size’ figure.The ‘data1-p1.txt’ list is of ‘ls -R’ form, hence the file sizes are obtained from the reference list (i.e.,‘listalldirs.txt’), but the ‘gw-data1-p1.list’ file is an ‘ls -lR’ list which contains its own file sizeinformation, hence the ‘list-size.pl’ script uses that information to caculate its total.6 Extracting a Sub-Set of Huge Daily Data FilesIf the data transferred to the ‘gwork’ area on ‘gale’ contains any daily data files, then we run anotherscript called ‘cut-to-oz’ on those daily data files (using another cut-down ‘ls -R’ list which containsonly entries for those daily files upon which we wish to operate).1. Use the ‘get-lsR-ents’ script to extract a list of the daily data files that need to be processed:% cd /bm/gwork/lih/bmrc/afm/p03% get-lsR-ents ’\/da\/’ gw-data1-p1.list > gw-daily.listNotice that the backslash character (’\’) must be used to escape or hide the meaning of characters(like ‘/’) which are considered special by AWK[1].5
Page 1 and 2: Transfer of Data from Terabyte Disk
Page 3: 3 Initial Setup On ‘gale’The in
Page 7 and 8: 8 Removing The Intermediate DataWhe
Page 9 and 10: References[1] A.V. Aho, B.W. Kernig
Page 11 and 12: 405406 A few minutes later:407408 %
Page 13 and 14: 881882 % cd /bm/gwork/lih/bmrc/afm/
Page 15 and 16: 8384 print STDERR "\n";85 print STD
Page 17 and 18: 203 print "${Prog}: Reference list
Page 19 and 20: 75 fi7677 if [ ! -f "${inFile}" ]78
Page 21 and 22: 203 exit204 }205 }206207208 # p G e
Page 23 and 24: 103 echo " that it already has all
Page 25 and 26: 25 gworkDir=/bm/gwork/afm/bmrc/laci
Page 27 and 28: 45 fi4647 currDir=‘pwd‘4849 uma

Transfer of Data from Terabyte Disk to BMRC Archives

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?