Transfer of Data from Terabyte Disk to BMRC Archives

More documents

Recommendations

Info

$LATEX Macros for Typewriter Font Characters$

141 # and if there were, then re-start the same data file part,142 # otherwise increment the part number an run that:143 #144 errCount=‘grep -c -i error ${LastFtpLog}‘145146 # Get the data file and part numbers from the previous run:147 #148 # Example:149 # nxt-run: 20041223,171054: Running part file: data1-p6.txt150 #151 LastRunLine=‘tail -1 nxt-run.log‘152153 # Note, AWK arrays are indexed starting from 1:154 #155 LastFileNum=‘echo $LastRunLine | awk ’{156 split($6,ary,"-")157 print substr(ary[1],5)158 }’ -‘159 LastPartNum=‘echo $LastRunLine | awk ’{160 split($6,ary,"-")161 split(ary[2],bry,".")162 print substr(bry[1],2)163 }’ -‘164165 if [ $warnCount -gt 0 ]166 then167 # There were some warnings, so send an email to the user:168 #169 LastPartFile="data${LastFileNum}-p${LastPartNum}.txt"170 export LastPartFile171 (172 echo "Data Transfer Warnings"173 echo "----------------------"174 echo "Please see log file: $LastFtpLog"175 echo176 echo "Data List Part File: $LastPartFile"177 echo "produced the following warning messages:"178 echo179 grep -C 3 -i warning $LastFtpLog180 ) | mail -s "Data Transfer Warnings" $MailTo181 fi182183 if [ $errCount -gt 0 ]184 then185 # There were some errors, but perhaps we have attempted re-running186 # the same data part file several times already (i.e., there may187 # be a non-existent file, such as "*.nc.bad", in the list):188 #189 PrevFtpLog=‘ls -1 ftp-*.log | tail -2 | head -1‘190191 result=‘cmp $PrevFtpLog $LastFtpLog‘192 stsOne=$?193194 if [ $stsOne -eq 0 ]195 then196 # The last two ftp log files are identical,197 # so lets try the one before that, too:198 #199 BeforeFtpLog=‘ls -1 ftp-*.log | tail -3 | head -1‘200201 result=‘cmp $BeforeFtpLog $LastFtpLog‘202 stsTwo=$?203204 if [ $stsTwo -eq 0 ]205 then206 # The last three ftp log files are identical,207 # so assume we have some unresolvable error,208 # and re-set the error count to zero209 # so that we can make some progress:210 #211 errCount=0212213 # And send some email to alert the user:214 #215 LastPartFile="data${LastFileNum}-p${LastPartNum}.txt"216 export LastPartFile217 (218 echo "Data Transfer Error Loop"219 echo "------------------------"220 echo "The last three ftp log files are identical,"221 echo "so assume we have some unresolvable error,"222 echo "and re-set the error count to zero"223 echo "so that we can make some progress:"224 echo225 echo "Please see FTP log files:"226 echo " $BeforeFtpLog"227 echo " $PrevFtpLog"228 echo " $LastFtpLog"229 echo230 echo "Data List Part File: $LastPartFile"231 echo "produced some error messages:"232 echo233 grep -i error $LastFtpLog234 echo235 ) | mail -s "Data Transfer Error Loop" $MailTo236 fi237 fi238 fi239240 if [ $errCount -gt 0 ]241 then242 # There were some errors, so re-run the same data file part:243 #244 PartFile="data${LastFileNum}-p${LastPartNum}.txt"245 else246 # No errors, that’s good. Run the next part:247 #248 NewPartNum=‘expr $LastPartNum + 1‘249 PartFile="data${LastFileNum}-p${NewPartNum}.txt"250 fi251252 if [ ! -f ${PartFile} ]253 then254 echo "${Prog}: ${StartHms}:" >> $LogFile255 echo " WARNING: ${PartFile}: file does not exist." >> $LogFile256257 # Try the next file number and reset part to 1:258 #259 NewFileNum=‘expr ${LastFileNum} + 1‘260 PartFile="data${NewFileNum}-p1.txt"261262 if [ ! -f ${PartFile} ]263 then264 echo "${Prog}: ${StartHms}:" >> $LogFile265 echo " WARNING: ${PartFile}: file does not exist." >> $LogFile266267 # Try the next file number with the same part number,268 # e.g., "data10-p92.txt" followed by "data11-p92.txt":269 #270 PartFile="data${NewFileNum}-p${LastPartNum}.txt"271272 if [ ! -f ${PartFile} ]273 then274 echo "${Prog}: ${StartHms}:" >> $LogFile275 echo " ERROR: ${PartFile}: file does not exist." >> $LogFile276 echo " Stopping ’crontab’ entry for $MyUser." >> $LogFile277 crontab -r278 exit 3279 fi280 fi281 fi282283 echo "${Prog}: ${StartHms}:" >> $LogFile284285 if [ $errCount -gt 0 ]286 then287 echo " Re-running part file: ${PartFile}" >> $LogFile288 else289 echo " Running part file: ${PartFile}" >> $LogFile290 fi291292 # Also need to record this in the "nxt-run.log" file:293 #294 NxtRunLog="nxt-run.log"295 StartDateTime="${StartYmd}-${StartHms}"296297 # NOTE:298 # Please DO NOT change the format of the following line,299 # this script depends on "${PartFile}" being the 6th argument:300 #301 echo "${Prog}: ${StartDateTime}: Running part file: ${PartFile}" >> $NxtRunLog302303 cp $PartFile $RemDFlist304305 NewFtpLogFile="${RunPath}/ftp-${StartDateTime}.log"306307 ## LD_LIBRARY_PATH=/home/strandwg/data/tcl8.5a2/lib308 ## export LD_LIBRARY_PATH309310 ## Tcl85sh=/home/strandwg/data/tcl8.5a2/bin/tclsh8.5311 ## export Tcl85sh312313 # Finally, run the selected part of the data file number:314 #315 ## ( umask 002; ${Tcl85sh} ${RunPath}/afm-ftp.tcl 2>&1 ) > $NewFtpLogFile &316 ##317 ( umask 002; tclsh ${RunPath}/afm-ftp.tcl 2>&1 ) > $NewFtpLogFile &318319 exit 0320B.10 Program: ‘gwstats’The ‘gwstats’ script runs a few commands which provide a simple snapshot view of the status of themultiple-terabyte file transfer process from a LaCie Terabyte disk on a Linux PC to the ‘gwork’ temporarystorage area on host ‘gale’:1 #!/bin/sh2 #3 # Program:4 # gwstats5 #6 # RCS-Strings:7 # $Source: /bm/gkeep/lih/src/sh/RCS/gwstats,v $8 # $Revision: 1.1 $9 #10 # Author:11 # Lawson Hanson, 20050209.12 #13 # Purpose:14 # Run a few commands which provide a simple snapshot view of15 # the status of the multiple-terabyte file transfer process16 # from a LaCIE Terabyte disk on a Red Hat Linux PC to gwork.17 #18 Prog=‘basename $0 .sh‘1920 MyUser=${USER:-${LOGNAME:-nobody}}2122 case $MyUser in23 afm)24 runDir=/bm/gkeep/afm/bmrc/lacie124
25 gworkDir=/bm/gwork/afm/bmrc/lacie126 ;;27 lih)28 runDir=/bm/gkeep/lih/bmrc/afm/p0329 gworkDir=/bm/gwork/lih/bmrc/afm/p0330 ;;31 *)32 echo33 echo "${Prog}: Error: Unknown user $MyUser"34 echo35 exit 136 ;;37 esac3839 cd $runDir4041 echo4243 ls -lrt | tail44 echo4546 tail nxt-run.log47 echo4849 date50 echo5152 echo "Storage space used on ’gwork’ in 1k blocks:"53 du -k -s $gworkDir54 echo5556 if [ ! -f nxt-run.log ]57 then58 echo59 echo "${Prog}: Error: nxt-run.log: file not found"60 echo61 exit 162 fi6364 LastRunPart=‘tail -1 nxt-run.log | awk ’{print $6}’ -‘6566 if [ ! -f $LastRunPart ]67 then68 echo69 echo "${Prog}: Error: ${LastRunPart}: file not found"70 echo71 exit 172 fi7374 echo "Number of files in this part: ‘grep -c ’\.nc$’ $LastRunPart‘"7576 LastFtpLog=‘ls -1 ftp-*.log | tail -1‘7778 if [ ! -f $LastFtpLog ]79 then80 echo81 echo "${Prog}: Error: ${LastFtpLog}: file not found"82 echo83 exit 184 fi8586 echo "Number of files already have: ‘grep -c -i ’already have’ $LastFtpLog‘"8788 echo "Number of files transferred.: ‘grep -c ’get file’ $LastFtpLog‘"89 echo9091 echo "Information from LastFtpLog.: $LastFtpLog"92 echo9394 numErrs=‘grep -c ’[Ee][Rr][Rr][Oo][Rr]’ $LastFtpLog‘9596 if [ $numErrs -gt 0 ]97 then98 isAre="is"99 msgWord="message"100101 if [ $numErrs -gt 1 ]102 then103 isAre="are"104 msgWord="messages"105 fi106 echo "There ${isAre} $numErrs FTP error ${msgWord}"107 echo108 fi109110 exit 0111B.11 Program: ‘get-lsR-ents’The ‘get-lsR-ents’ script can be used to extract particular entries from an ‘ls -R’ files list. See section6 on page 5 for an example of how to use this script to extract the daily (i.e., ‘/da/’) list of directoriesand associated files from a data list such as ‘data1-p1.txt’:1 #!/bin/sh2 #3 # Program:4 # get-lsR-ents5 #6 # RCS-Strings:7 # $Source: /bm/gkeep/lih/src/sh/RCS/get-lsR-ents,v $8 # $Revision: 1.2 $9 #10 # Author:11 # Lawson Hanson, 20050317.12 #13 # Purpose:14 # Given a defining string, extract the corresponding entries from an15 # FTP site’s "ls -R" command (or an "ls -lR" shell command) listing.16 # The script uses the technique of a simple finite state machine.17 #18 # Example:19 # To extract the list of "/da/" (i.e., daily) data files which need20 # to be further processed by the "cut-to-oz" script:21 #22 # % cd /bm/gwork/lih/bmrc/afm/p0323 # % ls -R data1 > gw-data1-p1.list24 # % get-lsR-ents ’\/da\/’ gw-data1-p1.list > gw-data1-p1-da.list25 #26 Prog=‘basename $0 .sh‘2728 usage ( )29 {30 echo31 echo "Usage: ${Prog} pattern file"32 echo33 }3435 case $1 in36 -h)37 usage38 exit 039 ;;40 *)41 ;;42 esac4344 pattern=$145 shift4647 awk ’BEGIN {48 gotMatch = 049 }5051 # Keep any comment ("#") lines that exist in the input file:52 #53 /^ *#/ {54 print55 next56 }5758 # Use directory entries to turn output listing "off":59 #60 /: *$/ {61 gotMatch = 062 }6364 # Use directory entries which match the specified pattern65 # to turn the output listing "on":66 #67 /’${pattern}’.*: *$/ {68 gotMatch = 169 }7071 # Ignore the "total" lines:72 #73 /^total / {74 next75 }7677 {78 if (gotMatch > 0) {79 print80 }81 }’ $*8283 exit 084B.12 Program: ‘cut-to-oz’The ‘cut-to-oz’ script is run on a list of huge daily data files to extract a portion of the data whichis relevant to Australian researchers. That sub-set of the data, (details of which was determined byDr.Aurel Moise), is that which is bounded by the region specified by latitudes 60 degrees South to 20degrees North, and longitudes 90 degrees East to 270 degrees East. This data set extraction is performed25
Page 1 and 2: Transfer of Data from Terabyte Disk
Page 3 and 4: 3 Initial Setup On ‘gale’The in
Page 5 and 6: 5 Consistency Checks Of The Transfe
Page 7 and 8: 8 Removing The Intermediate DataWhe
Page 9 and 10: References[1] A.V. Aho, B.W. Kernig
Page 11 and 12: 405406 A few minutes later:407408 %
Page 13 and 14: 881882 % cd /bm/gwork/lih/bmrc/afm/
Page 15 and 16: 8384 print STDERR "\n";85 print STD
Page 17 and 18: 203 print "${Prog}: Reference list
Page 19 and 20: 75 fi7677 if [ ! -f "${inFile}" ]78
Page 21 and 22: 203 exit204 }205 }206207208 # p G e
Page 23: 103 echo " that it already has all
Page 27 and 28: 45 fi4647 currDir=‘pwd‘4849 uma

Transfer of Data from Terabyte Disk to BMRC Archives

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?