Transfer of Data from Terabyte Disk to BMRC Archives

B.6 Program: ‘afm-ftp.tcl’The ‘afm-ftp.tcl’ script, although similar to that which has already been described before, is includedhere because it contains some changes which were necessary to enable it to run with the different versionof Tcl/Tk that we have at BMRC (compared to what is in use at NCAR). This script reads a specially(‘ls -R’) formatted list of directory paths and file names, and then connects via FTP to the Linux PCwhich (in this case) has the TB disk connected to it, and executes an FTP ‘get’ operation to copy eachfile over to the current host:1 # # #!/bin/sh2 #3 # Program:4 # afm-ftp.tcl5 #6 # Author:7 # Lawson Hanson, 20041220.8 #9 # Purpose:10 # Helps Dr.Aurel Moise to download and save data files using ftp11 # to connect from NCAR to PCMDI and get files back to NCAR.12 #13 # Input:14 # A file which was constructed from an edited "ls -R" output.15 # Basically, the file needs to supply a remote directory path16 # and a list of files required to be downloaded from that path.17 #18 # Example Input:19 # --+----------20 # |data1/some/where/blah:21 # |file1.nc22 # |file2.nc23 # |file3.nc24 # |data1/some/other/blah:25 # |file1.nc26 # |file2.nc27 # +----------28 #29 # Action:30 # Using the supplied data path, the program should:31 #32 # 1. Construct the asociated directories on the local machine,33 # 2. Open an FTP session to the remote host machine34 # 3. Get the specified list of files from the remote host (PCMDI)35 # back to the local host (NCAR) and store them in the named36 # location (as per the supplied data path)37 # 4. Close the FTP connection.38 #39 # Assumptions:40 # 1. The input text file will reside in the current directory41 # 2. The retrieved files will be located below a "tmp" directory42 # in the current directory of the local host machine (NCAR)43 #44 # Updates | By | Description45 # --------+----+------------46 # 20050111| LH | Modified "pGetDataFiles" to open/close FTP for each file,47 # | | because the FTP connection kept having time out errors.48 # 20050120| LH | Modified to read configuration data from a file.49 # 20050124| LH | Added the $openPortal variable to track ftp::Open50 # 20050125| LH | Modified to respond to request to cease operation.51 #52 # NOTE:53 # Do NOT remove the line with the ’#’ followed by ’\’,54 # nor separate it from the following ’exec wish ...’ line.55 # They are interpreted by the Tcl/Tk interpreter, "wish",56 # as a comment which continues on the next line, whereas57 # the shell simply executes the ’exec wish "$0" "$@" line.58 #\59 # exec tclsh "$0" "$@"6061 # The name of a Tcl/Tk script is held in the special variable "argv0"62 #63 set Prog [exec basename $argv0 .tcl]6465 # package require Tcl 8.566 # package require ftp676869 # Our version of Tcl/Tk is older than the required one,70 # hence, I downloaded the older "FTP" library code,71 # and have used "source" instead ... and needed to72 # change lowercase "ftp::..." to uppercase "FTP::...",73 # and remove most references to "$conn" in my code:74 #75 source /bm/gkeep/lih/src/tcl/ftp_lib1.2/ftp_lib.tcl767778 # Read in the configuration file:79 #80 set ConfigFile $env(HOME)/${Prog}.config8182 if [file exists $ConfigFile] {83 source $env(HOME)/${Prog}.config84 } else {85 puts stderr "${Prog}: ERROR: Configuration file not found"86 exit 487 }888990 # Set directory path and make if necessary:91 #92 set MyTmpDir [file nativename [file join $LocTopDir "tmp"]]9394 if {[file exists $MyTmpDir] == 0} {95 file mkdir $MyTmpDir96 }979899 # p R e a d D i r L i s t100 #101 # Purpose:102 # Read a list of directory paths and associated files103 # from a plain text file with entries such as:104 #105 # Example Input:106 # --+----------107 # |data1/some/where/blah:108 # |file1.nc109 # |file2.nc110 # |file3.nc111 # |112 # |data1/some/other/blah:113 # |file1.nc114 # |file2.nc115 # +----------116 #117 # Action:118 # Process each line:119 # 1. Ignore ("#") comments,120 # 2. Determine directory path,121 # 3. Split and use "mkdir" if required122 # to make local host directory path,123 # 4. Store list of files names124 #125 proc pReadDirList { } {126 global Prog127 global LocTopDir128 global ListFile129 global DirPath130 global FileName131 global NumDirs132 global NumFiles133134 set NumDirs 0135 set NumFiles($NumDirs) 0136137 if [file exists $ListFile] {138 if [catch {open $ListFile r} fileId] {139 puts stderr "${Prog}: ERROR: Opening $ListFile: $fileId"140 } else {141 while {[gets $fileId line] >= 0} {142 #143 # Ignore ("#") comments:144 #145 set sts [regexp {^ *#} $line match]146147 if {$sts > 0} {148 continue149 }150 # Ignore ("^ *$") blank lines:151 #152 set sts [regexp {^ *$} $line match]153154 if {$sts > 0} {155 continue156 }157 # Determine directory path:158 #159 set sts [regexp {run[1-9][0-9]*: *$} $line match]160161 if {$sts > 0} {162 set dirNum $NumDirs163 set NumFiles($dirNum) 0164 set DirPath($dirNum) [string trim $line { :}]165166 # Use "mkdir" to make local host directory path:167 #168 set locDir [file nativename [file join \169 $LocTopDir $DirPath($dirNum)]]170171 if {[catch {exec mkdir -p $locDir} result]} {172 puts stderr "${Prog}: ERROR(mkdir): $result"173 exit174 }175 incr NumDirs176 continue177 }178179 # Store list of netCDF (".nc") files names:180 #181 set sts [regexp {\.nc *$} $line match]182183 if {$sts > 0} {184 set FileName($dirNum,$NumFiles($dirNum)) [string trim $line]185 incr NumFiles($dirNum)186 continue187 }188 # Should only get here on unrecognized data list entries:189 #190 puts stderr "${Prog}: WARNING: Unrecognized Data List Entry:"191 puts stderr " Directory: $DirPath($dirNum)"192 puts stderr " Bad Entry: $line"193 }194 close $fileId195196 if {$NumDirs < 1} {197 puts stderr "${Prog}: WARNING: (${ListFile}) appears empty"198 exit199 }200 }201 } else {202 puts stderr "${Prog}: WARNING: (${ListFile}) file does not exist"20