ECMWF Data Handling System ECFS

ECMWF Data Handling System
ECFS
Anne Fouilloux
User Support
advisory@ecmwf.int
Slide 1

Content
• Introduction
• The ECFS client: a UNIX-like interface
• ECFS User commands
• Remarks
• Trouble-shooting
• Recommendations
• Status & Future plans
Slide 2
Slide 2

Introduction
•For many years (from 1983), ECMWF has operated a large-scale Data
Handling System (DHS), in which all ECMWF users can store and retrieve
data.
•The Data Handling System is composed of three main components:
- IBM’s High Performance Storage System (HPSS), used as the
underlying archiving system in which data is kept
- MARS - Meteorological Archive and Retrieval System
• GRIB and BUFR data, over 4.2 Petabyte of data (including backup copies),
more than 5 million files (approx. 1Gb/file) and about 3 TB stored every
day.
- ECFS - ECMWF’s File Storage system
• Any kind of data, 1400 TB of data (including backup copy), about 40 Million
files (~=23Mb/file) and between 2.5 and 3 TB stored daily
Slide 3
MARS and ECFS were developed by ECMWF to hide the complexities of HPSS and data
management from the users.
Slide 3

The ECFS client: a UNIX-like interface (1/2)
•A Unix style of file interface has been adopted by ECFS :
- Files are mapped to a Unix-compatible directory tree
- Either absolute and relative pathnames can be used
- Concept of current ECFS working directories ($ECDIR and
$EC_TMPDIR), analogous to the Unix current working
directory $PWD
- Limited wildcard support is provided. Wildcard characters are
NOT supported in ECFS DIRECTORY positions but CAN be
used for the rightmost (file) position. (For instance, you cannot
use els ec:t*/file*.out)
- The ECFS filesize limit is 16 GB. Be aware that certain Unix
Slide 4
systems (not at ECMWF) or software packages cannot handle
files over 2GB in size;
Slide 4

The ECFS client: a UNIX-like interface (2/2)
•But this is not a UNIX file system:
- Files are migrated off to tape(s) behind the scenes.
- There are (Network) overheads when files are transferred
to/from ECFS, UNLESS file is on disk cache (small and
recent data).
•ECFS commands: els, erm, ermdir, emkdir, ecd, epwd,
echmod, echgrp, ecp, emv, eumask and emove, ecfsdir,
ecfs_status
•ECFS commands are implemented as aliases/functions with the
execution mapping to Perl scripts. Environment is set up for both
Slide 5
C-shell and korn-shell users. Other shells are not supported!
Slide 5

Documentation & Availability at ECMWF
•ECFS commands are available on all our platforms (ecgate
and HPCF systems) except ecfs_status command for
monitoring your ECFS usage (available on ecgate only).
•Documentation is available on our website at
http://www.ecmwf.int/services/computing/docs/archives/ecfs/index.html
•ECFS man page:
man ecfs
•In addition there are man pages for each specific command
e.g.:
man els
Slide 6
Slide 6

ECFS domains
•ECFS files are stored in domains. Currently two domains exist:
ec: and ectmp:
- ec: is the permanent domain where files are stored indefinitely. This is the
default domain
- ectmp: is the temporary domain where files are stored for 90 days, after which
they are automatically deleted. Note that once a file has been automatically
deleted it CANNOT be recovered.
NB: Co-Operating states may ONLY use ectmp: domain.
•The domain names shown above (ec:, ectmp:) should used in
most of the ECFS commands to indicate which domain you are
working with.
•Note that, as an alternative, the ectmp: domain can be
referenced by ec:/TMP. Thus the following are equivalent:
- ectmp:/uid/newdir
- ec:/TMP/uid/newdir
Slide 7
Slide 7

ECFS user commands: Exploring the ECFS file system
•List (ls –al) ECFS files described by target:
els [-R]
•Change the current ECFS working directory for the specified
ECFS domain:
ecd
Target should be prefixed by an ECFS
domain either ec: or ectmp:
To list subdirectories recursively.
els can time out for very large ECFS directory trees. (see ecfs_audit)
Set the environment variables ECDIR
(ec:) or EC_TMPDIR (ectmp:)
NB: Defaults to login name of user if target omitted.
•Print name of the ECFS current working directory for the
specified ECFS domain:
Slide 8
epwd ec:
Or epwd ectmp:
Print the content of the environment
variables ECDIR (ec:) or EC_TMPDIR
(ectmp:)
Slide 8

Practical: Exploring the ECFS file system
•For this first practical, try the following commands on ecgate:
epwd ec:
epwd ectmp:
•Use els to list all the files contained in both domains:
els ec:
els ectmp:
• Change the working directories and use els to list their contents:
ecd ec:/trx
epwd ec:
els ec:
ecd ec:
Slide 9
Slide 9

ECFS user commands: Transferring files between ECFS
and client storage
Overwrite existing file
unconditionally
Create a backup copy in Disaster Recovery System (DRS)
Use sparingly, for files impossible/expensive to recreate.
Do not overwrite
if file exists.
Not an error
(DEFAULT)
Do not overwrite.
Treat as an error
if attempted
(Return 1)
file's timestamp, group and permission will be kept
(optional) destroy disk cache copy
after retrieval
ecp [-e|n|o|t] [-b] [-p] [-m]
emv [-e|n|o|t] [-b] [-p]
Either target or source should be prefixed by an
ECFS domain (ec: or ectmp:)
Overwrite only if target older Slide than10
requested source
(Time standards differ on local workstations and servers).
NB: emv is similar to ecp but files are removed after being transfered
Slide 10

Example: Transferring files between ECFS and client storage
> ecp $SCRATCH/my_file ectmp:Backup/Feb/ecfs_scratch_file
mk_ecdirs: Created ECFS directory ectmp:/TMP/trx/Backup
mk_ecdirs: Created ECFS directory ectmp:/TMP/trx/Backup/Feb
File: ecfs_scratch_file.EcFs_XfEr successfully moved
Transferred /scratch/ectrain/trx/my_file from ecgate to ECFS as
-> ectmp:/trx/Backup/Feb/ecfs_scratch_file
599894 bytes (599 Kbytes per second)
ecp returning 0
> emv ectmp:ecfs_scratch_file $SCRATCH/my_file
Transferred ectmp:/trx/ecfs_scratch_file from ECFS to ecgate as
-> /scratch/ectrain/trx/my_file
599894 bytes (599 Kbytes per second)
get: removed local source file /trx/ecfs_scratch_file
Slide 11
emv returning 0
Client storage
ECFS
Slide 11

Practical: ecp and emv
Client storage
ECFS
•Work in your $SCRATCH
cd $SCRATCH
•Make a copy of the practicals directory in your $SCRATCH
tar –xvf /scratch/ectrain/trx/ecfs_practicals.tar
•This will create a directory in your $SCRATCH containing the
data files for all the practicals
• Copy the files $SCRATCH/ecfs_practicals/data/file*.out in ectmp:
• Move the file ectmp:file1.out in your $SCRATCH
Slide 12
Slide 12

ECFS user commands: file deletion
erm
Target should be prefixed by an ECFS
domain either ec: or ectmp:
No client files are affected
>
erm ec:ecfs_scratch_file
It never prompt before any removal
Successfully deleted ec:/trx/ecfs_scratch_file
>
erm ec:test*
Successfully deleted ectmp:/trx/test1
Successfully deletec ectmp:/trx/test2
erm: 2 files deleted
Files are removed from ECFS with
a soft-delete: files will still be
kept for 28 days during which it
will be possible, on request, to undelete
any file that was deleted
by mistake. After that period any
removal will become permanent.
Slide 13
Contact us if you have large entire trees to remove
Slide 13

ECFS user commands: creation and removal of directories
•Make directory:
Creates all the
non-existing
parent directories
first
emkdir [-p] [-m octal_mode]
•Remove a specified empty directory:
ermdir
> emkdir –p ectmp:DIR1/DIR2/DIR3
mk_ecdirs: Created ECFS directory ectmp://TMP/trx/DIR1
mk_ecdirs: Created ECFS directory ectmp://TMP/trx/DIR1/DIR2
mk_ecdirs: Created ECFS directory ectmp://TMP/trx/DIR1/DIR2/DIR3
> ermdir ectmp:DIR1/DIR2/DIR3
Successfully deleted ectmp:/trx/DIR1/DIR2/DIR3
Specifies the mode to be used for new
directories. If not present, it uses ECFS
umask (775 by default)
Delete Slideempty 14 directories only
Slide 14

Examples
ECFS User Commands: changing permissions
•Change permissions: echmod octal_mode
> echmod 640 ec:myecdir
echmod: ec:myecdir now has mode 640
•Change the current ECFS eumask: eumask []
>
eumask 022
Only numerical values can be used as umasks.
The default umask for ECFS is set to 002.
One can directly define the ECUMASK environment
variable directly (without calling eumask).
•Change group of file(s): echgrp group
> echgrp mysecgrp ec:/uid/*
echgrp: ec:/uid/mydir1 now has group_id = mysecgrp
Slide 15
echgrp: ec:/uid/mydir2 now has group_id = mysecgrp
echgrp: ec:/uid/myfile1 now has group_id = mysecgrp
Slide 15

ECFS user commands: save or retrieve a complete UNIX
directory as one ECFS file
Date and time of last access
ecfsdir [-o] [-b] [-m|-a]
The date/time of last modification will
be used as time stamp. This is the
default for ecfsdir. Only meaningful at
retrieval.
> ecfsdir $SCRATCH/Results ectmp:Model/results_backup
/scratch/ectrain/Results
mk_ecdirs: Created ECFS directory ectmp:/TMP/trx/Model
file: results_backup.EcFs_XfEr successfully moved
Transferred /scratch/ectrain/trx/scratchdir/ecgate.139928/file287836/Results from ecgate to ECFS as
ectmp:/trx/Model/results_backup
341468160 bytes (18970 Kbytes per second)
ecp returning 0
Results directory saved
NB: ecfsdir uses cpio to “compact”the files.
Slide 16
Slide 16
Source orTarget should be
prefixed by an ECFS domain
either ec: or ectmp:
Results is a directory and all the files in
Results will be packed into a single file
called results_backup

ECFS user commands: save or retrieve a complete
UNIX directory as one ECFS file
> cat $HOME/ECFS/Results_1217.03Feb2006
Contents of the directory saved:
========================
./DIR1/DIR2/file1
./DIR1/DIR2/DIR3/file2
.
./DIRn/…/DIRm/filep
.
Name of the directory saved:
/scratch/ectrain/trx/Results
Ecfs backup in :
/trx/mp:Model/results_backup
Date : Fri Feb 3 12:19:04 GMT 2006
From : ecgate
Slide 17
This file is stored to in
$HOME/ECFS to give you
the list of files/directories
saved. However, you can
delete this file or move it
(it is not needed when
retrieved from ECFS).
Slide 17

Practical: ecfsdir
•Use ecfsdir to copy the content of the directory
$SCRATCH/ecfs_practicals/data in ec:mydata
>
ecfsdir $SCRATCH/ecfs_practicals/data ec:mydata
Client storage
ECFS
Faster than the
equivalent ecp
• Check the content your $HOME/ECFS (search for a file named
data_*25Feb2008).
>
cat $HOME/ECFS/data_*25Feb2008
• Then retrieve ec:mydata in your $SCRATCH/ecfs_practicals/mydata
>
>
ecfsdir ec:mydata $SCRATCH/ecfs_practicals/mydata
cd $SCRATCH/ecfs_practicals/mydata
Slide 18
Slide 18

ECFS user commands: renaming or moving files within
the same ECFS domain
emove [-o|t|n|e]
Source and target should be
prefixed by the same ECFS
domain (ec: or ectmp:)
>
emove ectmp:ecfs_file ectmp:DIR1/ecfs_fileFeb06
move: entered with cmd = emove –n ectmp:ecfs_file ectmp:DIR1/ecfs_fileFeb06
file: ecfs_file successfully moved
move: : /TMP/trx/ecfs_file renamed to /TMP/trx/DIR1/ecfs_fileFeb06
•DIR1 must exist!
Slide 19
•Not possible to move data between ec: and ectmp: domains
Slide 19

ECFS user commands: monitoring
•The ecfs_status command to be run on ecgate to get the most
recent usage by project account
Add this option to get a
detailed report per user
If omitted, print ECFS
status for you default
project account
ecfs_status [-h] [-u] [-a] [-d YYYYMMDD] [Project_name]
To display a help screen
to report summary for the
project account (default)
To print the ECFS status
around a given date (default
is the most recent)
•To get an overview on their ECFS usage, you can also refer to
the audit files ec:ecfs_audit and/or ectmp:ecfs_audit.tmp which
are created once per month and contain a complete list of a
Slide 20
user's files in each ECFS domain.
Slide 20

Examples: ECFS monitoring
•Running ecfs_status on ecgate:
ecfs_status
Client storage
ECFS
ECFS status on 20070110 for my_acc
Account my_acc Total: 963713 MB –135942 files Transfer previous month: 499690 MB –3613 files
Total: 963713 MB –135942 files Transfer previous month: 499590 MB - 3613 files.
•To read ecfs_audit or ecfs_audit.tmp, you need first to copy them locally (these two
files don’t exist for new accounts; they will be created after the first month):
>
ecp ec:ecfs_audit $SCRATCH/ecfs_audit
cat $SCRATCH/ecfs_audit
>
-- uid gid size(bytes) creation last_access path today=2006-01-09
* trx ectrain 1945665 2005-12-16 2005-12-16 /trx/test1
* trx ectrain 1305088 2005-12-16 2005-12-16 /trx/test2
…
Slide 21
Total files =20 megabytes = 116.864808082581
total directories = 2 total files not accessed since 20040708 = 0
Slide 21

Backup support
• No automatic backup copy is made of ECFS data. Specify the “-b”
option on the ECFS commands (ecp, emv, ecfsdir) to request a backup
copy to be made:
ecp –b myfile ec:essential_data
emv –b myfile ec:essential_data
ecfsdir –b $SCRATCH/results ec:essential_directory
• The existence of backup copies will be indicated by the first character
of the line listing:
b for files with a secondary copy
br--r----- 1 uid group 100 Nov 19 2003 essential_data
-rw-rw---- 1 uid group 512 Nov 19 2003 unimportant_data
- for files with no secondary copy
Slide 22
• NOTE: Irrespective of the existence of backup copies: any ECFS files
removed (deleted) by a user can only be recovered for a limited period
of time (28 days)
Slide 22

Recommendations
•Do not copy in/out the same files frequently. Use temporary local disk
space such as $SCRATCH (on ecgate) or $TEMP (on hpce) to keep a
local copy of these files (by default ecp will not overwrite a file if it exists;
do not use the –o option in that case)
•Create fewer larger files rather than many small files otherwise it can
adversely affect performance of the whole system for everybody
(overheads for EACH transfer).
•Keep files locally THEN “compact”them using ecfsdir or cpio or tar and
ONLY then store them into ECFS.
•Use ectmp: if files do NOT need to be kept for long periods.
•Delete files which you do not need in ec:. As there is no recursive
delete, you may contact us if you need to delete entire trees.
Slide 23
•Never use ECFS commands in parallel jobs on hpce (in serial jobs only)
Slide 23

ECFS within scripts (1/3)
• ECFS commands within scripts (tcsh, csh, sh, ksh, ksh93, bash):
- only C-shell and korn-shell supported
- you should NOT use csh –f which will not define aliases.
- When ECFS commands are not found, first check your environment:
• if you are a ksh user:
> echo $FPATH
/usr/local/share/ecmwf/share/ecksh_functions
• if you are a csh user:
> which els
els: aliased to (set noglob; $ECFS_SYS_PATH/els.p !{*};)
• ECFS commands within cron(tab) tasks: cron tasks do not execute
.profile, .cshrc, .login etc. Hence neither paths nor aliases would be setup.
We advice you to use our time critical framework (See Dominique’s
Slide 24
presentation Friday morning) instead of cron tasks.
Slide 24

ECFS within scripts (2/3)
• Error Handling: the following techniques are suggested for trapping
ECFS error codes when running batch scripts in the Korn shell
environment
- Set a trap for the entire script:
#!/bin/ksh
trap " echo ECFS call exited with RC= \$? " ERR
ecp nofile ec:
- or catch any errors on each call:
#!/bin/ksh
set +e
ecp nofile ec:
RC=$?
set -e
if [ $RC -gt 0 ]
then
Slide 25
echo " ECFS call exited with RC= $RC"
fi
To avoid the script to stop after the first error
Check the code returned by the ECFS command
Slide 25

ECFS within scripts (3/3)
•Check existence of a local copy before getting ECFS version of file:
#!/bin/ksh
if [ ! -r $SCRATCH/file2.out ]; then
ecp ectmp:file2.out $SCRATCH/.
fi
•Loop over ECFS directories to change mode
#!/bin/ksh
ECFSdir=ec:/$USER/TESTDIR-1
ECFSprefix=`dirname $ECFSdir`; dirs=`basename $ECFSdir`
while [ -n "$dirs" ]; do To skip the four first lines
newdirs=""
for dir in $dirs; do
for list in `els ${ECFSprefix}/$dir | tail -n +4 | tr -s ' ' ‘|'`; do
name=`echo $list | cut -f 9 -d ‘|'`
echmod 775 ${ECFSprefix}/${dir}/$name
newdirs="$newdirs ${dir}/$name"
done
Slide 26
done
dirs=$newdirs
done
Slide 26
Substitute space by |
Keep the 9 th field which
is the last one

Status & Future plans
•An ECFS access control system is now in place to manage and
control ECFS workload
•Recursive commands (erm, echmod) will be provided. We enabled
"soft delete" of files in preparation for the introduction later of
recursive delete of entire trees.
•The ECFS client part will be rewritten
•To match the Supercomputer capacity the following aspects still
have to be studied
- The amount of daily transfers of data in/out from ECFS
(not retransferring the same file in/out helps us a lot)
Slide 27
- The growth in the number of physical tapes/disk cache
Slide 27