Training Course - ComIntro - ECaccess

ECaccess 

A portal to access ECMWF 

Dominique Lucas 

1 Com Intro training course – 2005 ECaccess 

ECMWF

Content 

• ECaccess concepts 

• telnet, ssh and ftp access 

• X11 access 

• ECaccess file management 

• ECaccess job management 

• Unattended file transfers 

• Tutorial 


ECMWF

ECaccess - Concepts 

• ECaccess provides a portal to access ECMWF 

archiving and computing facilities 

• Strict authentication via SecurID card and 

X509 certificates 

• Data integrity/confidentiality guaranteed by SSL 


ECMWF

ECaccess - Concepts 

• ECaccess provides: 

File and job management in batch or interactive 

mode through an extended FTP server 

File and job management through a Web browser 

A secure telnet access to ECMWF 

A secure X Windows access to ECMWF 

• All this via internet or RMDCN. 


ECMWF

ECaccess - Architecture 

•Multi tiers 

ECaccess Client, ECaccess Gateway, ECaccess Server, 

ECaccess ecgate 


Tools 

MS 

Work- 

station 

ECcert 

Command 

FTP 

Client 

Telnet 

Client 

Web 

Browser 

FTP 

Server 

X 

Server 


Gateway 

 

MS 

Gateway 

I 

N R 

T M 

E D 

R C 

N N 

E 

T 


Server 

(Java 

app) 

JSSE-SSL 

OpenSSL-SSL 

ECproxy 

Server 

(C app) 

ECMWF 

Gateway 



F 

I 

R 

E 

W 

A 

L 

L 

ECcmd 

Server 

(Java app) 

ECios 

Server 

(C app) 


CA 

(OpenSSL) 

Ecgate 

ECMWF


• ECaccess client 

Standard Telnet, FTP, ssh and HTTP/S client 

Standard X Windows and FTP Server 

ECaccess tools (eccert …) 


ECMWF


• ECaccess gateway 

Entry point for all ECaccess users 

Verifies ECaccess users authentication 

(certificates or passcodes) 

Implements Telnet, FTP and HTTP/S protocols 

Secure tunnels through firewalls to ECaccess 

Server 

Either local gateway installed at your site or 

ecaccess.ecmwf.int for internet and 

msaccess.ecmwf.int for RMDCN. 


ECMWF

ECaccess – Architecture 

• ECaccess Server 

Located at ECMWF: ecaccess.ecmwf.int or 

msaccess.ecmwf.int 

Entry point for all ECaccess gateways 

Authenticate ECaccess gateways 

Provides job and file management functions 

Provides functions to monitor file transfers 

Provides system access functions 

Keeps track of users activity 


ECMWF


• ECaccess ecgate 

Runs the ECaccess Certificate Authority (CA) 

Provides job management through LoadLeveler 

(and NQS). 

Provides access to home, scratch and ECFS 

Keeps job input, output and error files 

Manages spool for unattended file transfers 


ECMWF

ECaccess – local gateways 

• Use local ECaccess gateways if installed 

• Internet ECaccess gateways available at most NMSs 

and at several other sites (29 gateways in total) 

• 12 ECaccess Gateways available over RMDCN 

• If local ECaccess gateway not installed, then use 

ecaccess.ecmwf.int or msaccess.ecmwf.int 


ECMWF

ECaccess – telnet/ssh 

{mshost}$ telnet ecaccess.ecmwf.int 

… 

login: xyz 

Passcode: XXXXXX 

Which system: ecgate 

… 

ecgate{~xyz}:1 

{mshost}$ ssh xyz@ecaccess.ecmwf.int 

… 

login: xyz 


Which system: ecgate 

ecgate1{~xyz}:1 

• Where xyz is your ECMWF User ID 

• On local gateways, port 9023 (telnet) or 9022 (ssh) may be needed. 


ECMWF

ECaccess - ftp 

{mshost}$ ftp ecaccess.ecmwf.int 

… 

login: xyz 


… 

ftp> ls 

227 Entering Passive Mode. (193,61,196,110,135,230) 

150 Opening ASCII mode data connection 

drwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00 ECJOBS 

drwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00 ECSCRATCH 

drwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00 ECTMP 

drwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00 ECFS 

drwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00 ECHOME 

226 Transfer complete 

ftp> 

• sftp not yet available. 

• On local gateways, port 9021 may be needed. 


ECMWF

ECaccess - ftp 

Local 

environment 

Local file 

can be 

dragged 

and 

dropped 

into 

ecaccess 

area. 


ftp entry for 

batch access 

ECMWF

ECaccess – X11 via telnet 

{mshost}$ echo $DISPLAY 

mshost:0.0 

{mshost}$ xhost +ecaccess.ecmwf.int 

ecaccess being added to access control list 

{mshost}$ telnet ecaccess.ecmwf.int 

… (Passcode validation required) 

Which host …: ecgate 

Which proxy … : X 

… 

ecgate{~xyz}:1 echo $DISPLAY 

(Something like … )ecaccess.ecmwf.int: 

ecgate{~xyz}:2 xcdp& 

• A control window showing the DISPLAY to be used will appear. 

This window or an X11 application should remain present to 

keep the X11 proxy. 


ECMWF

ECaccess – ecxterm via ssh 

{mshost}$ echo $DISPLAY 

mshost:0.0 

{mshost}$ ssh –X xyz@ecaccess.ecmwf.int 

… (Passcode validation required) 

Which host …: ecgate 

ecgate{~xyz}:1 echo $DISPLAY 

(Something like)ecaccess.ecmwf.int: 

ecgate{~xyz}:2 xcdp& 

• A control window will also appear. Keep it while X11 access 

required. 


ECMWF

ECaccess – ecxterm 


ECMWF

ECaccess – web – http://ecaccess.ecmwf.int/ 


ECMWF

ECaccess tools - ECtools 

• Must be installed at your end on each platform for which 

access to ECMWF is required. 

• Help for each command is available with “-help” option: 

 

{mshost}$ eccert –help 

• ECtools are also available at ECMWF. 


ECMWF

ECaccess - eccert Command 

{mshost}$ eccert –verbose 

echost: ecaccess.ecmwf.int 

ecport: 443 

eccert: /home/xyz/.eccert.crt 

Certificate request 

ECMWF user identifier: xyz 

Passcode from your SecurID card: 

Certificate saved (855 bytes) 

{mshost}$ 

• Certificate valid for 7 days. 

• Certificate only needed for file, job and transfer management, 

NOT required for unattended transfers (see later - ectrans). 

•Certificate not needed for ECMWF local use of ECtools. 


ECMWF

ECaccess environment 

Name Purpose Default value 

Environment Variables 

Oh 

ECDOMAIN 

Target domain (either 

ECFS, ECHOME, 

ECSCRATCH or 

ECMARS) and optionally 

target user. 

ECDEBUG Enable debug mode False 

ECHOME 

ECCERT Certificate file location $HOME/.eccert.crt 

ECHOST Gateway host name Local host 

ECPORT Gateway port number 443 


ECMWF

ECaccess File Management 

ecls Print a list of files at $ECDOMAIN. 

ecdir Same as the previous command but with any system-dependent 

information. 

ecget Retrieve an $ECDOMAIN file and store it on the local machine. 

ecreget Same as the previous command, but transfer is continued from the 

apparent point of failure. 

ecput Store a local file at $ECDOMAIN. 

ecdelete Delete a file at $ECDOMAIN. 

ecmkdir Create a directory at $ECDOMAIN. 

ecrmdir Remove an empty directory at $ECDOMAIN. 

ecmodtime Get the last modification date of a file at $ECDOMAIN. 

ecsize Get the size of a file at $ECDOMAIN. 

• No support for meta-characters. 


ECMWF

ECaccess file management 

{mshost}$ ecls 

script.sh 

ecaccess-tools.tar.gz 

ecaccess.doc 

ecaccess 

{mshost}$ ecdir ecaccess 

10838 drwxr-xr-- 4 xyz systems 96 Mar 14 09:30 . 

3194 drwxr-xr-x 47 xyz systems 4096 Mar 14 09:30 .. 

12721 drwxr-x--- 4 xyz systems 96 Mar 13 18:55 client 

124513 drwxr-x--- 11 xyz systems 2048 Mar 5 11:38 gateway 

{mshost}$ ecget ecaccess-tools.tar.gz 

{mshost}$ ecget ecaccess-tools.tar.gz tools.tar.gz 

{mshost}$ ls *.tar.gz 

ecaccess-tools.tar.gz 

tools.tar.gz 

{mshost}$ 


ECMWF


{mshost}$ ecdelete ecaccess-tools.tar.gz 

DELE command successful 

{mshost}$ ecput ecaccess-tools.tar.gz 

{mshost}$ ecmkdir ectest 

MKD command successful 

{mshost}$ ecrmdir ectest 

RMD command successful 

{mshost}$ ecrmdir ecaccess 

Directory not empty 

{mshost}$ 


ECMWF


{mshost}$ export ECDOMAIN=ecfs 

{mshost}$ ecdir 

10838 drwxr-xr-- 4 xyz systems 96 Mar 14 09:30 . 

3194 drwxr-xr-x 47 xyz systems 4096 Mar 14 09:30 .. 

12721 drwxr-x--- 4 xyz systems 96 Mar 13 18:55 backup 

124513 drwxr-x--- 11 xyz systems 2048 Mar 5 11:38 doc 

{mshost}$ export ECDOMAIN=“ecfs[zzz]” 

… 

{mshost}$ export ECDOMAIN=echome 

… 

{mshost}$ 


ECMWF

ECaccess Job and transfer management 

ecqls 

List Ecaccess target queues 

ecjreq Submit a batch request (file is at $ECDOMAIN). 

ecjget Get job input, job output and job error. 

ecjput 

ecjls 

Submit a batch request (file is local). 

Check status of a batch request. 

ecjdel Delete a batch request. 

ectreq Initiate a transfer from $ECDOMAIN, using the ectrans spooling 

mechanism. 

ectls List transfers carried out by ectrans. 

ectret Retry a transfer. 

ectdel Cancel an ectrans transfer (remove it from the spool). 

ectinfo Display the target location for a user-id. 


ECMWF

ECaccess Job Management 

{mshost}$ ecqls 

ecgate LoadLeveler submission on ecgate (INIT=23 ...) 

hpcd LoadLeveler submission on hpcd (INIT=21 ...) 

... 

{mshost}$ 

{mshost}$ ecqls hpcd 

diag Diagnostic jobs only 

temp Temporary small jobs < 5 Nodes 

inter_class Default interactive class 

debug Default interactive class 

os Operational serial/single task work 

ns Serial/single task work 

bench Benchmark class 

np Parallel work 

op Operational parallel work 

{mshost}$ 


ECMWF


{mshost}$ ecjreq -help 

Syntax: JREQ ECaccess-queue remote-script [args ...] 

-at - start date (yyyy-MM-dd HH:mm) 

-nd - no directives within the input script 

-tg - specify the target gateway name 

-tr - specifiy the access method (msuser[@destination]) 

-to - transfer output file when the request ends 

-te - transfer error file when the request ends 

-ti - transfer input file when the request ends 

-tk - keep in spool (default: deleted if transfer 

successful) 

-mu - send mail for the request to the stated address 

-mb - send mail when the execution/transfer begins 

-me - send mail when the execution/transfer ends 

-mf - send mail when the execution/transfer fails 


ECMWF


{mshost}$ ecjreq ecgate script.sh –me 

34850 

{mshost}$ ecjreq ecgate test.sh 

Error opening file 

{mshost}$ 

{mshost}$ ecjput ecgate test.sh –me 

34851 

{mshost}$ 


ECMWF


{mshost}$ ecjls 

4421 ecgate WAIT Nov 24 21:33 

3884 hpcd@hpcd.ecmwf.int DONE Nov 18 11:42 

{mshost}$ ecjls 3884 

Jobid: 3884 

Location: hpcd@hpcd.ecmwf.int 

Date/Time: Nov 18 11:42 

Status: DONE 

stdout size: 221 

stderr size: 219 

stdin size: 241Jobid: 4421 

{mshost}$ 


ECMWF


{mshost}$ ecjget o34852 

{mshost}$ ls JOB* 

JOB-o34852 

{mshost}$ ecjget o34852 job.out 

{mshost}$ ls –ail job.out 

71196 -rw-r----- 1 xyz group 686 Mar 13 19:10 job.out 

{mshost}$ 

{mshost}$ ecjdel 34852 

JDEL command successful 

{mshost}$ 

• ecjdel kills job running and removes all files. 


ECMWF

ECtrans Command 

• Unattended file transfers, issued from ECMWF 

• It is preferable if the ECaccess gateway is installed at your 

end. Alternatively use gateway at ECMWF. 

Internet 

RMDCN 


Server 

(ECMWF) 

ecgate or 

hpcd: 

MS Server 

running 

the gateway 

Shared File System 

MS Network 

file:// 


Gateway 

(Member State) 

ftp:// 

MS 

Workstation 

ms3 

$ ectrans –gateway ecaccess.ecmwf.int \ 

-remote msuser@destination \ 

–source [location/]filename \ 

[-target [location/]filename] 

MS 

Workstation 

ms1 

MS 

Server 

ms2 

ECAccess 

Member State 

Gateway: 

destination= 

ftp://$msuser[name]:$msuser[passwd]@$hostname/$location 


ECMWF

ECtrans - msuser maintenance 


ECMWF



ECMWF



ECMWF



ECMWF


•MS associations are specific to each ECaccess gateway. 


ECMWF

ECtrans Command – at ECMWF 

{echost}$ ectrans –help 

usage: ectrans [-get] -gateway name \ 

–remote msuser@destination \ 

-source name [args ...] 

-gateway str - target gateway name 

-source str - source file name 

-echost str - eccmd host name (default: localhost) 

-ecport num - eccmd port number (default: 644) 

-remote str - target user (default: same as ecuser) 

-target str - target file name (default: same a source) 

-verbose - verbose mode on 

-help - this message 

-get - transfer file from your site to ECMWF 

-status - check the status on file transfer 

Default values can also be set by the ECHOST, ECPORT, 

ECUSER or GATEWAY shell variables. 

{echost}$ 


ECMWF

ECtrans Command 

{ecgate}$ ectrans –gateway ecaccess.ecmwf.int \ 

-remote from_ecmwf@genericFtp 

–source ./data.grib \ 

–target data.grib \ 

-verbose 

echost: ecaccess.ecmwf.int 

ecport: 644 

ecuser: usl 

source: ./data.grib 

target: data.grib 

keep : false 

option: reject 

File to upload (585 bytes) 

10129250216245 

{ecgate}$ 

• Users have to maintain MS user’s associations on local 

gateway, via the web interface. 

• ectrans is available in both directions. 


ECMWF

ECaccess transfer Management 

{mshost}$ ectreq –help 

Syntax: TREQ source [args ...] 

-remote str - target user (default: current) 

-gateway str - target gateway name (default: current) 

-target str - target file name (default: same as source) 

-keep - keep the request in the spool 

-reject - if the target file already exists 

(default) 

-append - if the target file already exists 

-resume - if the target file already exists 

-erase - if the target file already exists 

{mshost}$ 

{mshost}$ ectreq data.grib 

10161191420729 

{mshost}$ 


ECMWF


{mshost/echost}$ ectret 10161191420729 

TRET command successful 

{mshost}$ 

• $SCRATCH used for temporary file storage. 


ECMWF


{mshost/echost}$ ectls 

10161191420729 STOP xyz@ecaccess.ecmwf.int Mar 14 15:19 

10164593445344 DONE xyz@ecaccess.ecmwf.int Mar 14 15:22 

10161193432443 DONE xyz@ecaccess.ecmwf.int Mar 14 15:19 

{mshost/echost}$ ectls 10161191420729 

Copyid: 10161191420729 

MS user: xyz 

Hostname: ecaccess.ecmwf.int 

Access: ECaccess gateway 

Status: STOP 

Error message: No target location for xyz 

Date/Time: Mar 14 15:19 

Source: ./job.out 

Target: ./job.out 

{mshost/echost}$ ectdel 10161191420729 

{mshost/echost}$ 


ECMWF

ECaccess pointers 

• http://www.ecmwf.int/services/ecaccess 

User’s manual 

Administrator’s manual 

Registration center (for local ECaccess gateway) 

• Quick start – check first if local gateway available 

http://ecaccess.ecmwf.int/ 

telnet ecaccess.ecmwf.int 

ssh [-X] xyz@ecaccess.ecmwf.int 

ftp ecaccess.ecmwf.int 

Ectrans 


ECMWF

Training Course - ComIntro - ECaccess

Create successful ePaper yourself

Delete template?

Save as template?