FTP Analysis via SMF Records, FTP Server Exits ... - Aesclever.com

FTP Analysis via SMF Records, FTPServer Exits and Logging, and CTRACESHARE Session 3753March 6, 2009David ChengApplied Expert Systems, Inc.davec@aesclever.com

AgendaFTP Background Data type, structure and mode Active FTP vs. Passive FTP FTP Commands and RepliesFTP Diagnostic/Performance DataFTP Server ExitsFTP SMF Records – Server, ClientTCP Connection SMF Records – INIT, TERMSMF Exits vs. NMI SYSTCPSMFTP Server LoggingTracing FTP Server Trace IP packet trace, OSAENTA trace Packet Trace AnalysisFTP Diagnosis, Tuning and AnalysisCopyright (C) 2009 Applied Expert Systems, Inc. 2

FTP Data Type – how data is interpretedby the receiver FTP always transfer data in 8-bit bytes; this is called the transfer size ASCII (TYPE A)Default data type Each line of data is terminated by CRLF (X’0D0A’)Translation is always required; even between 2 ASCII hosts:ASCII -> NVT-ASCII-> ASCII(NVT-ASCII : Network Virtual Terminal ASCII as defined in the TELNETprotocol.)If MVS is the receiving side, data will be translated to EBCDIC, CRLFreplaced with MVS record boundaries – according to SITE/LOCSITE parms:RECFM and LRECL EBCDIC (TYPE E) 8-bit EBCDIC bytes are transferred as they are – no translation IMAGE (TYPE I) Contiguous bits packed into the 8-bit FTP transfer byte sizeNormally used for binary data More efficient method to transfer data between 2 similar ASCII hostsCopyright (C) 2009 Applied Expert Systems, Inc. 3

FTP Data Structure – how data is storedby the receiver File (STRU F) Has no internal structure Contiguous sequence of bytes Most widely implemented Record (STRU R) File is made up of sequential records; ASCII type withCRLF sequences z/OS only supports Record structure with streammode transfer Page (STRU P) – not supported in z/OSCopyright (C) 2009 Applied Expert Systems, Inc. 4

FTP Data Mode – how data is transmitted Stream (MODE S)Transmitted as stream of bytes; with very little or no extraprocessing Block (MODE B) Transmitted as a series of data blocks, each block is precededby a header - count and descriptor z/OS only supports Block mode with data type EBCDIC A file transferred between 2 z/OS systems in Block mode willpreserve its record structure (e.g., variable length records) Compress (MODE C)Transmitted in a compressed format Simple compression algorithm – send duplicated bytes in a two-byte sequence, followed by a one-byte filler In z/OS Compress requires data type EBCDICCopyright (C) 2009 Applied Expert Systems, Inc. 5

Control / Data ConnectionsControl connection A communication path between the Client and Server for theexchange of commands & replies FTP Server Port 21 Connection stays up during the whole session, in whichmany files may be transferredData connection A full duplex connection over which data is transferred, in aspecified mode and type FTP Server Port 20 (for active FTP) Usually one for each file transferCopyright (C) 2009 Applied Expert Systems, Inc. 6

Active FTPServer initiates data connection to the client Client connects from a random unprivileged port (N >1024) to the FTP server’s port 21Client starts listening to port N+1 and sends the FTPcommand PORT N+1 to the FTP serverPORT h1,h2,h3,h4,p1,p2h1,h2,h3,h4 is the client’s IP address, p1,p2 is the clientport number in an 8 bit high, low bit orderThe Server will then connect back to the client’s specifieddata port from its local data port (port 20)Copyright (C) 2009 Applied Expert Systems, Inc. 7

Active FTPFTP ClientFTP Server1674 PORT 1675 21ACK1675 connect 20ACKCopyright (C) 2009 Applied Expert Systems, Inc. 8

FTP Active Mode in DetailFTP ClientPort XPort XPort XPort XPort XPort XPort YPort YPort YPort YFTP ServerSYN Port 21SYN ACK Port 21ACK Port 21PORT, IP address, port Y Port 21PORT command successful Port 21LIST, RETR, or STOR, etc. Port 21SYN Port 20SYN ACK Port 20ACK Port 20Data segments and ACKs Port 20Copyright (C) 2009 Applied Expert Systems, Inc. 9

Passive FTP Client initiates data connection to the server Firewall friendly When opening an FTP connection, the client opens 2random unprivileged ports locally (N > 1024 and N+1) The first port contacts the server on port 21 Client issues the PASV command (the PASV commandtakes no parameters) The server then opens a random port and sends ReplyCode 227 back to the client (similar to the PORTcommand) The client then initiates the connection from port N+1 toport P on the server to transfer dataCopyright (C) 2009 Applied Expert Systems, Inc. 10

Passive FTPFTP ClientFTP Server1673 PASV 21“227 Entering Passive Mode (IP Addr, Port #)”1674 connect 2020ACKCopyright (C) 2009 Applied Expert Systems, Inc. 11

FTP Passive Mode in DetailFTP ClientFTP ServerPort XSYN Port 21Port X SYN ACK Port 21Port XACK Port 21Port XPort XPort ZPort ZPort ZPort XPort ZPASV Port 21RC 227, IP address, Port Y Port 21SYNPort YSYN ACKPort YACKPort YLIST, RETR, or STOR, etc. Port 21Data segments and ACKsPort YCopyright (C) 2009 Applied Expert Systems, Inc. 12

FTP CommandsCommands and Replies are sent across the controlconnection and are in plain text.Commands are 3 or 4 bytes characters, each withoptional parameters.The FTP commands specify the parameters for: the data connection (port) transfer mode data representation type and structure the nature of file system operation (store, retrieve,append, delete, etc.)Copyright (C) 2009 Applied Expert Systems, Inc. 13

Sample FTP CommandsAccess Control:ABOR Abort a file transferUSERUser identification PORTData port specification QUITTerminates a user and the control connectionTransfer:TYPE STRUMODE RETR STORData representation (ASCII, EBCDIC, Image)Transfer structure (File, Record, Page)Transfer mode (Stream, Block, Compress)Server -> Client file transferClient -> Server file transferService:DELEDeletes a Server fileLIST Directory listingRNFRRenames from RNTO Renames to (RNFR must be immediately followed by a RNTO command)Copyright (C) 2009 Applied Expert Systems, Inc. 14

FTP Replies Synchronization of requests and actions in the filetransfer process Guarantee that the user process always knows the stateof ServerEvery command must generate at least one reply An FTP reply consists of a 3-digit number (i.e., 3alphanumeric characters) followed by some text The number is intended for use by the software todetermine what to do next; the text is intended for thehuman user There may be more than one reply, in which case thesemultiple replies must be easily distinguishedCopyright (C) 2009 Applied Expert Systems, Inc. 15

FTP Reply Code1yz2yz3yz4yz5yzPositive preliminary replyPositive completion reply (a new command may be sent)Positive intermediate reply (another command must be sent)Transient negative reply (command can be re- issued later)Permanent negative reply (command should not be retried)x0zx1zx2zx3zx4zx5zSyntax errorInformationConnectionsAuthentication and accountingUnspecifiedFile system statusCopyright (C) 2009 Applied Expert Systems, Inc. 16

Sample FTP Reply Codes 150 File status okay; about to open data connection. 226 Transfer complete 227 Entering passive mode {h1,h2,h3,h4,p1,p2} 250 Requested file action okay, completed. 257 "PATHNAME" created. 350 Requested file action pending further information. 450 Requested file action not taken. File unavailable (e.g., file busy). 550 Requested action not taken. File unavailable (e.g., file notfound, no access). 451 Requested action aborted. Local error in processing. 551 Requested action aborted. Page type unknown. 452 Requested action not taken. Insufficient storage space insystem. 552 Requested file action aborted. Exceeded storage allocation (forcurrent directory or data set). 553 Requested action not taken.Copyright (C) 2009 Applied Expert Systems, Inc. 17

C:\Windows>ftp 137.72.43.247Connected to 137.72.43.247.220-FTPD1 IBM FTP CS V1R7 at os17.aesclever.com, 21:05:48on 2006-0707-20.220 Connection will close if idle for more than 5 minutes.User (137.72.43.207:(none)): p390331 Send password please.Password:RC 220RC 331Service ready for new user- : continuationUser name OK, need passwordCopyright (C) 2009 Applied Expert Systems, Inc. 18

FTP VulnerabilitiesUser ID and Password are sent in clear textFTP Bounce Attack Arbitrary IP and port can be specified in the PORTcommand – this allows an attacker to open a (data)connection to a port on a machine that’s not theoriginal client Port scanning Bypass firewallsTo Control PORT or EPRT command:PORTCOMMAND ACCEPT | REJECTPORTCOMMANDIPADDR UNRESTRICTED | NOREDIRECTPORTCOMMANDPORT UNRESTRICTED | NOLOWPORTS(ports < 1024)Copyright (C) 2009 Applied Expert Systems, Inc. 19

FTP Diagnostics/Performance DataConnection attemptsLogon failuresClient identificationActive vs. Passive FTPFTP commandsFTP repliesThroughputData can also be used as audit trail and for monitoringsecurity breaches.Copyright (C) 2009 Applied Expert Systems, Inc. 20

FTP Diagnostics/Performance Data Published Record Types or API Non-intrusive, lower overhead Event-driven True real-time data FTP Server exits and SMF exits Waiting/”Polling” Comm Server Network Management API(SMF 119 only) TCP/IP session SMF data is not availableCopyright (C) 2009 Applied Expert Systems, Inc. 21

FTP Diagnostics/Performance DataFTP Server Exits FTCHKIP – open connection (invoked by FTP daemon) FTCHKPWD – password verification FTCHKCMD – FTP commandFTPOSTPR – FTP command completion FTCHKJES – Job submissionFTPSMFEX – FTP server SMF record SMF records (Type 118 or 119)FTP Server Logon FailureFTP Server Transfer CompletionFTP Client Transfer CompletionTCP Connection InitiationTCP Connection TerminationCopyright (C) 2009 Applied Expert Systems, Inc. 22

FTP Diagnostics/Performance Data Logging/Tracing FTP Server FTPLOGGING ANONYMOUSFTPLOGGING TRACE, DEBUG Packet trace – detailed analysis at protocol level NOT for monitoring purpose Performance penalty (e.g., APAR PQ84192)Copyright (C) 2009 Applied Expert Systems, Inc. 23

FTP Server User ExitsR1 -> parameter list, which is a series of pointers to valuesThe first word of the parameter list always points to thereturn code (RC). RC=0 upon entry to an exit. If RC is not0, user will receive a negative replyThe second word of the parameter list always points to aword containing the number of parameters that followAPF-authorizedSTEPLIB DD in the FTPD procRACF considerationSample code in TCPIP.SEZAINSTCopyright (C) 2009 Applied Expert Systems, Inc. 24

FTP Server Exit - FTCHKIPFTCHKIP is called at the initial stage of login orwhenever the user issues an OPEN command Client’s IP address (IPV4) and port Server’s IP address (IPV4) and port Socket address structure (IPV4 or IPV6) for theclient’s control conneciton Socket address structure (IPV4 or IPV6) for theserver’s control connection Session IDSecurity Control: control FTP access by IP addressCopyright (C) 2009 Applied Expert Systems, Inc. 25

FTP Server Exit - FTCHKPWDFTCHKPWD is called after the user enters the password Client’s user ID Client’s password User data Number of bad passwords input in this logon attempt Socket address structure for the client’s controlconnection Socket address structure for the server’s controlconnection Session IDSecurity Control: Control FTP access by User IDCopyright (C) 2009 Applied Expert Systems, Inc. 26

FTP Server Exit - FTCHKCMDFTCHKCMD is called whenever the client enters acommand Client’s user ID Command Command parameters Current directory type: MVS, HFS File type: SEQ, JES, SQL Current working directory Address of a buffer for command modificationSecurity Control: Control FTP access by FTP commandor file name, etc.Copyright (C) 2009 Applied Expert Systems, Inc. 27

FTP Server Exit - FTPOSTPRFTPOSTPR is called upon completion of the FTP commands RETR,STOR, STOU, APPE, DELE, and RNTOClient’s user IDClient’s IP addressClient’s portCurrent directory type: MVS, HFSCurrent working directoryCurrent file type: SEQ, JES, SQLFTP reply codeFTP reply stringFTP command codeCurrent CONDDISP setting: C for catalog, D for deleteCopyright (C) 2009 Applied Expert Systems, Inc. 28

FTP Server Exit - FTPOSTPRClose reason code: 0 – transfer completed normally 4 – transfer completed w/errorsee FTP reply code and text string 8 – transfer completed w/socket erros 12 – transfer aborted 16 – transfer aborted w/SQL file errorsDataset name or HFS file nameBytes transferredSocket address structure for the client’s control sessionSocket address structure for the server’s control sessionSession IDAddress of scratch pad area (256 bytes)Copyright (C) 2009 Applied Expert Systems, Inc. 29

FTP Server Exit - FTPSMFEXFTPSMFEX is called before a type 118 SMF (FTPserver) record is written to SMFType 119 SMF records must use the system-wide SMFexits: IEFU83, IEFU84 , (IEFU85)R1 -> the following parameter list: Pointer to the return code Pointer to the type 118 SMF record On entry, the return code is set to 0. A return code of 0specifies that the SMF record will be writtenCopyright (C) 2009 Applied Expert Systems, Inc. 30

FTP Server Exit Installation APF-authorize the load library Add the load library to STEPLIB in the FTPDproc If RACF Program Control is active: SETROPTSWHEN(PROGRAM), you must define FTP exitsto RACF class PROGRAM Restart the FTP Daemon (for FTCHKIP)Copyright (C) 2009 Applied Expert Systems, Inc. 31

FTP Server Exit InstallationSample RACF Definition for FTCHKIP:RDEFINE PROGRAM FTCHKIPADDMEM(‘loadlib’loadlib’/volser/NOPADCHK) UACC(READ)…SETR WHEN(PROGRAM) REFRESHWithout proper RACF definition, FTP client will get thefollowing error when logging in:550 PASS COMMAND FAILED - _PASSWD() ERROR: EDC5157I ANINTERNAL ERROR OCURREDCopyright (C) 2009 Applied Expert Systems, Inc. 32

Verify FTP Server Exits Start the FTP Server with the “TRACE”parameter; e.g., S FTPD,PARM=TRACE Check for the following messages in SYSLOG:BPXF024I (FTPD) Jan 5 18:01:34 ftpd 33619980 : DM1009 main:FTCHKIP successfully loadedBPXF024I (AESDJC1) Jan 6 02:01:57 ftps 16843115 : RX0625 main:chkpwdexit successfully loadedBPXF024I (AESDJC1) Jan 6 02:01:58 ftps 16843115 : RX0641 main:chkcmdexit successfully loadedBPXF024I (AESDJC1) Jan 6 02:01:58 ftps 16843115 : RX0696 main:FTPOSTPR successfully loadedCopyright (C) 2009 Applied Expert Systems, Inc. 33

FTP Server Transfer Completion SMF RecordFTP commandFTP type: SEQ, JES, SQLClient IP address and portServer IP address and portLocal user IDData format:A: ASCIIE: EBCDICI: image (binary)D: double byteU: UCS-2Data Structure: File or RecordTransmission mode – S: stream, B: block. C : compressedStart/End time of transmissionBytes transferredFTP reply codeDataset/member/file namesFile transmission durationCopyright (C) 2009 Applied Expert Systems, Inc. 34

FTP Client Transfer Completion SMF RecordFTP commandClient IP address and portServer IP address and portData formatData StructureTransmission modeStart/End time of transmissionByte countDataset/file name File transmission durationCopyright (C) 2009 Applied Expert Systems, Inc. 35

Enable TCP/IP SMF Recording SMFPRMxx – make sure that 118/119 is notbeing excluded from recording SMF Type 119 is available only in z/OS V1R2and later releases SMF Type 118 and Type 119 can co-exist To get FTP Server SMF record, configure FTPDATA as follows: 118: SMF STD 119: SMF TYPE119Copyright (C) 2009 Applied Expert Systems, Inc. 36

Enable TCP/IP Session SMF RecordingTo get FTP Client SMF record, configure TCP/IP PROFILE asfollows:SMFCONFIG TYPE119 FTPCLIENT …To get TCP INIT and TERM SMF records:SMFCONFIG TYPE119 TCPINIT TCPTERM …Do not collect duplicate records; either 118 or 119 (recommended)TERM records are “better” than INIT records: more informationCopyright (C) 2009 Applied Expert Systems, Inc. 37

Verify SMF RecordingSystem Level – issue the D SMF,O operator command, verify: SMFPRMxx member SMF parametersTCP/IP Level – issue the NETSTAT,CONFIG commandCheck the SMF Parameters listing; e.g.,SMF Parameters:Type 118:TcpInit: 00 TcpTerm: 00 FTPClient: 00TN3270Client: 00 TcpIpStats: 00Type 119:TcpInit: Yes TcpTerm: Yes FTPClient: YesTcpIpStats: Yes IfStats: Yes PortStats: YesStack: Yes UdpTerm: Yes TN3270Client: YesCopyright (C) 2009 Applied Expert Systems, Inc. 38

Verify SMF RecordingFTP Server – start the FTP server with the “TRACE” parameter;e.g., S FTPD,PARM=TRACELook for the write_smf_record messages; e.g.,250 Transfer completed successfully.BPXF024I (AESDJC1) Jan 6 02:02:08 ftps 16843115 : RU1463write_smf_record: entered with type 4BPXF024I (AESDJC1) Jan 6 02:02:08 ftps 16843115 : RU0754write_smf_record_119: entered with type 4.FTP Client – start the FTP client with the “trace” parameter, or issue the“debug” command from an FTP client session;e.g., ftp 137.72.43.247 (traceLook for the following messages: CU1963, CU1463, CU2241; e.g.,250 Transfer completed successfully.EZA1617I 2320 bytes transferred in 0.160 seconds. Transferrate 14.50 Kbytes/sec.CU1963 write_smf_record: entered with type 16.CU1463 write_smf_record_119: entered with type 16.CU2241 write_smf_record: length of smfrecord: 224Copyright (C) 2009 Applied Expert Systems, Inc. 39

Obtaining SMF data in real-time – SMF ExitsSMF Exits - receives control when caller invokes the macro towrite SMF records; Supervisor state, KEY 0 IEFU83 – SMFWTM, BRANCH=NO FTP Server SMF TCP/IP Session SMF (INIT and TERM) IEFU84 – SMFWTM, BRANCH=YES FTP Client SMF IEFU85 – SMFEWTM, BRANCH=YES MODE=XMEM, andASCB != home primary ASID.IEFU85 cannot issue any SVC MVS Dynamic Exits Facility allows multiple exits to co-existCopyright (C) 2009 Applied Expert Systems, Inc. 40

Installing SMF ExitsSpecify SMF Exits in SMFPRMxxBy default, ALL exits are invokedIf EXITS is specified on a SUBSYS for STC, OMVS, TSO or JES2, specify IEFU83,IEFU84 and IEFU85 on the EXITS parameterE.g, SUBSYS(STCSTC,EXITS(IEFU83,IEFU84,IEFU85,IEFACTRT))Specify SMF Exits to the Dynamic Exits FacilityDefine PROGxx in SYS1.PARMLIBEXIT ADD EXITNAME(SYS.IEFU83) MODNAME(module) DSNAME(dsn)EXIT ADD EXITNAME(SYSSTC.IEFU83) MODNAME(module) DSNAME(dsn)Issue the SET PROG=xx commandVerify SMF ExitsD PROG,EXITD PROG,EXIT,EX=SYS.IEFU83,DIAGCSV464I 21.37.38 PROG,EXIT DISPLAY 109EXIT SYS.IEFU83MODULE STATE EPADDR LOADPT LENGTH JOBNAMEIEFU83 A 838CA518 00000000 00000000 *AESSMF00 A 8FFB66F8 0FFB66F8 00000908 *Copyright (C) 2009 Applied Expert Systems, Inc. 41

Obtaining SMF data in real-time - NMI z/OS CS Network Management Interface (NMI) z/OS 1.4 w/APAR II13699, z/OS 1.5 or later SYSTCPSM interface Type 119 SMF records: FTP Server/Client, TN3270 Server andTSO Telnet Client TCP Connection records Requires waiting/”polling” – not as real-time asSMF exitsCopyright (C) 2009 Applied Expert Systems, Inc. 42

SYSTCPSM NMI SAF authorization for the SERVAUTH classEZB.NETMGMT.sysname.tcpprocname.SYSTCPSMEZB.NETMGMT.sysname.tcpprocname.SYSTCPSM profile TCP/IP Profile NETMONITOR ONor NETMONITOR … SMFSERVICE Connect to the AF_UNIX socket descriptor:/varvar/sock/SYSTCPSM.

SYSTCPSM Service Use the EZBTMIC1 service (in SYS1.CSSLIB) tocopy the data into application buffer Caller must be APF authorized Input is the token record received fromSYSTCPSM NMI Data is copied to application buffer as CTE(Component Trace Element) records: ,…For more info: “Comm Server IP Programmer’s Guide and Ref.”Copyright (C) 2009 Applied Expert Systems, Inc. 44

Sample FTP Session 1ftp 137.72.43.207EZY2640I Using 'TCPIP.FTP.DATA' for local site configuration parameters.EZA1450I IBM FTP CS V1R7EZA1554I Connecting to: 137.72.43.207 port: 21.220-FTPD1 IBM FTP CS V1R6 at OS16.AESCLEVER.COM, 14:51:31 on 2007-02-09.220 Connection will not timeout.EZA1459I NAME (137.72.43.207:AESDJC1):p390EZA1701I >>> USER p390331 Send password please.EZA1789I PASSWORD:EZA1701I >>> PASS230 P390 is logged on. Working directory is "AESDJC1.".EZA1460I Command:binEZA1701I >>> TYPE I200 Representation type is ImageEZA1460I Command:get 'aesdjc1.xmi' 'aesdjc1.xmi' (replaceEZA1701I >>> PORT 137,72,43,240,6,139200 Port request OK.EZA1701I >>> RETR 'aesdjc1.xmi'125 Sending data set AESDJC1.XMI FIXrecfm 80250 Transfer completed successfully.EZA1617I 166400 bytes transferred in 2.180 seconds. Transfer rate 76.33 Kbytes/sec.Copyright (C) 2009 Applied Expert Systems, Inc. 45

Sample FTP Session 1 – logging byFTP ExitsFTP OPEN CONNECTION,IP=137.72.43.240,PORT= 1674,TIME=14:51:13.67FTP CMD=USER ,USER= ,TIME=14:51:16.01,ARG=p390FTP CMD=PASS ,USER=P390 ,TIME=14:51:17.81,ARG=FTP LOGIN,USER=P390 ,TIME=14:51:17.81FTP CMD=TYPE ,USER=P390 ,TIME=14:51:23.03,ARG=IFTP CMD=PORTFTP CMD=RETR,USER=P390 ,TIME=14:51:34.37,ARG=137,72,43,240,6,139,USER=P390 ,TIME=14:51:34.40,ARG='aesdjc1.xmi'FTCHKIPFTCHKCMDFTCHKCMDFTCHKPWDFTCHKCMDFTCHKCMDFTCHKCMDFTP POST,CMD=RETR,USER=P390,IP=137.72.43.240,TYPE=MVS/SEQ,RC=250,REASON=0,TIME=14:51:36.93SESSIONID=FTPD100011, CPU TIME=0.829FTPOSTPRCopyright (C) 2009 Applied Expert Systems, Inc. 46

Sample FTP Session 1How to interpret the PORT commandPORT 137,72,43,240,6,139,6,139IP Address of the client: 137.72.43.240Port of the client: 256*6 + 139 = 1675Copyright (C) 2009 Applied Expert Systems, Inc. 47

Sample FTP Session 1: Active FTPFTP ClientFTP Server(137.72.43.240) (137.72.43.207)1674 PORT 1675 21ACK1675 connect 20ACKCopyright (C) 2009 Applied Expert Systems, Inc. 48

Sample FTP Session 1 – FTP ServerSMF dataFTPS:RETR,IP=137.72.43.240,PORT=21/1674,RC=250,User=P390,Format=S/S/IFormat=S/S/I,ABND=Start=15:51:34,End=15:51:34,Bytes=166400,Elapsed=0.010sec,Throughput=16640.00KB/secDSN1=AESDJC1.XMI/,DSN2=/Format:Data set type: P – partitioned, S – sequential, H – HFSMode: S – stream, B – block, C – compressedData format:A – ASCII, E – EBCDIC, I – image (binary),D – double-byte, byte, U – UCS-2Copyright (C) 2009 Applied Expert Systems, Inc. 49

Sample FTP Session 2EZA1460I Command:put 'aesdjc1.xmi' 'aesdjc1.small'EZA1701I >>> SITE FIXrecfm 80 LRECL=80 RECFM=FB BLKSIZE=3120200 SITE command was acceptedEZA1701I >>> PORT 137,72,43,240,6,142200 Port request OK.EZA1701I >>> STOR 'aesdjc1.small'125 Storing data set AESDJC1.SMALL451-System completion code and reason: D37-04451-Data set is out of space.451 Transfer aborted due to file error.EZA1460I Command:quitEZA1701I >>> QUIT221 Quit command received. Goodbye.READYCopyright (C) 2009 Applied Expert Systems, Inc. 50

Sample FTP Session 2 – logging byFTP ExitsFTP CMD=SITE ,USER=P390 ,TIME=14:53:28.45,ARG=FIXrecfm 80 LRECL=80RECFM=FB BLKSIZE=3120FTP CMD=PORT ,USER=P390 ,TIME=14:53:28.50,ARG=137,72,43,240,6,142FTP CMD=STOR ,USER=P390 ,TIME=14:53:28.52,ARG='aesdjc1.small'FTP POST,CMD=STOR,USER=P390,IP=137.72.43.240,TYPE=MVS/SEQ,RC=451,REASON=4,TIME=14:53:29.61FTP REPLY=Transfer aborted due to file error.FTP CMD=QUIT ,USER=P390 ,TIME=14:53:31.48,ARG=Copyright (C) 2009 Applied Expert Systems, Inc. 51

Sample FTP Session 2 – FTP ServerSMF RecordFTPS:STOR,IP=137.72.43.240,PORT=21/1674,RC=451RC=451,User=P390,Format=S/S/I,ABND=Start=15:53:28,End=15:53:29,Bytes=166400,Elapsed=0.500sec,Throughput=332.80KB/secDSN1=AESDJC1.SMALL/,DSN2=/Reply Code 451: Requested action aborted. Local error inprocessing.Copyright (C) 2009 Applied Expert Systems, Inc. 52

SMF 119 TCP Connection Termination and FTP Server RecordCopyright (C) 2009 Applied Expert Systems, Inc. 53

CleverView(TM) for TCP/IP BatchPRTCP Session RTT ReportAPPLIED EXPERT SYSTEMS, INC.System: OS16 Sysplex: ADCDPL TCP/IP Stack: TCPIPDate: 10/14/2005 (2005.287)Shift: 4, From: 18:00 To: 20:00# Round Trip Time (ms) Segments SegmentsLOCATION Remote IP AddrApplSessions Avg / Std Dev / Min / Max In Out======== =============== ======== ========== ============================ ============ ============AES2 137.72.43.239AESTCP80 237 3.09 15.119 0 174 475 741FTPD1 87 26.64 56.467 1 275 129 257HTTPD1 79 1.85 6.522 0 59 130 238NPMTCPI5 16 0.50 0.500 0 1 40 56TCPIP 87 5.00 32.140 0 301 94 217506 7.19 30.318 0 301 868 1509HOST16 137.72.43.252AESTCP80 79 9.28 44.817 3 404 158 237TCPIP 14 5.29 4.131 3 20 28 4293 8.68 41.362 3 404 186 279SUBNET1 137.72.43.*AESTCP80 316 4.64 26.091 0 404 633 978FTPD1 87 26.64 56.467 1 275 129 257HTTPD1 79 1.85 6.522 0 59 130 238NPMTCPI5 16 0.50 0.500 0 1 40 56TCPIP 103 5.49 29.919 0 301 3378 3964601 7.49 32.281 0 404 4310 5493# Round Trip Time (ms) Segments SegmentsApplSessions Avg / Std Dev / Min / Max In Out======== ========== ============================ ============ ============AESTCP80 316 4.64 26.091 0 404 633 978FTPD1 87 26.64 56.467 1 275 129 257HTTPD1 79 1.85 6.522 0 59 130 238NPMTCPI5 16 0.50 0.500 0 1 40 56TCPIP 103 5.49 29.896 0 301 3378 3964Copyright (C) 2009 Applied Expert Systems, Inc. 54

FTP Server Loggingz/OS 1.4 or later releaseFTP Server can log activities to SyslogD via thefollowing FTP.DATA options: FTPLOGGING TRUE ANONYMOUSFTPLOGGING TRUENine events are logged: CONNconnectivitySECUREsecurity (TLS/SSL, Kerberos)ACCESSlogin ALLOCfile and data set allocation DEALLfile and data set de-allocationTRANS file transfer SUBMITJES job submission QUERYSQL queryABEND abnormal terminationEach activity logging message has a message numberwithin the range of EZYFS50 to EZYFS95Copyright (C) 2009 Applied Expert Systems, Inc. 55

FTPLOGGING output – centralized BPXF024I messages19:34:05 BPXF024I (FTPD) Feb 28 01:34:06 ftpd 16842834 : EZYFS50I ID=FTPD10009819:34:05 CONN starts Client IPaddr=137.72.43.142 hostname=UNKNOWN19:34:12 BPXF024I (FTPD) Feb 28 01:34:13 ftps 16842834 : EZYFS57I ID=FTPD10009819:34:12 ACCESS fails USERID=BADUSER Reason=10 Text=The userid is unknown19:34:13 BPXF024I (FTPD) Feb 28 01:34:14 ftps 16842834 : EZYFS52I ID=FTPD10009819:34:13 CONN ends Input=0 bytes Output=0 bytes19:34:30 BPXF024I (FTPD) Feb 28 01:34:31 ftpd 50397260 : EZYFS50I ID=FTPD10009919:34:30 CONN starts Client IPaddr=137.72.43.142 hostname=UNKNOWN19:34:37 BPXF024I (AESDJC1) Feb 28 01:34:38 ftps 50397260 : EZYFS56I19:34:37 ID=FTPD100099 ACCESS OK USERID=AESDJC119:34:45 BPXF024I (AESDJC1) Feb 28 01:34:45 ftps 50397260 : EZYFS60I19:34:45 ID=FTPD100099 ALLOC OK Use MVS DSN=AESDJC1.MAIN.CNTL(ASM)19:34:45 BPXF024I (AESDJC1) Feb 28 01:34:45 ftps 50397260 : EZYFS61I19:34:45 ID=FTPD100099 ALLOC DDNAME=SYS00003 VOLSER=AES004 DSORG=PO19:34:45 DISP=(SHR,KEEP)19:34:45 BPXF024I (AESDJC1) Feb 28 01:34:45 ftps 50397260 : EZYFS70I19:34:45 ID=FTPD100099 DEALL OK Release MVS DSN=AESDJC1.MAIN.CNTL(ASM)19:34:45 BPXF024I (AESDJC1) Feb 28 01:34:45 ftps 50397260 : EZYFS81I19:34:45 ID=FTPD100099 TRANS MVS DSN=AESDJC1.MAIN.CNTL(ASM)19:34:44 BPXF024I (AESDJC1) Feb 28 01:34:45 ftps 50397260 : EZYFS84I19:34:44 ID=FTPD100099 TRANS Stru=F Mode=S Type=A Output=2460 bytes19:34:44 BPXF024I (AESDJC1) Feb 28 01:34:45 ftps 50397260 : EZYFS80I19:34:44 ID=FTPD100099 TRANS Reply=250 Transfer completed successfully.19:34:45 BPXF024I (AESDJC1) Feb 28 01:34:46 ftps 50397260 : EZYFS52I19:34:45 ID=FTPD100099 CONN ends Input=0 bytes Output=2460 bytesCopyright (C) 2009 Applied Expert Systems, Inc. 56

FTP Server TracingTRACE and DEBUG statements in FTP.DATA TRACE is equivalent to DEBUG BAS, which includes:DEBUG CMD - commandDEBUG INT – init and term of FTP sessionDEBUG FSC – details of APPE, STOR, STOU, RETR, DELE, RNFR, RNTODEBUG SOC – interface between FTP and networkUse the SITE command to turn on tracing dynamically only for the duration ofan FTP session Requires: DEBUGONSITE TRUE be specified in FTP.DATA z/OS example : site debug=basMS/DOS example: quote site debug=basUse the MODIFY command F jobname,DEBUG=(BAS) F jobname,DEBUG=(NONE)Output in SYSLOGCopyright (C) 2009 Applied Expert Systems, Inc. 57

FTP Server Tracing OutputFeb 8 04:29:51 os17 ftpd[83951678]: EZYFT82I ACTIVE SERVER TRACES - CMD INT FSC(1) SOC(1)Feb 8 04:29:51 os17 ftpd[83951678]: SD0359 accept_client: calling selectex for socket 6Feb 8 04:29:59 os17 ftpd[83951678]: SD0407 accept_client: accept()Feb 8 04:29:59 os17 ftpd[83951678]: SD0474 accept_client: accepted client on socket 7Feb 8 04:29:59 os17 ftpd[83951678]: SD1735 handle_client_socket: entered for socket 7Feb 8 04:29:59 os17 ftpd[83951678]: SD1958 handle_client_socket: new session for 137.72.43.122 port 4085Feb 8 04:29:59 os17 ftpd[83951678]: SD1099 spawn_ftps: enteredFeb 8 04:29:59 os17 ftpd[83951678]: SD0340 accept_client: prepare to accept another clientFeb 8 04:29:59 os17 ftpd[83951678]: SD0359 accept_client: calling selectex for socket 6Feb 8 04:29:59 os17 ftpd[67174598]: SD1119 spawn_ftps: my pid is 67174598 and my parent's is 83951678Feb 8 04:29:59 os17 ftpd[67174598]: SD1168 spawn_ftps: tmpmsg is 220-FTPD1 IBM FTP CS V1R7 at os17.aesclever.com,04:29:59 on 2007-02-08.08.Feb 8 04:29:59 os17 ftpd[67174598]: SD1254 spawn_ftps: tmpmsg is 220 Connection will not timeout.Feb 8 04:29:59 os17 ftpd[67174598]: SD0750 setup_new_pgm: enteredFeb 8 04:30:00 os17 ftpd[67174598]: SD0888 setup_new_pgm: msgcat->_name = /usr/lib/nls/msg/C/ftpdmsg.catFeb 8 04:30:00 os17 ftpd[67174598]: SD1049 setup_new_pgm: issuing execvFeb 8 04:30:00 os17 ftps[67174598]: GU1099 chkVerRel: system information for ADCD: z/OS version 1 release 7 (1247)Feb 8 04:30:00 os17 ftps[67174598]: PR0291 parse_cmd: enteredFeb 8 04:30:01 os17 ftps[67174598]: PR0470 parse_cmd: >>> USER aesdjc1Feb 8 04:30:01 os17 ftps[67174598]: SR2944 reply: --> 331 Send password please.Feb 8 04:30:02 os17 ftps[67174598]: PR0475 parse_cmd: >>> PASS ******Feb 8 04:30:02 os17 ftps[67174598]: RA0607 RACF_MIXED_CASE_PASSWORDS_ENABLED 0Feb 8 04:30:02 os17 ftps[67174598]: RA0670 pass: call user exit FTCHKPWD with these parameters:Feb 8 04:30:02 os17 ftps[67174598]: RA0673 ... exitrc 0, numparms 7Feb 8 04:30:02 os17 ftps[67174598]: RA0676 ... userid AESDJC1, numbadpw 0Feb 8 04:30:02 os17 ftps[67174598]: RA0696 ... client sockaddr addr:port 137.72.43.122:4085Feb 8 04:30:02 os17 ftps[67174598]: RA0698 ... server sockaddr addr:port 137.72.43.207:21Feb 8 04:30:02 os17 ftps[67174598]: RA0701 ... session id FTPD100140Feb 8 04:30:02 os17 ftps[67174598]: RA0795 pass: use __passwd() to verify the userFeb 8 04:30:02 os17 ftps[67174598]: RX1205 copyCntlBlks: enteredFeb 8 04:30:02 os17 ftps[67174598]: RX1226 copyCntlBlks: cbSaveArea at 7F5B7EB8Feb 8 04:30:02 os17 ftps[67174598]: RX1227 copyCntlBlks: ... for 37192 bytesCopyright (C) 2009 Applied Expert Systems, Inc. 58

CTRACE – Packet Tracing (SYSTCPDA)Set up External Writer ProcE.g., SYS1.PROCLIB(AESWRT)://IEFPROC EXEC PGM=ITTTRCWR,REGION=0K,TIME=1440,DPRTY=15//TRCOUT01 DD DISP=SHR,DSN=trace.datasetSet up tracing parametersE.g., SYS1.PARMLIB(CTAESPRM):TRACEOPTS ON WTR(AESWRT)Copyright (C) 2009 Applied Expert Systems, Inc. 59

CTRACE – Packet Tracing (SYSTCPDA)To Start Tracing:TRACE CT,WTRSTART=AESWRTV TCPIP,,PKT,CLEARV TCPIP,,PKT,LINKN=ETH1,ON,FULL,PROT=TCP,IP=TRACE CT,ON,COMP=SYSTCPDA,SUB=(TCPIP),PARM=CTAESPRMTo View Tracing Status:D TRACE,WTR=AESWRTVerify that the external writer is activeD TCPIP,,NETSTAT,DE Verify that TrRecCnt is non-zero and incrementingTo Stop Tracing:V TCPIP,,PKT,OFFTRACE CT,OFF,COMP=SYSTCPDA,SUB=(TCPIP)TRACE CT,WTRSTOP=AESWRT,FLUSHCopyright (C) 2009 Applied Expert Systems, Inc. 60

CTRACE – Packet Tracing (SYSTCPDA)To Start Tracing:TRACE CT,WTRSTART=AESWRTV TCPIP,,PKT,CLEARV TCPIP,,PKT,LINKN=ETH1,ON,FULL,PROT=TCP,IP=TRACE CT,ON,COMP=SYSTCPDA,SUB=(TCPIP),PARM=CTAESPRMTo View Tracing Status:D TRACE,WTR=AESWRTVerify that the external writer is activeD TCPIP,,NETSTAT,DE Verify that TrRecCnt is non-zero and incrementingTo Stop Tracing:V TCPIP,,PKT,OFFTRACE CT,OFF,COMP=SYSTCPDA,SUB=(TCPIP)TRACE CT,WTRSTOP=AESWRT,FLUSHCopyright (C) 2009 Applied Expert Systems, Inc. 61

CTRACE – OSAENTA Tracing (SYSTCPOT)Trace packets to a host attached to an OSA-Express2.The host can be an LPAR with z/OS, z/VM or Linux.The trace function is controlled by z/OS Communication Server, while the data iscollected in the OSA at the network port.Pre-Reqs:Install the required PTFs for z/OS V1R8 (APAR PK36947).Install the microcode for the OSA (2094DEVICE PSP and the 2096DEVICEPSP).Update the OSA using the Hardware Management Console (HMC) to:Define more data devices to systems that will use the trace function.Set the security for the OSA:LOGICAL PARTITION - Only packets from the LPARCHPID - All packets using this CHPIDVerify the TRLE definitions for the OSA that it has one DATAPATH addressavailable for tracing. Note that two DATAPATH addresses are required – one fordata transfers and the other for trace data.Copyright (C) 2009 Applied Expert Systems, Inc. 62

CTRACE – OSAENTA Tracing (SYSTCPOT)To Start Tracing:TRACE CT,WTRSTART=AESWRTAESWRTV TCPIP,,OSAENTAOSAENTA,PORTNAME=,CLEARV TCPIP,,OSAENTAOSAENTA,PORTNAME=,ON,NOFILTER=ALLTRACE CT,ON,COMP=SYSTCPOT,SUB=(TCPIP),PARM=,SUB=(TCPIP),PARM=CTAESPRMCTAESPRMTo Stop Tracing:V TCPIP,,OSAENTAOSAENTA,PORTNAME=,OFFTRACE CT,OFF,COMP=SYSTCPOTSYSTCPOT,SUB=(TCPIP)TRACE CT,WTRSTOP=AESWRT,FLUSHTo View Tracing Status:D TCPIP,,NETSTAT,DEVLINKSD TRACE,WTR=AESWRTAESWRTCopyright (C) 2009 Applied Expert Systems, Inc. 63

CTRACE Packet Trace Decoding – IPCS JCL//TSO EXEC PGM=IKJEFT01,DYNAMNBR=60,// PARM='%BLSCDDIR DSNAME(&SYSUID..BATCH.DDIR)VOLUME(AES003)'//SYSPROC DD DISP=SHR,DSN=SYS1.SBLSCLI0//TRACE DD DISP=SHR,DSN=trace.dataset

CTRACE Packet Trace Decoding – IPCS OutputIPCS PRINT LOG FOR USER AESDJC1 05:15:13 02/24/08_____________________________________________________________________________________________________________________COMPONENT TRACE FULL FORMATSYSNAME(ADCD)COMP(SYSTCPDA)SUBNAME((TCPIP))OPTIONS((FTP(20,21)))z/OS TCP/IP Packet Trace Formatter, (C) IBM 2000-2005, 2005, 2005.047FILE(TRACE)**** 2008/02/22RcdNr Sysname Mnemonic Entry Id Time Stamp Description----- -------- -------- -------- --------------- -----------------------------------------------------------------------------------------------------------------804059 ADCD PACKET 00000004 20:48:42.883175 Packet TraceFrom Interface : ETH1 Device: LCS Ethernet Full=52Tod Clock : 2008/02/22 20:48:42.883162 Intfx: 4Sequence # : 0 Flags: PktIpHeader: Version : 4 Header Length: 20Tos : 00 QOS: Routine Normal ServicePacket Length : 52 ID Number: AD04Fragment : DontFragment Offset: 0TTL : 64 Protocol: TCP CheckSum: 23F2 FFFFSource : 137.72.43.110Destination : 137.72.43.207TCPSource Port : 28265 () Destination Port: 21 (ftp)Sequence Number : 1439084340 Ack Number: 0Header Length : 32 Flags: SynWindow Size : 65534 CheckSum: 91D2 FFFF Urgent Data Pointer: 0000Option : Max Seg Size Len: 4 MSS: 1460Option : NOPOption : Window Scale OPT Len: 3 Shift: 0Option : NOPOption : NOPOption : SACK PermittedIP Header : 20000000 45000034 AD044000 400623F2 89482B6E 89482BCFCopyright (C) 2009 Applied Expert Systems, Inc. 65

z/VM Packet TraceTo enable the trace: NETSTAT OBEY PACKETTRACESIZE 256 NETSTAT OBEY TRACEONLY ETH0 ENDTRACEONLYTo start data collection:TRSOURCE ID TCP TYPE GT BLOCK FOR USER tcpip_useridTRSOURCE ENABLE ID TCPTo stop data collection: NETSTAT OBEY PACKETTRACESIZE 0 NETSTAT OBEY TRACEONLY ENDTRACEONLY TRSOURCE DISABLE ID TCPTo analyze a TRF trace file: IPFORMAT command Use the TRF2TCPD utility to convert the TRF file to pcap(tcpdump) formatCopyright (C) 2009 Applied Expert Systems, Inc. 66

Packet Trace Analysis Analyze one FTP session at a time Separate the Control Session from the DataSession Check session initiation and termination Check FTP commands and replies Look for packet retransmissions and unusuallong response times Elapsed time between packets TCP window sizeCopyright (C) 2009 Applied Expert Systems, Inc. 67

Packet Trace AnalysisRST flag in TCP header Abnormal condition and the receiver wants to abortthe connection; e.g., Receipt of any TCP segment from a host withwhich the receiver does not currently have aconnection Receipt of an invalid/incorrect Sequence Numberor Acknowledge Number Receipt of a SYN on a port without a listener Possible time-out on the Control Connection Issue netstat command on both sides to verifyconnectionsRun packet traces on both sides and compareCapture “good” traces for future comparisonsCopyright (C) 2008 2009 Applied Expert Systems, Inc.68

Packet Trace – Unfiltered by Application PortsCopyright (C) 2009 Applied Expert Systems, Inc. 69

Packet Trace – Filtered by Application PortsCopyright (C) 2009 Applied Expert Systems, Inc. 70

Control Connection (Active FTP)Copyright (C) 2008 2009 Applied Expert Systems, Inc.71

Packet Details – PORT commandData Connection Port =256*109 + 225 = 28129Copyright (C) 2009 Applied Expert Systems, Inc. 72

Control Connection (Passive FTP)Copyright (C) 2008 2009 Applied Expert Systems, Inc.73

Packet Details – Reply Code 227 in response to PASVData Connection Port =256*14 + 122 = 3706Copyright (C) 2008 2009 Applied Expert Systems, Inc.74

FTP Control Connection: exceptionsCopyright (C) 2009 Applied Expert Systems, Inc. 75

FTP Control Session: “451 transfer aborted”Copyright (C) 2009 Applied Expert Systems, Inc. 76

FTP Data Session: “connection reset by peer”Copyright (C) 2009 Applied Expert Systems, Inc. 77

FTP Data Session: “connection reset by peer”Copyright (C) 2009 Applied Expert Systems, Inc. 78

Trace Comparison – Host vs. WindowsCopyright (C) 2008 2009 Applied Expert Systems, Inc.79

FTP Diagnosis of Data Connection IssuesObtain a packet or OSAENTA trace.Obtain a server trace with options BAS and FLO. F ftpx,DEBUG=(FLO,BAS)Find the PORT or PASV command.Determine the IP address and port number for dataconnection.Check for session establishment: SYN, SYN ACK, etc.Missing SYN ACK may indicate firewall issue; e.g., forPassive FTP:FTP.DATA - PASSIVEDATAPORTS – should match firewall rangeTCP PROFILE – PORTRANGE E.g., firewall allows 5000-6000;PASSIVEPORTS(5000,6000) PORTRANGE 5000 1001 AUTHPORTCopyright (C) 2009 Applied Expert Systems, Inc. 80

FTP TuningUse the right Data Type (EBCDIC vs. ASCII)TCP Window Size: the maximum amount of data thatcan be in the network at any time for a single connection. Optimal TCP Window Size =Bottleneck Bandwidth * Round-trip Time (RTT)E.g., the slowest link=45 Mbit/sec, RTT=20ms45 Mbit/sec * 20ms= 45,000,000 bits/sec * .020 sec= 900,000 bits = 109.86 KBytesCopyright (C) 2009 Applied Expert Systems, Inc. 81

FTP TuningRTT Ping with default packet size; e.g., 256 Ping with “average” FTP packet size SMF 119 TCP Connection Termination Record (RTT attime of connection close) Packet traceWindow Size SMF 119 TCP Connection Termination Record Packet trace D NETSTAT,CONFIGTCPCONFIG TCPSENDBFRSIZE 64K TCPRCVBUFRSIZE 64K Windows tools: Dr. TCP, TCP Optimizer, etc.Copyright (C) 2009 Applied Expert Systems, Inc. 82

FTP AnalysisGlobal usage patterns:Total FTP sessions Total FTP bytes FTP server vs. FTP client activities When are heavy FTPs done?Are gigabyte file transfers done?Unauthorized attempts – logon failure reasonHeavy users: Typically may be responsible for 80% of the workload Which data sets are most heavily used?Certain type? SEQ/ JES/ SQL?Can reposition or copy data sets for better performance?Heavy data set usage: Often moved around or duplicates made forsecurity and other redundancy reasonsFTP performance analysis: throughput and response timeCopyright (C) 2009 Applied Expert Systems, Inc. 83

FTP AnalysisFTP Failure Analysis – Who and Why When do failures occur? At a specific time of day? Who are the top failing clients and datasets? Failure with one dataset or all datasets?Logon failure vs. data transfer failureCorrelated with heavy usage? Client side or server side – or in the middle?Control connection or data connection? Direction of flow – try reversing the role of client/serverFTP Historical Trending and Analysis Number of server/client sessionsTypes of transfers (SEQ/SQL/JES) Amount of transfers Failures Heavy usage hours ThroughputWorkload analysis = proactive problem diagnosticsCopyright (C) 2009 Applied Expert Systems, Inc. 84