Problem Reporting and Analysis Linux on System z ... - z/VM - IBM

<strong>Problem</strong> <strong>Reporting</strong> <strong>and</strong> <strong>Analysis</strong> <strong>Linux</strong> on System z - 

How to survive a <strong>Linux</strong> Critical Situation 

Sven Schuetz 

<strong>Linux</strong> on System z Development <strong>and</strong> Service 

sven@de.ibm.com 

1 

© 2011 IBM Corporation

IBM 

Live Virtual Class – <strong>Linux</strong> on System z 

Agenda 

Introduction 

How to help us to help you 

Systems monitoring 

How to dump a <strong>Linux</strong> on System z 

Real Customer cases 

2 




Introductory remarks 

<strong>Problem</strong> analysis looks straight forward on the charts but it might 

have taken weeks to get it done. 

A problem does not necessarily show up on the place of origin 

The more information is available, the sooner the problem can be 

solved, because gathering <strong>and</strong> submitting additional information 

again <strong>and</strong> again usually introduces delays. 

This presentation can only introduce some tools <strong>and</strong> how the tools 

can be used, comprehensive documentation on their capabilities is 

to be found in the documentation of the corresponding tool. 

Do not forget to update your systems 

3 




Describe the problem 

Get as much information as possible about the circumstances: 

– What is the problem 

– When did it happen (date <strong>and</strong> time, important to dig into logs ) 

– Where did it happen One or more systems, production or test environment 

– Is this a first time occurrence 

– If occurred before: how frequently does it occur 

– Is there any pattern 

– Was anything changed recently 

– Is the problem reproducible 

Write down as much information as possible about the problem! 

4 




Describe the environment 

Machine Setup 

–Machine type (z196, z10, z9, ...) 

–Storage Server (ESS800, DS8000, other vendors models) 

–Storage attachment (FICON, ESCON, FCP, how many channels) 

–Network (OSA (type, mode), Hipersocket) ... 

Infrastructure setup 

–Clients 

–Other Computer Systems 

–Network topologies 

–Disk configuration 

Middleware setup 

–Databases, web servers, SAP, TSM, (including version information) 

5 




Trouble Shooting First-Aid Kit (1/2) 

Install packages required for debugging 

– s390-tools/s390-utils 

• dbginfo.sh 

– sysstat 

• sadc/sar 

• iostat 

– procps 

• vmstat, top, ps 

– net-tools 

• netstat 

– dump tools crash / lcrash 

• lcrash (lkcdutils) available with SLES9 <strong>and</strong> SLES10 

• crash available on SLES11 

• crash in all RHEL distributions 

6 




Trouble Shooting First-Aid Kit (2/2) 

Collect dbginfo.sh output 

– Proactively in healthy system 

– When problems occur – then compare with healthy system 

Collect system data 

– Always archive syslog (/var/log/messages) 

– Start sadc (System Activity Data Collection) service when appropriate (please 

include disk statistics) 

– Collect z/VM MONWRITE Data if running under z/VM when appropriate 

When System hangs 

– Take a dump 

• Include System.map, Kerntypes (if available) <strong>and</strong> vmlinux file 

– See “Using the dump tools” book on 

http://download.boulder.ibm.com/ibmdl/pub/software/dw/linux390/docu/l26ddt02.pdf 

Enable extended tracing in /sys/kernel/debug/s390dbf for subsystem 

7 




dbginfo Script (1/2) 

dbginfo.sh is a script to collect various system related files, for 

debugging purposes. It generates a tar-archive which can be 

attached to PMRs / Bugzilla entries 

part of the s390-tools package in SUSE <strong>and</strong> recent Red Hat 

distributions 

– dbginfo.sh gets continuously improved by service <strong>and</strong> development 

Can be downloaded at the developerWorks website directly 

http://www.ibm.com/developerworks/linux/linux390/s390-tools.html 

It is similar to the RedHat tool sosreport or supportconfig from Novell 

8 

root@larsson:~> dbginfo.sh 

Create target directory /tmp/DBGINFO-2011-01-15-22-06- 

20-t6345057 

Change to target directory /tmp/DBGINFO-2011-01-15-22- 

06-20-t6345057 

[...] 




dbginfo Script (2/2) 

<strong>Linux</strong> Information: 

– /proc/[version, cpu, meminfo, slabinfo, modules, partitions, devices ...] 

– System z specific device driver information: /proc/s390dbf (RHEL 4 only) or 

/sys/kernel/debug/s390dbf 

– Kernel messages /var/log/messages 

– Reads configuration files in directory /etc/ [ccwgroup.conf, modules.conf, fstab] 

– Uses several comm<strong>and</strong>s: ps, dmesg 

– Query setup scripts 

• lscss, lsdasd, lsqeth, lszfcp, lstape 

– And much more 

z/VM information: 

– Release <strong>and</strong> service Level: q cplevel 

– Network setup: q [lan, nic, vswitch, v osa] 

– Storage setup: q [set, v dasd, v fcp, q pav ...] 

– Configuration/memory setup: q [stor, v stor, xstore, cpus...] 

– When the system runs as z/VM guest, ensure that the guest has the appropriate privilege 

class authorities to issue the comm<strong>and</strong>s 

9 




SADC/SAR 

Capture <strong>Linux</strong> performance data with sadc/sar 

– CPU utilization 

–Disk I/O overview <strong>and</strong> on device level 

–Network I/O <strong>and</strong> errors on device level 

–Memory usage/Swapping 

–… <strong>and</strong> much more 

–Reports statistics data over time <strong>and</strong> creates average values for 

each item 

SADC example (for more see man sadc) 

– System Activity Data Collector (sadc) --> data gatherer 

– /usr/lib64/sa/sadc [options] [interval [count]] [binary outfile] 

– /usr/lib64/sa/sadc 10 20 sadc_outfile 

10 




SADC/SAR (cont'd) 

– /usr/lib64/sa/sadc -d 10 sadc_outfile 

– -d option: statistics for disk 

– Should be started as a service during system start 

✱ SAR example (for more see man sar) 

– System Activity Report (sar) comm<strong>and</strong> --> reporting tool 

– sar -A 

– -A option: reports all the collected statistics 

– sar -A -f sadc_outfile >sar_outfile 

Please include the binary sadc data <strong>and</strong> sar -A output when 

submitting SADC/SAR information to IBM support 

11 




CPU utilization 

Per CPU values: 

watch out for 

system time (kernel time) 

iowait time (slow I/O subsystem) 

steal time (time taken by other guests) 

12 




Disk I/O rates 

read/write operations 

- per I/O device 

- tps: transactions 

- rd/wr_secs: sectors 

is your I/O balanced 

Maybe you should stripe your LVs 

13 




<strong>Linux</strong> on System z dump tools 

DASD dump tool 

14 

– Writes dump directly on DASD partition 

– Uses s390 st<strong>and</strong>alone dump format 

– ECKD <strong>and</strong> FBA DASDs supported 

– Single volume <strong>and</strong> multiple volume (for large systems) dump possible 

– Works in z/VM <strong>and</strong> in LPAR 

SCSI dump tool 

– Writes dump into filesystem 

– Uses lckd dump format 

– Works in z/VM <strong>and</strong> in LPAR 

VMDUMP 

– Writes dump to vm spool space (VM reader) 

– z/VM specific dump format, dump must be converted 

– Only available when running under z/VM 

Tape dump tool 

– Writes dump directly on ESCON/FICON Tape device 

– Uses s390 st<strong>and</strong>alone dump format 




DASD dump tool – general usage 

1. Format <strong>and</strong> partition dump device 

root@larsson:~> dasdfmt f /dev/dasd b 4096 

root@larsson:~> fdasd /dev/dasd 

2. Prepare dump device in <strong>Linux</strong> 

root@larsson:~> zipl d /dev/dasd 

3. Stop all CPUs 

4. Store Status 

5. IPL dump device 

6. Copy dump to <strong>Linux</strong> 

root@larsson:~> zgetdump /dev/ > dump_file 

15 




DASD dump under z/VM 

Prepare dump device under <strong>Linux</strong>, if possible on 64Bit environment: 

16 

root@larsson:~> zipl d /dev/dasd 

After <strong>Linux</strong> crash issue these comm<strong>and</strong>s on 3270 console: 

#cp cpu all stop 

#cp cpu 0 store status 

#cp i 

Wait until dump is saved on device: 

00: zIPL v1.6.0 dump tool (64 bit) 

00: Dumping 64 bit OS 

00: 00000087 / 00000700 MB 0 

... 

00: Dump successful 

Only disabled wait PSW on older Distributions 

Attach dump device to a linux system with dump tools installed 

Store dump to linux file system from dump device (e.g. zgetdump) 




DASD dump on LPAR (1/2) 

17 




DASD dump on LPAR (2/2) 

18 




Multi volume dump 

zipl can now dump to multiple DASDs. It is now possible to dump system 

images, which are larger than a single DASD. 

–You can specify up to 32 ECKD DASD partitions for a multi-volume dump 

What are dumps good for 

–Full snapshot of system state 

taken at any point in time 

(e.g. after a system has 

crashed, of or a running 

system) 

–Can be used to analyse 

system state beyond 

messages written to the 

syslog 

–Internal data structures not 

exported to anywhere 

Obtain messages, which have not been written to the syslog due to a 

crash 

19 




Multi volume dump (cont'd) 

How to prepare a set of ECKD DASD devices for a multi-volume 

dump (64-bit systems only) 

–We use two DASDs in this example: 

root@larsson:~> dasdfmt f /dev/dasdc b 4096 

root@larsson:~> dasdfmt f /dev/dasdd b 4096 

–Create the partitions with fdasd. The sum of the partition sizes must be 

sufficiently large (the memory size + 10 MB): 

root@larsson:~> fdasd /dev/dasdc 

root@larsson:~> fdasd /dev/dasdd 

–Create a file called sample_dump_conf containing the device nodes 

(e.g. /dev/dasdc1) of the two partitions, separated by one or more line 

feed characters 

–Prepare the volumes using the zipl comm<strong>and</strong>. 

root@larsson:~> zipl M sample_dump_conf 

[...] 

20 





To obtain a dump with the multi-volume DASD dump tool, perform 

the following steps: 

–Stop all CPUs, Store status on the IPL CPU. 

–IPL the dump tool using one of the prepared volumes, either 4711 or 

4712. 

–After the dump tool is IPLed, you'll see a messages that indicates the 

progress of the dump. Then you can IPL <strong>Linux</strong> again 



#cp ipl 4711 

Copying a multi-volume dump to a file 

–Use zgetdump without any option to copy the dump parts to a file: 

root@larsson:~> zgetdump /dev/dasdc > mv_dump_file 

21 





Display information of the involved volumes: 

root@larsson:~> zgetdump d /dev/dasdc 

'/dev/dasdc' is part of Version 1 multivolume dump,which is 

spread along the following DASD volumes: 

0.0.4711 (online, valid) 

0.0.4712 (online, valid) 

[...] 

Display information about the dump itself: 

root@larsson:~> zgetdump i /dev/dasdc 

Dump device: /dev/dasdc 

>>> Dump header information



SCSI dump tool – general usage 

1. Create partition with PCBIOS disk-layout (fdisk) 

2. Format partition with ext2 or ext3 filesystem 

3. Install dump tool: 

–mount <strong>and</strong> prepare disk : 

root@larsson:~> mount /dev/sda1 /dumps 

root@larsson:~> zipl D /dev/sda1 t dumps 

–Optional: /etc/zipl.conf: 

dumptofs=/dev/sda1 

target=/dumps 

4. Stop all CPUs 

5. Store Status 

6. IPL dump device 

23 

Dump tool creates dumps directly in filesystem 

SCSI dump supported for LPARs <strong>and</strong> as of z/VM 5.4 




SCSI dump under z/VM 

SCSI dump from z/VM is supported as of z/VM 5.4 

Issue SCSI dump 



#cp set dumpdev portname 47120763 00ce93a7 lun 47120000 

00000000 bootprog 0 

#cp ipl 4b49 dump 

To access the dump, mount the dump partition 

24 




SCSI dump on LPAR 

Select CPC image for LPAR to dump 

Goto Load panel 

Issue SCSI dump 

–FCP device 

–WWPN 

–LUN 

25 





The only method to dump NSSes or DCSSes under z/VM 

Works nondisruptive 

Create dump: 

#cp vmdump to cmsguest 

Receive dump: 

–Store the dump from the reader into CMS dump file: 

#cp dumpload 

–Transfer the dump to linux from CMS e.g. FTP 

–NEW: vmur device driver: 

root@larsson:~> vmur rec vmdump 

<strong>Linux</strong> tool to convert vmdump to lkcd format: 

root@larsson:~> vmconvert vmdump linux.dump 

<strong>Problem</strong>: Dump process relatively slow 

26 




How to obtain information about a dump 

Display information of the involved volume: 

root@larsson:~> zgetdump d /dev/dasdb 

'/dev/dasdb' is Version 0 dump device. 

Dump size limit: none 

Display information about the dump itself: 

root@larsson:~> zgetdump i /dev/dasdb1 

Dump device: /dev/dasdb1 

Dump created on: Thu Oct 8 15:44:49 2009 

Magic number: 0xa8190173618f23fd 

Version number: 3 

Header size: 4096 

Page size: 4096 

Dumped memory: 1073741824 

Dumped pages: 262144 

Real memory: 1073741824 

cpu id: 0xff00012320978000 

System Arch: s390x (ESAME) 

Build Arch: s390x (ESAME) 

>>> End of Dump header



How to obtain information about a dump (cont'd) 

Display information about the dump itself (dump header) <strong>and</strong> check if 

the dump is valid, use lcrash with options 

’-i’ <strong>and</strong> ’-d’. 

root@larsson:~> lcrash i d /dev/dasdb1 

Dump Type: s390 st<strong>and</strong>alone dump 

Machine: s390x (ESAME) 

CPU ID: 0xff00012320978000 

Memory Start: 0x0 

Memory End: 0x40000000 

Memory Size: 1073741824 

Time of dump: Thu Oct 8 15:44:49 2009 

Number of pages: 262144 

Kernel page size: 4096 

Version number: 3 

Magic number: 0xa8190173618f23fd 

Dump header size: 4096 

Dump level: 0x4 

Build arch: s390x (ESAME) 

Time of dump end: Thu Oct 8 15:45:01 2009 

28 

End Marker found! Dump is valid! 




Automatic dump on panic (SLES 10/11, RHEL 5/6): dumpconf 

The dumpconf tool configures a dump device that is used for automatic 

dump in case of a kernel panic. 

–The comm<strong>and</strong> can be installed as service script under /etc/init.d/dumpconf 

or can be called manually. 

–Start service: # service dumpconf start 

–It reads the configuration file /etc/sysconfig/dumpconf. 

–Example configuration for CCW dump device (DASD) <strong>and</strong> reipl after dump: 

ON_PANIC=dump_reipl 

DUMP_TYPE=ccw 

DEVICE=0.0.4711 

29 




Automatic dump on panic (SLES 10/11, RHEL 5): dumpconf 

(cont'd) 

–Example configuration for FCP dump device (SCSI disk): 

ON_PANIC=dump 

DUMP_TYPE=fcp 

DEVICE=0.0.4714 

WWPN=0x5005076303004712 

LUN=0x4047401300000000 

BOOTPROG=0 

BR_LBA=0 

–Example configuration for re-IPL without taking a dump, if a kernel 

panic occurs: 

ON_PANIC=reipl 

–Example of executing a CP comm<strong>and</strong>, <strong>and</strong> rebooting from device 4711 

if a kernel panic occurs: 

30 

ON_PANIC=vmcmd 

VMCMD_1="MSG Starting VMDUMP" 

VMCMD_2="VMDUMP" 

VMCMD_3="IPL 4711" 




Get dump <strong>and</strong> send it to service organization 

DASD/Tape: 

–Store dump to <strong>Linux</strong> file system from dump device: 

root@larsson:~> zgetdump /dev/ > dump_file 

–Alternative: lcrash (Compression possible) 

root@larsson:~> lcrash d /dev/dasdxx s 

SCSI: 

–Get dump from filesystem 

Additional files needed for dump analysis: 

– SUSE (lcrash tool): /boot/System.map-xxx <strong>and</strong> /boot/Kerntypes-xxx 

– Redhat & SUSE (crash tool): vmlinux file (kernel with debug info) contained in 

debug kernel rpms: 

• RedHat: kernel-debuginfo-xxx.rpm <strong>and</strong> kernel-debuginfo-common-xxx.rpm 

• SUSE: kernel-default-debuginfo-xxx.rpm 

31 




H<strong>and</strong>ling large dumps 

Compress the dump <strong>and</strong> split it into parts of 1 GB 

root@larsson:~> zgetdump /dev/dasdc1 | gzip | split b 1G 

Several compressed files such as xaa, xab, xac, .... are created 

Create md5 sums of the compressed files 

root@larsson:~> md5sum xa* > dump.md5 

Upload all parts together with the md5 information 

Verification of the parts for a receiver 

root@larsson:~> md5sum c dump.md5 

xaa: OK 

[....] 

Merge the parts <strong>and</strong> uncompress the dump 

root@larsson:~> cat xa* | gunzip c > dump 

32 




Transferring dumps 

Transferring single volume dumps with ssh 

root@larsson:~> zgetdump /dev/dasdc1 | ssh user@host "cat > 

dump_file_on_target_host" 

Transferring multi-volume dumps with ssh 

root@larsson:~> zgetdump /dev/dasdc | ssh user@host "cat > 

multi_volume_dump_file_on_target_host" 

Transferring a dump with ftp 

– Establish an ftp session with the target host, login <strong>and</strong> set the transfer mode to 

binary 

– Send the dump to the host 

root@larsson:~> ftp> put |"zgetdump /dev/dasdc1" 

 

33 




Dump tool summary 

Tool 

Environment 

St<strong>and</strong> alone tools 


DASD Tape SCSI 

VM&LPAR VM&LPAR VM 

Preparation 

Creation 

Dump 

medium 

Copy to 

filesystem 

Viewing 

zipl -d /dev/ 

ECKD or 

FBA 

Stop CPU & Store status 

ipl 

Tape cartridges 

zgetdump /dev/ 

> dump_file 

mkdir 

/dumps/mydumps 

zipl -D /dev/sda1 ... 

lcrash or crash 

LINUX file system 

on a SCSI disk 

--- 

--- 

vmdump 

VM reader 

Dumpload 

ftp ... 

vmconvert ... 

See “Using the dump tools” book on 

http://www.ibm.com/developerworks/linux/linux390/documentation_dev.html 

34 




Customer Cases 

35 




Availability: Guest spontaneously reboots 

Configuration: 

– Oracle RAC server or other HA 

solution under z/VM 

<strong>Problem</strong> Description: 

– Occasionally guests spontaneously 

reboot without any notification or 

console message 

Tools used for problem 

determination: 

– cp instruction trace of (re)IPL code 

<strong>Linux</strong> 1 

Oracle RAC 

Server 

HA Cluster 

communication 


Oracle RAC 

Server 

– Crash dump taken after trace was 

hit 

Oracle RAC Database 

36 




Availability: Guest spontaneously reboots - Steps to find root 

cause 

Question: Who rebooted the system 

Step 1 

–Find out address of (re)ipl code in the system map 

–Use this address to set instruction trace 

cd /boot 

grep machine_restart System.map2.6.16.600.54.5default 

000000000010c364 T machine_restart 

00000000001171c8 t do_machine_restart 

0000000000603200 D _machine_restart 

37 





cause (cont'd) 

Step 2 

–Set CP instruction trace on the reboot address 

–System is halted at that address, when a reboot is triggered 

CP CPU ALL TR IN R 10C364.4 

HCPTRI1027I An active trace set has turned RUN off 

CP Q TR 

NAME INITIAL (ACTIVE) 

1 INSTR PSWA 0010C3640010C367 

TERM NOPRINT NORUN SIM 

SKIP 00000 PASS 00000 STOP 00000 STEP 00000 

CMD NONE 

> 000000000010C364' STMF EBCFF0780024 >> 000000003A557D48 

CC 2 

38 






Step 3 

–Take a dump, when the (re)ipl code is hit 

cp cpu all stop 

cp store status 

Store complete. 

39 

cp i 4fc6 

Tracing active at IPL 

HCPGSP2630I The virtual machine is placed in CP mode due to a 

SOGP stop <strong>and</strong> store status from CPU 00. 

zIPL v1.6.30.24.5 dump tool (64bit) 

Dumping 64 bit OS 

00000128 / 00001024 MB 

...... 

00001024 / 00001024 MB 

Dump successful 

HCPIR450W CP entered, disabled wait PSW 00020000 80000000 

00000000 00000000 






Step 4 

–Save dump in a file 

zgetdump /dev/dasdb1 > dump_file 

Dump device: /dev/dasdb1 

>>> Dump header information > End of Dump header





Step 5 

–Use (l)crash, to find out, which process has triggered the reboot 

STACK: 

0 start_kernel+950 [0x6a690e] 

1 _stext+32 [0x100020] 

================================================================ 

TASK HAS CPU (1): 0x3f720650 (oprocd.bin): 

LOWCORE INFO: 

psw : 0x0704200180000000 0x000000000010c36a 

function : machine_restart+6 

prefix : 0x3f438000 

cpu timer: 0x7fffffff 0xff9e6c00 

clock cmp: 0x00c6ca69 0x22337760 

general registers: 

 

/var/opt/oracle/product/crs/bin/oprocd.bin 

41 

STACK: 

0 __h<strong>and</strong>le_sysrq+248 [0x361240] 

1 write_sysrq_trigger+98 [0x2be796] 

2 sys_write+392 [0x225a68] 

3 sysc_noemu+16 [0x1179a8] 




Availability: Guest spontaneously reboots (cont'd) 

<strong>Problem</strong> Origin: 

– HA component erroneously detected a system hang 

• hangcheck_timer module did not receive timer IRQ 

• z/VM 'time bomb' switch 

• TSA monitor 

z/VM cannot guarantee 'real-time' behavior if overloaded 

– Longest 'hang' observed: 37 seconds(!) 

Solution: 

– Offload HA workload from overloaded z/VM 

• e.g. use separate z/VM 

• or: run large Oracle RAC guests in LPAR 

42 




Network: network connection is too slow 


–z/VSE running CICs, connection to DB2 in <strong>Linux</strong> on System z 

–Hipersocket connection from <strong>Linux</strong> to z/VSE 

–But also applies to hipersocket connections between <strong>Linux</strong> <strong>and</strong> z/OS 


–When CICS transactions were monitored, some transactions take a 

couple of seconds instead of milliseconds 

Tools used for problem determination: 

–dbginfo.sh 

–s390 debug feature 

–sadc/sar 

–CICS transaction monitor 

43 




Network: network connection is too slow (cont'd) 

s390 debug feature 

–Check for qeth errors: 

cat /sys/kernel/debug/s390dbf/qeth_qerr 

00 01282632346:099575 2 00 0000000180b20218 71 6f 75 74 65 72 72 00 | qouterr. 

00 01282632346:099575 2 00 0000000180b20298 20 46 31 35 3d 31 30 00 | F15=10. 

00 01282632346:099576 2 00 0000000180b20318 20 46 31 34 3d 30 30 00 | F14=00. 

00 01282632346:099576 2 00 0000000180b20390 20 71 65 72 72 3d 41 46 | qerr=AF 

00 01282632346:099576 2 00 0000000180b20408 20 73 65 72 72 3d 32 00 | serr=2. 

dbginfo file 

–Check for buffer count: 

cat /sys/devices/qeth/0.0.1e00/buffer_count 

16 


–Too few inbound buffers 

44 




Network: network connection is too slow (cont'd) 


– Increase inbound buffer count (default: 16, max 128) 

– Check actual buffer count with 'lsqeth -p' 

– Set the inbound buffer count in the appropriate config file: 

• SUSE SLES10: 

- in /etc/sysconfig/hardware/hwcfg-qeth-bus-ccw-0.0.F200 

- add QETH_OPTIONS="buffer_count=128" 

• SUSE SLES11: 

- in /etc/udev/rules.d/51-qeth-0.0.f200.rules add ACTION=="add", 

SUBSYSTEM=="ccwgroup", KERNEL=="0.0.f200", 

ATTR{buffer_count}="128" 

• Red Hat: 

- in /etc/sysconfig/network-scripts/ifcfg-eth0 

- add OPTIONS="buffer_count=128" 

45 




High Disk response times 


–z10, HDS Storage Server (Hyper PAV enabled) 

–z/VM, <strong>Linux</strong> with Oracle Database 

–VM controlled Minidisks attached to <strong>Linux</strong>, LVM on top 

<strong>Problem</strong> description: 

–I/O throughput not matching expectations 

–Oracle Database shows poor performance because of that 

–One LVM volume showing significant stress 


–dbginfo.sh 

–sadc/sar 

–z/VM Monitor data 

46 




High Disk response times (cont'd) 

Observation in <strong>Linux</strong> 

Response time 

dm-9 0.00 0.00 49.75 0.00 19790.50 0.00 795.56 17.89 15.79 2.01 100.00 

Throughput 

Utilization 

Conclusion 

PAV not being utilized 

No Hyper PAV support in SLES10 SP2 

Static PAV not possible with current setup (VM controlled minidisks) 

Need to look for other ways for more parallel I/O 

–Link same minidisk multiple times to a guest 

–Use smaller minidisks <strong>and</strong> increase striping in <strong>Linux</strong> 

47 





Initial <strong>and</strong> proposed setup 

Physical Disk Logical Disk(s) VM Logical Disk(s) <strong>Linux</strong> 

1.Initial Setup 

2.Link Minidisks 

to guest 

multiple times 

Link multiple times 

multipath 

3.Smaller disks, 

more stripes 

striping 

LVM / 

Device mapper 

48 





New Observation in <strong>Linux</strong> 

Response times stay equal 

Throughput equal 

No PAV being used!! 

49 





Solution: check PAV setup in VM 

50 





Finally: 

dasdaen 133.33 0.00 278.11 0.00 12298.51 0.00 88.44 0.79 2.83 1.48 41.29 

dasdcbt 161.19 0.00 248.26 0.00 12260.70 0.00 98.77 0.91 3.47 1.88 46.77 

dasdfwc 149.75 0.00 266.17 0.00 12374.13 0.00 92.98 1.88 7.07 2.54 67.66 

dasdael 162.19 0.00 250.25 0.00 12483.58 0.00 99.77 1.90 7.57 2.86 71.64 

dasddyz 134.83 0.00 277.61 0.00 12431.84 0.00 89.56 0.75 2.71 1.68 46.77 

dasdaem 151.24 0.00 266.17 0.00 12595.02 0.00 94.64 2.01 7.61 2.82 75.12 

dasdcbr 169.65 0.00 242.79 0.00 12386.07 0.00 102.03 1.72 7.05 2.83 68.66 

dasdfwd 162.69 0.00 249.25 0.00 12348.26 0.00 99.08 1.92 7.70 2.83 70.65 

dasddyy 157.21 0.00 259.70 0.00 12409.95 0.00 95.57 2.58 9.96 3.05 79.10 

dasddyx 174.63 0.00 237.81 0.00 12374.13 0.00 104.07 1.76 7.38 2.93 69.65 

dasdcbs 144.78 0.00 272.14 0.00 12264.68 0.00 90.14 2.53 9.31 2.89 78.61 

dasda 0.00 0.00 0.00 1.00 0.00 3.98 8.00 0.01 10.00 5.00 0.50 

dasdq 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 

dasdss 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 

dasdadx 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0. 

dasdawh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 

dasdamk 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 

dasdaek 160.70 0.00 255.22 0.00 12382.09 0.00 97.03 2.27 8.95 2.88 73.63 

dasdcbq 148.76 0.00 265.67 0.00 12372.14 0.00 93.14 2.14 8.01 2.85 75.62 

dasddyw 162.19 0.00 254.23 0.00 12384.08 0.00 97.42 2.12 8.40 2.90 73.63 

dasdfwe 146.27 0.00 271.64 0.00 12419.90 0.00 91.44 2.63 9.71 2.80 76.12 

dasdfwf 162.19 0.00 249.75 0.00 12455.72 0.00 99.75 0.71 2.83 1.79 44.78 

dasdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 

dm0 0.00 0.00 1646.77 0.00 49494.53 0.00 60.11 5.08 3.04 0.36 59.70 

dm1 0.00 0.00 1665.17 0.00 49482.59 0.00 59.43 15.00 9.04 0.56 93.53 

dm2 0.00 0.00 1660.70 0.00 49432.84 0.00 59.53 13.46 8.11 0.55 90.55 

dm3 0.00 0.00 1647.26 0.00 49490.55 0.00 60.09 12.05 7.32 0.53 87.56 

dm4 0.00 0.00 1646.77 0.00 49494.53 0.00 60.11 5.08 3.04 0.36 59.70 

dm5 0.00 0.00 1665.17 0.00 49482.59 0.00 59.43 15.00 9.04 0.56 93.53 

dm6 0.00 0.00 1660.70 0.00 49432.84 0.00 59.53 13.46 8.11 0.55 90.55 

dm7 0.00 0.00 1647.26 0.00 49490.55 0.00 60.09 12.06 7.32 0.53 87.56 

dm8 0.00 0.00 0.00 1.99 0.00 7.96 8.00 0.00 0.00 0.00 0.00 

dm9 0.00 0.00 497.51 0.00 197900.50 0.00 795.56 7.89 15.79 2.01 100.00 

51 




Bonding throughput not matching expectations 


– SLES10 system, connected via OSA card <strong>and</strong> using bonding driver 


– Bonding only working with 100mbps 

– FTP also slow 


– dbginfo.sh, netperf 


– ethtool cannot determine line speed correctly because qeth does not report it 


– Ignore the 100mbps message – upgrade to SLES11 

bonding: bond1: Warning: failed to get speed <strong>and</strong> duplex 

from eth0, assumed to be 100Mb/sec <strong>and</strong> Full 

52 




Availability: Unable to mount file system after LVM changes 



– <strong>Linux</strong> HA cluster with two nodes 

– Accessing same dasds which are 

exported via ocfs2 



– Added one node to cluster, brought 

Logical Volume online 

– Unable to mount the filesystem from 

any node after that 

Tools used for problem 

determination: 

– dbginfo.sh 

OCFS2 

dasda b c d e f 

Logical Volume 

53 




Availability: Unable to mount file system after LVM changes 

(cont'd) 



– LVM metadata was 

overwritten when 

adding 3rd node 

– e.g. superblock not 

found 




– Extract meta data from 

running node 

(/etc/lvm/backup) <strong>and</strong> 

write to disk again 

OCFS2 

dasdf e c a b 

d 

{pv|vg|lv}create 

Logical Volume 

54 




Kernel panic: Low address protection 


– z10 only 

– High work load 

– The more likely the more multithreaded applications are running 


– Concurrent access to pages to be removed from the page table 


– crash/lcrash 


– Race condition in memory management in firmware 


– Upgrade to latest firmware! 

– Upgrade to latest kernels – fix to be integrated in all supported distributions 

55

Problem Reporting and Analysis Linux on System z ... - z/VM - IBM

Create successful ePaper yourself

Delete template?

Save as template?