Problem Solving and Troubleshooting in AIX 5L - IBM Redbooks

More documents

Recommendations

Info

Take Note! Before using this information and the product it supports, be sure to read the general information in “Special notices” on page 439. Second Edition (January 2002) This edition applies to IBM ^ pSeries and RS/6000 Systems for use with the AIX 5L Operating System Version 5.1 and is based on information available in August, 2001. Comments may be addressed to: IBM Corporation, International Technical Support Organization Dept. JN9B Building 003 Internal Zip 2834 11400 Burnet Road Austin, Texas 78758-3493 When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you. © Copyright International Business Machines Corporation 1999-2002. All rights reserved. Note to U.S Government Users – Documentation related to restricted rights – Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp.
Contents Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii The team that wrote this redbook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Special notice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix IBM trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Chapter 1. Problem determination introduction. . . . . . . . . . . . . . . . . . . . . . 1 1.1 Problem determination process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1.1 Defining the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1.2 Gathering information from the user . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1.3 Gathering information from the system . . . . . . . . . . . . . . . . . . . . . . . . 5 1.1.4 Resolving the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.1.5 Obtaining software fixes and hardware microcode updates . . . . . . . . 6 1.1.6 Other relevant documentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2 Troubleshooting starting point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.2.1 Boot path flowchart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.2.2 Symptom index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.3 Avoiding problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.3.1 System health check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Chapter 2. Error logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.1 Error logging overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.2 Error log file processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.2.1 Error templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.2.2 Error messages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.3 Error log management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.3.1 Viewing the error log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.3.2 Reading a summary error log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.3.3 Reading error logs in detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.3.4 Examples of detailed error reports . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.4 Finding a core dump. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Chapter 3. Boot problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.1 Types of machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.2 The boot process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 © Copyright IBM Corp. 2001 iii
Page 1: Problem Solving and Troubleshooting
Page 6 and 7: 3.2.1 Pre-boot state . . . . . . .
Page 8 and 9: 6.1.4 LVM device driver . . . . . .
Page 10 and 11: 8.1.1 Removing devices. . . . . . .
Page 12 and 13: 12.2.3 Use splp to look at printer
Page 14 and 15: xii Problem Solving and Troubleshoo
Page 16 and 17: xiv Problem Solving and Troubleshoo
Page 18 and 19: xvi Problem Solving and Troubleshoo
Page 20 and 21: KyeongWon Jeong is a Consulting IT
Page 22 and 23: We want our IBM Redbooks to be as h
Page 24 and 25: Although the title of this redbook
Page 26 and 27: The following questions should be a
Page 28 and 29: should be supplied to IBM. If possi
Page 30 and 31: Document Form number RS/6000 and pS
Page 32 and 33: pSeries and RS/6000 Model Form numb
Page 34 and 35: Does machine power on and stay powe
Page 36 and 37: From Figure 1-2 on page 13. Do you
Page 38 and 39: Symptom Possible cause Refer to LED
Page 40 and 41: ► Check disk space availability o
Page 42 and 43: 2.1 Error logging overview The erro
Page 44 and 45: 2.2.2 Error messages The template a
Page 46 and 47: 35BFC499 0821161601 P H cd0 DISK OP
Page 48 and 49: There are some new options introduc
Page 50 and 51: PERF The performance of the device
Page 52 and 53: VPD Vital product data. The content
Page 54 and 55:
SENSE DATA 0000 0000 0000 0000 0000
Page 56 and 57:
__filbuf 2C0 fgets 134 ftp_fgets 9C
Page 58 and 59:
AIX 5L Version 5.1 has changed the
Page 60 and 61:
usr/lib/libbsd.a /usr/lib/libbind.a
Page 62 and 63:
3.1 Types of machines Certain proce
Page 64 and 65:
42 Problem Solving and Troubleshoot
Page 66 and 67:
ackplane, and on the 7015 R series,
Page 68 and 69:
Boot process differences On the S70
Page 70 and 71:
3.2.4 Boot stage three Once the ico
Page 72 and 73:
Access boot list on PCI machine To
Page 74 and 75:
Machine hang shortly after power on
Page 76 and 77:
containing hd5, return to step 5 an
Page 78 and 79:
Note: Power cables must be removed
Page 80 and 81:
3.6 LED 551, 552, 554, 555, 556, an
Page 82 and 83:
7. If there is a version of v3fshel
Page 84 and 85:
# fsck -y /dev/hd4 # fsck -y /dev/h
Page 86 and 87:
3.6.3 LED 553 halt 12.If you copied
Page 88 and 89:
export PATH 8. /etc/environment is
Page 90 and 91:
3.7 No login prompt If the system b
Page 92 and 93:
4.1 Introduction Your system genera
Page 94 and 95:
Option 1: Using the Web-based Syste
Page 96 and 97:
After selecting this sequence, the
Page 98 and 99:
Note: AIX releases prior to Version
Page 100 and 101:
Option 3: Using the command line To
Page 102 and 103:
The -l flag logs the command output
Page 104 and 105:
Option 3: Using the command line Ru
Page 106 and 107:
Figure 4-6 Always allow system dump
Page 108 and 109:
LED code sysdumpdev status 0c5 -3 I
Page 110 and 111:
SYSTEM MESSAGES: AIX Version 5.1 St
Page 112 and 113:
The snap command supports the follo
Page 114 and 115:
Forcing a dump on MCA systems To fo
Page 116 and 117:
► Creation and deletion of proces
Page 118 and 119:
nid....... 4232044C time of crash:
Page 120 and 121:
Example 4-16 stack trace back (0)>
Page 122 and 123:
........... ganchor :00000000 THREA
Page 124 and 125:
.lowthresh_init 019E7028 .strthresh
Page 126 and 127:
► knlist [symbol] Displays addres
Page 128 and 129:
In this example, there are two leve
Page 130 and 131:
proc subcommand The proc subcommand
Page 132 and 133:
PID Process ID of the associated pr
Page 134 and 135:
Handling crash output Some crash su
Page 136 and 137:
114 Problem Solving and Troubleshoo
Page 138 and 139:
5.1 General advice 5.1.1 Diagnostic
Page 140 and 141:
- The GH value identifies a port, a
Page 142 and 143:
5.2.1 Making sense of the error log
Page 144 and 145:
ROS Level and ID............5 5A Se
Page 146 and 147:
http://www.ibm.com/servers/eserver/
Page 148 and 149:
Example 5-2 FUNCTION SELECTION menu
Page 150 and 151:
Concurrent mode provides a way to r
Page 152 and 153:
Once the keyboard input has been pr
Page 154 and 155:
The machine types that support boot
Page 156 and 157:
3. Select Service Aids or Task Sele
Page 158 and 159:
10.Enter the client address, server
Page 160 and 161:
The microcode download facility is
Page 162 and 163:
Note: The 7248 43P SMS microcode is
Page 164 and 165:
The CHRP machines models, 7017-S70,
Page 166 and 167:
Low-voltage differential/single-end
Page 168 and 169:
Inforamtion for Micro Channel Bus S
Page 170 and 171:
This message will be produced when
Page 172 and 173:
Failed The drive was failed by the
Page 174 and 175:
Feature Code 5.8.2 SSA devices 6219
Page 176 and 177:
Note: The cfgmgr command should not
Page 178 and 179:
5.9.2 External tapes and tape libra
Page 180 and 181:
condensation on the tape surface th
Page 182 and 183:
When checking the settings for thes
Page 184 and 185:
Page 186 and 187:
6.1 Logical Volume Manager (LVM) Th
Page 188 and 189:
specific program using the API cert
Page 190 and 191:
6.2.3 Physical volumes 6.2.4 Volume
Page 192 and 193:
6.3.1 Data integrity Depending on t
Page 194 and 195:
Here is an example: # varyoffvg dat
Page 196 and 197:
You can use this log file to determ
Page 198 and 199:
6.4 JFS problems As with the LVM, m
Page 200 and 201:
- Dot dot (..) link missing or not
Page 202 and 203:
If the jfslog device for a file sys
Page 204 and 205:
There are many reasons for the repl
Page 206 and 207:
0001 0110 hdisk2 0110 hdisk3 0002 0
Page 208 and 209:
2. Unmount all file systems on the
Page 210 and 211:
► For MCA machines Booting from a
Page 212 and 213:
You can see that the HOT SPARE flag
Page 214 and 215:
At this point, do the following: 1.
Page 216 and 217:
. Change/show the use of an SSA Phy
Page 218 and 219:
7.1 General network problem isolati
Page 220 and 221:
traceroute uses the UDP protocol an
Page 222 and 223:
If the answer is no, you might have
Page 224 and 225:
If external resolvers query your se
Page 226 and 227:
7.2.5 Dynamic or static routing If
Page 228 and 229:
The subsystem does not support the
Page 230 and 231:
Verify the interface state with the
Page 232 and 233:
or the SMIT fast path smit chinet t
Page 234 and 235:
In the General Statistics: section
Page 236 and 237:
This problem can occur if the ELAN
Page 238 and 239:
If you are receiving output similar
Page 240 and 241:
If the error message is: ping: send
Page 242 and 243:
Use the vi editor with the set list
Page 244 and 245:
Below is the normal output that is
Page 246 and 247:
If telnet is hanging after entering
Page 248 and 249:
3. Verify using the following comma
Page 250 and 251:
7.4.3 mbufs 7.4.4 thewall To check
Page 252 and 253:
7.4.8 traceroute 7.4.9 netstat The
Page 254 and 255:
udp: 269378 datagrams received 0 in
Page 256 and 257:
7.4.12 tcpdump 7.4.13 no The follow
Page 258 and 259:
VIPA can be configured with the mkd
Page 260 and 261:
9.3.187.185 sydney 9.3.187.186 napo
Page 262 and 263:
tiny YP_LAST_MODIFIED 0706832572 YP
Page 264 and 265:
The ypwhich command is inconsistent
Page 266 and 267:
Run nismatch username.domainname.cr
Page 268 and 269:
Logs grow too large Failure to regu
Page 270 and 271:
If an NIS+ database is corrupt, you
Page 272 and 273:
► Verify that the mountd, portmap
Page 274 and 275:
► mount: access denied ... Your m
Page 276 and 277:
The NFS server (host name) is down.
Page 278 and 279:
domain names of hosts so that users
Page 280 and 281:
7.8.1 AIX as a PPP client (outgoing
Page 282 and 283:
Where /tmp/ppp is the file set up w
Page 284 and 285:
Use the following command to connec
Page 286 and 287:
7.9 Sendmail troubleshooting AIX 5L
Page 288 and 289:
When a workstation that is behind a
Page 290 and 291:
8.1 Handling graphics devices Befor
Page 292 and 293:
8.3 X server 8.3.1 Start X server
Page 294 and 295:
3. Wrong display. The X server mayb
Page 296 and 297:
4. To find the location of the hang
Page 298 and 299:
► TCP sockets This is applicable
Page 300 and 301:
If this fails to start, check the f
Page 302 and 303:
. . number of extensions: 8 XInputE
Page 304 and 305:
8.6.3 Problem with login 8.6.4 CDE
Page 306 and 307:
While some of these filesets increa
Page 308 and 309:
In general, the causes of user appl
Page 310 and 311:
9.2.3 Name conflict Depending on ho
Page 312 and 313:
exist. 290 Problem Solving and Trou
Page 314 and 315:
AIX 5L Version 5.1 binary compatibi
Page 316 and 317:
► The nlist() interface ► DCE t
Page 318 and 319:
► POSIX compliant AIX is POSIX an
Page 320 and 321:
in the wget.spec file to: %define v
Page 322 and 323:
Finding memory-leaking programs A m
Page 324 and 325:
The following resource managers are
Page 326 and 327:
State = "Active" condition-response
Page 328 and 329:
Detection Time-out [2] # Process Pr
Page 330 and 331:
1735-023 shconf: Priority Problem C
Page 332 and 333:
► Hardware problems Problems with
Page 334 and 335:
10.2 Monitoring performance Due to
Page 336 and 337:
This command displays one line of o
Page 338 and 339:
Useful flags -P Proc ID, ... | ALL
Page 340 and 341:
Emulation and alignment detection I
Page 342 and 343:
There are several general rules tha
Page 344 and 345:
Use the following commands to deter
Page 346 and 347:
Example 10-6 The memory usage of a
Page 348 and 349:
Page 350 and 351:
means no pacing. A process cannot e
Page 352 and 353:
ARG/ENV list size in 4K byte blocks
Page 354 and 355:
scsi1 5415.8 1354.0 11644 0 Disks:
Page 356 and 357:
-o file: open output file (default
Page 358 and 359:
10.3.1 Using the netpmon command Th
Page 360 and 361:
The type of report can be customize
Page 362 and 363:
enice Alters priority of running pr
Page 364 and 365:
11.1 Introduction The trace system
Page 366 and 367:
In AIX 5L Version 5.1, trace now su
Page 368 and 369:
Interactive mode tracing If the tra
Page 370 and 371:
SAVE PREVIOUS log file? [no] + Omit
Page 372 and 373:
# trcstop To trace the same command
Page 374 and 375:
-O Options Allows more detailed opt
Page 376 and 377:
Copy the following files from the s
Page 378 and 379:
Page 380 and 381:
- Is the data formatted properly by
Page 382 and 383:
- PowerPC: lssrc -s qdaemon 12.1.2
Page 384 and 385:
the printer is attached to a local
Page 386 and 387:
- The right size of paper was not r
Page 388 and 389:
for “character special device.”
Page 390 and 391:
Make sure to look at the space that
Page 392 and 393:
Does the printer print correctly fo
Page 394 and 395:
12.1.6 The output spacing is bad Th
Page 396 and 397:
echo “\033)0B\c” The command us
Page 398 and 399:
Make sure that the lpd server is ac
Page 400 and 401:
Are the flags correct? This is a go
Page 402 and 403:
12.2.4 Printing to a file Printing
Page 404 and 405:
12.2.7 Using iptrace to trace netwo
Page 406 and 407:
Capturing a file as it goes over th
Page 408 and 409:
The other method to capture flags i
Page 410 and 411:
16-port adapter considerations The
Page 412 and 413:
System V defined printer attributes
Page 414 and 415:
The lpNet log shows you when networ
Page 416 and 417:
When looking at the file, it may be
Page 418 and 419:
Interface file banner selection For
Page 420 and 421:
These tools can be expanded until y
Page 422 and 423:
Remote: rs1230a.itsc.austin.ibm.com
Page 424 and 425:
12.3.14 Troubleshooting HP JetDirec
Page 426 and 427:
Extraneous files in qdir Another pr
Page 428 and 429:
produced this error message in msg1
Page 430 and 431:
Line descriptions for the file abov
Page 432 and 433:
This is also useful to see how diff
Page 434 and 435:
Configuration banner selection Bann
Page 436 and 437:
12.5.2 Cannot change attributes bec
Page 438 and 439:
Although the hpnpcfg script is undo
Page 440 and 441:
13.1 Debugging a network boot probl
Page 442 and 443:
ootps dgram udp wait root /usr/sbin
Page 444 and 445:
database shows the correct platform
Page 446 and 447:
For other NIM LED codes, see Table
Page 448 and 449:
13.2.1 Producing debug output from
Page 450 and 451:
13.2.2 Producing debug output from
Page 452 and 453:
Time: 35 LEDS: 0x539 Number of runn
Page 454 and 455:
3. Select the showlog operation. 4.
Page 456 and 457:
► 7015 R10/R20/R21 Service Guide,
Page 458 and 459:
► AIX 5L Version 5.1 System Manag
Page 460 and 461:
Page 462 and 463:
The following terms are trademarks
Page 464 and 465:
LES LAN Emulation Server LUM Licenc
Page 466 and 467:
Page 468 and 469:
adapter part number 146 adapter pla
Page 470 and 471:
more 23 mount 176 mpcfg 42 netpmon
Page 472 and 473:
du 176 dummy queue device 404 dump
Page 474 and 475:
hardware surveillance 41 hardware t
Page 476 and 477:
lo0 203 loader 289 local traffic 22
Page 478 and 479:
od 409 od subcommand 110 ODM 20, 25
Page 480 and 481:
eadelf 297 real memory 319 realloc(
Page 482 and 483:
System V print subsystem 390 stop b
Page 484 and 485:
serial pinouts 388 WLM 337 wlmmon c
Page 488:
Problem Solving and Troubleshooting
show all

Problem Solving and Troubleshooting in AIX 5L - IBM Redbooks

Create successful ePaper yourself

Delete template?

Save as template?