Guidelines on debugging and introduction to Totalview

<strong>Guidelines</strong> on debugging andintroduction to TotalviewDominique Lucas – George Mozdzynski1 Com hpcf training course - 2004 DebuggingECMWF

Outline• Other approaches to debugging• Debugging with totalview2 Com hpcf training course - 2004 DebuggingECMWF

Introduction• “Programming is an art” … and so is debugging“You don’t need a sledgehammer to crack a nut”• Time spent writing better code will be more thancompensated by reduced debug time3 Com hpcf training course - 2004 DebuggingECMWF

Don’t Panic• Most problems are trivial and easy to fix.• Look at stack trace/point of failure.• Intuition, experience, luck all play a part.• Check your code “again”.• Explain/show your code to somebody else.• Use make to build your libraries/models, to guarantee thatthe same options are used everywhere.4 Com hpcf training course - 2004 DebuggingECMWF

Some suggestions• Is the problem reproducible on a rerunDoes it fail in exactly the same way and place• Try THREADS=1• No stack trace?suspect ‘you’ have trashed memory• Try to reproduce problem at a lower resolution5 Com hpcf training course - 2004 DebuggingECMWF

The Universal Debug Tool(the print/write statement)• Will always be there for you!• What to write out.• Must be selective to keep file size(s) manageable.• Can be activated with an DEBUG variable or Namelistentry.6 Com hpcf training course - 2004 DebuggingECMWF

Memory Constraints• Task stack limit 4 Gbytes• Thread stack limitMaster thread 4 GbytesOther threads 256 Mbytes• Large arrays (>1 Mbyte) should be declared allocatableallocatable arrays use the (per task) heapno practical limit on heap size• Fortran 90/95 (xlf90/xlf95) put local data on stack.• Fortran 77 (xlf) puts statically allocated local data on heap.7 Com hpcf training course - 2004 DebuggingECMWF

Eoj – check memory and CPU utilisationeoj vers1.4 run at Wed Jun 11 17:35:46 GMT 2003 on hpca2501 for jobstep hpca2301.347796.0Queued : Wed Jun 11 17:11:09 GMT 2003 for 898 secondsDispatched : Wed Jun 11 17:26:07 GMT 2003 for 579 secondsJob Name : edhm_model_fcgroup1Step Name : 0Owner : rdxUnix Group : rdAccount : ecpreopsSTDIN : /dev/nullSTDOUT : /hpca/rdx_dir/log/mpm/edhm/fc/fcgroup1/model.1STDERR : /hpca/rdx_dir/log/mpm/edhm/fc/fcgroup1/model.1Class : npStep Type : General ParallelNode Usage : sharedStep Cpus : 8Total Tasks :Blocking :Node actual : 1Adapter Req. : (csss,MPI,shared,US)Resources : ConsumableCpus(1) ConsumableMemory(900.000 mb)#*#* Next 3 times NOT up-to-date (TOTAL CPU TIME given later IS accurate)Step User Time : 00:13:41.940000Step System Time : 00:00:33.760000Step Total Time : 00:14:15.700000 (855.7 secs)#*#* Last 3 times NOT up-to-date (TOTAL CPU TIME given later IS accurate)Context switches : involuntary = 28637, voluntary = 12378per second = 49 21Page faults : with I/O = 13056, without I/O = 1203613per second = 22 2078 Node ? #T #t secs/CPU (Eff%) (Now%) max/TSK mb (Eff%) (Now% - mb ) Task list-------- - -- -- ---------- ------ ------ ---------- ------ -------------- ---------hpca1503 M 8 1 120.33 ( 20%) ( 52%) 764.85 ( 84%) ( 88% - 7680) 0:1:2:3:4:5:6:7:-------- - -- -- ---------- ------ ------ ---------- ------ -------------- ---------Elapsed = 579 secs 900 mb = ConsumableMemoryCPU Tot = 962.64 ( 0+00:16:02) Average: 963 s/node, 120 s/taskSystem Billing Units used by this jobstep = 1.0378 Com hpcf training course - 2004 DebuggingECMWF

Debugging• checking:argument checking: -qextchk• xlf –qextchk prog.f –o progchecking done at compilation/linkingarray bounds checking: -C• xlf –C prog.f –o prog• ./progchecking done at runtimeundefined reference checking• xlf –qinitauto=FF –qflttrap:inv:en prog.f –o progchecking done at runtime9 Com hpcf training course - 2004 DebuggingECMWF

Debugging – floating point exceptions• nothing generated on floating point exception.• Floating point trapping% xlf –qflttrap=overflow:invalid:zerodivide:enable \ –qsigtrap prog.f –o prog% ./prog...CPU overhead of up to 20%.• Core files – how to get a traceback % dbx ./prog core

ECMWF local signal trap - ECLIBINTEGER*4 CORE_DUMP_FLAG, IRETURN, SIGNALS(1)REAL ACORE_DUMP_FLAG = 0SIGNALS(1) = 0IRETURN = SIGNAL_TRAP(CORE_DUMP_FLAG, SIGNALS)IF (IRETURN .LT. 0) THENPRINT *, 'ERROR'ELSE IF (IRETURN .EQ. 0) THENPRINT *, 'FPE TRAPPING IS NOT SET'ELSEPRINT *, 'FPE TRAPPING MODE =', IRETURNENDIFcall b(-2.)endsubroutine b(a)real awrite(*,*)sqrt(a)returnEND• Link using $ECLIB, e.g.xlf -c prog.fxlf prog.o -o prog $ECLIB11 Com hpcf training course - 2004 DebuggingECMWF

Signal_trap - arguments• CORE_DUM_FLAG:= 0: no core dumpedNot = 0: core dumped• SIGNALS – integer array with signals to trapSignals(1)=0 => SIGFPE, SIGILL, SIGBUS,SIGSEGV, SIGXCPU.See “kill –l “ for list of signals.12 Com hpcf training course - 2004 DebuggingECMWF

Totalview• Re-compile (part of) your application without optimization -qnooptimize for all routines -qsmp=noopt for routines with OpenMP Beware –qsmp=omp implies optimization• Command Line interface exists $ totalviewcli• Recommended use of totalview in batch mode, i.e. with theGUI version.13 Com hpcf training course - 2004 DebuggingECMWF

Totalview – GUI interface• Before submitting batch job to launch totalview, request anX11 proxy at login time via ECaccess.• Include in your batch job the display export DISPLAY=• Include source code searchpath -searchpath=“dir_1/,dir_2/,….dir_n/”• For MPI-parallel (load-leveller jobs) totalview –searchPath=$searchpath poe –a • For serial/OpenMP only (interactive) totalview –searchPath=$searchpath -a 14 Com hpcf training course - 2004 DebuggingECMWF

15 Com hpcf training course - 2004 DebuggingECMWF



View > Lookup_Function …fshortcutscan2mdm_ orscan2mdm.F9018 Com hpcf training course - 2004 DebuggingECMWF




To “restart” applicationGroup > Restart orGroup > Delete Ctrl+ZShortcut22 Com hpcf training course - 2004 DebuggingECMWF

File > Preferences23 Com hpcf training course - 2004 DebuggingECMWF

When breakpoint hit, stop: Process…is more practical24 Com hpcf training course - 2004 DebuggingECMWF


View > Dive Anew foradditional process windows26 Com hpcf training course - 2004 DebuggingECMWF

Tools > Fortran Modulesto view module data27 Com hpcf training course - 2004 DebuggingECMWF

Tools > Visualize28 Com hpcf training course - 2004 DebuggingECMWF

Set slice toreduce data29 Com hpcf training course - 2004 DebuggingECMWF


View > Sort > Descending31 Com hpcf training course - 2004 DebuggingECMWF

View module contents bydiving on module name32 Com hpcf training course - 2004 DebuggingECMWF

View all variables in aroutine by diving onname in Stack Frame33 Com hpcf training course - 2004 DebuggingECMWF

Tools > Memory UsageT L 255 model using 8 CPUs of standard memory node34 Com hpcf training course - 2004 DebuggingECMWF


Logicals0 = false, 1 = trueWill be displayed asT/F in future release36 Com hpcf training course - 2004 DebuggingECMWF



http://www.etnus.com/Support/docs/39 Com hpcf training course - 2004 DebuggingECMWF

…or just type totalview and use Help pages• …you don’t need a program to debug …40 Com hpcf training course - 2004 DebuggingECMWF

Guidelines on debugging and introduction to Totalview

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?