1LpjXGL

KOMEN CHATTANOOGA RACE FOR THE CURE 5KThe McKenzie Arena, Chattanooga, TN, September 24, 2006FEMALE OVERALL1 5 1103 Jessica Marlier Collegedale TN 18:12FEMALE MASTERS OVERALL1 11 1526 Jan Gautier Chattanooga TN 19:07FEMALE GRAND MASTERS OVERALL1 104 1536 Paula Cooper Hixson TN 23:49SURVIVORS1 85 17 Liza Graves 49 Knoxville TN 23:162 186 5 Linda Andreae 46 Chattanooga TN 26:143 190 19 Melinda Ellis 47 Loookout Mtn TN 26:204 245 4 Jean Thorn 49 Hixson TN 28:465 347 11 Sandra Roebuck 51 Ooltewah TN 33:336 407 10 Patricia Nowlin 54 Chattanooga TN 41:177 408 22 Marlene Larson 57 Chattanooga TN 41:188 423 12 Cynthia Scott 59 Ooltewah TN 57:21FEMALE AGE GROUP: 8 & Under1 415 1579 Presley Batchelor Chattanooga TN 45:162 420 1525 Emilia Valenzuela Rinngold GA 49:58FEMALE AGE GROUP: 9 - 141 22 1505 Hannah Jumper Lookout Mtn TN 20:102 86 1440 Blair Harden Ringgold GA 23:173 99 1293 Haley Popp Chattanooga TN 23:394 127 1731 Shelby Balch Hixson TN 24:325 129 1374 Abby Gibbons Chattanooga TN 24:346 169 1744 Katy Richardson Chattanooga TN 25:537 181 1528 Sharroll Chappel Ringgold GA 26:038 188 1066 Crystal West Ringgold GA 26:149 198 1457 Hagan Harman Lookout Mnt. TN 26:3610 210 1419 Mary Halley Magee Lookout Mtn TN 27:1911 221 1059 Jenny Dodds Ringgold GA 27:5312 222 1214 Mollie Amerson Ooltewah TN 27:5613 225 1385 Kaitlin Wade Dalton GA 28:0914 239 1301 Ashley Riner Chattanooga TN 28:4015 240 1420 Felton Rachel Hixson TN 28:4116 253 1479 Cornelius Shelby Chattanooga TN 29:0117 257 1478 Shrum Jordan Harrison TN 29:1018 261 1061 Jordan Jones Tunnel Hill GA 29:1319 279 1456 Margaret Harman Lookout Mnt. TN 29:4520 287 1111 Anna Behrends Ringgold TN 29:5821 293 1286 Meghan Melia Hixson TN 30:2322 315 1432 Ally Fien Soddy Daisy TN 31:4623 351 1371 Caitlin Duggan Chattanooga TN 33:4624 360 1516 Rory Hutchloon Rocky Face GA 34:26

A little humble OpenStack

Typical environment• CentOS 6 or Ubuntu 14.04• CPython 2.6 or 2.7• eventlet-based concurrency model for Pythonservices• MySQL (Galera), memcache [, MongoDB]• RabbitMQ

Credits• “Debugging Python applications in Production”by Vladimir Kirillov (https://www.youtube.com/watch?v=F9FHIghn_Vk)• Brendan Gregg’s Blog (http://www.brendangregg.com/blog/index.html)

printf() debugging

printf() debugging: python-memcachedef _get_server(self, key):if isinstance(key, tuple):serverhash, key = keyelse:serverhash = serverHashFunction(key)if not self.buckets:return None, Nonefor i in range(Client._SERVER_RETRIES):server = self.buckets[serverhash % len(self.buckets)]if server.connect():# print("(using server %s)" % server,)return server, keyserverhash = serverHashFunction(str(serverhash) + str(i))return None, None

printf() debugging: just don’t do that!• the most primitive way of introspection at runtime• either always enabled or explicitly commented inthe code• limited to stdout/stderror streams• information is only (barely) usable for developers• pollutes the code when committed to VCSrepositories

Logging

Logging: basicsimport loggingFORMAT = "%(asctime)-15s %(clientip)s %(user)-8s %(message)s"logging.basicConfig(format=FORMAT)d = {'clientip': '192.168.0.1', 'user': 'fbloggs'}logging.warning("Protocol problem: %s", "connection reset", extra=d)2006-02-08 22:20:02,165 192.168.0.1 fbloggs Protocol problem: connection reset

Logging: log levelsif is_pid_cmdline_correct(pid, conffile.split('/')[-1]):try:_execute('kill', '-HUP', pid, run_as_root=True)_add_dnsmasq_accept_rules(dev)returnexcept Exception as exc:LOG.error(_LE('kill -HUP dnsmasq threw %s'), exc)else:LOG.debug('Pid %d is stale, relaunching dnsmasq', pid)LevelNumeric valueCRITICAL 50ERROR 40WARNING 30INFO 20DEBUG 10NOTSET 0

Logging: log records propagationimport loggingLOG = logging.getLogger('sqlalchemy.orm')...LOG.debug('Instance changed state from `%(prev_state)s` to `%(new_state)s`',prev_state=prev_state, new_state=new_state)sqlalchemy.orm -> sqlalchemy -> (root)

Logging: context matterscfg.StrOpt('logging_context_format_string',default='%(asctime)s.%(msecs)03d %(process)d %(levelname)s ''%(name)s [%(request_id)s %(user_identity)s] '‘%(instance)s%(message)s’)2015-06-10 12:42:00.765 27516 INFO nova.osapi_compute.wsgi.server [req-58f233abf2b6-452f-b4fe-0c781ce8f8d0None] 192.168.0.1 "GET /v2/fc7f78f1c53d4443976514d2fd16e5cb/images/detail HTTP/1.1" status: 200 len: 905 time: 0.10439712015-06-10 12:41:57.004 2760 AUDIT nova.virt.block_device[req-209db629-0d06-4f81-92ad-b910f1a72b36 None] [instance: a0d1c6ef-1fa8-46f9-a19df8fb7d2df6a2]Booting with volume 8bad9533-9d6f-4be8-939d-b7a28a536a1a at /dev/vda

Logging: log processing• logs are collected from different sources and parsed(Logstash)• then they are imported into a full-text search system(ElasticSearch)• Web UI is used for providing easy access to results andquerying (Kibana)

Logging: log processingtitle: Kernel Neighbour table overflowquery: >filename:kernel.logAND level:warningAND message:neighbour AND message:overflowtitle: Neutron Skipping router removalquery: >filename:neutron-l3-agent.logAND location:neutron.agent.l3_agentAND message:skipping AND message:removaltitle: Neutron OVS lib errors and warningsquery: >filename:neutron-openvswitch-agent.logAND location:neutron.agent.linux.ovs_libAND level:(error OR warning)title: Neutron race condition at subnet deletionquery: >filename:neutronAND level:traceAND message:AttributeError

Logging: summary• useful for both developers and operators• developers define verbosity by the means oflogging levels• configurable handlers (file, syslog, network, etc)• advanced tooling for log processing / monitoring

Logging: useful links• General info: https://docs.python.org/3.3/howto/logging.html#logging-howto• Adding contextual information: https://docs.python.org/2/howto/loggingcookbook.html#adding-contextual-information-toyour-logging-output• Logstash/ElasticSearch/Kibana: http://www.logstash.net/docs/1.4.2/tutorials/gettingstarted-with-logstash

pdb

pdb: basicsdef _binary_search(arr, left, right, key):if left == right:return -1middle = left + (right - left) / 2if key == arr[middle]:return middleelif key > arr[middle]:return _binary_search(arr, middle, right, key)else:return _binary_search(arr, left, middle, key)def binary_search(arr, key):return _binary_search(arr, 0, len(arr), key)l = list(range(10))assert binary_search(l, 5) == 5assert binary_search(l, 0) == 0assert binary_search(l, 9) == 9assert binary_search(l, 10) == -1assert binary_search(l, -5) == -1

pdb: basicsRomans-MacBook-Air:03-pdb malor$ python -m pdb basics.py> /Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py(1)()-> def _binary_search(arr, left, right, key):(Pdb) break binary_searchBreakpoint 1 at /Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py:15(Pdb) continue> /Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py(16)binary_search()-> return _binary_search(arr, 0, len(arr), key)(Pdb) argsarr = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]key = 5(Pdb) step--Call--> /Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py(1)_binary_search()-> def _binary_search(arr, left, right, key):(Pdb) next> /Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py(2)_binary_search()-> if left == right:

pdb: basics(Pdb) list1 def _binary_search(arr, left, right, key):2 -> if left == right:3 return -145 middle = left + (right - left) / 267 if key == arr[middle]:8 return middle9 elif key > arr[middle]:10 return _binary_search(arr, middle, right, key)11 else:(Pdb) where/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/bdb.py(400)run()-> exec cmd in globals, locals(1)()/Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py(20)()-> assert binary_search(l, 5) == 5/Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py(16)binary_search()-> return _binary_search(arr, 0, len(arr), key)> /Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py(2)_binary_search()-> if left == right:

pdb: post-mortem debuggingRomans-MacBook-Air:03-pdb malor$ python -m pdb basics.py> /Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py(1)()-> def _binary_search(arr, left, right, key):(Pdb) continueTraceback (most recent call last):File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pdb.py", line 1314, in mainpdb._runscript(mainpyfile)File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pdb.py", line 1233, in _runscriptself.run(statement)File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/bdb.py", line 400, in runexec cmd in globals, localsFile "", line 1, in File "basics.py", line 1, in def _binary_search(arr, left, right, key):File "basics.py", line 16, in binary_searchreturn _binary_search(arr, 0, len(arr), key)…RuntimeError: maximum recursion depth exceededUncaught exception. Entering post mortem debuggingRunning 'cont' or 'step' will restart the program-> return _binary_search(arr, middle, right, key)py.test --pdbnosetest --pdb -s. . .

pdb: commands(Pdb) breakNum Type Disp Enb Where1 breakpoint keep yes at /Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py:15breakpoint already hit 2 times(Pdb) commands 1(com) args(com) where(com) end(Pdb) continuearr = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]key = 5/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/bdb.py(400)run()-> exec cmd in globals, locals(1)()/Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py(20)()-> assert binary_search(l, 5) == 5> /Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py(16)binary_search()-> return _binary_search(arr, 0, len(arr), key)> /Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py(16)binary_search()-> return _binary_search(arr, 0, len(arr), key)

pdb: conditional break points(Pdb) break binary_searchBreakpoint 1 at /Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py:15(Pdb) breakNum Type Disp Enb Where1 breakpoint keep yes at /Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py:15(Pdb) condition 1 key == 10(Pdb) continue> /Users/malor/Dropbox/talks/kharkivpy-debugging/examples/03-pdb/basics.py(16)binary_search()-> return _binary_search(arr, 0, len(arr), key)(Pdb) argsarr = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]key = 10

pdb: summary• bread and butter of Python developers• usually the easiest and the quickest way of debugging scripts/apps• integrated with popular test runners• greenlet-friendly• requires stdin/stdout, thus not usable for debugging daemons orembedded Python code (like Gimp or Blender plugins)• not suitable for debugging of multithreaded/multiprocessingapplications• can’t attach to a running process (if not modified in advance)

winpdb

winpdb: attaching to a processrpodolyaka@rpodolyaka-pc:~/sandbox/debugging$ rpdb2 -d search.pyA password should be set to secure debugger client-server communication.Please type a password:r00tmePassword has been setrpodolyaka@rpodolyaka-pc:~$ rpdb2RPDB2 - The Remote Python Debugger, version RPDB_2_4_8,Copyright (C) 2005-2009 Nir Aides.> password r00tmePassword is set to: "r00tme"> attachConnecting to 'localhost'...Scripts to debug on 'localhost':pid name--------------------------3706 /home/rpodolyaka/sandbox/debugging/search.py> attach 3706> *** Attaching to debuggee...

winpdb: attaching to a process> bp binary_search> blList of breakpoints:Id State Line Filename-Scope-Condition-Encoding------------------------------------------------------------------------------0 enabled 15 /home/rpodolyaka/sandbox/debugging/search.pybinary_search> go> *** Debuggee is waiting at break point for further commands.> stackStack trace for thread 140416296978176:Frame File NameLine Function------------------------------------------------------------------------------> 0 ...ndbox/debugging/search.py 15 1 ....7/dist-packages/rpdb2.py 14220 StartServer2 ....7/dist-packages/rpdb2.py 14470 main3 /usr/bin/rpdb2 31

winpdb: embedded debuggingdef add_lease(mac, ip_address):"""Set the IP that was assigned by the DHCP server."""import rpdb2; rpdb2.start_embedded_debugger('r00tme')api = network_rpcapi.NetworkAPI()api.lease_fixed_ip(context.get_admin_context(), ip_address, CONF.host)dnsmasq daemon forks and executes this like:nova-dhcpbridge add AA:BB:CC:DD:EE:FF 10.0.0.2

winpdb: debugging of threadsdef allocate_ips(engine, host):while True:with engine.begin() as conn:result = conn.execute(ip_addresses.select() \.where(ip_addresses.c.host.is_(None))).first()if result is None:# no IPs leftbreakid, address = result.id, result.addressrows = conn.execute(ip_addresses.update() \.values(host=host) \.where(ip_addresses.c.id == id) \.where(ip_addresses.c.address == address) \.where(ip_addresses.c.host.is_(None)))if not rows:# concurrent updatecontinue

winpdb: debugging of threadst1 = threading.Thread(target=allocate_ips, args=(eng, 'host1'))t1.start()t2 = threading.Thread(target=allocate_ips, args=(eng, 'host2'))t2.start()t1.join()t2.join()> attach $PID…> threadList of active threads known to the debugger:No Tid Name State-----------------------------------------------0 140456866166528 MainThread waiting at break point> 1 140456389068544 Thread-1 waiting at break point2 140456380675840 Thread-2 waiting at break point

winpdb: debugging of threads> thread 2Focus was set to chosen thread.> stackStack trace for thread 140456380675840:Frame File NameLine Function------------------------------------------------------------------------------> 0 /home/rpodolyaka/sa.py 30 allocate_ips1 ...ib/python2.7/threading.py 763 run> go> break> *** Debuggee is waiting at break point for further commands.> stackStack trace for thread 140456380675840:Frame File NameLine Function------------------------------------------------------------------------------> 0 ...alchemy/engine/default.py 409 do_commit1 ...sqlalchemy/engine/base.py 525 _commit_impl2 ...sqlalchemy/engine/base.py 1364 _do_commit

winpdb: summary• allows to debug multithreaded Pythonapplications• remote debugging (which effectively means, nostdout/stdint limitations as with pdb)• wxWidgets-based GUI• to attach to a running process you need tomodified it in advance (embedded debugging) orstart it with rpdb2

cProfile

cProfile: basicsdef count_freq(stream):res = {}for i in iter(lambda: stream.read(1), ''):try:res[i] += 1except KeyError:res[i] = 1return resdef build_tree(stream):queue = [Node(freq=v, symb=k) for k, v in count_freq(stream).items()]while len(queue) > 1:queue.sort(key=lambda k: k.freq)first = queue.pop(0)second = queue.pop(0)queue.append(Node(freq=(first.freq + second.freq), left=first, right=second))return queue[0]

cProfile: Amdahl's law

cProfile: basicsRomans-MacBook-Air:07-cprofile malor$ python -m cProfile -s cumtime huffman.py ~/Downloads/kharkivpy-debugging.key24868775 function calls in 14.059 secondsOrdered by: cumulative timencalls tottime percall cumtime percall filename:lineno(function)1 0.008 0.008 14.059 14.059 huffman.py:1()1 0.001 0.001 14.051 14.051 huffman.py:33(build_tree)1 5.029 5.029 14.035 14.035 huffman.py:23(count_freq)12417038 3.863 0.000 9.006 0.000 huffman.py:25()12417038 5.143 0.000 5.143 0.000 {method 'read' of 'file' objects}255 0.009 0.000 0.014 0.000 {method 'sort' of 'list' objects}32895 0.005 0.000 0.005 0.000 huffman.py:36()511 0.001 0.000 0.001 0.000 huffman.py:7(__init__)510 0.000 0.000 0.000 0.000 {method 'pop' of 'list' objects}1 0.000 0.000 0.000 0.000 functools.py:53(total_ordering)1 0.000 0.000 0.000 0.000 {open}256 0.000 0.000 0.000 0.000 {len}255 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}1 0.000 0.000 0.000 0.000 {dir}1 0.000 0.000 0.000 0.000 {method 'items' of 'dict' objects}3 0.000 0.000 0.000 0.000 {setattr}3 0.000 0.000 0.000 0.000 {getattr}1 0.000 0.000 0.000 0.000 {max}

cProfile: visualisation

cProfile: context mattersimport cProfile as profilerimport gc, pstats, timedef profile(fn):def wrapper(*args, **kw):elapsed, stat_loader, result = _profile(“out.prof”, fn, *args, **kw)stats = stat_loader()stats.sort_stats('cumulative')stats.print_stats()return resultreturn wrapperdef _profile(filename, fn, *args, **kw):load_stats = lambda: pstats.Stats(filename)gc.collect()began = time.time()profiler.runctx('result = fn(*args, **kw)', globals(), locals(),filename=filename)ended = time.time()return ended - began, load_stats, locals()['result']

cProfile: context mattersfrom werkzeug.contrib.profiler import ProfilerMiddlewareapp = ProfilerMiddleware(app)

cProfile: context mattersPATH: '/6e0f43cd74db46f5b95f2142fe0c9431/flavors/detail'2732 function calls (2602 primitive calls) in 1.294 secondsOrdered by: cumulative timencalls tottime percall cumtime percall filename:lineno(function)1 0.000 0.000 1.287 1.287 /usr/lib/python2.7/dist-packages/nova/api/compute_req_id.py:38(__call__)2/1 0.008 0.004 1.287 1.287 /usr/lib/python2.7/dist-packages/webob/request.py:1300(send)2/1 0.000 0.000 1.287 1.287 /usr/lib/python2.7/dist-packages/webob/request.py:1262(call_application)1 0.000 0.000 1.287 1.287 /usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py:121(__call__)1 0.000 0.000 1.271 1.271 /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:686(__call__)1 0.000 0.000 1.270 1.270 /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:829(_validate_token)1 0.000 0.000 1.270 1.270 /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:1669(get)1 0.000 0.000 1.270 1.270 /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:1726(_cache_get)

cProfile: summary• easy CPU profiling of Python code with lowoverhead• text/binary representation of profiling results (thelatter can be used for merging results and/orvisualisation done by external tools)• can’t attach to a running process• can’t profile Python interpreter-level code(Py_EvaluateFrameEx, etc)

objgraph

objgraph: basicsIn [1]: import objgraphIn [2]: objgraph.show_most_common_types()function 4530dict 2483tuple 1428wrapper_descriptor 1260weakref 981list 911builtin_function_or_method 897method_descriptor 705getset_descriptor 531type 473

objgraph: basicsIn [3]: objgraph.show_growth()function 4530 +4530dict 2412 +2412tuple 1353 +1353wrapper_descriptor 1272 +1272weakref 985 +985list 904 +904builtin_function_or_method 897 +897method_descriptor 706 +706getset_descriptor 535 +535type 473 +473In [4]: objgraph.show_growth()weakref 986 +1list 905 +1tuple 1354 +1

objgraph: graphs>>> x = []>>> y = [x, [x], {‘x’: x}]>>> objgraph.show_refs([y], filename='sample-graph.png')

strace

strace: tracing syscallsrpodolyaka@rpodolyaka-pc:~$ strace -e network python sa.py. . .socket(PF_INET6, SOCK_STREAM, IPPROTO_IP) = 5setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0setsockopt(5, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0connect(5, {sa_family=AF_INET6, sin6_port=htons(5432), inet_pton(AF_INET6, "::1",&sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EINPROGRESS (Operation nowin progress)getsockopt(5, SOL_SOCKET, SO_ERROR, [0], [4]) = 0getsockname(5, {sa_family=AF_INET6, sin6_port=htons(36894), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0sendto(5, "\0\0\0\10\4\322\26/", 8, MSG_NOSIGNAL, NULL, 0) = 8recvfrom(5, "S", 16384, 0, NULL, NULL) = 1. . .

strace: tracing syscallsroot@node-13:~# strace -p 1508 -s 4096 -tt. . .16:53:29.532770 epoll_wait(7, {}, 1023, 0) = 016:53:29.532832 epoll_wait(7, {}, 1023, 0) = 016:53:29.532892 epoll_wait(7, {}, 1023, 0) = 016:53:29.532953 epoll_wait(7, {}, 1023, 0) = 016:53:29.533022 epoll_wait(7, {{EPOLLIN, {u32=9, u64=39432335262744585}}}, 1023,915) = 116:53:29.596409 epoll_ctl(7, EPOLL_CTL_DEL, 9, {EPOLLRDNORM|EPOLLWRBAND|EPOLLMSG|0x28c45820, {u32=32644, u64=22396489217113988}}) = 016:53:29.596494 accept(9, 0x7ffe1ef32b10, [16]) = -1 EAGAIN (Resource temporarilyunavailable)16:53:29.596638 epoll_ctl(7, EPOLL_CTL_ADD, 9, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP,{u32=9, u64=39432335262744585}}) = 016:53:29.596747 epoll_wait(7, {{EPOLLIN, {u32=9, u64=39432335262744585}}}, 1023,851) = 116:53:29.611852 epoll_ctl(7, EPOLL_CTL_DEL, 9, {EPOLLRDNORM|EPOLLWRBAND|EPOLLMSG|0x28c45820, {u32=32644, u64=22396489217113988}}) = 016:53:29.611937 accept(9, 0x7ffe1ef32b10, [16]) = -1 EAGAIN (Resource temporarilyunavailable). . .

strace: summary• allows tracing of applications interactions with `outsideworld`• points possible problems with performance (likeexcessive system calls, polling of events with toosmall timeout, etc)• limited to tracing of system calls of one process andits forks• use cautiously on production environments as itgreatly affects performance

gdb

gdb: prerequisites• Ubuntu/Debian:• sudo apt-get install gdb python-dbg• CentOS/RHEL/Fedora (separate debuginfopackage repository):• sudo yum install gdb python-debuginfo

gdb: basics• python-dbg is a CPython binary built with‘--with-debug -g’ options. It’s slow and verboseabout memory management• you can debug regular CPython processes inproduction using the debug symbols shipped separately• gdb has Python bindings to write scripts for it• CPython is shipped with a gdb script allowing toanalyse interpreter-level stack frames to get app-levelbacktraces

gdb: `hanging` appdef allocate_ips(eng, host):while True:with eng.begin() as conn:row = conn.execute(ip_addresses.select() \.where(ip_addresses.c.host.is_(None))).fetchone()if row is None:breakid, address = row.id, row.addressupdated_rows = conn.execute(ip_addresses.update() \.values(host=host) \.where(ip_addresses.c.id == id) \.where(ip_addresses.c.host.is_(None)))if not updated_rows:continuet = threading.Thread(target=allocate_ips, args=(eng, 'host1'))t.start()t.join()

gdb: `hanging` apprpodolyaka@rpodolyaka-pc:~$ strace -p 20267Process 20267 attachedfutex(0x7fea50000c10, FUTEX_WAIT_PRIVATE, 0, NULLrpodolyaka@rpodolyaka-pc:~$ gdb /usr/bin/python3.4 -p 20216(gdb) t a a frameThread 2 (Thread 0x7f7702c83700 (LWP 20353)):#0 sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101101 ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S: No such file ordirectory.Thread 1 (Thread 0x7f770a03b700 (LWP 20350)):#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:8585 ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

gdb: `hanging` app(gdb) t a 2 py-btThread 2 (Thread 0x7f7702c83700 (LWP 20353)):Traceback (most recent call first):File "/usr/lib/python3.4/threading.py", line 294, in waitgotit = waiter.acquire(True, timeout)File "/home/rpodolyaka/venv3/lib/python3.4/site-packages/sqlalchemy/util/queue.py", line 157, in getself.not_empty.wait(remaining)File "/home/rpodolyaka/venv3/lib/python3.4/site-packages/sqlalchemy/pool.py", line1039, in _do_getreturn self._pool.get(wait, self._timeout)File "/home/rpodolyaka/venv3/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2037, in contextual_connectself._wrap_pool_connect(self.pool.connect, None),File "/home/rpodolyaka/venv3/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1906, in beginconn = self.contextual_connect(close_with_result=close_with_result)File "sa.py", line 31, in allocate_ipswith eng.begin() as conn:File "/usr/lib/python3.4/threading.py", line 868, in runself._target(*self._args, **self._kwargs)File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_innerself.run()File "/usr/lib/python3.4/threading.py", line 888, in _bootstrapself._bootstrap_inner()

gdb: virtualenv pitfallsrpodolyaka@rpodolyaka-pc:~$ gdb -p 20656# WARN: executable not passed!(gdb) py-btUndefined command: "py-bt". Try "help".(gdb) bt#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85#1 0x00000000004cdff5 in PyThread_acquire_lock_timed ()#2 0x0000000000522039 in ?? ()#3 0x00000000004ee01a in PyEval_EvalFrameEx ()#4 0x00000000004ec9fc in PyEval_EvalCodeEx ()#5 0x00000000004f25a9 in PyEval_EvalFrameEx ()#6 0x00000000004ec9fc in PyEval_EvalCodeEx ()#7 0x00000000004f25a9 in PyEval_EvalFrameEx ()#8 0x00000000004ec9fc in PyEval_EvalCodeEx ()#9 0x0000000000581115 in ?? ()#10 0x00000000005ab019 in PyRun_FileExFlags ()#11 0x00000000005aa194 in PyRun_SimpleFileExFlags ()#12 0x00000000004cb4cb in Py_Main ()#13 0x00000000004ca8ef in main ()

gdb: summary• allows to debug multithreaded applications• allows to attach to a running process at any given moment of time• can be used for analysing of core dumps (e.g. if we don’t want tostop a process, or if it died unexpectedly)• can be used for debugging of C-extensions, CFFI calls, etc• success depends on how CPython was built and whether youhave installed debug symbols or not• used by pyringe to provide pdb-like experience (https://github.com/google/pyringe)

htop

htop

lsof

lsof: lsof -p $PIDnova-api 5910 nova mem REG 252,0 141574 3586 /lib/x86_64-linuxgnu/libpthread-2.19.sonova-api 5910 nova mem REG 252,0 149120 3582 /lib/x86_64-linuxgnu/ld-2.19.sonova-api 5910 nova mem REG 252,0 26258 52555 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cachenova-api 5910 nova 0u CHR 1,3 0t0 1029 /dev/nullnova-api 5910 nova 1u CHR 136,13 0t0 16 /dev/pts/13nova-api 5910 nova 2u CHR 136,13 0t0 16 /dev/pts/13nova-api 5910 nova 3w REG 252,0 34967268 135756 /var/log/nova/nova-api.lognova-api 5910 nova 4u unix 0xffff880850b92a00 0t0 260406 socketnova-api 5910 nova 5r FIFO 0,8 0t0 260407 pipenova-api 5910 nova 6w FIFO 0,8 0t0 260407 pipenova-api 5910 nova 7u IPv4 260408 0t0 TCPnode-13.domain.tld:8773 (LISTEN)nova-api 5910 nova 8r CHR 1,9 0t0 1034 /dev/urandomnova-api 5910 nova 9u IPv4 260409 0t0 TCPnode-13.domain.tld:8774 (LISTEN)nova-api 5910 nova 10u IPv4 260420 0t0 TCP *:8775 (LISTEN)nova-api 5910 nova 15u 0000 0,9 0 7380 anon_inode

netstat

netstat: netstat -nlaptcp 8 0 192.168.0.16:52819 192.168.0.11:5673 ESTABLISHED5975/pythontcp 0 0 192.168.0.16:36901 192.168.0.11:5673 ESTABLISHED1513/pythontcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN3042/sshdtcp 0 0 0.0.0.0:4567 0.0.0.0:* LISTEN13888/mysqldtcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN7433/mastertcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN19704/tgtdtcp 0 0 192.168.0.16:35357 0.0.0.0:* LISTEN5546/python

perf_events

perf_events: perf top

perf_events: perf trace254.663 ( 0.001 ms): sshd/22802 clock_gettime(which_clock: 7, tp: 0x7ffd0e807970) = 0254.666 ( 0.003 ms): sshd/22802 read(fd: 14, buf: 0x7ffd0e8038b0,count: 16384 ) = 4095254.672 ( 0.243 ms): chrome/11973 epoll_wait(epfd: 16, events: 0x6a6a1b73480,maxevents: 32, timeout: 4294967295) = 1254.678 ( 0.003 ms): chrome/11973 read(fd: 24, buf:0x6a6a2d5b018, count: 4096 ) = 32254.685 ( 0.003 ms): chrome/11973 write(fd: 11, buf:0x7f940dfa55e7, count: 1 ) = 1254.688 ( 0.001 ms): chrome/11973 read(fd: 24, buf:0x6a6a2d5b018, count: 4096) = -1 EAGAIN Resource temporarily unavailable254.691 ( 0.001 ms): chrome/11973 epoll_wait(epfd: 16, events: 0x6a6a1b73480,maxevents: 32 ) = 0254.693 ( 0.001 ms): chrome/11973 epoll_wait(epfd: 16, events: 0x6a6a1b73480,maxevents: 32 ) = 0

perf_events: perf statPerformance counter stats for 'python sa.py':125.242831 task-clock (msec) # 0.004 CPUs utilized945 context-switches # 0.008 M/sec14 cpu-migrations # 0.112 K/sec6,996 page-faults # 0.056 M/sec408,133,256 cycles # 3.259 GHz213,117,410 stalled-cycles-frontend # 52.22% frontend cycles idle stalled-cycles-backend432,245,331 instructions # 1.06 insns per cycle# 0.49 stalled cycles per insn91,417,607 branches # 729.923 M/sec3,937,108 branch-misses # 4.31% of all branches30.130596204 seconds time elapsed

Questions?slides: http://bit.ly/1LpjXGLtwitter: @amd4ever