• YouTube

Summary

UNIX core-dumps, Windows minidumps or Java heap-dumps are well established technologies for post-mortem defect analysis. I'll present a similar technology for Python. An improved pickling mechanism makes it possible to serialise the state of a Python program for subsequent analysis with a conventional Python-debugger.

Description

UNIX core-dumps, Windows minidumps and analogous solutions of other operating systems are well established technologies for post-mortem defect analysis of native-code processes. In principle those dumps can be used to analyse „interpreted“ programs running within a native-code interpreter-process. However in practise this approach is tedious and not always successful [1]. Therefore operating system independent dump methods were developed for some „interpreted“ languages [2]. A prominent example are Java heap dumps [3].

Unfortunately up to now there was no practically usable dump-method for Python. Various attempts were made to utilise OS-level dump methods [4, 5]. In 2012 Eli Finer published the Python module pydump [6]. This module pickles the traceback of an exception and subsequently uses the pdb debugger to analyse the unpickled traceback. Unfortunately pydump fails on PicklingErrors.

In my talk I'll present the Python package *pyheapdump*. It has the same operation principle as Eli's pydump, but is an independent implementation. pyheapdump uses an extended pickler (sPickle) to serialise all relevant objects of a Python process to a file. Later on a fault tolerant unpickler recreates the objects and a common Python debugger can be used to analyse the dump. The pickler extensions make it possible to:

• pickle and unpickle many commonly not pickleable objects [7].
• replace the remaining not pickleable objects by surrogate objects so that the resulting object graph is almost isomorphic to the original object graph.

Which objects are relevant? In its default operation mode pyheapdump uses the frame-stacks of all threads as start point for pickling. Following the usual rules for pickling the dump includes all local variables and all objects reachable from a local variable and so on. That is usually enough for a successful defect analysis.

Compared with other Python post-mortem debugging methods pyheapdump has several advantages:

• It is a pure Python solution and independent from the operation system.
• Creation of the pyheapdump and fault analysis can be performed different computers.
• It is not obstructive. It does not modify / monkey-patch or disturb the dumped process in any way, with the exception of loading additional modules.
• If used with the Pydev-debugger, it supports multi-threaded applications.
• If used with the Pydev-debugger and Stackless Python, it supports tasklets.

The implementation of pyheapdump is fairly small, because it draws most of its functionality from the underlying sPickle package and from the new Stackless-Support [8] of the Pydev-Debugger. Therefore it is - despite of its short history - already a useful piece of software.