color version - PET: Python Entre Todos - Python Argentina

Memory debugging and defragmentation
Author: Claudio Freire
How would you say Python manages memory Malloc (a C function that reserves
memory for those unfamiliar with C)
Well, no. Python has unusual and very specific requirements, if it used malloc for all of
its memory needs, it would be too inefficient. Python uses, instead, a series of techniques
and strategies designed to minimize malloc calls, which are very slow to be used in the
core of Python’s virtual machine.
• Pools
Introduction
This lecture was planned to be given in 20 minutes. If you can’t read it in 20 minutes,
reread, reread and re-reread until it takes 20 minutes :-p
Well, actually the text here is only slightly based in the real lecture. I’m one known to
improvise in lectures, all I have are the slides as a guide, the general guidelines for the
lecture, and this format doesn’t lend itself to improvisation. Besides, no freaking way I’m
going to remember the exact words I used ;)
Now, really… this is a deep convoluted subject. Reread and re-reread until you get it.
When I started looking for lecture subjects, I decided for memory fragmentation because
not long ago I though it was a myth. Ok, yes, it can happen… but, does that happen to
anybody Is it a problem How naive of me to doubt that could answers could be “yes”.
In PyConAr 2010 (and here in this space now), not only I had the chance to transmit
around 4 months of picking Python’s guts apart, trying to find out what was going on
with my applications (where I suffered from memory fragmentation), I also learned that
this issue is much, much more common than I’d expected.
When the lecture started, I asked: Does anybody know what memory fragmentation is.
Exactly zero attendants raised their hand. Exactly the amount I had expected. But, in the
end, many approached me saying that the same thing that was happening to me (and
that I described, and will describe below) was happening to them.
• Integer
• Floating point
• Free lists
• Tuples
• Lists
• Frames (yes, the frames are objects and need to be managed too)
• Arenas
Lets see what each is about, one by one
Pools
Not for swimming.
They’re arrays, object vectors of the same type, that Python uses usually to accelerate
the creation and destruction of common objects, like integers and floating point
numbers.
When python needs one of these objects, it doesn’t create one, it creates many (as many
as they fit in a pool block), and keeps a linked list of which objects are free within each
block.
That, according to my post-PyCon estimations, means a 10% of everyone present
(assuming they’re Python developers) suffer from that issue, and to a 3% of them it
means serious operative problems. So the issue was far more relevant than I had
expected.
How does Python manage memory
Before going into this fragmentation thing, we must study some internal details of how
memory management works in Python.