color version - PET: Python Entre Todos - Python Argentina

If black is used space, and white is free space, you can see here how there’s a lot of
free space, but unusable for objects beyond some size, because it’s not contiguous
space.
Put simply, memory fragmentation happens when there’s lots of free space, but it’s not
contiguous. Like in the above map, much of it is free space, but it’s unusable for big
objects because, in contrast with a file you can split into several blocks on disk, an
object’s memory region needs to be contiguous.
So, in contrast with fragmentation in a file system, memory fragmentation makes
portions of it unusable. If I needed memory for a big object, say a few megabytes, I would
have to use the area towards the end of the map (that is, extend the virtual image of the
process). This is effectively what malloc does when it’s faced with this situation.
The immediate, visible effect here, is an inefficiency in the use of available memory. If my
program needed 2GB of memory in theory, it could be reserving 4GB from the operating
system (because it has many small pieces reserved that it cannot use). If I have bad luck,
this could make my system swap. If I have too much bad luck, it could trash, and die.
Lets see code that fragments memory:
>>> l = []
>>> for i in xrange(1,100):
... ll = [ " " * i * 16 for j in xrange(1000000 / i) ]
... ll = ll[::2]
... l.extend(ll)
... print sum(map(len,l))
...
8000000
16000000
…
792005616
>>>
But there’s something worse. Suppose I do:
>>> del l
>>> del ll
I get from top:
10467 claudiof 20 0 1532m 1.5g 1864 S 0 75.6 1:17.96 python
If I repeat the fragmentation example, I can confirm that those 1.5G are effectively free
for python:
10467 claudiof 20 0 1676m 1.6g 1864 S 0 82.8 2:33.39 python
But if I try to free them (to the operating system), I can’t.
¿WTF
Enter Guppy
Guppy is a little red fish commonly seen in fish tanks everywhere. Those little fish there,
those are called guppy.
Really.
It’s also an extension library for Python that contains a module, heapy, which allows me
to do memory diagnostics.
Really.
After this, top says:
10467 claudiof 20 0 1676m 1.6g 1864 S 0 82.7 1:17.07 python
Guppy
Meaning, even though according to our calculations the program had to consume 800M
of memory, it effectively consumes 1.6G. Double that.
Why
Well the example was specifically tailored to create 50% of unusable holes. The memory
is fragmented, then, by 50%.
So, let’s try to use it:
>>> from guppy import hpy
>>> hp = hpy()
>>> hp.heap()