Building Machine Learning Systems with Python - Richert, Coelho

Chapter 12
A task could be "call double with argument 3". Another task would be "call double
with argument 642.34". Using jugs, we can build these tasks as follows:
from jug import Task
t1 = Task(double, 3)
t2 = Task(double, 642.34)
Save this to a file called jugfile.py (which is just a regular Python file). Now, we
can run jug execute to run the tasks. This is something you run on the command
line, not at the Python prompt! Instead of Python's jugfile.py file (which should
do nothing), you run jug execute.
You will also get some feedback on the tasks (jug will say that two tasks named
"double" were run). Run jug execute again and it will tell you that it did nothing!
It does not need to. In this case, we gained little, but if the tasks took a long time to
compute, it would be very useful.
You may notice that a new directory named jugfile.jugdata also appeared on
your hard drive, with a few other weirdly named files. This is the memoization
cache. If you remove it, jug execute will run all your tasks (both of them) again.
Often it is good to distinguish between pure functions, which simply
take their inputs and return a result, from more general functions
that can perform actions such as reading from files, writing to files,
accessing global variables, modifying their arguments, or anything that
the language allows. Some programming languages, such as Haskell,
even have syntactic ways in which to distinguish pure from impure
functions.
With jug, your tasks do not need to be perfectly pure. It is even
recommended that you use tasks to read your data or write your
results. However, accessing and modifying global variables will not
work well; the tasks may be run in any order in different processors.
The exceptions are global constants, but even this may confuse the
memoization system (if the value is changed between runs). Similarly,
you should not modify the input values. Jug has a debug mode (use
jug execute—debug), which slows down your computation, but
would give you useful error messages if you make this sort of mistake.
The preceding code works but it is a bit cumbersome; you are always repeating the
Task(function, argument) construct. Using a bit of Python magic, we can make
the code even more natural:
from jug import TaskGenerator
from time import sleep
@TaskGenerator
def double(x):
[ 243 ]