Building Machine Learning Systems with Python - Richert, Coelho

Big(ger) Data
We can use the same jugfile.py system as before, except that now, instead of
running it directly on the master, we schedule it on the cluster. First, write a very
simple wrapper script:
#!/usr/bin/env bash
jug execute jugfile.py
Call it using run-jugfile.sh and use chmod +x run-jugfile.sh to give it an
executable permission:
For c in 'seq 16'; do qsub run-jugfile.sh; done
This will create 16 jobs, each of which will run the run-jugfile.sh script, which
will simply call jug. You can still use the master as you wish. In particular, you can at
any moment run jug status and see the status of the computation. In fact, jug was
developed in exactly such an environment, so it works very well in it.
Eventually, the computation will be finished and we can kill off all the nodes. Be sure
to save the desired results somewhere and run the following:
starcluster terminate mycluster
Note that terminating will really destroy the filesystem and all your results. Of
course, it is possible to change this default. You can have the cluster write to a
filesystem which is not allocated and destroyed by starcluster but is available to you
on a regular instance; in fact the flexibility of these tools is immense. However, these
advanced manipulations could not all fit in this chapter.
Starcluster has excellent documentation online at http://star.mit.edu/cluster/,
which you should read for more information about all the possibilities of this tool.
We have seen only a small fraction of the functionality and used only the default
settings here.
[ 258 ]