Linux Journal | May 2012 | Issue 217 - ACM Digital Library

Lua | App Inventor | LTSP | Pure Data | C and PythonHIGHLY AVAILABLEiSCSI STORAGEWITH DRBD ANDPACEMAKERSince 1994: The Original Magazine of the Linux CommunityUSE DESIGNFRAMEWORKS TOIMPROVE YOUR SITEMAY 2012 | ISSUE 217 | www.linuxjournal.comPROGRAMMINGLUA FOR OBJECT-ORIENTED PROGRAMMINGHOW TO: APP INVENTOR FOR ANDROIDINTRO TO PARALLEL PROGRAMMING WITH C AND PYTHONPD—THE MODERN AND FLEXIBLE LANGUAGE FOR AUDIOFREE TOSUBSCRIBERSEPUB, Kindle, Android, iPhone & iPad editionsTRACK DOWNBandwidth-HoggingConnections with iftopSCALINGLTSP in LargeEnvironmentsREVIEWED:ZaReason’sValta X79

Learn Drupal & jQueryFROM THE COMFORT OFYOUR LIVING ROOMThe Lullabot Learning Series includes everything you need to become aDrupal & jQuery expert from the comfort of your living room! The videosare available in both DVD format and high-definition video download.Purchase the videos at http://store.lullabot.com

Executive EditorSenior EditorAssociate EditorArt DirectorProducts EditorEditor EmeritusTechnical EditorSenior ColumnistSecurity EditorHack EditorVirtual EditorJill Franklinjill@linuxjournal.comDoc Searlsdoc@linuxjournal.comShawn Powersshawn@linuxjournal.comGarrick Antikajiangarrick@linuxjournal.comJames Graynewproducts@linuxjournal.comDon Martidmarti@linuxjournal.comMichael Baxtermab@cruzio.comReuven Lernerreuven@lerner.co.ilMick Bauermick@visi.comKyle Rankinlj@greenfly.netBill Childersbill.childers@linuxjournal.comContributing EditorsIbrahim Haddad • Robert Love • Zack Brown • Dave Phillips • Marco Fioretti • Ludovic MarcottePaul Barry • Paul McKenney • Dave Taylor • Dirk Elmendorf • Justin RyanProofreaderGeri GalePublisherAdvertising Sales ManagerAssociate PublisherWebmistressAccountantCarlie Fairchildpublisher@linuxjournal.comRebecca Cassityrebecca@linuxjournal.comMark Irgangmark@linuxjournal.comKatherine Druckmanwebmistress@linuxjournal.comCandy Beauchampacct@linuxjournal.comLinux Journal is published by, and is a registered trade name of,Belltown Media, Inc.PO Box 980985, Houston, TX 77098 USAEditorial Advisory PanelBrad Abram Baillio • Nick Baronian • Hari Boukis • Steve CaseKalyana Krishna Chadalavada • Brian Conner • Caleb S. Cullen • Keir DavisMichael Eager • Nick Faltys • Dennis Franklin Frey • Alicia GibbVictor Gregorio • Philip Jacob • Jay Kruizenga • David A. LaneSteve Marquez • Dave McAllister • Carson McDonald • Craig OdaJeffrey D. Parent • Charnell Pugsley • Thomas Quinlan • Mike RobertsKristin Shoemaker • Chris D. Stark • Patrick Swartz • James WalkerAdvertisingE-MAIL: ads@linuxjournal.comURL: www.linuxjournal.com/advertisingPHONE: +1 713-344-1956 ext. 2SubscriptionsE-MAIL: subs@linuxjournal.comURL: www.linuxjournal.com/subscribeMAIL: PO Box 16476, North Hollywood, CA 91615-9911 USALINUX is a registered trademark of Linus Torvalds.

iXsystems Servers + Intel® Xeon®Processor E5-2600 Family =Unparalleled performance densityE5-2600iXsystems is pleased to present a range of new, blazinglyfast servers based on the Intel® Xeon® Processor E5-2600family and the Intel® C600 series chipset.The Intel® Xeon® Processor E5-2600 Family employs a new microarchitecture toboost performance by up to 80% over previous-generation processors. Theperformance boost is the result of a combination of technologies, including Intel®Integrated I/O, Intel® Data Direct I/O Technology, and Intel® Turbo Boost Technology.The iXR-1204+10G features Dual Intel® Xeon® E5-2600 Family Processors, andpacks up to 16 processing cores, 768GB of RAM, and dual onboard 10GigE NICsin a single unit of rack space. The robust feature set of the iXR-1204+10G makes itsuitable for clustering, high-traffic webservers, virtualization, and cloud computingapplications.For computation and throughput-intensive applications, iXsystems now offersthe iXR-22x4IB. The iXR-22x4IB features four nodes in 2U of rack space, each withdual Intel® Xeon® E5-2600 Family Processors, up to 256GB of RAM, and a Mellanox®ConnectX QDR 40Gbp/s Infiniband w/QSFP Connector. The iXR-22x4IB is perfect forhigh-powered computing, virtualization, or business intelligence applications thatrequire the computing power of the Intel® Xeon® Processor E5-2600 family and thehigh throughput of Infiniband.IXR-1204+10G-10GbE On-BoardiXR-1204+10G• Dual Intel® Xeon® E5-2600 FamilyProcessors• Intel® X540 Dual-Port 10 GigabitEthernet Controllers• Up to 16 Cores and 32 process threads• Up to 768GB Main Memory• 700W Redundant high-efficiencypower supplyiXR-22x4IB• Dual Intel® Xeon® E5-2600 FamilyProcessors per node• Mellanox® ConnectX QDR 40Gbp/sInfiniband w/QSFP Connector per node• Four server nodes in 2U of rack space• Up to 256GB Main Memory per servernode• Shared 1620W Redundant highefficiencyPlatinum level (91%+)power supplyIXR-22x4IBIntel, the Intel logo, and Xeon Inside are trademarks or registered trademarks of Intel Corporation in the U.S. and other countries.Call iXsystems toll free or visit our website today! 1-855-GREP-4-IX | www.iXsystems.com

Current_Issue.tar.gzRubies,Pythons and Perls!SHAWN POWERSIt may sound like a new Indiana Jonesmovie or possibly a cheesy platformstylevideo game from the 1990s, butthe title of this column actually refers toour focus this month—programming! Notthat there’s anything wrong with daringadventures in remote locations, it’s justthat all the red tape can be overwhelming.You know that somewhere there wasa college intern filing environmentalimpact study reports, negotiating workvisas for multi-country searches andadding venomous predator riders on theinsurance policies. Dr Jones just grabbedhis hat and found the treasure!Here in the real world, we prefer tocreate our own treasures. That’s wherethis issue comes into play. ReuvenM. Lerner is actually a bit of a rebelthis month, and in an ironic twist, hefocuses on Web design frameworks. Theworlds of Web design and programmingcertainly overlap, so we’ll give him apass. Dave Taylor wraps up his WordsWith Friends series by demonstratinghow to calculate word point values.Dave’s articles always are fun for me,because scripting is the only sort ofprogramming I ever do (usually out ofsysadmin need), and watching him createstuff that I understand is fun.Kyle and I are system administratorsby trade, so it’s not a surprise thatour columns this month show it. Kyledescribes how to use iftop, which islike the monitoring tool top, except forbandwidth usage. If you are struggling tofind where all your bandwidth is going,iftop can be invaluable, and in his article,you’ll learn how to use it. For my column,it’s quite clear where the bandwidth isgoing—to the thin clients! This month,I finish my three-part series on LTSP anddiscuss how to scale your thin-clientenvironment. As with most complexproblems, there isn’t a single solution toLTSP scaling. I talk about a handful ofmethods and help you choose the bestoption for your environment.I’m sure you’re thinking, “I thoughtthis was the programming issue!” Restassured, it is. Alejandro Segovia exploresLua, a multiparadigm programminglanguage. No, he doesn’t take us to aLuau, with grass skirts and coconuts, butin his words, he “presents a reusable8 / MAY 2012 / WWW.LINUXJOURNAL.COM

lettersNotRenewingMysubscriptionwas a gift,and is inthe nameof my thenfuturewife,sometime in1996. Withthe demiseof the paper edition, you have lost acustomer. Electronic “search” capabilitydoes not compensate for “flicking throughand finding something interesting”. I canuse Google to seek something in which Iknow I am interested. Goodbye.—Ian StirlingIan, sorry to see you go. We wish youthe best!—Ed.Digital SubscriptionLinux Journal was the only Linuxmagazine I could find that didn’t insultmy intelligence by putting a CD (withsoftware I could download for free) inthe magazine and tripling the price. Andalthough this may have contributed tothe exit of the print version, I thought itshowed a great respect for your readers.So, although I’m not crazy about thedigital version, I’ll adapt/evolve and enjoyit for many years to come because ofyour mentality to offer a high-qualityproduct at a fair price.—R. Scott SandersThe adjustment is frustrating at times,I agree. I’m excited, however, when Idream about the future possibilities withdigital. My ideas for flip-book animationswere about as crazy as we could get withpaper, but the digital format is new. Whoknows what the future might hold!—Ed.LJ Digital FormatI have no plans to buy a Kindle or anyother tablet-type gizmo. That is moneybetter spent on oscilloscopes and stufflike that. I was struggling with the PDFformat on my Kubuntu Lucid laptop,when I discovered the EPUB format andCalibre. Now we are cooking. I preferpaper, but I want LJ to stay around agood long time, so I bought the CD tohelp support you. I do miss reading it inthe bathroom though.—MikeFor quick bathroom reading, smartphonesaren’t too bad either. I actually like toread the EPUB version using EPUBReaderon Firefox. We’re glad to have you as asubscriber, even if you’re not interestedin tablets!—Ed.10 / MAY 2012 / WWW.LINUXJOURNAL.COM

[ LETTERS ]Now I’m a BelieverI couldn’t figure out why I wasunable to find Linux Journal on themagazine racks at my local bookstore.I even made a formal complaint withthe manager. So when I found outyou had made a permanent transitionto digital format only, I wasembarrassed. Intent on keeping thismagazine and its articles flowing, Iwent out and bought a NOOK (I hadbeen toying with the idea anywaydue to many interesting hacks I hadread about).My one concern is regarding longevity.Eventually this device, the cloud servicerunning it, or the corporation backingit, will fall. What then will I do whenI want to read something printed byyour magazine a few years before? Doyou at Linux Journal not worry aboutthe direction that the cloud and digitaldistributionmentality is taking us? Am Ithe only IT professional in the world thatsees the problem with this trend?—Tex StandridgeHacker DietI love the picture on the table contentspage of the March 2012 issue: iPad,remote keyboard and half-eatensnack. This must truly symbolize thegentrification (I’d say “yuppinization”, butI think yuppie is passé already) of hacking.Twinkies and Mountain Dew replaced bychocolate croissant and latte?—John McClatchieI’m a tea drinker myself, but I won’tpass up a latte if offered. I can’t sayI’ve ever been a fan of Mountain Dewor Twinkies though. I’m not sure whatthat makes me!—Ed.wc Is One Too ManyI was just reading my favorite columnin Linux Journal, Work the Shell byDave Taylor, and he pointed out thatwc doesn’t give an accurate count ofcharacters [see Dave’s column in theMarch 2012 issue]. For example:$ echo linux | wc -c6Nope, you’re not alone. I keep digitalbackups of every .pdf and .epub editionon my computer and backed up on myserver. I’m actually really happy that asa company we’ve decided to offer thecloud-based interactive version, butalso provide a DRM-free downloadableversion for our customers. It helps mesleep better at night!—Ed.This should be 5. First off, I want tothank Dave. I’ve used wc many times,and I never noticed this before (I guess Imostly use it to count lines and words).After realizing this, I used Google to findout why this is happening. It turns out,wc is counting the newline character.This is because echo, by default, endsthe line with a new line “\n”.WWW.LINUXJOURNAL.COM / MAY 2012 / 11

[ LETTERS ]A simple fix for this, in the case of usingecho, would be to use the -n argument,which tells echo not to put a new line atthe end of the line. For example:$ echo -n linux|wc -c5Thanks again to Dave Taylor for hisgreat shell column. I love that hedoesn’t just show you how it’s done,but that he takes you through histhought processes so you see why he’sdoing what he is doing.—Kristofer OcchipintiDave Taylor replies: Thanks forthe kudos, Kristofer. I’ve neverreally understood why wc doesn’thave a simple flag to skip countingnonprinting characters, which wouldsave vast amounts of fuss. It’s evenmore odd if you think about how Cstarts array indexing at zero, not one,so there’s even more oddness in howthings are counted.Anyway, you’re of course right that -naddresses the problem when you’re usingwc with an echo statement, but if youhave a more complicated pipe like this:find . -name "*.doc" -print | rev | sed 's/[aeiou]//g' | wc -cthen it’s hard to figure out where to putthe -n flag. Extra credit: what does theabove actually do?iPad and LinodeAfter reading the article on replacingyour laptop with an iPad [see “Swap YourLaptop for an iPad + Linode” by MarkO’Connor in the March 2012 issue], Itook advantage of the new iPad releaseto pick up an iPad 2, and I was surprisedto find myself enjoying it quite a bit.Of course, I had trouble with iSSHbecause of issues with the Alt andCtrl key, so I’d recommend using theapp called “Prompt” instead. My onlyproblem with it is that I have to use Escas my meta-key. Otherwise, great article!—Chris KingWhy I Will Not Be Renewing MySubscription after Many YearsIn September 2011, you decided to stoppublishing a portable, permanent, easyto-useedition of Linux Journal. For sixmonths, I have received monthly e-mailsannouncing new issues, and when I’vehad time, I’ve downloaded the electronicedition, but—and this is crucial—I havenot read it. I don’t have time to read whenI have a computer in front of me. I don’thave a touchpad, and not being able to putthree or more bookmarks in the magazine,so I can easily move around in it (and seenon-adjacent pages almost simultaneously)all make it much less convenient. I wasallowed to take paper into areas wheremy classified computers are maintainedtoo, but I can’t take electronic devices inwith me (not if I hope to take them out12 / MAY 2012 / WWW.LINUXJOURNAL.COM

[ LETTERS ]again). Print resolution and contrast canbe better on paper too, so the physical actof reading has become less pleasurable. Ihaven’t read Linux Journal in six months,and I see no likelihood of doing so anytime soon. If I’m not reading it, I don’t getany benefit from it.By eliminating the print edition, you havereduced the usefulness of Linux Journalto the point where I might as well notsubscribe. This is not a threat; I don’texpect you to reconsider. I simply wantedyou to know why you have lost a reader.I hope you retain enough readers tocontinue publishing. It was fun whileit lasted.—Mark FishmanSorry the new format doesn’t work foryou. I know a few folks are printingissues out so they can leaf through them,but that seems to be a bit of a kludge atbest. Perhaps someday digital readers willevolve enough to create the tactile feelmost people miss.—Ed.LJ Digital and TabletsSince you went digital, I have enjoyedreading Linux Journal in PDF format onmy desktop or notebook computer, butthis past weekend, I purchased a ToshibaThrive tablet, and one of the first thingsI did was scp the latest issue to thetablet. I have to say, the digital versionand tablets are a marriage made in digitalheaven, and I can once again take my LJwith me wherever I go. I vote “yes” tothe digital version.—Howard PepperI agree. Although I enjoy Linux Journalon an E Ink device, there’s somethingmagical about seeing it on a full-colortablet. It’s almost a bit surreal for somereason. I’m glad you’re enjoying thetablet experience!—Ed.Subscription Site and TrueCryptI just got my February 2012 issue,and the digital edition is what mostof the Letters are about. I’m justgetting used to it. The only complaintI have is that I need my mailinglabel to access the subscription site(https://www.pubservice.com/SubInfo.aspx?PC=LJ&do=D), and Ihave to fetch a printed issue to get it.My second comment is related to theNon-Linux FOSS column in regards toTrueCrypt [see the February 2012 Upfrontsection]. The distro I work with is Fedora,and there is no TrueCrypt package.The reason literally is “The TrueCryptsoftware is under a poor license,which is not only non-free, but has thepotential to be actively dangerous toend users or distributors who agreeto it, opening them to possible legalaction even if they abide by all of thelicensing terms, depending on theintent of the upstream copyright holder.WWW.LINUXJOURNAL.COM / MAY 2012 / 13

[ LETTERS ]Fedora continues to make efforts to tryto work with the TrueCrypt upstreamto fix all of the issues in their licenseso that it can be considered Free, buthave not yet been successful.” (Source:http://fedoraproject.org/wiki/Forbidden_items#TrueCrypt.)On the same page, Fedora suggests“tc-play (https://github.com/bwalex/tc-play) is an independently developedTrueCrypt-compatible program under theBSD license. A tc-play package has beensubmitted for package review for possiblefuture inclusion in Fedora.”Otherwise, I really enjoy reading LJ frombeginning to end. Keep up the great work!—JSchiavonWe have fixed the subscriber loginpage, so you now are able to log in withyour e-mail address and zip code. Inaddition to the login page, the monthlyissue download notification e-mail thatyou receive contains links to your issuearchive, so you can download any issuesyou may have missed, as well as accessyour subscription account. There is also aManage your subscription link that allowsyou to modify your account, renew oraccess the digital archive.Also, thanks for the info on TrueCryptand tc-play. I always feel a bit like I’min uncharted waters when I highlighta Non-Linux FOSS program. Yourfeedback is appreciated by me and ourreaders as well.—Ed.LJ Digital and MonitorsI was, like many of February’s letterwriters,somewhat dismayed by the endof the print edition, but as the compositorfor a couple printed journals (no colorpermitted, adequate paper for halftones,though not great, and certainly notglazed), which have had to be priced ata great deal more per issue than LJ everwas, I do understand the pressures LJ wasfacing. Some of the problems mentioneddon’t seem to me really to exist, however.Jim Fuqua, for instance, condemns thefull-spread layout of the PDF version, butwith any recent version of Adobe Reader,the button just to the right of the zoompercentage box brings up the pagessequentially, one by one, and you can runthrough the entire journal with the mouseif you want to. (That makes me dizzy. TheEnter key works fine for me.) A properlyshaped Reader panel will ensure that thepaging really works in “fit page” mode.I am using FC15 at present, and thatleads to another consideration. FC15does a neat job of setting up an auxiliaryscreen, which can be turned fromlandscape to portrait mode easily. It isa bit pricier than an e-reader and can’tbe carried around, but I, like Jim, am ageezer. I ride free on Philadelphia buses,and by this time, I don’t even have toshow ID. I can’t survive the eyestrain14 / MAY 2012 / WWW.LINUXJOURNAL.COM

[ LETTERS ]caused by any of the e-readers I haveyet seen (I fall asleep), but my 24"diagonal auxiliary screen has put upthis “Contact Linux Journal” page atsomething more than 18" diagonal. I canread it with mid-range glasses, sittingback comfortably and not squinnying—atabout a 30" distance from the text.Of course, this is not in the least bitportable, but neither, in any real sense,is my Dell Latitude laptop. If there is anLJ article that I really find useful, I printit off in black and white. One couldrun through color ink cartridges prettyfast with the saturated colors that thePDF comes in, so I don’t even think ofprinting the entire journal.I did like the printed version, but as Iprune down my set to the last print issue,I will have some much-needed shelf spacefor other things. The digital edition hasthe virtue that it doesn’t take up muchroom. I am close enough to the publishingindustry and its woes to know that LJ hasmade the only possible decision.—PierreI love a secondary monitor in portraitmode. One of these days, I’m actuallygoing to buy a monitor designed forLinux Journal Exclusive Offer:SAVE 60%* on the newIntel E5-2600 dual processor server.NEW Intel Xeon E5-2620Dual Processor Server:• 16GB DDR3-1333 ECC RAM• 1u Rackmount Chassis(4 bay hotswap)• 1TB Enterprise 7.2K SATA HDD• 10TB Bandwidth• CIDR/29 IP Reassignment• $199.95 Monthly (compared to$499 at our largest competitor)* This 60% savings is strictly limited to the first 15 reservations.Go to www.ace-host.net/linuxjournal to claim yours now.

[ LETTERS ]it (Dell makes a really nice one, possibly the oneyou use). For anyone interested in trying it with a“regular” monitor, it’s important to realize somemonitors have poor viewing angles when you tipthem on their side. For looking at a document inportrait mode, however, it’s hard to beat!—Ed.eJournal Is Still Pleasurable to ReadAt first, I wasn’t sure about the paperless LJ; I reallyprefer to read stuff on paper. PDFs tend to have smallprint, even on my old 19" 1280x1024 LCD. But youseem to have boosted the point size a little, andreading on-screen really is rather pleasurable.As to paying for a subscription to the magazine, Ipaid for access to the information, not the paper. Infact, I started with IEEE Micro. When that went away,I subscribed to Embedded Linux Journal. When thatfolded, my subscription was transferred to LJ, whichhas now become, in a way, eLJ. And, I’m still readinginteresting articles about different aspects of Linux,hardware and computing in general.The more things change, the harder it is to peel theorange. Change is inevitable; what matters is howwe manage that change and cope with it.—fest3er8Thank you for the kind words! I have noticed thelayout, while still magazine-shaped on the PDF, is abit different and looks nice on a computer screen. Idon’t know the exact details about what changed,but I think it looks good both on a computer and ona color e-reader. I’m glad you’re enjoying it.—Ed.WRITE LJ A LETTER We love hearing from our readers. Please send usyour comments and feedback via http://www.linuxjournal.com/contact.At Your ServiceSUBSCRIPTIONS: Linux Journal is availablein a variety of digital formats, including PDF,.epub, .mobi and an on-line digital edition,as well as apps for iOS and Android devices.Renewing your subscription, changing youre-mail address for issue delivery, paying yourinvoice, viewing your account details or othersubscription inquiries can be done instantlyon-line: http://www.linuxjournal.com/subs.E-mail us at subs@linuxjournal.com or reachus via postal mail at Linux Journal, PO Box16476, North Hollywood, CA 91615-9911 USA.Please remember to include your completename and address when contacting us.ACCESSING THE DIGITAL ARCHIVE:Your monthly download notificationswill have links to the various formatsand to the digital archive. To access thedigital archive at any time, log in athttp://www.linuxjournal.com/digital.LETTERS TO THE EDITOR: We welcome yourletters and encourage you to submit themat http://www.linuxjournal.com/contact ormail them to Linux Journal, PO Box 980985,Houston, TX 77098 USA. Letters may beedited for space and clarity.WRITING FOR US: We always are lookingfor contributed articles, tutorials andreal-world stories for the magazine.An author’s guide, a list of topics anddue dates can be found on-line:http://www.linuxjournal.com/author.FREE e-NEWSLETTERS: Linux Journaleditors publish newsletters on botha weekly and monthly basis. Receivelate-breaking news, technical tips andtricks, an inside look at upcoming issuesand links to in-depth stories featured onhttp://www.linuxjournal.com. Subscribefor free today: http://www.linuxjournal.com/enewsletters.ADVERTISING: Linux Journal is a greatresource for readers and advertisers alike.Request a media kit, view our currenteditorial calendar and advertising due dates,or learn more about other advertisingand marketing opportunities by visitingus on-line: http://ww.linuxjournal.com/advertising. Contact us directly for furtherinformation: ads@linuxjournal.com or+1 713-344-1956 ext. 2.16 / MAY 2012 / WWW.LINUXJOURNAL.COM

UPFRONTNEWS + FUNdiff -uWHAT’S NEW IN KERNEL DEVELOPMENTThe GNU General Public Licenseis pretty powerful. By ensuring thatderivative works can be distributed only ifthe source code is made available as wellunder the same license, it ensures thata piece of free software can’t suddenly“go dark”, if a given person or companywants to fork the code and release aproprietary version.Since the very early days of Linux,however, there has been a naggingquestion surrounding this issue: arekernel drivers actually derivative works?If the courts ever decide they are, the vastand growing array of binary-only kerneldrivers suddenly will find themselves facedwith a tough question: either releasetheir source code under the GPL or stopdistributing the driver to anyone.Linus Torvalds has said that he’s finewith binary drivers, but other folks havesaid that it’s really the license itself, andnot his opinion, that determines whethersomeone has violated the GPL. Thus far,the courts have not ruled either way, sothe question remains open.From a practical standpoint,however, it’s possible for binary driversto insinuate themselves so deeplyinto the kernel that they really doconstitute a derived work, anyway youslice it. So, a long time ago, the kernelstarted using EXPORT_SYMBOL_GPLto mark kernel interfaces that trulywere considered intrinsic, and thatcould be used only to code thatidentified itself as being releasedunder a GPL-compatible license. So, fora long time now, binary-only drivershave had access only to a controlledsubset of kernel interfaces that preventthem from essentially forking thekernel into a proprietary version.That’s the history, and it’s a balance thatseems to work well enough for everyone.But recently, Christoph Hellwigsubmitted a patch, which was accepted,that dramatically changed that balance inthe Virtual Filesystem (VFS). As AntonAltaparmakov said in response to thepatch, Christoph’s code would make itvirtually impossible for binary-only driversto read or write files the way they alwayshad before. The drivers would have tore-implement from scratch all the VFSinterfaces for reading and writing.Ultimately, it turned out thatChristoph had not intended to add theserestrictions, but the discussion revealed afairly big gap between the opinions and18 / MAY 2012 / WWW.LINUXJOURNAL.COM

[ UPFRONT ]Open Formats,Open EditorsThey Said ItBuying the rightcomputer and gettingit to work properly isno more complicatedthan building anuclear reactor fromwristwatch parts in adarkened room usingonly your teeth.—Dave BarryThere is only onesatisfying way toboot a computer.—J.H. GoldfussE-books are currently quite a hot topic inthe publishing world. Heck, for the pastfew months, it’s been quite a hot topichere as well! Thankfully, digital publicationdoesn’t have to mean proprietary formatsand DRM-laden files.If you want to delve into the world ofe-book creation, you should check out Sigil.It’s a cross-platform WYSIWYG tool forcreating EPUB-format e-books. It supportsimages, table of contents, and all the otherfeatures that make e-books so powerful. Ifyou have a book you want to convert intoEPUB format, or if you just have a text fileyou’d like to add markup language to fore-readers, Sigil is a powerful tool you won’twant to miss.Download the latest version fromcode.google.com/p/sigil. —SHAWN POWERSIn a way, staring intoa computer screen islike staring into aneclipse. It’s brilliantand you don’t realizethe damage until itstoo late.—Bruce SterlingIf the automobilehad followed thesame developmentas the computer, aRolls-Royce wouldtoday cost $100, geta million miles pergallon, and explodeonce a year killingeveryone inside.—Robert CringelyA computer oncebeat me at chess, butit was no match forme at kick boxing.—Emo Philips20 / MAY 2012 / WWW.LINUXJOURNAL.COM

[ UPFRONT ]That’s a Beautiful$DOCUMENT_TYPEYou’ve Got ThereOne of the biggestfrustrations mostnew LibreOffice(or OpenOffice.org)users have is thelack of templatesand clip art. We’veaddressed thisproblem before,but with therecent surge ofLibreOffice, it’simportant to knowhow to improveyour powerfuloffice suite!LibreOfficeoffers bothtemplates andextensions toadd creativity and flair to yourdocuments. Whether you’re lookingfor a template to aid with your homebudgeting spreadsheet or want someclip art to advertise your garagesale, the LibreOffice Web site holdsplenty of options.A quick click to templates.libreoffice.orgor extensions.libreoffice.org will giveyou a huge list of available add-ons.You also will have the opportunity toadd your own templates or extensionsto the Web site so others can benefitfrom your creations! The site featuresa voting system to help promote thehighest quality submissions, and asearch feature narrows down theavailable items to fit your needs. Checkit out $DATE_THIS_ARTICLE_IS_READ!—SHAWN POWERSWWW.LINUXJOURNAL.COM / MAY 2012 / 21

[ UPFRONT ]Programming for ScientistsMy last several articles have covereddifferent software packages that are usefulto scientists trying to do computationalscience. I tried to explore as broad aspectrum of subjects as I could in thosepieces, and I even covered some basicprogramming constructs like MPI or scipy.But, I always have been limited by theamount of space a venue like this allows.All I can do is provide a taste of what isout there and hope that readers take itaway and learn more on their own. Also, inmany cases people find themselves doingresearch in areas that never have beendone by anyone else before. This meansthere will not be an appropriate softwarepackage, and researchers will need to writetheir own software from scratch.One major problem for computationalscience researchers is that they simply do nothave the time to attend normal classes overthe span of a term or two in order to learnthe skills they need to do their work. Theyneed to be able to jump-start their researchand essentially go from zero to 100mph inno time flat. Part of my day job is to helpthem do this. I provide crash courses in mostof the subjects that they may need. But,what can they do when they leave and try toapply this information several days or weekslater? Enter the Software Carpentry site(software-carpentry.org), a resource thatshould be on every researcher’s bookmarklist. I have no association with the authorand maintainer of the site. I’m just glad tohave a high-quality source of information towhich I can point my users.The first level of resources available isa set of self-paced on-line workshops.These workshops are distributed under acreative commons license, specifically theCreative Commons Attribution License. Thismeans you are free to use the material andremix it, as long as you properly attributethe author. These workshops cover a vastnumber of subjects and are available asboth PDF and PowerPoint files. For someof the workshops, video screencasts evenare available, so it’s almost like having aninstructor right there with you. Each topic isbroken down into smaller sections to makedigesting them easier. Additionally, exercisesare available so you can review the material.Many new researchers, graduatestudents and post-docs have had littleor no experience in computationalscience at all. Many never even have seenany type of UNIX environment. This isquite a stumbling block, as most highperformancecomputing centers that Iknow of run Linux. So you probably willwant to start with the workshop TheShell. This workshop is broken down intothe following sections:n Introductionn Files and Directories22 / MAY 2012 / WWW.LINUXJOURNAL.COM

[ UPFRONT ]Python is relatively similar to otherlanguages (like C), so you should be ableto apply what you learn here to thoseother languages with just minor syntaxtranslations. Also, Python is growing inpopularity in scientific programming circlesdue to its clean formatting rules and therelative ease of incorporating externalhigh-performance libraries written in C orFORTRAN. In this sense, you almost canconsider Python to be a glue language,but it has quite a lot of capability availabledirectly through external libraries likenumpy and scipy. You could do worse as acomputational scientist than learn Python.With its growing popularity, there is alsoa greater chance that the specific problemarea you’re researching already has tools orlibraries available.Once you have at least one languageunder your belt, it is time to learn more ofthe details involved in programming itself.These workshops cover the following:n Program Design: goes througha simple example of designing,debugging and improving a program.n Testing: how you should test yoursoftware, handle exceptions and dounit tests.n Make: how to use rules, patterns andmacros to build your software.These topics cover a lot of the extraitems you need to know in order toprogram effectively, but they aren’tstrictly programming proper. That topic iscovered by the following workshops:n Sets and Dictionaries: using associativedata structures to represent data thatdoesn’t really fit into a list.n Regular Expressions: how to useregular expressions for patternmatching.n Databases: an introduction to SQL.n Data Management: an introduction tomanaging your data.n Matrix Programming: using numpy tohandle numerical processing.n Multimedia Programming:programming using sound, picturesand other media files.n Spreadsheets: using spreadsheets foranalysis and visualization.With these workshops, you will learnmany of the programming elements andstructures that will be of use to you inscientific programming. After this, youshould have covered enough, hopefully,to be able to program a solution to theproblem you are studying. Again, all of theseworkshops include exercises, so you actuallycan try applying what you have learned. I’ma firm believer that you don’t learn anything24 / MAY 2012 / WWW.LINUXJOURNAL.COM

[ UPFRONT ]until you actually try to use it.In-person workshops and boot campsalso are available. Because all of thematerial is available for free reuse, yousimply can use the workshop materials toput on your own workshop or boot camp.The team behind Software Carpentry also isavailable to do in-person boot camps. Youcan contact them through the Web site tomake arrangements. These boot camps aretwo- or three-day crash courses to coverthe bulk of the material, and they’re alwaysbeing offered at different places aroundthe globe—follow the blog to see whenone is being offered in your neck of thewoods. If you do decide to run your own,the team at Software Carpentry is happyto help out and spread the word throughits network. A forum is available at theWeb site for each of the workshop topicswhere you can discuss the material withother attendees or other presenters.Finally, I suggest that you actuallysubscribe to the blog RSS feed. Newworkshops always are being added,and new boot camps always are beingplanned. Watching the RSS feed will keepyou informed about these additions. Asalways, feel free to contact me if youhave anything specific you’d like to seecovered here. Hopefully, I’ve been able toplant the seed and give you ideas on howyou can pick up the skills you need.—JOEY BERNARDNon-Linux FOSS(Image from www.xonotic.org.)Some say Windows has all the bestgames, and in the case of Xonotic, it’spartially true. Xonotic is a free, opensource,first-person shooter designed bythe developers of Nexuiz. Xonotic usesa highly modified version of the Quakeengine called DarkPlaces. Sporting bothsingle and multiplayer modes, the fastpacedaction and detailed maps will pleaseeven the fussiest gamers. Download theWindows version of Xonotic from theWeb site: www.xonotic.org.Oh, and for those folks not runningWindows? Yeah, it’s available for youtoo. Xonotic runs on Windows, OS X and Linux. Get your frag on!—SHAWN POWERSWWW.LINUXJOURNAL.COM / MAY 2012 / 25

COLUMNSAT THE FORGEDesignFrameworksREUVEN M.LERNERWant your site to look good, even though you’re not adesigner? Try a design framework.For as long as I can remember, I’veknown how to use a pencil. I can writewith it, and I even can draw with it—although in my case, saying I can draw issomething of a sad exaggeration. I mightknow how to use a pencil and thoroughlyunderstand its technology, but thattechnical knowledge doesn’t mean I candraw something aesthetically pleasing.I mention this because I always thinkof my poor drawing abilities when I workwith CSS. I understand the technologyand have used it for many years. I’mcomfortable working with and modifyingstylesheets, and using complex selectors.And yet, any Web application that Icreate on my own looks ugly. Despite allof my technical knowledge of CSS, I’mgenerally unable to make a nicer, morepleasing design.Fortunately for people like me, anew type of framework exists to helpwith such problems. Just as serversideframeworks like Ruby on Rails andDjango make it easy to create Webapplications, and client-side frameworkslike Backbone.js and Knockout.js makeit easy to create in-browser applications,design frameworks make it easy todecorate and design your applications.Now, when I first heard about designframeworks (sometimes known as CSSframeworks), I wondered how they couldpossibly help. After all, I already knowCSS; what could they contribute? Theanswer is: a great deal. By embracing adesign framework, you gain a number ofCSS classes and IDs that make it very easyto lay out your page. By choosing theright classes, you can make a site lookmore than reasonable, even if you havemy graphic design skills. Using a designframework means that even if you don’thave any design skills, you can make asite that looks fairly pleasing. If you dohave design skills, the framework willallow you to do more with less effort.In this article, I describe some of theleading design frameworks, leading upto the latest and most interesting one,26 / MAY 2012 / WWW.LINUXJOURNAL.COM

COLUMNSAT THE FORGE“last” class, the layout will look prettygood. This is far better, in every way,from what many people would do toensure a certain layout, namely the useof tables. Tables are great, but not forlaying out a page. I can’t tell you howmany times I’ve seen tables triple-nestedinside other tables, just to get the layoutto work appropriately.Another advantage of Blueprint isthat it has tried and tested these CSSdefinitions on a variety of browsers. Youno longer need to worry about whetherthings will work on Chrome, Firefox, IEand Safari, because Blueprint has takencare of that for you. The special IE-onlyCSS file ensures that Internet Exploreralso will work correctly, even if your usersare running an old version.Of course, because Blueprint is abunch of CSS files, you always canchange it to suit your needs. Andindeed, much of the (extensive)Blueprint documentation describeshow you can customize not only thelayouts, but also the CSS itself.The biggest criticism of Blueprint isthat its class names violate the spirit ofCSS. Sure, you can have classes named“span-5” and “span-10” to indicate thatyou want text to extend across five orten columns of your 24-column layout.But, wasn’t CSS supposed to free youfrom such calculations, allowing you tothink more in terms of semantic names?My response is that, yes, this is a goodpoint, but Blueprint’s layouts are souseful and so easy to understand, it’sunhelpful to focus on the original intentof CSS’s designers.Sass, SCSS and CompassAnother problem with Blueprint isn’tinherent to Blueprint itself, but ratherto the technology on which it’s based.CSS is a very clever technical standard,but it has been demonstrably difficultfor people to learn and understand,and also for many designers to use.True, some people are able to makebeautiful, clever designs with CSS,and I don’t mean to diminish theircapabilities in any way. But, CSS has anumber of deficiencies, many of which Ihadn’t thought about until I discoveredSass a few years ago.Sass (Syntactically AwesomeStylesheets) is a CSS preprocessor. Thatis, it is a file format that you compile,with the output of the compilation beinga CSS file. Sass has a large number ofadvantages over straight CSS, startingwith the fact that you can set variables.This is a tremendous thing when you’recreating a site that should have aconsistent color scheme. You can set thecolors in one place and then use them inmany different places.Sass also lets you nest CSSdefinitions, such that if you haveseveral elements that share definitions,you no longer have to repeat thosesimilarities in different places, but cantake advantage of something akin to28 / MAY 2012 / WWW.LINUXJOURNAL.COM

COLUMNSAT THE FORGEvariable scoping in a programminglanguage. Nesting also means thatcomplex selectors become easier bothto write and to read, since they breakthe path down into digestible parts.Sass also offers a type of functionor macro definition, known as a“mixin”, allowing you to keep yourCSS DRY (“Don’t Repeat Yourself”).This means you can define a CSS rulein one place and then include it inmultiple other places.Now, Sass has been around forseveral years, and it has workedwell during that time. It originallywas written by developer HamptonCatlin (who also developed the Hamltemplate format), but has since beentaken over by Nathan Weizenbaumand Chris Eppstein. The original Sasssyntax resembled Python in someways, in that trailing semicolonswere removed and indentation was amandatory part of the syntax. That is,the indentation of your declaration, oreven of the definition of your mixin,mandated indentation, and the level ofindentation reflected the scope withinwhich your definitions would takeeffect. For example, here is a simple setof definitions in Sass, taken from thesass-lang.com home page:table.hlmargin: 2em 0td.lntext-align: rightlifont:family: serifweight: boldsize: 1.2emYou then would run a Sass compiler overthe Sass file, producing a CSS file that couldbe interpreted by a Web browser.This syntax still exists, and if youprefer it, Sass happily will work withyou. However, the preferred syntaxis now known as SCSS, and it is asuperset of the existing CSS syntax. Myimpression is that the syntax changedpartly because the differences betweenCSS and Sass were too great for manypeople, and the mandatory indentationcaused problems for a number of users.Thus, the Sass shown above can beexpressed in SCSS as follows:table.hl {margin: 2em 0;td.ln {text-align: right;}}li {font: {family: serif;weight: bold;size: 1.2em;}}WWW.LINUXJOURNAL.COM / MAY 2012 / 29

COLUMNSAT THE FORGEbut they’re pretty minor.One difference between LESS andSass is that LESS files can be compiledin a number of different ways. They canbe turned into CSS using a compiler—originally written in Ruby, but currentlywritten in JavaScript—running on thedeveloper’s computer, either beforeor during the deployment process.However, because the implementation isin JavaScript, another option is available.You can download the LESS stylesheet,as is, into your browser and then use aJavaScript library to translate it into CSSon the fly. For example:ConclusionGiven the similarity between LESS andSCSS, I’m not surprised a frameworkemerged that is based on LESS. Butthe framework that was created(Bootstrap, written by two engineersat Twitter and released under theApache license) has turned out tobe a huge hit among developers anddesigners alike. Bootstrap not onlyincludes Blueprint’s grid, but also alarge number of other conveniencesand stylings that make it easy to havegood-looking navigation bars, tables,forms and even widgets for dynamic,For performance reasons, it’s bestto compile the LESS file into CSSbeforehand. However, this is useful whenthe compiler is not available, if you wantto generate stylesheets dynamically, andalso during development.Why would you prefer LESS over SCSS,or vice versa? I honestly cannot givea compelling reason for one over theother. They’re so similar and have suchadvantages over plain-vanilla CSS thatit doesn’t matter which you choose, solong as you go with one of them. BeforeBootstrap, I would have put my moneyon SCSS becoming the de facto standard,but Bootstrap might have given LESS anew lease on life, and might even help itovertake SCSS.LINUX JOURNALon yourAndroid deviceDownloadapp now inthe AndroidMarketplacewww.linuxjournal.com/androidWWW.LINUXJOURNAL.COM / MAY 2012 / 31

COLUMNSAT THE FORGEclient-side applications. Bootstrap evenis capable of modifying the size andshape of menus depending on screensize, making it a “reactive” frameworkappropriate for mobile devices as muchas desktop computers.Some people even are getting tiredof Twitter Bootstrap, because it hasbecome so popular and makes iteasy to create designs that look likemany other Bootstrap sites. But forsomeone like me, who just wants toget a simple site up and running, andfor it to be relatively nice-looking, I’vefound Bootstrap to be a huge help.In my next article, I’ll actually lookat Bootstrap—what it provides, howto install and use it, and even howto use it along with Ruby on Rails,which (as I mentioned above) comeswith support for SCSS, rather thanLESS. Regardless of which of thesesystems and frameworks you usethough, I strongly encourage youto try them and incorporate theminto your own work. An experienceddesigner presumably will know howto integrate these technologies, whiledesign-challenged programmers willwelcome the chance to create a nicelookingsite on their own.■Reuven M. Lerner is a longtime Web developer, consultantand trainer. He is also finishing a PhD in learning sciences atNorthwestern University. His latest project, SaveMyWebApp.com,went live this spring. Reuven lives with his wife and children inModi’in, Israel. You can reach him at reuven@lerner.co.il.ResourcesIf you are interested in CSS, it might be usefulto read up on the subject. The W3C’s standardis at http://www.w3.org/Style/CSS. Youalso might be interested in HTML & CSS: TheGood Parts, a book written by Ben Henick andpublished by O’Reilly that tries to point readersin the direction of useful and appropriatetechniques. Another good O’Reilly book onthe subject of CSS is the CSS Cookbook(3rd edition) by Christopher Schmitt.Blueprint has not been updated sinceMay 2011, but it still works and is welldocumented. You can read about it anddownload it from http://blueprintcss.org.Sass/SCSS is documented athttp://sass-lang.com. The site includesmany recipes for common things that peoplewant to do. Compass, the framework built onSass/SCSS, is at http://compass-style.org.Note that many on-line tutorials andscreencasts about Compass use the oldSass syntax and, thus, might be a bit hard tounderstand if you know only the new syntax.Two books about Sass and Compass areSass and Compass in Action by WynnNetherland, Nathan Weizenbaum andChristopher Eppstein, published by Manning;and Pragmatic Guide to Sass, written bySass creator Hampton Catlin, published by thePragmatic Programmers.Finally, if you are interested in comparingthe finer points of Sass and LESS,a nice chart and introduction is athttps://gist.github.com/674726.32 / MAY 2012 / WWW.LINUXJOURNAL.COM

COLUMNSWORK THE SHELLCalculatingWord Point ValuesDAVE TAYLORDave continues improving the Scrabble and Words WithFriends script.My last article ended by wrappingup the word finder utility for Scrabbleand Words With Friends, a fun andcomplicated project that ended uprequiring about 65 lines of Bourne Shell.That’s not too long, but for a shell script,actually, it is rather long, and it was oneof the scripts we’ve written that mostbegged to be coded in Perl or anothermore string-friendly language. Still, wepersevered, right?In this article, let’s wrap things up bylooking at word values based on theindividual letter values in both Scrabbleand Words With Friends.To start, here’s how the point chartlooks for Words With Friends:A=1 D=2 G=3 J=10 M=4 P=4 S=1 V=5 Y=3B=4 E=1 H=3 K=5 N=2 Q=10 T=1 W=4 Z=10C=4 F=4 I=1 L=2 O=1 R=1 U=2 X=8This contrasts with the somewhatlower point value of letters in thegame Scrabble:A=1 D=2 G=2 J=8 M=3 P=3 S=1 V=4 Y=4B=3 E=1 H=4 K=5 N=1 Q=10 T=1 W=4 Z=10C=3 F=4 I=1 L=1 O=1 R=1 U=1 X=8If you look closely, you’ll see that justabout every word is going to be worthless in Scrabble than in Words WithFriends—interesting. I always assumedthat the two used the same basic letterfor-letterpoint values, so that calculatingpoint values in one game taught you howto calculate points for the other—not so!Calculating Point ValuesThere are a lot of ways to translate asequence of letters into their individualpoint values and calculate the sum ofthose values. Indeed, an array comesto mind immediately, but the problemis that you still have to write the codeto step through the individual letters,and if you’re doing that, why not usesed for the letter → numeric valuesubstitution instead?It turns out that’s a fast and elegant34 / MAY 2012 / WWW.LINUXJOURNAL.COM

COLUMNSWORK THE SHELLNow that you have a bunch of numeric values,however, what do you do with them?solution, and I’ve coded it also tocompare easily what letters have a givenpoint value in wwf (aka Words WithFriends) and s (aka Scrabble).Here’s how I coded the one-pointletters in each:wwf1="s/[asetior]/ 1 /gs1="s/[asentiloru]/ 1 /g";As you can clearly see, Scrabble hasa lot more one-point letters than WordsWith Friends does. Again, I had no ideauntil I started this analysis.With sed expressions coded up thisway, it’s easy to turn a word like “cat”into “4 1 1”. Here’s the full set ofsubstitutions:wwf1="s/[asetior]/ 1 /g"wwf2="s/[dlnu]/ 2 /g"wwf3="s/[ghy]/ 3 /g"wwf4="s/[bcfmpw]/ 4 /g"wwf5="s/[kv]/ 5 /g"wwf8="s/[x]/ 8 /g"wwf10="s/[jqz]/ 10 /g"s1="s/[asentiloru]/ 1 /g"s2="s/[d]/ 2 /g"s3="s/[mpbc]/ 3 /g"s4="s/[vyhwf]/ 4 /g"s5="s/[k]/ 5 /g"s8="s/[jx]/ 8 /g"s10="s/[qz]/ 10 /g"Quite honestly, entering all that datais the hardest part of creating the wordpoint-valuescript. It’s tedious work, soyou’ll be smart to copy and paste.Now that you have a bunch of numericvalues, however, what do you do withthem? It turns out that’s easy too:sed 's/ / + /g'Think about the original substitution,and you’ll see what I’m doing: the firstletter substitutes to letter, as does every subsequentletter. Therefore, individual spacesdenote the very beginning and end ofthe word, while double spaces are onlybetween digits. Replace those doublespaces with a +, and that sequenceof digits translates into a simplemathematical formula:4 + 1 + 1To solve simple math, you can use$(( )) as a shell notation, or you cancall the built-in function expr—sameWWW.LINUXJOURNAL.COM / MAY 2012 / 35

COLUMNSWORK THE SHELLbasic result.Before we get there, however, here’show I’ll use that sequence of sedsubstitutions in the script:wwfexpr=$(echo $1 | sed "$wwf1;$wwf2;$wwf3;$wwf4;$wwf5;$wwf8;$wwf10"➥| sed 's// + /g')sexpr=$(echo $1 | sed "$s1;$s2;$s3;$s4;$s5;$s8;$s10" |➥sed 's/ / + /g')Let me explain the sh script.shnotation briefly, in case you haven’tsee this approach before. A classic waythat hackers Trojan Horse a Linux orUNIX system is to drop a shell script likevi or ls into somewhere like the /tmpdirectory. It’s not a problem, unless yourPATH looks like this:.:/bin:/usr/bin:/usr/local/binYou can see I’ve tucked the doublespace-to-plus-signsubstitution into thesame subshell invocation too—short andneat. In fact, those two lines are theheart of the script. There’s only one lineleft actually, and it both calculates theactual values and shows the result:echo "\"$1\" has a base point value of $(expr $wwfexpr) in WwF➥and $(expr $sexpr) in Scrabble"This is what I really like aboutprogramming with the power of theentire Linux shell at your fingertips: thisscript that calculates point values for agiven word in both Scrabble and WordsWith Friends is actually only three lineslong, if you don’t count the variables weset up at the beginning.Now, let’s run it to see what kind ofvalues we see:in which case you can unwittingly runthe invasive script and possibly createa setuid root copy of the shell forthe bad guys to exploit at their laterconvenience. Security’s a bit far afieldfor this particular column, but suffice itto say that for security reasons, I neverhave “.” in my PATH.Therefore, I could use ./script toinvoke the script in my current directoryif it’s marked as executable, but sinceI have so many scripts lying around, Ifind it even safer to not mark them asexecutable until I’m 100% sure they’redone and tested. Instead, sh scriptworks just as well, although it spawns asubshell for execution.Now you know. And, here are moreexamples:$ sh wordvalue.sh word"word" has a base point value of 8 in WwF and 8 in Scrabble$ sh wordvalue.sh calculate"calculate" has a base point value of 18 in WwF and 13 in Scrabble$ sh wordvalue.sh linux36 / MAY 2012 / WWW.LINUXJOURNAL.COM

COLUMNSWORK THE SHELL"linux" has a base point value of 15 in WwF and 12 in Scrabble$ sh wordvalue.sh journal"journal" has a base point value of 19 in WwF and 14 in ScrabbleAt this point, I’ll leave it as an exercise foryou, the reader, to figure out how to graftthis functionality onto the script we wrotein the previous few articles that calculatedpossible words from a set of letters.The additional bonus task is to beable to analyze the board so you canfigure out how to cover DL, TL, DW andTL squares, as available (that stands fordouble letter, triple letter, double wordand triple word, in case you’re not ahard-core word-gamer). Beware though,it’s considerably more difficult, becausenow you have to figure out how to enterthe current state of the board—definitelyextra credit!■Dave Taylor has been hacking shell scripts for more than 30 years.Really. He’s the author of the popular Wicked Cool Shell Scriptsand can be found on Twitter as @DaveTaylor and more generallyat www.DaveTaylorOnline.com.

COLUMNSHACK AND /The Sysadmin’sToolbox: iftopKYLE RANKINWho’s using up all the bandwidth, and what are they doing?Use iftop to find out.Longtime system administrators oftentake tools for granted that they’ve usedfor years and assume everyone else hasheard of them. Of course, new sysadminsjoin the field every day, and even seasonedsysadmins don’t all use the same tools.With that in mind, I decided to write a fewcolumns where I highlight some commonbut-easy-to-overlooktools that make lifeas a sysadmin (and really, any Linux user)easier. My last article covered sar, a toolyou can use to collect and view systemmetrics over time. This time, I discuss aprogram that’s handy for viewing real-timenetwork performance data: iftop.Anyone who’s had to use a networkat a conference has experienced whathappens when there just isn’t enoughnetwork bandwidth to go around. Whileyou are trying to check your e-mail, otherpeople are streaming movies and TVshows, downloading distribution installdisks, using p2p networks, upgradingtheir distributions or watching cat videoson YouTube. Although it’s certainlyfrustrating to try to use one of thosenetworks, imagine how frustrating itwould be to be the admin in chargeof that network. Whether you run aconferenceWhether you run a conference network, alocal office network or even a Web serverat your house, it can be really nice to knowwhat is using up all of your bandwidth.network, alocal officenetwork oreven a Webserver atyour house,it can be really nice to know what isusing up all of your bandwidth.iftop is a Linux command-line programdesigned to give you live statistics aboutwhat network connections use the mostbandwidth in a nice graphical form. As youmay realize from the name, iftop borrows38 / MAY 2012 / WWW.LINUXJOURNAL.COM

COLUMNSHACK AND /For me, the really great thing about iftop is that it’sa relatively simple command-line tool.each network connection between a pairof hosts. In between the two hosts arearrows that let you know the directionthe traffic is flowing. The final threecolumns provide average bandwidthfor each connection during the last 2,10 and 40 seconds, respectively. So forinstance, the very top connection inFigure 1 has averaged around 2.83Mbduring the last 2 seconds, 3.32Mbduring the last 10 seconds and 3.11Mbduring the last 40 seconds. Underneathall the transmit and receive columns atthe bottom of the screen are a seriesof statistics for overall transmitted andreceived traffic (TX and RX, respectively)including 2-, 10- and 40-secondaverages for both those and, finally,the totals for the interface.Note: if you have a server withmultiple interfaces, you may want iftopto monitor a different interface fromthe default. Just add -i followed by theinterface to monitor when you launchiftop. For instance, to monitor eth2, Iwould type iftop -i eth2.Disable DNS LookupsBy default, when you run iftop, it will tryto translate all of the IP addresses intohostnames. Sometimes this can be usefulif you are diagnosing issues on a localnetwork; however, like with a lot of othernetwork diagnostics tools, resolving allof those IPs can slow down the programand also may contribute to the trafficyou see in the output. The solution is torun iftop with the -n argument, so it justshows you IP addresses for everything(you always can run a DNS query againstan IP you are interested in, in anotherwindow). Alternatively, if you alreadyhave iftop running, you can press n todisable DNS lookups.Show Port DataWhen you run iftop on a server thatmight serve multiple purposes, it canbe handy to know whether all of thatupstream traffic is accessing your Webserver, your mail server or somethingelse. Alternatively, if you are trying tofigure out what’s using up all of yourdownload bandwidth, it can be handyto see whether the top connectionsare Web connections or some rsyncjob you have running. To figure all ofthis out, iftop allows you to toggle theport display on and off. Press the p keywhile iftop is running, and it will displaythe ports used for both the source anddestination IP for all traffic.The one big downside to showingboth the source and destination portsused for a connection is that you’ll findin many cases you are concerned only40 / MAY 2012 / WWW.LINUXJOURNAL.COM

COLUMNSTHE OPEN-SOURCE CLASSROOMLTSP, Part III:Servers Unite!SHAWN POWERSWhat’s better than a lab full of thin clients? Ten labs full of thinclients! This article shows how to scale LTSP. If you think moreis better, read on!My last two articles have focused onLinux thin clients. I’ve covered how toset them up, how to administer themand even how best to tweak your serverto meet your needs. This article finishesthe series by describing how to scaleLTSP in large environments. There are afew different methods, and each has itsadvantages and disadvantages.The Ants Go Marching One by OneOne of the options for a large LTSP rolloutis simply not to scale at all. Thismay seem like a cop-out, but since LTSPworks so well in a classroom (or similar)environment, simply adding a secondnetwork card to your workstation classcomputers is a simple way to serve4–5 thin clients per classroom. In fact,if your network infrastructure is oldand can’t support the large bandwidthrequirements thin clients demand,this type of setup is perfect. The highbandwidth is managed by a cheapdesktop switch, and the only trafficon your main building network is forInternet and file sharing.When you’re setting up your networkin this way, it’s important to realize thethin clients in one classroom won’t beable to see the thin clients in anotherclassroom. Since every classroom hasits own server, every server is setting upits own NAT for the thin clients to live.If you plan to do something fancy likesharing a printer connected to a thinclient, keep in mind only the server willbe visible to the rest of the network.(You still can get around this limitationby sharing the thin-client-connectedprinter via the classroom server. Linux isincredibly flexible!)If a classroom-server-type setup soundsideal for you, setting it up is as simple asconfiguring the classroom workstation(often the “teacher station”) as an LTSPserver. It still will function perfectlyfine as a workstation, but because it42 / MAY 2012 / WWW.LINUXJOURNAL.COM

COLUMNSTHE OPEN-SOURCE CLASSROOMon a file from an LTSP server, so you getall the pertinent thin-client options in theglobal section of the conf file.In the above section of the conffile, you’ll notice the next-serverstatement. That tells the thin clientwhere to go for its TFTP boot file. Thefilename directive tells the clientwhat boot file to download from theTFTP server. It is possible to host thePXE boot files on a central server, andthen point the thin clients to anotherserver for the rest of the boot process,but it gets complicated really quickly.I find it’s much easier to hand the thinclient off to the LTSP server right insidethe dhcpd.conf file. It means each LTSPserver must be running TFTPD, but sinceeach server is already waiting for thinclients to connect, it’s not a big deal.If you do the DHCP configuration/trickery above, configuring each LTSPserver is pretty much the same as thestandard server model. You still needto create the chroot, and you still needto run ltsp-update-image anytimeyou make a change to the chroot, andany changes you make must be doneto each and every LTSP server on yournetwork. It isn’t an efficient way tomanage a network, but it’s conceptuallyvery simple and easy to troubleshootwhen something goes wrong. The bigdownside is replicating changes to everyserver and updating each chroot. Ifyou’re thinking there must be a betterway, well, you’re right!One Server to Rule Them AllA few years ago, some of the brilliantfolks working on the LTSP Project(Stéphane Graber was the main herohere) realized that NBD was so efficientat serving the chroot image, it reallywouldn’t be a bottleneck to use asingle NBD chroot image and spread theactual workload of running applicationsacross a cluster of servers. As with mostgreat ideas, this introduced a bunch ofproblems, but for the most part, theseproblems have been addressed, and LTSPCluster is a viable, easily scalable way toimplement a large LTSP install.There is a Web site for LTSP Cluster(http://www.ltsp-cluster.org), but tobe honest, it’s not the most up-to-datesite from which to learn. The best walkthroughI’ve been able to find is on theUbuntu site: https://help.ubuntu.com/community/UbuntuLTSP/LTSP-Cluster.Following the directions on that site willget you a fully functional cluster justwaiting to be tweaked. For the purposeof this article, I’ll explain what happensin a cluster environment, so hopefullythe concept makes sense. Because clustersupport is arguably the least-developedaspect of LTSP, you might decide it’s notworth the effort.When running LTSP Cluster, you mustdedicate a server (or virtual machine)as the NBD server for the entirenetwork. You also can create a highlyavailable scenario with fancy DHCP/DNS magic, but for most scenarios, a44 / MAY 2012 / WWW.LINUXJOURNAL.COM

COLUMNSTHE OPEN-SOURCE CLASSROOMsingle NBD server is the way to go. Thechroot environment is very similar to thetraditional LTSP model, but when runningthe ltsp-build-client script, youhave to add the --ltsp-cluster flagso the clustering options are built in.Once the NBD image is created, separateservers, referred to as “applicationservers”, are added to the clusterconfiguration. When a client boots up,the NBD server queries the attachedapplication servers, determines whichserver is least taxed and assigns the thinclient to that server. From that point on,until reboot, the client runs its applicationsvia SSH from the application server itwas assigned. It still is possible (andrecommended) to use local apps in order toutilize the power of the thin-client device.The main configuration differencewith LTSP Cluster is that the lts.conffile has been replaced with a Postgresdatabase and a Web interface. Onceyou get the hang of using the slightlyconfusing interface, you’ll find it is apowerful way to organize configurationsbased on individual client machines(Figure 1). I also created a short video acouple years ago, demonstrating how toadd nodes in the Web interface, becauseit can be a bit daunting at first glance:http://youtu.be/7QdYW-NT_sw.LTSP Cluster is truly the mostadvanced way to scale your thin-clientenvironment. Because it does add acouple layers of complexity to yourrollout, I recommend that before youimplement Cluster, you at least arefamiliar with how LTSP works with itstraditional installation method. In myexperience, when LTSP Cluster works, itworks very well. When something goeswrong, troubleshooting is tougher thanwith a traditional setup. As with mostthings, your mileage may vary.Thin Shmin, We Like Our Clients Fat!Finally, there is arguably the least-serverintensivemodel for implementing thinclients, and that is not to use thin clients.Yes, yes, my mastery of the obvious isastounding, but seriously, LTSP supports aOpen Frame PPCNew - PPC-E10Ÿ10 " 1024x600 TFT LED Backlit LCDŸFanless ARM9 400Mhz ProcessorŸAnalog Resistive TouchscreenŸUp to 1 GB Flash & 256 MB RAMŸ10/100 Base-T EthernetŸ3 RS232 & 1 RS232/422/485 PortŸ1 USB 2.0 (High Speed) Host portŸ1 USB 2.0 (High Speed) OTG portŸ2 Micro SD Flash Card SocketsŸSPI & I2C portsŸI2S Audio Interface w/ Line-in/outŸOperating Voltage of 12 to 26 VdcŸOptional 2D Accelerated Video & DecoderŸPricing starts at $595 for Qty 12.6 KERNELThe PPC-E10 Compact Panel PC comes ready to run with theOperating System installed on Flash Disk. Apply power and watcheither the Linux X Windows or the Windows CE User Interfaceappear on the vivid 10” color LCD. Interact with the PPC-E10using the responsive integrated touch-screen. Everything worksout of the box, allowing you to concentrate on your application,rather than building and configuring device drivers. Just Write-Itand Run-It. For additional information please contact EMAC.www.emacinc.com/panel_pc/ppc_e10.htmSince 1985OVER27YEARS OFSINGLE BOARDSOLUTIONSEQUIPMENT MONITOR AND CONTROLPhone: ( 618) 529-4525 · Fax: (618) 457-0110 · Web: www.emacinc.comWWW.LINUXJOURNAL.COM / MAY 2012 / 45

COLUMNSTHE OPEN-SOURCE CLASSROOMFigure 1. The Web interface is confusing, but useful.“fat-client” model. Basically, rather thandepending on the LTSP server for runningapplications, the NBD chroot imagecontains a complete Linux install, so everyapplication, even down to the windowmanager, is run from the client. In thismodel, all the server is responsible for isserving out the NBD image. To enable thefat-client mode, you need to add the--fat-client flag when runningltsp-build-client to create your chroot.As with every scenario, this hasits shortcomings. Because absolutelyeverything is run from the client, the fatclientmodel requires powerful machines.In fact, the specs for a fat client aresimilar to that of a standalone computer,with the exception of a hard drive. Thefat clients require a substantial networkconnection to the server, and unless youhave a very fast networkinfrastructure, you mightbe better off with actualstandalone computers.If you have a bunchof workstations withouthard drives, or if themaintenance time savedby having a single chrootenvironment for all yourworkstations is worth it,fat clients can be incrediblyuseful. I’ve noticed,however, that even topendAtom-based thin-clientdevices tend to bog downin fat-client mode. If you’regoing to depend on your clients to run thecomplete OS, it’s important to test themfirst to make sure they’re up to snuff.Because you may have a combinationof new and old client machines, it isalso possible to mix and match the fatclientand thin-client model from thesame server. It means creating two NBDimages, but assigning the proper image isas simple as specifying a new filename inyour dhcpd.conf file. This is a nice way totake advantage of a few really powerfulworkstations without making your slowerhardware obsolete.All Things Are Possible, Not All ThingsAre WiseMy last few articles have torn theLTSP system apart and looked at itsjuicy insides. Perhaps that’s a gross46 / MAY 2012 / WWW.LINUXJOURNAL.COM

COLUMNSTHE OPEN-SOURCE CLASSROOMTable 1. Scaling Models, the Pros and ConsMODEL DESCRIPTION BEST FOR ADVANTAGES DISADVANTAGESClassroom ServerWorkstation classmachines with extraEthernet card server, 3–8clients per classroom.Environments with poornetwork infrastructureand/or lack of big servers.Minimal server costs,utilizes existingworkstations as miniservers,keeps thin-clientbandwidth off mainnetwork.Classroom thin clientsisolated from each other,potential for double-NAT problems, moreservers means moreconfigurations to manage.Single EthernetCentralized servers havea single Ethernet port,with a separate DHCPserver allocating clientsto servers.Large networkinstallations withsufficient networkinfrastructure.Troubleshooting is simple,clients can be reassignedby modifying dhcpd.conffile, configuration is similarto standalone model.Manual load balancing iscumbersome and oftenincorrectly done, multipleservers to configuremeans repeating the sameadmin tasks over andover, no automatic failoverif server goes down.LTSP ClusterCentralized NBD serverload balances over anynumber of applicationservers.Large installationswith sufficient networkinfrastructure andexperienced LTSPadministrator.Clustering allows forcentral configuration of allclients and single chrootfor quick updates.Web interface slightlycumbersome, layersof complexity maketroubleshooting moredifficult, NBD serverfailure can take downentire network.LTSP Fat ClientPowerful clientmachines load entireoperating systemover NBD, running allapplications locally.Powerful clientmachines withsufficient networkinfrastructure.Client machineshandle all computing,so server requirementsare minimal.Requires quite powerfulclient machines and solidnetwork infrastructure.Standalone hard driveinstalls should beconsidered.metaphor, but hopefully, it’s a littlemore clear how powerful thin clientscan be for your organization. Myfavorite part is how many differentways LTSP can be implemented. Ifyou’re just starting the process, a fewclassroom labs are the perfect way tosee if LTSP will work for you. In fact,by recycling older workstations andusing them as thin clients, the cost ofimplementation is often zero. It justtakes a few adventurous users willingto learn a new way to compute.As the system administrator, you haveseveral ways to set up your LTSP system.Table 1 gives some pros and cons ofeach scaling method discussed here,but if you’re just starting, I recommendgoing with a single server and a handfulof clients. Once you get to the point ofscaling, you’ll have a much better idea ofwhat fits into your environment best. If youhave questions along the way, feel free todrop me an e-mail or ask the professionalsin the IRC #ltsp channel on Freenode.■Shawn Powers is the Associate Editor for Linux Journal. He’s alsothe Gadget Guy for LinuxJournal.com, and he has an interestingcollection of vintage Garfield coffee mugs. Don’t let his silly hairdofool you, he’s a pretty ordinary guy and can be reached via e-mailat shawn@linuxjournal.com. Or, swing by the #linuxjournal IRCchannel on Freenode.net.WWW.LINUXJOURNAL.COM / MAY 2012 / 47

NEW PRODUCTSLogic Supply’s LGX Extended TemperatureAU115 Atom SystemThe key selling points for Logic Supply’s LGXExtended Temperature AU115 Atom Systeminclude the fanless system’s enhancedvibration and thermal tolerances and itsvery compact footprint. The AU115 system,says Logic Supply, offers reliable computingin an operating temperature environmentof –20°C to 60°C and is easily mountable, making it ideal for deployment in remotemonitoring, outdoor kiosks, data acquisition and networking applications. With no fans orvent openings, the AU115 is protected from exposure to the dust, dirt and grime typicallyfound on factory floors or in mobile environments. The device also conforms to theiMIL-STD-810F standards for vibration and shock, and it can be configured with a solid-statedevice for storage to extend the lifetime of the system and ensure data integrity.http://www.logicsupply.com/au115

NEW PRODUCTSAdaCore’s GNAT Pro, CodePeerAdaCore recently announced two major product upgrades—GNAT Pro7.0 and CodePeer 2.1—both of which support the upcoming Ada 2012language standard. GNAT Pro is AdaCore’s full-featured, multilanguage,flagship development environment complete with libraries, bindingsand a range of supplementary tools. Besides the Ada 2012 support,GNAT Pro’s key new features include the new GNATtest automatictest framework and numerous enhancements to several existing tools.Meanwhile, CodePeer is AdaCore’s advanced static analysis tool thathelps developers detect potential runtime and logic errors in Adaprograms. By mathematically analyzing every line of software andconsidering every possible input and every path through the program,CodePeer can be used early in the development life cycle to identifydefects when they are much less costly to repair.http://www.adacore.comRoderick W. Smith’s Linux Essentials (Sybex)It’s a positive sign for our revolution to see thecollection of basic Linux books continuing to expandand diversify. LJ writer Roderick W. Smith’s newtitle Linux Essentials, published by Sybex, adds aunique twist by adding full-color images to themix of resources that aid readers in understandingthe essentials of Linux. Linux Essentials featuresa learn-by-doing approach that clearly identifiesthe concepts to be covered and explores them in ahands-on tutorial format. Topics include installation,desktop configuration, management of files andfilesystems, remote administration, security andmore. Fedora is the distribution discussed in thebook. Linux Essentials is a great way to share the giftof Linux with our new, less-technical friends and family members.http://www.sybex.comWWW.LINUXJOURNAL.COM / MAY 2012 / 49

NEW PRODUCTSJames A. Whittaker, Jason Arbonand Jeff Carollo’s How Google TestsSoftware (Addison-Wesley)Things done at “Google scale” are mind boggling. Hundreds ofmillions of lines of code distributed across millions of source files.Billions of build actions prompting millions of automated tests daily.Hundreds of thousands of browser instances daily. Apps releasedcontinuously into production for massive worldwide use. Folks, software testing doesn’tget any tougher than this. To do all of this right, Google is pioneering the future of testingand automation. Enter the new book How Google Tests Software from author trio JamesA. Whittaker, Jason Arbon and Jeff Carollo, which seeks to offer mere mortals the chanceto learn from the Google experience. The authors distill wisdom from legendary GoogleTesting leader James Whittaker and other top Google experts to reveal exactly how theInternet giant tests software and offer new best practices anyone can use. Some of thesepractices include techniques for analyzing risk and planning tests; implementing exploratory,black-box, white-box and acceptance testing; testing “Docs & Mocks”; and much more.http://www.informit.comThe Portland Group’s PGI 2012 Compilers andDevelopment ToolsThe Portland Group says that its new PGI 2012 line ofparallelizing compilers and development tools helpsdevelopers leverage the huge performance potential of GPUsbetter than ever before. PGI 2012, which supports Linux,OS X and Windows, is the Portland Group’s first general release to include support forthe OpenACC directive-based programming model for NVIDIA CUDA-enabled GPUs.The updated PGI Accelerator FORTRAN and C compilers are targeted at scientistsand engineers who are not full-time programmers, freeing developers to focus onoptimizing their algorithms. The 2012 release is also the company’s first to include thefully feature-enabled PGI CUDA C/C++ compiler for multicore x64 CPUs from Intel andAMD. Finally, PGI 2012 includes a number of performance and feature enhancementsfor multicore x64 processor-based HPC systems.http://www.pgroup.com50 / MAY 2012 / WWW.LINUXJOURNAL.COM

NEW PRODUCTSCodeWeavers’ CrossOver XIThe only thing not simple about CrossOver XI is “how wepronounce the name of the product”, said CodeWeavers’ever irreverent CEO, Jeremy White, recently about hiscompany’s new complete package for running Windowsbasedapps on Linux or Mac. The boss is unsure whether itscustomers will pronounce the name “CrossOver Ex-Eye”, or“CrossOver Eleven”, “CrossOver Exy”, or his own favorite,“CrossOver Zigh”. Regardless of what you call it, CrossOverXI combines all previous versions of CodeWeavers’ CrossOversoftware into one simple download, giving “everyone in our universe immediate accessto thousands of Windows applications otherwise unavailable without a Microsoftlicense”, added White. Applications, which number in the thousands, range fromWizard 101, to Microsoft Office 2010 to nostalgia-oozing WordPerfect 3.1.http://www.codeweavers.comLinux JournaLon youre-ReaderCustomizedKindle and Nookeditionsnow availablee-ReadereditionsFREEfor SubscribersLEARN MORE

NEW PRODUCTSInductive Automation’s IgnitionDeploy the new Ignition 7.4, says maker Inductive Automation,and you’ll feel as if you had an extra pair of hands helpingyou knock out your HMI/SCADA/MES development projects.Inductive Automation bills the Java/Web-based Ignition asthe “simplest and most innovative software for HMI, SCADAand MES applications in the automation industry”. Ignition connects to nearly any majorSQL database or operating system and features a built-in OPC-UA server with drivers forthe most common PLC brands. New in v7.4 are innovations, such as dynamic componenttemplates, user-defined types, scripting with Python 2.5 and project templates. All of theabove, says Inductive Automation, carve out repetitive development tasks, leaving moretime for innovating, improving testing and securing projects.http://www.inductiveautomation.comSoftIntegration’s Ch 7.0 and Embedded ChSoftIntegration notes that an instructorusing its Ch C/C++ interpreter, now inversion 7.0, has called it “as close to aperfect teaching environment” he has everseen. A programmer customer said usingCh is “the fastest way to write and deployembeddable C/C++ scripting programs acrossplatforms”. Ch 7.0 and Embedded Ch 7.0are an embeddable C/C++ interpreter forcross-platform scripting, 2-D/3-D plotting,numerical computing, shell programming and embedded scripting. The 7.0 release addsnew features to make Ch especially appealing for secondary school and college studentsto learn computing and math. These include many new math functions and features forplotting, QuickAnimation, ChIDE and Embedded Ch. Embedded Ch allows users to embedCh into an C/C++ application, such as game, semiconductor ATE, SoC, CAM, IC, PCB, RF,MEMS or LED as a scripting engine. Commercial and academic licenses are available.http://www.softintegration.comPlease send information about releases of Linux-related products to newproducts@linuxjournal.com orNew Products c/o Linux Journal, PO Box 980985, Houston, TX 77098. Submissions are edited for length and content.52 / MAY 2012 / WWW.LINUXJOURNAL.COM

2012Back for 2012, USENIX is combining establishedconferences and workshops into a week of research,trends, and community interaction. Events include://///USENIX ATC ’122012 USENIX Annual Technical ConferenceWednesday–Friday, June 13–15 www.usenix.org/atc12WebApps ’12 /////3rd USENIX Conference on Web Application DevelopmentWednesday, June 13 www.usenix.org/webapps12/////HotCloud ’124th USENIX Workshop on Hot Topics in Cloud ComputingTuesday–Wednesday, June 12–13 www.usenix.org/hotcloud12HotStorage ’12 /////4th USENIX Workshop on Hot Topics in Storage and File SystemsWednesday–Thursday, June 13–14 www.usenix.org/hotstorage12/////TaPP ’124th USENIX Workshop on the Theory and Practice of ProvenanceThursday–Friday, June 14–15 www.usenix.org/tapp12/////Why AttendYou’ll leave the week with insights into the future of many hot topics.Your access to leading researchers and industry experts will not onlyprovide answers to your most burning questions, but will also help youbuild lasting connections across multiple disciplines. Take full advantageof this opportunity and learn from the top researchers, practitioners, andauthors all in one place, at one time.Networked Systems for Developing RegionsUSENIX Federated Conferences Weekwww.usenix.org/fcw12June 12–15, 2012 Boston, MAWeb ApplicNetworked Systems for Developing RegionStorage ProvenanceSystems Research Cloud ComputingWeb Application DevelopmentProvenance Systems ResearchCloud Computing StorageEArly BIrdNetworked dISCoUNT Systems for DeveloprEgISTEr byMAy 21 &NSdr ’12 /////6th USENIX/ACM Workshop on Networked Systems for Developing RegionsFriday, June 15 www.usenix.org/nsdr12Stay Connected...www.usenix.orghttp://www.usenix.org/facebookhttp://twitter.com/usenixhttp://www.usenix.org/linkedinhttp://blogs.usenix.org

REVIEWThe ConstructionGenerally, if you want a case that aregular human can change parts on,you build your own machine. That’smore of a guideline than a rule, but itseems most big computer companieslike to make their cases as proprietaryas possible. Sure, you might be ableto fit your computer system into asize 4 shoebox, but good luck if youever want to upgrade your hardware.ZaReason, on the other hand, includesa standard CoolerMaster brand mid-sidetower case. I use the term “mid-size”rather sheepishly, as it’s the quite large“G-Lite 430 Black” (Figure 1).ZaReason has nicely branded the casewith its company name, but otherwiseleft it the standard CoolerMaster unit—refreshing, no? To be fair, the case itselfis a bit more flexible than I’d like, but itdoesn’t feel cheap by any means. Partof its flexibility is due to the massiveamount of ventilation it boasts (Figure2). Because the X79 is meant to be apowerhouse, the cooling considerationis greatly appreciated. In fact, it comeswith an additional case fan installed onthe side to help keep the beast withinon the cool side.On the front panel, the case sports apower button and reset button, as onewould expect. It also has the following:Figure 2. This ventilation is appropriate, but itdoes make the case feel a bit flimsy.n Headphone and microphone jacks (analog).The back of the case provides quitea generous set of connections (Figure3), including:n Six USB 2.0 ports.n Two USB 3.0 ports (clearly marked).n One PS/2 port.n Gigabit Ethernet port.n 8 channel, 7.1 analog jacks.n Coaxial SPDIF-out.n SPDIF-optical out.n Two USB 2.0 ports.n One 1394 port (400Mbit).WWW.LINUXJOURNAL.COM / MAY 2012 / 55

REVIEWn 16GB DDR3 RAM @1600MHz (4x4GB).Figure 3. The ports are easy to access, and theexternal CMOS reset button is very convenient.n External CMOS reset button.Of course, all those ports really tell thestory of the motherboard more than thecase, so I cover that next.The Mother (Board)The specifications on the Valta X79don’t specify a brand name for themotherboard, so I won’t focus on thename printed on the board, but ratherthe features it boasts. First off, in orderto handle the 3930K second-generationSandy Bridge i7 CPU, the motherboardhas the Intel X79 chipset and LGA-2011socket. (More on the CPU in a bit.)Adding to the CPU itself, the X79 has:Another pleasant surprise was thepower supply included with the ValtaX79. Because this build included a sizableGPU along with the zippy CPU, ZaReasoninstalled an 850 watt PSU. Although thewattage is about what I’d expect, I washappy to see the power supply was an 80Plus Gold-rated unit. Often when buildinga machine for brute horsepower, littlethought is given to efficiency. Thankfully,that’s not the case here. 80 Plus Goldcertification means the power supply is87% efficient at 20% load, 90% efficientat 50% load, and 87% efficient at 100%load. As it happens, when this system isrunning full bore with both the GPU andCPU maxed out, it’s using almost exactly425 watts. Whether it was a coincidenceor not, it means the PSU was sizedexactly right for maximum efficiency. Wellplayed, ZaReason.n Three PCI-E 3.0 x16 slots.n Four PCI-E x1 slots.n Four DDR3 Quad-Channel RAM slots.Figure 4. The interior is roomy and cabling neat,allowing for maximum airflow.56 / MAY 2012 / WWW.LINUXJOURNAL.COM

REVIEWSoftwareLinux. ’Nuff said (Figure 5).Seriously though, ZaReason is acompany that sells Linux computers.When configuring your system, youget to choose from the most commondistributions. By name, they offer all the’buntus, Debian, Fedora, Linux Mint, nooperating system and my favorite, “tellus what kind of Linux you want.”Earl, being a Linux Journal readerhimself, knew my feelings regardingUbuntu’s Unity interface. For the reviewunit, ZaReason installed Ubuntu 11.10,but instead of the default Unity interface,they installed GNOME 3 with extensionsto look much like my beloved GNOME 2(Figure 6). You may think the review unitgot special treatment, but Earl actuallytold me this interface is what they shipto many customers, especially thoseuncomfortable with Unity.but basically, it’s a form of virtualcurrency. That currency is createdand secured by a large network offolks donating processing power tocryptographically verify transactions.In return for that donated processingpower (aka “mining”), virtual coinageis produced and distributed to theminers. This both introduces thecurrency fairly into the economyand gives people an incentive fordonating their processing power.n Hashrate: this is the speed at whichWe’ve Secretly Replaced Our BenchmarkingTools with Cryptocurrency!It’s easy to look up benchmarks onspecific CPUs and GPUs. BecauseZaReason sent me this unit after readingmy article on cryptocurrency, it seemsonly fair to benchmark it using thatmethod. Before I delve into hashratesand profits/losses, however, let me definea few terms for those not familiar withcryptocurrencies:n Mining: cryptocurrency is somethingI’ve written about in past months,WWW.LINUXJOURNAL.COM / MAY 2012 / 59

REVIEWThe second-generation i7 processor, with itsnew AVX instruction set, is the fastest Litecoinminer I’ve ever seen.miners can verify transactions, or moreprecisely, the speed at which they cansolve the math problems that verifytransactions. The faster the processor,the higher the hashrate.n Bitcoin: the most widely acceptedand traded cryptocurrency. Thealgorithm used to verify transactionsis SHA256 and is most efficientlymined with GPUs, specificallyAMD/ATI-brand GPUs.n Litecoin: created to be the silver toBitcoin’s gold, Litecoin uses the Scryptalgorithm with the intent of beingmore efficient to mine with a CPU.Scrypt is more memory-intensive thanSHA256, so CPUs have an easier timeverifying the transactions.Another important thing tounderstand about cryptocurrencyis that the speculation market forindividual currencies is extremelyvolatile. Bitcoins have traded for aslittle as fractions of a penny each,all the way to more than $30 each.Litecoins peaked at about a nickel,and at the time of this writing, theyare selling for about a half-cent each.Cryptocurrencies are fun to play with,but I certainly don’t recommend investingmore than you can afford to lose.Torture Testing, for Profit!Let me start with the graphics card.First, the AMD 6970 is an incredibly fastgaming card. I know that’s not what I’mreviewing, but it’s worth noting I couldn’tget this thing to drop a frame no matterhow hard I tried.When it comes to Bitcoin mining,the 6970 is no slouch either. Runningat stock clockrates and voltages, Iwas able to mine Bitcoins at around375 million hashes per second, orMH/s. Using my Kill-A-Watt meter, Imeasured the difference in wattage atabout 225 watts versus the idle GPU.Using the profitability calculator athttp://allchains.info, given the currentBitcoin difficulty and trading price, itturns out I can mine about $1.10 worthof Bitcoins every day. Keep in mind,however, that my electricity costs about$0.11 per kilowatt hour, so while Imight make $1.10 in Bitcoins, it costsme $0.59 in electricity. At current rates(end of March 2012), that means this60 / MAY 2012 / WWW.LINUXJOURNAL.COM

REVIEWcomputer will make about $0.51 perday in profits, along with quite a bit ofnoise and heat.The second-generation i7 processor,with its new AVX instruction set, isthe fastest Litecoin miner I’ve everseen. Running all six cores (12 withhyperthreading), the Core i7-3930Kuses right around 200 watts accordingto my Kill-A-Watt. That’s actually alittle lower than some benchmarksI’ve seen, but nonetheless, it’s whatI recorded. The impressive part isthat the CPU was able to sustainabout 72 thousand hashes per second(KH/s), which is more than twiceas fast as my AMD 6-core CPU canmuster. Using the same profitabilitycalculator from allchains.info, this CPUcan mine around 42 Litecoins a day.Since the price of Litecoins is so low,that equates to only around $0.23.Unfortunately, at 200 watts, it costsaround $0.53 per day to mine, so if Iwere to mine Litecoins today, I’d losearound $0.30 a day.That is where the speculation comesin. Just a few short weeks ago, Litecoinswere selling for 3–4 cents each, whichmeans it was profitable to mine.Depending on your cost for electricity,perhaps it never will be profitable tomine Bitcoins or Litecoins. Again, Istress not to spend more money thanyou can afford to lose. Still, if you’reinto cryptocurrencies, it’s a neat way tostress-test a computer!ConclusionThe Valta X79 is a screaming-fast computer.To be honest, if I were going to build apowerhouse computer from scratch, it’s theexact type of system I would build. I thinkthat’s what I like best about it. ZaReasonhas taken the essence of building your owncomputer, using standard parts, and donethe testing to make sure everything workswell together. Then ZaReason assembles it,configures it and, most important, supportsit. If you’ve ever wanted to build a customsystem, but wished you didn’t have to giveup a warranty and tech support, you’lllove this Linux beast from ZaReason.The price for the unit as reviewed is$2,396. To configure and price yourown custom Valta X79, head overto http://www.zareason.com.■Shawn Powers is the Associate Editor for Linux Journal. He’s alsothe Gadget Guy for LinuxJournal.com, and he has an interestingcollection of vintage Garfield coffee mugs. Don’t let his silly hairdofool you, he’s a pretty ordinary guy and can be reached via e-mailat shawn@linuxjournal.com. Or, swing by the #linuxjournal IRCchannel on Freenode.net.ResourcesZaReason: http://www.zareason.comBitcoin: http://www.bitcoin.orgLitecoin: http://www.litecoin.orgMarket Price and Profit Calculator:http://www.allchains.infoWWW.LINUXJOURNAL.COM / MAY 2012 / 61

Object-OrientedProgramming withLuaIn spite of being a multiparadigm programming language, Luaprovides a nontrivial object-oriented programming model thatmay prove cumbersome to programmers who want to applyOO software engineering practices to development. This articlepresents a reusable mechanism through which you can implementan object-oriented model using Lua’s built-in constructs.ALEJANDRO SEGOVIALua is a dynamic scripting languageand has been developed at the PUC-RioUniversity in Brazil since 1993. In itscurrent 5.2 version, it is lightweightand portable, and it has become verysuccessful for both open-source andcommercial software projects, particularlyin the field of video games, where it isbeing used by companies like EA.Lua also is becoming increasinglyimportant for mobile device softwaredevelopment and recently has beenchosen by Wikipedia as its new scriptinglanguage. I first became interested in Luawhen I learned about how easy it was toembed into C and C++ applications.But, enough introduction—let’s startcreating some objects!62 / MAY 2012 / WWW.LINUXJOURNAL.COM

FEATURE Object-Oriented Programming with LuaLet’s provide the points the abilityto pretty-print themselves to thestandard output.First, give Points a print method:function Point.print(p)print("("..p.x..","..p.y..")")endNow, you can print points like so:p1 = Point.new({x=10, y=10})Point.print(p1) --prints: "(10,10)"In a similar fashion, you can adda range of methods like translating,scaling and rotating points. The onlylimitation is that you always will haveto call these methods as shown above:prefixing every invocation with the classname and explicitly supplying the pointon which to operate.It is natural to ask, “can you improvethis syntax?” It turns out, you can byusing Lua’s colon operator (:).The colon operator can be used ina statement like p1:print(), and itwill be expanded to p1.print(p1)automatically. Just like there are twosemantically equivalent options foradding methods to your classes, thesetwo expressions are semanticallyequivalent as well.Now, consider this: if you couldhave Lua associate whatever name isat the left of the colon operator tothe object class, you would be able tosimulate message passing to objects.This would allow using the followingsyntax with your objects:p1:print() --prints p1p2:print() --prints p2Thus far, however, you can’tassociate p1 and p2 with Point.When p1:print() is expanded top1.print(p1), the Lua interpreterwill print an error message statingthat "p1" has no attributecalled "print".Who knows about “print”? Well,“Point” does. What you need to do istell Lua that when it fails to find a givenattribute or method in “p1”, it has tocontinue searching in “Point”. Thisassociation can be declared using theconcept of metatables.Assuming “p1” is a point like before,this snippet will set everything up sothe Lua interpreter continues searchingfor attributes and methods missing in“p1” in your Point class:setmetatable(p1, Point)Point._ _index = PointAfter this little change, thefollowing code will start to behavethe way you want:p1.print(p1) --prints p1But more important, its semantically64 / MAY 2012 / WWW.LINUXJOURNAL.COM

Although Lua doesn’t provide an out-of-the-boxobject model, it turns out that implementingPolymorphism in Lua is very easy due to thedynamic nature of the language.equivalent sibling will work too:p1:print() --prints p1endself._ _index = selfreturn pWhat sadly will not work is trying todo this on “p2”, because “p2” has nometatable. Therefore, you must set itsmetatable to “Point” as well. And, notjust for “p2”, but also for every singlePoint instance you create.Although manually setting themetatable for every Point instancedefinitely is an option, it might proveto be cumbersome and error-prone.Furthermore, because this is an operationthat has to be performed on every pointinstance, why not do it in the Pointconstructor? Let’s do that.This is the updated Point constructor.It first will check whether an object hasbeen supplied and will create a defaultone if it hasn’t. Then, it will set theobject’s metatable to be Point, makingunrecognized messages be rerouted tothe Point class for every point created:Note: I am passing the metatable as thefirst argument of the Constructor (andcalling it “self”). This not only will allowyou to create new points using the colonoperator (as shown in the next example),but it also lets you do some advancedobject-oriented tricks that I’ll discuss later.With this updated constructor, thefollowing is now valid:p1 = Point:new({x=10, y=10})p2 = Point:new({x=20, y=20})That’s it as far as declaring,instantiating and using point objectsgoes. Try running this example usingthe Lua interpreter, and make sure youunderstand the concepts. I will buildupon them in the next sections toshow how to implement Polymorphismand Inheritance.function Point.new(self, o)local p = o or {x=0, y=0}setmetatable(p, self)PolymorphismIn the previous section, I described amechanism by which you can model aWWW.LINUXJOURNAL.COM / MAY 2012 / 65

FEATURE Object-Oriented Programming with Luaclass “Point” by means of Lua tables. Inthis section, let’s build on the previousconcepts to support Polymorphism withyour objects.Polymorphism is one of the mostpowerful concepts in object-orientedprogramming. It allows the programmerto change the behavior of a systemdynamically based on the type of theobjects to which you send your messages.Although Lua doesn’t provide anout-of-the-box object model, it turnsout that implementing Polymorphismin Lua is very easy due to the dynamicnature of the language.Suppose you have a global function“printpoints()”, which receives a listof points and has them printed to thestandard output. Furthermore, supposeyou had two different point types in yoursystem: Point, which models 2-D pointsjust like you have been using so far, andPoint3D, a new class that represents 3-Dpoints that live in 3-D space.For this example, let’s imagine thatthe “printpoints()” function will berequired to handle mixed lists of 2-Dand 3-D objects.Well, if you can have both pointtypes present in a single list and youhave a statically typed programmingbackground, you already might bethinking that you need both points toascribe to a common interface thatallows them to be printed. The term“IPrintable” may even come to mind.Indeed, what you require is for bothpoints to share a set of messages theyrespond to. In other words, you needboth types to agree to a contract that“certifies” that they can be “printed”.In statically typed programminglanguages, this contract can be expressedby means of Interfaces. In Lua, however,such a construct is not necessary, as youcan use the dynamic foundation of thelanguage to ask, at runtime, whether anobject responds to any given message.In this case, the message wouldbe “print”.This concept of objects conforming toa common Interface that is not declaredanywhere explicitly is known as anImplicit Protocol. Both point types willabide to the Implicit Protocol of knowinghow to print themselves.Let’s see how this “printpoints()”function could go:function printpoints(points)for _, p in ipairs(points) doif p.print thenp:print()endendendHere, you receive the list of points toprint, and independently of what thelist contains, iterate over its elements.For each element, you ask whether itresponds to the message “print”. If itdoes, you send it the “print” message.Let’s use the interactive interpreter66 / MAY 2012 / WWW.LINUXJOURNAL.COM

Unlike Polymorphism, when modeling typeInheritance, it is not enough to have theobjects ascribe to the same Implicit Protocoland rely on the dynamism of Lua.to see how this function behaves whenhandling a list consisting of two 2-DPoints and one 3-D Point:$ luaLua 5.1.4 Copyright (C) 1994-2008 Lua.org, PUC-Rio> require "point"> points = { Point:new({x=10,y=10}),Point:new({x=20,y=20}),➥Point3D:new({x=30,y=30,z=1}) }--The "points" list contains 2D and 3D points.> printpoints(points)(10,10)(20,20)(30,30,1) -- This is a 3D point!Note how the points in the list wereable to respond to the “print” message,each using its corresponding behavior,meaning 2-D points printed their (x,y)pairs, whereas the single 3-D pointcorrectly printed its (x,y,z) tuple. This is,effectively, Polymorphism in Lua.If you are feeling uneasy about theconcept of Interfaces, remember thatthe true meaning of Polymorphism isnot in implementing Interfaces, butrather in having objects dynamicallyrespond to the messages receivedaccording to their type. Dynamicallytyped programming languages are,therefore, an ideal fit for Polymorphism.InheritanceThe last major concept common toobject-oriented programming that I’mgoing to tackle here is Inheritance.How can you add an Inheritancemechanism to the object-orientedmodel I’ve been discussing?Unlike Polymorphism, when modelingtype Inheritance, it is not enough to havethe objects ascribe to the same ImplicitProtocol and rely on the dynamism ofLua. In order to implement Inheritance,you need to describe a relationshipeffectively between the types of the basedata type (the base class) and the deriveddata type (the derived class).What you need to do is be able to“extend” an object type with a newer,“more concrete” type that addsspecific logic.The way to achieve this using LuaWWW.LINUXJOURNAL.COM / MAY 2012 / 67

FEATURE Object-Oriented Programming with Luamight seem strange at first, but if youbear with me, you will see it naturallymatches the concepts used here thus far.To continue with this example,let’s add to the system the ability torepresent 3-D points in homogeneouscoordinates. In homogeneouscoordinates, each 3-D point isrepresented using four values (x,y,z,w).When converting from Cartesiancoordinates to homogeneous coordinates,you just need to set the w value to 1. Toconvert from homogeneous coordinatesback to Cartesian coordinates, you needto divide all components of the point bythe w value, therefore taking (x,y,z) backto Cartesian coordinates.The 3-D point (1,1,1) in homogeneouscoordinates would be (1,1,1,1), as well as(2,2,2,2), (3,3,3,3) and so on. If you arenot convinced, try dividing by w in eachcase and see what happens.Points in 4-D homogeneouscoordinates are 3-D points,meaning, Points in homogeneouscoordinates (PointH) have an “is a”relationship with class Point3D. Thisis an Inheritance relationship you canrepresent in the object model.Assuming class Point3D exists and issimilar to Point, you start by stating thatPointH (4-D homogeneous coordinatepoints) are 3-D Points:PointH = Point3D:new({x=0, y=0, z=0, w=1})What I did here, conceptually, wasto declare PointH to be a new classthat “Inherits” all the behavior fromPoint3D. In particular, its metatablewill point to Point3D.What’s interesting is the fact that whenyou create a new PointH instance called“ph”, by using the PointH:new() method,Point3D’s constructor will be called, butwith the “self” object pointing to (table)PointH instead. Therefore, the “ph”instance will have its metatable pointingto PointH and not Point3D.This achieves a “chain” of metatables.PointH instances will have PointH as theirmetatable, whereas class PointH will havePoint3D as its metatable. This chain ofmetatables allows the Lua interpreterto conduct the following searchesautomatically when a message is sent toa PointH instance:n 1) First, search in the instance itself.n 1.1) If the method is present, call it.n 1.2) Otherwise, search in the PointH class.n 1.2.1) If the method is present, call it.n 1.2.2) Otherwise, search Point3D class.n 1.2.2.1) If present, call it.n 1.2.2.2) Otherwise, fail.In this example, if you send “ph” a“print” message, “ph” doesn’t know68 / MAY 2012 / WWW.LINUXJOURNAL.COM

“print”, so it delegates the messageto its metatable: PointH. PointHdoesn’t know how to “print” either,so it delegates the message to its ownmetatable: Point3D. Point3D knows howto “print”, so its method is used.Now, suppose you taught PointH howto interpret the “print” message bydefining the function PointH.print(p).In that case, PointH’s print would havebeen used instead of continuing thesearch in Point3D.What this means is that yourInheritance model effectively supports“overriding” a base class’ methods.This is how PointH’s completeimplementation would look:Here, Point3D’s print is overridden withPointH’s, meaning that when calling printon a PointH instance, the w value will beprinted along the (x,y,z) tuple.ConclusionThis article described one of themechanisms through which object-orientedprogramming can be implemented in theLua programming language. Polymorphismand Inheritance, two of the main pillarsupon which object-oriented programmingis based, was discussed as well.Hopefully, you will be able to usethese ideas, or build on top of them,to bring your own object-orientedsoftware designs to Lua.■PointH = Point3D:new({x=0, y=0, z=0, w=1})function PointH.print(p)print("("..p.x..","..p.y..","..p.z..","..p.w..")")endAlejandro Segovia Azapian is a computer engineer fromUruguay. He specializes in computer graphics, and he hasconducted several undergraduate courses at the UruguayanCatholic University on the topic. At the time of this writing, heis working as a software engineer for Autodesk, Inc.ResourcesProgramming in Lua: http://www.lua.org/pil/index.htmlLua: http://www.lua.org“Why ‘Lua’ is on everybody’s lips, and when to expect MediaWiki 1.19”: http://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2012-01-30/Technology_reportLua (Programming Language)—Applications (Wikipedia): http://en.wikipedia.org/wiki/Lua_%28programming_language%29#ApplicationsLua-Scripted Video Games (Wikipedia): http://en.wikipedia.org/wiki/Category:Lua-scripted_video_gamesWWW.LINUXJOURNAL.COM / MAY 2012 / 69

AndroidProgrammingwithAPP INVENTORDrag and drop your wayto Android programming.AMIT SAHAMIT App Inventor, re-releasedas a beta service (as of March5, 2012) by the MIT Centerfor Mobile Learning after taking overthe project from Google, is a visualprogramming language for developingapplications for the Android mobilecomputing platform. It is based on theconcept of blocks, and applicationsare designed by fitting together blocksof code snippets. This may sound likea very childish way of programming,especially for seasoned readers of LinuxJournal. But then again, App Inventorwill tickle the child programmer in youand make you chuckle at the ease withwhich you can develop applicationsfor your Android device. In this article,I describe how to use the camera onthe Android device, develop e-mail andtext-messaging-based applications andalso show how to use location sensorsto retrieve your current geographicallocation. Let’s get started.70 / MAY 2012 / WWW.LINUXJOURNAL.COM

GETTING STARTEDApp Inventor has minimum setuprequirements and is completely browserbased.You need a working Java installationon your system, as it uses Java Web Startfor its functioning. Point your browser tohttp://appinventor.mit.edu, and onceyou sign in with your Google account,you should see a screen as shown inFigure 1. This is called the Projects Pagewhere you can see your existing projectsand create new ones.Now, let’s develop and deployan Android application using AppInventor and in the process learn thebasic development-deployment cycle.Create a New Project using the NewProject button, and enter a name foryour project, say “Project1”. Now youshould see the Designer window foryour project. The Designer window iscomposed of four sub-components.The Palette on the leftmost side of thewindow is the placeholder for all theavailable components for your project.The Viewer is where the application willbe designed by placing together variouscomponents (this is where you designthe user interface for your application).The Components show the currently usedcomponents in your project, and theProperties column is where you assignthe properties of the components.First, let me briefly explain the notionof components. An App Inventor projectis made up of building blocks calledcomponents, such as a text label todisplay text, a text box to take userinputs, a camera component to clickpictures and so on. Currently, you willsee a few categories of components—basic components, such as those foruser input and display of text to morespecialized components, such as thosefor displaying media and animations, andcomponents acting as an interface tothe device sensors. A complete referencefor all the components is available athttp://appinventor.mit.edu/learn/reference/index.html. Componentshave associated behavior, methods andproperties. Some of the properties can beset; whereas others can be only read.In this first project, let’s use thefollowing components: Camera, Buttonand Image. The code usually shows itbetter, but briefly here is what you’regoing to do: clicking the button startsthe camera on your device, which youuse to click a picture, which then isFigure 1. App Inventor’s Projects PageWWW.LINUXJOURNAL.COM / MAY 2012 / 71

FEATURE Android Programming with App InventorFigure 2. User Interface for Project1displayed using the Image component.Here are the steps:1. Drag a Camera component fromthe palette to the Viewer. Itshould show up under Non-visiblecomponents below the Viewer.By default, it will be named asCamera1, which you can, of course,change to something else.2. Drag a Button to the Viewer, andfrom the Properties, change its Textto “Click”.3. Drag an Image component ontothe Viewer.4. You can play around with the Screenproperties to set things like title,background color and orientation.For the purpose of this project, setthe Title to “Click!”.That completes the design of the userinterface (Figure 2). Next, let’s programthe components using Blocks.Open the Blocks Editor, which shouldstart downloading the JAR file for theeditor. It will ask you for the locationof the App Inventor setup commandsif you have not installed them in thestandard location under /usr/google.The Blocks Editor for the current projectwill look like Figure 3. Going back tothe description for this project, thegoal is to activate the device camerawhen the button is clicked. This is donewith the code block “When Button1.click”, which you dragged from the72 / MAY 2012 / WWW.LINUXJOURNAL.COM

Figure 3. Blocks Editor for Project1Blocks pane on the left. When thebutton is clicked, you want the device’scamera to be activated, so drag the“call Camera1.TakePicture” block insidethe previous block. Once the picture istaken, you will want it to be displayedusing the Image component. So, insertthe block “when Camera1.AfterPicture”into the editor, and then set the“Image1.Picture” to the location of thesaved image.Now that you have designed theuser interface and programmed theapplication’s logic, you’re ready to testit. Go back to the Designer window,and on the right, click on Package forPhone→Download to this Computer.That should initiate the download ofthe Android package (.apk file) foryour project. Now, transfer this fileto your Android device, and install it.Then, try it out.A PEEK UNDERTHE HOODNow you have designedand deployed your firstAndroid application,and you have usedcomponents (the cameracomponent and theimage components),assigned them behaviorand set properties. Ifyou are familiar withthe idea of event-drivenprogramming, youalready will have realizedthat App Inventor is an event-drivenprogramming framework. The eventcan be the user clicking a button or thereception of a text message. For example,when the button is clicked, an event issaid to have occurred, and in responseto this event, the camera is activated.Again, when the camera finishescapturing a picture and saving it, theresponse code uses the image location todisplay it using an image component.Earlier, I mentioned that componentshave associated behavior, methodsand properties. You can find thesefor a component by clicking thecomponent in the Blocks Editor. Forexample, Figure 4 shows the methodavailable for the Camera component(Camera1.TakePicture) and the behavior(Camera1.AfterPicture).Besides the blocks associated withcomponents, more fundamentalWWW.LINUXJOURNAL.COM / MAY 2012 / 73

FEATURE Android Programming with App InventorFigure 4. Blocks Available for the Camera Componentprogramming blocks are available:Math blocks, Logic blocks, Controlblocks and others. (I’ll demonstrateusing a few of these in one of theprojects later in this article.)Now that you have a basic idea ofdeveloping applications using AppInventor, let’s look under the hood a bit,starting from the source. Download thesource code for “Project1” by going tothe Projects Page and selecting Project1and clicking on More Actions→DownloadSource. That should start downloadingthe sources in a zip file. When youunzip the file, you will have twodirectories: src and youngandroidproject.Under the src directory, you will have asubdirectory called appinventor, whichhouses the subdirectories, and thenai_droidery/Project1 (note that “droidery”is my Google user name). In this directory,you will see the source files Screen1.blk,Screen1.scm and Screen1.yail. Screen1.blkis an XML-based representation of thevisual blocks that was created earlier;Screen1.yail is an intermediate languagebased on the Scheme language usedby App Inventor, which is then fed toKawa to create the Android packagefor installation on Android devices. TheScreen1.scm file is a JSON representationof the components used in the projectwith details about the components,such as the version information. Ifyou are keen to understand how AppInventor really works, you also maywant to check out App Inventor’ssource code (see Resources).74 / MAY 2012 / WWW.LINUXJOURNAL.COM

FEATURE Android Programming with App InventorFigure 6. Final Blocks for the LocationOnClickEmail Projectand customized. In the process, you’lllearn to make use of a couple more AppInventor components.E-MAIL YOUR CURRENT LOCATIONConsider a not-so-fictional scenariowhen you might want to tell your friendexactly where you are at the moment sothat she can drive down to meet you.Or, you simply may be lost. Either way,the Location Sensor can help. Let’s callthis project “LocationOnClickEmail”.The user interface for this projectlooks like the one shown in Figure 5.Besides the basic components, suchas text labels and buttons, add theLocationSensor component (foundunder the Sensors category) and anActivityStarter component (foundunder the Other Stuff category). TheActivityStarter component, which hasbeen named “MailAppStarter” will beused to start the e-mail application onthe Android device. For details on theActivityStarter Component, refer tohttp://appinventor.mit.edu/learn/reference/other/activitystarter.html.Now you need to add the project logicusing the Blocks Editor as before. Figure6 shows the final state of the BlocksEditor for this project.The application logic can be dividedinto two steps—obtaining the locationusing the Location Sensor when the GetLocation button is clicked. This is done inthe “when GetLocationButton.Click”76 / MAY 2012 / WWW.LINUXJOURNAL.COM

code block. When this button is clicked,the Location Sensor is enabled. Once theLocation Sensor has been able to obtain thelocation information, it invokes the “whenLocationSensor1.LocationChanged” methodwhere the text labels are updated with thelocation data. Next, when the Email Locationbutton is clicked, the MailAppStartercomponent’s DataUri property is set tostart the mailing application. Here, therecipient is set to “droidery@gmail.com”,the subject to “My Location” and thebody of the message to the obtainedaddress. The recipient and subject canbe changed in the mailer applicationon the device.That completes the current project. Formore details on using the Location Sensorand the App Starter components, refer tothe App Inventor Reference (see Resources).TEXT-MESSAGING-BASED LOCATIONSENSOR APPLICATIONIn the last application, you initiatedthe location sending event. What ifyou want to design an application thatwill run as a service, such that whenit receives a request via text message,it sends your current location to thesender? Even with privacy being sucha sensitive issue in today’s connectedworld, such an application can beuseful if you want to make sure yournot-so-grown-up kid isn’t lost, forexample. In addition to the LocationSensor component, you will becomefamiliar with the Texting component inthis application.Here is the idea: on receipt of a textmessage with “location” in its body,the application replies with the currentFigure 7. Action Taken When a Text Message Is ReceivedWWW.LINUXJOURNAL.COM / MAY 2012 / 77

FEATURE Android Programming with App Inventorlocation as a text message. The actionstaken upon receipt of a text messageare shown in Figure 7. This is thecore logic for the application. In the“Texting1.MessageRecieved” procedure,the “number” and “messageText”are available as arguments. If the“messageText” is “location”, thencheck whether the location has beenobtained. If yes, then construct a replyusing the address and send the text;otherwise, send an error message backas the reply.The complete application, alongwith others, can be downloaded fromhttps://bitbucket.org/amitksaha/articles_code/src. You can upload thesource (.zip) archives to App Inventordirectly and try out the applications afterpackaging them. In this article, I havestrictly concentrated on using an Androiddevice for testing the applications. Forbasic uses, you also can use the emulatorthat is available in App Inventor and alsouse a live development methodologywhere you can install the applicationdirectly to your device. See the AppInventor Web site to try these out. Itested these applications on my SamsungGalaxy-SII running Android 2.3, but Ihope there won’t be any issues withrunning them on other devices runningAndroid 2.2 and higher.LOOKING AHEADI started this article with the intentionof having some fun programming forthe Android platform, and I hope it hasbeen so thus far. If you’re interestedin looking into App Inventor further,the first things that you might wantto check out, apart from extendingthe projects to something more funand useful, are the various othercomponents. Of special note is theData Store component that allowsyou to store data on the device, theWeb components for interacting withremote Web content, other Sensorcomponents and Media components.App Inventor is fun, but you mightfeel that although it’s good as astarting point, you would prefer a moretraditional programming language asyou become more familiar with Androiddevelopment. Instead of completelythrowing your App Inventor projectaway, consider using the App InventorJava Bridge to use your App Inventorcomponents while you write Androidapplications using the more traditionalway of programming in Java.If you feel the need to run your ownApp Inventor service, the MIT Center forMobile Learning has made available theApp Inventor JARs to enable you to hostyour own service (see Resources).If you want to keep exploring AppInventor itself, two excellent books areavailable: David Wolber, Hal Abelson,Ellen Spertus and Liz Looney’s AppInventor: Create your own Androidapps (O’Reilly) and Jason Tyler’s AppInventor for Android: Build Your Own78 / MAY 2012 / WWW.LINUXJOURNAL.COM

Apps—No Experience Required! (Wiley).If you enjoyed App Inventor, you mightwant to look at some other tools forprogramming your Android device visually,such as DroidDraw and Corona. And ifyou want to program Android visually onthe device itself, check out Catroid.■Amit Saha is currently a PhD research student in the area ofEvolutionary Algorithms and Optimization. Like his random echoesshow (http://echorand.me), he has been writing on myriad Linuxand open-source technologies for the past five years. Overall, heloves playing around with a bit of this and a bit of that. He welcomescomments on this article and beyond at amitsaha.in@gmail.com.ResourcesMIT App Inventor: http://appinventor.mit.eduSetting Up Your Computer: http://appinventor.mit.edu/learn/setup/index.htmlApp Inventor Reference: http://appinventor.mit.edu/learn/reference/index.htmlAPK File Format: http://en.wikipedia.org/wiki/APK_%28file_format%29Under the Hood of App Inventor:http://googleresearch.blogspot.com/2009/08/under-hood-of-app-inventor-for-android.htmlApp Inventor Open-Source Project: http://code.google.com/p/app-inventor-releasesActivity Starter Component: http://appinventor.mit.edu/learn/reference/other/activitystarter.htmlApp Inventor Java Bridge Project: http://groups.google.com/group/app-inventor-instructors/browse_thread/thread/10a64e64b7886afbRunning Your Own App Inventor Service: http://appinventoredu.mit.edu/developers-blogs/andrew/2011/nov/running-your-own-app-inventor-serviceApp Inventor Course: http://sites.google.com/site/appinventorcourse“Android App Development” presentation by Peter McNeil: http://www.cjugaustralia.org/September+2011Catroid Project: http://code.google.com/p/catroidCode for This Article (available in the appinventor_article subdirectory):https://bitbucket.org/amitksaha/articles_code/srcWWW.LINUXJOURNAL.COM / MAY 2012 / 79

A PUREDATAMISCELLANYAn introduction to Pd—a graphicpatching environment designed foraudio production and processing,with extensibility for the additionof other media-related features.DAVE PHILLIPSAfter reviewing recent trends in the world of SuperCollider3 forLinuxJournal.com, I decided to investigate activity in the world ofPure Data, better known to its users and abusers as Pd. However,instead of offering up yet another tutorial on this or that aspect of Pd,I’ve opted for a nonlinear account of my experience with the system.80 / MAY 2012 / WWW.LINUXJOURNAL.COM

WHAT IS PURE DATA?Pd often is described as an example ofa “visual programming language”, adefinition that is at least broad enoughto avoid inaccuracy. Personally, I definePd as a graphic patching environmentdesigned for audio production andprocessing, with extensibility for theaddition of other media-related features,such as tools for processing images,text and video. Pd’s graphic patchingcapabilities consist of a small collectionof basic units—objects, messages,numbers, symbols and comments—thatare placed on a canvas-like work areaand graphically connected (patched) intosignal processing networks (Figure 1), aprocess similar to constructing a soundproducingarrangement of modules on anold-school modular patching synthesizer.Of course, Pd is a little more modern anda lot more flexible. Its basic units are notpredefined synthesis or audio processingprimitives. They can be those things (andmuch more), but you need to definethem as such, and you need to knowwhat definitions and values are allowablefor each type of unit. Figure 1 illustratesFigure 1. A Simple Pd PatchWWW.LINUXJOURNAL.COM / MAY 2012 / 81

want to try Pd in an optimized risk-freeenvironment, check out the Puredynedistribution. Puredyne is a live Linuxsystem designed for multimedia artistsand musicians who want a fast, lightsystem that’s also loaded with some of thebest audio/video programs available in thefree software universe.For my re-introduction to Pd, I decidedto use the packages available from theL2Ork site maintained by Professor IvicaBukvic at Virginia Tech. This flavor isbased on the Pd-extended package, withcustomizations for the hardware used bythe L2Ork performance group. Pd-L2Orkincludes almost all the extra packages Ineeded for this article. The Tech group’sflavor works well, but I encounteredsome significant problems when I tried torun examples from those extra libraries(see below).GETTING INTO ITRegardless of which Pd package youFigure 2. Pd’s Audio/MIDI Test SuiteWWW.LINUXJOURNAL.COM / MAY 2012 / 83

FEATURE A Pure Data Miscellanychoose, your first step into its worldshould be taken in the Media menu.First, configure your audio and MIDIdevices—JACK audio and ALSA MIDIare recommended for Linux users—thenselect the Test Audio And MIDI itemfrom the menu. This selection runs thetesttone.pd patch, a suite of tests toverify your audio/MIDI configuration(Figure 2). When you’re satisfied with itsresults, start opening and running theexample files found in /usr/lib/pd/doc/ or/usr/local/lib/pd/doc/.LEARNING TOOLSSearching on Google and GoogleVideos provides plenty of tutorials andpresentations on Pd (see Resources forsome of my favorites); however, Pd’s onlinedocumentation is one of the system’sgreat strengths. Cool and useful examplefiles abound in /usr/lib/pd/doc/ (or /usr/local/lib/pd/doc/), complete and readyto run. They include a wonderful varietyof synthesis techniques, effects anddynamics processors, GUI methods andso forth. I find the synthesis examplesof special interest. Many techniques arepresented that are not so easily accessedin other systems—for example, SSBmodulation and four different FM synths.Users are encouraged to use the exampleinstruments as the basis for their owninstrument designs and compositions,a matter accomplished with simple cutand-pasteoperations. Editing patchesis equally simple, with all edits donedirectly on the patch canvas. Pd patchescan be represented as plain-text files, butthey rarely are edited directly and moreoften are used for exchanging patchesbetween users. The GUI is where theediting action takes place in Pd.By the way, when you don’t wantto break your work flow to look up amodule’s definition, simply right-clickover any box to summon a pop-up menuthat includes a selection for the object’sHelp window. The Help file describesthe object’s functions and provides asummary of its parameters. New usersshould note that the Help file itself isan active patch, so be careful aboutchanging its default values.Given Pd’s longevity, it’s no surprise tofind a rich set of third-party resources forits study and expansion. If the program’sown documentation isn’t enough, youcan find a great FLOSS Pd manual on theNet, and other Pd-oriented books abound(see Resources). I recommend especiallyDesigning Sound, by Andy Farnell. Apartfrom its great significance to users ofPd, Designing Sound is a most engagingand deeply fascinating book. Ostensiblytargeted toward game sound designers,this book is a treasure for anyone whoworks with digital audio. It covers a vastrange of subject material, all relevantto his topic at large, much of which ispotentially difficult to impart to a nonspecialistreader. However, Mr Farnell isa master of exposition and explanation,and I’m absorbed by whatever I read84 / MAY 2012 / WWW.LINUXJOURNAL.COM

in his book. Designing Sound has aparticular appeal for Pd people—everyexample in the book is presented in Pd,and although the author states it is nota Pd tutorial, beginners may find hispresentation especially enjoyable.MAKING CONNECTIONSPd loves external connections. Indeed,it’s all just pure data to Pd—if you canget the data into Pd, the program willmake every attempt to process it. That’sgood news for me, because I want to useit for processing input from a variety ofexternal devices.For external MIDI control over mypatches, I have a Behringer BCF2000control surface and an Akai LPK252-octave keyboard, both of which wereinstantly recognized by my Arch systemand by QJackCtl. I used QJackCtl’s ALSAconnections panel to hook the hardwareFigure 3. Hardware Controller Inputs in a Pd PatchWWW.LINUXJOURNAL.COM / MAY 2012 / 85

FEATURE A Pure Data MiscellanyFigure 4. Pd Plays with IanniXinto Pd where notein and ctlin objectsreceive data from my MIDI hardware(Figure 3). Thus, when I move a slideron the BCF, the slider in my Pd patchmoves in smooth sync with the hardwarecontroller. I use the LPK to make suddenchanges in visual displays, and sometimesI use it simply to play notes into a Pdsynthesizer patch.I tested Pd’s OSC support byconnecting it to the IanniX OSCsequencer. Thanks to user/developercodex99, I tested a pre-release ofIanniX that includes a neat examplefor working with Pd. The screenshot inFigure 4 shows the connection at work.Pd controls the speed of the sequencerwhile IanniX controls the animationin the gemwin display window. It’s allvery cool, and it opens the way to somefascinating possible uses. Perhaps you’dlike an OSC-enabled arrangement of Pd +IanniX + Ardour3? No problem, and youmight as well throw in connections withProcessing and Renoise too, perhaps witha machine per program and everythingconnected by netsend/netreceive objects.I also have a Webcam and a miniDVcamera that I wanted to hook into Pd. AsI’ve noted already, Pd itself has no videoprocessing capabilities, but many of itsexternals are dedicated to visual dataacquisition and massage. The good newsis that the physical connections—USB86 / MAY 2012 / WWW.LINUXJOURNAL.COM

and FireWire ports and drivers (V4L2 andIEEE1394)—are well supported by thoseexternals. The bad news is that I had tojump through more than a few hoops tomake it all work.VIDEO PROBLEMS AND SOLUTIONSThe packaged version of GEM workedwell for everything except patchesinvolving my cameras. Video file playbackworked well enough, but I had troubleswith live video input. Neither my Webcamnor my miniDV camera were recognized,although both worked fine in the Cheeseand Kino programs. A bit of research ledme to conclude that my installed versionof GEM was borked, so I decided to takethe opportunity to upgrade. I built andinstalled GEM 0.93.3 (with support for itsPd external, of course), and my Webcamcame to life in my experiments withthe pix_video module. Alas, my miniDVcamera remained moribund. Once again,a little research led to the discovery thatmy Arch system’s 3.0 kernel introducedsome changes in its support for FireWiredevices. So, I ran this command:chmod 666 /dev/fw1Then, I specified /dev/fw1 as myFireWire device for pix_video. When Iselected /dev/fw1 in my patch, the Pdconsole window complained with thiserror message:However, the device worked perfectlyin the patch, and now I have video I/Ofor both devices.PDP, PiDiP and GridFlow similarlywere crippled by problems with therelevant kernel modules. I rebuilt andre-installed all three externals, butI also needed to change my patchesto accommodate pd-v4l2 instead ofpd-v4l. At last, my test patch openedmy Webcam with this series ofinformative messages:pdp_v4l2: opening /dev/video0pdp_v4l2: driver info: uvcvideo 1.1.0 / UVC Camera (046d:09a1)➥@usb-0000:00:0b.1-5pdp_v4l2 : input 0 : Camera 1pdp_v4l2: device has 1 inputspdp_v4l2: switched to input 0pdp_v4l2: device supports 0 standardspdp_v4l2 : format 0 : MJPEGpdp_v4l2 : format 1 : YUV 4:2:2 (YUYV)pdp_v4l2: device supports 2 formatspdp_v4l2 : current format is : MJPGpdp_v4l2 : setting format : index : 0 : pixel format : MJPGpdp_v4l2 : capture format : width : 320 : height :240 :➥bytesperline : 0 : image size : 102400control 9963776 active (i:0)control 9963777 active (i:1)control 9963778 active (i:2)control 9963788 active (i:12)control 9963795 active (i:19)control 9963800 active (i:24)control 9963802 active (i:26)control 9963803 active (i:27)control 9963804 active (i:28)error: Cannot open '/dev/fw1': 25, Inappropriate ioctl for devicepdp_v4l2: got 8 buffers type 1 memory 1WWW.LINUXJOURNAL.COM / MAY 2012 / 87

FEATURE A Pure Data MiscellanyFigure 5. The #dp-camera Abstractionpdp_v4l2 : mapped 8 bufferspdp_v4l2 : queued 8 bufferspdp_v4l2 : device initializedpdp_v4l2 : created thread : 2412164864pdp_v4l2 : capture startedEverything looked good up to thispoint, but then the console printedthis line:pdp_v4l2: unsupported color model: 1196444237My patch’s rendering window opened,the Webcam’s LED was on, but I hadno picture until I figured out that the“unsupported color model” resulted fromthe MJPG format setting. I changed theformat value to 1 (for YUYV support),and voilà, my smiling face appeared inthe patch’s rendering display. Finally, Ihave support for my Webcam in my Pdvideo patches. Figure 5 shows the basiccamera configuration for my PDP/PiDiP88 / MAY 2012 / WWW.LINUXJOURNAL.COM

video patches.Incidentally, I use this patch now asan abstraction in my own versions ofthe tutorial patches.Alas, the situation with my miniDVcamera is not so good. PDP/PiDiPincludes the pdp_ieee1394 object, butit doesn’t like /dev/fw1, and my Archsystem provides no other appropriatedevice node (such as /dev/dv1394). I ranPd as superuser, but still got no joy withthe camera. I’ll continue to search for asolution, and in the meanwhile, I can useGEM when I want DV support.IT TAKES TWOIn one of my experiments, I created apatch that joined a video processor withan audio synthesizer, both of whichelements were controlled by a singleslider. The patch worked, but the videoplayback seriously interfered with theaudio continuity. Fortunately, I founda solution: I separated the patch intotwo patches, then I ran those patchesin two instances of Pd, launching oneinstance with the -noaudio option forthe video processor and starting theother with the -rt flag for the audiosynthesis. Both patches include a ctlinobject that receives data from a slideron an external MIDI control surface(see above). With this arrangement, thegraphics are updated smoothly, and theaudio playback glitches are gone. Thissolution is not as elegant as the singlepatchdesign, but it’s definitely the betterperformer on my aged single-core CPU. Ilearned later that a two-machine solutionis available with the netsend/netreceiveobjects. I haven’t tried it yet, but it’s onmy short list of neat things to do with Pd.THE PD EVERYWHERE PROJECTPd can be decoupled from its defaultGUI, a feature that has drawn theattention of developers Peter Brinkmannand Peter Kirn. Thanks to their libpdProject, Pd-based software can be runon mobile devices and systems, suchas the Android and the iOS. The librarycan be embedded directly into devices,promising a new generation of Pd-basedaudio software for the little and notnecessarily-so-littlethings. You cankeep up with libpd’s development atthe Pd Everywhere site and on PeterBrinkmann’s blog (see Resources). It’scool—check it out.As this article went to press, Ireceived a copy of Making MusicalApps: Real-Time Audio Synthesis onAndroid and iOS, a new book writtenby Peter Brinkmann. It’s a must-haveitem if you’re looking for a way to useLinux to program audio applicationsfor the new mobile devices. The bookhas also persuaded me that I reallyneed to get an Android.MORE POWER PDPd has its power users, and some ofthose users have Web sites dedicatedto their Pd-related productions andWWW.LINUXJOURNAL.COM / MAY 2012 / 89

FEATURE A Pure Data Miscellanyinvestigations. Frank Barknecht is atrue Pd wizard (and keen cyclist); you’llwant to look at his footils.org site andcheck out his work on the RRADicaland RjDj Projects. Chris McCormick’sWebPd Project is very cool, as is hisSqueakyshoecore music (made with Pd,of course), and longtime user/developerGuenter Geiger recently has expanded hisPd-related work into the PDa (Puredataanywhere) Project, an effort to bring Pd’spower to Linux-friendly PDA devices.Dr Albert Graef has lent hisconsiderable talents to Pd in the formof his Pd-Faust software. The projectincludes a new system for running Faustplugins in Pd, with dynamic reloadingand on-the-fly GUI generation. It’savailable from the Pure Web site (seehttp://code.google.com/p/pure-lang/wiki/Addons#pd-faust) for furtherdetails. Dr Graef notes that getting it allup and running may be a bit tedious—ithas dependencies not usually found in theaverage package repos—but the latestrelease with all needed dependencies isavailable in the PlanetCCRMA repositoryat http://ccrma.stanford.edu/planetccrma/mirror/fedora/linux/planetccrma/14/x86_64/repoview/pd-faust.html.These days, it seems everyone wantsto get in touch with Pd. I’ve alreadynoted that the Renoise tracker/DAWand the IanniX OSC sequencer includeexamples for connecting those worthiesto the power of Pd, and even the mightyCsound provides a Csound-to-Pd bridge.These definitely are good days, and goodcompany, for Pd.By the way, if you’re looking fora quick introduction to some of Pd’ssonic capabilities, check out someof the offerings on Soundcloud andthe contributions at Puredata.info.Soundcloud’s Pure Data communityhas uploaded more than 60recordings, and you’re bound to findsomething of interest there.THE WRAPAs a veteran Csounder, I must say I amimpressed with Pd’s synthesis capabilities.However, I also must confess that I’mmost attracted to Pd for its extendedservices, particularly in the graphics andvideo realms. I continue to look intothe possibilities of the PDP/PiDiP andGridFlow packages, and I have so muchmore to learn about GEM’s capabilities.I’m also interested in using it for poetryand prose text manipulation—Pd is thatflexible, as evidenced by its many uses inreal-world deployment.■Dave Phillips is a musician/author/teacher living and working inFindlay, Ohio. He has worked with computers for musical purposessince 1985. He discovered Linux in 1995 and created thehttp://linux-sound.org Web site shortly after. He has written onaudio topics for Linux Journal and other publications since 1998and wrote The Book Of Linux Music & Sound (NoStarch Press) in2000. His extracurricular activities include practicing t’ai-chi,reading Latin poetry, walking a shar-pei named Maximus, andspending as much time as possible with a woman named Ivy.90 / MAY 2012 / WWW.LINUXJOURNAL.COM

Resources“Super Collision At Studio Dave: The New World ofSuperCollider3, Part 1” by Dave Phillips:http://www.linuxjournal.com/content/super-collisionstudio-dave-new-world-supercollider3-part-1Pure Data: http://puredata.infoMax (software): http://en.wikipedia.org/wiki/Max/MSPCycling74: http://cycling74.com“Vanilla” Pd Release (maintained by Miller Puckette):http://crca.ucsd.edu/%7Emsp/software.htmlPd-extended: http://puredata.info/community/projects/software/pd-extendedPd-L2Ork Project: http://l2ork.music.vt.edu/main/?page_id=56Puredyne: http://puredyne.orgProgramming Electronic Music in Pd by JohannesKreidler: http://www.pd-tutorial.comObiwannabe Home: http://obiwannabe.co.ukvreahli’s channel (YouTube): http://www.youtube.com/user/vreahliPure Data: Lesson 01, Hello World! by Dr. RafaelHernandez (YouTube): http://www.youtube.com/watch?v=rtgGol-I4gABorgmekanik’s Pd Videos (YouTube):http://www.youtube.com/user/borgmechanikerPd + GEM Creations from Boris.volant and VincentPrijent (YouTube): http://www.youtube.com/user/trahisonM/videosFLOSS Pure Data Manual: http://flossmanuals.net/puredatapd-graz Group’s bang Book: http://pd-graz.mur.at/label/bookThe Theory and Technique of Electronic Music byMiller Puckette: http://www.amazon.com/Theory-Technique-Electronic-Music/dp/9812700773/ref=sr_1_1?s=books&ie=UTF8&qid=1325192262&sr=1-1Composition: Pure Data as a Meta-Compositional Instrument by Michael Barkl:http://www.amazon.com/Composition-Pure-Data-Meta-Compositional-Instrument/dp/3838316479/ref=sr_1_1?s=books&ie=UTF8&qid=1325192847&sr=1-1Designing Sound by Andy Farnell:http://www.amazon.com/Designing-Sound-Andy-Farnell/dp/0262014416/ref=sr_1_1?s=books&ie=UTF8&qid=1325192902&sr=1-1“An Introduction to OSC” by Dave Phillips:http://www.linuxjournal.com/content/introduction-oscIanniX: http://www.iannix.org/en/index.phpArdour: http://ardour.orgProcessing: http://processing.orgRenoise: http://www.renoise.comCheese: http://projects.gnome.org/cheeseKino Video Editor: http://www.kinodv.orgPd Everywhere: http://noisepages.com/groups/pd-everywherePeter Brinkmann’s Blog—Pure Data, Android Audio,and Random Stuff: http://nettoyeur.noisepages.comMaking Musical Apps: Real-Time Audio Synthesison Android and iOS by Peter Brinkmann:http://shop.oreilly.com/product/0636920022503.doFrank Barknecht: http://footils.orgRRADical: http://puredata.info/community/projects/rradicalRjDj: http://rjdj.me/music/Frank%20BarknechtChris McCormick’s WebPd Project: http://mccormick.cx/projects/WebPdChris McCormick’s Squeakyshoecore Music:http://sciencegirlrecords.com/chr15m/squeakyshoecorePd-Faust: http://docs.pure-lang.googlecode.com/hg/pd-faust.htmlPDa Project: http://gige.xdv.org/public_html/pdaCsound-to-Pd Bridge: http://booki.flossmanuals.net/csound/csound-in-pdOfferings on Soundcloud: http://soundcloud.com/groups/pure-data-exclusiveContributions at Puredata.info: http://puredata.info/community/tracksWWW.LINUXJOURNAL.COM / MAY 2012 / 91

ParallelProgrammingin C and PythonA fast-track guide to writingparallel programs in C and Python.AMIT SAHAAdvancements in computingtechnologies have enabled even“resource-constrained” Netbooksto have four CPU cores. Thus, parallelcomputing is not about having a largenumber of computer nodes in a clusterroom anymore. To write programs thattake advantage of such easily availableparallel computing resources, C andPython programmers have libraries attheir disposal, such as OpenMP (C) andmultiprocessing (Python) for sharedmemory parallel programming, andOpenMPI (C/Python) for distributedmemory parallel programming. And, incase you don’t have hardware resourcesbeyond your quad-core personal computer,cloud-computing solutions, such asPiCloud, can be of tremendous use.For the purposes of this article, Iassume you have one computer atyour disposal and, thus, focus onOpenMP, multiprocessing and PiCloud.Note that the article’s code examplesdemonstrate parallel programming andmay not be the most optimal strategypossible for parallelization. I also haverefrained from any benchmarking toshow speedups of the parallel programsversus their serial counterparts.92 / MAY 2012 / WWW.LINUXJOURNAL.COM

Shared Memory ParallelProgrammingShared memory parallelprogramming libraries enable writingprograms that can take advantage ofthe multiple CPUs on your computer.OpenMP (Open specificationsfor Multi-Processing) is a standardAPI specification that may be usedexplicitly to achieve multithreaded,shared memory parallelism. TheAPI is defined for C/C++ andFORTRAN, and it is composed ofthree primary API components:compiler directives (#pragma in C),runtime libraries and environmentvariables. An OpenMP programbegins life as the master threaduntil it encounters a parallel block.A team of parallel threads is thencreated; this operation is calledthe fork operation. The statementsin this parallel block are executedin parallel by these threads. Oncethe team of threads has finishedexecution, the master thread is backin control, which is called the Joinoperation. This model of executionis referred to as the Fork-Join model.Let’s use the GNU implementationof OpenMP, called libgomp. Yourdistribution’s package manageris the easiest way to install thislibrary. To enable the compilerdirectives that you use in yourOpenMP programs, specify-fopenmp, and to link to the runtimeListing 1. ompdemo.c#include #include int main (int argc, char **argv) {int nthreads, tid, i;/* Get the number of processors */printf("Number of processors available::➥%d\n",omp_get_num_procs());/* Set the number of threads to the number of processors */omp_set_num_threads(omp_get_num_procs());/* Fork a team of threads with each thread having aprivate tid variable * */#pragma omp parallel private(tid)}{/* Obtain and print thread id */tid = omp_get_thread_num();printf("Hello World from thread = %d\n", tid);/* Only master thread does this */if (tid == 0){}nthreads = omp_get_num_threads();printf("Number of threads = %d\n", nthreads);} /* All threads join master thread and terminate */return 0;library, a -lgomp has to be added duringthe compilation of your OpenMP programs.WWW.LINUXJOURNAL.COM / MAY 2012 / 93

FEATURE Parallel Programming in C and PythonLet’s start by creating a team of threads (Listing1). First, use the omp_get_num_procs()function to get the number of processorsavailable, which you use to set the number ofthreads using the omp_set_num_threads()function. Next, specify the parallel block of theprogram using #pragma omp parallel.You want each thread to have a privatecopy of the tid variable, so append thisinformation to the #pragma directive. Inthe parallel block, let’s simply print a helloworld message from all of the threads,including the master thread. The masterthread has a tid of 0, and it can be usedto execute statements specific to it. Thethread id of a particular thread is obtainedusing the omp_get_thread() function.Once you compile and execute thisprogram, the output, depending on thenumber of processors on your computer,will be similar to the following:$ gcc -o ompdemo ompdemo.c -lgomp -fopenmp$ ./ompdemoNumber of processors available:: 2Hello World from thread = 0Number of threads = 2Hello World from thread = 1Next, let’s consider a code snippet in yourprogram that may look like this:New: Intel Xeon E5 Based ClustersBenchmark Your Code on Our Xeon E5 BasedTesla Cluster with:AMBER, NAMD, GROMACS, LAMMPS, or Your Custom CUDA CodesUpgrade to New Kepler GPUs Now!Microway MD SimCluster with8 Tesla M2090 GPUs8 Intel Xeon E5 CPUs and InfiniBand2X Improvement over Xeon 5600 SeriesGSA ScheduleContract Number:GS-35F-0431N

A -> Large arrayFor every element e in array Acall function, fun(e)end forInstead of calling the function, fun forevery element, e, serially, what if you coulddo it in parallel? OpenMP’s work-sharingconstructs enable you to do that. Considerthe code snippet below:A -> Large array#pragma omp for schedule(dynamic,chunk)For every element e in array Acall function, fun(e)end forThe directive, #pragma omp forschedule(dynamic,chunk) specifies thatthe ensuing for loop will be executed in parallelby the thread team, with the workload specifiedby the schedule(dynamic, chunk)clause. The chunk variable specifies the unitof division—that is, the number of iterations ofthe for loop that will be executed by a thread,and dynamic specifies that the chunks will beassigned dynamically to the threads.Let’s use this directive to process a largearray, calling a function for each arrayelement (Listing 2). The parallel section of thecode begins at the directive #pragma ompparallel where we specify that the arrays aand b and the variable chunk will be sharedHarness Microway’s Proven GPU ExpertiseThousands of GPU cluster nodes installed.Thousands of WhisperStations delivered.Award Winning BioStack – LSAward Winning WhisperStation Tesla – PSC with 3D‘11AWARDBESTBest NewTechnologyns/Day (Higher is Better)CPU + GPUCPU Only1.070.332.020.653.541.301 Node2 Nodes 4 NodesNAMD F1-ATP Performance GainConfigure Your WhisperStation or Cluster Today!www.microway.com/tesla or 508-746-7341

FEATURE Parallel Programming in C and PythonListing 2. omp_for_eval.c/* Work Sharing constructhttps://computing.llnl.gov/tutorials/openMP/#DO/* Get the number of processors */printf("Number of processors available::Distributes an array of elements across threads, where each➥%d\n",omp_get_num_procs());element is passed as a parameter to a function to be evaluated/* Set the chunk size*/*/chunk = CHUNKSIZE;#include #include /* Set the number of threads to the number of processors*/omp_set_num_threads(omp_get_num_procs());#define N 100000#define CHUNKSIZE 100/* dummy function*/#pragma omp parallel shared(a,b,chunk) private(i){float myfun(float a){#pragma omp for schedule(dynamic,chunk)return a*a;for (i=0; i< N; i++)}{b[i] = myfun(a[i]);int main (int argc, char **argv){}} /* end of parallel section */int i, chunk,tid;float a[N], b[N];printf("For evaluation completed, the result➥has been stored in array B\n");for (i=0; i < N; i++)return 0;a[i] = i * 1.0;}among the threads. This is required, becauseall the threads will be accessing the arrays aand b, and chunk will be used by the ensuingparallel for loop to determine the unit ofdivision. The variable, i, specifying the loop,will be private to each thread.The result of the function evaluationwill be available at the end of the parallelsection in the array b:$ gcc -o omp_for_eval omp_for_eval.c -lgomp -fopenmp$ ./omp_for_evalNumber of processors available:: 2For evaluation completed, the result has been stored in array B96 / MAY 2012 / WWW.LINUXJOURNAL.COM

Listing 3. pi_openmp.c/* Program to compute Pi using Monte Carlo method:(http://math.fullerton.edu/mathews/n2003/montecarlopimod.html)*/int i,count=0; /* # of points in the 1stdouble z;quadrant of unit circle */double pi;#include #include #include #include /* Get the number of processors */printf("Number of processors available::➥%d\n",omp_get_num_procs());/* Returns the value of count with niter iterations*/int part_count(int niter){int i, count=0;printf("Enter the number of iterations used to➥estimate pi (multiple of %d please):➥",omp_get_num_procs());scanf("%d",&niter);float x,y,z;for ( i=0; i

FEATURE Parallel Programming in C and Pythonof a unit circle. The number of samplesgenerated have a direct effect on the accuracyof the estimate of the value of Pi. With thehelp of OpenMP, let’s divide the number ofsamples equally among multiple threads andfinally accumulate the result in the masterthread to calculate the value of Pi (Listing 3).In this code, notice the use of thereduction clause, #pragma omp parallelshared(chunk) reduction(+:count).This effectively means to perform the +operation on the private copies of the variablecount, and store it in the global sharedvariable count. The value of count then isused to calculate the value of Pi:and creates that number of processes.A process is created by creating an object ofthe Process class: p= Process(target=f,name='Process'+str(i), args=(i,)).Listing 4. mpdemo.py'''Create number of processes using the multiprocessing module'''import multiprocessingfrom multiprocessing import Process# dummy functiondef f(id):#This is a dummy function taking a parameter$ gcc -o pi_openmp pi_openmp.c -lgomp -fopenmpreturn$ ./pi_openmpNumber of processors available:: 2if _ _name_ _ == '_ _main_ _':Enter the number of iterations used to estimate pi➥(multiple of 2 please): 10000# of iterations = 10000 , estimate of pi is 3.1636[gene@zion openmp]$ ./pi_openmp# get the number of CPUsnp = multiprocessing.cpu_count()print 'You have {0:1d} CPUs'.format(np)Number of processors available:: 2Enter the number of iterations used to estimate pi➥(multiple of 2 please): 1000000# of iterations = 1000000 , estimate of pi is 3.14222Python’s Multiprocessing ModulePython’s multiprocessing module effectivelyside-steps the Global Interpreter Lockthat inhibits true multithreaded Pythonprograms. With this module, your Pythonprograms can make use of multipleprocessors on your computer. Listing 4 is aPython program that uses this module toget the number of CPUs on your computer# Create the processesp_list=[]for i in range(1,np+1):p = Process(target=f, name='Process'+str(i), args=(i,))p_list.append(p)print 'Process:: ', p.name,p.start()print 'Was assigned PID:: ', p.pid# Wait for all the processes to finishfor p in p_list:p.join()98 / MAY 2012 / WWW.LINUXJOURNAL.COM

Listing 5. mp_queue.py'''Queue: http://docs.python.org/library/➥multiprocessing.html#pipes-and-queues# chunk sizeif num_el%np != 0:print "The current chunking mechanism will not work"exitDemonstrates the usage of Queue to share data between processes.Splits up a large array into chunks and calculates the partialelse:chunk = num_el/npdot products.'''# Create the processesp_list=[]import multiprocessingfrom multiprocessing import Process, Queuefrom numpy import *# Create the Queue which will have the partial productsproduct=Queue()# dot product of the partial data chunksfor i in range(1,np+1):def add(chunk1,chunk2,product):a = chunk1.get()b = chunk2.get()# A pair of queues per process for the two arraysaq = Queue()bq = Queue()prod = a*bproduct.put(sum(prod))# push the chunks into the queueaq.put(a[(i-1)*chunk:i*chunk])if _ _name_ _ == '_ _main_ _':bq.put(b[(i-1)*chunk:i*chunk])#size of the arraysnum_el = 100000# create the processp = Process(target=add, args=(aq,bq,product))p.start()# Create two arraysp.join()a = linspace(0,100,num_el);b = linspace(0,1,num_el);# collect the individual sumsitems=[]# get the number of CPUs and assign it as the number of# processes to createfor i in range(product.qsize()):items.append(product.get())np = multiprocessing.cpu_count()print 'You have {0:1d} CPUs'.format(np)# final product: sum of individual productsprint "Dot product:: ",sum(items)WWW.LINUXJOURNAL.COM / MAY 2012 / 99

FEATURE Parallel Programming in C and PythonThe target argument specifies thecallable object to be called by the process’srun() method, which is invoked bythe start() method; name specifies acustom name for the process; and argsspecifies the argument tuple for the target.Depending on the number of processorson your computer, you should see outputsimilar to the following:$ python mpdemo.pyYou have 2 CPUsProcess:: Process1 Was assigned PID:: 29269Process:: Process2 Was assigned PID:: 29270This program maintains a list of thecreated process objects so as to wait forthem to finish using the join() method. Ifyou attempt to use this example as a startingpoint to actually do something, you will wanta way to send back information from thespawned processes to the master process.There are a few ways to achieve this. Theeasiest, but not really recommended way,is to use shared state variables. Anotherway to establish a communication channelbetween the master and the spawnedprocesses is to use Queues. Listing 5 usesthe multiprocessing module to calculate thedot product of a large array by splitting thecalculation across multiple processes.For every process created, a pair of queueobjects are created, one each for the twoarrays. The whole array is divided into equalchunks, and a process is assigned a chunkto calculate the partial product. The partialproducts are stored in the queue objectproduct. After the processes have finishedexecution, partial products are retrieved fromthe queue and summed to get the actual dotproduct. Here is sample output:$ python mp_queue.pyYou have 2 CPUsDot product:: 3333350.00017Next, let’s take a look at Process pools.As the name implies, a pool of processesis created using the Pool class, andeach member of this process pool is thenassigned a task to execute using methods,such as apply() or map(). Listing 6demonstrates using Process pools tocalculate the value of Pi using the samealgorithm as shown in Listing 3.The pool of processes is created using thestatement pool = Pool(processes=np),and each process in this pool is asked to invokethe function monte_carlo_pi_part usingcount=pool.map(monte_carlo_pi_part,part_count), where part_count is a listwith the number of samples to be generatedin each process. The returned value from eachprocess is stored in the list, count. Finally,let’s sum this list and calculate the estimatedvalue of Pi:$ python pi_mp.pyYou have 2 CPUsEstimated value of Pi:: 3.1412128The call to map() is a blocking call—thatis, the process doesn’t terminate until the resulthas been received. For other applications, the100 / MAY 2012 / WWW.LINUXJOURNAL.COM

Listing 6. pi_mp.py'''Multiprocessing-based code to estimate the value of PI#returnreturn countusing Monte Carlo sampling.Ref: http://math.fullerton.edu/mathews/n2003/montecarlopimod.htmlUses workers:if _ _name_ _=='_ _main_ _':http://docs.python.org/library/➥multiprocessing.html#module-multiprocessing.pool'''np = multiprocessing.cpu_count()print 'You have {0:1d} CPUs'.format(np)import randomimport multiprocessing# Number of points to use for the Pi estimationn = 10000000from multiprocessing import Pool# iterable with a list of points to generate in each worker# each worker process gets n/np number of points to#calculate the number of points in the unit circle# calculate Pi from#out of n pointsdef monte_carlo_pi_part(n):part_count=[n/np for i in range(np)]count = 0for i in range(n):x=random.random()y=random.random()#Create the worker pool# http://docs.python.org/library/➥multiprocessing.html#module-multiprocessing.poolpool = Pool(processes=np)# if it is within the unit circleif x*x + y*y

FEATURE Parallel Programming in C and Pythonaccount on the project Web site, and theninstall the client from the PyPi packageindex. The Quickstart documentationshould help you set up PiCloud on yourcomputer (see Resources).Let’s write a simple function and run it onPiCloud. First, define a function sort_num()in the file demo.py as follows:import numpydef sort_num(num):sort_num = numpy.sort(num)return sort_numFire up the Python interpreter andimport the function you just defined andthe modules cloud and numpy:>>> from demo import *>>> import cloud>>> import numpysort_num to be executed with the argumentnum. This method returns an integer that isan identifier for this particular job. The formalspecification of the cloud.call methodis available at http://docs.picloud.com/moduledoc.html#cloud.call:>>> jid58The returned value of the executedfunction can be obtained using the functioncloud.result using the obtained jid:>>> cloud.result(jid)array([ 10, 11, 11, ..., 10000, 10000, 10000])As you can see, the returned array isobtained. Next, consider a function (definedas a file mapdemo.py), which returns thevolume of a cylinder when the radius(r) andheight(h) is passed to it:Next, create an array of 50000 integerswith each integer chosen randomly from therange (10, 10000):>>> num=numpy.random.random_integers(10,10000,50000)Now comes the important step,invoking the cloud.call method tospecify the function that you want to runand its arguments:>>> jid=cloud.call(sort_num,num)This implies that you want the functionimport numpydef vol_cylinder(r,h):return numpy.pi*r*r*hIf you had one cylinder configuration—that is, one pair of (r,h)—the cloud.callmethod can be used for the purpose. Whatif you had 100 different configurations tosquare? You could use 100 cloud.callinvocations, but there is a more efficientway of doing it: using the cloud.mapfunction, which accepts a sequence ofparameters (or sequences of parameters):102 / MAY 2012 / WWW.LINUXJOURNAL.COM

from mapdemo import *>>> import cloud>>> import numpy>>> r=numpy.random.random_sample(100)>>> h=numpy.random.random_sample(100)>>> jids=cloud.map(vol_cylinder,r,h)>>> jidsxrange(359, 459)As you can see, jids is a list of 100elements with values from 359 to 459 (notethat the values you have may be different).Here, cloud.map has created 100 jobs withthe arguments formed from the 100 pairs ofthe two sequences, r and h. You can pass thislist of jids directly to cloud.result to getthe results as a list:>>> result=code.result(jids)>>> result[0.35045338986267927, 0.0043144690799585004,➥0.094018119621969765, 0.93579722612039329,➥0.0045154147876736109, 0.018836324478609345,➥0.0027243595262778321, 1.5049675511377265,➥0.37383416636274164, 0.24435487403102638,➥0.28248315493701553, 1.2879340600324913,➥0.68406526971023041, 0.14338739850272786,...In more practical situations, the functioncomputation may not be so trivial, andyou may not have a definite idea when allthe jobs will be over. In such a case, youcan wait until all the jobs have finishedto retrieve the results using the functioncloud.join. The PiCloud documentationuses the map function to calculate thevalue of Pi (yes, again!) using the map()method (see Resources).Among a host of other interestingfeatures, PiCloud allows your Pythonapplication to expose your functions inPiCloud via a REST API, which will allowyou to call this function from any otherprogramming language. See the Resourcessection of this article for an example.PiCloud allows you to set up cron jobs,use different computing resources and useenvironments to install any native librariesfor use in your application.Distributed Memory ParallelProgrammingThe past few sections in this article give ataste of parallel programming from a deviceas resource-constrained (relatively!) as adual-core desktop. What if you already have a30-node computing cluster at your disposal?In that case, you would use the tried-andtestedOpenMPI, an implementation of theMPI-2 specification. Another solution forsetting up a computing cluster is the ParallelVirtual Machine (PVM). If you don’t havemultiple computers and still want to try outOpenMPI programming, you simply canrun it on a standalone computer or use avirtualization solution, such as VirtualBox, tocreate a cluster of virtual machines.ConclusionMy intention with this article was to providean introduction to parallel programming.Because the hardware requirements are reallybasic, I mostly looked at shared memoryparallel programming. I also covered a moreWWW.LINUXJOURNAL.COM / MAY 2012 / 103

FEATURE Parallel Programming in C and Pythonmodern way of parallel computing usinga cloud-computing service. Another veryinteresting project, Parallel Python, enableswriting shared memory as well as distributedmemory parallel programs in Python. I hopethe examples in this article help you getstarted exploring parallel programming fromthe comfort of your Netbook!■Amit Saha is currently a PhD research student in the area ofEvolutionary Algorithms and Optimization. Like his random echoesshow (http://echorand.me), he has been writing on myriad Linuxand open-source technologies for the past five years. He welcomescomments on this article and beyond at amitsaha.in@gmail.com.ResourcesThe code from this article is available athttps://bitbucket.org/amitksaha/articles_code/src/6c95473182a5/parallel_article.Parts of the section on PiCloud were adapted from myarticle on PiCloud, titled “PiCloud: Easy way to the Cloud”published in Linux For You (http://www.lfymag.com),March 2012.OpenMP Tutorial: https://computing.llnl.gov/tutorials/openMPGNU OpenMP Implementation: http://gcc.gnu.org/onlinedocs/libgompMonte Carlo Sampling Technique: http://math.fullerton.edu/mathews/n2003/montecarlopimod.htmlProcess Class: http://docs.python.org/library/multiprocessing.html#process-and-exceptionsShared State Variables: http://docs.python.org/library/multiprocessing.html#sharing-state-between-processesQueues: http://docs.python.org/library/multiprocessing.html#exchanging-objects-between-processesPython Multiprocessing Documentation:http://docs.python.org/library/multiprocessing.htmlPiCloud Home Page: http://www.picloud.comPiCloud Documentation: http://docs.picloud.comPyPi Package Index: http://pypi.python.org/pypi/cloudPiCloud Quickstart Documentation:http://docs.picloud.com/quickstart.htmlcloud.map Function: http://docs.picloud.com/client_basic.html#mappingcloud.join Function: http://docs.picloud.com/moduledoc.html#cloud.joinCalculating the Value of Pi Using themap() Method: http://docs.picloud.com/basic_examples.html#calculating-piPiCloud Features: http://www.picloud.com/product/#featuresUsing a C Client to Invoke a Function Published viathe REST API: http://echorand.me/2012/01/27/picloud-and-rest-api-with-c-clientRunning an Evolutionary Algorithm in the Cloudwith PiCloud: http://echorand.me/2012/01/26/picloud-pyevolve-evolutionary-algorithms-in-the-cloudOpenMPI Home Page: http://www.open-mpi.orgParallel Programming Tutorial:https://computing.llnl.gov/tutorials/parallel_comBeginner MPI Tutorial: http://www.mpitutorial.com/beginner-mpi-tutorialMPI Python Bindings (mpi4py):http://mpi4py.scipy.orgPVM Home Page: http://www.csm.ornl.gov/pvmPVM Book: http://www.netlib.org/pvm3/book/pvm-book.htmlParallel Python: http://www.parallelpython.com104 / MAY 2012 / WWW.LINUXJOURNAL.COM

INDEPTHReplicate Everything!Highly AvailableiSCSI Storagewith DRBD andPacemakerIn high-availability clusters, redundancy of data is just ascrucial as redundancy of nodes. In the second installmentof his series on high availability, Florian Haas explainsbuilding rock-solid, block-replicated iSCSI data storagewith Pacemaker and DRBD. FLORIAN HAASAny high-availability solution isonly as good as its data. Historically,high-availability clusters often havesingle-instance, hardware-based storagesolutions—the classic and widespreadSAN. Interestingly, this is one of the fewremaining strongholds of proprietarysolutions, even in environments that areotherwise dominated by open-sourcetechnology. It’s also often an extremelyexpensive stronghold.The first article in this series on Linuxhigh availability covered the basics ofthe stack, taking storage for granted[see Florian’s article “Ahead of thePack: the Pacemaker High-AvailabilityStack” in the April 2012 issue]. Thisinstallment covers building redundant,automatically replicated, highly availablecluster storage that can act as a drop-inreplacement for costly and proprietarySAN-based solutions.Linux iSCSI: a Tale of Four TargetsThe highly available storage stack onLinux centers predominantly on iSCSI,an implementation of the SCSI protocolover IP networks. iSCSI, defined in RFC106 / MAY 2012 / WWW.LINUXJOURNAL.COM

INDEPTH3720 back in 2004, has become a widelyadopted and supported standard forSAN connectivity, largely displacing FibreChannel-based SANs in many shops.iSCSI, normally TCP-based with optionalsupport for RDMA-capable protocols,such as InfiniBand, could be describedas a client/server protocol, but that’sconfusing, as the iSCSI “client” is veryoften an application “server” in theconventional sense. Hence, iSCSI sticks tothe conventional SCSI terms of “target”(the thing that stores the data physically)and “initiator” (the thing that accessesthe data over the wire).As far as the initiator side isconcerned, there’s a clearly dominantsolution in a Linux environment,and that is the open-iscsi Project.Supported by all major distros andiSCSI vendors, open-iscsi, although notflawless throughout its existence, isnow ubiquitous and widely accepted asthe reference iSCSI initiator on Linux.For targets, things are a bit morecomplicated. No fewer than fouropen-source iSCSI targets currently areavailable to the Linux storage specialist.IET, the iSCSI Enterprise Target, is aniSCSI-only, in-kernel implementationof a target that is available as anout-of-tree kernel module only. IETcame out of a proprietary iSCSI targetimplementation from Netherlands-basedArdis Technologies, forked under GPLterms. It enjoys a remarkable proliferationamong vendors of cheap iSCSI storageappliances. Several distributions,including SUSE and Debian/Ubuntu,ship IET. Its development is ongoing,but humming along quietly undermost people’s radar. Scott Walkerand Arne Redlich are currently theproject’s main developers.With STGT, ex-IET maintainer FujitaTomonori intended to write an alluserspace,multiple-protocol replacementfor IET. It uses only a tiny, generic stubof in-kernel code that made it into Linuxat the 2.6.20 release. Everything elseabout the target happens in userland.Red Hat was quick to adopt STGT as itsdefault iSCSI target during the Red HatEnterprise Linux (RHEL) 5 release cycle,and continues to support it throughRHEL 6. SUSE also has adopted it forSUSE Linux Enterprise Server (SLES) 11,and it also ships in Debian and Ubuntu.However, active development on STGTseems to have largely ceased, and theproject is in maintenance-only mode.SCST, maintained primarily by VladislavBolkhovitin and Bart van Assche, isanother IET fork with the goal of fixing“all the problems, corner cases issues andiSCSI standard violations that IET has”, asthe project boldly states on its Web site.SCST has a large and devoted following,and SCST users primarily laud itsperformance benefits over the competingWWW.LINUXJOURNAL.COM / MAY 2012 / 107

INDEPTHtarget implementations. Contrary toSTGT and similar to IET, SCST does alot of its work in the kernel, and itsdevelopers have repeatedly submittedthe target for inclusion in the mainlinekernel. Thus far, however, these effortshave been unsuccessful—at leastpartially because a fourth target beatdevelopers to it.That fourth target is LIO, namedafter the “linux-iscsi.org” domain thatthe project owns. Its main developer isNicholas Bellinger. LIO is a generic, inkerneltarget driver, of which the iSCSItarget is merely one of many front ends.Of the four targets mentioned here, it isthe only one not interrelated with anyother, and that’s not its only interestingtrait. LIO uses an unusual in-kernelconfiguration approach using ConfigFS,which has produced some interestingflame wars on the linux-kernel mailinglist in years past. However, in early 2011,LIO beat out SCST to become the blessedreplacement of STGT as the upstreamkernel’s preferred target subsystem. Asupstream kernel releases trickle intodistributions slowly, some distros alreadyhave incorporated this upstream change,but most have not.DRBD: Storage Replication forthe MassesAny of the four iSCSI targets justmentioned would enable us to buildstable, fully open-source, single-instanceiSCSI storage on Linux. But that wouldviolate the three Rs of high availability:Redundancy, Redundancy, Redundancy.Storing all of our data just once isn’tgood enough. We’ll need at least twocopies of every block we store to remainavailable even in the face of a storagenode failing.That’s where DRBD comes in. Originallythe “Distributed Replicated BlockDevice”, it’s a synchronous replicationfacility in the kernel block layer. DRBDprovides a virtual block device type (withthe LANANA-registered major numberof 147) with unique characteristics: anywrite I/O it receives passes down into thelocal block layer and simultaneously intothe network stack. Over the wire, datareplicates to another node—the DRBDpeer—where it again passes through thelocal I/O subsystem and on to directlyattached storage. The synchronousnature of DRBD’s replication ensuresthat any writes to the device completeonly when completed on bothnodes. As such, its replication is fullytransparent to applications.Obviously, if one of the nodes fails,DRBD automatically switches itselfinto disconnected mode: rather thanpersisting in its temporarily pointlessquest to replicate, it starts recordingaddresses of out-of-sync blocksin a persistent bitmap. Once thepreviously failed node recovers, DRBDre-synchronizes the changed dataquickly and efficiently.DRBD’s overall overhead is surprisingly108 / MAY 2012 / WWW.LINUXJOURNAL.COM

INDEPTHsmall. Its impact on read performanceis practically nil, as reads are local tothe host, without any network layerinvolved at all. For writes, throughputis practically unaffected by a well-tunedDRBD, although some highly latency-criticalapplications will be adversely affected byrunning on DRBD.DRBD has had a somewhat colorfulhistory with regard to its mainlineLinux integration. For the better partof its lifetime, the developers chose tomaintain the project as an out-of-treemodule. Then in mid-2007, it made itsfirst push toward mainline integration,meeting initial rejection from the kernelcommunity—just like many projects without-of-tree baggage. The developers thenembarked on a massive cleanup process,culminating in a final pull requestfor the 2.6.32 kernel merge window.Following some last-minute opposition,Linus declined to pull DRBD for just thatrelease, and DRBD finally made it into thekernel for 2.6.33 in February 2010.Since then, the “official” DRBDcodebase and the Linux kernel have againdiverged, with the most recent DRBDreleases remaining unmerged into themainline kernel. A re-integration of thetwo code branches is currently, somewhatconspicuously, absent from Linux kernelmailing-list discussions. That situation hascaused interesting disparities regardingthe state of vendor support for DRBD.Some distributions, like Debian andUbuntu, currently ship DRBD kernel codeas part of their standard kernel packages.Others, like SLES, ship the out-of-treekernel module version. Still others, likeRHEL, do not ship DRBD at all and relyon outside packaging contributions fromCentOS and ELRepo. At any rate, to useDRBD, two components are requiredat a minimum: the DRBD kernel modeitself and a set of userspace utilities andshell scripts that ship as part of thedrbd-utils (drbd8-utils on Debian-likeplatforms) package. On a system wherethose packages are available, setting upa replicated data set—in DRBD-speak, a“resource”—amounts to a sequence ofthese steps:n Set aside a block device for DRBD’slocal use. DRBD refers to this as its“backing device”. This can be an SCSILUN, a disk partition, an LVM logicalvolume or any other Linux blockdevice you can possibly imagine.n Create a resource definition file in/etc/drbd.d.n Initialize DRBD’s metadata.n Enable the DRBD resource.n Kick off the initial full-devicesynchronization to make sure yourdata sets are in perfect unison (thisstep can be skipped if the devices arefresh off the assembly line and knownto be identical in content).WWW.LINUXJOURNAL.COM / MAY 2012 / 109

INDEPTHthe device as any other block device, theongoing background synchronizationnotwithstanding (this will sound highlyfamiliar to mdadm users).For most use cases, the next stepwould be to create a filesystem on theDRBD device. But because DRBD is justanother block device, you can treat it asexactly that, and you’re essentially free todo anything you would with a “normal”,non-replicated block device. In this case,let’s create an LVM Physical Volume (PV)signature and a Volume Group (VG):pvcreate /dev/drbd0vgcreate vg1 /dev/drbd0Once that volume group exists, youcan use it like any other—for example,you can create a 10-Gigabyte LogicalVolume from it:lvcreate -L 10G -n lun1 vg1And this logical volume, along withthe rest of its VG, is now availableon whichever node currently has theunderlying DRBD resource in the Primaryrole. You easily can try this out:mike:# vgchange -a n vg1mike:# drbdadm secondary vg1nancy:# drbdadm primary vg1nancy:# vgchange -a y vg1Thereafter, /dev/vg1/test is available asa block device on nancy. The steps youjust manually completed are, of course,a cluster manager’s task to handleautomatically, as I’ll explain in a moment.In combining DRBD and LVM with aLinux iSCSI target, you can create fullyreplicated, redundant SAN storage inLinux. You need only a cluster managerto tie everything together.Pacemaker-Based iSCSI TargetHigh AvailabilityAs for any Pacemaker cluster, youfirst need to set up the underlyingCorosync cluster messaging layer. Let’suse a configuration that is substantiallyidentical to that in the first article in thisseries, where I explained the basics ofCorosync in more detail:totem {# Enable node authentication & encryptionsecauth: on# Redundant ring protocol: none, active, passive.rrp_mode: active# Redundant communications interfacesinterface {}ringnumber: 0bindnetaddr: 192.168.0.0mcastaddr: 239.255.30.11mcastport: 5405interface {ringnumber: 1bindnetaddr: 192.168.43.0mcastaddr: 239.255.43.0WWW.LINUXJOURNAL.COM / MAY 2012 / 111

INDEPTHmcastport: 5405}}amf {mode: disabled}service {# Load Pacemakerver: 1name: pacemaker}Allocating a new multicast group toeach cluster on the network eliminatesthis issue.The actual Pacemaker configurationis one that you can, again, eitherimplement on a step-by-step, interactivebasis, or you can import it all in one fellswoop, thanks to the crm shell beingfully scriptable:primitive p_drbd_vg1 ocf:linbit:drbd \params drbd_resource="vg1" \op monitor interval="30" role="Slave" \op monitor interval="10" role="Master"logging {}fileline: offto_stderr: yesto_logfile: noto_syslog: yessyslog_facility: daemondebug: offtimestamp: onAttentive readers immediately willrealize that this example uses multicastaddresses different from the ones inmy previous article. This is a generallyestablished Corosync best practice: neverreuse cluster communications multicastaddresses across multiple distinct clusterssharing one host network. Cluster nodeswill not join rogue clusters due to mutualnode authentication via the Corosync“authkey” shared secret, but they willmulticast cluster communications tohosts that aren’t supposed to see them.ms ms_drbd_vg1 p_drbd_vg1 \meta interleave="true" notify="true"primitive p_lvm_vg1 ocf:heartbeat:LVM \params volgrpname="vg1" \op monitor interval="30"primitive p_target_vg1 ocf:heartbeat:iSCSITarget \params implementation="tgt" \iqn="iqn.2001-04.com.example:example.vg1" \tid="1" \additional_parameters="DefaultTime2Retain=60 DefaultTime2Wait=5" \op monitor interval="10"primitive p_lu_vg1_lun1 \ocf:heartbeat:iSCSILogicalUnit \params lun="1" path="/dev/vg1/lun1" \target_iqn="iqn.2001-04.com.example:vg1" \op monitor interval="10"primitive p_ip_vg1 ocf:heartbeat:IPaddr2 \params ip="192.168.0.100" cidr_netmask="24" \op monitor interval="10"group g_vg1 \p_lvm_vg1 p_target_vg1 p_lu_vg1_lun1 p_ip_vg1order o_drbd_before_vg1 inf: \ms_drbd_vg1:promote g_vg1:start112 / MAY 2012 / WWW.LINUXJOURNAL.COM

INDEPTHFigure 1. Normal operation: mike hosts iSCSI target; nancy receives synchronous replication stream.colocation c_vg1_on_drbd inf: \g_vg1 ms_drbd_vg1:MasterThis configuration creates an iSCSItarget using the STGT implementation—the default on RHEL/CentOS and anavailable choice on SLES, Debian andUbuntu—which uses the iSCSI QualifiedName “iqn.2001-04.com.example:vg1”.The target contains one Logical Unit withthe Logical Unit Number (LUN) 1, and isavailable through the portal IP address192.168.0.100 using the default iSCSITCP port of 3260.As usual, you can check Pacemaker’sstatus with the crm_mon utility:============Last updated: Sun Mar 4 21:59:23 2012Last change: Sun Mar 4 21:58:38 2012 via crm_attribute on mikeWWW.LINUXJOURNAL.COM / MAY 2012 / 113

INDEPTHFigure 2. mike has failed; nancy transparently takes over iSCSI target. DRBD replication is suspended.Stack: openaisCurrent DC: nancy - partition with quorump_target_vg1 (ocf::heartbeat:iSCSITarget): Started mikep_lu_vg1_lun1 (ocf::heartbeat:iSCSILogicalUnit): Started mikeVersion: 1.1.6-4.el6-89678d4947c5bd466e2f31acd58ea4e1edb854d5p_ip_vg1(ocf::heartbeat:IPaddr2): Started mike2 Nodes configured, 2 expected votes6 Resources configured.============Master/Slave Set: ms_drbd_vg1 [p_drbd_vg1]Masters: [ mike ]Slaves: [ nancy ]Online: [ mike nancy ]Resource Group: g_vg1p_lvm_vg1 (ocf::heartbeat:LVM): Started mikePacemaker, LVM and DRBD now jointlyensure that this specific target and LUNalways are available on one of the clusternodes. If the node hosting the target goes114 / MAY 2012 / WWW.LINUXJOURNAL.COM

INDEPTHClones and Master/Slave SetsBesides regular cluster resources (“primitives” in Pacemaker parlance), Pacemaker alsosupports two advanced resource types: clones and master/slave sets.Clones are Pacemaker’s “define once, run anywhere” feature. They allow you to definea specific resource—say, a local monitoring dæmon—just once, and then merely set thenumber of instances of this dæmon that you want to run in the cluster. You define clonessuch that they “wrap” a primitive. The example below defines a cluster monitoringresource that will run in three incarnations, but only one instance is ever allowed onany node. Thus, if more than three nodes are on-line, Pacemaker can place one of eachinstance on any three nodes available. Should the cluster fall short of three nodes,Pacemaker would run as many instances as possible, one per node:primitive p_mon ocf:pacemaker:ClusterMon \op monitor interval="10"clone cl_mon p_mon \meta clone-max=3 clone-node-max=1Master/slave sets are a special case of clones. In master/slave sets, Pacemaker “promotes”one or more of the clone instances to a master role, with the other instances serving asslaves. What exactly differentiates a master from a slave is entirely application-dependent—Pacemaker simply operates on generic “promote” and “demote” actions. The Pacemakerresource agent to manage DRBD resources is an example of a master/slave set: its Masterand Slave roles map to the Primary and Secondary resource roles in DRBD proper:primitive p_drbd_vg1 ocf:linbit:drbd \params drbd_resource="vg1" \op monitor interval="30" role="Slave" \op monitor interval="10" role="Master"ms ms_drbd_vg1 p_drbd_vg1 \meta interleave="true" notify="true"The interleave and notify meta attributes are specific to clones. For more information seethe Resources for this article.WWW.LINUXJOURNAL.COM / MAY 2012 / 115

INDEPTHFigure 3. mike has recovered; nancy continues to host iSCSI target. DRBD resynchronizes andcontinues to replicate in the reverse of its original direction.down—for reasons of node failure orscheduled maintenance—the entire stacksimply shifts over to the other node:2 Nodes configured, 2 expected votes6 Resources configured.========================Last updated: Sun Mar 4 22:01:15 2012Online: [ nancy ]OFFLINE: [ mike ]Last change: Sun Mar 4 22:01:12 2012 via crm_attribute on mikeStack: openaisResource Group: g_vg1Current DC: nancy - partition with quorump_lvm_vg1(ocf::heartbeat:LVM): Started nancyVersion: 1.1.6-4.el6-89678d4947c5bd466e2f31acd58ea4e1edb854d5p_target_vg1 (ocf::heartbeat:iSCSITarget): Started nancy116 / MAY 2012 / WWW.LINUXJOURNAL.COM

INDEPTHp_lu_vg1_lun1 (ocf::heartbeat:iSCSILogicalUnit): Started nancyp_ip_vg1 (ocf::heartbeat:IPaddr2): Started nancyMaster/Slave Set: ms_drbd_vg1 [p_drbd_vg1]Masters: [ nancy ]Stopped: [ p_drbd_vg1:0 ]Adding more LUNs to the stackis a simple matter of creating moreLogical Volumes in the DRBDbackedVG, then adding additionalocf:heartbeat:iSCSILogicalUnitresources to the g_vg1 group.Ensuring Seamless Failover TransparencyThe iSCSI standard prescribes a perconnectioniSCSI parameter, Time2Retain,that governs the length of a connectioninterruption that the initiator musttolerate. If the target goes away for alimited time—as in the case of a shortnetwork hiccup—the initiator simplyblocks I/O on the affected iSCSI links.I/O resumes when the target returns.Only if the Time2Retain expires doesit drop the connection and flag an I/Oerror. You can use this feature in buildinghighly available iSCSI targets: as longas the cluster manages to completefailover within the time frame allottedby Time2Retain, the initiators simplyresume I/O when the target comes aliveon the second node. DRBD, for its part,guarantees through its synchronousreplication that the new node has exactlythe same data as the old one, so thisresumption of I/O is perfectly safe as longas no volatile caches are involved.The iSCSI initiator and target negotiatethe Time2Retain parameter whenconnecting initially. Both present theirpreferred value, and the minimumpresented value wins. Sadly, manyiSCSI initiator implementations (theubiquitous open-iscsi included) set thisvalue to zero by default, thus forgoinga useful high-availability feature builtright in to the iSCSI standard. However,they usually present a simple meansActive/Active ClusteringThe example configuration in this article is Active/Passive: the iSCSI target portal group isonly ever active on one node; the other serves as a standby and DRBD replication target.Expanding this to an Active/Active configuration is straightforward: dedicate one more SCSILUN on each host for DRBD replication, and duplicate the example setup with a new DRBDresource, an additional LVM Volume Group, another iSCSI target-portal group, and as manyLVM LVs/iSCSI Logical Units as you wish. Pacemaker then has the capability to balance theresource groups between the nodes, improving hardware utilization.WWW.LINUXJOURNAL.COM / MAY 2012 / 117

INDEPTHDRBD 8.4Syntax ChangesDRBD 8.4 users will notice that syntaxchanges apply to certain commandsmentioned in this article. Specifically,in DRBD 8.4 drbdadm -- --forceprimary turns into drbdadmprimary --force . At thetime of this writing, DRBD 8.4 users reportedstability issues and performance regressionson the relevant project mailing lists; hence,DRBD 8.3 remains the preferred DRBDrelease for most users.of re-enabling it—in open-iscsi it’s theDefaultTime2Retain parameter in theiscsid configuration file:node.session.iscsi.DefaultTime2Wait = 60node.session.iscsi.DefaultTime2Retain = 5Setting these configuration parametersenables target failover that is smooth,transparent and almost interruption-free(except for a short, nonfatal I/O hang)to initiators.The highly available, replicatedstorage stack I explored in this articleis enormously solid, runs in productionprobably thousands of times overand has wide support from a varietyof distributions. However, it doeshave its shortcomings when there arerequirements for multinode replication,wide-area replication and scalability.In the next and final installment of thishigh-availability series, I will explorealternative approaches to replicatedstorage that promise to fix these issues.■Florian Haas is a Principal Consultant and co-founder at hastexo, anindependent professional services organization specializing in Linuxhigh availability, storage replication and the open-source cloud. Priorto launching hastexo, he spent four years at Linbit, the companybehind the DRBD Project. He is the original author of the DRBD User’sGuide and continues to contribute to the project; however, neither henor his company are affiliated or associated with Linbit.ResourcesIET Project Web Site: http://iscsitarget.sourceforge.netSTGT Project Web Site: http://stgt.sourceforge.netSCST Project Web Site: http://scst.sourceforge.netLIO Project Web Site: http://www.linux-iscsi.orgDRBD Community Web Site: http://www.drbd.orgDRBD User’s Guide (DRBD 8.3 version):http://www.drbd.org/users-guide-8.3“Clones” (Pacemaker Configuration Explained):http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-resource-clone.html“Options” (Pacemaker Configuration Explained):http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/ch10s02s02.html“Notifications” (Pacemaker Configuration Explained):http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/ch10s02s08.html118 / MAY 2012 / WWW.LINUXJOURNAL.COM

EOFDOC SEARLSFreedomand IdentityIt’s time for the former to liberate the latter.Two big things happen for me thismonth (May 2012). One is thatmy first book comes out. (That is,the first one written by me alone, withno co-authors.) The title is The IntentionEconomy: When Customers Take Charge.The publisher is Harvard Business ReviewPress, and the publication date is MayDay. The other is IIW, the Internet IdentityWorkshop in Mountain View, California.It’s also on May Day, plus the two daysfollowing. This will be our fourteenth IIW(we have two each year). Like the others,it will be an unconference, long onconversation, collaboration and hacking,and short on BS (no speakers, no panels,no keynotes, no booths). And I’m sureit will be, as all of them are, better andmore productive than the last one.There is a thread that runs betweenthese two things—the book andidentity—going back to 1994, when PhilHughes began vetting the idea of doinga magazine about free software. Thatmagazine is the one you’re reading now.Since the beginning of that thread,two things have been clear. One is thatfreedom and identity are both personal.The other is that neither are simple.“Free will does not mean one will, butmany wills conflicting in one man”,Flannery O’Connor wrote (in Wise Blood).“Freedom cannot be conceived simply.”Same for identity. Writes Michael Ventura(in Shadow Dancing in the USA):...there may be no more importantproject of our time than displacingthe fiction of monopersonality. Thisfiction is the notion that each personhas a central and unified “I” whichdetermines his or her acts. “I” havebeen writing this to say that I don’tthink people experience life that way.I do think they experience languagethat way, and hence are doomed tospeak about life in structures contraryto their experience.120 / MAY 2012 / WWW.LINUXJOURNAL.COM

EOFWe hold these identity-defining contexts like cardsin a hand, and we play them all differently.The “fiction of monopersonality”struck home for me when I read thatpassage in the early 1980s. Who I was to mymother, my wife, my kids, my readers, myco-workers—and even to my self in differentsettings—was different. What we callsanity is unifying all our different identitiesbehind a first-person pronoun: “I”, “me”,“my” or “mine”. Even our names are morevaried than those. In my own case I am“Doc” to some people, “David” to others,“Dave” to others, “Searls” to others, and“dsearls” or “@dsearls” on the Web. InJunior High, I was “Sleepy”. (Thus, I haveshared nicknames with two of the SevenDwarfs.) Ventura’s point is not just thatmonopersonality is fiction and that weexperience life and language differently,but that who we are is context-dependent,and there are many of those, most ofwhich take the form of relationships withothers, all of which come and go—andchange along the way.I recently listened to a tape of aconversation I had with my mother manyyears ago. I found myself wishing thatI could not only talk to her again, butalso that I could again be who I was withher. But I couldn’t, because that partof me—the son-self in my portfolio ofpolypersonal identities—died when sheleft the planet in 2003. I was already inmy fifties by then, but I was still her son:to her, and to me. My father, with whomI also had a fun and loving relationship,died in 1979, half my life ago. So I amnow nobody’s son.My wife and I have wedding ringsinside each of which is inscribed “thecouple decides”. Who I am as a husbandis a member of a relationship that has anidentity of its own, but one sustained bytwo sovereign selves.We hold these identity-definingcontexts like cards in a hand, andwe play them all differently. Where itgets complicated is with the cards—often literal ones—that are loanedto us. These are what Devon Lofretto(posting on the ProjectVRM list) calls“administrative” identities. In my walletI have administrative identities fromthe State of California, United Airlines,Harvard University, the AutomobileClub of Southern California, VISA,Costco/American Express, Chase Bankand Blue Cross. Not in my wallet,but in my possession, are otheradministrative identities from HudsonCounty, New Jersey (where I wasborn), UC Santa Barbara and dozens ofretail establishments. If you count thehundreds of login/password combinationsrecalled by my browsers, I total severalhundred administrative identities inall. None of them are who I am inWWW.LINUXJOURNAL.COM / MAY 2012 / 121

EOFSolving the conflict between sovereign sourceand administrative identities has been theprimary challenge for digital identity developmentfrom the start.the deepest and most original sense:the one that was born free. That is,none of them are what Devon calls my“sovereign source” identity, because theadministrator of that identity providesthe whole context, starting with whatthey call me, which often aligns withwhat my parents named me, but doesn’talways. In either case, the a prioriresponsibility is theirs, not mine. We arenot a couple, and the couple can decideonly what the administrative party aloneallows it to decide.Solving the conflict betweensovereign source and administrativeidentities has been the primarychallenge for digital identitydevelopment from the start. Becausewe are so accustomed to being a zillionnamespaces in a vast administrativesphere (now as large as the Webitself), we acquiesce to the insanityof it. In fact, we actually mistake thenormative nature of this insanity forits opposite. That’s why we keep tryingto come up with ways to managemultiple logins and passwords—orto create yet another administrativesystem (the largest of which today areGoogle, Facebook and Apple)—ratherthan to replace the whole crazy systemwith one that builds upward from oursovereign source identities.Albert Einstein said no problem canbe solved at the level of consciousnessthat created it. That means we can’tmeet the challenge of fully enablingsovereign source identity at theadministrative level. It just can’t bedone. There is no administrative answerto the identity mess.Yes, we do need administrativecompliance to whatever answers wecome up with. But our answers needto come from our free, sovereign andindependent selves. Devon lays outour challenge this way: “How can youadminister an identity ecosystem ifyou cannot properly define sovereignsource authority? Understanding thestructural power of John Hancockshould be a job requirement in theAmerican government.”We may not have defined “sovereignsource” yet, but we have experienced it,and that helps. Wrote Walt Whitman:You make too much of articulation.Encompass worlds but never try to encompass me.122 / MAY 2012 / WWW.LINUXJOURNAL.COM

EOFI crowd your noisiest talk by looking toward you.Writing and talk do not prove me.I carry the plenum of proof and everything elsein my face.With the hush of my lips I confound the topmost skeptic.Also:I know this orbit of mine cannot be sweptby a carpenter’s compass,I know that I am august.I do not trouble my spirit to vindicate itselfor be understood.I see that the elementary laws never apologize.Strong stuff. But can we make thatinto what the wonks call “policy”?No, we can’t, because the wonks’answer will be an administrative one. Butmost of us here aren’t wonks. Instead,we write code. Here in Hackerland, wechange practice, not law.When John Hancock signed theDeclaration of Independence, he wasdeclaring the independence of a nationwith a signature that also declared hissovereign self. He presented his plenumand proof in script several decades beforeWhitman declared his own in verse.My goal for the last six years has beento change practice through inventionsLinux JournaLnow availablefor the iPad andiPhone at theApp Store.linuxjournal.com/iosFor more information about advertising opportunities within Linux Journal iPhone, iPad andAndroid apps, contact Rebecca Cassity at +1-713-344-1956 x2 or ads@linuxjournal.com.

y his colleague Christina Cacioppo(http://www.usv.com/2011/11/what-comes-next.php). She writes:One reason to create firms is thecoordination and signaling problemsof situations with imperfectinformation and transaction costs.As technology increases informationflows and decreases transactioncosts, individuals can leave theirold employers and strike out ontheir own. Their livelihoods will stilldepend on providing valuable servicesin exchange for fees, but they’ll doso as freelancers—and on their own,they’ll capture more of the valuegenerated by their work....These free agents, disaggregatedand newly empowered, canpromote and sustain themselveswith new tools....Sound familiar? How many LinuxJournal readers (especially ones whowrite code) began doing this longago? Christina also visits the subjectof identity:Advertiser IndexThank you as always for supporting ouradvertisers by buying their products!ADVERTISER URL PAGE #Acenet http://www.ace-host.net 15Drupal Business http://www.drupalsummit.com/city/vancouver 33Summit VancouverEmac, Inc. http://www.emacinc.com 45EmperorLinux http://www.emperorlinux.com 37iXsystems, Inc. http://www.ixsystems.com 7Linaro Connect Q2.12 http://connect.linaro.org/register 17Logic Supply, Inc. http://www.logicsupply.com 48, 59Lullabot http://www.lullabot.com 2Microway, Inc. http://www.microway.com 94, 95O'Reilly Fluent Conference http://fluentconf.com/fluent2012 105Silicon Mechanics http://www.siliconmechanics.com 3SouthEast LinuxFest http://www.southeastlinuxfest.org 119Usenix FederationConference Weekhttps://www.usenix.org/conference/fcw1253ATTENTION ADVERTISERSBetween identified, liberatedindividuals and the nameless,faceless drones of Mechanical Turklies identity: does it matter whoperforms the task at hand? If theworker’s background, skills, orexperience matter, there’s likely toThe Linux Journal brand’s following hasgrown to a monthly readership nearlyone million strong. Encompassing themagazine, Web site, newsletters andmuch more, Linux Journal offers theideal content environment to help youreach your marketing objectives. Formore information, please visithttp://www.linuxjournal.com/advertising.WWW.LINUXJOURNAL.COM / MAY 2012 / 125

EOFbe higher variance in demand fora particular person’s services, andfree agents will be sought after andchosen by reputation on servicesbuilt for those purposes. Less-skilledpeople are likely better suited fortasks for which identity doesn’tmatter, and other marketplacesthat don’t include a concept ofreputation will provide access to aglobal pool of workers.In other words, there is a growingdistinction between dronework byfaceless sources of simple labor anddistinctive constructive work by trulyfree (and faceful) agents. This post isfor the latter.To get full respect for the freeagents that all of us were born as, thefreedom-loving hackers among us needto help build the tools that give oursovereign sources full authority overthe administrative namespaces currentlymanaged by second and third parties.(We’re the first parties here.)A number of these tools are alreadyin the works, and many of them areopen source. Singly’s Locker Project(http://lockerproject.org), for example,is an open-source personal data lockeron which apps are already being built.Individuals (or their friendly programmers)can write rules for managing relationshipsthrough countless potentially interrelatedevents on the Net, using KRL (kineticrules language), written by Phil Windley(http://windley.com). KRL and its rulesengine are both open source (at GitHub).On the legal front, Joe Andrieu, Judi Clarkand Iain Henderson (along with morethan 40 others) have been working onan Information Sharing Agreement(http://kantarainitiative.org/confluence/display/infosharing)that treats sovereign and administrativeidentities as equals from the start.The group and its work are both open.Drummond Reed and colleagues atConnect.me have been working onthe Respect Trust Framework(http://connect.me/c/trust)—builtaround freely made assertions bysovereign individuals regarding thetrustworthiness of others—which islisted with the Open Identity Exchange(OIX) and hosted by the Personal DataEcosystem Consortium. That’s open too.(That is, if others want to use the sameframework, they can. Individuals’ datais also their own.)These developers will be at IIW.Same will go, I hope, for the VCs I justmentioned as well. (Christina and hercolleague Brad Burnham were bothat the last IIW.) So, if you’re up forraising the Barn of the Sovereign Selfwith the rest of us, come give us evenmore to celebrate on May Day.■Doc Searls is Senior Editor of Linux Journal. He is also afellow with the Berkman Center for Internet and Society atHarvard University and the Center for Information Technologyand Society at UC Santa Barbara.126 / MAY 2012 / WWW.LINUXJOURNAL.COM