TWISTING - Linux Magazine

TWISTING - Linux Magazine

COVER STORYTwistedwww.photocase.comCreating network apps with TwistedTWISTINGPYTHONThe Twisted framework makesit so easy to create networkawareapplications in Python.Twisted speaks all the majorInternet protocols, from mailthrough chat, and it can handleencryption. We’ll show youhow to set up a personal webserver with Twisted. BYMARKUS FRANZMost programmers eventuallyface the task of adding networkcommunication capabilities totheir applications. If it is simply a case ofmanipulating web content, there aresimple answers, but more complex functions,such as adding full-fledged emailcapabilities or a complete web server, involvemuch more effort.Although the Python standard libraryhas modules to match most walks of aprogrammer’s daily life (batteries included,in Python-speak), special applicationsrequire external packages.Twisted [1] is a well-organized and powerfulcollection of modules for addingnetworking capabilities to Python programs.If you are coding an email client withmultiple protocol support (POP 3, SMTP,IMAP), the Twisted framework will removethe need to start from scratch.Twisted Mail has all the major mail protocols.Twisted also has ready-to-runmodules for SSH, SFTP, HTTP (includingHTTP/ 1.1), DNS, NNTP, and Jabber.If you insist on re-inventing thewheel – by implementing your ownprotocols for example – the twisted.cred and twisted.spread Twisted modulescan help to simplify the job.AsynchronousNetworkingTwisted is basically an asynchronousnetwork framework. In contrast toother libraries, the Twisted functions donot block when they are called. Applicationsjust keep on running until they aretold that the required data is available.twistedtwisted.conchtwisted.webtwisted.web2twisted.mailtwisted.namestwisted.newstwisted.wordsAlthough Python’s standard librariescould handle this (the asyncore modulehas basic functionality for switching betweenmultiple I/ O channels within athread), Twisted implements this designat a higher level in its protocols, interfaces,and components. This lets programmerswrite network applicationsthat do without additional processes andthreads while at the same time handlingmultiple I/ O channels.Highly ModularTo use Twisted to make your own softwarenetwork-aware, you first need tofind out which part of the frameworkyou will need. The developers splitTwisted into multiple sub-projects whenmoving from version 1 to 2, with the aimof adding more clarity. Table 1 has a listof the most important elements, andthere is a complete list at [2].Twisted requires the Zope Interfacemodule, which implements interfacesTable 1: Major Twisted ModulesFramework for asynchronous applications, the basis for all Twisted subprojectsImplementation of the SFTP and SSH protocols for clients and serversHTTP protocol for clients and serversSupport for the HTTP/ 1.1 protocol as a server framework; this package is stillunder development at present, and should not be used for critical applicationsImplementation of the SMTP, IMAP, and POP protocols for clients and serversDNS protocol support for clients and serversNNTP protocol for clients and serversModule for chat or instant messaging applications3030 ISSUE 64 MARCH 2006 W W W. L I N U X- M A G A Z I N E . C O M

TwistedCOVER STORYthat the Python language core lacks. Ifyou just need to use a single project,such as Twisted Web or Twisted Mail,you can download the required modulefrom the project homepage. Alternatively,Twisted Sumo [3] has all (stable)modules, including a Zope interface.After unpacking, give the install command. If the Zopeinterface is not installed, the installerwill display an error message and quit.The twisted.cred module handles authenticationin client-server communications.It allows multiple network protocolsto connect to a system, to authenticate,and to exchange data. For example,POP 3 support in Twisted provides acombination of the username and passwordto open the requested mailbox.The so-called Perspective Broker is importanthere; the broker provides accessto remote objects and implements objectcopying, referencing, and caching.Database Connectiontwisted.enterprise provides a databaseinterface that is compatible with Python-DB-API 2.0. This makes access toMySQL, Oracle, or PostgreSQL databaseschild’s play. The module uses an asynchronousinterface, which can run inmultiple threads, without losing threadsafety in an event-based Twisted mainloop. The main loop occurs within thetwisted.internet module and is known asthe reactor. It implements the infiniteloop of the program in which Twistedhandles various events. The reactor providesthe underlying interface to Twisted’smajor internal capabilities, such asnetwork connections, threading, orevent handling.Other modules, such as twisted.protocolsor twisted.manhole, are very rarelyneeded in practical applications. Conchimplements version 2 of the Secure Shellprotocol for Twisted. The how-to at [5]discusses the implementation of an SSHclient with Conch in a few simple steps.Web or application servers are one ofTwisted’s most interesting fields of applicationand use twisted.web or twisted.web2. The powerful template toolkit forthis task is titled Nevow [6]. TwistedTwistedFigure 1: Web request process within theTwisted framework.Client (Browser)HTTP»web.server.Site«»web.server.Request«»web.resource.Resource«also has the full range of functions requiredfor programming HTTP clients.The twisted.web API supports multipleadvertisementW W W. L I N U X- M A G A Z I N E . C O MISSUE 64 MARCH 200631

COVER STORYTwistedabstraction layers: from simple web servers,through session support, interactingapplication servers, and distributed websites– the options are many.When a client request reaches a webserver, the server creates a request objectand hands the object back to the resourcesystem, which creates the response(Figure 1).Besides twisted.web, there is also thetwisted.web2 module, which is stillunder development. The major improvementsare as follows:• HTTP 1.1 support• Internal and predefined output filters,for gzip-compressed serving of webpages• Separation of high and low level requesthandling• Correct analysis of HTTP headers• Vastly improved URL rewriting, if usedin combination with a proxyAlthough there are major enhancementsin twisted.web2 in comparison to theprevious version, the developers do notrecommend using the new module atpresent, one of the reasons being thatthe new module is slower than its predecessor.The last major module is Twisted Mail,which implements SMTP, POP 3, andIMAP 4. Besides protocols, the modulecan also read and write the Maildir mailboxformat. There is also a preconfiguredcombination of SMTP and POP 3for mail servers and virtual hostingsystems. And Twisted Mail understandsmost Sendmail options, which can come01 # Load modules02 from twisted.internet importreactor03 from twisted.web import04static, server05 # Set root directory06 my_server = static.File('/var/07Listing 1: Simple WebServerwww/htdocs')08 # Launch web server on port777709 reactor.listenTCP(7777,server.Site(my_server))10 handy for downwardly compatible applications.01 # Load modules02 from twisted.internet import reactor03 from twisted.web import static, server, twcgi0405 # set root directory06 my_server = static.File('/var/www/htdocs')0708 # Evaluate Perl scripts09 class PerlScript(twcgi.FilteredScript):Frozen ServerThe Twisted framework has a collectionof command line programs, which prepareTwisted applications for special scenarios[7]. The mktap and tapconverttools create Tap, Tas, or Tax formattedfiles from the Python source code. Thecode can then be used with the variousTwisted servers, for example: Web, FTP,or IRC.The twistd tool brings Tap files to life.Strictly speaking, twistd is not requiredto run Twisted applications, but it doesmake things easier, as it takes control ofthe reactor and handles starting andquitting the application. Additionally,twistd supports the selection of a differentreactor type, allowing an applicationto run in daemon mode or write logfiles.The tap2deb and tap2rpm programscompress finished server applications fordistribution in Debian or RPM packageformats. The tools automatically generatescripts for installing and removingthe server.Listing 1 shows a simple web serverand demonstrates how powerful Twistedis. You could use this code to enableweb-based configuration of your application.The first couple of lines load the requiredmodules. Line 6 creates a newserver with a web root directory forHTML documents – /var/www/htdocs inour example. The following instructiontells the reactor to listen for requests onport 7777. The calllaunches the web server.Listing 2 demonstrates a web serverthat adds a number of options to thebasic framework in Listing 1. First of all,lines 9 through 11 enable Perl script support:we want the server to hand fileswith a .pl extension to the Perl interpreter,/usr/bin/perl, and to return itsoutput. Support for PHP or other languagescould just as easily be enabled atthis point.The putChild() method in line 14 setsthe CGI directory to /var/www/cgi-bin,10 filter = '/usr/bin/perl' # path to Perl interpreter11 my_server.processors = {'.pl': PerlScript}1213 # Set and enable CGI directory14 my_server.putChild('cgis', twcgi.CGIDirectory('/var/www/cgi-bin'))1516 # Directories for other targets17 my_server.putChild('doc', static.File('/var/www/doc'))1819 # Index files20 my_server.indexNames = ['index.html', 'index.htm', '']2122 # Launch web server on port 777723 reactor.listenTCP(7777, server.Site(my_server))24 2: Extended Web Server32 ISSUE 64 MARCH 2006 W W W. L I N U X- M A G A Z I N E . C O M

TwistedCOVER STORYthus allowing access to the scripts athttp://servername:7777/cgis. Themethod also specifies the path for thedoc directory, /var/www/doc. The index-Names variable specifies which files theserver looks for during a directory request.The order is defined by the filenamesspecified.The following Twisted tools could beused as an alternative to the web servercode:mktap web U--path /var/www/htdocs U--port 7777twistd --file web.tapmktap creates a pre-configured Tap file.In this case, mktap sets the root directoryto /var/www/htdocs and the port to7777. The results end up in the web.tapfile, which the twistd server programuses as a parameter. After launching, theserver, the logfile, twistd.log, and the PIDfile, which can be used to kill the server(kill `cat`), reside in the currentdirectory. mktap web -help gives youmore information on the available options.Simple but PowerfulThis article can only give you a quickoverview of Twisted. The framework,along with Twisted Mail, Conch, TwistedWeb(2), and the other sub-projects, formthe basis for many professional networkapplications – even NASA uses TwistedMatrix [8].Helping Python applications reach forthe stars makes life a lot easier for overworkedprogrammers back here onearth. Twisted clearly reduces the effortinvolved in developing a client or serverby removing the need to implementTHE AUTHORMarkus Franz runs a business advisoryservice for companies involvedin IT projects. He develops Internetsearch engines, including Metager2,, which was implementedin Python throughout.Markus is just 17 years old and stillattends the Armin Knab High Schoolat Kitzingen, Germany.known protocols. This gives programmerssecure, fast, stable, and flexiblenetwork applications at the click of amouse. ■INFO[1] Twisted Matrix project page:http:// www. twistedmatrix. com[2] Overview of all Twisted projects:http:// www. twistedmatrix. com/projects[3] Twisted Sumo:http:// twistedmatrix. com/ projects/ core[4] Zope Interface:http:// www. zope. org/ Wikis/ Interfaces[5] SSH client with Conch:http:// twistedmatrix. com/ projects/conch/ documentation/ howto/ conch_client. html[6] Nevow (template system):http:// divmod. org/ projects/ nevow[7] Tools: http:// twistedmatrix. com/projects/ core/ documentation/ howto/basics. html[8] Twisted users: http:// services/ successadvertisement

More magazines by this user
Similar magazines