Hacking the Linux Automounter

Proceedings of theLinux SymposiumVolume TwoJuly 19th–22nd, 2006Ottawa, OntarioCanada

Conference OrganizersAndrew J. Hutton, Steamballoon, Inc.C. Craig Ross, Linux SymposiumReview CommitteeJeff Garzik, Red Hat SoftwareGerrit Huizenga, IBMDave Jones, Red Hat SoftwareBen LaHaise, Intel CorporationMatt Mackall, Selenic ConsultingPatrick Mochel, Intel CorporationC. Craig Ross, Linux SymposiumAndrew Hutton, Steamballoon, Inc.Proceedings Formatting TeamJohn W. Lockhart, Red Hat, Inc.David M. Fellows, Fellows and Carr, Inc.Kyle McMartinAuthors retain copyright to all submitted papers, but have granted unlimited redistribution rightsto all as a condition of submission.

Hacking the Linux Automounter—Current Limitationsand Future DirectionsIan Maxwell KentRed Hat, Inc.ikent@redhat.comJeff MoyerRed Hat, Inc.jmoyer@redhat.comAbstractThe IT industry is experiencing a considerableshift from proprietary operating systemsto Linux. As a result, the features and functionalitythat people have come to expect ofthese systems now must be provided for onLinux. An automounter provides a mechanismfor automatically mounting file systems uponaccess, and umounting them when they are nolonger referenced. The Linux automounter isnot feature-complete and there are cases whereLinux autofs is just plain incompatible with implementationsfrom other proprietary vendors.In order to solve the current automounter limitations,we start by developing an understandingof how things work today. We explainthe basic configuration of autofs for a clientmachine using simple examples. Then walkthrough the the internals for basic operations,such as the mounting, or lookup, of a directoryand the umounting, or expiry, of a directory.This includes a description of where autofsfits into the VFS layer. Next we discussthe two main deployment difficulties. The firstis that the Linux automounter implements directmount maps in a way that is incompatiblewith that of every other implementation.We discuss the desired behavior and compareit with that of the Linux automounter. We willthen discuss the current development effort toovercome this by extending autofs and its kernelinterface. The second major problem surroundsthe use of multi-mount entries for the/net, or -hosts mount maps. Because ofthe nature of multi-mount maps, the Linux implementationmounts and umounts these directoryhierarchies as a single unit. This meansthat clients mounting exported filesystems fromlarge servers can experience resource starvation,causing failed mounts. The root problemis described and we show how the kernel andLinux automounter can be modified to addressthis issue also. We conclude with a review ofthe progress of the work outlined above andgive a summary of future directions.1 IntroductionWith even a modest amount of information,network clients often need many mount entriesin their tables to make the organization’s informationavailable. To make matters worse, themount tables often change. The administrativeoverhead is not workable. This leads to heavyuse of an automounter in many enterprises.An automounter provides the ability to managemount tables centrally, automatically mountingentries on demand and umounting them aftera predefined period of inactivity. In addition tothe reduction in administrative overhead, an automounterprovides a dramatic reduction in the

2006 Linux Symposium, Volume Two • 47mount-optionsmount-options is an optional comma separatedlist of mount options to be applied tothe entries in the map unless entries in themap specify their own options.Lines beginning with a # are comments and areignored. Long lines may be broken by quotingthe new line character with a backslash, as iscommon practice in configuration files.The special mount point /- is reserved to indicatethat the map is a direct mount map and isnot associated with any specific top-level directory.2.2 Mount mapsMount maps consist of two types—indirect anddirect—and have the following basic format:keykey [mount-options] locationkey is the name used to look up mount tableentries in the map. For indirect mountentries, this is the name of the directoryupon which the mount will be made. Fordirect mount entries, this is the full pathleading to the directory upon which themount will be made.mount-optionsmount-options is an optional commaseparatedlist of mount options to be appliedto the map entry.locationlocation specifies the file system that is tobe mounted on key. It can be a single filesystem or a number of file systems to selectfrom using availability and proximitymetrics. It may also consist of multiplekey [mount-options] locationoffsets that each must start with a slash (/).If the first offset is /, then it is optional.These offset mount entries are referred toas multi-mount entries in Linux autofs.There are a number of standard macro substitutionsavailable for use in location specifications.They are commonly used in multiple architectureenvironments. A description of thosenormally available can be found in [2] on page190. For those understood by Linux autofs, seeautofs(5).As in the master map, lines beginning with a #are comments and are ignored, and long linesmay be broken by quoting the new line characterwith a backslash.A map key of * denotes a wild-card entry. Thisentry is consulted if the specified key does notexist in the map. A typical wild-card entrylooks like this:* server:/export/home/&The special character & will be replaced by theprovided key. So, in the example above, alookup for the key foo would yield a mount ofserver:/export/home/foo.The timeout on mounts points defaults to tenminutes and can be changed using a commandline option when the service is started.3 Linux automounter—autofsThe Linux automounter differs in relatively fewways from traditional Unix automounter implementations.In fact, all of the information providedin the last sections regarding configurationdata apply to the Linux automounter as

48 • Hacking the Linux Automounter—Current Limitations and Future Directionswell. This section begins with a description ofthe Linux-specific details of the master map,and then moves on to the architecture of theLinux automounter.3.1 Linux autofs master mapThe Linux autofs master map syntax is a superset of the standard automount master map syntax.This is partly because Linux autofs doesnot utilize the name service switch to locate thesource of maps and so must allow it to be specified.The syntax is:mount-point \[maptype:]map-name \[mount-options]The fields above are the same as those describedin Section 2.1 (“The master map”), exceptfor the maptype, which can be one offile, program, yp, nisplus, hesiod orldap. The daemon supports the specificationof a map format within the maptype parameter.It can be sun or hesiod, but the initscript doesn’t cater for it. The default formatis sun, and it is a subset of the standard sunautomount map format. Linux autofs understandsmuch of this map format, and when a fullimplementation of direct mounts is added, theonly things missing will be special maps suchas the -hosts and -null.3.2 ArchitectureThe automounter is implemented in two mainparts: a user-space daemon, which is responsiblefor parsing map options and issuing mountand umount commands, and a filesystem, implementedin the kernel. The daemon is furtherbroken up into the daemon proper and a set ofloadable modules. To understand how the daemonoperates, we will walk through the daemonstartup for a minimal setup.Consider the following auto.master map:/net/etc/auto.netWe will not show the contents of the programmap, auto.net, as it is shipped with autofs.Autofs startup begins with the init script.This script parses the auto.master map andspawns one automount daemon for each mountpoint listed. The example given above will resultin an automount command with the followingparameters:/usr/sbin/automount \/net program /etc/auto.netAs shown above, the daemon takes as its optionsa mount point, the type of the map to beloaded, and the name of the map to be loaded.Now we will look at the loadable modules.There are three types of modules: lookup,parse, and mount. Lookup modules are usedto look up a given key in a map. The lookupmodule has code that understands how to getinformation from a map source. For example,lookup_file.so is able to read in entriesfrom a file map. Map entries are storedas a key value pair. The key, as noted above,corresponds to a directory. The parse moduleis then responsible for parsing the value part ofthe key value pair. Finally, the mount moduletakes care of doing the actual mounting. Thismodule has to know how to pass arguments onto the mount command. In the case of NFS, thismodule is also responsible for parsing replicatedserver entries.Returning to the example above, the daemonknows that it needs to load the lookup_

2006 Linux Symposium, Volume Two • 49program module, since the program map typewas specified in the command line. It callsthe module’s lookup_init routine, passing amap format (or none, in this case), and all argumentsthat the daemon itself did not process.These leftover arguments are considered to bemap arguments.The lookup module will perform its initializationand hand a context pointer back to thecaller. Before returning, though, it loads theparse module, calling its parse_init function.It then passes the map format down, aswell as any options it did not handle. The parsemodule will load the mount_nfs module, if ithasn’t already been loaded. This module is alwaysloaded, since the primary file system typemounted via autofs has historically been NFS.3.3 Multi-mountsMulti-mount entries allow the user to specify adirectory hierarchy that will be mounted. Forexample:net generates multi-mount entries on thefly, and the daemon mounts them when/net/ is accessed. The is used as the key.3.4 VFS interfaceTo understand the kernel interface used by autofs,it is necessary to know a little about theVirtual Filesystem Switch (VFS). The VFS is asoftware layer that handles all system calls relatedto standard Unix file systems. It does thisby defining several data structures that containinformation about the file system and objectsthat provide callback functions. The VFS usesthe callback functions to carry out standard filesystem operations. The primary objects are thesuperblock, the inode, the dentry, and the fileobject. For the interested reader, a descriptionof the VFS, its data structures, and the operationsthey define can be found in Chapter 12 of[7].mydir -rw \/ server:/export/mydir \/src server2:/export/home/src \/tmp :/usr/tmpThis example demonstrates how to cobble togethera single directory structure from multipleservers. One point to note here is thatthe mydir directory contains both an NFSmountedfile system, and mount points beneathit.Currently, when any directory in this hierarchyis accessed, the automount daemon mounts everyentry in the directory hierarchy. The expiryof a multi-mount entry also happens atomically.This is the mechanism used to implement-hosts. The program map auto.The dentry object represents a single componentof a directory path. One of the main functionsof the VFS is to resolve a given file systempath to its dentry by walking each of its pathcomponents.The VFS kernel interface of autofs is conceptuallystraightforward. The automount functionalityis provided largely in the inode operationlookup to lookup a new dentry, the dentry operationrevalidate to revalidate an existing dentry,the file operation readdir to read a dentrydirectory, and with a file system specific ioctlto check for dentrys that have not been used fora given timeout.The bulk of the work done in autofs is themounting and expiring of file systems.

52 • Hacking the Linux Automounter—Current Limitations and Future Directionsoption [4]. A code review is needed to establishwhether other services, such as mountd andportmap, can be configured to allow insecureports for their connections. But of course, usinginsecure ports is generally not a good idea.Autofs and mount also perform RPC probing todiscover whether the target server is availablebefore performing a mount. This process leadsto as many as nine ports per mount being usedduring a mount, which causes rapid exhaustionof reserved port space. The RPC port allocationalgorithm allows for a maximum of 800concurrently mounted file systems when usingUDP.The situation is somewhat different with TCP.For each TCP mount attempt, a client usesmultiple reserved ports, and each TCP socketmust transition through the TIME_WAIT stateto ensure the completion of the TCP three-wayhandshake. This process ensures that lost duplicatesdon’t cause errors on subsequent connections.The TIME_WAIT state is 2*MSL (maximumsegment lifetime) [3, Ch. 2, Sec. 7],which is 60 seconds for the Linux TCP stack.After this timeout, these reserved ports are freefor use again.This leads to a practical limit of around 100TCP protocol mounts performed in rapid succession.If the mounts are performed muchmore slowly, as is expected in normal operation,this number is somewhat larger. Nevertheless,it generally falls somewhat short of thetheoretical limit of 800 before port allocationproblems appear.4.4 Handling multi-mountsMulti-mounts were discussed in Section 3.3.These map entries must be handled atomically,mounted and umounted as a single unit. Problemsarise when using the auto.net programmap if the servers have a large number of exports,or if there are a large number of mountpoint offsets in a multi-mount entry. They mustbe handled as a single unit due to possible nestingdependencies within the mount hierarchy.The anonymous device and reserved port exhaustiondescribed in previous sections are thesource of the problem. We will present a partialsolution to this problem in Section 6, wherelazy mount/umount of multi-mount map entriesis described. Even with the improvements thereis still a limit on the total number of mounts thatcan be active at any one time due to resourceexhaustion. The only real solution to this problemis multiplexing of RPC connections.4.5 Parsing nsswitch.confCurrently, the Linux automounter does onlylimited parsing of the nsswitch.conf file. Itis only referenced when trying to locate themaster map during startup. The script justchecks what sources are present in the automountentry in nsswitch.conf, and looks forthe auto.master map in each location.There are a couple of reasons for this. First, allother consumers of the nsswitch.conf fileuse the standard glibc interfaces for accessingthe nsswitch.conf file. This interface is notconducive to the use that automount makes ofit.The format is described in the nsswitch.conf(5) man page. It includes basic usage,such as:subsystem: lookup_listIt also contains some more complex usages,such as:

54 • Hacking the Linux Automounter—Current Limitations and Future Directions6 Autofs Version 5Work is well underway to resolve most of thelimitations described above. In order to implementthe new functionality in a clean and sensibleway, it has been necessary to incrementthe kernel protocol version to 5.00. It seemedsensible then, to avoid confusion, to incrementthe version of the user space daemon to 5.0.0 aswell. Given the decision to increment the majorversion, it follows that the development priorityshould be to implement missing functionalityrather than attempt to retain compatibilitywith older versions of autofs. Hence, the newfunctionality will work only with version 5.00of the kernel module. Existing indirect mountmaps will continue to work as in previous versions.6.1 Direct mount implementationThe first and most important task has been toimplement fully functional direct mounts. Thisis particularly important because it paves theway for lazy mount/umount of multi-mountsand host map implementations.Two methods are available to do this. Thefirst is to use file system stacking similar tothat found in Wrapfs from the FiST [5] system.Although using Wrapfs from FiST wasvery compelling, in the end it was decided itwould increase the complexity too much whencompared to the chosen method.The method that has been used is to treat eachdirect mount entry as a distinct mount and takeadvantage of the VFS inode method follow_link to trigger mounts. This method is safe touse for this purpose because a directory cannotbe a symbolic link; therefore the method cannototherwise be in use. Since mount point directoriesare created in the host file system, the VFSdoesn’t call the autofs lookup, revalidate,or readdir methods when the directory is accessed,but calls the follow_link method(which follows the lookup during a path walk)to trigger the mount before walking into thenext directory. This implementation is surprisinglysimple but effective.The changes needed in the daemon are relativelystraightforward as well. A mount optiondirect has been added so the kernel moduleknows it is a direct mount and can send mountrequests to the daemon at the right time. Inthe daemon an additional entry point has beenadded to each of the lookup modules to enumeratea map so that the mount triggers can beset up.The changes in the communication protocol betweenthe kernel and the daemon also allow asingle process or thread to handle an entire directmount map.One difference comes in the expiry of directmounts. Each direct mount that has had amount triggered over mounts the direct mountpoint. Because of this it is passed over whenthe kernel walks the path. Therefore the busynesstimeout can only be updated during an expirerun. As a result, only truly busy mountpoints (ie. with open files or a processes workingdirectory) will prevent expiry. Changingthis expire semantic does not seem to be a problemand will hopefully help with graphical environmentspreventing mounts from expiringdue to the way they often scan file systems forchanges.Another issue is that because direct mounts aremade on directories within the underlying filesystem, changes to direct maps cannot be seenuntil the map is re-read (by sending the daemona HUP signal).It is interesting to note that existing industryimplementations implement direct mounts in a

2006 Linux Symposium, Volume Two • 55similar way.6.2 Lazy mount/umountLazy mount/umount of multi-mount map entrieshas been a difficult problem to solve forsome time now. But with the direct mountchanges above, we can see how it can be done.The basic problem to be solved is that of nestedmounts. Let’s revisit the example of Section 3.3on multi-mounts with a couple of small modificationsto demonstrate the problem:mydir -rw \/ server:/export/mydir \/src server2:/export/src \/src/f77 server2:/export/src/f77 \/src/c server2:/export/src/c \/tmp :/usr/tmpWhen mydir is accessed, the file system correspondingto the offset / is mounted. But nowthe file system is not necessarily an autofs filesystem, so we can never get a callback from thekernel. So autofs never knows another mount isneeded. Therefore, we must treat the entry as asingle unit and mount everything. Clearly thisnecessity applies equally when there is nestingat lower levels in the offsets, such as the offsetsin the src directory.We can deal with this issue by partitioningthe offsets and installing direct mount triggerswithin each of the file systems. In our example,when mydir is accessed we mount the entrycorresponding to / and install direct mounttriggers for each offset within the list boundedby nesting points. In this case, we install directmounts for /src and /tmp. Similarly, whenone of these mounts is triggered we mount itand install the corresponding triggers. In theexample we mount the entry for /src and theninstall triggers for /scr/f77 and /src/c andso on.Expiring these is a little trickier, because formulti-mounts like these we need to expire thedirect mounts themselves as well as the file systemsthat may be mounted on them. To solvethis problem, we need a way for the kernel todistinguish multi-mounts from standard directmounts. The obvious way to do this is to addan additional mount option, “offset” to distinguishthem from other direct mounts.The interesting thing about this scenario is thatwhen a file system is mounted on a trigger thatis perhaps itself nested, it will always be seenas busy by the expire system because there isa file handle open for communication with themount. On the other hand, a direct mount triggerwithout such a mount doesn’t hold open apipe but creates it at mount time. So multimountscan expire independently in a naturalway without further complication.6.3 Host mapsSince the lazy mount/umount has been implementedmany of the the resource issues withhost maps should be resolved. A separate moduleis devoted to handling host maps. The implementationamounts to little more than enumeratingthe local hosts table, then enumeratingtheir exports and using this informationfor lookups when they are accessed. The currentsimple implementation will no doubt needmuch refinement, such as filtering out non-NFSservers from the local hosts list to reduce clutter.6.4 Nsswitch integrationA parser for handling /etc/nsswitch.confmap source lookups has been added. Integratingthe parser amounted to adding a layer to

56 • Hacking the Linux Automounter—Current Limitations and Future Directionsperform lookups between the daemon and thelookup modules. The daemon now calls thecommon lookup module instead of calling thelookup modules directly and iterates throughthe list of sources found during the parseof /etc/nsswitch.conf. There where, ofcourse, a number of side affects that had to beovercome but generally it appears to work quitewell.6.5 Master map parsing6.6 Included map supportIncluded map support has also been implemented.The design fits into the map readingand lookup modules by just watching for a “+”as the first character of a map key and callingthe higher common lookup function to do thework, then continuing after it returns. This hasbeen done for both the master map and mountmaps but plus map inclusion is allowed only infile maps as is the case with other industry standardautomounters.Another important issue is the parsing of mastermaps in the init script. The init script isclearly not the right place for parsing the mastermap. As is the case in other industry automountimplementations, parsing should be done in autility designed specifically for that purpose orin the automounter proper.Another requirement is to use the name serviceswitch to read maps and lookup entries inmap sources. The code developed above alsoworks well for this which resolves another ofour long-standing limitations.The other feature that is expected of an automounteris that when there are multiple entriesfor a key in the master map, these entries shouldbe merged as described in Section 2.1. Thishas been achieved by leveraging the functionalityimplemented for handling nsswitch semantics.It follows fairly naturally from the need tohandle multiple map sources required by nssswitch.Implementing this feature has also provideda way to implement -null map supportin a fairly straightforward way. However, thereare difficulties identifying a map that needs tobe refreshed when there has been a change.But otherwise this should end up working fairlywell.6.6.1 LDAP supportThe LDAP lookup module has been a concernfor a long time and it has finally got some ofthe attention it so badly needed. The areasthat have been improved are the ability to specifythe schema used for storage of automountmaps, integration of master map parsing intothe daemon, encrypted TLS connections andthe ability authenticate to the LDAP server.One long standing problem is the need to supporttwo distinct LDAP schema used to storeautomount maps as well as some variationswithin these schema. The schema to be usedcan now be set in the autofs configuration forthe five class and attribute names used to queryan LDAP server. This will reduce the over headof using an LDAP server for autofs quite a bitand allow the use of other schema if it is required,as long as it is consistent with the waythe base schema are used.There have been a number of requests to addthe ability to use encrypted connections and tobe able to use authentication when connectingto an LDAP server. First, encrypted and optionallycertified connections can now be madeusing the START_TLS mechanism. The configurationfor the location of certificates must be

2006 Linux Symposium, Volume Two • 57done using the method required by the clientLDAP library and settings in an autofs authenticationconfiguration must be used to enableit. Authentication uses the SASL library andthe authentication method to use along withthe login name and secret are also held in thesame file as the TLS options above. So farthe only method tested has been DIGEST-MD5but other common methods available in SASLshould work or be relatively straightforward toadd.7 Concluding remarksThe astute reader will have noticed that theabove implementation of direct mounts andlazy mount/umount of multi-mount maps willuse a lot of anonymous devices. This use hasbecome possible since the limit on the numberof these devices was greatly increased in theearly stable release cycle of the 2.6 kernel. Itcould be possible for this to function with a 2.4kernel, but no work has been done to estimatethe effort to back port the anonymous devicechanges. So initially at least, direct mounts willonly be available for 2.6 kernels.This paper has described a good number ofachievements and identified the challenges inrounding out the Linux automount implementation.We don’t mean to say that these challengesare the only ones we face—just the mostdifficult to address, as well as those that are fundamentalto having a functional automounteron Linux.some challenging problems with respect to maprefresh.References[1] Linux Kernel source, Versions 2.4 and2.6, http://www.kernel.org/.[2] Hal Stern, Mike Eisler and RichardoLabiaga, Managing NFS and NIS, 2ndEdition, O’Reilly, June 2001.[3] W. Richard Stevens, Bill Fenner, andAndrew M. Rudoff, UNIX NetworkProgramming, The Sockets NetworkingAPI, Volume 1, Third Edition,Addison-Wesley Professional ComputingPress, 2004.[4] Travis Bar, Nicolai Langfeldt, Seth Vidaland Tom McNeal, Linux NFS-HOWTO,http://nfs.sourceforge.net/nfs-howto/, 2002-08-25.[5] FiST: Stackable File System Languageand Templates, Eraz Zadok et al.,http://www.filesystems.org/.[6] Sun TM Microsystems NFSAdministration Guide, Chapter 5,http://docs.sun.com/, 1995.[7] Robert Love, Linux Kernel Development,Second Edition, Novell Press, 2005.The current status of the changes outlinedabove for autofs version 5 is that the directmounts, nsswitch handling, lazy mount/umount, integration of master parsing, nsswitchintegration and the LDAP improvements haveall been done but have seen limited testing. Theplus map inclusion has also been done but has

58 • Hacking the Linux Automounter—Current Limitations and Future Directions