UNIX Network Programming Overview of STREAMS Douglas C ...

STREAMS Overview STREAMS is a exible communication subsystemframeworkUNIX Network Programming{ Originally developed by Dennis Ritchie for ResearchUNIXOverview of STREAMS STREAMS provides a uniform infrastructurefor developing and conguring characterbasedI/O{ e.g., networks, terminals, local IPCDouglas C. Schmidt STREAMS supports the addition and removalof processing components at installationtimeor run-time{ Via user-level or kernel-level commands12STREAMS Overview (cont'd) The STREAMS paradigm is data-driven, notdemand-driven{ i.e., asychronous in the kernel, yet synchronous atthe applicationSTREAMS Benets STREAMS provides an integrated environmentfor developing kernel-resident networkingservices Supports both immediate and deferred processing STREAMS promotes denition of standardservice interfaces{ e.g., TPI and DLPI Internally, data are transferred by passingpointers to messages{ Goal is to reduce memory-to-memory copying overhead STREAMS supports dynamic \service substitution"controlled by user-level commands34

A Simple StreamSTREAMS Benets (cont'd) Message-based interfaces enable o-boardprotocol migration Permits layered and de-layered multiplexing More recent implementations take advantageof parallelism in the operating systemand hardware56A Module on a StreamThe Stream Head A \stream head" exports a uniform serviceinterface to other layers of the UNIX kernel{ Including the application \layer" running in userspace General stream head services include1. Queueing(a) Provides a synchronous interface to asychronousdevices2. Datagram- and stream-oriented data transfer3. Segmentation and reassembly of messages4. Event propagation{ i.e., signals78

Message StructureComposite Message1314Message Buer SharingSTREAMS Message TypesM DATAM PROTOM PASSFPM IOCTLM BREAKM SIGM DELAYM CTLM SETOPTSM RSEuser dataprotocol informationpass le pointeruser ioctl() requestrequest line breaksignal process grouprequest transmit delaymodule-specic control messageset Stream head optionsreserved for RSE use Normal priority messages{ M DATA, M PROTO, M PASSFP, and M IOCTLmay be generated from user-level{ Typically subject to ow control1516

STREAMS Message Types(cont'd)Queue StructureM PCPROTOM FLUSHM IOCACKM IOCNAKM COPYINM COPYOUTM IOCDATAM PCSIGM READM HANGUPM ERRORM STOPM STARTM STOPIM STARTIM PCRSEprotocol informationush queuesacknowledge ioctl() requestfail ioctl() requestrequest to copyin ioctl() datarequest to copyout ioctl() datareply to M COPYIN and M COPYOUTsignal process groupread noticationline disconnectfatal errorstop output immediatelyrestart outputstop input immediatelyrestart inputreserved for RSE use High priority messages*Typically not ow controlled*MPCPROTO may be generated from user-level* Others passed between STREAM components1718Queue Structure (cont'd)Queue and Message Linkage queue t{ Primary data structure. Each module contains a write queue and a readqueue{ Stores information on ow control, scheduling, max/mininterface sizes, linked messages, private data qinit{ Contains put(), service(), open(), close()subroutines qband{ Contains information on each additional messageband > 0 and < 256 module info{ Stores default ow control information1920

Queue SubroutinesQueue Subroutines (cont'd) Four standard subroutines are associated withqueues in a module or driver, e.g., Standard subroutines (cont'd){ open(queue t *q, dev t *devp, int oag, int sag,cred t *cred p);. Called when Stream is rst opened and on anysubsequent opens. Passed a pointer to the new read queue{ put(queue t *q, mblk t *mp). Performs immediate processing. Supports synchronous communciation servicesi.e., further queue processing is blocked untilput() returns. Also called any time a module is \pushed" ontothe Stream{ close(dev t dev, int ag, int otyp, cred t *cred p);. Called when last reference to a Stream is closed. Also called when a module is \popped" o theStream{ service(queue t *q). Performs deferred processing. Supports asynchronous communication servicesUses the message queue available in a queue. Runs as a \weightless" process: ::2122Queue Flow ControlFlow Control and the service()Procedure Typical non-multiplexed exampleint service (queue t *q)fmblk t *mp;while ((mp = getq (q)) != 0)if (queclass (mp) == QPCTL jjcanputnext (q)) f/* Process message */putnext (q, mp);gelse fputbq (q, mp);return 0;ggput() and service() work together to support advisoryow control Flow control is more complex with multiplexersand concurrency2324

Flow Control and the canput()Procedure canputnext() is used by put() and service()routines to test advisory ow control conditions e.g.,Flow Control and the put()Procedure Typical put() exampleint put (queue t *q, mblk t *mp)fint canputnext (queue t *q)fgnd closest queue with a service() procedureif (queue is full) fset ag for \back-enabling"return 0;greturn 1;gif (queclass (mp) == QPCTL jjcanputnext (q)) f/* Process message */putnext (q, mp);elseputq (q, mp);/* Enables service routine */return 0; Note that non-MP systems may use canput(): ::2526putq() The int putq(queue t *, mblk t *) functionenqueues a message on a queue{ It is typically called by a queue's put() procedurewhen it can no longer proceed: ::getq() The mblk t *getq(queue t *) function dequeuesa message from a queue{ It is typically called by a queue's service() procedure putq() automatically handles1. priority-band allocation Messages are dequeued in priority order{ i.e., higher priority messages are stored rst in thequeue!2. priority-band message insertion3. ow control getq() handles1. Flow control Enqueueing a high priority message automaticallyschedules the queue's service()procedure to run at some point{ Diers on MP vs. non-MP system272. Back-enabling getq() returns 0 when there are no availablemessages28

Multiplexing STREAMS provides rudimentary support formultiplexing via multiplexor driversMultiplexor Links (before){ Unlike modules, multiplexors occupy a le-systemnode that can be \opened". Rather than \pushed" Multiplexors may contain one or more upperand/orlower connections to other STREAMmodules and/or multiplexors{ Enables support for layered network protocol suites . e.g., "/dev/tcp" Note there is no automated support for propagatingow control across multiplexors2930Multiplexor Links (after)Internetworking Multiplexor 3132

Driver Data Structure LinkageModule Data Structure Linkage3334Pipes and FIFOsPipes and FIFOs (cont'd) In SVR4 and Solaris, the traditional localIPC mechanisms such as pipes and FIFOshave been reimplemented using STREAMS This has broaded the semantics of pipes andFIFOs in the following ways:1. Pipes are now bidirectional (STREAM pipes)2. Pipes and FIFOs now support both bytestreamorientedand message-oriented communication{ ioctl()s exist to modify the behavior of STREAMdescriptors to enable or disable many of the newfeatures3. Pipes can be explicitly named and exported intothe le system4. Pipes and FIFOs can be extended by having modulespushed onto them3536

Mounted Streams and CONNLDMounted Streams and CONNLD(cont'd) In earlier generations of UNIX, pipes wererestricted to allowing multiplexed communication{ This is overly complex for many client/server-styleapplications SVR4 UNIX and Solaris provide a mechanismknown as \Mounted Streams" thatpermits non-multiplexed communication{ This provides semantics similar to UNIX domainsockets{ However, Mounted Streams are more exible sincethey incorporate other features of STREAMS3738Layered MultiplexingLayered Multiplexing (cont'd) Advantages{ Share resources such as control blocks { Supports standard OSI and Internet layering models Disadvantages{ More processing involved to demultiplex in deepprotocol stacks{ May be more dicult to parallelize due to locksand shared resource contention{ Hard to propagate ow control and QOS info acrossmuxer3940

De-layered MultiplexingDe-layered Multiplexing (cont'd) Advantages{ Less processing for deep protocol stacks { Potentially increased parallelism due to less contentionand locking required { Easier to propagate ow control and QOS info Disadvantages{ Violates layering (e.g., need a packet lter){ Replicates resources such as control blocks4142STREAM ConcurrencyConcurrency Alternatives Modern versions of STREAMS support multiprocessing{ Since modern UNIX systems have multi-threadedkernels Dierent levels of concurrency support include1. Fine-grainSTREAMHeadsLP_XDR::svc (void){ /* outgoing */ }ModulesLP_TCP::put(Message_Block *mb){ /* outgoing */ }APPLICATIONC1C2APPLICATIONLP_XDR::svc (void){ /* incoming */ }ModulesLP_TCP::put(Message_Block *mb){ /* incoming */ }* Queue-level* Queue-pair-level2. Coarse-grainLP_DLP::svc (void){ /* outgoing */ }NETWORK DEVICESOR PSEUDO-DEVICESLP_DLP::svc (void){ /* incoming */ }STREAMTail* Module-level* Module-class-level* Stream-levelMODULEOBJECTWRITE READQUEUE QUEUEOBJECT OBJECTPROCESSOR THREADMESSAGEOBJECT Note, developers must use kernel lockingprimitives to provide mutual exclusion andsynchronization Layer Parallelism4344

Concurrency Alternatives (cont'd)STREAMHeadsCP_XDR::put(Message_Block *mb){ /* outgoing */ }CP_TCP::put(Message_Block *mb){ /* outgoing */ }CP_IP::put(Message_Block *mb){ /* outgoing */ }ModulesCP_DLP::svc (void){ /* outgoing */ }NETWORK DEVICESOR PSEUDO-DEVICESAPPLICATIONC1APPLICATIONC2CP_XDR::put(Message_Block *mb){ /* incoming */ }CP_TCP::put(Message_Block *mb){ /* incoming */ }CP_IP::put(Message_Block *mb){ /* incoming */ }STREAMTailModulesCP_DLP::svc (void){ /* incoming */ }MODULEOBJECTWRITEQUEUEOBJECTREADQUEUEOBJECTPROCESSOR THREADMESSAGEOBJECT Connectional Parallelism45

Concurrency Alternatives (cont'd)APPLICATIONAPPLICATIONSTREAMHeadsMP_XDR::put(Message_Block *mb){ /* outgoing */ }ModulesMP_UDP::put(Message_Block *mb){ /* outgoing */ }MP_XDR::put(Message_Block *mb){ /* incoming */ }ModulesMP_UDP::put(Message_Block *mb){ /* incoming */ }MP_DLP::svc (void){ /* outgoing */ }NETWORK DEVICESOR PSEUDO-DEVICESMP_DLP::svc (void){ /* incoming */ }STREAMTailMODULEWRITEQUEUEREADQUEUEPROCESSOR THREAD Message Parallelism46

STREAMS Evaluation Advantages{ Portability, availability, stability{ Kernel-mode eciency Disadvantages{ Stateless process architecture. i.e., cannot block!{ Lack of certain support tools. e.g., standard demultiplexing mechanisms{ Kernel-level development environment. Limited debugging support: ::{ Lack of real-time scheduling for STREAMS processing: ::. Timers may be used for \isochronous" service47

Summary STREAMS provides a exible communicationframework that supports dynamic con-guration and reconguration of protocolfunctionality Module interfaces are well-dened and reasonablywell-documented Support for multi-processing exists in Solaris2.x48