The DTN Reference Implementation - Delay Tolerant Networking ...

The DTN Reference 

Implementation 

Michael Demmer 

IETF DTNRG Meeting - March 9, 2005

Implementation Overview 

• Written primarily in C++ 

– ~22,000 non-comment lines of C++ (~5,000 C) 

– 10K in generic system support classes (oasys) 

– 189 individual classes 

– Multithreaded (pthreads), mutex, spin lock 

– STL for data structures (string, list, hashtable, …) 

• Emphasize clarity, cleanliness, flexibility 

• Ported to Linux, Solaris, Win32 (Cygwin), 

Linux on PDA (ARM), FreeBSD, Mac OSX 

(Source line statistics generated using David A. Wheeler's 'SLOCCount’) 

3/13/05 DTN Reference Implementation 2

Bundle Daemon (dtnd) Components 

API Server 

Contact 

Manager 

TCP 

UDP 

Sneakernet 

Convergence Layers 

Tcl Console / 

Configuration 

Event Queue 

Bundle 

Daemon 

Fragmentation 

Manager 

Persistent Store 

Bundle 

Router 

3/13/05 DTN Reference Implementation 3 

dtnd 

Bundle Actions 

= Thread

Bundle Daemon Components 

• Bundle Daemon: core event queue and dispatcher 

• Bundle Router: all “smarts” in the system, effects 

policies via a the Bundle Actions interface 

• API Server: handler for application IPC interface 

• Convergence Layers: transport-specific mechanism 

for transmitting bundles to a next-hop 

• Contact Manager: service for detecting connection 

opportunities with neighbors 

• Fragmentation Manager: proactive and reactive 

fragmentation, maintains reassembly buffers 

• Persistent Store: object database abstraction for 

durable storage, uses Berkeley DB or SQL backends 


Bundle Daemon Events 

API Server 

Contact 

Manager 

TCP 

UDP 

Sneakernet 


Tcl Console / 

Configuration 

Event Queue 

Bundle 

Daemon 

Fragmentation 

Manager 


Bundle 

Router 


dtnd 


All events (e.g. bundle arrival, 

contact disruption, reassembly 

completed, etc.) posted to 

the daemon event queue. 

= Thread

Bundle Router Actions 

API Server 

Contact 

Manager 

TCP 

UDP 

Sneakernet 


Tcl Console / 

Configuration 

Event Queue 

Bundle 

Daemon 

Fragmentation 

Manager 


Bundle 

Router 


dtnd 


Events are then forwarded to 

the router, which responds 

with a set of actions that are 

dispatched to components. 

= Thread

Bundle Router 

• Not just traditional routing – all 

“policy” decisions in the daemon 

– e.g: where / when to send bundles, whether to 

accept custody, apply fragmentation, store 

bundles, etc… 

– Insulated by event/action interface 

– Multiple implementations, chosen at config 

• Why this design? 

– Believe routing is the hardest problem, may need 

all events 

– Easier to change policies if consolidated in one 

place 


Evaluation 

HOP 

E2E 

user 

kernel 

user 

kernel 

user 

kernel 

• Approaches compared (both e2e and hop): 

– SFTP (simple ftp-like TCP app) – kernel IP or ‘plug proxy’ 

– MAIL (Sendmail, no ckpoint/restart) both end to end and 

– DTN (store-and-forward with restart) hop configurations 

user 

kernel 

• Compare robustness to interruption / link errors 

• Link up/down patterns: aligned, shifted, sequential, random 


DTN Overhead – No Disruptions 

DTN does best of any for small msgs, modest overhead overall 


Interruption Tolerance 

Zero throughput for e2e 

Note zero 

throughput for all 

E2E cases 

Up/down 1m/3min; 40KB messages; shift: 10s 


Simulator Framework 

• Implementation also includes basics of a discrete 

event simulator for prototyping and testing 

– TCL based simulation script to generate contact schedules, test 

traffic patterns, link bandwidth / delay, etc. 1 

• Uses the same events and abstractions 

– Implemented via a SimConvergenceLayer, Bundle Payload option 

for no data, modified event queue, and SimBundleActions 

– Other core classes and architecture are unmodified 

– Allows testing of the whole code base except convergence layers 

• Key characteristic is that to the router, there is no 

distinction between simulation or deployment 

1 not all planned simulation features are yet implemented 


Current Status 

• Basic system operational and (fairly) stable 

– Primitives for in-memory and on-disk Bundle management 

– Extensible naming and addressing framework 

– Next-hop taxonomy of peers, links, contacts 

– TCP and UDP convergence layers (with pending ID specs) 

– Implemented (mostly) Bundling protocol v3 

– Berkeley db and sql storage implementations 

– “Sockets-like” application interface 

– Sample applications: dtnping, dtnsend, dtnrecv, dtncp 

– Proactive and reactive fragmentation and reassembly 


Major ToDo Items 

Bundle 

Router 

Bundle 

Management 

API Server 

Convergence 

Layers 

Contact 

Manager 

Simulator 

−Split into Bundle Engine / Bundle Router 

−Implement a non-static routing protocol 

−Implement Bundle expirations 

−Custody transfer, delivery status notification 

−Class of Service based forwarding decisions 

−Improve error reporting and robustness 

−Change from loopback UDP to TCP 

−Handle application disconnection 

−Some more ad-hoc protocol (e.g. raw 802.11) 

−More non-traditional networking (e.g. USB key) 

−Multicast or broadcast based peer discovery 

−Scheduled links and contacts 

−Complete implementation 


Implementation Details 

• ISO Stack View 

• Example Event / Action Flow 

• Naming / Address Families 

• Bundle Class Representation 

• Bundle Consumers (next hop) 

• Bundle Lists 

• Convergence Layer API 

• Persistent Storage Interface 


ISO Stack View 

Application 

IPC 

Bundle 

Bundle Daemon 

Presentation Bundle Protocol 

Transport 

Sneakernet 

(in progress) 

DTN Application 

XDR 

UDP 

Embedded Application 

TCP UDP 

to/from 

next hop 

DTN2 

socket-like API 

(plan to change to TCP) 

Embedded 

Application 

Application can also run the 

daemon code as a thread. 


Example Event / Action Flow 

User configures a new 

ONDEMAND TCP link 

and a route for the link. 

event: LINK_CREATED 

event: LINK_AVAILABLE 

Router adds link to 

route table. 

TCPCL interface accept()’s 

a new connection and 

constructs a new bundle 

from the wire. 

event: BUNDLE_ARRIVED 

Router matches the link in 

the route table as the only 

next hop for the bundle. 

action: STORE_ADD 

action: OPEN_LINK 

Database stores the bundle. 

TCPCL resolves the destination 

admin string and establishes a 

connection to the next hop. 

event: CONTACT_UP 

Router checks the link, finds 

a pending bundle, moves 

the bundle to the contact. 

action: ENQUEUE_BUNDLE 

TCP Connection thread picks 

bundle off contact queue, 

sends it to the next hop. 

Router checks bundle 

mappings and finds 

no other destinations 

action: STORE_DEL 

Database removes 

the bundle. 


event: BUNDLE_TRANSMITTED 

Disruption occurs on the 

connection, TCPCL times 

out on a keepalive message 

event: CONTACT_DOWN 

event: LINK_UNAVAILABLE 

Router checks queues for 

any bundles that need to be 

re-routed to other links.

Bundle Router Events 

BUNDLE_RECEIVED // New bundle arrival 

BUNDLE_TRANSMITTED // Bundle or fragment successfully sent 

BUNDLE_EXPIRED // Bundle expired 

BUNDLE_FORWARD_TIMEOUT // A Mapping timed out 

CONTACT_UP // Contact is up 

CONTACT_DOWN // Contact abnormally terminated 

LINK_CREATED // Link is created into the system 

LINK_DELETED // Link is deleted from the system 

LINK_AVAILABLE // Link is available 

LINK_UNAVAILABLE // Link is unavailable 

REGISTRATION_ADDED // New registration arrived 

REGISTRATION_REMOVED // Registration removed 

REGISTRATION_EXPIRED // Registration expired 

REASSEMBLY_COMPLETED // Reassembly complete 

ROUTE_ADD // Add a new entry to the route table 

ROUTE_DEL // Remove an entry from the route table 


Bundle Action Interface 

enqueue_bundle(Bundle* bundle, BundleConsumer* nexthop, 

bundle_fwd_action_t fwdaction, 

int mapping_grp, u_int32_t expiration, 

RouterInfo* router_info) 

dequeue_bundle(Bundle* bundle, BundleConsumer* nexthop) 

move_contents(BundleConsumer* source, 

BundleConsumer* dest) 

store_add(Bundle* bundle) 

store_del(Bundle* bundle) 

open_link(Link* link) 

close_link(Link* link) 


Naming / Address Families 

URI (scheme:scheme-specific-part) 

bundles:/// 

Bundle Destination 

“dns-style” 

string 

Local Interface (admin only) 

RouteTable (destination pattern) 

RouteTable (next hop) 

RouteTable (default pattern) 

address 

family (AF) 

Examples 

bundles://internet.dtn/host://test.dtnrg.org/myapp1 

host://0.0.0.0:5000 

bundles://dtnrg.org/smtp://*@dtnrg.org/* 

bundles://some_region/host://xyz.dtnrg.org:5000 

bundles://*/* 

internet 

fixed 

smtp 1 

wildcard 

Admin String Format 

host://:/demux 

0x0102 

smtp://@/demux 

* or */* or *:* 


AF 

1 smtp addressing has not actually been implemented (yet)

Naming / Address Families (2) 

- tuple: string 

- region: string 

- admin: string 

- proto: string 

parse(string) 

valid() 

BundleTuple 

- family: AddressFamily 

compare(tuple) 

BundleTuplePattern 

match(tuple) 

AddressFamily 

- schema: string 

validate(admin_str) 

resolve(admin_str) 

match(admin_str, pattern) 

InternetAddressFamily 

FixedAddressFamily WildcardAddressFamily 

• AF abstraction allows essentially any 

format for admin strings 

• BundlePattern class implements AFspecific 

wildcard matching 


serialize() 

Bundle Class Representation 

SerializableObject 

Bundle 

- source: BundleTuple 

- destination: BundleTuple 

- custodian: BundleTuple 

- reply-to: BundleTuple 

- payload: BundlePayload 

- mappings: BundleMappings 

- priority: int 

- custody_req: boolean 

- return_rcpt: boolean 

- creation_ts: timestamp_t 

- refcount: int 

… 

add_ref(), del_ref() 

add_mapping(), del_mapping() 

• Bundle Metadata always in memory 

– Reference-counted C++ object for safety 

• Various fields “parsed” from the 

wire format into member variables 

– BundleProtocol class to convert to and from 

– Object serialization layer for synchronizing 

with the database 

• BundlePayload abstraction 

– Manages on-disk representation of payload 

– Methods for read_data(), write_data() 

– For small bundles, keeps in-memory copy 


Bundle Consumers 

BundleConsumer 

next_hop: string 

bundle_list: BundleList 

enqueue_bundle(Bundle*) 

dequeue_bundle(Bundle*) 

Contact Link Peer Registration 

OndemandLink ScheduledLink 

OpportunisticLink 

• Routing can return any consumer as a valid next-hop 

link0: Scheduled, UDP 

link1: Opportunistic, SneakerNet 

link2: OnDemand, TCP 

– virtual override of enqueue_bundle to implement different abstractions 

• Allows router to use most-natural expression of an algorithm 

– enqueue bundle on peer implies send on next contact on any link to the peer 

peer0 

peer1 

a next-hop dtn daemon 


peer 

link 

contact 

registration 

means to communicate with a peer 

created when link is “open” 

locally attached application

Bundle Lists / Bundle Mappings 

Bundle Lists 

bundle_list: Pending Bundles 

list_position: 1 

op: NONE 

bundle_list: Contact 1 


op: FORWARD_ONCE 

group id: 0x10 

timeout: 1097563442.125245 

bundle_list: Contact 2 


op: FORWARD_ONCE 

group id: 0x10 

timeout: 1097553431.324565 

BundleMappings 

Pending Bundles Contact 1 Contact 2 

Bundle 1 Bundle 2 Bundle 3 


Convergence Layer API 

add_interface(Interface) 

add_link(Link, argc, argv) 

open_contact(Contact) 

send_bundles(Contact) 

del_interface(), del_link(), close_contact() 

Convergence Layer API 

create a new ingress point for bundle arrivals 

prepare the given link for transmission 

open a connection for the given contact 

try to send bundles queued on the given contact 

remove the given interface/link/contact 

• Though the abstraction is intended to be general, to date only 

two functional implementations (TCP and UDP) 

– API may (will likely) need to be extended to account for other transports 

• Other things of note: 

– May add an MTU (e.g. UDP limited to 16K max bundle) 

– Likely need abstractions for neighbor discovery, peer identification, etc. 


Persistent Storage Interface 

BundleStore 

store: 

PersistentStore 

load() 

add(Bundle*) 

del(int) 

get(int) 

update(Bundle*) 

BerkeleyDBBundleStore 

RegistrationStore 

store: PersistentStore 

load() 

add(Registration*) 

del(int) 

get(int) 

update(Registration* 

) 

SQLBundleStore 

• Several layers of abstraction for 

implementation transparency 

store: 


load() 

GlobalStore 

next_bundleid() 

next_regid() 

update() 

• Easy to extend with new storage types 

(e.g. routing state) or storage 

implementations 

load() 

add(int, SerializableObject*) 

del(int) 

get(int) 

BerkeleyDBStore 


update(int, 

SerializableObject*) 

SQLStore 

impl: 

SQLImplementation 

SQLImplementation 

PostgresSQLImpl MYSQLImpl 


Persistent Storage Interface (2) 

• SerializableObject interface allows a single function 

to define marshal / unmarshal for all DB types 

int Bundle::Serialize(SerializeAction* a) { 

} 

a->process(“bundleid”, &bundleid_); 

a->process(“source”, &source_); 

a->process(“dest”, &dest_); 

a->process(“priority”, &priority_); 

a->process(“custreq”, &cust_req_); 

a->process(“return_rcpt”, &return_rcpt_); 

// ... 

• Adding a new DB implementation requires 

implementing a handful of functions for different 

basic types (int, string, etc) 

– Berkeley DB implementation packs fields into a byte array 

– SQL implementations generate an update command e.g. 

update bundles set custreq = true where bundleid = 10

The DTN Reference Implementation - Delay Tolerant Networking ...

Create successful ePaper yourself

Delete template?

Save as template?