Communication Paradigms - Connect

Advanced Distributed Systems 

Communication Paradigms 

MSc in Advanced Computer Science 

Gordon Blair (gordon@comp.lancs.ac.uk) 

[http://research.lumeta.com/ches/map/]

Overview of the Session 

A problem led approach 

The role of communication in distributed systems 

Different styles of communication paradigm 

Interprocess communication 

Remote invocation 

Indirect communication 

Comparing and contrasting the approaches 

Associated Reading: CDK, chpts 4 and 5, pp. 609-615, 

677-683, chpt 18, TvS, pp. 145-157 (overall, rather 

fragmented coverage!) 

Adv. Dist. Systems G. Blair/ F. Taiani 2

A Framework for Understanding 

Distributed Systems 

What are the entities that are communicating in the 

distributed system? 

Nodes? Processes? Objects? Components? Web services? 

How do they communicate or, more specifically, what 

communication paradigm is used? 

What (potentially changing) roles and responsibilities do they 

have in the overall architecture? 

Client or server? Peers? 

How are they mapped on to the physical distributed 

infrastructure (what is their placement)? 

Partitioning? Replication? Caching? Code mobility? 


Context: Middleware 

Structures 


The Problem: A 

Social Network Systems 

Architecture for a Social 

Network 


Design Space 

invocation (RMI) 


Interprocess 

Communication

Sockets Programming 

Choice of UDP or TCP communication models 

Possible support for (IP) multicast 


Other Approaches 

Message passing, e.g. MPI 

Overlay networks 


Remote Invocation

Remote Invocation: Remote 

Procedure Calls

What is RPC? 

Higher level mechanism supporting the construction of distributed 

applications (see also discussion on transparency later) 

Supports the calling of a procedure in a separate address space 

(process) as if it exists in the local address space; the process may 

or may not be on the same machine 

But what is the semantics of a local procedure call? 


Styles of RPC 

First class RPC 

Integrated into the language => normal language 

mechanisms can be used for exceptions etc 

Examples include Java RMI, Ada and Argus 

Second Class RPC 

A special Interface Definition Language (IDL) is used 

to define communications (see later) 

Language independence 

Examples include Sun RPC, CORBA and DCE 


Programming with Interfaces 

Separation of interface and implementation 

Client software does not need to know the details of 

the implementation, cf. abstraction 

Important for platform and language independence 

Also important to support the evolution of software 

someInterface 

myObject:myClass 


Interface Definition Languages 

What is an IDL? 

A language independent means of specifying 

an interface 

Similar to abstract data type specification 

Dealing with Parameters 

IN parameters = value passed to server 

OUT parameters = value returned from server 

IN/OUT parameters = combination of above 


Implementing RPC 

Communications modules 

Object A 

Proxy 

Request 

Dispatcher 

Remote 

Object B 

Client Stub 

Reply 

Server Stub 

N.B. Proxies, stubs, dispatchers are generated 

automatically by an appropriate IDL compiler 


Remote Invocation: Remote 

Method Invocation

From RPC to Distributed Objects 

(Remote Method Invocation) 

obj 

obj 

obj 

OO programming: 

process 

Objects "speak" to one another by invoking each other's methods 

But they can only speak to objects located in the same process! 


From RPC to Distributed Objects 

(Remote Method Invocation) 

obj 

obj 

obj 

obj 

process 

obj 

process 

machine 

machine 

Distributed OO programming: breaking the process boundary 

Objects can "speak" to objects in remote processes 

Is this magic? 

Adv. Dist. Systems G. Blair/ F. Taiani 19 

No: It is all layering and abstraction

Essential Characteristics of RMI 

Full integration with object-oriented programming language 

Ability to exploit objects, class and inheritance 

Added benefits from exploiting built in (object-oriented) 

approaches to, for example, exception handling 

From procedure calling to method invocation 

Added expressiveness of supporting object references 

More sophisticated options for parameter passing 

E.G. See lecture on Java RMI 

Pass by (object) reference 

Pass by value (exploiting serialisation) 

Often integrated with code (object) mobility 

E.G. Use of class loading in RMI 


Indirect Communication: 

General

What is Indirect Communication? 

Communication between entities in a distributed 

system through an intermediary with no direct 

coupling between the sender and the 

receiver(s) 

But what is the intermediary 

Groups 

Event-based abstractions (e.g. publish-subscribe) 

Message queues 

Shared memory abstractions (e.g. DSM, tuple spaces) 

Note the optional plural in this definition: 

Often intrinsically provides multiparty 

communication 


A Closer Look at Coupling 

Remote invocation paradigms all imply a direct coupling 

between the client and server 

Indirect communication paradigms seek a level of uncoupling: 

Space uncoupling in which the sender does not know or need to 

know the identity of the receiver(s) and vice versa 

Because of this, the system developer has many degrees of freedom in 

dealing with change: participants (senders or receivers) can be replaced, 

updated, replicated or migrated 

Time uncoupling in which the sender and receiver(s) can have 

independent lifetimes (in other words, the sender and receiver(s) 

do not need to exist at the same time to communicate) 

Important benefits particularly in volatile environments where senders and 

receivers may come and go. 

Many uses in distributed systems including supporting mobility, 

dependability and event dissemination 


A Few Words on Indirection 

“All problems in computer science 

can be solved by another level of 

indirection” 

Roger Needham et al 

“There is no performance problem 

that cannot be solved by 

eliminating a level of indirection” 

Jim Gray 



Group Communication

Group Communication: An Initial 

Example of Indirection 

What is group communication? 

Based on the concept of a group abstraction, with operations 

provided to join and leave the group (cf. group membership) 

Messages sent to a group rather than to any individual process 

and messages are then delivered to each member of the group 

Often enhanced by guarantees in terms of message ordering 

and reliability 

Uses in distributed systems 

Important in supporting fault-tolerance 

e.g. supporting replication 

Also used heavily in dissemination of events 

e.g. in financial systems 

Key examples include JGroups and Isis 

See lecture on 

coordination and 

agreement 


A Typical Group Service 

interface GroupCommunicationService { 

appli 

appli 

} 

// creates a new group and returns the groups ID 

public GroupID 

groupCreate(); 

// Adding & Removing a member to/from a group 

public void groupJoin ( GroupID group, Participant member); 

public void groupLeave ( GroupID group, Participant member); 

// multicasts a message to the named group with 

// the specified delivery semantics, and 

// optionally collects a number of replies 

group 

comm 

public Messages[] multicast ( GroupID group, OrderType order, 

Messages message, int nbReplies) 

group 

comm 


Example 

When replicating servers: collective coordination needed 

either for fault-tolerance or scalability 

server 1 

server 2 

warehouse 

I’ve sold 

the last hat 

buy 

OK 

coming 

soon 

I’ve sold the last hat 

user 


Problems 

Reliability (against loss / crash) 

Ordering 

Scalability 


Reliability Guarantees 

Unreliable multicast 

Message sent to all members and may or may not arrive 

Reliable multicast: Protection against faulty network 

Reasonable efforts are made to ensure delivery in spite of 

message losses 

Can be based on positive or negative acknowledgements 

No guarantees is the sender crashes during multicast 

Atomic multicast: Protection against faulty participants 

All members receive message, or none do 

Main issue: tolerate the sender’s crash 


Implementing Reliable Multicast 

 

 

 

 

 

1) Originator sends a message to each member of the group, 

and awaits acknowledgements (ACK) 

2) If some acknowledgements are not received in a given 

period of time, re-send message; repeat this n times if 

necessary 

3) If all acknowledgements received 

then report success to caller 

Works fine if: 

Network problems are transient (msgs eventually get through) 

No crash (and no spurious behaviour) 

Not very scalable: ACK explosion! 

ack 

A 

B msg msg 

ack 

C 

msg 

ack 

D 


Avoiding ACKs Explosion 

Using negative ACKs (abbreviated in NACKs) 

If everything is fine the receiver does say anything 

If a message is lost the receiver complains to the sender 

Problem: How do we know that a message should be there? 

B 

counter = 01 

m 1 

m 1 

last msg from B = 0 

C 

A 


? 

B 

counter = 12 

NACK(m 1 ) 

m 2 

m 21 

C 


A 

m 2 


I’ve missed 

1 message! 

A 

B 

C 


Ordering Guarantees 

Unordered multicast 

No guarantees 

FIFO (First In, First Out) 

Messages sent from the same process are delivered in the order they 

were sent at different sites 

Messages sent from different processes may be delivered in different 

orders at different sites 

Totally ordered multicast 

Consider messages m1 and m2 sent to the group by (potentially) 

different processes 

Either m1 will be delivered before m2 or vice versa for all members of 

the group 

Causally ordered multicast 

As above, except the ordering of m1 and m2 is only important if a 

“happened-before” relationship exists between the messages 


Total Ordering vs Causal 

Ordering 


Implementing Total Ordering 

 

 

The sequencer approach (centralised) 

1. All requests sent to a sequencer, where they are given an ID 

2. The sequencer assigns consecutive increasing IDs 

3. Requested arriving at sites are held back until they are next in 

sequence 

Problems: sequencer = bottleneck + single point of failure 

Other approaches 

Distributed agreement to generate ids (as in Isis) 

Assign timestamps from a (global) logical or physical clock 

These are complex 



Publish-Subscribe

Publish-Subscribe Systems 

What is publish-subscribe? 

A key example of a distributed event-based system whereby: 

Publishers publish an event e: publish(e) 

Subscribers express interest in a set of events specified by a filter 

f: subscribe(f) 

Events are delivered asynchronously: notify(e) 

Publish optionally advertise what they will produce: advertise(f) 

The system acts as a broker to deliver events to the right 

subscribers 


As with groups, used in financial information systems and 

related news feeds applications 

Feature heavily in systems supporting cooperative working 

Increasingly used in ubiquitous computing/ monitoring 

Examples include JMS, Scribe, Siena, Gryphon and Hermes 


Publish-Subscribe Systems 

(continued) 


A Closer Look at the Subscription 

Channel-based 

Model 

Publishers publish events to named channels and subscribers 

then subscribe to one of these named channels and therefore 

receive all events sent to that channel 

Rather primitive scheme and the only one that defines a 

physical channel (all other schemes employ some form of 

filtering over the content of an event as we will see below) 

Topic-based (also referred to as subject-based) 

Each notification is expressed in terms of a number of fields 

with one field denoting the topic and subscriptions defined in 

terms of this topic of interest 

Similar to channel-based approaches (implicit vs. explicit) 

Can be enhanced by introducing hierarchies of topics 


Subscription Models (continued) 

Content-based 

Generalization of topic-based allowing the expression of 

subscriptions over a range of fields in an event notification. 

The filter is a query defined in terms of compositions of 

constraints over the values of event attributes 

Significantly more expressive but with significant new 

challenges introduced in terms of implementation 

Type-based 

Intrinsically linked with (typed) object-based approaches 

Subscriptions defined in terms of this type (signature/ methods/ 

attributes), with matching defined in terms of types or subtypes 

of the given filter 

Can be integrated elegantly into programming languages and 

can also check type correctness of subscriptions. 


Implementing Publish-Subscribe 

The key decision is whether to go for a centralised, 

distributed or fully peer-to-peer architecture 

Most systems have distributed architectures consisting of a 

network of brokers 


Flooding 

Realising Content-based 

Approaches 

Send all published events to all possible recipients, or 

alternatively all subscriptions back to all possible senders 

Can be supported by underlying multicast service as available 

Simple but significant message overload 

Filtering 

Filters propagated back through the broker network with the 

intuition that notifications are only forwarded through the broker 

network if there is a path to a valid subscriber 

Advertisements 

In above scheme, filters need to be sent to all possible publishers 

Overhead can be reduced by use of advertisements 


Realising Content-based 

Approaches (continued) 

Rendezvous 

Consider the set of all possible events as an event space 

Partition this event space into pieces and allocate 

responsibility for each piece to a given broker (known as 

the rendezvous node for that event) 

Implementation requires two functions to be defined: 

SN(s) which takes a given subscription, s, and returns one or more 

rendezvous nodes which take responsibility for that subscription 

EN(e) which takes a given event, e, and returns one or more rendezvous 

nodes responsible for matching e against subscriptions in the system 

Can lead to highly scalable implementations 


An Optional Exercise 

The following is an actual algorithm to implement filtering 

This algorithm assume each node maintains 

A neighbours list containing a list of all connected neighbours in 

the network of brokers 

A subscription list containing a list of all directly connected 

subscribers serviced by this node 

A routing table – maintaining a list of neighbours and valid 

subscriptions for that pathway 

An implementation of matching on each node in the network of 

brokers, in particular a match function takes a given event 

notification and a list of nodes together with associated 

subscriptions and returns a set of nodes where the notification 

matches the subscription 


The Algorithm 

upon receive publish(event e) from node x 

matchlist := match(e, subscriptions) 

send notify(e) to matchlist; 

fwdlist := match(e, routing); 

send publish(e) to fwdlist - x; 

upon receive subscribe(subscription s) from node x 

if x is client then 

add x to subscriptions; 

else add(x, s) to routing; 

send subscribe(s) to neighbours - x; 


Some Additional Notes 

 

 

 

 

 

 

When a broker receives a publish request from a given node, it must pass 

this subscription to all connected nodes where there is a corresponding 

matching subscription and also decide where to propagate this event 

through the network of brokers. 

Lines 2 and 3 achieve the first goal by matching the event against the 

subscription list and then forwarding the event to all the nodes with 

matching subscriptions. 

Lines 4 and 5 then use the match function again, this time matching the 

event against the routing table and forwarding only to the paths that lead 

to a subscription. 

Brokers must also deal with incoming subscription events. 

If the subscription event is from an immediately connected subscriber then 

this subscription must be entered in the subscriptions table (lines 7 and 8). 

Otherwise, the broker is an intermediary node and now knows that a 

pathway exists towards this subscription and hence an appropriate entry is 

added to the routing table (line 9). In both cases, this subscription event is 

then passed to all neighbours apart from the originating node (line 10). 



Message Queues

What is a message queue? 

Message Queues 

An alternative paradigm for indirect communication based on 

distributed queues 

Messages are sent to a queue 

Processes can then access messages in the queue either by 

receiving a message (blocking), polling for messages (non-blocking) 

or being notified when messages arrive 

Messages are persistent and message delivery is reliable 

Fundamentally a point-to-point service (not multi-party) 


Enterprise Application Integration (EAI), i.e. integration between 

applications in a given enterprise utilising the loose coupling 

Also used heavily in commercial transaction processing systems 

Examples include JMS, IBM’s Websphere MQ, Microsoft’s MSMQ 

and Oracle’s Streams Advanced Queuing AQ) 


Message Queues (continued) 



Distributed Shared Memory

Distributed Shared Memory 

What is a distributed shared memory? 

Provides an abstraction of shared memory in a distributed system 

If data is not available locally, it must be fetched (similar to a page fault in 

traditional virtual memory systems) 

Hides distribution entirely from the programmer 

No new programming abstractions to learn 

Can be costly to implement though in terms of maintaining consistency of 

shared data 


Tends to be more specialist, for example for parallel and 

distributed computation in cluster computers (i.e. relatively tightly 

coupled distributed architectures) 

Examples include JMS, IBM’s Websphere MQ, Microsoft’s MSMQ 

and Oracle’s Streams Advanced Queuing AQ) 


Distributed Shared Memory 

(continued) 



Tuple Spaces

Tuple Spaces 

What is a tuple space? 

Another paradigm for indirect communication offering an 

abstraction of a semi-structured shared space consisting of a 

number of tuples 

Processes can write tuples to the tuple space 

Processes can then read a tuple from tuple space which also leaves 

a copy in the tuple space 

Alternatively, they can take the tuple which removes it from the space 

Both the read and take operations are based on pattern matching 

(associative access) 


Influential in the areas of mobile and ubiquitous computing 

Also used for systems integration 

Examples include Linda, JavaSpaces, IBM’s TSpaces and 

L2imbo 


Tuple Spaces(continued) 


Indirect Communication: A 

Comparison

Comparison of Approaches 


Expected Learning Outcomes 

At the end of this session: 

You should understand the range of low level services offered by 

, what they might be useful for and their 

intrinsic limitations 

You should understand the services offered by 

paradigms and the essential difference between 

and 

You should understand the essence of 

, why it is 

different from remote invocation (for example) and the role of time and 

space uncoupling in supporting these characteristics 

You should appreciate the of indirect communication services 

available and be able to assess their strengths and weaknesses 

You should be able to select appropriate communication paradigms for 

a given distributed systems application mapping requirements on to 

underlying communication services and being aware of the trade-offs 

associate with such decisions

Communication Paradigms - Connect

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?