Tutorial on Fault Tolerant CORBA

<strong>Tutorial</strong> onFault Tolerant CORBADownload specifications fromhttp://www.omg.org/cgi-bin/doc?ptc/2000-03-04http://www.omg.org/cgi-bin/doc?ptc/2000-03-05Download tutorial fromhttp://www.omg.org/cgi-bin/doc?orbos/2000-09-14OMG MeetingBurlingame, CA September 2000<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Outline1. Introduction to Fault Tolerance2. Fault Tolerance Mechanisms3. Fault Tolerance Properties4. Fault Tolerance Management5. Fault Tolerant Applications6. Fault Tolerant Hello Server Example<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Outline1. Introduction to Fault Tolerancea. Objectivesb. Limitationsc. Types of Faultsd. Basic Concepts<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

What is Fault Tolerance?• Murphy’s Law of Fault Tolerance:– The only thing that is certain is thatthe system is going to fail• The best that we can dois to reduce the probability of failure(but not to zero)<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Objectives of FT CORBA• Wide range of fault tolerance– Simple low-cost clients– Highly reliable server clusters– Many systems will contain both– Other systems will contain external clients thatknow nothing, or little, about fault tolerance• Local Clusters and alsoWide-area Systems• Large-scale Servers and alsoEmbedded Controllers<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Limitations of the Specification• Interoperability limitations– All replicas of an object must be hosted byinfrastructure from the same vendor• Non-determinism may compromisestrong replica consistency• No support for partitioned systems• No commission (wrong result) faults• No software design faults• Vendors can provide proprietary productsthat overcome these limitations<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Types of Faults• Processor faults– Crash faults– Commission faults (very expensive)• Network faults– Multiple network connections• Operating System hangs• Memory leaks• Software design errors(beyond the state-of-the-art)<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Consistency• Redundancy is the basis of fault tolerance• The Fault Tolerant CORBA standard is based onfault tolerance by object replication• Strong replica consistency– All of the replicas have the same state– Greatly simplifies the application system design– Requires careful design of, and strong mechanisms in,the infrastructure<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Object Groups• Replicas of an object form an object group• Each object group has an InteroperableObject Group Reference (IOGR)• Object group abstraction provides– Replication transparency– Failure transparency<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Identity Model• CORBA supports a weak identity model• Fault Tolerant CORBA requiresa strong identity model• Object groups identified by– FTDomainId, ObjectGroupId• Members of object groups identified by– FTDomainId, ObjectGroupId, Location<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Replication Styles• Passive Replication– Only one replica processes each requestOther replicas are available as backups if required– Lower memory and processing costs slowerrecovery from faults• Active Replication– Several replicas process each request– Fastest recovery from faults• Underlying mechanisms are the same for both<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Who Has Control?• Infrastructure-controlled fault tolerance– Automatic creation and allocation of replicas– Automatic maintenance of replica consistency– More sensible for complex programs on servers• Application-controlled fault tolerance– Precise control over object creation and allocation– Application algorithms maintain replica consistency– May be necessary for embedded systems<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Fault Tolerance for the Client• Failover– If Server does not respond, Client should tryagain using the same or an alternate address– If Client transmits its request more than once,it should not be executed more than once• Addressing– If Client uses an obsolete address, Servershould supply an up to date address• Loss of Connection– If Client’s connection to Server fails,the Client’s ORB should be informed prompty<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Fault Tolerance for the Server• Object Replication• Object Group Properties– Property Manager interface• Creating Fault-tolerant Objects– Generic Factory interface– Object Group Manager interface• Detecting Faults• State Transfers<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Fault Tolerance Domains• Aid application management and provide forscalability• Each Fault Tolerance Domain is managed by asingle Replication ManagerHawaiiLocationA1Host 1IIOPMessageover TCP/IPORBwithoutsupport forFault ToleranceHost 2GatewayB1San JoseDomainHost 3B2D1C1C2Host 4Wide AreaDomainHost 5C3E1E2E3F1Host 7Host 6BostonDomainF2<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

set_properties()create_object()Architectural OverviewReplicationManagernotificationsFaultNotifierFaultDetectorcreate_object()fault reportsis_alive()ClientServerServerCS1S2FactoryFaultDetectorFactoryFaultDetectorCORBA ORB CORBA ORB CORBA ORBLoggingMechanismRecoveryMechanismLoggingMechanismRecoveryMechanismLoggingMechanism<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Outlineâ2. Fault Tolerance Mechanismsa. Addressingb. Failover<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Interoperable Object Group Reference(IOGR)• An IOGR is a multiple profile IOR• Each profile contains a TAG_GROUPcomponent, consisting of– FTDomainId– ObjectGroupId– ObjectGroupRefVersion• At most one profile may contain aTAG_PRIMARY component, which gives a hintas to which profile corresponds to the primary<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Interoperable Object Group ReferenceType_idNumber ofProfilesIIOP ProfileIIOP ProfileIIOP ProfileMultipleComponents ProfileTAG_INTERNET_IOPProfileBodyIIOPVersion Host PortObjectKeyComponentsNumber ofComponentsTAG_GROUPComponentTAG_PRIMARYComponentOtherComponentstag_group_versionft_domain_idobject_group_idobject_group_version<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Profiles AddressObject Group MembersInteroperable Object Group ReferenceProfile S1 Profile S2 Profile S3ServerReplicaS1ServerReplicaS2ServerReplicaS3<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Access via IIOPDirectly to PrimaryInteroperable Object Group ReferenceProfile S1 Profile S2 Profile S3ServerReplicaS1ClientIIOPmessageServerReplicaS2ServerReplicaS3<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Profiles Address GatewaysInteroperable Object Group ReferenceProfile G1 Profile G2GatewayG1GatewayG2ServerReplicaS1ServerReplicaS2ServerReplicaS3<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Access via IIOP and a GatewayInteroperable Object Group ReferenceProfile G1 Profile G2ClientIIOPmessageGatewayG1GatewayG2ProprietarymulticastprotocolServerReplicaS1ServerReplicaS2ServerReplicaS3<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Direct Access viaProprietary Multicast ProtocolInteroperable Object Group ReferenceProfile G1 Profile G2ProprietarymulticastprotocolServerReplicaS1GatewayG1ClientServerReplicaS2GatewayG2ServerReplicaS3<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Most Recent Object Group Reference• ProblemObject Group Reference may not correspond tocurrent membership of the server object group• SolutionGROUP_VERSION Service ContextTAG_GROUP component of IOGR containsGroup Version Number (GVN) for theserver object groupClient ORB puts GVN in the GROUP_VERSIONService Context of the client’s request messagefor the server object group<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Most Recent Object Group Reference• Server ORB extracts the GVN from therequest message• If server GVN = GVN from client– Primary: Process request– Backup: Log request• If server GVN > GVN from client– Throw LOCATE_FORWARD_PERM with IOGR• If server GVN < GVN from client– Get new IOGR from ReplicationManager<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Outlineâ2. Fault Tolerance Mechanismsa. Addressingb. Failover<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Failover Semanticswith Fault TolerancePermitted Failover ConditionsCompletion StatusCOMPLETED_NOCOMPLETED_MAYBECORBA ExceptionCOMM_FAILURETRANSIENTNO_RESPONSEOBJ_ADAPTER<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Transparent Reinvocation• ProblemWith reinvocation for COMPLETED_MAYBE,at-most-once semantics might be violatedif no extra mechanisms are in place• SolutionREQUEST Service Context– Client Id– Retention Id– Expiration TimeAllows server ORB to recognize that a request isa repetition of a previous requestIf it is, server does not reexecute the request butreturns the reply that was previously generated<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Transport Heartbeats• Problem– Host or connection fails during clientinvocation– TCP/IP connection not cleanly torn downand Client ORB hangs on the connection• Solution– Periodic heartbeat messages over theconnection<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Transport HeartbeatsClient Side• HeartbeatPolicy– Heartbeat – On/Off– Heartbeat Interval– Heartbeat Timeout• If profile hasTAG_HEARTBEAT_ENABLED set to true,– Client can setHeartbeatPolicy values– Client ORB invokes_FT_HB() on serverServer Side• TAG_HEARTBEAT_ENABLED componentin profile• HeartbeatEnabledPolicyallows server to turnheartbeats on and off• Server ORB respondsto _FT_HB()<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Outline3. Fault Tolerance Propertiesa. Replication Styleb. Membership Stylec. Consistency Styled. Fault Monitoring Stylee. Fault Monitoring Granularityf. Factoriesg. Initial Number of Replicash. Minimum Number of ReplicasI. Fault Monitoring Interval and Timeoutj. Checkpoint Interval<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Replication Style• Stateless– Read-only access to static data• Cold passive replication– Recovery from faults using state information andmessages recorded in a message log– Slowest recovery from faults• Warm passive replication– Current state of the "primary" replica is transferredperiodically to the "backup" replicas– More rapid recovery from faults• Active replication– Every replica executes the invoked methods– Very rapid fault recovery<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Active ReplicationClientinvokes a method ofServer AServer AObject ObjectObject Object ObjectEternalEternalEternal Eternal EternalReliable totally ordered multicastServer BEternal Eternal EternalSTOPObject Object ObjectSTOPDuplicate repliessuppressedSTOPSTOPDuplicate invocationssuppressedReliabletotally orderedmulticasts forrequests and replies<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Passive ReplicationClientinvokes a method ofServer AEternalEternalPrimaryreplicaServer AObject ObjectObject Object ObjectOnly primary replicaof Server Aexecutes the methodEternal Eternal EternalReliable totally ordered multicastServer BObject Object ObjectEternal Eternal EternalPrimaryreplicaOnly primary replicaof Server Bexecutes the methodReliabletotally orderedmulticastfor state transferReply returnedfrom primary replica of Server Bto primary replica of Server A<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Membership Style• Infrastructure-Controlled– Fault Tolerance Infrastructure creates multiplereplicas of an object (members of an object group)and allocates them to appropriate hosts• Application-Controlled– The application determines when and how manyreplicas to create and the hosts on which theyshould be created<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Infrastructure-ControlledMembership StyleIOGRProfile Profilecreate_object()ApplicationObject AApplication directsReplication Manager tocreate an object groupReplication Managercreates and adds themembers to the groupReplicationManagercreate_object()Host PObject BB1Host QObject BB2FactoryFactory<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Application-ControlledMembership StyleIOGRProfile Profilecreate_member()ApplicationObject AApplication directsthe Replication Managerto create a memberat a specific locationand add it to the groupReplicationManagercreate_object()Host PObject BB1Host QObject BB2Factory<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Application-ControlledMembership StyleIOGRProfile Profileadd_member()ApplicationObject AApplication createsa member and directsthe Replication Managerto add it to the groupReplicationManagerHost PObject BB1Host QObject BB2create_object()Factory<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Consistency Style• Infrastructure-Controlled– Fault Tolerance Infrastructure maintainsstrong replica consistency of the object replicasusing logging, checkpointing, activation, andrecovery• Application-Controlled– The application is responsible for maintainingwhatever consistency it requires, using its ownmechanisms– No logging, checkpointing, activation or recoveryare provided by the Fault Tolerance Infrastructure<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Strong Replica Consistency• Maintained for object groups that have theInfrastructure-controlled Consistency Style• For Active replication, at the end of eachoperation, all of the members of theobject group have the same state• For Passive replication, at the end ofeach state transfer, all of the members ofthe object group have the same state<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

FaultMonitoringGranularityObject BObject AB1A1• Member• Location(proxy objectfor the location)• Location and Type(proxy objectof given typefor the location)Object BObject AB1A1Object BObject AB1A1<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Factories• Sequence of FactoryInfo– Factory that can be used to create amember of the object group– Location at which factory is to createa member of the object group– Criteria that the factory is to use whencreating the member of the object group,e.g. initialization values, constraints onthe member, etc<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Outline4. Fault Tolerance Managementâa. Replication Managementb. Fault Managementc. Logging and Recovery Management<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Replication Management• Replication Manager maintainsobject groups (replicated objects)and fault tolerance propertiesof the object groups– Replication Style– Membership Style– Consistency Style– etc<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Replication Management• Replication Manager interface providesmethods to register and obtain Fault Notifier– register_fault_notifier()– get_fault_notifier()• Replication Manager interface inherits from– Property Manager– Object Group Manager– Generic Factory<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Property Manager• Fault tolerance properties may be defined– For all replicated objects (object groups)– For all replicated objects of a type– For a specific replicated object at creation– For executing replicated objects• More specific definitions override more generaldefinitions<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Property Manager Interface• set_default_properties()• get_default_properties()• remove_default_properties()• set_type_properties()• get_type_properties()• remove_type_properties()• set_properties_dynamically()• get_properties()<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Property Manager Interfacevoid set_type_properties(in TypeId type_id,in Properties overrides)raises(InvalidProperty, UnsupportedProperty);Properties get_type_properties(in TypeId type_id);void remove_type_properties(in TypeId type_id,in Properties props)raises(InvalidProperty, UnsupportedProperty);<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

When Can Properties Be Set?Replication StyleMembership StyleConsistency StyleFault Monitoring StyleFault Monitoring GranularityFactoriesInitial Number of ReplicasMinimum Number of ReplicasFault Monitoring Interval andTimeoutCheckpoint IntervalDefault✔✔✔✔✔✔✔✔✔Type✔✔✔✔✔✔✔✔✔✔Creation✔✔✔✔✔✔✔✔Dynamic✔✔✔✔✔<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Generic Factory Interface• Inherited by Replication Managerand invoked by applicationto create or delete an object group• Implemented by Application and invokedby Replication Manager or Applicationto create or delete an individual object replica• create_object()• delete_object()<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Generic Factory Interfacetypedef Object ObjectGroup;typedef any FactoryCreationId;Object create_object(in TypeId type_id,in Criteria the_criteria,out FactoryCreationId factory_creation_id)raises(NoFactory, ObjectNotCreated, InvalidCriteria,InvalidProperty, CannotMeetCriteria);void delete_object(in FactoryCreationId factory_creation_id)raises(ObjectNotFound);<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

create_object()Generic FactoryReplicationManagercreate_object()ServerS1ServerS2FactoryFactoryCORBAORBCORBAORB<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Object Group Manager Interface• create_member()• add_member()• remove_member()• set_primary_member()• locations_of_members()• get_object_group_ref()• get_object_group_id()• get_member_ref()<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Object Group Manager InterfaceObjectGroup create_member(in ObjectGroup object_group,in Location the_location,in TypeId type_id,in Criteria the_criteria)raises(ObjectGroupNotFound, MemberAlreadyPresent,NoFactory, ObjectNotCreated, InvalidCriteria,...);ObjectGroup add_member(in ObjectGroup object_group,in Location the_location,in Object member)raises(ObjectGroupNotFound, MemberAlreadyPresent,ObjectNotAdded);<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

create_member()Object Group ManagerReplicationManagercreate_object()ServerS1ServerS2FactoryCORBAORBCORBAORB<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Outline4. Fault Tolerance Managementa. Replication Managementâ b. Fault Managementc. Logging and Recovery Management<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Fault Management• Fault Detector– Part of Infrastructure– Supplier of fault reports to FaultNotifier• Fault Notifier– Receives fault reports from Fault Detectors andFault Analyzer• Fault Analyzer– Specific to Application– Both a consumer and a supplier of fault reports<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Fault Detection & NotificationReplicationManagerStructuredPushConsumerSequencePushConsumerpush_structured_event()push_sequence_event()FaultAnalyzerPullMonitorableis_alive()Application ObjectFaultDetectorFaultNotifierpush_structured_fault()push_sequence_fault()<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Fault Event Propagation• Fault Event Propagation– CosNotification::StructuredEvent– CosNotification::EventBatch• Types of Fault Event– ObjectCrashFault• If all objects ata Location failed,TypeId and ObjectGroupIddoes not exist• If all objects of a TypeIdat a Location failed,ObjectGroupId does not existDomain_name = FT_CORBAType_name = ObjectCrashFaultFTDomainIdLocationTypeIdObjectGroupIdmydomainmyhost/myprocessIDL:Bank:1.0<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 20001

Fault Event Suppliers & Consumers• Fault Event Supplier– Fault Detector– Pushes fault events• Fault Event Consumer– ReplicationManager, Consumer Object created byReplicationManager, or Application– Registers using connect methods– Adds constraints to filter fault events propagatedto it by the FaultNotifier<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Fault Notifier Interface• Supplier End– push_sequence_fault()– push_structured_fault()• Consumer End– connect_structured_fault_consumer()– connect_sequence_fault_consumer()– create_subscription_filter()– disconnect_consumer()<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Fault Notifier Interfacevoid push_structured_fault(in CosNotification::StructuredEvent event);void push_sequence_fault(in CosNotification::EventBatch events);<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Fault Notifier Interfacetypedef unsigned long long ConsumerId;CosNotifyFilter::Filter create_subscription_filter(in string constraint_grammer)raises(CosNotifyFilter::InvalidGrammer);ConsumerId connect_structured_fault_consumer(in CosNotifyComm::StructuredPushConsumer consumer,in CosNotifyFilter::Filter filter);void push_structured_fault(in CosNotification::StructuredEvent event);<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Outline4. Fault Tolerance Managementa. Replication Managementb. Fault Managementc. Logging and Recovery Managementâ<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Logging & RecoveryManagementLogging for Active ReplicationClientCServerS1ServerS2CORBA ORB CORBA ORB CORBA ORBLoggingMechanismRecoveryMechanismLoggingMechanismRecoveryMechanismLoggingMechanism<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Logging & RecoveryManagementLogging for Warm Passive ReplicationClientCServerS1ServerS2CORBA ORB CORBA ORB CORBA ORBLoggingMechanismRecoveryMechanismLoggingMechanismRecoveryMechanismLoggingMechanism<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Logging & RecoveryManagementLogging for Cold Passive ReplicationClientCServerS1ServerS2CORBA ORB CORBA ORB CORBA ORBLoggingMechanismRecoveryMechanismLoggingMechanismRecoveryMechanismLoggingMechanism<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

• get_state()• set_state()Checkpointable Interface• get_update()• set_update()Updateable Interface<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Logging & RecoveryManagementState Transfer for Cold Passive ReplicationServerS1ServerS2get_state()CORBAORBCORBAORBRecoveryMechanismLoggingMechanismRecoveryMechanismLoggingMechanism<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Logging & RecoveryManagementRecovery for Cold Passive ReplicationServerS1ServerS2set_state()CORBAORBCORBAORBRecoveryMechanismLoggingMechanismRecoveryMechanismLoggingMechanism<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Outline5. Fault Tolerant Applicationsa. Pool of Processorsb. Internet Serverc. Telco Switching<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Pool of Processors• Multiple replicas of each application object• The replicas of an application object areassigned to different processors• No need for all objects to have the samenumber of replicas, or the same typeof replication• Replication Manager is replicatedjust like any other object<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Pool of ProcessorsThe replicas of an application object areassigned to different processorsReplicationManagerObj DObj EReplicationManagerReplicationManagerObj CObj EObj AObj CObj BObj AObj BObj FObj A Obj C Obj CObj H Obj DObj FObj G Obj BObj EObj H Obj G Obj FFaultTolerantORBFaultTolerantORBFaultTolerantORBFaultTolerantORBFaultTolerantORB<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Internet Server• Use pool of processors• Most clients will be outside our system andwill not understand fault tolerance• They communicate using IIOP/TCP/IP andenter the FT Domain through a gateway• If a gateway fails, the clients can failover toanother gateway<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Internet ServerInternetHost1Host2Host3UnreplicatedClientsGatewayGatewayGateway<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Internet Server• Must also provide back-end databaseto record inventory, orders, etc.• Do not attempt to replicate a database• Use a COTS fault-tolerant database• Access the database through a gateway• The gateway ensures that– The database is accessed once only– Replies from the database are multicastto all replicas<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Internet ServerInternetHost1Host2Host3UnreplicatedClientsGatewayGatewayGatewayGatewayGatewayCOTSDatabase<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Simple Switching Application• Line cards plugged into dual-bus backplaneEach card has embedded processor with ORB• Each line card is distinct; they are not replicas• Two control processors use active replication• Either control processor can control the switchThey are true replicas• Line cards communicate with bothcontrol processors<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Simple Switching ApplicationReplicated Control Computersuse embedded fault tolerancewith active replicationORBwith faulttoleranceGatewayUnreplicated computers on cardsuse client fault toleranceORB withclient faulttoleranceBackplane with dual bus interconnect<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Larger Switching ApplicationSwitchcontrollercardsShelf controller cardsLine cardsRedundantintershelfinterconnectShelf controller cardsLine cardsShelf controller cardsLine cards<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Larger Switching ApplicationSwitch control functionis shared betweenshelf control processorsShelf controller cardsLine cardsRedundantintershelfinterconnectShelf controller cardsLine cardsShelf controller cardsLine cards<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Outline6. Fault-Tolerant Hello Server Examplea. Hello Server Launcherb. Hello Server Factoryc. Hello Serverd. Hello Client<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server ExampleHelloServerLauncherPublishHello Serverobject groupreferenceReturnHello Serverobject groupreferenceObtainHello Serverreferenceset_type_properties()create_object()ReplicationManagerHelloClientInvokeHelloServerInvokecreate_object()ReturnHello ServerreplicareferencesHelloServerHelloServer HelloFactoryCreateHello Serverobject<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server Launcher1. Initialize the ORB2. Obtain a reference to the Replication Manager3. Narrow the reference to the Property Manager4. Invoke the set_type_properties() method of theProperty Manager to set the propertiesfor the Hello Server type5. Narrow the reference to the Generic Factory6. Invoke the create_object() method of theGeneric Factory to create a Hello Serverreplicated object7. Publish the Hello Server's IOGR in a filefor the client to read<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server LauncherHelloServerLauncherPublishHello Serverobject groupreferenceReturnHello Serverobject groupreferenceObtainHello ServerreferenceInvokeset_type_properties()create_object()ReplicationManagerHelloClientInvokehello()Invokecreate_object()ReturnHello ServerreplicareferencesHelloServerHelloServer HelloFactoryCreateHello Serverobject<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server Launcher Main// Set type properties for the Hello Server typetry{helloServertId = CORBA::string_dup("IDL:omg.org/HelloServer:1.0");helloServerProp.length(10);helloServerProp[0].nam.length(1);helloServerProp[0].nam[0].id= CORBA::string_dup("org.omg.ft.ReplicationStyle");helloServerProp[0].nam[0].kind = CORBA::string_dup("string");helloServerProp[0].val

Hello Server Launcher MainhelloServerProp[6].nam.length(1);helloServerProp[6].nam[0].id =CORBA::string_dup("org.omg.ft.InitialNumberReplicas");helloServerProp[6].nam[0].kind = CORBA::string_dup("string");helloServerProp[6].val

Hello Server LauncherHelloServerLauncherPublishHello Serverobject groupreferenceReturnHello Serverobject groupreferenceObtainHello Serverreferenceset_type_properties()Invokecreate_object()ReplicationManagerHelloClientInvokehello()Invokecreate_object()ReturnHello ServerreplicareferencesHelloServerHelloServer HelloFactoryCreateHello Serverobject<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server Launcher Main// Narrow the Replication Manager’s object reference// to a Generic Factory reference and invoke create_object()genFact = FT::GenericFactory::_narrow(repMgr);if (!CORBA::is_nil(genFact)){theCriteria.length(0);helloServerRef = genFact->create_object(helloServerTId, theCriteria, fcId);<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server LauncherHelloServerLauncherPublishHello Serverobject groupreferenceReturnHello Serverobject groupreferenceObtainHello Serverreferenceset_type_properties()create_object()ReplicationManagerHelloClientInvokehello()Invokecreate_object()ReturnHello ServerreplicareferencesHelloServerHelloServer HelloFactoryCreateHello Serverobject<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server Launcher Mainif (!CORBA::is_nil(helloServerRef)){// Publish the IOR of the Hello Server for the Client to readbuffer = orb->object_to_string(helloServerRef);fd = fopen( "HelloServerIOR", "w" );if (fd == NULL){ cerr

Hello Server Factory1. Invoked by FTCORBA rather than directly by the user2. Extract the ObjectId from the criteria3. Check the type_id to determine the object to create• The factory may be able to create several types of objects4. Create the object and activate it5. Record the object locally to enable deletion• The index of the object in this local sequence is returnedas an out parameter to enable deletion• A more sophisticated implementation of the factorywould reuse indices6. Return the object reference of the object just created<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server FactoryHelloServerLauncherPublishHello Serverobject groupreferenceReturnHello Serverobject groupreferenceObtainHello Serverreferenceset_type_properties()create_object()ReplicationManagerHelloClientInvokehello()Invokecreate_object()ReturnHello ServerreplicareferencesHelloServerHelloServer HelloFactoryCreateHello Serverobject<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server Factory ImplementationCORBA::Object_ptr FactoryImpl::create_object(const char* type_id, const FT::Criteria& the_criteria,FT::GenericFactory::FactoryCreationId_out factory_creation_id)throw(FT::NoFactory, …){ CORBA::Object_ptr helloServerRef;PortableServer::ObjectId *objId;int i, n, found;try{ i = 0; found = 0; n = the_criteria.length();while ((i>= objId);}i++;}if (found == 0) { …<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

if (strcmp(type_id, "IDL:omg.org/HelloServer:1.0") == 0){helloServerServant = new HelloServerImpl();myPoa->activate_object_with_id(*objectId, helloServerServant);helloServerRef = myPoa->create_reference_with_id(*objectId, “IDL:omg.org/HelloServer:1.0");}else{throw FT::NoFactory();}<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server FactoryHelloServerLauncherPublishHello Serverobject groupreferenceReturnHello Serverobject groupreferenceObtainHello Serverreferenceset_type_properties()create_object()ReplicationManagerHelloClientInvokehello()Invokecreate_object()ReturnHello ServerreplicareferencesHelloServerHelloFactoryCreateHello Serverobject<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server Factory Implementation// create_object() continuednumberOfObjects++;objectSeq.length(numberOfObjects);objectSeq[numberOfObjects-1] = CORBA::Object::_duplicate(helloServerRef);factory_creation_id = new FT::GenericFactory::FactoryCreationId();(*factory_creation_id)

Hello Server Factory Main1. Initialize the ORB2. Initialize the POA3. Create the Factory object• The Factory object is only invoked locally by FTCORBAIts object reference must never escape from this process4. Initialize FTCORBA• Connects to the Replication Manager• Receives commands to create objects andinvokes the Factory to create the local replica5. Set the ORB running<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server Factory Mainint main(int argc, char** argv){try{myOrb = CORBA::ORB_init(argc, argv);if (!CORBA::is_nil(myOrb)){rpObj = myOrb->resolve_initial_references("RootPOA");myPoa = PortableServer::POA::_narrow(rpObj);<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server Factory Mainif (!CORBA::is_nil(myPoa)){// Create myFactory servantmyFactoryServant = new FactoryImpl();myFactoryOid = myPoa->activate_object(myFactoryServant);// Create an object reference for myFactory servanttempMyFactory = myPoa->create_reference_with_id(*myFactoryOid, "IDL:omg.org/Factory:1.0");// Narrow the reference to the Generic Factory interfacemyFactory = FT::GenericFactory::_narrow(tempMyFactory);<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server Factory Main// Must NOT export the object reference for myFactoryServant// New replicated objects are created by invoking the// create_object() method of the Replication Manager// Initialize FTCORBAFTCORBA_init(myOrb, myPoa, myFactory, argc, argv);}// Using the ORB, make myFactory ready to receive requestsmyOrb->run();<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server ImplementationThe implementation of Hello Server is very simple1. Obtain the name of the client as a parameter2. Append “Hello” to the front of it3. Append “!” to the back of it4. Return the reply<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Server ImplementationCORBA::String HelloServerImpl::hello(const char* hellostring){char hellostring2[200];strcpy(hellostring2, "Hello ");strcat(hellostring2, hellostring);strcat(hellostring2, "!");}return CORBA::string_dup(hellostring2);<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Client Main1. Initialize the ORB2. Obtain a reference to an active Hello Server3. Invoke the hello() method of the Hello Server<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello ClientHelloServerLauncherPublishHello Serverobject groupreferenceReturnHello Serverobject groupreferenceset_type_properties()ObtainHello ServerreferenceHelloClientcreate_object()ReplicationManagerInvokehello()Invokecreate_object()ReturnHello ServerreplicareferencesHelloServerHelloServer HelloFactoryCreateHello Serverobject<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000

Hello Client Main// Obtain the Hello Server Object Reference: obj…// Narrow the object to a Hello ServerHelloServer_var server = HelloServer::_narrow(obj);if (!CORBA::is_nil((HelloServer_ptr)server)){CORBA::String_var returned;const char* hellostring = "client";// Invoke the hello() method of the remote serverreturned = server->hello(hellostring);cout

Fault Tolerant CORBA• For more information, contact:Louise Moser, Michael Melliar-Smith, Priya NarasimhanEternal Systems, Inc.P.O. Box 13963Santa Barbara, CA 93107Phone: +1-805-893-4897Fax: +1-805-893-3262Email: moser @eternal-systems.compmms @eternal-systems.compriya @eternal-systems.com<strong>Tutorial</strong> on Fault Tolerant CORBA © Eternal Systems, Inc, 2000