Enterprise Integration Using the IEC CIM - CIMug - UCAIug

EnterpriseIntegration Usingthe IEC CIMScott Neumann, UISOLMargaret Goodrich, SISCOLucas Malesku, BC HydroFrank Wilhoit, AEP

Introduction• The purpose of this presentation is todescribe a process for using the CIM forenterprise integration• This process is reflective of IECstandards efforts, best practices andintegration experiences at major utilitiesincluding AEP and BC Hydro• This is an attempt to provide an overviewof key topics given time constraintsProprietary and Confidential

What are we integrating?• Above is a common example for distribution• Other examples are transmission or market systems• Can not assume one size fits allProprietary and Confidential

People and Process• Integration success involves people and process as much astechnology• It is necessary to have people from different ‘silos’ work,recognizing that not everyone has the same knowledge,responsibility, focus or skill set• The architecture and processes need to be well documented,where opportunities for training and discussion are provided to besupportive of ‘buy in’• A strong governance process is needed to insure architectureconformance, but…• Processes are also needed to address cases where exceptions areneeded• Time and budget are factors that must be recognized, and mayfactor into specific integration priorities and decisionsProprietary and Confidential

Documentation• Documentation is key for design, governance, development, testing,deployment and support… avoiding dependencies upon ‘tribal’ knowledge• Documentation must be useful and provide value to a given audience, usinga set of standard templates and avoiding replication of information• Use of industry standards helps avoid reinventing wheels• The architecture should be documented in a manner that provides guidanceall those doing modeling, design and development• Things that need to be documented to support integration include:• Use Cases• Functional requirements• Non-functional requirements• System Interaction Specification• Interface designs for source and target(s)• Models and mappings• Integration patterns• In general, if something isn’t documented, take the view that it doesn’t exist!Proprietary and Confidential

Uses Cases• Everything needs to start with use cases… as they help definerequirements and are supportive of testing• In the simplest terms, a use case answers the question: ‘WHOdoes WHAT to WHO, WHEN, and WHY do they do it?’• The ‘WHO’ are called actors, which may be persons with specificroles or systems• A sequence diagram should be defined to describe the informationflows between actors, describing the WHAT and the WHEN• Each WHAT is typically viewed as a potential message• The WHY should be provided as background information• The use case does NOT say HOW!• Uses cases can be first defined at a business level, whichthen can guide the development of more detailed technicaluse cases as needed for implementationProprietary and Confidential

Modeling Levels7Proprietary and Confidential

Conceptual Models• Very important that it can be readily understood by business stakeholders• Provides the needed vocabulary for business-level use cases• Can be readily mapped to a logical model where a more detailed technicalview is providedProprietary and Confidential

Logical Models• The IEC CIM is the primary logical model ofinterest for this audience• However, there are other logical models that maybe leveraged for integration purposes, especiallywhen we get outside of the core electrical domain• Logical models are usually, but not alwaysexpressed as UML (e.g. OAGIS defined usingXSDs)• Sometimes logical models are derived fromlegacy XML schemas, entity-relationship models,etc.Proprietary and Confidential

Logical Models• CIM fragment with classes, inheritance, associations,attributes and data typesProprietary and Confidential

Canonicals• Canonical data models are used for informationexchanges to provide a common reference• Components convert to and from the canonical datamodel as needed to facilitate a given informationexchange• Using the CIM, canonicals are defined using ‘contextualprofiles’, where a subset of the CIM (and potentiallyother logical models) are leveraged• Each element of a profile/canonical can be traced backto a source element in a logical model• A canonical data model is a physical data model that isoften realized as an XSDProprietary and Confidential

Example Modeling ProcessProprietary and Confidential

Tools• Process needs to be supported by tools• Need to avoid any ‘unique’ modeling approaches thatcould potentially result in rework, lock-in or inability toleverage tools of choice• Should understand the consequences of the use ofany tool or any forced migration to another tool• Must avoid models being a ‘dead end’, where theyare only useful as a reference instead of being usedfor the generation of useful design artifacts• Tools and approaches must minimize the impact ofdesign changes or normal evolution• Don’t use a hammer for everything, use the right toolfor the right jobProprietary and Confidential

Enterprise Architect• A.k.a. ‘EA’• The UML tool of choice for maintenance of theIEC CIM, where CIM releases are now availableas ‘.eap’ files• Low cost, but extremely powerful commercialUML tool• Common uses are for development of classdiagrams and sequence diagrams• Can export models in a variety of formats,including many XMI options• Now the most widely used UML tool in the utilityindustry in large part because of the CIMProprietary and Confidential

CIMTool• Open source Eclipse plug-in that provides an IDE for profiledevelopment and management• Freely available from cimtool.org• Can import models from UML tools such as EA• Has model validation capabilities• Follows the draft TC57 XML NDR• Can also merge models together, and is not limited to CIM• Can apply a variety of syntax rules to profile definitions,generating XSDs, RDFS, HTML, DDL, etc.• Agreements on an IEC TC57 XML Naming and DesignRules specification are both validating the use of CIMToolnow and creating opportunities for other compliant tools (e.g.CIMContextor)Proprietary and Confidential

CIMTool IDEProprietary and Confidential

Using the CIM• The CIM can be used in many ways:1. Where it is the basis of a predefined, standardexchange profile or interface definition2. To extend an existing standard profile or interfaceto meet local requirements3. Where it is used to create new exchange profilesand interfaces where standards do not exist• For items 2 and 3 above, it is often necessary toextend the CIM• The way in which the CIM is extended andprofiles defined must be carefully consideredProprietary and Confidential

CIM Extension Methods• Extensions can be directly applied to a givenCIM release, in terms of new classes,attributes and associations• There are two ways to identify the extensionswithin a UML model file:• Stereotyping• Tagged values• It is also possible to put the extensions into aseparate model file in conjunction with theabove methodsProprietary and Confidential

CIM Extensions: Stereotypes• Use of Stereotyping Functions in EA• Use as the label for customtop-level package• Use as the label for each newclass or attribute added to an existing CIMpackage where name designates the entity orproject that needs the extension. Forexample - or the name of theentity.Proprietary and Confidential

CIM Extensions: Tag Values• Use Tagged Value Function in EA• Tagged Values may be preferred overStereotype since EA allows simple viewingusing the “View | Tagged Values” feature• Tagged Values may be added to bothAttributes and Associations• Tagged Values is the method already usedby CIMTool to manage namespaces in UMLProprietary and Confidential

CIM Extensions: Extension Models• Extensions can be put into a separate file• One option is to leverage CIMTool merge capabilities, whichthen makes it easier to take on new base CIM versions• CIMTool will ‘merge’ where two models have classes thatmatch in namespace and class name.Proprietary and Confidential

Namespaces• Namespaces are used in models, XSDs, profile definitions and WSDLs• Changing namespaces can force significant levels of reimplementationeffort… that might otherwise be avoided• Along the lines of OASIS recommendations, avoid changingnamespaces unless there is an absolute need to do so to reflect aversion that would otherwise be incompatible• Use the ‘version’ attribute to track version numbers, e.g.:version="1.0.0"• Each profile should/will have its own namespace, avoiding conflictswhere a CIM class might be used differently in different profiles, e.g.:targetNamespace="http://iec.ch/TC57/2011/EndDeviceControls#"Proprietary and Confidential

Traceability• Need to be able to trace from a canonical/physical modelback to a logical model• SAWSDL is a standard for semantic annotations• It is used to provide traceability within a contextual profile• This helps to avoid any semantic confusion• XML Schema fragment using SAWSDL:Proprietary and Confidential

Integration Choices• Depending upon the integration scenario, there are many factorsthat may drive the implementation of an information flow• Some of the implementation options include:– Messages through an ESB that convey transactions to create, change or deleteof objects– Event notification messages through an ESB that indicate the create, changeor delete of objects– Messages through an ESB that support query requests, potentially withasynchronous replies– Web service requests for queries or transactions– Data set exchanges via CIM/XML files– Data set updates using incremental CIM/XML– Information transfer to a data warehouse using ETL mechanisms– HTTP requests• There are a variety of technical, performance, availability andsecurity considerations that would need to be factored/addressedfor any integration choiceProprietary and Confidential

CIM/XML• See IEC 61970-452/501/552• Used primarily for model exchanges in contrast toexchange of dynamic information, queries ortransactions• Uses RDF Schema to define the Profiles to exchange• Also supports model changes using incrementalCIM/XML• While typically exchanged using a ‘Claim Check’pattern, can also be conveyed within messagepayloads• Does many things that are difficult to do using XMLSchema-based messagesProprietary and Confidential

Incremental CIM/XML Example

Common Message Envelope• IEC 61968-1 defines a general approach for transport independentmessages that minimally consist of a verb, noun and payload• Messages using a common message envelope (CME) can be wrapped byother wrappers (e.g. SOAP, WSDL, JMS)• Usage of different parts of the CME (e.g. Request, Reply Payload) aredependent upon messaging pattern and verb• Payload is defined using an XML Schema ‘any’Proprietary and Confidential

Message ExamplecreatedMyBusinessObjectSourceApp353371More stuff• This design wraps a CMEinside the SOAP body• This example shows whatwould be seen ‘on thewire’• You can’t tell if thismessage was generatedusing a WSDL or not!• The CME even allows forpayloads that do not havea corresponding XSDProprietary and Confidential

Basic Integration PatternsSynchronous Request/ReplyPoint-to-PointMulticastingAsynchronous Request/ReplyProprietary and Confidential

Patterns and Verbs• Queries are indicated by the use of a ‘get’ verb on therequest in a request/reply pattern, where the payloadof the response message should return the requestedinformation• Transactions are indicated by the use of verbs suchas ‘create’, ‘change’, ‘close’, ‘cancel’, ‘delete’, andcould use request/reply or point-to-point patterns• Both queries and transactions may result in responses(verb=‘reply’), which may indicate success or errors• Events are indications that something has happenedof potential interest, using past tense verbs such as‘created’, ‘changed’, ‘closed’, ‘canceled’, ‘deleted’and typically use a multicast patternProprietary and Confidential

Transports• IEC 61968 is by definition transport independent, but the reality is thata wide range of messaging technologies have largely given way toJMS and HTTP• JMS provides a powerful messaging API, where the worst thing aboutJMS is the name, as it is not limited to use by Java applications• HTTP and JMS have some trade-offs:– JMS provides a reliable transport with the capability to queue andmulticast messages– HTTP provides a standard ‘on the wire’ protocol, but JMS does not• SOAP may be layered on both of these transports, where the IEC61968 message envelope may be placed within the SOAP body• When SOAP is used, it can be possible to define a WSDL that alsoprovides a description of the interface• Other transports can also be considered, such as AMQP and REST• ESBs support the use of intermediaries• IEC 61968-100 is being drafted to define specific, transport-dependentimplementation profiles and integration patternsProprietary and Confidential

Web Services• IEC 61968 messages can be ‘wrapped’ by a WSDL• Two basic approaches: Loose coupled (coarse grained) or tightlycoupled (strongly typed, fine grained)• For broadest compatibility, WSDL should be WS-I compliant anduse document-wrapped style• Remember that HTTP is not the only transport possible… as JMScan be used to provide delivery guarantees, multicasting and otherfeatures not possible with HTTP• The use of a SOAP envelope enables the use of many security tools• WSDLs are sometimes less important than one might think,especially given the emergence of RESTProprietary and Confidential

Names• Object naming is an important aspect of integration that is oftenoverlooked especially where all systems do not use common mRIDs• Issues:– The same object may be known in different system by different names– Messages need to use an identified that is known by source and targets– Some systems may need to be aware of multiple identifiers for a single object(e.g. local ID and external ID)• Sources and targets MUST agree upon names to be used, or rely uponthe use of bilateral mapping tables for object names• A naming service could be used for allocating and managing names• The performance implications of name lookups needs to be carefullyconsidered• Name class introduced in CIM v15 to help address these issues, providinga more realistic and useful model of object naming• In theory, an mRID is just another type of name!Proprietary and Confidential

CIM Name Class• Identified Objects can have many Names• Each Name.name instance can be associated with a differentNameType• NameTypes can be associated with a NameTypeAuthoritythat can be responsible for allocating Name.name valueswithin a NameType• IdentifiedObject.name preserved for legacy reasonsProprietary and Confidential

Names in Messages• Common usage of Name class in a message XSD• NameType often option, but should be provided by source ifmore than one Name instance provided• Target process will map the object using one of the names, asthe name is either used locally or defined within a bilateralmapping tableProprietary and Confidential

Name Mapping Example• Source sends amessage related toobject ‘ABC’• Within messageName.name valueis set to ‘ABC’• Name.NameType.name can be usedto specificallyidentify the type ofnameExternalIDLocal IDABC 1XYZ 2RTY 345INB 67433MJG 222ECV 555GGG 225Note: This is one example, there are many options• Target receives themessage• Given that theName.name value isunknown, a tablelookup is required toknown that objectnamed ‘ABC’ asprovided by theexternal system isequivalent to locallynamed object ‘1’• This is a bilateralagreement requiredfor integration whereall systems do notshare a globallyunique IDProprietary and Confidential

Common Services• Common services are often implementedwithin an integration infrastructure to providecommon functionality that is supportive ofintegration• Common examples include:– Logging and Exceptions– Claim Check– Notifications– Identity Management– Large message handlingProprietary and Confidential

Claim Check Pattern• Commonly used for exchange of large files, such as thosecontaining CIM/XML• Functionality may wrapped within a serviceProprietary and Confidential

Security• Key terms: Authentication, Authorization, Confidentiality,Integrity, Non-Repudiation, Denial of Service, Trust, …• If properly planned and designed, the implementation of security canbe a last step (depending upon the implementation approach)• Different security policies will be needed in different situations, suchas in the cases of intra-enterprise, inter-enterprise or extra-enterpriseintegration or where data has sensitivity and/or confidentialityrequirements• Tools like TIBCO Policy Manager (as an example) make it verysimple to implement necessary security policies using agents, wheredevelopers need otherwise not be concerned with security policy• If you are going to use SSL/TLS, how do you create and manage yourcertificates?• Does the integration impact critical infrastructure (e.g. as per NERC)?• One size does not fit all integrationsProprietary and Confidential

Validation• How is an XML message validated?• XML Schema validation is very limited and can only beused to validate the form of the XML and very little withrespect to content• Schematron (an ISO standard) provides for moresophisticated validation rules• For example, how do you validate an object identifierthrough simple rules?• Ultimately the target application will need to decidewhether from its perspective a message is validProprietary and Confidential

Error Handling• Error handling is dependent upon the integrationpattern• CME provides detailed structures for reporting errorsthat are both human and machine readable• In a request/reply pattern, the server can simply detectand report an exception back to the client• In a publish/subscribe or point-to-point pattern a meansto log and report exceptions is needed• A notification infrastructure may also be needed toreport errors to support staffProprietary and Confidential

Integration Beyond the ESBProprietary and Confidential

Data Warehouse Challenges• The IEC CIM defines a normalized logical data model• Data warehouses are typically denormalized• The CIM is complex:– Deep inheritance relationships– Many associations between classes• Objects within models need to be versioned as theychange over time, and historical views are needed• Need many dimensions that allow facts to be analyzed inmany ways• Facts may involve summarization and aggregation ofsource data in order to be useful• Many sources of data need to be staged and correlatedProprietary and Confidential

ETL• Extract, Transform and Load (ETL) is used to movedata from sources to staging tables to tables within thedata warehouse• ETL processes are typically defined for both bulk andincremental updates• Incremental updates are important so that history canbe preserved within the data warehouse• Microsoft SQL Server Integration Services is but onecommonly used tool to implement an ETL integrationProprietary and Confidential

ETL Pattern• Non-messaging integration used to populate a database, such as a datawarehouse• Data may undergo significant level of transformation and/or summarization• In the case of a data warehouse target there may be a significant level ofdenormalizationProprietary and Confidential

More Information• IEC TC57 Sharepoint (for CIM UG and IECTC57 members): http://iectc57.ucaiug.org• E-mails:‣ Scott: sneumann@uisol.com‣ Margaret: margaret@sisconet.com‣ Lucas: lucas.malesku@bchydro.com‣ Frank: fewilhoit@aep.com46Proprietary and Confidential