Syndication Troubleshooting Guide for Common Scenarios

Syndication Troubleshooting Guide for Common ScenariosAuthor: Sunil Bhatnagar, WCM US L2 supportLast Updated: 28 th June 2010The purpose and usage of this guide is to provide quick troubleshooting tips and steps to address commonlyreported scenarios for syndication failure,1

Table of contents1. Understand the syndication environment.....................................................32. Verify the enviroment and configuration......................................................43. Facts to keep in mind.......................................................................................54. Reading the traces............................................................................................85. EventLog and syndication...............................................................................116. Most Common Issues (and how to address them).........................................136.1 Automatic syndication is not happening6.2 Syndication updates complete but with errors6.3 Syndication performance is not good6.4 Syndication changes donot show up on all cluster members6.5 Syndication of a particular item(s)is not happening6.6 Syndication completes but not all items showup on the subscriber sideor Customer needs to verify if all items on the syndicator match with all items on thesubscriber7. Syndication and Clustering...........................................................................218. Syndication Resources....................................................................................232

1. Understand the syndication environment• Is the syndication being done for the first time to the subscriber ?• Is the syndication being done between One Syndicator and One Subscriber ormultiples ? If multiples, get details in terms of a topology of the syndication setup,detailing number of syndication servers, number of syndicator/subscriber objects on aserver and the flow of syndication.• Is either the syndicator or the subscriber a part of a cluster ? If so is syndication beingdone from/to one node on a cluster or the entire cluster through httpserver ?• Is the syndication being done for one library or multiple libraries ? If multiple, do theitems in one library reference the other ?• Is this synidcation being done for All items or All Live Items ?• Is this automatic or manual syndication ?• Are the syndicator and subscriber pointing to the same ldap ? If different ldaps then arethe users and groups identical ?• Are the syndicator(s) and subscriber(s) at the same fix level ?• Has any authoring been doen on the subscriber ?• If doing All items syndication - is versioning enabled on the syndicator ?• Did syndication ever work completely ?• Are any of the libraries being syndicated, Migrated libraries, if so, from whichversion ?• Collect the Syndication MustGather information as below:for V6.0.Xhttp://www-01.ibm.com/support/docview.wss?rs=688&uid=swg21243980for V6.1.Xhttp://www-01.ibm.com/support/docview.wss?rs=688&uid=swg21316241• Capture a screenshot of the syndicator and subscriber objects3

2. Verify the enviroment and configurationFrom the MustGather files:From the versioninfo.log verify both syndicator and subscriber have the same exact WCM version andWCM fixes installed. Both the syndicator and subscriber systems should match with regards to WCMversion and WCM fix level.Check for loose class files on the syndicator and subscriber system as this will impact the fix levels.location for these loose class files:V6.0.X: \wcm\shared\appV6.1.X: \wcm\prereq.wcm\wcm\shared\appCheck for the following properties in the syndicator and subscriber's -WCMConfigService.properties filelocation for this file:V6.0.X: \wcm\shared\app\config\wcmservices\WCMConfigService.properties fileV6.1.X: \PortalServer\wcm\shared\app\config\wcmservicesdeployment.itemDispatcherUrl=http://${WCM_HOST}:${WCM_PORT}/${WCM_CONTEXT_ROOT}/connect/?MOD=ItemDispatcherdeployment.syndicatorUrl=http://${WCM_HOST}:${WCM_PORT}/${WCM_CONTEXT_ROOT}/connect/?MOD=Synddeployment.subscriberUrl=http://${WCM_HOST}:${WCM_PORT}/${WCM_CONTEXT_ROOT}/connect/?MOD=Subsdeployment.subscriberOnly=false (this should never be set to true on the syndicator)deployment.itemChangedTaskDelay=30 ( in seconds – this is the default value)Items to verify from the screenshots:Syndicator ObjectName of the syndicator - (Keep a note of this as this will be referenced in the logs)Name of the subscriber - (Keep a note of this as this will be referenced in the logs)Subscriber ID The ID of the subscriber, used to setup the subscription - Make sure this matches the ID onthe subscriber objectSubscriber URL The URL of the subscriber - this should be a fully qualified URL (in some cases an IPaddress) and should match the value default on the subscriber object. (Note: This value reflect the values ofthe variables WCM_HOST and WCM_PORT defined in WAS.These variables can be overridden byhardcoding the values in the WCMConfigService.properties file)Libraries selected for syndication.Configuration of the item gatherer - all items or all live items syndicationSyndicator is enabled or not – the checkbox should be checkedSubscriber ObjectName of the subscriberThe name of the syndicatorThe ID of the syndicator used to setup the subscription.Syndicator URLEnable the subscriber4

3. Facts to keep in mind1. Syndication is just transferring content items stored in the JCR repository of the syndicator server to theJCR repository of the subscriber server2. The UUIDs of each of these items is maintained the same across the Syndicator and Subscriberdatabases.3. To have a simple picture in mind the syndicator and subscriber are sort of gatekeepers to the JCRDBthat stores the WCM content and processes these items through syndication as deemed necessary by theevent log.4. Copying of the JCR repository itself from the syndicator to subscriber is a brute force method ofensuring that all the data on the syndicator showsup on the subscriber. Here are somethings to considerwhen doing so:• This will impact PDM/PZN as well since they use the same JCRDB• We suggest this ONLY if this is a first time syndicationRefer to the following link for details on performing this:http://www-10.lotus.com/ldd/portalwiki.nsf/dx/18082009091727PMWEB3JG.htm5. Syndication uses the http protocol and does not support the use of https although it may worktechnically as long as the certificates are setup correctly between the servers.6. Resetting of the event log will cause a FULL REBUILD ( if automatic syndication is enabled or ifmanual syndication is initiated)7. Once the event log gets created, any content item saved to the JCR repository for a library is logged tothe event log as one or more entries. (depending on it's state and other settings, for example a live item islogged twice: once against the live item gatherer and again against the all items gatherer, there will alsobe additional entries for versions).8. The event log provides critical information to determine if an object is not getting syndicated.More information in the section: EventLog and Syndication9. IBM Support tools portlet has some useful tool that can be used in syndication troubleshootingThe portlet can be downloaded from here (P.S: Requires registering with the site):https://greenhouse.lotus.com/plugins/plugincatalog.nsf/assetDetails.xsp?action=editDocument&documentId=AE2BB2412F20AA318525772E006F7014Some of the tools available in this portlet are:DisplayNodesDisplayEventLogClearPersistenceTableClearcaches10. Authoring on the subscriber side can cause issues with syndication since the last modified date on theitem on the subscriber will change, as a result when the subscriber processes this item being pushedfrom the syndicator it might skip saving this item as it has a current or newer version of the item in itsdatabase. (Authoring on the subscriber side can be ok if the client is aware of the limitations andmanages them properly like in two way syndication scenarios where authoring is done on both serversbut in different libraries)5

11. Here is an example of a very typical syndication setup:Topology:Syndicator (Authoring server - Hasthe WCM Authoring portlet installed) =one way syndication=>Subscriber (Rendering server that hasthe WCM Local Rendering Portlet installed) to display the content - probably does not have theauthoring portlet installed) .Syndicating two libraries one content and one design. The content libraryhas the content items while the design library has the AT, PT, components etc.Typically the contentlibrary will reference the items in the design library. Both these libraries will need to be syndicatedsimultaneously for the syndication to succeed. Typically both these libraries are added to the syndicatorin live items scope. For typical setups such as above it does make sense to syndicate live items onlysince there is no authoring being done on the rendering server side and it is being used only to rendercontent - LRP/RRP will only display live content.So if a customer is doing ALL items syndication it isvalid to ask the reason for it and if they have a setup similar to above whether they would like toconsider doing live items12. Both subscription partners should share the same user repository (such as LDAP) if they use automaticsyndication. If this is not the case, disable automatic syndication and perform manual syndication. Thelatter should be followed by executing memberfixer. Please note that the purpose of automaticsyndication is defeated if the LDAPs are not the same or at the very least the users and groups are notidentical in both LDAPs with identical DNs.13. Key traces to be enabled ( on BOTH syndicator and subscriber):com.aptrix.deployment.*=allcom.aptrix.syndication.*=allFor event log related issuescom.ibm.workplace.wcm.services.eventlog.*=finest:com.ibm.workplace.wcm.services.transaction.steps.eventlog.*=finest(The eventlog trace will show if an item got added to the event log and how.)14. Syndication can handle the syndicator and subscriber servers being in different time zones as long as theserver clocks for both are not more than a minute apart with the timezone differences factored in. Forthe servers on the same timezone it will be the same rule of the server times not being more than aminute apart.15. Items that were deleted and then purged will not be syndicated if the event log is reset. To give anexample if an item was deleted and purged on the syndicator side, the entry is recorded in the event logso that the same action can be performed on the subscriber side upon syndication. However if theeventlog is reset prior to the syndication then the rebuilt event log will not have this entry and hence thisitem will not get syndicated.16. If you are resetting the event log then make sure that the event log traces:com.ibm.workplace.wcm.services.eventlog.*=finest:com.ibm.workplace.wcm.services.transaction.steps.eventlog.*=finest are applied prior to resetting the event log. Enable eventlog traces from WAS Adminconsole in config mode and save the configuration, then stop the portal server and reset eventlog. Makesure to remove traces from WAS Admin console in config and runtime mode, once eventlog rebuild iscomplete and the server has started up.The server startup with the eventlog traces set and the event logbeing built provides vital information.17. If the repository( library) is large ( large would mean in the range of 20,000-30,000 items)then thecustomer would want to increase the number of historical trace files and trace files size in WebSphereApplication Server admin console. A setting of 20 historical files and a file size of 50 MB is a goodnumber and should be adequate for most scenarios. The idea is to ensure that trace information is notlost due to file rollovers.18. Use the DisplayNodes tool included in IBM Support Tools Portlet for WCM to collect the output forany failing uuid. The displayNodes.jsp output will show all the references to that UUID and hence it is auseful tool in certain scenarios.6

19. Library access permissions are not syndicated. If the library does not exist on the subscriber, it will becreated during syndication but no access control settings are specified on the new library. On renderingservers you might want to specify different access control settings, for example you may want todisallow content entry for all users.20. On the syndicator, the PackageGenerator class generates the packages to send to the subscriber and onthe subscriber, the PackageProcessor class processes packages sent by the syndicator.21. When eventlog is rebuild, it processes every WCM library (enabled) regardless of whether it is in asyndication definition or not. This makes it long time to rebuild eventlog depending on how many WCMlibraries (enabled) are there. Disabled libraries are not processed and not added to eventlog.22. Syndication cannot be scheduled but the frequency of syndication occurrence can be set. By default it isset for 30 seconds but can be changed by changing the value deployment.itemChangedTaskDelay in theWCMConfigService.properties file. Also the occurrence of syndication in certain scenarios can becontrolled by using the publish date of content - the publishing of the content can be delayed and hencesyndication can be delayed. This can used for example - to allow syndication to happen only during nonpeak hours23. Prior to running the configuration task wcm-reset-event-log the Portal server needs to be stopped andthen restarted when the task has completed. The config task merely clears the event log DB, the eventlogis rebuilt when the portal server is started and may take a while to comeup. To address this PK88848introduced the module to reset the eventlog DB while the portal server is running. This fix is available inWCM Cumulative iFix 21(PK90387) for 6.0.1.X and WCM cumulative iFix 15 (PK90388) for V6.1.x.The FAQs related to this module are technoted here: http://www-01.ibm.com/support/docview.wss?uid=swg2139669124. One of the common misconception is that LIVE items syndication means syndicating only publisheditems. The Live Items syndication includes published items, expired items and non workflowed items(but not drafts, versions and deleted items)25. When performing a first time or large syndication it is recommended to turn of the JCR textsearch.Refer to the following technote on how to do that:http://www-01.ibm.com/support/docview.wss?rs=6h88&uid=swg212994987

4. Reading the traces1. Most activity occurs on the subscriber side since that is where the subscriber will go through the list ofitems to be saved, fetch these items from the syndicator database, add them to the respository and thensave the document in its own respository. Basically here is what happens in very very layman's terms(this maynot be technically very accurate but the idea is to give a mental picture of the syndicationfunctioning in the most common scenario): On the syndicator side the itemchangedtaskdelay valuedetermines when the system will check for updates to the event log. If it finds some updated items it willthen contact the subscriber and provide it with the list of items to be updated. The subscriber then startsprocessing each item in this list. It skips items that are already at the most current version (Note: youwill see the statement -"skipping item" in the logs) and it adds the newer modified items to the items thatneed to be updated. After doing that it will fetch theses items from the syndicator server's database andthen try to save these documents( items) on the subscriber JCRDB.Most syndication errors are seenduring this time. If the subscriber is able to process all the items sent by the syndicator it will send thesyndicator a 603 message code on receipt. If for some reason there are some items that failed to beprocessed by the subscribers those items will be processsed the next time the itemgatherer queries theeventlog.2. New items are saved first, followed by updates and lastly by removals3. Typically you will see a message at the end of the syndication in the subscriber trace logs such as this[6/4/08 22:20:02:141 EDT] 00000084 PackageProces I ****************** Start of subscription updatereport ******************[6/4/08 22:20:02:156 EDT] 00000084 PackageProces I Subscription update started:2008-06-04T22:19:17.125[6/4/08 22:20:02:156 EDT] 00000084 PackageProces I Subscription update finished:2008-06-04T22:20:02.141 (45016 milliseconds)[6/4/08 22:20:02:172 EDT] 00000084 PackageProces I Syndicator: syndicator6_to_subscriber4, URL=http://testsyndication1.raleigh.ibm.com:10038/wps/wcm/connect/?MOD=Synd[6/4/08 22:20:02:172 EDT] 00000084 PackageProces I Subscriber: subscriber4_from_syndicator6,URL=http://testsyndication2.raleigh.ibm.com:10038/wps/wcm/connect/?MOD=Subs[6/4/08 22:20:02:172 EDT] 00000084 PackageProces I Full update[6/4/08 22:20:02:188 EDT] 00000084 PackageProces I Updates sent: 21[6/4/08 22:20:02:188 EDT] 00000084 PackageProces I Removes sent: 0[6/4/08 22:20:02:203 EDT] 00000084 PackageProces I Items up to date: 20[6/4/08 22:20:02:203 EDT] 00000084 PackageProces I No errors were detected during this syndication.[6/4/08 22:20:02:219 EDT] 00000084 PackageProces I -------------------------- High Level Details---------------------------[6/4/08 22:20:02:234 EDT] 00000084 PackageProces I Start of library updates,time=2008-06-04T22:20:02.109, count=1[6/4/08 22:20:02:250 EDT] 00000084 PackageProces I Total items: 1[6/4/08 22:20:02:250 EDT] 00000084 PackageProces I End of library updates,time=2008-06-04T22:20:02.109, elapsed=0 milliseconds[6/4/08 22:20:02:250 EDT] 00000084 PackageProces I Start of draft item removal,time=2008-06-04T22:20:02.109, count=0[6/4/08 22:20:02:266 EDT] 00000084 PackageProces I Total items: 0[6/4/08 22:20:02:266 EDT] 00000084 PackageProces I End of draft item removal,time=2008-06-04T22:20:02.109, elapsed=0 milliseconds[6/4/08 22:20:02:281 EDT] 00000084 PackageProces I Start of draft item removal,time=2008-06-04T22:20:02.109, count=0[6/4/08 22:20:02:281 EDT] 00000084 PackageProces I Total items: 0[6/4/08 22:20:02:281 EDT] 00000084 PackageProces I End of draft item removal,time=2008-06-04T22:20:02.109, elapsed=0 milliseconds[6/4/08 22:20:02:297 EDT] 00000084 PackageProces I Start of new published items,time=2008-06-04T22:20:02.109, count=0[6/4/08 22:20:02:297 EDT] 00000084 PackageProces I Total items: 08

[6/4/08 22:20:02:312 EDT] 00000084 PackageProces I End of new published items,time=2008-06-04T22:20:02.109, elapsed=0 milliseconds[6/4/08 22:20:02:312 EDT] 00000084 PackageProces I Start of update published items,time=2008-06-04T22:20:02.109, count=0[6/4/08 22:20:02:312 EDT] 00000084 PackageProces I Total items: 0[6/4/08 22:20:02:328 EDT] 00000084 PackageProces I End of update published items,time=2008-06-04T22:20:02.109, elapsed=0 milliseconds[6/4/08 22:20:02:328 EDT] 00000084 PackageProces I Start of new draft items,time=2008-06-04T22:20:02.125, count=0[6/4/08 22:20:02:344 EDT] 00000084 PackageProces I Total items: 0[6/4/08 22:20:02:344 EDT] 00000084 PackageProces I End of new draft items,time=2008-06-04T22:20:02.125, elapsed=0 milliseconds[6/4/08 22:20:02:344 EDT] 00000084 PackageProces I Start of update draft items,time=2008-06-04T22:20:02.125, count=0[6/4/08 22:20:02:359 EDT] 00000084 PackageProces I Total items: 0[6/4/08 22:20:02:359 EDT] 00000084 PackageProces I End of update draft items,time=2008-06-04T22:20:02.125, elapsed=0 milliseconds[6/4/08 22:20:02:375 EDT] 00000084 PackageProces I Start of item removal,time=2008-06-04T22:20:02.125, count=0[6/4/08 22:20:02:375 EDT] 00000084 PackageProces I Total items: 0[6/4/08 22:20:02:375 EDT] 00000084 PackageProces I End of item removal,time=2008-06-04T22:20:02.125, elapsed=0 milliseconds[6/4/08 22:20:02:391 EDT] 00000084 PackageProces I Start of deferred new published content links,time=2008-06-04T22:20:02.125, count=0[6/4/08 22:20:02:391 EDT] 00000084 PackageProces I Total items: 0[6/4/08 22:20:02:391 EDT] 00000084 PackageProces I End of deferred new published content links,time=2008-06-04T22:20:02.125, elapsed=0 milliseconds[6/4/08 22:20:02:406 EDT] 00000084 PackageProces I Start of deferred new draft content links,time=2008-06-04T22:20:02.125, count=0[6/4/08 22:20:02:406 EDT] 00000084 PackageProces I Total items: 0[6/4/08 22:20:02:422 EDT] 00000084 PackageProces I End of deferred new draft content links,time=2008-06-04T22:20:02.125, elapsed=0 milliseconds[6/4/08 22:20:02:438 EDT] 00000084 PackageProces I Start of resolving references,time=2008-06-04T22:20:02.125, count=0[6/4/08 22:20:02:438 EDT] 00000084 PackageProces I Total items: 0[6/4/08 22:20:02:438 EDT] 00000084 PackageProces I End of resolving references,time=2008-06-04T22:20:02.125, elapsed=0 milliseconds[6/4/08 22:20:02:453 EDT] 00000084 PackageProces I Start of item renaming,time=2008-06-04T22:20:02.141, count=0[6/4/08 22:20:02:453 EDT] 00000084 PackageProces I Total items: 0[6/4/08 22:20:02:469 EDT] 00000084 PackageProces I End of item renaming,time=2008-06-04T22:20:02.141, elapsed=0 milliseconds[6/4/08 22:20:02:469 EDT] 00000084 PackageProces I Start of create versions,time=2008-06-04T22:20:02.141, count=0[6/4/08 22:20:02:469 EDT] 00000084 PackageProces I Total items: 0[6/4/08 22:20:02:484 EDT] 00000084 PackageProces I End of create versions,time=2008-06-04T22:20:02.141, elapsed=0 milliseconds[6/4/08 22:20:02:484 EDT] 00000084 PackageProces I Start of update versions,time=2008-06-04T22:20:02.141, count=0[6/4/08 22:20:02:500 EDT] 00000084 PackageProces I Total items: 0[6/4/08 22:20:02:500 EDT] 00000084 PackageProces I End of update versions,time=2008-06-04T22:20:02.141, elapsed=0 milliseconds[6/4/08 22:20:02:500 EDT] 00000084 PackageProces I Start of version removal,time=2008-06-04T22:20:02.141, count=0[6/4/08 22:20:02:516 EDT] 00000084 PackageProces I Total items: 0[6/4/08 22:20:02:516 EDT] 00000084 PackageProces I End of version removal,time=2008-06-04T22:20:02.141, elapsed=0 milliseconds[6/4/08 22:20:02:531 EDT] 00000084 PackageProces I Start of item status update,time=2008-06-04T22:20:02.141, count=0[6/4/08 22:20:02:531 EDT] 00000084 PackageProces I Total items: 0[6/4/08 22:20:02:531 EDT] 00000084 PackageProces I End of item status update,time=2008-06-04T22:20:02.141, elapsed=0 milliseconds[6/4/08 22:20:02:547 EDT] 00000084 PackageProces I --------------------------- Low Level Details---------------------------[6/4/08 22:20:02:547 EDT] 00000084 PackageProces I Start of library updates,time=2008-06-04T22:20:02.109, count=1 [DepRef(id:409ae28049c897ccab84bf55786cef3c type:com.ibm.workplace.wcm.services.library.Library nonDraft:true draft:false purged:false parentId:409ae28049c897ccab84bf55786cef3c timeStamp:1211200237696 stateUpdate: false versions:null)]9

[6/4/08 22:20:02:562 EDT] 00000084 PackageProces I Start of draft item removal,time=2008-06-04T22:20:02.109, count=0 [][6/4/08 22:20:02:562 EDT] 00000084 PackageProces I Start of draft item removal,time=2008-06-04T22:20:02.109, count=0 [][6/4/08 22:20:02:578 EDT] 00000084 PackageProces I Start of new published items,time=2008-06-04T22:20:02.109, count=0 [][6/4/08 22:20:02:578 EDT] 00000084 PackageProces I Start of update published items,time=2008-06-04T22:20:02.109, count=0 [][6/4/08 22:20:02:578 EDT] 00000084 PackageProces I Start of new draft items,time=2008-06-04T22:20:02.125, count=0 [][6/4/08 22:20:02:594 EDT] 00000084 PackageProces I Start of update draft items,time=2008-06-04T22:20:02.125, count=0 [][6/4/08 22:20:02:609 EDT] 00000084 PackageProces I Start of item removal,time=2008-06-04T22:20:02.125, count=0 [][6/4/08 22:20:02:609 EDT] 00000084 PackageProces I Start of deferred new published content links,time=2008-06-04T22:20:02.125, count=0 [][6/4/08 22:20:02:609 EDT] 00000084 PackageProces I Start of deferred new draft content links,time=2008-06-04T22:20:02.125, count=0 [][6/4/08 22:20:02:625 EDT] 00000084 PackageProces I Start of resolving references,time=2008-06-04T22:20:02.125, count=0 [][6/4/08 22:20:02:625 EDT] 00000084 PackageProces I Start of item renaming,time=2008-06-04T22:20:02.141, count=0 [][6/4/08 22:20:02:641 EDT] 00000084 PackageProces I Start of create versions,time=2008-06-04T22:20:02.141, count=0 [][6/4/08 22:20:02:641 EDT] 00000084 PackageProces I Start of update versions,time=2008-06-04T22:20:02.141, count=0 [][6/4/08 22:20:02:641 EDT] 00000084 PackageProces I Start of version removal,time=2008-06-04T22:20:02.141, count=0 [][6/4/08 22:20:02:656 EDT] 00000084 PackageProces I Start of item status update,time=2008-06-04T22:20:02.141, count=0 [][6/4/08 22:20:02:656 EDT] 00000084 PackageProces I ******************* End of subscription updatereport *******************which provides the subscription summary4. Sometimes instead of the 603 -" no more confirmations to send" the subscriber responds back to thesyndicator with a 200 message code where you will see the phrase "OK" being used. This phraseindicates that the subscriber is still processing the updates that the syndicator sent in earlier and doesnotwish to process any new updates. This will happen when say the syndicator sends in a large number ofupdates, the subscriber is processing each one of them ( and this is taking time) meanwhile theitemchangedtaskdelay which is set to 30 seconds causes the syndicator to contact the subscriber againfor the eventlog updates.5. Also while the subscriber is processing the updates sent by the syndicator the status of the subscriberwill showup as active while the syndicator will showup as idle for most part.6. To read the traces, start with the syndicator log to identify the syndication initiation point and co-relatethe corresponding message on the subscriber systemout.log. Once identified, use the systemout.log toverify where the error occurs ( it will be easier to scan through the systemout.log than the trace.log tofind this). Once identified then find the corresponding entry in the trace.log to see if additionalinformation is available.7. As mentioned the majority of the activity in the most common scenarios occurs on the subscriber side.Hence the subscriber logs typically are bigger than the syndicator ones. And the errors will mostprobably showup in the subscriber logs.8. Edit the WCMConfigService.properties file and change the deployment.enableReport setting to "true".This enables high level reporting of syndication to the SystemOut log on the subscriber server. Itprovides a summary of items that were processed, and which items failed syndication to help withtroubleshooting syndication issues.10

4. Eventlog Database and SyndicationFirst of all syndication has changed dramatically in version v6 compared to v5 and although some parts usethe same names (the item gatherers for example), they are completely different to what they were inprevious releases.Syndication begins at a library level. Each WCM library is associated with two item gatherers: the "AllItems" gatherer and the "Live Items" gatherer. These item gatherers are unique for every library and providethe means to track items by status in specific libraries. Unlike in previous releases, the item gatherers doNOT contain any information about the items, they are simply a label used to associate a syndicator objectwith the items it must syndicate.When an item is created or modified, the state of the item is updated in the event log database. It isimportant to understand that the event log stores the STATE of items, NOT the actions that were performedon them. To illustrate, let's look at the following example:If you create a new content document, an entry similar to the following will be added to the event log:Where:ITEMID: The ID of the content document.TYPENAME: The class type of the content document.ITEMTYPE: Whether the content document is published (P) or draft (D)STATE: A number representing the current state of the content document.DELETED: Whether the content document has been deleted (Y), purged (P) or moved (M).PARENTID: The ID of the content document's parent node (i.e. a sitearea id).TSTAMP: The timestamp of when the content document was last modified.ITEMGATHERERID: The ID of the appropriate item gatherer (All or Live) associated with the contentdocument's library.When the content document is modified, the entry in the event log is updated to reflect the updated state ofthe item, for example:Syndication uses the information in the event log to determine what items it must send to a subscriber.When you create a syndicator object you will select a library and a scope (All items or Live item). Thisprocess will associate the syndicator object with a particular ITEMGATHERERID. When syndicationoccurs, the syndicator will obtain all the items in the event log that match it's associated item gatherer andsend them to the subscriber to be processed. The full process will use the STATE, DELETED flag andother information in the event log to determine what action to perform on the subscriber.Basically the event log maintains a list of items in the system by state and associated with an item gathererid and that syndication uses the event log to determine what items it must send to a subscriber. When yourun the wcm-reset-event-log task you are actually clearing all the entries from the event log database andtelling WCM to rebuild the event log when the server restarts. This is possible because the event logdatabase stores the current state of all the items in the system and not the actions that were performed on theitems so no information is lost when the event log is reset, WCM will simply get every item in the systemand add it to the event log using its current state.11

Ideally the event log database should always be kept in sync with all the items in the system and never needto be reset. In practice however, sometimes an item is not added to the event log or an entry in the event logis not updated due to a defect or error in the system. This can cause the item to not be syndicated andresetting the event log will correct the problem by re-adding all items in the system in their current state.12

6. Most Common Issues (and how to address them)In this section we try to troubleshoot issues keeping the following in mind:• Questions to ask• Things to verify from the existing information• Additional information to request (if needed)• Actions to be taken13

6.1 Automatic syndication is not happeningSymptoms ReportedSyndication not even initiatingSyndicator/subscriber connect but there is no data being transferredSyndication starts up from the syndicator but no response from the subscriberSyndication updates donot show up on the subscriber at alla) On the syndicator machineCheck for 6.0.X/wcm/shared/app/config/wcmservices/WCMConfigService.properties fileCheck for 6.1.X/PortalServer/wcm/shared/app/config/wcmservices/WCMConfigService.properties fileon syndicator machine to make sure that inittask for syndication is set to trueconnect.moduleconfig.syndication.inittasks=trueb) On the syndicator machineCheck for 6.0.X\wcm\shared\app\config\wcmservices\WCMConfigService.properties fileCheck for 6.1.X\PortalServer\wcm\shared\app\config\wcmserviceson syndicator machine to make sure that subscriberOnly is NOT set to truedeployment.subscriberOnly=falsec) If a) and b) checkout ok then the issue can be one of the three:a) EventLog is emptyb) Persistence Tables need to be clearedc) EJBTimers are not firingCheck these using the following steps:1. Make sure the following message logged in the syndicator SystemOut.log at startup.SCHD0133I: Scheduler WPSTaskScheduler (wps/Scheduler) has acquired the lease and will run all taskson this application server.In a cluster environment, this message should log in either cluster member SystemOut.log. If this messagenot found in the SystemOut.log, then EJBTimer won't be able to fire the scheduled timers to sendsyndication update to subscriber. WAS L2 should help customer to configure scheduler for clusterenvironment.2. Make sure there are no SQL exceptions in the trace. For example:com.ibm.websphere.ce.cm.DuplicateKeyException. If SQL exception found in trace use theClear_WCM_Persistence_Service_table.jsp to clear the WCM_OBJECTS table from persistence database.This tool is available in the IBM Support Tools Portlet.3. Get the output of the following command and EJBTimers_nn.nn.log from the syndicator server:$WAS_PROFILE_HOME/bin directory - ./findEJBTimers.sh WebSphere_Portal -all -username -password %WAS_PROFILE_HOME%\bin directory - findEJBTimers.bat WebSphere_Portal -all -username -password Typically you should see something like this:[4/30/09 19:43:25:916 CEST] 0000000a EJBTimersComm A EJB : wcm, WCM_EJBs.jar, EJBScheduler[4/30/09 19:43:25:916 CEST] 0000000a EJBTimersComm A Info : com.ibm.workplace.wcm.util.scheduler.Schedulable|com.aptrix.syndication.business.TaskManager$CheckSessionsWork|com.aptrix.syndication.business.TaskManager$CheckSessionsWork|-5312fffa8c8dffc29c9092d19e8f8b8d9687d18c86919b969c9e8b969091d19d8a8c96919a8c8cd1ab9e8c94b29e919e989a8ddbbc979a9c94ac9a8c8c9690918ca8908d94fffffffffffffffefdffff878dffc79c9092d1969d92d188908d948f939e9c9ad1889c92d18a8b9693d18c9c979a9b8a939a8dd1be9d8c8b8d9e9c8bac9c979a9b8a939e9d939aaeaec3847c6da2cffdfff9a5fff493909198ad8a9191969198b5fff68b96929a8c8b9e928fb3fff699968d8c8bab96929a8bffefb3959e899ed08a8b9693d0bb9e8b9ac4b3ffef96918b9a8d899e93bb8a8d9e8b9690918bffefb3959e899ed0939e9198d0b3909198c4b3fffb919e929a8bffedb3959e899ed0939e9198d0ac8b8d969198c4b3fff6919e929a8c8f9e9c9a8eff81fffb878ffffffffedf08918d5c8c8dfff1959e899ed18a8b9693d1bb9e8b9a9714

957efeb4a68be6fcffff878f88f7fffffedf08918d5c878c8dfff1959e899ed1939e9198d1b3909198c4741b6f3370dc20fdfffeb5fffa899e938a9a878dffef959e899ed1939e9198d1b18a929d9a8d79536ae2f46b1f74fdffff878fffffffffffff8acf8bffc29c9092d19e8f8b8d9687d18c86919b969c9e8b969091d19d8a8c96919a8c8cd1ab9e8c94b29e919e989a8ddbbc979a9c94ac9a8c8c9690918ca8908d948eff81fff5[4/30/09 19:43:25:916 CEST] 0000000a EJBTimersComm A 1 EJB Timer tasks foundIf not found then try restarting Portal and run the command again.If still not present then run cancelEJBTimers and restart Portal.4. If there are many EJBTimer tasks scheduled, then run following command on the syndicator server$WAS_PROFILE_HOME/bin directory - ./cancelEJBTimers.sh WebSphere_Portal -all -username -password %WAS_PROFILE_HOME%\bin directory - cancelEJBTimers.bat WebSphere_Portal -all -username -password (P.S: For ejbtimer issues enable the folowing trace: com.ibm.workplace.wcm.util.scheduler.*=all)5. Make sure that the event log is populated with entries and is not empty. Run the displayEventLog toolfrom the IBM Support Tools portlet on the syndication server. Review the output of the jsp to ensure that itis not empty. If it is empty then make sure that you double check step2 on the subscriberOnly value set totrue. Possibly a restart of the portal server maybe needed.6. If the eventlog is empty and the subscriberonly is set to false and the restart doesnot fix the issue andeverything above checked out okay thena) Make sure additional traces are added on the syndicator server:com.ibm.workplace.wcm.services.eventlog.*=finest:com.ibm.workplace.wcm.services.transaction.steps.eventlog.*=finest(P.S: As mentioned earllier ensure that customer has set the traces in WAS and the number of historical files and file size of the trace files isadequate)b) Reset the event logc) Restart portald) Run the displayEventLog tool and check if the eventlog got populated now - if so then reattemptsyndication, if not then e)e) Collect the trace logs and the jsp output and provide it to IBM support15

6.2 Syndication updates complete but with errorsThese are some of the most typical errors seen in the logs:DuplicatePathExceptionRefer to this technote: http://www-01.ibm.com/support/docview.wss?rs=688&uid=swg21260741Solution is to do the following1. Remove the failing object(s) from the Subscriber instance.2. Then re-save the failing object on the syndicator and do an "Update", or you can simply do a"Full Rebuild" of the syndication pair.ReferentialIntegrity ErrorsThis would indicate that the references for an item being saved could not be found. Typically this wouldmean that the an item may be referencing another item in another library and most likely that library is notgetting syndicated over. Make sure to ask the customer to check if all the libraries that have itemsreferencing each other are being syndicated.DuplicateKeyExceptioncom.ibm.websphere.ce.cm.DuplicateKeyException: [IBM][CLI Driver][DB2/AIX64] SQL0803N One or more values in the INSERT statement, UPDATEstatement, or foreign key update caused by a DELETE statement are not valid because the primary key, unique constraint or unique index identified by "1"constrains table "JCR.EV_VERSION" from having duplicate rows for those columns.SQLSTATE=23505Resetting of the event log should clearup this error. Also use the clear persistence table jsp available in theIBM Support tools portlet.ItemNotFoundjavax.jcr.ItemNotFoundException: A node does not exist with UUID: 60b1f58044f3bee49956b9f420fa6210 atcom.ibm.icm.jcr.WorkspaceImpl.getNodeByUuid(WorkspaceImpl.java:1117)While retrieving the an item from the syndicator database the subscriber logs throw this error is it is unableto find the item in the JCRDB of the syndicator server. This could possibly indicate that the eventlog is notin sync with the JCRDB for the syndicator. Ask the customer to reset the event log and then try thesyndication again.Parent Not Found [[5/6/09 16:32:24:577 EDT] 00000041 PackageProces W Could not save item with id DepRef(id:16c567804590753c8bd68f0b6a6ea584 type:com.aptrix.pluto.site.SiteArea nonDraft:true draft:false purged:false parentId:6babef80458422afb958b9fb02923ee3 timeStamp:1227863963980stateUpdate: false versions:null moved: false) because it could not find its parent.Typically you will see this error as a symptom of the problem where the parent item or its parent could notbe saved. Use the parent UUID mentioned in the string above and check the logs to see the reason for theparent failing. If a top level site or a workflow fails to be saved it will cause the failure to save several itemsand the above error will showup in the logs.InvalidClassExceptionIWKSY1059X: detail: IWKPD1088X: Exception occurred while trying to retrieve document from item.IWKSY1060X:(Cause:java.io.InvalidClassException: com.aptrix.pluto.content.Content; local class incompatible: stream classdesc serialVersionUID =1595762135600321749, local class serialVersionUID = -1089925949120930392)at com.aptrix.deployment.subscriber.SerialisationTransformer.transformDocument(SerialisationTransformer.java:88)http://www-01.ibm.com/support/docview.wss?rs=1041&uid=swg21265346From the versioninfo.log verify the fixes installed on both the syndicator and subscriber matchIf the fix levels match and we still see the error then, check for loose class files on the syndicator orsubscriber system this will impact the fix levels.Location for these loose class files:V6.0.X: \wcm\shared\appV6.1.X: \wcm\prereq.wcm\wcm\shared\app16

6.3 Syndication performance is not good1. Increase the default syndication frequency of 30 seconds for servers that don’t require frequentsyndication. Take care not to space the intervals TOO far apart, 10 to 30 minutes is a good range.2. Don’t enable a syndicator on the production rendering server.Set the subscriber.only property to “true” inWCMConfigServices.properties . This stops the monitoring task from running on the IWWCM instance thattracks object changes for later syndication to another server.Whenever the subscriber.only setting ischanged, also reset the EventLog (\config\WPSConfig wcm-reset-event-log) beforerestarting the server3. Avoid having too many Subscribers (>5) linked to the one Syndicator.Create a tiered syndicationstructure, e.g. the first syndicator syndicates to two servers and those servers further syndicate onto the rest.This improves performance because you are decreasing the load on the syndicator server.4. Consider disabling ‘Versioning’ for all items on both authoring and rendering servers.If storing previous versions of items isn’t required, then disabling this feature can improve performance ofsaving and syndication. Refer to the following link in the infocenter that mentions how to achieve this:http://publib.boulder.ibm.com/infocenter/wpdoc/v6r0/topic/com.ibm.wp.zos.doc/wcm/wcm_config_author_options.htmlhttp://publib.boulder.ibm.com/infocenter/wcmdoc/v6r0/topic/com.ibm.lotus.wcm.doc/wcm/wcm_config_prop_authoring.htmlSet the Version control options in the WCMConfigservice.properties file:* versioningStrategy.AuthoringTemplate= never (or manual)* versioningStrategy.Component= never (or manual)* versioningStrategy.Content= never (or manual)* versioningStrategy.PresentationTemplate= never (or manual)* versioningStrategy.Site= never(or manual)* versioningStrategy.Taxonomy= never (or manual)* versioningStrategy.Workflow= never (or manual)* versioningStrategy.Default= never (or manual)17

6.4 Syndication changes donot show up on all cluster membersTypically a customer will report a scenario in which they are syndicating from their stand alone authoringserver to a single node in a two node horizontal cluster. The syndication completes successfully but thechanges are not reflected when accessing the cluster through the front end httpserver or when accessing thenon syndicator node directly through its internal http transport port.1. This is not a syndication issue but an issue with the WCM cache objects defined in WASdynacache. A simple test to verify this is to either restart the non syndicator node ( the changesshould reflect now) or use the clearcaches.jsp available here: https://w3.tap.ibm.com/w3ki08/display/wcm/Tools2. A technote that helps in such a scenario:http://www-01.ibm.com/support/docview.wss?rs=688&uid=swg213040203. This could be a WAS dynacache settings issue however if the customer is still reporting issuesafter following the technote, this needs to be investigated as a possible WCM cache issue.18

6.5 Syndication of a particular item(s)is not happeningSometimes in an already running syndication setup. Customer will report that when they editing and resavingan existing content itemon the syndicator, it doesnot showup on the subscriber side. Ask thecustomer to do the following:1. Use the IBM Support Tools Portlet to determine the UUID for the item that is not getting syndicated.2. Use the displayEventlog from the portlet and get a screen capture of the output3. Re-edit the item and re-save the item4. Use the displayEventlog tool and get a screen capture of the output5. Check between the screencaptures and see if the UUID got added to the eventlog or not - A lot of thetimes just resaving the item corrects the issue.6. If it has not been added then there might be an issue with eventlogQuestions to ask:6a. Does this happen for an item in a particular library ?6b. If so, then does this happen for all items in that library ?6c. If so, is this a migrated library ?To resolve this do one of the following:6.1) Make sure additional traces are added on the syndicator:com.ibm.workplace.wcm.services.eventlog.*=finest:com.ibm.workplace.wcm.services.transaction.steps.eventlog.*=finest6.2) Reset the event log6.3) Restart portal ( please note - restarting portal after restting event log may take sometimedoesnot mean that the portal is hung)6.4) Use the displayEventLog tool to check if the eventlog got populated now with the UUID ofthe item - if so then retry syndication, If not, then 6.5)6.5) Attempt the following steps for a new item with the above mentioned traces in place:1. Create a new item in the library being syndicated.2. Use the IBM Support Tools portlet to determine the UUID for this item3. Use the displayEventlog tool and get a screen capture of the output - does this item getadded to the eventlog ?6.6) Collect the trace logs, review the traces and the jsp output and consult with IBMP.S: In scenarios such as above always try the syndication with a new item to ensure its notcontent/library specific issue.7. If the item is added to the eventlog, then attempt the syndication with the MustGather traces enabled.Review the subscriber logs to see the cause of the failure. Typically the item would have failed to beprocessed by the subscriber due to the syndication errors already covered in the Section "CommonIssues->Syndication updates complete but with errors"19

6.6 Syndication completes but not all items showup on the subscriber side or Need to verify if allitems on the syndicator match with all items on the subscriber:1. In this scenario, please consult IBM Support for a LibraryExport.jsp and run it against the syndicatorand subscriber nodes. This will generate a CSV file each on the syndicator and subscriber servers.2. IBM support will compare the outputs of these and identify if certain items are missing.3. Use troubleshooting techniques discussed so far to determine why certain items didnot make it throughthe syndication. That is based on the difference identify the missing UUIDs. Review the logs and see theerrors associated with the syndication of these UUIDs. If no errors in the logs then see the eventlog todetermine if it got added to event log or not. And if not then enable traces to determine why.4. Keep in mind a large number of syndication issues can be identified by comparing the LibraryExportoutput from both syndicator and subscriber along with the displayEventLog output on the syndicatorside.20

7. Syndication and ClusteringA clustered environment typically consists of several nodes and a load balancer but can include morecomplex relationships between several clusters with multiple levels of load distribution as shown in figure 2below. It is therefore important to understand the topology of the environment before making any decisionsabout how to configure syndication.The term load balancer is often used loosely to mean any system which distributes load to multiple sources.To prevent confusion, we will use the term load balancer to mean a system which distributes load acrossseveral nodes in a single cluster and IP sprayer to mean a system which distributes load across severalclusters as shown in figure 2 above.Configuring syndication in a clustered environmentWhen configuring syndication it is best to keep the ultimate goal in mind, which is to update the JCRrepository of a subscribing environment with changes from the JCR repository of a syndicatingenvironment. This means that for complex setups like the one shown above where you have more than oneJCR repository to update in what is conceptually a single subscription environment, you need to treat eachcluster as a separate environment and create multiple syndication pairs. For this reason IP sprayers shouldnever be used in configuring syndication. In other words, the syndicator and/or subscriber URLs shouldnever point to a system that distributes load between multiple servers or clusters with different JCRrepositories.The remainder of this section will describe configuring syndication between the single authoring cluster andone cluster in the rendering environment. This process should be repeated to achieve syndication to theother rendering cluster.21

The syndicator and subscriber objects can be created by accessing the interface via the load balancer or anynode directly. Because all nodes in a cluster share the same repository, it doesn't matter which sends (for asyndicator) or receives and processes (for a subscriber) the syndication request. This means that thesyndication URLs can be configured to/from any single node in the cluster, however it is recommended thatit be configured to/from the load balancer to facilitate fail over should a node go down. The syndicationURLs can be set to any specific node or the load balancer irrespective of how you accessed the system tocreate the objects.Syndication scheduling in a clustered environmentAutomatic syndication it initiated by the scheduler service which runs on each node in a clusteredenvironment. Each scheduler competes for a lease which is only granted to one node in the cluster. Thenode that obtains the lease is then responsible for running all tasks in the system until the lease is releasedand granted to another node (see the "High availability" section at http://www.ibm.com/developerworks/websphere/techjournal/0404_johnson/0404_johnson.htmlfor more information).This means that any node in the cluster could initiate automatic syndication which might be confusing if youhave configured the syndication URLs to a specific node in the cluster. The important thing to realise hereis that the syndication URLs do not restrict which nodes are used to perform syndication but simply indicatehow each environment should contact the other.For example, the syndication pair may be configured between cluster 1, node 1 (C1N1) and cluster 2, node3 (C2N3) but the scheduler may initiate syndication on cluster 1, node 2 (C1N2). C1N2 will contact thesubscriber using the configured URL - C2N3 which will receive and process the initial packet. C2N3 willthen use the configured syndicator URL - C1N1 to request updated items. Because all syndicationoperations happen as individual HTTP request/responses, it is ok for the process to switch between nodes inthis fashion. Interestingly this is also why it is ok to configure syndication to a load balancer which maydistribute requests for updated items across different nodes in the cluster.22

8. Syndication Resources1. Syndication MustGather, the learn more and analyze data section provides several technoteshttp://www-01.ibm.com/support/docview.wss?rs=1041&uid=swg21265396http://www-01.ibm.com/support/docview.wss?rs=1041&uid=swg212653942. Syndication Whitepaper:http://www.ibm.com/developerworks/lotus/library/wwcm-syndication/3. Troubleshooting section for Syndication in the InfoCenter for V6.1http://publib.boulder.ibm.com/infocenter/wcmdoc/v6r0/topic/com.ibm.lotus.wcm.doc/wcm/wcm_syndication_troubleshooting.html4. IBM support Tools portlet for Lotus Web content Managementhttps://greenhouse.lotus.com/plugins/plugincatalog.nsf/assetDetails.xsp?action=editDocument&documentId=AE2BB2412F20AA318525772E006F701423