TIBCO MDM Customization Guide - TIBCO Product Documentation

Using FileWatcher| 149Check for Duplicate FilesFileWatcher does not load any file it has previously processed. It maintains ahistory of the last 100 files encountered, and checks each new file against thehistory list. If a duplicate file is detected, an email with the error message is sent tothe administrator, and the duplicate file is moved to the rejected directory.Concurrent File LoadingIn a clustered environment, if more than one FileWatcher attempts to process anincoming file, it would lead to concurrency issues and possible data corruption.This is addressed by the lock mechanism which establishes a semaphore bycreating a lock file for the DataSet and releasing the lock once the processing isdone.Before processing an incoming file, FileWatcher attempts to acquire a lock on thelock file specified as a part of DataSet parameters. To acquire a lock, it attempts tocreate a file with the DataSet name suffixed with name specified in the LockFileparameter (_). If the lock is acquired for thespecified DataSet, FileWatcher processes the incoming files for that DataSet anddeletes the lock file after the processing is done.If the LockFile for a DataSet already exists and lock can not be established, theFileWatcher skips the DataSet processing and works on any other DataSets thathave been configured.The lock file is created in the / path provided in the URIInfosection of the DataSet. It contains information about the instance which acquiredthe lock so that the abandoned lock can be cleaned up by the correct instance. Theinstance ID used in the lock file is given by the JVM argument NODE_ID.A LockFile can be specified for each DataSet. If no LockFile parameter is specifiedin the DataSet information, the global LockFile parameter is used to determinelock file extension.Import Data using FileWatcherData is imported into the catalog by defining data sources and mapping the datasources into a repository. New data is imported through the user interface (UI);you upload a data source, and then import it into the repository. FileWatcherexpedites importing data by polling for new files, and uploading themautomatically. These files contain typically master data, which can uploaded intoa data source and then imported into the repository.TIBCO MDM Customization Guide