24757 MXF White Paper v2

This paper describes the implementation of MXF-based operations (Material eXchangefile Format) in the TV studio environment. The first proposal for MXF was presented ata technical meeting of the Pro-MPEG Forum in Beaverton, Oregon, in July 1999. Thisfirst proposal consisted of simply a wrapper of the AV streams for program exchange inan IT environment. After this submission, the specification of MXF was expanded tocarry rich meta-data. After completion of its technical work in the Pro-MPEG Forum,MXF was submitted to SMPTE for further standardization. MXF is expected to becomeone of the most important methods for carriage of meta-data and AV contents for thetelevision industry. This paper further specifies the characteristics and advantages ofMXF for multimedia operations in an IT environment.At present the broadcast and production industries face significant changes due to theintroduction of IT technologies. The use of IT technology has the potential to createefficiencies and improvement of data workflow in the television productionenvironment, with the addition of significant cost reductions. It is believed that the firstwave of IT technology will be realized by the use of MXF and rich meta-data to createworkflow innovation. This paper discusses the harmonization of MXF file transfer andmeta-data applications.The initial work for the creation of a unified file exchange scheme, MXF, wasestablished by the Pro-MPEG Forum, but currently the AAF (Advanced Authoring Format)association and the EBU (European Broadcasting Union) have joined not only thestandardization activities but also the development of toolkits. This document describesthe concept of an MXF development toolkit and the roadmap for future MXF products.MXF represents a key technological development in the migration to advancedproduction systems that make use of extensive meta-data. This paper specifies howMXF/AAF and meta-data work together in an IT-based production system, and furtherclarifies its advantages for end-users.

Program materials are usually stored in the form of packagedmedia. A/V tapes are used mostly for this purpose. Programmaterials are then ingested and transferred to servers via thedigital interfaces, such as, SDTI-CP, i.LINK®‚*. This type ofoperation is known as “legacy style” and is illustrated in theupper part of Fig. 1. In this case, a data transfer is carried outon a peer-to-peer connection. In other words, dedicateddevices store the program materials and the exchangeoperation is performed on a standalone basis.Once the program materials are ingested onto the fileservers, these materials can be handled as files. The files aretransferred and shared via the network connecting the fileservers. In this case the key technologies are theinterchangeable file format and the IP network. Thecompatibility of data files is very important to guaranteeinteroperability among the devices that constitute theproduction system. The first objective of the MXF format isfile interchange via a network connection. This operation isspecified as “network style” in the lower part of Fig. 1.The networking operation provides several benefits to users.For example, operators can share all program contents thatare ingested as files. This means each production operatorcan collaborate with others flexibly. In the near future,therefore, the “legacy standalone” work style will migrate tothe network style by the gradual conversion of physicalmedia packages to ingested multimedia data files.It should be noted that in the case of package media,exchange interoperability is guaranteed by strict adherenceto a given tape-based standard - because the A/V signalformat is tightly related to the physical layer of the media.However, in the case of ingested data files, the format islogical and separated from the physical specification of themedia. Currently there are some open and proprietarymethods for file format specification, such as AVI, OMFI,Quicktime, DPX, GXF, etc.Fig. 1 Program Material Ingestion and File interchangeamong serversIn today’s television operations, complete programs aremostly exchanged by package media, because it can simplyguarantee the interoperability between equipment of thesame standard. However, as the TV industry migrates toend-to-end applications of IT technologies, file exchangeswill replace package media. Therefore, the urgent need fora flexible and open multimedia file format satisfying therequirements of end-users and standards organizations isneeded. To achieve the latter, MXF has been submitted toSMPTE for its consideration.The basic objective of the MXF file format is theexchange of (intermediate or final) program material inIT-based environments.*i.LINK is a trademark of Sony used only to designate that a product contains anIEEE 1394 connector. All products with an i.LINK connector may not communicatewith each other. Please refer to the documentation that comes with any devicehaving an i.LINK connector for information on compatibility, operating conditionsand proper connection. For information on any Sony device having an i.LINKconnector contact Sony at 1-800-686-7669.

MXF has been defined as a flexible wrapper for Video,Audio streams and meta-data. Fig. 2 depicts the basic filestructure of MXF. MXF can contain AV streams in the FileBody. Rich meta-data can be included in the header aswell as in the body of the file. The initial standardizationprocess focussed on the wrapping of Type D10 (MPEG-ES,SDTI-CP based) and DV-DIF streams (common formatfor DVCAM and DVC-Pro VTRs). The MXF file formatkeeps a neutral position towards the type of compressionscheme (DV, MPEG, etc.) and source signal format(525/625-line, SDTV/HDTV, etc.) in use for the programmaterial. Furthermore, the MXF body is expandable forvarious other types of AV stream formats. Currentlyuncompressed source body, MPEG PS/TS/PES body andType D11 stream (HDCAM®) body are under discussion inthe Pro-MPEG Forum.The SDTI-CP (Type D10 VTR compatible MPEG-ES), DV(DVCAM and DVC-Pro) and HDCAM (Type D11 VTR)streams will use an intermediate container format, calledMXF Generic Container, before final encapsulation by theMXF wrapper. This GC (Generic Container) is a commonpacket format which can include lots of useful meta-data.Existing stream formats don’t have extensible meta-dataspace so that this common meta-data carriage willsupport interoperability functions for various compressionschemes currently only implemented in stream form.The details of GC will be described in Meta-data FormatMXF section.The concept of compression independence provides animportant advantage for users of MXF. Even if a receiverunit cannot decode the body format, the decoder can atleast recognize the meta-data in the MXF file header forcontent identification. This means the search for contentmaterial using meta-data is available without decodingthe file body.MXF removes the boundaries between compressionschemes and the source signal format.Fig. 2 Compression independent format

Since the initial activities for the development of MXF,the members of the Pro-MPEG Forum have expandedthe capabilities of the format, with special emphasis oninteroperability with the Advanced Authoring Format(AAF). Pro-MPEG had a joint meeting with membersof the AAF association in May 2000. At this meeting,Pro-MPEG members decided to make MXF a definedsubset of AAF. At present, members of the AAFassociation and the EBU have joined the team of MXFdevelopers, which include many broadcast, manufacturerand academic organizations. Pro-MPEG, AAF and theEBU share the opinion that the technical advantagesbrought by MXF will lead to its adoption asthe common interchange file format for IT-basedproduction applications.Since these three organizations do not have as part of theircharter the creation of industrial standards, they decidedto submit their developmental work on MXF to SMPTE forfinal standardization. The first proposed draft standard toSMPTE was submitted in March 2001. The relationshipamong these activities is depicted in Fig. 3 below.The four organizations in Fig. 3 have conducted goodcollaborative technical work and are supportive of thestandardization activities of MXF. Therefore MXFpromises to become the next generation standard forprogram exchange with the support of these keystandardization groups.One of the important applications of MXF is its use inarchive systems. Users are very sensitive to the selectionof the file format for the archive system, because theycannot easily change the file format after starting thearchiving process. It would be extremely difficult andcostly to convert a large database of program materialat a later time. With this consideration in mind, MXFprovides the confidence needed in a “future-proof” fileformat, for users to commit to archive their materials.In particular, MXF is an advanced, open standard, andmany manufacturers have pledged their support to thisstandard. In the near future, even if users want to changetheir archiving systems, they will easily find alternativesdue to the interoperability of the MXF file format.Fig. 3 Relationship of the organizations regarding MXF development and standardization

As said earlier, Pro-MPEG forum and AAF associationmembers agreed to develop MXF as a subset of AAF.The AAF file format has been developed for highperformance authoring applications, including non-linearediting and multi-layer compositional work. Thisrelationship is shown in Fig. 4.AAF allows for a very flexible description of its contentsand specifies all edit processes during a non-destructiveedit operation. This description can include all EDL-like(Edit Decision List) information, special effects and variouselements of rich meta-data. While AAF offers a highlevel of versatility and performance for authoringwork, it is, however, too flexible and rich for the purposeof file interchange. If AAF files are exchanged, numerousconstraints are required for file interoperability. Therefore,from an AAF viewpoint, a simpler version of AAF(i.e. with limited functionalities and specific exchangecharacteristics) is required for file interchange. MXF issuitable for this purpose.On the other hand, MXF is a new format in the industry,requiring a relation to an existing format. AAF is the onlyavailable file format with capabilities to handle richmeta-data. Therefore, MXF has been developed as a subsetof the AAF format - just like a light version of AAF.MXF allows the description of an editing operation for acut edit only, but it cannot specify complex special effects.In MXF the sequence of body contents is placed in thepresentation order. This comes from the requiredfunctionality of streaming of the file contents oversynchronous stream interconnections. During thestandardization process, the EBU requested the inclusionof specific meta-data for use by broadcasters, which isknown as “Geneva Scheme meta-data.” While thismeta-data is very useful for broadcast operations, thecurrent specification of AAF does not include it. However,since AAF is a very flexible and expandable file format,it is expected that the Geneva Scheme for meta-datawill be adopted by AAF in future revisions of the format.Fig. 4 Relationship between AAF and MXF

The Geneva Scheme meta-data is specified in part 4of the MXF documents. The purpose of this scheme formeta-data is to add content descriptions scene by scene.This document is simply a collection of various types ofexisting meta-data. For example, the Location information,Language information, Image formats, Scriptingdescriptions, Contracts and Rights, Sub-title descriptions,Owner information, etc. This document will become adynamic standard, which means it will be expanded infunctionality by future revisions of its meta-data set.The Geneva Scheme meta-data is the first step for theglobal exchange of meta-data for contents description.The Geneva Scheme meta-data is therefore a keyadvantage for the use of MXF file format.Both AAF and MXF are file formats, but theirfunctionalities do not conflict with each other.AAF is a very flexible format that is suitable forcomplex edit operations; it is, however, verydifficult to exchange AAF files between differentmanufacturers’ devices. Therefore, an acceptablestandard file format is necessary for the exchangeand archive of program materials. This concept isspecified in Fig. 5. MXF is a file format specificallydesigned for simple exchange and archive of final,complete, material packages.Fig. 5 The relationship of AAF and MXF format

Different types of file formats are used in today’sproduction environments. Each processing island adoptsa different format based on different operational policiesand technical requirements. There is, however, no standardfor file interchange among these processing islands.In order to implement file exchange, a neutral andadvanced format is necessary. This neutral characteristicdefines an intermediate common file format, which can beconverted to and from all the existing file formats with aminimum of computational efforts.This is one of the strengths of MXF and is shown inFig. 6. MXF can carry the equivalent informationcontained within an OMF, GXF or any other type offile format. At the same time, MXF is a subset ofAAF, for optimum interoperability with this powerfulauthoring file format. Therefore, the choice of MXF asan inter-island exchange format is a good choice, forit not only resolves today’s interoperability problemsbut also leads to the implementation of highly efficient,IT-based, production environments.Fig. 6 Inter-island format for various production formats

One of the typical models of IT-based production systemis shown in Fig. 7 below. All work servers are connectedvia an IT-based network (i.e. IP network). In addition, theingested materials are stored on near-line archives asMXF files, as well as shared with Production andOn-Air systems. These near-line MXF files will also bestored in long-term archive systems. The ingestedMXF files can then be transferred to the branches(affiliates) or other studios for re-purposing applications.In order to design this type of system, the key technologiesof MXF/AAF and meta-data are required as part of itsinfrastructure. The meta-data element UMID (UniqueMaterial IDentifier) is also essential to maximize therelationship (linkage) between the MXF/AAF file andmore extensive external meta-data. UMID specifies theidentification of each program material contained in thefiles, on a frame-by-frame or, scene-by-scene basis, or anyother form. UMID is just a pointer to link the packagedprogram segments to any associated external meta-data.Of course, MXF can carry rich meta-data, however, usershave valuable legacy meta-data in their own formats. ThisUMID linkage system supports the use of such externalmeta-data in conjunction with the ingested/archivedprogram materials.This meta-data friendliness is key for the migration toa future IT-based system.Fig. 7 Typical Model for IT based Platform

MXF can contain meta-data in the header and the bodyportions of the file. Header meta-data consist of EDL-likeinformation, Content identification meta-data, Scene/Shotmeta-data, Descriptive meta-data (Geneva schememeta-data), etc. These descriptors represent sideinformation for the whole of the file, shots and scenes.On the other hand, MXF adopt a GC (Generic Container)as the intermediate container for the body. All AVstreams and meta-data are encapsulated into this GC.The GC consists of the following items: System Item,Video Item, Audio Item and Auxiliary Item. The essentialitem is the System Item. The System Item containsspecific parameters for the processing of AV streamsand meta-data. The basic packing of GC is on a frameby-framebasis. The meta-data that requires frameaccuracy (e.g. time code, UMID, etc.) should be storedin the System Item. Also, if a larger space for meta-datais required, Auxiliary Items are available to carry theadditional meta-data.Non-linear editors are required to edit both types ofmeta-data (header & body). The general idea for the editoperation is to utilize the intermediate format as shownin Fig. 8. XML is a human-readable language, which isuseful to record the header meta-data description. Hence,non-linear editors will use the XML file as the firstoutput of the edit process description. On the other hand,the body streams are stored in Audio/Video servers whereframe accurate meta-data is multiplexed/de-multiplexedvia SDTI-CP and/or, i.LINK interfaces. Those XML filesand AV streams are used to generate an MXF file.Since MXF is just a file interchange format, manufacturersdon’t have to store the contents of an MXF file in its nativefile structure. MXF import/export operations are, therefore,very important to enable interoperation between differentmanufacturers’ machines.Fig. 8 Meta-data format in MXF

The expected meta-data operations are shown in Fig. 9.The acquisition process is the first step for the generationof meta-data in the production chain. For example,camcorders will record the conventional SMPTE12Mtimecode along with UMID meta-data. UMID meta-data isattached to every shot for the purpose of materialidentification. Those shot materials (AV streams and metadata)are ingested onto servers in the form of an MXF file.The program shots are stored as File Packages in the MXFfile. File packages are just shot materials, which havedifferent timelines. Once these MXF files are stored intoservers, non-linear editors can start editing the filepackages. The file packages are mapped onto the singletimeline that defines the Material package in MXF. Non-Linear editors will add some meta-data for the descriptionof scenes and generate EDL-like data. These are includedin the header meta-data of MXF files. This edit process willbe repeated to generate the final complete package, asspecified in Fig. 9.The relationship between File package and Material packageis depicted in Fig. 10. All File packages are generatedduring the acquisition process as program materials.Some meta-data (for example, shot meta-data) havealready been attached to each File package.Fig. 9 Meta-data operation in MXFNon-linear editors will generate the single timeline for thereference of the edit process. This is defined as Materialpackages. File packages are cut and mapped onto thisreference timeline that defines the Material package.Non-linear editors can define some scenes on theirtimeline independently from the boundaries of shotsin File packages. Non-linear editors can also attachmeta-data to each scene. Finally the mapping decisionsare converted to EDL-like data and stored in the MXFheader meta-data. This is, in broad terms, the editingprocess for MXF and associated meta-data.Fig. 10 Meta-data operation in MXF

According to our model for an IT-based system, the editedfiles are stored into the near-line and deep archivemodules. MXF files basically contain UMID information;therefore UMID data can be used to link the MXF essencesand the external meta-data. The basic idea is shown inFig. 11, which describes the linkage (synchronization)between the external meta-data database system andessences in an AV server. MXF header meta-data isduplicated and stored in the external MXF database. Forexample, a user can then search the AV contents fromthis external database. The found data contains the searchkeys (UMIDs), so that the corresponding program entitycan be retrieved from the AV servers (archive systems).This is just an example, but this external meta-datadatabase can also include user specific data such asscripts, text, subtitles, closed captions, XML files, etc.Fig. 11 UMID Linkage between MXF file and External meta-data

Fig. 12 shows a comparison between the currentproduction model and a future, IT-based model. It canbe seen that the basic difference is the workflow of theproduction process.The workflow of the current production system follows asequential process, because all operations are pipelineoriented. In addition, all production processes areconnected and share materials via the archive system.However, in the case of IT-based production systems, alldata are handled by means of file-based processes andall files are shared among the horizontal productionoperators. These production operators can process theircreative work concurrently without knowledge of thestorage location of their materials. In essence, all fileoperations are managed with shared files, providing aninnovative workflow improvement for end users and highefficiency in the production process.Fig. 12 Current and Future production modelIT-based production will improve the following areas in thecurrent workflow:• Concurrent production work with file sharingCreators can process their work independently withminimum waiting time. In addition the created files can beshared with worldwide production locations creatingopportunities for effective collaboration work.• Re-use and re-sale of archived contentsIT-based production will generate archived files on a dailybasis. Meta-data will catalogue this material for re-use andre-sale in a convenient manner. MXF will be used as the fileinterchange standard for this purpose.• Management of large amount of AV assetswith meta-dataUMID information will be stored inside MXF files. Externalmeta-data databases will be associated with AV materials byusing UMID data. This mechanism will improve the use ofasset management systems.• Reliable contents transferFile transfer technology via IP networks will improve thereliability of content transfer. Error-free file transfer willimprove the ingestion process of incoming material.

The MXF format is defined as a wrapping format forexisting stream-based material via digital interfaces, e.g.SDTI, i.LINK, etc. Current digital stream formats (TypeD10/D11 stream, DV DIF, etc.) can be easily converted toMXF files. The migration process from stream-basedinterfaces to network connections will occur in a gradualmanner, balancing the preservation of legacy investmentsand the acquisition of new IT-based equipment.maintaining material interoperability. The maturedinteroperation of SDI/SDTI/MXF data formats has beenshown to work well, with numerous benefits for the entireproduction process. Therefore, it is the hope of many, thatthese standards will be supported by many leadingmanufacturers of professional broadcast equipment.The Pro-MPEG forum has made very effectivedemonstrations at NAB-2000/2001 to prove theharmonization of the SDTI and MXF worlds. This is shownin Fig. 13. SDI/SDTI signals and MXF files can be exchangedbetween different manufacturers’ equipment whileFig. 13 Interoperability of MXF and SDTI equipment = Migration to the Future

As mentioned above, MXF is defined as a subset of AAF.The AAF association has developed an SDK (SoftwareDevelopment Kit) for developing AAF systems. Therefore,with some constraints added to the AAF SDK, it willresult in an SDK for MXF. The notable differencebetween AAF and MXF is the binary notation employedin their definitions. In the case of AAF, a Microsoft®Structured Storage notation is recommended. But in thecase of MXF, a KLV (Key-Length-Value, defined by SMPTE336M) protocol is used for binary notation. Therefore,a translation of binary format is required. Fortunately,the AAF SDK has a plug-in mechanism to implementthis capability.Currently Sony is developing the plug-in for MXF supportin collaborative work with the AAF association.Fig. 14 MXF support by AAF SDK

The EBU PITV group has started the development of MXFreference software. Their purpose is to accelerate theproduct level implementation by distributing the sourcecode of the reference MXF file generators and MXF samplefiles. Their approach is very flexible, because the metadatadescription is specified by means of an XMLdictionary. They assume the use of future extensions tosupport evolving sets of meta-data.The rough structure of the software module in the AAFSDK is shown in Fig.14. The AAF SDK provides theapplication interfaces required by the upper layers ofsoftware by means of Windows® API. The AAF SDK canoutput the file in several formats, such as MicrosoftStructured Storage, XML, etc. In order to support MXF fileimport/export functions in the AAF SDK, a KLV Binaryinterpreter is necessary to import/export the MXF filedirectly to/from AAF modules. In addition, the codecplug-in for MXF bodies is also necessary.Basic AAF capability has already been implemented in thereleased of AAF SDK Ver.1.0. Sony is planning toimplement the additional modules shown above to supportMXF import/export functions. These additional moduleswill be provided as a plug-in for the AAF SDK.

MXF/AAF and meta-data are key technologies for theimplementation of IT-based production applications. Thesetechnologies will provide significant levels of improvementin data workflow for all acquisition, production, andasset management operations.The first step towards the practical implementation of anIT-based production will start by the multi-manufactureradoption of the MXF file format. The introduction ofSony’s e-VTR at NAB’02 will help the migration to IT-basedproduction by bridging linear, digital VTRs to the ITnetworking world. Valuable assets stored in 1 /2” tape formwill then be converted, with minimum efforts, to MXF files,for communication over IT networks.1. SMPTE 336M Data Encoding Protocol Using KLV2. SMPTE 330M Unique Material Identifier (UMID)3. SMPTE 298M Universal Labels for UniqueIdentification of Digital Data4. SMPTE 326M SDTI-CP Format5. SMPTE 331M SDTI-CP Element &Meta-data Definitions6. SMPTE RP204 SDTI-CP MPEG-2Decoder Template7. ISO/IEC 13818-2: Amendment 2: (MPEG-2,4:2:2P@ML).8. SMPTE 356M-2001, D10 Type Stream Specifications:MPEG-2 4:2:2P@ML for525/60 and 625/509. SMPTE 322M Format for Transmission of DVCompressed Video Audio andData over a Serial DataTransport Interface10. SMPTE 314M Data Structure for DV-basedAudio, Data and CompressedVideo, 25 Mb/s and 50 Mb/s11. SMPTE xxxM MXF format document12. SMPTE yyy.1M MXF Operational Pattern 113. SMPTE yyy.2M MXF Operational Pattern 214. SMPTE abc MXF Descriptive Meta-data Set 115. SMPTE zzz.mM DV Essence Container16. SMPTE zzz.nM CP Essence Container17. SMPTE RPxxx MPEG-2 InteroperabilityRanges (tbc)

Sony Electronics Inc.1 Sony DrivePark Ridge, NJ 07656www.sony.com/professional© 2002 Sony Electronics Inc. Reproduction in whole or in part without written permission is prohibited. All rights reserved.Sony, DVCAM, HDCAM and i.Link are trademarks of Sony. All other trademarks are the property of their respective owners.BC-MXFWP