GStreamer for Video Processing Applications - Mil-OSS

GStreamer for VideoProcessingApplicationsMil-OSS WG3, August 30, 2011Stephen D. Burks, Joshua M. DoeUS Army RDECOM CERDEC NVESDoss@nvl.army.mil(Distribution A: Public Release Unlimited)

Outline• Overview of GStreamer• Image capture• Real-time image processing• Image quality analysis

What is GStreamer?• Graph based multimedia framework (~Microsoft DirectShow)• Open source – LGPL license• Can be used in proprietary products• Cross-platform• Linux, Windows, Mac OS X, Android, Symbian, …• Multiple languages• C library, many bindings (C++, Python, Java, Perl,…)http://www.gstreamer.net/

GStreamer:Pervasive Technology• Media players, video editors, VOIP and video calls, musiceditors/composers, camera capture, transcoding, …• Texas Instruments, Nokia, Intel, Motorola, Freescale,Tandberg (Cisco), …• GNOME, Totem, Rythmbox, Banshee, Songbird, PiTiVi, …

GStreamer Basics• Element• Source – file, test pattern generator, network• Processing element – encoder/decoder, transform,colorspace• Sink – file, display, network• Plug-in – collection of elements• Bin / pipeline – connected set of elements

GStreamer Overview• Capabilities: describes input/output data• Example: “video/x-raw-gray, bpp=16, depth=16,endianness=BIG_ENDIAN, width=640, height=480”• video/x-raw-rgb, video/x-raw-bayer, image/tiff, audio/x-raw-int, etc.• Properties• Communication• Buffers – sample of audio, frame of video• Events/Messages/Queries – communication between elements and/or application

GStreamer Strengths• Modularized processing• Can combine elements in arbitrary ways for rapid prototyping• Already developed data processing steps aid in rapid development• Single stream of data can be displayed, written to file, processedmultiple times, or any combination of them.• Quick testing of pipeline through command line• Automatic negotiation of capabilities• Handles time synchronization• Automatic plug-in discovery• Agnostic to data type• Arbitrary capabilities

Common GStreamer Elements• filesrc/filesink – stream data from/to a file• multifilesrc/multifilesink – stream data from/to multiple files• tee – duplicate a stream• queue – FIFO buffer• videotestsrc – test pattern generator• autovideosink - display• videoconvert – convert between colorspaces/formats

Making a GStreamer plug-in• Most written in C, but steep learning curve• Need to learn GLib, GObject (object oriented C)• Lots of boilerplate code• Rapid prototyping• Can be written in Python• NVESD engineers developed MATLAB wrapper

Visual Processing andExploitation Framework (VPEF)• Focus on the enterprise and cloud computing• VBench application for graphical creation ofpipelines

Image Capture:Overview• Want multi-spectral imagery that is spatially and temporallyregistered• Researchers also like to have multiple variants of a certain type ofsensor, such as a smaller and larger pixel pitch• Some data collections have as many as 10 sensors going into asingle capture computer• Need a means to display and save all this imagery without any loss

Image Capture:Vendor Card Support• Currently have support for the following capture cards– National Instruments IMAQ– Euresys– Blackmagick DeckLink• We are adding support for the following capture cards– Active Silicon– Coreco– BitFlow– Matrox– Foresight Imaging

Image Capture:Metadata and Ground Truth• In order to simplify data collections, it is beneficial to knowall of the following:– Target, location, range, aspect– Weather, atmospheric transmission– Time of collection, sensor mode, sensor parameters• There is no NVESD standard for this information, so weare looking into MISB/SMPTE (MXF)• This information will be embedded with the video

Image Capture:Cost Savings of GStreamer Approach• Software– Capture software from two major vendors cost 5K and 10K per license– Captured data usually has to be re-processed once collected• Hardware– Since GStreamer will support multiple vendors, users can choose lessexpensive options– Example: Need 4 RS-170 signals captured simultaneously. 4 NI-1410cards cost approximately 4K, while one Euresys Picolo card costs 0.6K.They will give identical imagery in GStreamer.• Personnel– Less time re-processing imagery– Fewer individuals needed manning computers– Less need for data-loggers since that data can be saved with imagery

Real-Time Image Processing• Most data is collected at one site, processed at another site, and thenanalyzed using additional software• It would be in our best interests to combine all of these steps so long as:• Data is not lost due to processing or analysis• The interface is simple• It is simple to add and modify processing or analysis steps

Real-Time Image Processing:Image FusionWaveband AFusionProcessWaveband B

Real Time Image Processing:Sample Graphical Pipelines• Image capture pipeline• Timing negotiated by GStreamer• This pipeline has 2 video inputs, 4 processingelements, 3 video outputs, and 3 file outputs

Real-Time Image Processing:Image Registration• Affine transformations (shift, scale, rotate)• Once we find algorithms for more advancedimage registration, we will add them as elements

Real-Time Image Processing:Cost Savings of GStreamer Approach• Rapid Prototyping– Can add, change, and remove GStreamer elements with ease– Image quality can be strongly affected by some image processingelements, while others will be benign to image quality– Faster time from prototype to engineering test unit• Re-Use of Elements– Most image processing tasks are now standardized, so there is noneed to re-code them multiple times.– It is the relative weights of these elements that make them unique for aparticular program.

Image Quality Analysis:Introduction• To complete the task of determining what is the best image process,one needs to “score” imagery based upon certain metrics.winner• The Image quality tasks could be added as discrete elements into the pipeline forreal time scoring of imagery• If a certain image quality metric is determined to correlate well to an observer task(like navigation or identification), then a user can make image processing changesto raise that score for a particular task.

Image Quality Analysis:Objective Measurements• Want to have ability to do “on the fly” measurements ofobjective image quality measurements– Resolution (MTF)– Noise (3D Noise)– Responsivity (SITF)• These measurements can determine if your sensor isfocused properly and if its electronics are workingproperlyMTFCyc/mrad

Integration with DoD OTD• Plan to put all appropriate code on Software.Forge.mil• Migration of some code to public GStreamer repositorypending document reviews– Bug fixes/minor enhancements to existing plug-ins– Commercial hardware support– Simple elements (non-proprietary technology)• Interested in collaboration with other militaryorganizations focused on image processing

Demonstration