Motion compensated frame rate conversion - White Paper - Snell

Innovation in the
Multi-Screen World
White Paper
Motion Compensated Frame Rate
Conversion
Paola Hobson, Snell Ltd
8 th February 2013
Introduction
Motion compensated frame rate conversion has proven to be
of great benefit to broadcasters, content owners and media
organizations around the world. The superior quality of motion
compensated frame rate conversion allows viewers everywhere to
enjoy pristine pictures which in turn enable broadcast and media
organizations to enjoy international sales of their content.
In this White Paper we describe some of the technologies used in
motion compensated frame rate conversion, and discuss the balance
of cost and quality which customers need to consider when choosing
a converter.
Frame rate conversion : motion estimation
As viewers and subscribers demand ever-higher quality in their SD,
HD, and 3D content, the quality of the source material is more
important than ever before. Frame rate and format conversions are
needed by any content broadcaster or distributor when the content
arrives at a different frame rate and format than is required for
transmission or distribution to end users. For example, European
HD 50Hz format programs need to be converted to 59.94Hz for
distribution in the US. TV content at 50 or 59 Hz needs to be
converted to 23.98 Hz for DVD and Blu-ray distribution.
Changing the frame rate of the content essentially means creating
additional frames where none existed at the input. Similarly,
changing the number of lines in the picture (e.g. as in a 720 to 1080
conversion) requires creating additional lines where none existed
at the input. Simple linear interpolation of new lines or frames from
those immediately preceding and following gives a very blurred
result when the objects in the scene move, as shown in Figure 1.
Other effects can also arise such as motion judder, where objects no
longer have smooth motion from frame to frame.
Figure 1 : simple linear standard conversion
© 2013 Snell

Innovation in the
Multi-Screen World
Input frame 1
Interpolated frames
Figure 2 : illustration of motion-compensated frame rate conversion.
Originally launched in 1994, Snell’s Alchemist was the world’s first
motion-compensated standards converter. In very simple terms, in
a motion compensated converter, the system calculates the motion
between frames in the content, and works out where to move
objects to when creating new frames in between, as shown in Figure
2.
In practice, the simple linear motion shown in Figure 2 is rare, and
motion is often much more complex. Typical video scenes contain
multiple objects, of different sizes moving at different speeds, which
could have very complex shapes. The objects may be passing across
detailed backgrounds, and may cross each other. Therefore accurate
calculation of motion is very challenging and different motion
vectors are required for different parts of the picture.
The most accurate motion estimation method developed so far is
called Phase Correlation (Ph.C). Ph.C is used in Alchemist Ph.C-HD
to deliver the highest-quality output with sharp detail, free from
artefacts. Figure 3 shows the results of frame rate conversion using
phase Correlation, yielding perfect output.
Phase Correlation provides a correlation surface from the
comparison of images. This enables the identification of motion on
a pixel-by-pixel basis for correct processing of each motion type.
Since phase correlation operates in the frequency rather than the
spatial domain, it is able to zero in on details while ignoring such
factors as noise and grain within the picture. In other words, the
system is highly tolerant of the noise variations and rapid changes in
luminance levels that are found in many types of content – resulting
in high-quality performance on fades, objects moving in and out of
shade, and light flashes.
Frame rate conversion: the difficult bits
Motion estimation is only a part of the processing needed when
frame rate converting video. Movies and TV programs are
composed of multiple sequences which are edited together, with
various effects such as cuts, fades etc, and often graphics, logos
and text appear over the content for short periods of time (e.g.
a scoreboard) then disappear. Scrolling captions are also very
common, not just at the start or end of a program, but also during a
program to provide instant updates e.g. a stock ticker at the foot of
a news program.
In such cases new content appears in the picture without any
history, and as we explained above, the simple example of Figure
2 is not very common in practice. Therefore motion compensated
frame rate converters also have to include processing which
creates perfect picture content in areas where new objects appear,
disappear or obscure each other.
Figure 3 : standards conversion using Phase Correlation
Another area where motion estimators have problems is where
the picture contains stationary repetitive patterns, which may be
interpreted as a moving object. In the example shown in Figure 4a,
a motion estimation algorithm examining the scene frame by frame
by looking at an area about the size of one of the bridge supports
(as could be the case for some block-based motion estimation
methods) would have difficulty knowing whether to interpret the
information as a static pattern or a moving object. The moving
bridge support object in Figure 4b could give the same apparent
motion analysis results as the bridge itself (Figure 4a) if it is moving
exactly its own width each frame.
© 2013 Snell

Innovation in the
Multi-Screen World
Frame 1 Frame 2 Frame 3
Figure 4a : stationary object containing repetitive patterns
Frame 1 Frame 2 Frame 3
Figure 4b : object moving at its own width per frame
Alchemist Ph.C-HD conversion
As mentioned above, Alchemist Ph.C-HD uses a highly accurate
method called Phase Correlation for calculation of motion vectors.
This is a frequency domain method, which requires spectral analysis
of the content followed by further processing. In Phase Correlation,
the distance and direction of the motion is measured accurately, but
the area of the screen in which it took place is not. Thus in practical
systems the phase correlation stage is followed by a pixel-by-pixel
assignment stage.
One reason why phase correlation is so accurate is that it uses
frequency information to calculate the motion in the scene. In this
way, small objects are not missed because they will generate high
frequency components in the analysis. A further advantage of
phase correlation is that it is sub-pixel accurate, meaning that object
movement in fractions of pixels can be calculated. This leads to
smoother motion profiles and higher quality results.
In addition to highly accurate motion estimation, Alchemist Ph.C-
HD integrates many additional steps to manage the difficult cases
we described above such as obscured/revealed areas, stationary
repetitive patterns, and highly complex motion. After calculation
of the initial motion vectors, Alchemist Ph.C-HD applies processing
to check for consistency of vectors, discarding those that appear to
be erroneous or not fitting the pattern of vectors in the surrounding
area. Through such techniques, Alchemist Ph.C-HD can avoid errors
often associated with motion compensated frame rate converters
such as blurring or introduction of the wrong content into an area of
the scene.
As may be expected, the additional algorithms and methods used in
Alchemist Ph.C-HD to get such accurate results require considerable
processing resources, and as a result, the Alchemist Ph.C-HD is
a relatively expensive solution. However, the investment cost in
Alchemist Ph.C-HD is rapidly repaid through its unmatched quality
of conversion and its handling of difficult program material.
KudosPro
The KudosPro range of motion compensated converters (MC500,
MC1000 and MC2000) offers cost-effective conversion in a 1U
frame. In order to reduce the amount and cost of components used,
KudosPro products use a different processing method to Alchemist
Ph.C-HD.
KudosPro MC500/MC1000MC2000 perform motion estimation
using a special transform analysis which is proprietary. The
transform is designed to detect the broad motion profile of a scene
and is followed by further processing stages to maximise accuracy
and reduce the impact of signal noise.
KudosPro processing was designed to achieve high quality
conversions while keeping the overall processing latency as low
as possible. Even with challenging content, the KudosPro MC500/
MC1000/MC2000 deliver optimal results whilst saving power and
resources through the use of sophisticated analysis techniques.
Conversion results
Both Alchemist Ph.C-HD and KudosPro MC500/1000/2000 offer
good quality conversion for a range of typical content. Many types
of content such as news, drama, current affairs and talk shows
contain relatively large objects with slow, regular motion, which does
not challenge a frame rate converter.
Where motion becomes faster, more complex, and where there are
multiple small moving objects in the scene, the requirements placed
on the frame rate converter become more demanding. Particularly
difficult are fast action sports and music clips where there are rapid
dance movements.
Figure 5 shows an example where Alchemist Ph.C-HD and KudosPro
MC1000 were used to frame rate convert some mainly static and
slow moving content. As can be seen in the figure, the content is
sufficiently non-challenging for there to be no visible difference
between the two outputs.
© 2013 Snell

Innovation in the
Multi-Screen World
(a)
(b)
Figure 5 : (a) Alchemist Ph.C-HD (b) KudosPro conversion of HD content 1080 50i to 1080 59i
However, in the case of more challenging content with fast
movement, as illustrated in Figure 6, we can see that the background
stationary objects look largely similar between the two converters.
However more complex processing capability of Alchemist Ph.C-
HD leads to better rendition of the fast moving objects such as the
motorbike and cyclist.
Figure 6 : (a) Alchemist Ph.C-HD (b) KudosPro conversion of HD content 1080 50i to 1080 59i
Conclusions
Both Alchemist Ph.C-HD and KudosPro motion compensated frame
rate converters offer state-of-the-art processing, with conversion
quality unmatched by any other converter in the market. Incredibly
accurate processing combined with years of refinement make
Alchemist Ph.C-HD the true “gold standard” in frame rate conversion,
where converted output is virtually indistinguishable from the input
for most content.
KudosPro uses simpler processing in order to offer a cost-effective
and compact alternative. Good quality results are obtained for
most types of content, but for the most demanding content, such as
sports and dance, Alchemist Ph.C-HD gives better results.
Further reading
For more detail on phase correlation and other motion
estimation methods, please see “The Engineer’s Guide to Motion
Compensation” by John Watkinson http://www.snellgroup.com/
documents/engineering-guides/emotion.pdf
© 2013 Snell