Real-time feature extraction from video stream data for stream ...

ai.cs.uni.dortmund.de

Real-time feature extraction from video stream data for stream ...

6. Feature Extraction

6.3. Image Feature Extraction with the streams framework

RapidMiner does not provide the user with any stream functionality. But given that

video data should be processed in real-time, we need to extract features in a streaming

manner. This can be achieved by using the streams framework (see section 3.5.3). The

following section gives an overview of the processors, which I have implemented for

processing video data and extracting features. A full list of all implemented processors,

including more details about parameters and the API, can be found in the appendix.

6.3.1. Representing images

By using stream.io.MJpegImageStream, I can read in decoded video data and create

a stream of data items. Each of these data items corresponds to one frame of the video

stream. It contains the frame itself as a serialized image plus some additional information.

As the image type java.awt.image.BufferedImage, which is usually used in

JAVA to store images, is not serializable, it can of course not be used. Hence the images

included in the data items are of the self-implemented type stream.image.ImageRGB.

An ImageRGB image can be obtained from a BufferedImage by using the ImageRGB

(BufferedImage img) constructor. On the other hand an ImageRGB image can be

converted to a BufferedImage using the provided BufferedImage createImage()

function.

In RapidMiner image types vary depending on the image representation. Hence IMMI

operator can either process color images, monochrome images or black and white images.

In contrast to this concept, I decided to represent all types of images by the ImageRGB

class. Monochrome images can be stored by calculating the grayscale value for each pixel

and setting the value of all three RGB channels of the image to this grayscale value. Black

and white images get stored by setting each pixel either to black or white. The advantage

of this approach is that every image processor can handle every image and the processors

can be combined in any order. When using RapidMiner the combination of operators is

sometimes difficult, due to the fact that images of different types have to be converted

into each other to ensure the outcome of one operator matches the income of the next

one.

6.3.2. The AbstractImageProcessor

Data items, containing ImageRGB images, can be processed by special processors, extending

the stream.image.AbstractImageProcessor processor. The advantage

of using the AbstractImageProcessor instead of the AbstractProcessor is that

the AbstractImageProcessor allows an easy access to the image, included in the

data item, without searching for the image first. Nevertheless it is still possible to access

the full data item. This is necessary in order to store new features or access further

information, that is included in the data item. Figure 6.10 shows the source code of the

AbstractImageProcessor.

68

More magazines by this user
Similar magazines