Download full issue (PDF 2.1MB) - Council of European ...

Information Technologies for Visually Impaired People
3 Sketch Understanding
In this section we describe the tool for understanding
sketched drawings, its applications and the pattern recognition
techniques used in it.
3.1Purpose and Application Framework
Usually, visually impaired children use a plastic sheet
to make drawings so that the drawings can be seen but also
perceived by touching the relief. Also, computers can "see"
the drawings input by touching devices like a digital tablet
or a tablet PC. Our idea was to join both drawing modes by
adding a plastic sheet on the screen of a digital tablet. Thus,
while the strokes made by the user are digitally acquired by
the electronic device as a sequence of 2D coordinates, they
are also marked in relief in the sheet so can be touched by
the user.
Our system allows teachers to define new forms as exercises
to practice with the sketching input possibilities.
These forms can be designed to interactively learn geometric
shapes, graphical symbols and diagrams, letters, or any
other symbol consisting of strokes. In addition, our system
can be used to include diagrams in documents after cleaning
up. Forms have different zones with associated functions.
For example a zone can have a drawing and the associated
function of aligning the input strokes with the lines
of it. The aim would be to measure the ability of the user to
follow (redraw) the lines of a given drawing. In addition
this activity zone has two audio messages depending whether
the action is right or not.
Once the form has been designed, it can be printed with
a special printer that makes the reliefs of the Braille dots
and associates textures to colours. The form is printed on a
plastic sheet that is placed on the digitizing tablet during
the activity. Thus the child can "read" the instructions by
touching the relief and interact with the system by touching
the cells and getting audio feedback, or drawing with the
digital pen on the tablet by following the drawing, or copying
it into another cell, etc.
The double system, relief plastic sheet and digital tablet,
allows the visually impaired user first to touch the figure,
and then to draw it with the digital pen. Digital strokes
are captured from the computer and interpreted and the computer
gives audio feedback accordingly. Thus, for example
the system compares the input figure with a database of
models like square, circles, animal silhouettes, capital letters,
etc.
The teacher, when designing the forms, can record audio
responses or visual ones like emoticons associated to
activity actions. Visual outputs allow the system to be used
by non-visually impaired or low vision users. An outline of
the system is shown in Figure 7. An interesting survey on
similar interfaces can be found in [10].
3. 2 A Framework Based on a Sketching Interface
The technological basis of our system is the recognition
of graphical symbols. It can be defined as the identification
and interpretation of visual shapes depending on the do-
main. It means that the same symbol can have different
meanings depending on the context in which it appears.
According to the input mode, we can classify the recognition
of graphical symbols in two categories, namely offline
or static recognition, and on-line or dynamic recognition.
The former uses a scanned document and only works
with the spatial information of the image strokes. The latter
interactively obtains the document, using an input device
as a digitizing tablet or Tablet PC, so it additionally has
dynamic information of the temporal sequence of strokes,
speed, pressure, etc. Our work is based on on-line recognition
of hand drawn graphical symbols by a digitizing tablet.
Such type of man-machine interfaces are called sketching
(or calligraphic) interfaces.
Related works can be read in [11] [12]. On-line recognition
consists of three steps: acquisition, primitive recognition
and interpretation. The acquisition step consists of
extracting the strokes from the graphic device and encoding
them as a sequence of coordinates or lines approximating
the input as well as dynamic information such as speed,
pressure, etc. Primitive recognition consists of associating
the strokes with basic graphic structures as polylines, circumference
arcs, basic geometric shapes, etc. Finally the
interpretation step aims to associate a semantic meaning to
the graphical structure consisting of basic primitives. This
step requires the use of semantic knowledge of the working
domain. Let us further describe these steps in our system.
a) Acquisition of drawing strokes: when the user
draws on the digitizing tablet, the electronics of it convert
the input to a sequence of coordinates that are sent to the
computer. Thus, each stroke of the drawing is encoded as a
sequence of coordinates, see Figure 8. Taking as curves such
sequences of coordinates, singular points as maximum curvature
points, crossings, end points, etc. are detected. It allows
conversion of the strokes in sequences of segments as
Figure 8 illustrates.
b) Primitive recognition: The extracted segments are
grouped into primitives. These primitives depend on the
application domain. For example, in flowchart diagrams
primitives can be squares, circles, arrows, etc. In our case,
the alphabet of primitives consists of the valid shapes as
squares, diamonds, circumferences, etc. In the example
shown in Figure 8 the detected primitive is an open polyline
identified as a square
c) Interpretation: The last step is the validation of the
drawing after comparing it with possible models. For example,
in the case of forms requesting a particular drawing,
the input is compared with the ideal pattern with a distortion
tolerance allowed for due to the hand drawn input, and
an acceptance decision is made. In Figure 8 we show a shape
that was drawn and recognized as square, and the corresponding
model of Figure 7a). On the contrary in Figure
7b), we can see another square but with a high degree of
distortion regarding the model and that has been rejected.
In freehand drawings, each input shape is compared with
all the models in the database and is recognized as the model
with minimum distance, under a given threshold. The mod-
60 UPGRADE Vol. VIII, No. 2, April 2007 © Novática