Collaboration by Illustration: Real-Time Visualization in Web3D

Abstract
Collaboration by Illustration: Real-Time Visualization in Web3D
Frank Goetz ∗
University of Paderborn
Bernd Eßmann † and Thorsten Hampel ‡
Heinz Nixdorf Institute, University of Paderborn
Figure 1: Visualization object embedded in Java-based Whiteboard of CSCW system.
Graphical representation of complex data sets is an important
method for supporting scientific collaboration. Following this idea,
we developed a novel Web3D-based visualization framework that
supports synchronous as well as asynchronous cooperation, even of
spatially separated scientist. Our openVisaar framework is based on
open-source technology and combines innovative techniques from
the area of computer-generated visualization with modern achievements
of CSCW (Computer Supported Cooperative Work). Therefore,
we extended OpenSG with the functionality for supporting
cooperative and remote features. OpenSG is a portable scene graph
system to create real-time graphics applications. openVisaar is designed
to accompany state-of-the-art CSCW systems in the collaboration
process by embedding the visualization features into the cooperation
environment. For this purpose openVisaar provides a rich
set of interfaces. By combining both worlds – visualization and
CSCW – users of our system get an up-to-date all-in-one solution
for collaborating with real world data.
CR Categories: H.5.3.b [Information Interfaces and Representation]:
Group and Organization Interfaces—Collaborative
Computing; H.5.3.c [Information Interfaces and Representation]:
Group and Organization Interfaces—Computer-Supported Cooperative
Work ; I.3.2.a [Computer Graphics]: Graphics Systems—
Distributed/Network Graphics; I.3.2.b [Computer Graphics]:
Graphics Systems—Remote Systems
Keywords: collaborative visualization, remote visualization, realtime
rendering, computer supported cooperative work
∗ e-mail: frank.goetz@uni-paderborn.de
† e-mail: bernd.essmann@uni-paderborn.de
‡ e-mail: hampel@uni-paderborn.de
1 Introduction
Many different libraries, frameworks, and even dozens of systems
and browser plug-ins are available for bringing three-dimensional
content into the Web. In this paper we do not plan to describe
another toolkit, framework, or library for visualizing threedimensional
scenarios in Web3D – we want to explain how the chosen
components work together and what their benefits are. The
basic ideas and concepts of the here presented system are very intuitive.
Perhaps, they can help other developers and scientists to
realize their own Web3D-based collaboration system.
1.1 GeoVis and ESkimo
In 1997 we began with the developments of a visualization system
named GeoVis. The need for GeoVis [Koenig et al. 1998] was
based on a corporation with geologists who wanted complex threedimensional
real-time visualizations of their time-variant data sets.
GeoVis was implemented on SGI workstation in C++, Open Inventor
2.0, Iris Viewkit, and Motif 2.0 and featured the visualization of
terrain, surfaces, and 3d glyphs from the underlying data set (see
Figure 2 (left side)). An interactive and time dynamic representation
could not be accomplished with GeoVis, though the data themselves
were multi-temporal.
The data set was calculated by climate simulation software developed
at the University of Münster and shows the geographic and

geological data of an urban area in North Rhine-Westphalia (Germany)
[Bernarnd and Streit 1995]. Each time step of the data set
consists of data for wind velocity (wind direction and wind speed),
temperature, moisture, air pressure, and more, for regular spatial locations
over ground. Climate simulation produces continuous time
steps in 30 seconds intervals. Such a geological data set has a size
of a few 100 megabytes.
Two years later (1999), SGI workstations had become obsolete.
Now, Windows NT was the modern operating system and most of
the SGI-based implementations could not be used directly on the
new platform. Therefore, we redesigned GeoVis and ported most
of its routines to a new visualization system. Additionally, slices,
isolines, streamlines, and stream ribbons were realized as new visualization
techniques in our second visualization system called ESkimo.
With the new system we wanted to accomplish two things:
Firstly, we were to use only libraries that are freely available (and
ideally would be open source) for non-commercial use and secondly,
we wanted to separate the application itself from the GUI.
ESkimo is based on MAM/VRS (Modeling and Animation Machine
/ Virtual Rendering System) [Doellner and Hinrichs 2002].
Our visualization system, which calculates and generates the threedimensional
view is the only part of the toolkit that has to be compiled
for different platforms. The Tcl/Tk-based user interface looks
the same on all platforms (see Figure 2 (right side)).
The main reason for separating the visualization part of the toolkit
from the user interface was to offer users (who are not involved
in the ESkimo project) the possibility to build their own user interface.
ESkimo runs on SUN Solaris, SGI Irix and different Microsoft
Windows operating systems.
Figure 2: Glyphs visualized with GeoVis (left) and ESkimo’s
graphical user interface (right).
1.2 In-Between Phase
So far so good – ESkimo worked fine for the next years and was
even runnable on computers in 2002. After reflecting the past years
it became clear that also ESkimo needed updating.
At that time, collaboration was demanded in every scientific field
and of course for all kinds of applications. VRML (Virtual Reality
Modeling Language) showed us during the last years that threedimensional
multi-user scenarios are possible and even work in
Web3D. Its successor X3D (eXtensible 3D) was ready for launch
to reach new dimensions in Web3D.
At this point, we rethought ESkimo and debated its strength and
weakness for a next-generation visualization system. Then we
started developing a Java-based visualization system. For OpenGL
rendering we decided on GL4Java (OpenGL binding for Java)
whereas we used X3D as format and for scene graph structure. At
this time neither Xj3D nor X3D enabled browser plug-ins were officially
available. After a few weeks we saw that our implementation
would not fullfill our goals, because:
• Java as programming language is too slow,
• Java memory management can not handle large data sets,
• and loading data from a server to a client took too long.
Here we stopped implementing a pure Java-based solution. Our focus
now was to have a real-time Web3D-based visualization system
for synchronously collaborative work with large data sets.
2 Ideas, Requirements and Concepts
Before discussing related work, we want to point out what kind of
system and features we were looking for, what needs we anticipated
from possible cooperation partners and what additional desires we
had:
1. a maintenance-free visualization system that runs on all kinds
of possible platforms (even on laptops with no or moderate
rendering resources) and various operating systems,
2. rendering of visualizations should be as fast as possible and
GPU-based (Graphics Processor Unit) rendering algorithms
should be no problem at all,
3. the system must handle large data sets, which should be available
for users immediately,
4. even locally separated cooperation partners should use it,
5. and finally join together CSCW (Computer Supported Cooperative
Work) and visualization to give users a working environment
for optimal collaboration.
After the collection of ideas and requirements we made a first concept.
A distributed visualization framework should fulfill all of
our requirements. To achieve a fast and high quality rendering our
server services should be implemented in C++ and OpenGL. There
should exist a SceneServer that handles and distributes the threedimensional
visualization scene (maybe a scene graph), manages
users and their rights, and communicates with SceneRenderes and
remote clients. Individual SceneRenderes (optimally one for each
client) should get the distributed scene from the SceneServer, render
it, grab the OpenGL framebuffer, encode the framebuffer into a
video frame, and stream the video to a specific remote client. The
client itself should be realized as a Java-based applet that can be
embedded into a web browser or as Java-based standalone application.
An overview of the concept can be found in Figure 3.
3 Related Work
There are many different visualization systems and toolkits available
that can be used for scientific visualization. We only want
to mention a few like AVS/Express by Advanced Visual Systems
Inc., IRIS Explorer by Numerical Algorithms Group Ltd., Open Visualization
Data Explorer by OpenDX formerly by IBM, and the
Visualization Toolkit (VTK) by Kitware.
Amira [Stalling et al. 1998] is an Open Inventor based visualization
toolkit for processing three-dimensional data, e.g. in medicine,
biology, physics, or engineering. Graphics hardware support is utilized
for displaying even very large data sets at interactive frame

Figure 3: Concept of the distributed visualization framework.
rates. Also, there exists a distributed Open Inventor implementation
[Hesina et al. 1999], unfortunately not directly supported
by Amira. Furthermore, Amira provides no specific collaboration
functionality. A possibility for working cooperatively with Amira
is using Microsoft’s NetMeeting. NetMeeting offers features such
as white boarding and application sharing. Microsoft stopped development
work on NetMeeting in 2003 and instead pushed Office
Live Meeting, formerly known as PlaceWare.
A generic solution for hardware-accelerated remote visualization
from [Stegmaier et al. 2002] works transparently for all OpenGLbased
applications and OpenGL-based scene graphs and does not
require any modifications of existing applications. They use a similar
approach as [Richardson et al. 1998] in their paper virtual network
computing. [Ma and Camp 2000] developed a solution for
remote visualization of time-varying data over wide area networks.
This system involves a display daemon and a display interface.
Data from the renderer is automatically compressed, transported
and afterwards decompressed. By using a custom transport method,
they are able to employ arbitrary compression techniques. In [Engel
et al. 2000] Engel et al. describe a visualization system that uses
image compression technologies for transporting visualization data
from a server to a client computer. The visualization parameters and
GUI events from the clients are applied to the server application by
sending CORBA requests.
An extensive overview of distributed and collaborative visualization
and the available systems gave [Brodlie et al. 2004] in their
talk at Eurographics 2003. Their gViz project [Wood et al. 2003]
aims to provide today’s scientists with visualization software that
works within modern Grid environments. A major idea is the Gridenabling
of existing visualization systems. Therefor IRIS Explorer,
a modular visualization environment, was extended. Now, a number
of scientist at different locations can join in a collaborative visualization
session. [Riding et al. 2005] present a Grid-based abstract
visualization system for applications scientists. Comparable
to Eskimo the user interface consists of two components; one for
controlling the pipeline parameters by means of the gViz computational
steering library, and one to display the visualization output.
GVK (Grid Visualization Kernel), which addresses the connection
of grid applications and visualization clients on the grid, was influenced
and motivated by the common grid services approach as
described in the scope of the OGSA (Open Grid Service Architecture)
[Foster et al. 2002]. The concept of GVK and its core functionality
for grid visualization can be found in [Kranzlmuller et al.
2004]. The infrastructure of GVK features a portal for arbitrary
simulation servers and visualization clients, while the actual pro-
cessing of the visualization pipeline is transparently performed on
the available grid resources.
After presenting automatic distribution of rendering workloads in
a Grid enabled collaborative visualization environment in [Grimstead
et al. 2004], Grimstead et. al. use in [Grimstead et al. 2005]
their distributed collaboration Grid enabled visualization environment
called RAVE (Resource-Aware Visualization Environment) to
collaborate with wireless PDA. RAVE supports on one hand active
clients (computers with graphics processor) and on the other hand
thin clients (computer with no or modest local render resources).
Clifton shows in [Presser 2005] a Java web application for allowing
multiuser collaboration and exploration of existing VRML worlds.
For this solution no Java is necessary at the client side, only a web
browser with a VRML plug-in that support ECMAscript has to be
installed.
The CoVis project (learning through collaborative visualization)
was finished 1998 at the Northwestern University [Ramamurthy
et al. 1995]. A principal purpose of the project is the use of new
technologies for the extension and improvement of the learning
possibilities for humans. Technically the project offers a collection
of tools that can be divided into two fundamental areas, collaborative
communication tools and tools for scientific visualizations.
The visualization tools can be started from the network, but deliver
no direct mechanisms for collaborative work. Habanero [Chabert
et al. 1998], which was developed at the Software Development
Division at the NCSA (National Center for Supercomputing Applications),
offers a framework-architecture for the production of
cooperative environments based on Java. Habanero brings along
some tools like a whiteboard, telnet, and audio chat. After connecting
to a server the user can open his/her own session or join already
existing sessions, where active sessions are presented in a list for
selection. There are no authorization or cooperation mechanisms.
OpenGL Vizserver from SGI [Sil 2005] is a client-server system
that allows users to remotely view and interact with large data sets
from any other system at any location in an organization and to collaborate
with multiple colleagues using the same application data.
The three-dimensional rendered scenes are transported as streams
of compressed pictures from the server to the individual clients.
There exist already some interesting visualization systems for cooperative
work. Visualization systems and frameworks in the past
mainly focused on the visualization itself, on handling the data that
have to be visualized, and on providing distribution functionality.
Some systems also featured collaboration, e.g. CoVis and Habanero,
but both were discontinued a few years ago. Today, SGI
Vizserver is the outstanding system for performing cooperative distributed
visualization. Comparing available systems and our list
of requirements we had to conclude that no single system fits our
needs.
4 Scenario of Use
Two spatial separated domain specialists (in our case geologists)
try to understand the same climate phenomenon. Following the old
fashioned way, the scientists would have to communicate sequentially,
according to workflow specified earlier. This can be realized
e.g. by email. In the beginning, a visualization expert processes the
data set into a meaningful representation. Then, the resulting representation
will be delivered to the two geologists. Now, the representation
is analyzed and discussed by the geologists. Additional enhancements
will be made by a visualization expert. Again, the geologists
will discuss the phenomena asynchronously via email. This
procedure will be repeated until the geologists are satisfied with the

esults. Finally, the results have to be stored and distributed to other
scientists. As a final step the scientists publish their results. They
would have to manually create a web page including the pictures,
annotations, and any meta-data. Overall this suspended process is a
very time consuming cooperation task.
Using our system the domain specialists and the visualization expert
work in one shared workspace (see Figure 1). The workspace
is presented within a whiteboard, allowing graphical editing and annotating
of the embedded objects. All objects are persistently stored
on a CSCW server. A remote visualization system renders a threedimensional
representation of the data set into an interactive object.
The visualization object is visible as an interactive picture embedded
in the shared workspace, which can be manipulated as any other
object. Additionally, the visualization object may be attached with
control panels for manipulating the visualization. These include a
navigation pane, a moderation pane, and a preference pane.
It is possible to create snapshots from the actual scene and store
them in the shared workspace. Measurements of single data entities
within the data set can be selected and stored as a cooperation
object. These features help the domain specialists to exchange their
ideas while working with the representation. The exchange itself
can be realized by storing interesting visual bookmarks to the scene
and annotating them.
An embedded chat facility derived from the CSCW system allows
communication while cooperatively exploring the visualization.
Additionally, to coordinate the exploration of the shared visualization,
users may use the moderation function, which allows
reserving timeslots for exclusive navigation in the scene. The results
of the visualization process may be published on the fly in
form of a website at any time of the process. For this purpose
the CSCW server generates a website containing the objects in the
shared workspace, e.g. the annotated screenshots. On a reload,
the website changes dynamically as a result of the changes in the
whiteboard.
Figure 4: openVisaar Java client.
5 The New Visualization Framework
As mentioned in the introduction, until now we developed two different
visualization systems. Our new approach is more than only a
toolkit for generating and displaying interactive three-dimensional
visualizations of geographical and meteorological data. open-
Visaar, which is the name of our latest development, can be used to
present all kinds of interactive visualizations. To achieve this, only
a few adaptations have to be made to the server and the client software.
Nearly all changes at the client side can be provided directly
by the server by sending all relevant data like menus and toolbars to
the client. Therefore the user does not have to install a new version
of his client program. He is able to use the same client software
for different types of visualizations, e.g. geographical or medical
visualizations.
The visualization framework offers a server-client based architecture,
at which the server is responsible for the scene graph management,
user management, and communication. The client application
only displays a video stream of the scene rendered in realtime
and delivers functionality for interaction and navigation. The
server/cluster side is completely written in C++ to obtain the highest
performance that is necessary for high quality three-dimensional
visualizations, whereas the client is completely written in Java.
The user interface of openVisaar (see Figure 4) is based on Java and
Swing. Because of using Swing and SkinLF (www.l2fprod.com)
the Look and Feel of the graphical user interface can be changed
and modified easily. By only editing the XML file that represents
the SkinLF skin, a modification of the skin can be made.
openVisaar uses OpenSG [Reiners et al. 2002] for the rendering of
the three-dimensional view. OpenSG (Open Source Scene Graph)
is a portable scene graph system to create real-time graphics programs.
Actually the OpenSG server-client architecture was designed
to distribute the scene graph in only one direction, e.g. for
rendering different views of a cave or power wall on separate computers.
To provide additional cooperation functionality the original
unidirectional protocol of OpenSG was extended to a bidirectional
one. As a result, miscellaneous OpenSG applications, running on
various computers, are able to communicate with each other in both
directions. One important feature of our visualization software is
the platform independence. Indeed, OpenSG is available for various
platforms like Microsoft Windows, Mac OS X, SGI Irix and
different Linux derivates. But after all, users have to install the
OpenSG package if they want to use our visualization framework.
Furthermore, we want to provide remote visualization functionality.
Therefore, in addition to the OpenSG package, real-time video
encoder software and a video-streaming server are installed on each
computer of the visualization cluster. This means that openVisaar
is part of a framework that enables the transport of real-time generated
visualizations from a server to a client computer by simply
grabbing the OpenGL framebuffer of the OpenSG based server application.
The grabbed images will be sent through a virtual video
device to a video encoding software.
The virtual video device is realized with the Video4Linux Loopback
Driver. This driver implements a video pipe using two
Video4Linux devices. The program supplying the data uses the
first device. The second device acts as a normal Video4Linux device.
It is utilizable by any application that fulfills the Video4Linux
specifications.
The encoding of the video stream is done with the MP4Live server
of the MPEG4IP package [Mackie 2002]. The MPEG4IP project
was started by Cisco’s Technology Center to further the adoption
of audio and video streaming standards and to serve as a toolkit
to track the ISMA (Internet Streaming Media Alliance) specifications
and requirements. MPEG4IP provides an end-to-end system
to explore MPEG-4 multimedia. The MPEG4IP package includes
many existing open source packages and offers the possibilities of
integrating them. MPEG4IP is a tool for streaming video and audio
that are standards-oriented and free from proprietary protocols and
extensions. It is available for Linux, Microsoft Windows, Mac OS
X and SGI Irix. One component of the MPEG4IP package is the
MP4Live server. The MP4Live server is able to produce MPEG-4
conform video streams in real-time.
Additionally, our framework uses the Apple Darwin Streaming
Server to multicast the encoded ISO compliant MPEG-4 stream

from the server to the client. The Apple Darwin Streaming Server
(developer.apple.com/darwin/) is a server technology, which allows
sending streaming video data to clients across the Internet using the
industry standard RTP (Real-Time Transport Protocol) and RTSP
(Real-Time Streaming Protocol).
There are different possibilities for the video playback; we prefer
an embedded Apple QuickTime Player canvas, since it provides all
features of a common Java canvas. Another method is to download
a dynamic link library based on the MP4Player. The MP4Player
is part of the MPEG4IP package. The MP4Player supports different
video/audio codecs and RTP/RTSP. This library is completely
written in C++. The complete communication between the client
software and the video player library is done by using JNI (Java
Native Interface). The player library generates a frame with the
video stream that is delivered by the server. This frame can be integrated
in our Java-based client as shown in Figure 4 on the right
side.
The architecture of the openVisaar framework is shown in Figure 5.
In the diagram two further clients are shown, which will be presented
in the next chapter. Until now only the first client (Java
Client at the top of Figure 5 or in Figure 4) was introduced.
Figure 5: Architecture of openVisaar.
Their exists the possibility to view the delivered video stream with
a common MPEG-4 video player (even embedded in a web page),
if the user does not want to download any extra software package.
Tests with the MP4Player, the Apple QuickTime player and the RealPlayer
from RealNetworks delivered positive results. Thus, in
principle, all RTSP capable video player software, which supports
ISO compliant MPEG-4 video streams, can be used for the realtime
decoding at the client side. Of course, for a non-video based
solution the OpenSG based server can directly be started at the remote/client
side. In fast networks with a high bandwidth and low
latency a direct interaction and accurate navigation in the threedimensional
scene is possible (this is not limited to local networks,
also high bandwidth internet connections like ADSL (Asynchronous
Digital Subscriber Line) can be used).
openVisaar allows every collaborator to move the viewing position
and to change the viewing angle to gain better comprehension of
the displayed data and to look at details. Different visualization
techniques can be altered and changed individually, attachments
and notes can be made at any three-dimensional point, and high
quality screenshots can be grabbed of a distinguished viewport.
The different collaborators can change all visualization techniques
and parameters. In this case, techniques and objects that are being
changed by one of the users are locked for all other users.
There are two possibilities for a cooperative work scenario. In the
first one a moderator controls all interaction of the users and gives
the moderation to distinguished ones. In the second scenario all
users are coequal. This means that an individual user can make
attachments, make annotations, or enable visualization techniques
whenever and wherever s/he wants. Objects will be locked by the
user manager and respectively by the SceneServer to avoid a coeval
changing of any object in the three-dimensional scene. The other
cooperation partners will be informed and also the objects will be
signed as unchangeable until a user has completed his work. Depending
on their rights, individual users are restricted in their interaction
possibilities. Specific menus, tabs, control elements, and
navigation elements will only be enabled and made visible in the
Java client after the rights were checked by the user manager. The
navigation itself can be controlled by using the navigation control
element (see Figure 4 left side) or by directly interacting on the
video panel. There are three different types of mouse navigators
available; trackball, walk and fly.
Finally we present some of the visualization techniques that are
supported by our openVisaar visualization framework so far. In
addition to the features provided by OpenSG and OpenSG Plus,
like low level base functionality, large scene support with methods
for efficiently handling large scene models, high level primitives
that provide complex primitives (e.g. free-form surfaces and volume
data), and high level shading enabling new rendering qualities
through efficient hardware use, we implemented a lot of important
algorithms for utilizing a wide range of well-known visualization
techniques (see Figure 4, 5, 6). E.g. height fields, particles, streamlines,
stream ribbons, and simple geometric objects for displaying
buildings and/or trees are available. Furthermore, we support particle
tracing, direct volume rendering, large texture maps, isosurface
rendering [Goetz et al. 2005b], and even point-based rendering
[Goetz et al. 2005a] in a hardware accelerated manner. All of
these methods are implemented as normal OpenSG NodeCores and
can be arbitrarily combined to gain a better view of the meteorological
data. Even these techniques can be combined among each
other, e.g. to combine point-based and polygon-based objects for
a mixed level-of-detail. In this case an object at a far distance is
rendered as points, whereas at a near distance using a triangle representation
renders the same object. While using large texture maps
(a few gigabytes of image data), e.g. to render a high quality earth
globe, a dynamic caching method that supports mip mapping, bump
mapping, and texture compression is available.
Now we want to take a closer look at specific visualization techniques
of openVisaar. The underlying terrain data and the resulting
model data is displayed in a three-dimensional space. Buildings in
the landscape are represented as boxes or as simplified houses on a
surface. Textures are used to show differences in land usage. Color
can show differences in height or in ground temperature.
Abstract glyphs are used to show air attributes like wind speed,
direction, and temperature. Glyphs (in other contexts also called
icons) are graphical objects. Their attributes – like position, shape,
color, size etc. – can be connected to data variables. The design
of glyphs follows strict guidelines in the creation of expressive and
effective visualizations. Data model and visualization goals are the
basis on which several glyph types were developed and evaluated

efore deciding on a special shape and attributes. In our approach
we use a wind tube with an additional raindrop as three-dimensional
glyph (see Figure 4). Our glyph can show different scalar variables
like temperature, pressure, moisture, and wind velocity depending
on its position and the currently viewed time step. The glyph is
pointing in the direction resulting from the three wind speed values.
To get additional information from the visualized data a user
has the possibility to select a glyph at a distinguished position in
the three-dimensional visualization. All data values at the selected
position appear immediately in the window of the remote client.
Figure 6: openVisaar visualization with landscape, glyphs and
streamlines.
openVisaar allows the generation of as many streamlines and stream
ribbons as needed. Particles can be inserted by placing an emitter
at an arbitrary position in the volume data set. Then streamlines
and/or particles as points or small triangles will be calculated and
animated in real-time for static or time-varying time steps. The best
way to display streamlines and particles is to combine them with
other visualization techniques. The particle system can be used to
simulate moving clouds that are generated from the air moisture
data of different time steps.
The hardware accelerated direct volume rendering, particle tracing
and isosurface rendering can be even used to display time varying
data sets in real-time in order to receive a better understanding of
the temporal change.
These are the basic features of the openVisaar visualization framework,
which was first introduced in [Goetz and Domik 2004]. In
the next chapter the new achieved functionality by integrating our
framework into an existing CSCW system is described in detail.
6 Embedding openVisaar into sTeam
To deliver the whole functionality of the server to the client, every
visualization technique of the framework and all functions of the
visualization framework itself have to offer an interface to receive
and send all important parameters from the server to the client and
vice versa. The user at the client computer can modify the values of
these parameters. For the mouse navigation and interaction in the
rendered scene all parameters have to be sent from the client computer
to the server by using the existing connection. Furthermore
all menus, toolbars and popup windows offer functionality to control
the specified visualization technique and settings to define the
appearance of the different visualizations. The server program has
the ability to change the look and feel of the graphical user interface
at the client side. Here, new functionality in the server program can
be made available to the user at run-time without an explicit update
of the client software.
In our combined solution the collaborative functionality (user management,
access rights, off-the-visualization annotation and structuring)
is provided by the sTeam CSCW system [Hampel and Keil-
Slawik 2002], the visualization is provided by the openVisaar visualization
framework (see Figure 7). sTeam has its own protocol
called COAL (Client Object Access Layer) that is used for the communication
between the sTeam server and the sTeam clients, e.g.
whiteboard or web browser. Also the communication between the
openVisaar SceneServer and the sTeam server was realized with the
COAL protocol. The communication between the SceneServer and
an individual SceneRenderer is still done by using TCP/IP.
The sTeam server provides the cooperation functionality. The
server consists of an object repository, which is stored in a relational
database and a core that manages the access to the stored cooperation
objects. This includes an event system. Clients may register
changes of objects. In case of a change the clients will be notified
by the system. This allows them to react directly to changes of any
object in the cooperation system. The core provides the rights management
for accessing the object by using ACLs (Access Control
Lists). This allows a flexible access right structure.
The visualization objects are embedded as a video stream (a kind of
interactive picture) in the sTeam whiteboard as shown in Figure 1
on the first page of this paper. The remote rendering and video
streaming is realized with openVisaar. As mentioned before open-
Visaar is an OpenSG based visualization framework that is able to
multicast in real-time rendered three-dimensional visualizations as
MPEG-4 compliant video streams using RTSP [Goetz and Domik
2003]. The openVisaar server coordinates the whole visualization
process. Each node of the visualization cluster is able to render and
encode a viewpoint independent three-dimensional graphical representation
for one connected client. The visualization of the data
itself is rendered in the same way on all render nodes (SceneRenderer),
but a choice between different or synchronized views is possible.
Figure 7: Architecture of the integrative cooperative visualization
system.
6.1 A Closer Look at openVisaar
If we take a closer look at openVisaar, we have to divide it into
two parts. On the one side there is the server, normally a cluster

composed of a few powerful computers, equipped with up-to-date
graphics accelerator boards and appropriate main memory. On the
other side there is the client, most times the computer of the user.
This can be a standard pc, laptop, or handheld without high demands
on the hardware. This is an important feature for mobile
settings. The only demand is that the client computer is able to
decode ISO compliant MPEG-4 video streams in real-time and advantageously
supports Java.
Lets take a closer look at the design of the visualization framework
(see Figure 5). The framework itself consists of a SceneServer
that distributes the three-dimensional scene to the SceneRenderer.
Each SceneRenderer is able to render from the users point of view,
whereby changes of all users are synchronized in real-time:
• Both, the SceneServer and SceneRenderer use OpenSG for
their scene graph management and rendering. As mentioned
before OpenSG is a portable scene graph system that provides
fundamental functionality to create real-time graphics
programs. One central part of the OpenSG design is
multithreaded asynchronous scene graph manipulation. The
OpenSG data structures are set up in a way that allows multiple
independent threads to manipulate the scene graph independently
without interfering with each other. This feature
allows synchronizing the manipulations of each user with the
manipulations of other users. Finally, every user of the collaborative
working community gets the same view on the current
data set.
• Our SceneServer manages the centrally stored scene graph.
Changes are executed in agreement with the user manager.
Modifications are journalized in a simple version controlling
system. On every change to the scene, the SceneServer sends
synchronized messages to all SceneRenderer to update their
replicated scene graph. The user manager of the SceneServer
provides the elementary coordination functions and a simple
rights management. If the remote client provokes an action
message, first the user manager checks the legitimacy of the
message with a check up of the existing rights of the user. If
the check up is correct, the action message will be forwarded
to the user specific SceneRenderer.
• Every SceneRenderer contains a replicated scene graph that
will be regularly updated with the scene graph of the Scene-
Server. Every modification of another user can be seen immediately.
The SceneRenderer generates an individual view
of the shared visualization scene and transports it as a video
stream to the remote client. Chosen user positions and orientations
are sent from the remote client to the SceneRenderer
to calculate correct positions and orientations for data manipulation
tools, selection-marks, and text messages. All graphics
are rendered into a virtual OpenGL framebuffer. The so
called pbuffer (pixel buffer) is used for hardware-accelerated
off-screen rendering, followed by the grabbing, encoding, and
sending of the pictures as video stream. Every frame of the
currently rendered picture is sent through the virtual video device
to the video-encoding server. Finally, the MP4Live server
generates a real-time video stream by using an ISO compliant
MPEG-4 video codec. The Apple Darwin Streaming Server
(gets the encoded video stream) is a server technology, which
allows sending streaming video data to clients across the Internet.
Alternative to the Apple Darwin Streaming Server we
support the Apache Web Server. Because of security and performance
issues, Apple Darwin Streaming Server is our first
choice. It is optimal, if the amount of the SceneRenderer is
equal to the amount of available remote clients. Otherwise
individual users have to share their views with other users.
A single sTeam object represents each data set for a specific visualization.
The sTeam access control system is used for the rights
management of the visualization techniques. The rights are structured
in a hierarchy order. Functionality for manipulating the visualization
becomes available according to these access rights. If a
cooperation partner has not the essential rights, then participation
at the cooperation process is not granted.
6.2 Platform Independent Video Integration
For displaying the visualization as video stream on the whiteboard
we chose the MPEG-4 standard, which also is used by the conventional
openVisaar client. MPEG-4 (an extension of the MPEG-
2 technology) is an ISO standard combining high-quality video
streaming with minimal data rates [Koenen 2002]. By using H.264
encoding, also known as MPEG-4 part 10, it contains a number
of new features that allow much more effective video compression
and provides more flexibility for applications in a wide variety of
network environments. As an open industry standard, anyone can
create an MPEG-4 player or encoder. openVisaar uses the open
source codec XviD for the encoding of the video stream and the
open source MP4Live server [Mackie 2002] for the video streaming
to the remote clients.
On the client side, the video is seamlessly integrated into the
sTeam whiteboard as a Java and GEF (Graphical Editor Framework)
[Moore et al. 2004] component avoiding media discontinuities
in the cooperation process and allowing direct interaction with
the visualization scene. Two different kinds of integrations were
realized:
1. Integration of an openVisaar object into the old Java-based
sTeam whiteboard using Apple QuickTime for Java as video
decoder (see Figure 1)
2. Integration of an openVisaar object into the new Eclipsebased
sTeam whiteboard using a pure Java-based MPEG-4
video decoder (see Figure 8)
Figure 8: The Eclipse-based Whiteboard with its standard components:
whiteboard view (1), user view (2), and chat (3), extended
with the new visualization plug-ins (visualization object (4), visualization
bookmark objects (5), navigation control (6) and visualization
preferences (7)).
The standalone version of the openVisaar client and the one embedded
into the old sTeam whiteboard use Apple QuickTime SDK

for integrating video streaming into the graphical user interface.
Apple’s QuickTime SDK provides good support for high quality
MPEG-4 decoding with low CPU (Central Processor Unit) usage,
but it is based on platform dependent runtime libraries. It was chosen,
because Sun’s Java Media Framework platform independent
video streaming lacks support for efficient streaming protocols and
formats like RTSP and MPEG-4.
In our latest approach we use IBM Toolkit for MPEG-4. It is based
on pure Java code and is therefore platform independent. Plus,
it provides decoding facilities fast enough for our purposes. The
drawback is the missing support of off-screen rendering, which is
necessary to nest video streaming into GEF components. This leads
to the problem that additional to the video rendered into a GEF
component, there exists always a source window providing the decoded
picture. While no other solution is available at the moment,
this window is switched to the background or is minimized after
creation. In addition, QuickTime is integrated as an alternative rendering
technique.
6.3 Adaptations to openVisaar Client
The openVisaar client was originally implemented as a standalone
client, using Sun’s Swing components for providing the GUI. To
adapt the functionality of this client to the novel whiteboard plugin,
large parts had to be reimplemented from scratch. First the user
interface had to be adapted to the Eclipse SWT (Standard Widget
Toolkit). Here the separation of the user interface code and the
functional code in the openVisaar client application proved useful.
A redesign of the code for managing connections to the openVisaar
SceneServer and for handling remote events was necessary. These
code supported one-to-one connections only, because the client operated
in an SDI (Single Document Interface) like way. In conjunction
with sTeam, one workspace can contain multiple open-
Visaar objects. Therefore multiple connections based on the same
environment must be supported. Finally, we used the provided
extension-point mechanism to extend the Java-based and Eclipsebased
whiteboard clients with a new component for displaying the
visualization. This component handles the complete graphical representation
of our openVisaar based visualization object within a
sTeam area.
7 Results
Approaches like presented in this paper suffer under the complexity
of the integrated software packages and used protocols. For example,
just the openVisaar framework bases on fifty non-standard
packages installed on the server side. Furthermore, using different
software packages and libraries poses the question of the overall
performance.
In this section we present some results of the rendering and encoding
performances, frame rates, image quality, and latency of the
video streaming. Of course, most measured values depend on the
hardware configuration, e.g. computational power or network bandwidth.
To overcome the problem of more collaborators than available
SceneRenderers at the server side, it is possible to share one view
on the three-dimensional scene. Therefor, only a single SceneRenderer
is necessary for the rendering. In this case the moderation
modus should be used in order to coordinate simultaneous actions
of different users. To deliver the video to all clients without sending
it more than one time, multicasting must be enabled. Today,
this is no problem if a powerful backbone with up-to-date routers
is used. The multicast video streams often can be received even
using network connections provided by local ISPs (Internet Service
Providers), as our tests showed.
7.1 Rendering Performance
The rendering works in the same way as on a local computer, hence
it depends on computational power of the CPU and GPU. Using
graphics hardware accelerated algorithms, we can e.g. render up
to 1.000.000 particles or 64.000 streamlines at an interactive frame
rate on a 2.8 GHz Intel processor with NVIDIA GeForce 6600GT
graphics card. The more computational power a rendering node
(SceneRenderer) provides, the more complex the three-dimensional
scene can be. By using frameworks like Chromium [Humphreys
et al. 2002] it is even possible to use a cluster as back-end for rendering
or simulation.
7.2 Encoding Performance and Image Quality
A major problem of our solution is that encoding costs processor
time. If you want to get high image quality, high resolution, and
little kilobit rate the latest video codecs e.g. X264 (H.264) are
advantageous, but they cost extra processing time. Figure 9 depicts
a comparison of the image quality of four XviD (H.263) encoded
real-time video streams from 50 kilobits to 2000 kilobits.
It is remarkable that the image quality barely improves within the
range from 500 kilobits to 4000 kilobits (highest encoding rate using
XviD). So even an ADSL connection can be used at the client
side to receive video streams in a adequate quality. Nowadays, the
monthly costs for a six-megabit flatrate (no extra traffic costs) are
very low in Germany, even 4000 kilobits video streams are receivable.
Figure 9: Image quality of the video stream at 50, 200, 500, and
2000 kilobit (from top left to bottom right).
At the moment our utilized computer system for each rendering
node (SceneRenderer) is a Dell Precision 380 Essential with Intel
Pentium D Processor 820 (two processor cores with 2.8GHz) and

NVIDIA Quadro FX540 graphics card. One core is used for the rendering
of the visualization and the second core encodes the grabbed
OpenGL framebuffer in real-time as video. Another possibility is
to use a one-core processor in combination with real-time video encoding
hardware. If the rendered scenes are not too complex and
a lower kilobit rate is acceptable, even only one processor does the
job. This works fine for most of our visualization scenarios at a
resolution of 352x288 pixels and an encoding rate of 1000 kilobits.
While the grabbing of the framebuffer is very fast on up-to-date
graphics card equipped with PCI Express, latency between rendering
the three-dimensional scene and encoding it as video costs only
a delay one frame. This means that by rendering with a frame rate
of 20 frames-per-second the latency is 50 milliseconds for this task.
7.3 Latency of Video Streaming
We identified five potential places (from video encoding to video
decoding) where latency can take place: At the MP4Live server, in
the kernel IP transmit stack, in the network, in the kernel IP receive
stack, and in the video player.
• MP4Live has very little latency. There is no latency from
when the video frame is encoded with MPEG-4. With MPEG-
2, the B-Frames (Bidirectional Frames) require a two-frame
latency period.
• The kernel IP stacks can have some latency built in. In normal
case latency is very low and does not need to be considered.
• Network latency will occur, as well. We made some tests in
Germany, using the University of Paderborn as backbone for
the MP4Live server and an Arcor ADSL connection in Dortmund
for the client (direct distance between the two cities is
100 kilometers). The standard ping time on this connection
is around 40 milliseconds. Whereby the signal is routed over
ten hops including the two cities Frankfurt and Leipzig. The
round trip of one TCP/IP packet has a complete length of 1000
kilometers and differs a lot from the direct connection of the
two cities. As remark, the ping time in the internal network of
Arcor is only 20 milliseconds. These low latencies are possible
by using FastPath, which means to disable interleaving, a
method for error correction in ADSL connections.
• The Apple Darwin Streaming Server introduces another latency
to the communication chain. By disabling buffering of
the video streams this latency can be reduced.
• Finally, the video player (decoder) at the client side produces
latency, because often two seconds buffering is enabled for
streaming. This latency can also be reduced to a few milliseconds
by disabling the buffering of the video player (decoder).
The latency between server and client can be reduced to 100-300
milliseconds using a LAN connection. Depending on traffic and
collisions this time can raise up to 1.000-2.000 milliseconds, especially
by using slow wireless LAN connections. In normal case
the latency between client and server is lower than 50 milliseconds,
because the network packages are very small and often consist of
only a few bytes.
As a workaround to high latency network connections (greater than
500 milliseconds) we provide the possibility to work on previously
recorded video streams in an asynchronously manner. A collaborator
has the possibility to choose a start and an end point in the
three-dimensional scene. Afterwards, a camera track is calculated
and a corresponding video is generated. This video can be annotated
and used for discussion in the whiteboard in the same way as
with the synchronous video streams. This method is not as powerful
as the interactive work with live streams, since in-scene annotations
are not possible and only the two-dimensional previously recorded
video can be annotated.
If the latency is below 500 milliseconds a user can work with our
system in a synchronous manner. We found out that a latency below
200 milliseconds is acceptable, which is the normal latency in
our university’s LAN (not WLAN). Collaborators using a high latency
network connection are still able to watch the video stream
and to change some attributes of the visualization. But a navigation
through the three-dimensional scene becomes very difficult,
because of the delayed visual response.
One benefit of using multicasting for providing video streams is
that it scales very well, even for large groups of spectators. The
bandwidth of the streaming server must not increase in the same
way as the group of spectators. If the video stream is published
by the collaborators, the spectators can watch the live video by just
pointing their MPEG-4 capable video player to the streaming server
(one SceneRenderer). A very nice feature, especially for e-learning
scenarios, where lectures present visualizations to many students.
8 Conclusion
Facing the gap between full-featured CSCW and visualization systems,
this paper presents an approach towards an integration of a
remote visualization framework into an existing CSCW environment.
All components are based on open libraries that can be used
freely for non-commercial purposes. The visualization framework
is based on a state of the art scene graph system (OpenSG) and supports
remote as well as distributed rendering. By combining the visualization
system with a sophisticated CSCW system the complete
solution offers a wide range of new and unique features. Especially
the cooperation facilities of the visualization system are extended
in an extraordinary way. With openVisaar plus sTeam spatially separated
domain specialists can analyze complex visualizations and
organize their gained insights cooperatively by using the integrative
cooperation environment. Last but not least research results can be
published in an easy and seamless way. By using a Java based remote
client, collaborators have the possibility to work spatially separated
with the same visualization in Web3D and are able to share
their insights directly. Our solution uses a high quality MPEG-4
conform video stream to distribute the pictures. Furthermore RTSP
offers a robust protocol that can stream multimedia (video and/or
audio) over multicast in one-to-many applications. This is even
possible, if some users use limited computers like standard desktop
computers, laptops, or handhelds. Furthermore we have a high
level of security by using a server-based remote rendering solution
with video streaming. All cooperation partners get only encoded
pictures and some selected values at distinguished positions in the
data set. There is no possibility to access the raw data set, not even
at a certain timestamp. This feature is especially important when
working with valuable and safety-relevant data sets, which may be
stolen when transmitted between hosts.
As a final conclusion, we may state that we achieved a high level
of cooperation support for visualization processes by combining
two sophisticated systems of each domain. The chance to improve
the system in a fast and simple way by adding several useful features
shows the flexibility of this solution. In future work we will
improve our system by integrating new cooperation features and
more visualization techniques into the system. In addition, we will
change the focus from geography and geology to medical visualizations,
since one of our current cooperation partners consists primarily
of radiologists.

References
BERNARND, L., AND STREIT, U. 1995. Three-dimensional
boundary layer modeling and gis. In GISDATA Specialist Meeting
of the ESF.
BRODLIE, K. W., DUCE, D. A., GALLOP, J. R., WALTON, J. P.
R. B., AND WOOD, J. D. 2004. Distributed and collaborative
visualization. Computer Graphics Forum 23, 2, 223–251.
CHABERT, A., GROSSMAN, E., JACKSON, K., AND PIETROVICZ,
S. 1998. Ncsa habanero - synchronous collaborative framework
and environment. In White Paper, Software Development Division
at the National Center for Supercomputing Applications.
DOELLNER, J., AND HINRICHS, K. 2002. A generic rendering
system. IEEE Transactions on Visualization and Computer
Graphics 8, 2, 99–118.
ENGEL, K., SOMMER, O., AND ERTL, T. 2000. A framework for
interactive hardware accelerated remote 3d visualization. In Proceedings
of EG/IEEE TCVG Symposium on Visualization VisSym
00, Eurographics Association, 167–177.
FOSTER, I., KESSELMAN, C., NICK, J. M., AND TUECKE, S.
2002. The physiology of the grid: An open grid services architecture
for distributed system integration. In Open Grid Service
Infrastructure WG, Global Grid Forum.
GOETZ, F., AND DOMIK, G. 2003. A framework for video-based
and hardware-accelerated remote 3d-visualization. In Proceedings
of Eurographics 2003 Short Presentations, Eurographics
Association, 263–269.
GOETZ, F., AND DOMIK, G. 2004. openvisaar - enriching cooperative
visualization by combining open technologies. In 1st
International Conference on Cooperative Design, Visualization
and Engineering, Springer-Verlag GmbH, 38–46.
GOETZ, F., HILLEBRAND, L., AND DOMIK, G. 2005. Integration
of point based rendering into a common scene graph system. In
Proceedings of the 5th ACTA press International Conference on
Visualization, Imaging, and Image Processing, ACTA press.
GOETZ, F., JUNKLEWITZ, T., AND DOMIK, G. 2005. Real-time
marching cubes on the vertex shader. In Proceedings of Eurographics
2005 Short Presentations, Eurographics Association.
GRIMSTEAD, I. J., AVIS, N. J., AND WALKER, D. W. 2004.
Automatic distribution of rendering workloads in a grid enabled
collaborative visualization environment. In Proceedings
of ACM/IEEE conference on Supercomputing 2004, IEEE Computer
Society.
GRIMSTEAD, I. J., AVIS, N. J., AND WALKER, D. W. 2005. Visualization
across the pond: how a wireless pda can collaborate
with million-polygon datasets via 9,000km of cable. In Proceedings
of the 10th international conference on 3D Web technology,
ACM Press, 47–56.
HAMPEL, T., AND KEILSLAWIK, R. 2002. steam: Structuring
information in a team - distributed knowledge management in
cooperative learning environments. ACM Journal of Educational
Resources in Computing 1, 2, 1–27.
HESINA, G., SCHMALSTIEG, D., FUHRMANN, A., AND PUR-
GATHOFER, W. 1999. Distributed open inventor: a practical
approach to distributed 3d graphics. In Proceedings of the ACM
symposium on Virtual reality software and technology, ACM
Press, 74–81.
HUMPHREYS, G., HOUSTON, M., NG, R., FRANK, R., AH-
ERN, S., KIRCHNER, P. D., AND KLOSOWSKI, J. T. 2002.
Chromium: a stream-processing framework for interactive rendering
on clusters. In Proceedings of SIGGRAPH 2002, 693–
702.
KOENEN, R. 2002. Mpeg-4 overview. In ISO/IEC
JTC1/SC29/WG11 N4668.
KOENIG, M., LENZ, C., AND DOMIK, G. 1998. Visualization
of meteorological data using an interactive flight. In Computer
Graphics International’98.
KRANZLMULLER, D., HEINZLREITER, P., AND VOLKERT, J.
2004. Grid-enabled visualization with gvk. In Proceedings
of Grid Computing: First European Across Grids Conference,
Springer-Verlag GmbH, 139–146.
MA, K.-L., AND CAMP, D. M. 2000. High performance visualization
of time-varying volume data over a widearea network. In
Proceedings of Supercomputing 2000.
MACKIE, D. 2002. Streaming video and mpeg4ip. In Presentation
of MPEG4IP at the Silicon Valley Linux User’s Group, Cisco
Technology Center, Silicon Valley, USA.
MOORE, W., DEAN, D., GERBER, A., WAGENKNECHT, G., AND
VANDERHEYDEN, P. 2004. Eclipse Development using the
Graphical Editing Framework and the Eclipse Modeling Framework.
IBM Press.
PRESSER, C. G. M. 2005. A java web application for allowing
multiuser collaboration and exploration of existing vrml worlds.
In Proceedings of the 10th international conference on 3D Web
technology, ACM Press, 85–92.
RAMAMURTHY, R. K. B., WILHELMSON, R., PEA, R., GOMEZ,
L. M., AND EDELSON, D. C. 1995. Covis: A national science
education collaboratory. In Proceedings of the American
Meteorological Society 4th Conference on Education joint with
the 11th Conference on Interactive Information and Processing
Systems for Meteorology, Oceanography, and Hydrology, 15–20.
REINERS, D., VOSS, G., AND BEHR, J. 2002. Opensg - basic
concepts. In 1. OpenSG Symposium.
RICHARDSON, T., STAFFORD-FRASER, Q., WOOD, K. R., AND
HOPPER, A. 1998. Virtual network computing. IEEE Internet
Computing 2, 1.
RIDING, M., WOOD, J., BRODLIE, K., BROOKE, J., CHEN, M.,
CHISNALL, D., HUGHES, C., JOHN, N. W., JONES, M. W.,
AND ROARD, N. 2005. e-viz: Towards an integrated framework
for high performance visualization. In Proceedings of the UK
e-Science All Hands Meeting 2005, EPSRC, 1026–1032.
SILICON GRAPHICS INC. 2005. OpenGL Vizserver 3.1:
Application-Transparent Remote Interactive Visualization and
Collaboration, Technical White Paper, http://www.sgi.com.
STALLING, D., ZOECKLER, M., AND HEGE, H. C., 1998. Amira,
the software formerly known as hyperplan. Technical report,
Konrad-Zuse-Zentrum fuer Informationstechnik Berlin (ZIB).
STEGMAIER, S., MAGALLON, M., AND ERTL, T. 2002. A generic
solution for hardware-accelerated remote visualization. In Proceedings
of EG/IEEE TCVG Symposium on Visualization Vis-
Sym’02, Eurographics Association.
WOOD, J., BRODLIE, K., AND WALTON, J. 2003. gviz - visualization
middleware for e-science. In Proceedings of the 14th
IEEE Visualization, IEEE Computer Society.