FME Server Geodata Warehose White Paper

GeoGR - FME Server e-commerce Integration
Summary
GeoGR Inc. needed a system which could provide users with immediate online
access to the Graubünden area’s spatial data. Using FME Server for data translation
and distribution, Safe Software Gold Partner TYDAC developed a custom data
accessibility solution that enables GeoGR to achieve this primary goal. Data users,
including both professionals and the general public, can now view, order and
download datasets in the format they require, without burdening data authors.
The Organization
Graubünden, the largest canton (or partially sovereign state) in Switzerland, formed
GeoGR Inc. in 2007 alongside private surveyors and planners in an even-thirds
ownership structure. This public-private partnership was founded specifically to
address the increasing need for all kinds of data users, including various
professionals and the general public, to access the area’s spatial data.
The Situation
Spatial data relating to land within Graubünden is stored in many disparate
locations. The datasets are managed by various authors including cantonal and
federal institutions, private surveying companies, and public utility companies
focused on areas such as transportation.
These authors create and maintain a wide range of data types including raster
maps, high precision digital elevation maps, orthographic imaging, and cadastral
maps. Their datasets contain spatial information related to zoning, including danger
areas and protected areas; ski resorts and other tourism sites; agricultural land;
water management; natural resources management; forestry management; and
wildlife protection.
All of this valuable information is regularly used by the general public and an
extensive group of professionals across Graubünden including land developers,
infrastructure firms and various levels of government. Prior to the creation of
GeoGR Inc., accessing this data required a lengthy process in which data
users were required to locate the institution that held the data they required, submit
a paper-based request, and then wait to receive the data, manually prepared by a
staff member at that institution, 2-3 days later by email or postal mail.

The Challenge
The GeoGR mission was to eliminate the time consuming, manual process of spatial
data distribution by designing both a centralized data repository as a single access
point, and a web-based, automated data delivery system.
To centralize data and ensure real-time updates, the system would need to be
capable of:
Translating and transforming data from an extensive list of formats and
sources into a central database.
Integrating various data types such as vector and raster.
Providing an efficient process for keeping the system up-to-date.
For end users, the web-based system would need to provide an intuitive, selfservice
way to request any required spatial data.
www.geogr.ch - The Solution
Summary
GeoGR commissioned Safe Software Gold Partner TYDAC to develop the GeoGR data
access system based on TYDAC’s own data accessibility solution, Neapoljs, which
uses FME Server as the translation and distribution backbone. Through scheduled
batch updates, FME Server automatically loads disparate vector and raster data
from their original sources into the required data model for storage in a central
PostGIS database. This ensures that when end users access the Graubünden area’s
spatial data using the system’s UMN MapServer based web viewer, they are getting
the most up-to-date information available. Users who wish to purchase data can do
so using the system’s WebGIS viewer which allows them to geographically select
the data using a rectangle, polygon, or existing object such as a municipal border.
The viewer also enables the users to select the required format output such as
ASCII Grid, AutoCAD DXF, GeoTIFF, the Swiss INTERLIS format, Geodatabase, and
Shape, as well as the required data model. Alternatively, users can choose preprocessed
“off the shelf” datasets which are available in a variety formats for
download. Once a purchase request is submitted, FME Server quickly transforms the
selected data on-the-fly and makes the output dataset available in the web interface
for the user to download. Payment is handled by the Drupal back-end which allows
for purchase via PayPal or special project accounts.

PostgreSQL / PostGIS
The heart of the solution is a PostgreSQL/PostGIS system which hosts three main
databases:
The Geodatabase: contains all vector data (approximately 1GB)
The Drupal CMS and Shop database
The Neapoljs WebGIS CMS database
FME Server - Data Upload
The following illustration shows the data upload process:
Data is delivered by ftp up to a daily frequency from around 30 different data
suppliers
Vector data is delivered in different formats, such as INTERLIS (Swiss
"standard"), ESRI Geodatabases (mdb) and ESRI Shape
Raster data is delivered at a low frequency in GeoTIFF format
Grid data (DEMs) is delivered at a low frequency in ESRI ASCII Grid and ASCI
XYZ formats
Figure 1: Upload Schema

The FME Challenge: Arcs, compound polygons
To upload data to a PostGIS database using FME is normally a "piece of cake".
However, Swiss survey and planning data contains "compound polygons" (polygons
built by arcs and straights, the lately integrated new PostGIS type "curvepolygon").
As at the time PostGIS did not support that, therefore we had to store the arc
information as attributes, which meant dissolve the arcs when reading and rebuild
them when writing. In addition Swiss survey data allows errors (means selfintersecting
polygons, in cases where arcs overlap). As FME is very strict when it comes
to topology errors, we had to eliminate the errors, in addition to the arcs dissolve
and rebuild task. This tough FME challenge was solved, but at the end we had sort
of an FME Workbench burnout. The image below shows the quite complex custom
transformer doing the overlap cleaning:
Figure 2: FME Custom Transformer

FME Server: Data Download
Once an order has been completed (ordered and paid) the data is processed by FME
Server and other batch routines:
Data ordered in the original format is basically just zipped and put in the
users' download area.
Data in other formats or in sections is processed by FME Server batch
routines, zipped and put in the users' download area.
Very large datasets (over 500MB) can be ordered on DVD. This concerns only
aerial images and high resolution DEMs.
Once the data is ready (which normally only takes a few minutes) the user is
informed by e-mail.
Download Process
The figure below shows the download process when data is order using the viewer:
Choice of section: rectangle, manually drawn polygon or existing object.
FME Server batch is created and put in the shopping cart.
Once purchased, the FME Server batch starts and puts the result in the users'
download area.
Figure 3: Download Process

UMN MapServer - Web Mapping
UMN MapServer is used as the WebGIS Viewer engine. UMN MapServer accesses
the PostGIS database and for specific applications as well ESRI Shape and MapInfo
TAB files. Raster data is integrated as GeoTIFF and ECW. FME was used for creating
overviews of large raster datasets such as aerial images (original resolution, 20cm)
to the high performing ECW format. In addition UMN MapServer is used as WMS
client and server. In total over 150 MapServer Map-Files were created and the
amount is still growing.
Thank to a LUNIX 64bit server with 10GB of memory, users would be able to print
maps up to very large A0 format plots in terms of a few minutes. Currently the
application allows plots up to A3 format, due to copyright restrictions.
Neapoljs
TYDAC's Neapoljs is the Web Mapping Framework and CMS which sits on top of UMN
MapServer. Using Neapoljs technology, a "Desktop GIS like" application was
created, with a constantly growing tree of layers:
Figure 4: WebGIS Viewer

The viewer functionality can compare to a state of the art Desktop Mapping
Software and more:
Create maps of choice, combining up to 100 layers
Common zoom functions, such as in, out, pan, previous, next and full extent
Info-Tool: Querying information on layers
Display and print legends
Measuring: Distance and Area
Redlining: Drawing of points, lines, polygons and text
Printing as HTML and PDFs in a sophisticated way (choice of several paper
formats and layouts, capability to rotate the map)
Numerous Search Options and Query by Example (SQL-like queries)
Data Download of sections (rectangle, polygon) via FME Server
Figure 5: Data Download Interface for high resolution DEMs (Lidar Data)

WebGIS Viewer for Municipalities
In addition to the overall viewer, the Neapoljs CMS allows the configuration of any
number of viewers in different look & feel. This was used to offer to the 180
municipalities a low cost WebGIS solution. In the first year of the offer, more than
thirty municipalities signed up for the service.
As the CMS is web-based, the administration of the site can be done by the
communes themselves or by third party service providers such as surveying
companies. Beside the low cost, other advantages are that the municipalities can
use the base data of GeoGR at no charge (over hundred layers), but on top of that
they can add their own data in different GIS formats.
Figure 6: WebGIS for municipalities

Neapoljs CMS
As mentioned above, the Neapoljs CMS allows anybody to administer the sites just
using a browser. The CMS is very easy to use and can be learned in a few hours, so
that many administer the sites themselves. The sites can not only be maintained
but can be extended as one likes. New sites (i.e. for other purposes) can be created
at any time:
Definition of individual graphical user interfaces (GUIs, "corporate identity")
Creation of print layouts, up to almost any paper format
Upload of new or updated datasets and integration in form of maps or layers
Definition of any number of search criteria creating powerful search tools
Assignment of individual GUIs and functionality to different user groups
Management of user groups and users
Abbildung 7: Neapoljs CMS

Drupal
Drupal is an open source software package that allows an individual, a community
of users, or an enterprise to easily publish, manage and organize a wide variety of
content on a website. Hundreds of thousands of people and organizations are using
Drupal to power an endless variety of web sites. The built-in functionality, combined
with thousands of freely available add-on modules, enables many features such as
the ones used in GeoGR:
Electronic commerce
Collaborative authoring environments
File uploads and downloads
The image below shows the page about surveying data, containing a preview (using
OpenLayers accessing the PostGIS data via UMN MapServer), offering information
such as availability, metadata, WMS service etc. and stating the ways this data can
be ordered.
Figure 8: Surveying data: preview, information, ordering

TYDAC extended the available functionality with additional modules permitting:
Integration of map previews (see image above) and tile ordering (image
below) using OpenLayers and UMN MapServer
Different paying methods requested by GeoGR:
o Credit Cards using PayPal as payment engine
o Accounts: The canton and municipalities can create accounts, where
user can obtain without payment (data used in projects for the canton
and municipalities)
o Prepayment: Users can prepay a certain amount and therefore get
discounts of up to 30% on purchases
Neapoljs and FME Server integration: Users select off the shelf data packages
in the shop or sections in the viewer which are placed in their shopping cart.
Once a payment has been done, FME batch translation routines are
processed. The FME translation results are put in the download area of the
user, normally after a few minutes. In additions, users are informed by mail
once the data is ready.
Figure 9: Ordering of orthophoto tiles inside Drupal

The Benefits
The data accessibility system now provides users with immediate access to the
Graubünden area’s spatial data through an easy-to-use, web based system which is
available 24/7. The old manual process, which often required users to wait days for
the ordered data, has now been replaced with an automated system that allows
users to gain access to the data they need in just minutes. Moving forward, FME’s
flexible transformation capabilities and its support for a growing list of 200+ formats
will ensure that the system continues to scale as users’ needs evolve.
For data authors, the new system provides a hands-off approach by automatically
gathering their data and integrating it into a central repository. Because of the
system’s automated data ordering process, authors no longer need to manage
invoices and payments or the paperwork required to sell their data to end users.
These system capabilities ensure that data can be gathered and distributed without
impacting the current infrastructure or requiring human intervention.
What They’re Saying
“The biggest fear of the data authors in our area was that our new system would
force them to change their spatial data storage infrastructures. Using FME Server,
we were able to centralize our area’s geo related data and make it accessible to
users while leaving our existing databases intact. During implementation, we were
impressed by FME Server’s surprisingly fast performance and how easy it was to use
for altering data structures.” Peter Huser, CEO of GeoGR Inc.
“I can finally say goodbye to the old, lengthy data ordering process. FME Server has
made it possible for us to access our area’s geodata any time, day or night, in the
formats our applications require.” Sascha Flury, Planner
TYDAC Inc.
TYDAC is Safe Software reseller since 1997 and today Safe Gold Partner. In addition
TYDAC is MapInfo Business Partner Switzerland, Autodesk AND member and ESRI
EDN member.
Similar solutions to GeoGR have been realized or are currently under work. Some of
them use other basic components, such as Autodesk MapGuide along with
Topobase.
www.tydac.ch - www.mapserver.ch - www.geogr.ch
www.safe.com - www.mapserver.org - postgis.refractions.net - www.drupal.org