ArcSDE - Northeast Arc Users Group

New technologies better enablingWorkflow integrationPlanningManagementScienceOperationsEngineeringRegulation. . . Supporting Collaboration

GIS Is Deployed In Many WaysTraditionalEmerging

ESRI’s s GIS Platform

ESRI Server ProductsGISApplicationServicesArcGISServerArcIMSGeoPublishingArcSDEGeoDataManagement. . . Three Complementary Systems. . . Supporting Multi-user user Implementation

ArcObjects and GeodatabaseArcGIS is built and extendedusing software components calledArcObjects. ArcObjects components aresoftware objects that have multiple developerAPIs. These include ComponentObject Model (COM), .NET, Java, , andC++.• ArcObjects is at the core of all the ArcGISproducts:– ArcGIS Desktop– ArcGIS Engine– ArcGIS Server• The term “GeoDatabase” has multiple meanings.– Core ArcGIS data model– Set of COM components in ArcGIS for accessing data– A physical store of geographic data“Geodatabase”ArcObjectsArcSDE C-APIThe Geodatabase

The GeodatabaseAn ordered collection of GIS datasets used with ArcGISHeld as an Accessfile, a folder, orwithin a relationaldatabase.The native data structure of ArcGIS

Model Real World Features for betterManagementMap LayersModel inputs and derived datasetsGlobe LayersPrimary unit of information management and exchange

EditingHow Users work with individual dataelementsQuery, Identify, Analysis, …

Processes and Workflows• With increased complexity to model real worldfeatures and associated processes it is importantto carefully define:– Data architecture and models– Data creation workflows and standards• Data and Business process integration• Versioned Editing (will discuss later)– Architecture– Management processes– Metadata• The above must be defined in the context of thebusiness applications they need to support

An Example:Some Lessons Learned and Best Practicesfrom Major US City project• Coordination between project lifecycle and softwaredevelopment lifecycle required careful configurationmanagement– Project began with the release of ArcGIS 8.0 and DB2 7.1 and wasdelivered on ArcGIS 8.3 and DB2 8.1– Each new release of any component required regression testing ofapplication and review of configuration parameters– Numerous service packs and fix packs added complexity to theconfiguration management process– As a result of the project, ArcSDE for DB2 and DB2 SpatialExtender are solid and performance is very strong– Recommendation is to design, develop, and deploy on a releasedversion of ArcGIS and underlying RDBMS and understandimplications of upgrading any component

Enterprise and Knowledge• Education becomes critical and must be part ofthe planned system implementation• The planned system implementation must haveand understood context (fit) within theorganization’s s overall enterprise architectureand workflows

Storing the Geodatabase• Personal geodatabase– Built on Microsoft Jet Engine– Microsoft Access file• Enterprise geodatabase– Stored within an RDBMS– Requires ArcSDEPersonalGeodatabaseArcGISArcSDE• Differences:– Personal GDB size limit: 2GB– Enterprise GDB allows versioningand multi-user user editingEnterpriseGeodatabaseIBM DB2InformixOracleSQL Server• Use same ArcGIS tools with either storage type

Three kinds of Geodatabases1. PersonalGeodatabase2. FileGeodatabase(9.2)Stored in Access. 2 GB Limit, but effective size is 250 to500 MB. Windows only. Single editor and a few readers.Legacy support.Stored in file folder. Up to 1 TB per dataset. Any platform.Single editor and a few readers. Recommended.3. ArcSDEGeodatabaseStored in RDBMS (Oracle, SQL Server, DB2, Informix).Uses ArcSDE. Versioning and Multi-user user Support.Recommended. “Personal” and “Workgroup” at 9.2

ArcSDE and personalgeodatabasesArcSDE• RDBMS (Oracle,SQLServer, DB2, Informix)• Requires ArcSDE licenseto edit data• Multiuser editing/versioning• Scalable• Supports raster storage• RDBMS security• Fault tolerance/reliabilityPersonal• MS Access (.mdb)• Single-user editing only• Size limit of 2 GB• File system security

Spatial Data Formats• Storing spatial data has changed with time andtechnology• Different formats have their advantages anddisadvantages• Data formats are used to represent real-worldobjectsArcInfo LIBRARIANCoverageGridTINShapefileImage CatalogSDEArcStormGeodatabase

• ArcSDE vs. File SharingArcSDE vs. File– Compare amount of data transferred over networkfor a request (example using ArcIMS HTMLbrowser)Image 1 (vector only)ArcSDE ApplicationServerArcSDE Direct ConnectUNC file sharingImage 2 (vector and image)Image 1 Image 2137 KB 542 KB179 KB 686 KB1124 KB 116095 KB

Database ArchitecturesSQL Server 7/2000DB2 UDBOracle 8i/9i/10gInformixInstanceInstanceDB DB DB DBOnly SQL Server andDB2

Multiple Spatial Database Support forSQL ServerMultiple Spatial DatabasesSingle SpatialDatabaseArcGISArcGISArcGISArcGISArcGISArcGISArcGISArcGISPlanningPlanningEnvironEnvironSafetySafety

ArcSDE Connection ArchitecturesApplicationServerDirectConnectlibraries (dll(dll)RemoteServerTCP/IPDB NetProtocolTCP/IPgsrvrDB ServerDB NetProtocolgsrvrgiomgrgiomgr

Direct Connect from ArcIMS• Author uses Application Server connections• Edit map service .axlfile directly– SDEWORKSPACE element– No encrypted passwords• Set encrypted="false" and use valid password– SQL Server– Oracle 9i

Spatial Referenceby Feature Class or Feature Dataset• Precision/Scale - ArcSDE stores geometry coordinatesas positive integers– Faster processing, better compression, aids coincident geometry– Coordinates rounded to nearest integer– Property of a stand alone feature class or a feature dataset• Coordinate System– Geographic, UTM, or State Plane andassociated projection Information if aprojected coordinate system• Spatial Domainallowable coordinate range (2 32 postive integerdomain)– Precision (Scale)– Minimum X, Y (Offsets)00OFF LIMITS2 32 ,2 32

Applying Precision to data• Precision is a scale factor– Precision multiplies coordinates for integer storage– Used to preserve decimal places before roundingto integer– Larger precision preserves more digitsX = 123.456Floating PointPrecision = 1000Coordinate inArcGIS Desktop(123.456) (123.46)Multiply by precision(123.456 (123.46 * 1000) 100)Divide by precision(123456 // 1000) 100)Integer Coordinate ingeodatabase(123456) (12346)A

ArcGIS Desktop HelpPrecision and Extent

The Geodatabase Supports All Spatial DataTypesTopologyFeaturesCadastralDatasets107’DimensionsNetworksRastersGeodatabaseDomains, SubtypesRelationshipsABCAnnotation27 Main St.3D ObjectsAddressLocatorsTerrainAttributeTables

ArcSDE data types• Storage of:– Point– Line– PolygonPoint– Annotation– Dimension– Raster– Multi-patch– Geometric networks– Polygon topology– SurveyTARBORO STRaster136.98AnnotationLinePolygonDimension

Geodatabase ComponentsGeodatabaseTablesFeature datasetSpatial referenceFeature classesPolygonLinePointAnnotationRelationship classesRouteDimensionRaster datasetsRaster catalogsSurvey datasetsSurvey folder SurveyToolboxesTool Model ScriptGeometric networksTopologyBehaviorAttribute defaultsAttribute domainsSplit/Merge policyConnectivity rulesRelationship rulesTopology rules

Common GIS Dataset TypesFeature ClassesRastersTablesSimple geometry storage plus rich operators

Geodatabase Foundation: Simple FeaturesFeature• Feature class is a table• Features are held as rows• Attributes are recorded in columns• A shape column holds the geometryShapeColumn

Attribute domains, relationships, andsubtypes – To model descriptive dataRelationshipsSubtypes and Domains

Topology – To model how featuresshare geometryCounties cannot overlap one anotherCounties must be covered by StatesRoad centerlines must not overlapParcel lot lines must be covered by parcel polygonboundaries

Networks - To model navigation, flow, and connectivityA Network Dataset is an interconnected series of edges, junctions, turns, andnavigation properties.

Feature Classes• Table that also stores shapes for features– Each row represents a feature and its attributes– Stores features with one type of geometry(point, line, or polygon)• Allows for x, y, z, and m coordinates– Has an associated spatial reference• Projection, coordinate system, and spatial extents

Feature Datasets• Contain feature classes– All share the same spatial reference– Never tables no geometry• Required to model spatialrelationships– Geodatabase topology– Geometric network

Raster Datasets – To manage very largecontinuous imageryBIDBlockIDShapeImageDateXfer89871 -- Index 12-1212-04BOC100100100100101Etc.Etc.

Raster Catalogs – To manage a tiled image layerImage TilesIDShapeImageDateAbstract11.jp212.jp213.jp22101 11.jp2 12-1212-98Stuff2102 12.jp2 12-1212-98More stuff21.jp222.jp223.jp22103 13.jp2 12-1212-98Etc.2201 21.jp2 12-1515-98Etc.31.jp232.jp233.jp22202 22.jp2 12-1515-98Anz. . . . . . . . . . . . . . .

Geodatabase Data Types• Domains, Relationships, Subtypes• Topology• Raster Catalog• Linear Referencing• Network• Terrain• 3D Object• Address Locator• Cartographic Representations• Cadastral Dataset• VersioningGeodatabaseObjectsRulesDataApplication Tier:• Ensure Integrity• SpatialRelationships• BehaviorDBMS:• Spatial Types• Indexes• AttributesRDBMSIntegrity, Spatial Relationships, and Behavior

ArcCatalogManaging Workspaces and Geodatabases: Organizing GIS contentWorkspaceGeodatabaseMap 2CatalogGlobe 1Data 1Map 1Data 2ModelFile folders holding:Layer• Datasets and documents• Maps, Layers, Metadata, Models,DatasetsAn ordered collection of featureclasses and datasets held in:• File folders• Relational databases: Oracle, SQLServer, Informix, DB2

ArcSDE Administrative Schema• ArcSDE Data Dictionary– GeoDatabase tables (“GDB_– ArcSDE tables– associated indexesGDB_”)• ArcSDE Packages/Stored Procedures– Stored Procedures/Packages– Triggers– Sequences/Identity

ArcSDE client accessArcObjectsArcMapArcView 3xMapObjectsArcIMSsdecommandsArcSDE APISDE tables• Layer Registry• Table Registry• Logs• Raster• Spatial Columns• VersioningUser tables• Business Tables• Business Tables withSpatial Column• Raster TablesGDB_ tables(Geodatabase only)• Feature Classes• Feature Datasets• Topology• Rules• Metadata• Domains & Subtypes• Geometric Networks• Relationship Classes• Disconnected EditingUser tables• Geometric NetworkTables• Topology TablesArcCatalogArcToolboxArcGIS ServerArcGIS EngineA

ArcSDE 9 Data Dictionary Tables• GDB_ANNOSYMBOLS• GDB_ATTRRULES• GDB_CODEDDOMAINS• GDB_DEFAULTVALUES• GDB_DOMAINS• GDB_EDGECONNRULES• GDB_EXTENSIONS• GDB_FEATURECLASSES• GDB_FEATUREDATASET• GDB_FIELDINFO• GDB_GEOMNETWORKS• GDB_JNCONNRULES• GDB_NETCLASSES• GDB_NETWEIGHTASOCS• GDB_NETWEIGHTS• GDB_NETWORKS• GDB_OBJECTCLASSES• GDB_RANGEDOMAINS• GDB_RASTERCATALOGS• GDB_RELCLASSES• GDB_RELEASE• GDB_RELRULES• GDB_REPLICADATASETS• GDB_REPLICAS• GDB_SPATIALRULES• GDB_STRINGDOMAINS• GDB_SUBTYPES• GDB_TABLES_LAST_MODIFIED• GDB_TOOLBOXES• GDB_TOPOCLASSES• GDB_TOPOLOGIES• GDB_TOPORULES• GDB_USERMETADATA• GDB_VALIDRULES• COLUMN_REGISTRY• COMPRESS_LOG• DBTUNE• GCDRULES• GEOMETRY_COLUMNS• LAYERS• LAYER_LOCKS• LINEAGES_MODIFIED• LOCATORS• METADATA• MVTABLES_MODIFIED• OBJECT_LOCKS• PROCESS_INFORMATION• RASTER_COLUMNS• SDE_EXCEPTIONS• SDE_LOGFILES• SDE_LOGFILE_DATA• SDE_LOGFILE_POOL• SDE_TABLES_MODIFIED• SDE_XML_COLUMNS• SDE_XML_INDEXES• SDE_XML_INDEX_TAGS• SDE_XML_TAGS• SERVER_CONFIG• SPATIAL_REFERENCES• STATES• STATE_LINEAGES• STATE_LOCKS• TABLE_LOCKS• TABLE_REGISTRY• VERSION• VERSIONS

\arcgis\ArcSDE\Documentation\Admin_Cmd_Ref\Support_files\arcsde_gdb_tables.pdf or http://arcsdeonline.esri.com

Editing the SDE.DBTUNE table• The SDE.DBTUNE table holds the storageparameters used by ArcSDE to create DBMStables and indexes.– ArcSDE appends DBTUNE storage parameter tothe CREATE TABLE and CREATE INDEX SQLstatements• Edit a file (SDEHOME/etc/dbtune.sde) bydefault.• Use sdedbtune import operation to updatethe SDE.DBTUNE table.

Primary Administrative Tables

Vector Data• Vector data stored as feature classes• Geodatabase container for vector data– Includes geometry and attributes– All geometry is of the same type• Logically one table with spatial column– Physically stored as many tablesPointLineDimensionPolygonAnnotation

Storage model overview• ArcSDE feature class comprisedof 3 tables + 7 indexes– Business table (2)– Feature table (3)– Spatial index table (2)22 221115SP_FID GX1 15316GY21……OBJECTID SHAPE …1212……33…FID123…………POINTSxy,xy,xyxy,xy,xyxy,xy,xy122216 211515212216 21…………SpatialcolumnCoordinateInformation2316 2215 21……

Spatial index• Spatial index is used to improve performance ofretrieving data• IBM DB2, Oracle, and SQL Server use grid tiles– Goal• Optimize the size of the grid tiles– Balance• Number of features referenced by each tile (features per tiles)• Number of tiles referencing each feature (tiles per feature)• Informix and Oracle Spatial use R-tree Rindexing– R-tree index constructs the index using the featureenvelope• Do not need to calculate any parameters for building R-tree R(built automatically)

Storage model overview• Versioned data gains two additional tables:– Adds table– Deletes table• Allows for the isolation of edits• Compress migrates rows to base table– Depends on work flowSFBusinessD91A91A

Feature class storageVECTOR DATAArcSDE BinaryDatatypeSpatialDatatype andFunctionsOracle 8i/9i/10g (BLOB, LONG RAW)SQL Server 7/2000 (IMAGE)Oracle 8i/9i/10g (SDO_GEOMETRYand ST_GEOMETRY at 9.2)DB2 UDB (ST_GEOMETRY)Informix (ST_GEOMETRY)* ST_GEOMETRY – SDE shapeAvailability of storage options depends on yourRDBMS

ArcSDE and Spatial column typesLAKESBusiness tableSHAPE NAME CLASSLakeSuperior12 Geometry stored in business table No feature table is created (no F# and S# tables present) Spatial indexes created by the RDBMS Spatial queries and analysis can be performed with APIor spatial SQL functionsSELECT count(*)FROM buildings, bridgesWHERE Contains(Buffer(bridges.position, 15.0),buildings.footprint) = 1;

Spatial Datatype Issues• Single version has single table schema design– Can simplify views• User defined datatypes and advanced databaseconfigurations– ST_Geometry (BLOB datatypes) ) more widely supported– SDO_Geometry – Composite datatype - be careful with moreadvanced database architecture configurations – see databasevendor documentation.• Example – SDO_GEOMETRY and 10g r2 Data Guard LogicalStandby configuration• Can provide spatial join functionality

Spatial Datatype Query examplespatial join – DB2 ST_GEOMETRY data type• create view spatialjoin as select SDATTRACTIONS.SHAPE,SDATTRACTIONS."NAME", SDFREEWAYS.STREET_NAM fromSDATTRACTIONS, SDFREEWAYS wheredb2gse.st_distance(sdattractions.shape,sdfreeways.shape)

Raster Data• Raster data stored as raster datasets• Geodatabase container for raster data– Stores all pixel values• Logically one table with spatial and raster column– Physically stored as many tables

Raster Data SchemaNew at9.0

ArcSDE Raster DatasetRaster tile• Divide raster into tiles• Each tile stores pixel datafor a band• Tile size defined by user– Specified in pixels– Cannot alter after loading• Data stored in theSDE_BLK table as a BLOB• Sizing Guideline: 128x128• Smaller tiles result insmaller BLOB fields andmore rows in the SDE_BLKtablePixel

General Storage Options• Raster Dataset– Individual set of pixels• 1 or more bands covering the exact same area• Fundamental storage type– Use as separate datasets or…– Mosaic 2 or more inputs into one seamless image• Raster/Image Catalog– Collection of Raster Datasets organized with a table• Each row represents a different raster dataset– Named “raster catalog items”– Individuals are stored separately, but presented as asingle entity– Use entire collection or use a layer definition for sub-setting

• Alternatives:General Storage Options(continued)– Raster Catalog (Hybrid)• A raster catalog is created and inputs are mosaicked together– Each row of the raster catalog is mosaicked into• Each new large raster dataset becomes one item of the rastercatalog– Results in fewer rows within the raster catalog table– Raster Datasets as Attributes of Features• “Pictures of features” stored with the feature• i.e. Snapshot of a home, associated with a land parcel polygon• i.e. Scanned pump diagram, associated with a well point

Raster Catalogs – To manage any series ofimages in a relational databaseCatalog of Many ImagesIDShapeImageDateAbstract2101 11.jp2 12-1212-98Stuff12.jp22102 12.jp2 02-1212-02More stuff2103 13.jp2 01-1313-95Etc.21.jp222.jp22201 21.jp2 03-0404-98Etc.11.jp233.jp22202 22.jp2 06-1515-04Hot stuff. . . . . . . . . . . . . . .

Raster Catalogs – To manage raster time seriesTime SeriesIDShapeImageTimeAbstract2101 11.jp2 11:00 Stuff2102 12.jp2 12:00 More stuff2103 13.jp2 13:00 Etc.2201 21.jp2 14:00 Etc.2202 22.jp2 15:00 Hot stuff. . . . . . . . . . . . . . .

Raster Columns in TablesTo hold pictures or scans as attribute informationIDShapePINAssessedValueStructure2101 12-101101 1500002102 12-102102 1750002103 13-111111 1460002201 21-112112 1730002202 22.122 156000. . . . . . . . . . . . . . .

When Do I Use Each One?• Raster Dataset– Continuous analysis and clipping (one large raster)– Retaining all areas of overlap is not important– Retaining metadata of each input is not important– Fast viewing at any scale• Raster Catalog– Overlapping areas of individual inputs are important– Metadata of individual inputs is important• Query on attributes/metadata (i.e. % cloud cover)– Simply want to keep/store individual images• Raster Catalog Hybrid– Want the best of both RD and RC in ArcGIS– Need to load data to ArcSDE very quickly• Rasters as Attributes– Picture of the real-world feature offers value

Compressions DescribedLoss-lessLossyType: • LZ77 • JPEG (quality 5 – 95)• JPEG 2000 (quality 1- 255)Applies to:Amount ofcompression:Best for:• All data types• Moderate (2:1, 3:1)• Higher for morehomogenous data• Further data processing• Data with sharp edges• JPEG – 8 bit unsigned• JP2 – 8 + 16 bit unsigned• High (5:1, 15:1)• User controlscompression / quality ratio• Data only used as backdrop• Smooth, continuous imagesExamples:• Analytical surfaces/DEMs• Analytical satellite /aerial images• Scanned linework andcolormapped images• Digital photos• Imagery for base-mapping/cartographic output• Scanned true-color maps

Compression Recommendations• Always use compression for best displayspeeds– Data transmission over the network is expensive– Un-compressing raster data happens on client and isfast• Load uncompressed or loss-less less compressedsource data, if available– Reduces excessive image degradation– Data is uncompressed/re-compressed as it getsloaded into ArcSDE

Compression Recommendations(continued)• Serve ArcSDE raster data via ArcIMS– To avoid too much loss (prototype your solution)– If you store lossy, , serve image with loss-lesslesscompression• i.e. PNG– If you store loss-less, less, serve either lossy or loss-lessless• When loading lossy-compressed files such asJPEG, MrSID, , ECW, JPEG2000– Load and view sample, then verify visual quality– May require use of loss-less less compression in ArcSDE• Lossy compression may affect scale of use– Due to degradation of raster quality

Scale of Use vs Lossy CompressionMAPSCALE1 : 10001 : 50001 : 120001 : 240001 : 633601 : 1500001 : 250000more loss… JPEG Compression …less loss6 15 25 50 75 94

Pyramids• Reduced resolution copies of original raster• Improves display performance– Clients only request requiredresolution• Stored re-sampling vs.display re-sampling• Set at creation time– Adds about 33% more storage• NOTE: Pyramids are notused during analysis

Pixel Resampling Methods• Used in both pyramid creation and display resampling• Interpolates pixel values from original raster valuesForDiscreteData Nearest neighbor• Value of nearest pixel• Does not create new values• Fastest overallForContinuousData Bilinear interpolation• Weighted average of fournearest pixels• Faster than cubic convolution• Smooths most imagery Cubic convolution• Sixteen nearest pixels• Usually most pleasing result• Sharpens most imagery

Pyramid Enhancements• Raster Dataset Partial Pyramid Updating– Pyramids are updated for only the area that is updated• IF the area updated is to the south and west of the pyramidreference point• By default, this is the top-left hand corner of the imageRebuild all pyramidsRebuild all pyramidsUpdates in this areacause pyramids to bebuilt for this area only

ArcSDE Logfiles• Logfile maintain sets of selected records– Object IDs• Used by ArcGIS– Selections greater than selection threshold• ArcIMS• Default is 100• http://support.esri.com/index.cfm?fasupport.esri.com/index.cfm?fa=knowledgebase.techarticles.articleShow&d=22668– Selection operations• ArcSDE– Versioning

Shared• Default architecture• Each user owns two tables– SDE_logfiles• Logfile table name and the logfile ID– SDE_logfile_data• OBJECTID of selected records• Created when user’s s first selection exceedsthreshold– User requires CREATE TABLE permission– Revoke permission once the tables are created– If insufficient permissions, selection cannot be made• Logfiles no longer checked on connection

Session1. Pool of sde-owned logfile tables– Shared among users– Connection uses first available logfiletable– Number of logfile tables in pool isconfigurable2. Temporary session logfile table createdper connectionuse tempdbgoselect * from information_schema.tables• Recommended if many gsrvrs start under sameuser account (e.g.,ArcIMS(e.g.,ArcIMS)

Standalone1. Created as temporary tables– Number of stand-alone alone logfile tables per connectionis configurableuse tempdbgoselect * from information_schema.tables– One column used for ObjectIds of selected layer2. Uses pool of sde-owned logfile tables– Shared among users– One pool table per layer selection

Configuring• Configuration stored in sde.SDE_server_configtable• Change with sdeconfig command• ALLOWSESSIONLOGFILE• TRUE gsrvr CAN create session logfile• FALSE gsrvr CANNOT create session logfile (usesstandalone or user logfiles)• SESSIONLOGPOOLSIZE• Number of sde-owned session logfiles• MAXSTANDALONELOGS• Maximum number of standalone logs per connection• HOLDLOGPOOLTABLES• TRUE gsrvr will hold table until disconnect• FALSE gsrvr releases table if no current selection

Default settings• SDE_server_config table– ALLOWSESSIONLOGFILE = FALSE– SESSIONLOGPOOLSIZE = 0– MAXSTANDALONELOGS = 0– HOLDLOGPOOLTABLES = TRUE• SDE_DBTUNE table– LOGFILE_DEFAULTS SESSION_TEMP_TABLE = 1 (TRUE)• No session logs• No standalone logs• Users must have create table permission

• For OracleRecommendations– Use shared logfiles (default)• For SQL Server– Use session logfiles in tempdb• Set ALLOWSESSIONLOGFILE = TRUE• Do not configure a logfile pool– Advantages• No increased transaction log load• Transactions logged more efficiently in tempdb– Not required for recovery• No additional tables to manage• All temporary tables deleted on disconnect– Watch for hot tempdb

What is Versioning?• Provides long term transaction editing• An alternative view of the database that has:– an owner– a description– a permission– a parent version• Versions are not affected by changes occurring inother versions of the database• Version:– A window into a database– A view into a database• Foundation for– Disconnected Editing (8.3 – 9.x)– Geodatabase Replication (9.2)Geodatabase

Versioning requirements• Multiuser geodatabases only– Not available with single user geodatabases• Need ArcEditor or ArcInfo license to edit data– ArcView license can only view data• Need ArcSDE and RDBMS softwareArcObjectsArcSDEGeodatabaseRDBMS

Versioned geodatabase• Method for tracking and presenting changes– Accessed through versions• Multiple, alternate versions may coexist– Appears to users as if they have their own copy ofthe dataUSER1 versionOriginal versionUSER2 version

What versioning offers• Long transactions– Isolate work across multiple sessions• Save, reconnect, continue edits• Example – Edits do not impact others• Model what-if scenarios– Simulate situations with versions• Example – Disaster event planning• Workflow management– Create versions for stages• Example – Land development• HistoryPublicAnalystPublicFloodFarmland Residential– Save snapshot of database state• Example – Wildlife habitat monitoring Year 1 Year 2

Versions and States• Short Term vs. Long Term Transaction– RDBMS – Short Term – SCN, etc...– ArcSDE/Versioning– Long Term – State ID• Version:– Isolated view of data– A window into a database– A view into a database– Explicit (“named(named”) ) vs. Implicit (edit session)• State:– Increment of work– An edit or unit of change– Multiple states make up a version– States are recorded in the A and D tables

Versioning Architecture andWorkflow are Critical• Versioning supports a variety of workflows– Workflow will vary by organization• Maintain separate stages of a project• Comply with business procedures• Quality Assurance (QA)– Ensure timely and accurate database changes• Security – version permissions– Preserve geodatabase integrity– Vulnerability of versions to inadvertent modifications• Maintain geodatabase performanceVersioningworkflowBusinessworkflow

Integration and Performance• ArcSDE RI• ArcSDE Views• Moving Data– XML Recordset and Workspace• Performance

ArcSDE Schema and RIGeodatabase, Behaviors and Referential Integrity• ArcSDE schema “living” with or in DW schemas– You must KNOW everything about your ArcSDE and Data schemas.– “lives” alongside, integrated or “connected” into the overall Data Warehouseschemas and structures, embedded.– Need to be careful of the complex relationships and interdependencies ncies withinthe ArcSDE/GeoDatabase schema objects.• Referential Integrity and PK-FK– ArcObjects, ArcSDE and RDBMS are maintaining various pieces dependingupon the functionality (versioning, relationship classes, feature e linkedannotation, etc…) ) implemented.• Entity Integrity - define a row as a unique entity• Domain Integrity - define validity of values for columns• Referential Integrity - ensure key values are consistent across tables• User-Defined Integrity - enforce other business rules– Understand the schema and what is implemented, be careful and test.– Some customizations will work but may not necessarily be formally y supported.

Geodatabase Constructs• Consider GDB constructs when implementingbusiness logic.• Business rules modeled as geodatabase behavior• Rules stored in the database, enforced by ArcGIS– Domains enforce database integrity– Relationships enforce referential integrity– Subtypes enforce user-defined integrityRDBMSCHECK constraintPrimary keyForeign keyUser-defined procedureTriggerGeodatabaseDomain/SubtypeRelationship classSubtype, topology, geometricnetworkRelationship class withreferential integrity

Views: Types of joins in ArcGIS• ArcSDE– Spatial and non-spatial viewsdetable –o create_view vs.sdetable –o create_mv_view• ArcMap– Join– Relate– Relationship class (editing)• ArcIMS– and ArcXML tags

Views: Functional comparisonHeterogeneousformatsMultiversionedCardinality1:1 1:M M:1N:MArcMaprelate:ArcMapjoin:Relationshipclass:ArcXML:X X X XX X XX X XX X X *X XXXX X *ArcSDEview:X X*X X** Not supported in ArcMap

ArcSDE spatial views• Comprised of three views• Registered withArcSDE repository– Feature class properties inLAYERS / SPATIAL_REFERENCES– Dependent objects flagged inTABLE_REGISTRYsdetable –o o describe_reg ^–t t road_view ^–u u wilson ^–p p wilson• Considerations:– Read-only– Only use with single-versionedtables• Knowledge Base: 24439also in sdehelp.chmROAD_VIEWROAD_VIEWF12S12ROADSROADSF1S1

Different “windows” to datatypes• ArcSDE API• MDAC(Microsoft Data AccessComponents)– ODBC– OLE/DB– ADO• ArcGIS/ArcObjects• RDBMS– Between versions– Between databases (gateways other than MDAC)• ArcSDE 8.3 vs. ArcSDE 9.0 (rareto 9i upgrade)• You will need to test(rare - known issue in Oracle 8i

ArcObjects Datatypes: ArcGIS to RDBMSArcGISShort Integer(-32K to +32k)Long Integer(-2B to +2B)Float(-3.4E38 to 1.2E38)Double(-2.2E308 to 1.8E308)Text(up to 64,000 characters)Date(will vary by RDBMS)BLOB (varies)GUID (36 charactersenclosed in curly brackets)ObjectIDShape Field in BusinessTableAccessnumber(integer)number (longinteger)number(single)number(double)textdate/timeole objectnumberautonumber(long)ole objectSQL Serversmallintintnumericnumericvarchar(up to 8,000 characters)datetimeimageuniqueidentifierInt(created by storedprocedure and i table)int (sde binary)Also see arcgisdeveloperonline.esri.com (Ifield.Type property)and ArcGIS 9.0 Desktop Help for limitsOraclenumbernumbernumbernumbervarchar2datelong raw or blobcharnumber(created by anOracleSequence)number orsdo_geometryDB2smallintintegerdecimaldecimalvarchartimestampblobcharacterInteger(created fromsde_object_idstable)DB2GSE.st_datatype

Moving or Exporting Data• Copy and Paste• XML Workspace and Recordset• Personal Geodatabase• Watch out for existing objects with same name…

Performance Factorsmore at tommorrow’s session: 10:30-12:0012:00• Enterprise System Architecture• Hardware configuration and sizing• Network• DBMS Configuration• ArcSDE Configuration• Type and complexity of the data and the datamodel• Client applications Customization• Workflows• Volume and nature of operations

Future - The Road Ahead“good reasons to get started now in the GeoDatabase”• Geodatabase Replication8.3/9.0/9.1 – Single Generation, 9.2 – Multiple Generation• History based on ArchivingArchive Version• Support for multiple transaction models“non-versioned” editing – 3 models: Versioned, Non-Versioned, Hybrid• File GeodatabasePersonal Geodatabase solution for UNIX/LINUX• Interoperabilityexport to many formats including GML• RDBMS projectsPostgreSQL, , DBMS Simple Feature, Unicode/ANSI, IPV6• Integrated ArcGIS / ArcSde / RDBMS runtime offeringArcGIS “GeoDatabase Manager” – embedded SDE/RDBMS runtime• ArcGIS Server IntegrationPublish Geodatabases and Datasets to catalog service

Conclusion• Make Geodatabase part of GIS datamanagement – personal or enterprise or both• Remember ArcSDE is a “phone line” to thedatabase and all user actions are generatingSQL statements going across that “phone line” –reducing the activity will improve performanceand scalability (e.g. scale dependant layers,etc…)• Thank you…