Integrating ArcGIS desktop in data interoperability environments with ...
Hochschule für Technik Stuttgart
Data-Interoperability based on Common Databases
Integrating ArcGIS desktop in data interoperability environments with heterogeneous GIS clients
M.Sc Abdurasyid Abd id Moestofa M f / Prof. Rainer Kettemann Stuttgart University of Applied Sciences Schellingstraße 24 70174 Stuttgart Germany [email protected]
[email protected]
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
1
Hochschule für Technik Stuttgart
Stuttgart University of Applied Sciences
Stuttgart University of Applied Sciences 11 bachelor and 13 master programs Photogrammetry and Geoinformatics and 3 more GIS related courses
GIS Laboratory with current focus on data interoperability p Host of an annual ESRI User Group Meeting in the State Baden-Württemberg ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
2
Hochschule für Technik Stuttgart
Data-Interoperability based on Common Databases
Content Types of and reasons for data interoperability
Data interoperability based on common data stores
Experiences with ArcGIS 9.3.1 and other GIS in using ORACLE 9g as a common database
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
3
Hochschule für Technik Stuttgart
Data - Interoperability
Data interoperability based on common (open) data formats in DBMS, DBMS e e.g. g Oracle SDO SDO_Geometry Geometry
Data interoperability based on OGC Web Services, Web Map Service (WMS) and Web Feature Service (WFS) Presentation “Replacing Replacing Local Data by Web Services Using ArcGIS Desktop Desktop“, Biniam Neguse, Wednesday Jul 14, 2010, 1:30 – 2:45, Room 28 D
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
4
Hochschule für Technik Stuttgart
Spatial Data Infrastructure (SDI) Scenarios
Regional separation –
interoperability in hierarchical structures
European Union Germany states A
B
Independent units in federal organizations Equivalent tasks solved with independent
C
D
regions / counties / municipalities
GIS solutions Interoperability solution: OGC Web Services in a Wide Area Network
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
5
Hochschule für Technik Stuttgart
Spatial Data Infrastructure (SDI) Scenarios
Task based separation –
interoperability in coequal structures
Independent organizations ministries / departments
Department ABC D
Same spatial responsibility
Individual tasks solved with independent GIS solutions
Interoperability p y solution: Common data store in a Local Area Network
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
6
Hochschule für Technik Stuttgart
Data classification and connection types
A project jproject td data t data project data for others for others f others for th others fromfrom others from others
data for others is a subset of the own project data
direct connection
basic b ibasic d data t data basic data
B project p j data project j data d project data for others for others for others from others from others from others basic data b i d basic data t basic data
Internet
Basic data is the common spatial reference
OGC Service
data from others is required to solve own tasks k
state maps / cadastral data ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
7
Hochschule für Technik Stuttgart
Data classification and connection types
A project j t data d t
read / write connection to maintain the data
for others from others b i d basic data t
read only connections data is maintained by others
Usually there is no need to maintain (read / write) data with heterogeneous clients
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
8
Hochschule für Technik Stuttgart
Architecture in General
GIS clients A
GIS clients B
GIS clients B
middleware
middleware
middleware
interface for geodata RDBMS with data and metadata
client related metadata
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
9
Hochschule für Technik Stuttgart
Architecture of the test environment
ArcSDE
Version 10
Version 6.1
Version 9.3.1
Data Server
FDO
Middleware tasks • Connect to database • Create Spatial tables • Register Spatial Data • Works like MDSYS Metadata
SDO Geometry SDO_Geometry
Restriction: Onlyy works with its application pp
Version 11g MDSYS.SDO_GEOMETRY MDSYS SDO GEOMETRY USER_SDO_GEOM_METADATA
client related metadata
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
10
Hochschule für Technik Stuttgart
Data and Meta Data in ORACLE
SSpatial i l Table T bl ID………….Number Attributes…. String, Integer, etc. Geometry … SDO SDO_GEOMETRY GEOMETRY with element info and coordinates section
Oracle O l M Metadata d Table T bl • Table Name • Geometry column Name • Coordinate Reference System
Application A li i Middleware Middl Schema S h • Table Name • Geometry type • Coordinate Reference System • Owner of the table • Geometry column name
Oracle Database
ArcGIS 9.3.1 offers • auto registration to create its own meta data section for all data in the RDBMS. • manuall registration i t ti ffor selected l t dF Feature t Classes Cl ArcGIS reads the information from the first feature
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
11
Hochschule für Technik Stuttgart
Created by
Interoperability Test RReadable d bl with
ArcGIS Desktop 9.3.1 (2D geometry only, no g geometry y collection))
GeoMedia Professional 6.1 (3D geometry only, geometry collections)
AutoCAD Map 3D 2010
ArcGIS Desktop 9.3.1
GeoMedia Professional 6.1
AutoCAD Map 3D 2010
uDig 1.1.1
Yes Yes
Yes
Yes
(Only Polygon, Line, and Point)
Yes No
Yes
Yes
Yes
(Only Polygon Polygon, Line, and Point)
Yes
(2D and 3D geometry, geometry g y collections))
(2D only, no geometry collections)
Yes
Yes
(Only Polygon, Line, and Point)
uDig 1.1.1
-
-
-
-
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
12
Hochschule für Technik Stuttgart
Limitation and Anomaly
Limitation with 3D data
RRegistered it d with ArcSDE
Result in ArcGIS 9.3.1
GeoMedia Professional Always stores spatial data in 3D mode with z=0
Dimension info value is ignored. All data is read as 2D Geometry y So the original z=0 is interpreted as X=0 or Y=0
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
13
Hochschule für Technik Stuttgart
Applicable solution
W kfl Workflow for f read/write d/ it data d t interoperability i t bilit 1. Creating all common feature classes using ArcGIS
2. Registering g g the Feature Classes with other systems y here: AutoCAD, GeoMedia, and uDig
3. Populate and modify the existing feature classes with any system AutoCAD still allows to store multiple geometry types within all Feature Class. So it’s users have to be careful. GeoMedia recognizes the geometry type set by ArcGIS and allows only to populate with features of this geometry type ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
14
Hochschule für Technik Stuttgart
Applicable solution
Results in using a common data base in read/write mode 1. It is possible to maintain data in Oracle using multiple client software if all users take care of agreed geometry types and coordinate dimensions
2 Oracle always enables a mix of geometry types and 2. coordinate dimensions in every feature class
3. Systems with restrictive own meta data like ArcGIS or GeoMedia ensure fixed geometry types in a feature class
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
15
Hochschule für Technik Stuttgart
Access tests with real world datasets
S = mall dataset L = large dataset shown here
Building First pulse polygons lidar
Last pulse lidar
S
1 412
116 034
356 665
L
14 975
4 943 011
22 483 498
Without Spatial Filter Small dataset
Large Dataset
AutoCAD Map 3D 2010
4m 8s
Failed
ArcGIS Desktop 9.3.1
1m 15s
24m 45s
GeoMedia Pro. 6.1
1m 5s
Failed
uDig
2m 21s
Failed
Lidar data was used as point features to have large datasets ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
16
Hochschule für Technik Stuttgart
Access tests with real world datasets
S = mall dataset L = large dataset shown here
Building First pulse polygons lidar
Last pulse lidar
S
1 412
116 034
356 665
L
14 975
4 943 011
22 483 498
With Spatial Filter 1*1 km² Small dataset
Large Dataset
AutoCAD Map 3D 2010
2m 16s
4m 3s
ArcGIS Desktop 9.3.1
14s
56s
GeoMedia Pro. 6.1
30s
2m 36s
Lidar data was used as point features to have large datasets ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
17
Hochschule für Technik Stuttgart
Access tests with real world datasets
D t sets Data t
Building B ildi polygons
Firstt pulse Fi l lidar
Lastt pulse L l lidar
Small: 1 km² test data
1 412
116 034
356 665
Large: Municipality WN
14 975
4 943 011
22 483 498
Lidar data was used as point features to have large datasets Without Spatial Filter
With Spatial Filter 1 * 1 km²
Small dataset
Large Dataset
Small Dataset
Large Dataset
AutoCAD Map 3D 2010
4m 8s
Failed
2m 16s
4m 3s
ArcGIS A GIS Desktop 9.3.1
1 15 1m 15s
24 45 24m 45s
14 14s
56 56s
GeoMedia Pro. 61 6.1
1m 5s
Failed
30s
2m 36s
uDig
2m 21s
Failed
-
-
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
18
Hochschule für Technik Stuttgart
Processing time on spatial data
The task: Calculate the height of the buildings using their footprints and lidar preclassified data (first pulse = on roof or vegetation, last pulse = on ground) Top p
= mean of all first p pulse p points within a building (spatial join) Ground = mean of all ground points within a 3 m buffer around each building (spatial join) Height = Top - Ground
Loaded data from large dataset Buildings: 1 412 records, First Pulse 152 512 records, Last Pulse 342 304 records
The intermediate table applies spatial filtering before joining ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
19
Hochschule für Technik Stuttgart
Conclusions 1
+ It is possible to share data by using Oracle’s SDO GEOMETRY based on OGC standards SDO_GEOMETRY
+ Using a common database avoids copying data
+ Data is always up to date
+ The responsibility for all data is always by the owner
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
20
Hochschule für Technik Stuttgart
o
o
o
o
Conclusions 2
Users have to agree to a common structure of the data (community structure) The common structure should be b based d on the th possibilities ibiliti off the th system with most restrictions (e.g. ArcGIS) Avoid joins – necessary attributes should be stored together with the geometry to benefit from the spatial index Comparing the results with former ones shows, that al vendors are on a good way to support data interoperability
+ ArcGIS 10 will allow to read geometry data from standard RDBMS without ArcSDE ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
21
Hochschule für Technik Stuttgart
Further research
1. Extend the research to other RDBMS supporting spatial information 2. Testing the new GIS versions, especially ArcGIS10 3 S 3. Set up an environment i for f students d to get experience with data sharing based on common data bases from various vendors
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
22
Data-Interoperability based on Common Databases
Integrating ArcGIS desktop in data interoperability environments with heterogeneous GIS clients
M.Sc Abdurasyid Abd id Moestofa M f / Prof. Rainer Kettemann Stuttgart University of Applied Sciences Schellingstraße 24 70174 Stuttgart Germany [email protected]
[email protected]
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
1
Hochschule für Technik Stuttgart
Stuttgart University of Applied Sciences
Stuttgart University of Applied Sciences 11 bachelor and 13 master programs Photogrammetry and Geoinformatics and 3 more GIS related courses
GIS Laboratory with current focus on data interoperability p Host of an annual ESRI User Group Meeting in the State Baden-Württemberg ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
2
Hochschule für Technik Stuttgart
Data-Interoperability based on Common Databases
Content Types of and reasons for data interoperability
Data interoperability based on common data stores
Experiences with ArcGIS 9.3.1 and other GIS in using ORACLE 9g as a common database
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
3
Hochschule für Technik Stuttgart
Data - Interoperability
Data interoperability based on common (open) data formats in DBMS, DBMS e e.g. g Oracle SDO SDO_Geometry Geometry
Data interoperability based on OGC Web Services, Web Map Service (WMS) and Web Feature Service (WFS) Presentation “Replacing Replacing Local Data by Web Services Using ArcGIS Desktop Desktop“, Biniam Neguse, Wednesday Jul 14, 2010, 1:30 – 2:45, Room 28 D
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
4
Hochschule für Technik Stuttgart
Spatial Data Infrastructure (SDI) Scenarios
Regional separation –
interoperability in hierarchical structures
European Union Germany states A
B
Independent units in federal organizations Equivalent tasks solved with independent
C
D
regions / counties / municipalities
GIS solutions Interoperability solution: OGC Web Services in a Wide Area Network
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
5
Hochschule für Technik Stuttgart
Spatial Data Infrastructure (SDI) Scenarios
Task based separation –
interoperability in coequal structures
Independent organizations ministries / departments
Department ABC D
Same spatial responsibility
Individual tasks solved with independent GIS solutions
Interoperability p y solution: Common data store in a Local Area Network
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
6
Hochschule für Technik Stuttgart
Data classification and connection types
A project jproject td data t data project data for others for others f others for th others fromfrom others from others
data for others is a subset of the own project data
direct connection
basic b ibasic d data t data basic data
B project p j data project j data d project data for others for others for others from others from others from others basic data b i d basic data t basic data
Internet
Basic data is the common spatial reference
OGC Service
data from others is required to solve own tasks k
state maps / cadastral data ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
7
Hochschule für Technik Stuttgart
Data classification and connection types
A project j t data d t
read / write connection to maintain the data
for others from others b i d basic data t
read only connections data is maintained by others
Usually there is no need to maintain (read / write) data with heterogeneous clients
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
8
Hochschule für Technik Stuttgart
Architecture in General
GIS clients A
GIS clients B
GIS clients B
middleware
middleware
middleware
interface for geodata RDBMS with data and metadata
client related metadata
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
9
Hochschule für Technik Stuttgart
Architecture of the test environment
ArcSDE
Version 10
Version 6.1
Version 9.3.1
Data Server
FDO
Middleware tasks • Connect to database • Create Spatial tables • Register Spatial Data • Works like MDSYS Metadata
SDO Geometry SDO_Geometry
Restriction: Onlyy works with its application pp
Version 11g MDSYS.SDO_GEOMETRY MDSYS SDO GEOMETRY USER_SDO_GEOM_METADATA
client related metadata
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
10
Hochschule für Technik Stuttgart
Data and Meta Data in ORACLE
SSpatial i l Table T bl ID………….Number Attributes…. String, Integer, etc. Geometry … SDO SDO_GEOMETRY GEOMETRY with element info and coordinates section
Oracle O l M Metadata d Table T bl • Table Name • Geometry column Name • Coordinate Reference System
Application A li i Middleware Middl Schema S h • Table Name • Geometry type • Coordinate Reference System • Owner of the table • Geometry column name
Oracle Database
ArcGIS 9.3.1 offers • auto registration to create its own meta data section for all data in the RDBMS. • manuall registration i t ti ffor selected l t dF Feature t Classes Cl ArcGIS reads the information from the first feature
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
11
Hochschule für Technik Stuttgart
Created by
Interoperability Test RReadable d bl with
ArcGIS Desktop 9.3.1 (2D geometry only, no g geometry y collection))
GeoMedia Professional 6.1 (3D geometry only, geometry collections)
AutoCAD Map 3D 2010
ArcGIS Desktop 9.3.1
GeoMedia Professional 6.1
AutoCAD Map 3D 2010
uDig 1.1.1
Yes Yes
Yes
Yes
(Only Polygon, Line, and Point)
Yes No
Yes
Yes
Yes
(Only Polygon Polygon, Line, and Point)
Yes
(2D and 3D geometry, geometry g y collections))
(2D only, no geometry collections)
Yes
Yes
(Only Polygon, Line, and Point)
uDig 1.1.1
-
-
-
-
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
12
Hochschule für Technik Stuttgart
Limitation and Anomaly
Limitation with 3D data
RRegistered it d with ArcSDE
Result in ArcGIS 9.3.1
GeoMedia Professional Always stores spatial data in 3D mode with z=0
Dimension info value is ignored. All data is read as 2D Geometry y So the original z=0 is interpreted as X=0 or Y=0
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
13
Hochschule für Technik Stuttgart
Applicable solution
W kfl Workflow for f read/write d/ it data d t interoperability i t bilit 1. Creating all common feature classes using ArcGIS
2. Registering g g the Feature Classes with other systems y here: AutoCAD, GeoMedia, and uDig
3. Populate and modify the existing feature classes with any system AutoCAD still allows to store multiple geometry types within all Feature Class. So it’s users have to be careful. GeoMedia recognizes the geometry type set by ArcGIS and allows only to populate with features of this geometry type ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
14
Hochschule für Technik Stuttgart
Applicable solution
Results in using a common data base in read/write mode 1. It is possible to maintain data in Oracle using multiple client software if all users take care of agreed geometry types and coordinate dimensions
2 Oracle always enables a mix of geometry types and 2. coordinate dimensions in every feature class
3. Systems with restrictive own meta data like ArcGIS or GeoMedia ensure fixed geometry types in a feature class
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
15
Hochschule für Technik Stuttgart
Access tests with real world datasets
S = mall dataset L = large dataset shown here
Building First pulse polygons lidar
Last pulse lidar
S
1 412
116 034
356 665
L
14 975
4 943 011
22 483 498
Without Spatial Filter Small dataset
Large Dataset
AutoCAD Map 3D 2010
4m 8s
Failed
ArcGIS Desktop 9.3.1
1m 15s
24m 45s
GeoMedia Pro. 6.1
1m 5s
Failed
uDig
2m 21s
Failed
Lidar data was used as point features to have large datasets ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
16
Hochschule für Technik Stuttgart
Access tests with real world datasets
S = mall dataset L = large dataset shown here
Building First pulse polygons lidar
Last pulse lidar
S
1 412
116 034
356 665
L
14 975
4 943 011
22 483 498
With Spatial Filter 1*1 km² Small dataset
Large Dataset
AutoCAD Map 3D 2010
2m 16s
4m 3s
ArcGIS Desktop 9.3.1
14s
56s
GeoMedia Pro. 6.1
30s
2m 36s
Lidar data was used as point features to have large datasets ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
17
Hochschule für Technik Stuttgart
Access tests with real world datasets
D t sets Data t
Building B ildi polygons
Firstt pulse Fi l lidar
Lastt pulse L l lidar
Small: 1 km² test data
1 412
116 034
356 665
Large: Municipality WN
14 975
4 943 011
22 483 498
Lidar data was used as point features to have large datasets Without Spatial Filter
With Spatial Filter 1 * 1 km²
Small dataset
Large Dataset
Small Dataset
Large Dataset
AutoCAD Map 3D 2010
4m 8s
Failed
2m 16s
4m 3s
ArcGIS A GIS Desktop 9.3.1
1 15 1m 15s
24 45 24m 45s
14 14s
56 56s
GeoMedia Pro. 61 6.1
1m 5s
Failed
30s
2m 36s
uDig
2m 21s
Failed
-
-
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
18
Hochschule für Technik Stuttgart
Processing time on spatial data
The task: Calculate the height of the buildings using their footprints and lidar preclassified data (first pulse = on roof or vegetation, last pulse = on ground) Top p
= mean of all first p pulse p points within a building (spatial join) Ground = mean of all ground points within a 3 m buffer around each building (spatial join) Height = Top - Ground
Loaded data from large dataset Buildings: 1 412 records, First Pulse 152 512 records, Last Pulse 342 304 records
The intermediate table applies spatial filtering before joining ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
19
Hochschule für Technik Stuttgart
Conclusions 1
+ It is possible to share data by using Oracle’s SDO GEOMETRY based on OGC standards SDO_GEOMETRY
+ Using a common database avoids copying data
+ Data is always up to date
+ The responsibility for all data is always by the owner
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
20
Hochschule für Technik Stuttgart
o
o
o
o
Conclusions 2
Users have to agree to a common structure of the data (community structure) The common structure should be b based d on the th possibilities ibiliti off the th system with most restrictions (e.g. ArcGIS) Avoid joins – necessary attributes should be stored together with the geometry to benefit from the spatial index Comparing the results with former ones shows, that al vendors are on a good way to support data interoperability
+ ArcGIS 10 will allow to read geometry data from standard RDBMS without ArcSDE ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
21
Hochschule für Technik Stuttgart
Further research
1. Extend the research to other RDBMS supporting spatial information 2. Testing the new GIS versions, especially ArcGIS10 3 S 3. Set up an environment i for f students d to get experience with data sharing based on common data bases from various vendors
ESRI User Conference 2010 - M.Sc Aburashid Moestofa, Prof. Rainer Kettemann
22
