Data Interoperability: An Introduction
2013 Esri International User Conference July 8–12, 2013 | San Diego, California Technical Workshop
Data Interoperability An Introduction Bruce Harold – Esri Dale Lutz – Safe Software
Esri UC2013 . Technical Workshop .
Background Why Data Interoperability?
Esri UC2013 . Technical Workshop .
What is the Data Interoperability Extension? ESRI & SAFE Co-Development based on FME
• Format Support – Adds support for 200+ data formats – Plus Your Custom Formats – Direct use in ArcGIS Desktop
• Quick Data Translation – Data Interoperability Tools Toolbox – Convert between data formats
• Spatial ETL (Extract, Transform, Load) – Schema Mapping (may be Dynamic) – Transformation of Geometry – Logic Controls Data Flow – Per Feature Esri UC2013 . Technical Workshop .
ArcGIS Data Interop Common Tasks • • •
• • •
Data Integration - Bring multiple formats together Data Migration - Process of moving data between systems Data Cleanup - Clean up errors in the data and streamline data preparation Data Distribution - Distribute data to different systems Data Merge - Merge changes from edited data Data Validation - Geometry and Attributes
Esri UC2013 . Technical Workshop .
ETL
Data Interoperability Differentiators Not Just About Schema Manipulation
•
Change Detection
•
3D PDF Writing
•
XML Documents as Features or Attributes
•
JSON Documents as Features or Attributes
•
Protocol Support (HTTP, RSS…)
•
Web and Compressed Inputs
•
Parallel Processing
•
Vertical Coordinate Systems
•
BLOB Attributes
Esri UC2013 . Technical Workshop .
Basics: OGC Support What you get just by installing
Esri UC2013 . Technical Workshop .
OGC GML Simple Feature Format Support Support for GML-SF in ArcGIS Desktop
•
GML – SF – simple features
•
WFS – optionally non-cached for auto refresh
•
WFS – cache timeout
•
Note: Installation of the extension is required
Esri UC2013 . Technical Workshop .
Demo
OGC Formats: GML & WFS Bruce Harold
Esri UC2013 . Technical Workshop .
Demo Summary: GML & WFS •
Comes Free But you have to install - Simple Features Only -
•
Better with the extension licensed -
Non-simple parameters supported
•
Direct Read of GML, consume WFS
•
Important in many jurisdictions
Esri UC2013 . Technical Workshop .
Using Formats Direct Read of any format
Esri UC2013 . Technical Workshop .
Direct Read of Formats •
Formats recognized by Data Interop. File-based, by extension, like .tab, .mid, .skp, .accdb - Visible in Catalog window browse tree - May be extended with Custom Formats -
•
Directly usable in ArcGIS -
•
Save time converting data
Consume external data ArcPad AXF - Other application formats - Web feeds -
Esri UC2013 . Technical Workshop .
Easy Format Selection •
Browse or Search
Esri UC2013 . Technical Workshop .
When to Browse and when to Connect Browse & Direct Read uses format defaults
•
Many formats have optional parameters -
Select subsets, set conditions, passwords, filters…
•
Connections give you access to format parameters
•
Connections let you set Coordinate Systems
Esri UC2013 . Technical Workshop .
Interoperability Connections Persist between application sessions
•
Convenient to access in browse tree
•
Retain your settings
•
Off-disk data -
•
Web Feeds Google Spreadsheets
Share with colleagues -
.FDL file created C:\Users\<user>\AppData\Roaming\Safe Software\Interoperability
Esri UC2013 . Technical Workshop .
Demo
Direct Read Bruce Harold
Esri UC2013 . Technical Workshop .
Demo Summary: Direct Read •
Direct Read Summary -
Directly read, inspect and browse within ArcGIS
-
Works like native ESRI formats (no GDB behavior)
-
Access directly in Desktop Apps, Geoprocessing
-
No development or customization required
Esri UC2013 . Technical Workshop .
Quick Translation Quick Import & Quick Export GP Tools
Esri UC2013 . Technical Workshop .
Quick Data Import/Export Enabling Extension Provides ‘Quick’ Tools •
Quick Export -
•
Quick Import -
•
To any format To staging Geodatabase
Not Configurable -
Is this OK?
Esri UC2013 . Technical Workshop .
Automation and Quick Translations •
Use Quick Tools in ModelBuilder -
Get any format into a temporary File GDB Handle Multiple Formats In/Out
Esri UC2013 . Technical Workshop .
Demo
Quick Tools Bruce Harold
Esri UC2013 . Technical Workshop .
Demo Summary: Quick Import/Export •
“Quick” GP Tools -
Useful in ModelBuilder
-
Also Scriptable in Python
-
Easy interaction with others - Departments - Contractors - 3rd Party Software Users
Esri UC2013 . Technical Workshop .
Leveraging Formats Custom Formats
Esri UC2013 . Technical Workshop .
What if you need your own format? Save time by “canning” a custom format workflow
•
Commonly handled text-based sources -
CSV XML
•
Build format recognition workflow once
•
Re-use the persisted workflow any time
•
Lets you develop a file format for sharing
Esri UC2013 . Technical Workshop .
Demo
Custom Formats Bruce Harold
Esri UC2013 . Technical Workshop .
Demo Summary: Custom Formats •
Based on an existing format
•
May be shared .FDS file in profile directory for each format - C:\Users\<user>\My Documents\FME\Formats -
•
Editable later Get the custom Quake format by downloading the slides!
Esri UC2013 . Technical Workshop .
The Power of Transformation Spatial ETL
Esri UC2013 . Technical Workshop .
The Full Power of Spatial ETL •
Formats are only part of the story
•
Often the format does not change * -
•
Transformation is key -
•
But the schema does Many categories of transformer Feature by Feature or Dataset by Dataset Geometry or Attribute transformations
Many Inputs Many Outputs is OK *like shapefile to shapefile
Esri UC2013 . Technical Workshop .
Workbench Application •
Graphically Design Data Flows
READ
Esri UC2013 . Technical Workshop .
TRANSFORM
WRITE
It’s a Pipeline •
Readers send features into the workspace -
•
Transformers “do something’ -
•
Geometry – X:Y:Z:M Attributes – with a schema May operate “per feature” and pass it immediately May work on sets or all features together Some offer parallel processing
Writers send features to the destination DATA
Esri UC2013 . Technical Workshop .
Some Transformers Are All You Need •
This example derives Adds, Deletes and NoChanges in one step -
Esri Community Maps contributor data is processed this way
Esri UC2013 . Technical Workshop .
The Power of Transformation •
Using Spatial ETL •
100’s of Transformers -
Categorized
•
Browse
•
Search
•
Popup in Workbench
•
Navigate to in Workbench
•
Don’t see what you need? -
Esri UC2013 . Technical Workshop .
Ask for it!
Authoring Spatial ETL Tools Readers, Transformers, Visualizers, Writers
•
Add Spatial ETL tools into a ToolBox -
Right Click context menu
•
Wizard to stub out your Workbench
•
Refine the workspace
•
Publish Parameters -
•
Inputs are automatic
Run the workspace -
In Workbench As a GP Tool
Esri UC2013 . Technical Workshop .
Demo
ETL Bruce Harold
Esri UC2013 . Technical Workshop .
Demo Summary: Spatial ETL •
Complements Core Functionality
•
Analogous to ModelBuilder, except… Feature by Feature granularity - Feature geometry manipulable - Non-Esri & Web data sources are OK -
Esri UC2013 . Technical Workshop .
Wrapping Up More on use cases…
Esri UC2013 . Technical Workshop .
A Word on XML XML is a Document, not a Format
•
XMLTemplater -
•
Easily use an XML file to stub out XML Populate attributes with XQuery expressions
XQuery Extract, Update, Explode {fme:get-attribute("feederName")} {fme:get-attribute("feederNo")} <eaLoc>{fme:get-attribute("eaLoc")}
Esri UC2013 . Technical Workshop .
Organizational Interaction Regional / County National GIS State ETL
Federal
Esri UC2013 . Technical Workshop .
ArcGIS Server and Data Interoperability 64bit
•
Non-Native formats in Map Services
•
Publish ETL as GP services
•
Esri WFS services more flexible -
Non-cached streaming option Setting in Interop Connection
Esri UC2013 . Technical Workshop .
New in 10.2 See the Help too
•
Based on FME 2013 SP1 + -
•
New Formats – highlights only -
•
Improved performance Improved FME Workbench user experience Salesforce OData RIEGL Laser Scan SpatiaLite* ZIP Files
New or Updated Transformers -
Many condensed (e.g. AttributeCreator)
Esri UC2013 . Technical Workshop .
Resources: Learn More •
How do I try it? -
•
How do I learn it? -
•
Request software evaluation (60 days): http://www.esri.com/software/arcgis/extensio ns/datainteroperability/evaluate New course (another coming): http://training.esri.com/gateway/index.cfm?fa =catalog.webCourseDetail&courseid=2571
Who do I talk to? -
Esri UC2013 . Technical Workshop .
Visit the Analysis Island in the Esri Showcase or the SAFE Software booth (#1608) in the Expo.
Thank you… Please fill out the session evaluation
Wednesday Offering ID: 1343 Thursday Offering ID: 1430
Online – www.esri.com/ucsessionsurveys Paper – pick up and put in drop box Esri UC2013 . Technical Workshop .
Esri UC2013 . Technical Workshop . Data Interoperability - An Introduction
Data Interoperability An Introduction Bruce Harold – Esri Dale Lutz – Safe Software
Esri UC2013 . Technical Workshop .
Background Why Data Interoperability?
Esri UC2013 . Technical Workshop .
What is the Data Interoperability Extension? ESRI & SAFE Co-Development based on FME
• Format Support – Adds support for 200+ data formats – Plus Your Custom Formats – Direct use in ArcGIS Desktop
• Quick Data Translation – Data Interoperability Tools Toolbox – Convert between data formats
• Spatial ETL (Extract, Transform, Load) – Schema Mapping (may be Dynamic) – Transformation of Geometry – Logic Controls Data Flow – Per Feature Esri UC2013 . Technical Workshop .
ArcGIS Data Interop Common Tasks • • •
• • •
Data Integration - Bring multiple formats together Data Migration - Process of moving data between systems Data Cleanup - Clean up errors in the data and streamline data preparation Data Distribution - Distribute data to different systems Data Merge - Merge changes from edited data Data Validation - Geometry and Attributes
Esri UC2013 . Technical Workshop .
ETL
Data Interoperability Differentiators Not Just About Schema Manipulation
•
Change Detection
•
3D PDF Writing
•
XML Documents as Features or Attributes
•
JSON Documents as Features or Attributes
•
Protocol Support (HTTP, RSS…)
•
Web and Compressed Inputs
•
Parallel Processing
•
Vertical Coordinate Systems
•
BLOB Attributes
Esri UC2013 . Technical Workshop .
Basics: OGC Support What you get just by installing
Esri UC2013 . Technical Workshop .
OGC GML Simple Feature Format Support Support for GML-SF in ArcGIS Desktop
•
GML – SF – simple features
•
WFS – optionally non-cached for auto refresh
•
WFS – cache timeout
•
Note: Installation of the extension is required
Esri UC2013 . Technical Workshop .
Demo
OGC Formats: GML & WFS Bruce Harold
Esri UC2013 . Technical Workshop .
Demo Summary: GML & WFS •
Comes Free But you have to install - Simple Features Only -
•
Better with the extension licensed -
Non-simple parameters supported
•
Direct Read of GML, consume WFS
•
Important in many jurisdictions
Esri UC2013 . Technical Workshop .
Using Formats Direct Read of any format
Esri UC2013 . Technical Workshop .
Direct Read of Formats •
Formats recognized by Data Interop. File-based, by extension, like .tab, .mid, .skp, .accdb - Visible in Catalog window browse tree - May be extended with Custom Formats -
•
Directly usable in ArcGIS -
•
Save time converting data
Consume external data ArcPad AXF - Other application formats - Web feeds -
Esri UC2013 . Technical Workshop .
Easy Format Selection •
Browse or Search
Esri UC2013 . Technical Workshop .
When to Browse and when to Connect Browse & Direct Read uses format defaults
•
Many formats have optional parameters -
Select subsets, set conditions, passwords, filters…
•
Connections give you access to format parameters
•
Connections let you set Coordinate Systems
Esri UC2013 . Technical Workshop .
Interoperability Connections Persist between application sessions
•
Convenient to access in browse tree
•
Retain your settings
•
Off-disk data -
•
Web Feeds Google Spreadsheets
Share with colleagues -
.FDL file created C:\Users\<user>\AppData\Roaming\Safe Software\Interoperability
Esri UC2013 . Technical Workshop .
Demo
Direct Read Bruce Harold
Esri UC2013 . Technical Workshop .
Demo Summary: Direct Read •
Direct Read Summary -
Directly read, inspect and browse within ArcGIS
-
Works like native ESRI formats (no GDB behavior)
-
Access directly in Desktop Apps, Geoprocessing
-
No development or customization required
Esri UC2013 . Technical Workshop .
Quick Translation Quick Import & Quick Export GP Tools
Esri UC2013 . Technical Workshop .
Quick Data Import/Export Enabling Extension Provides ‘Quick’ Tools •
Quick Export -
•
Quick Import -
•
To any format To staging Geodatabase
Not Configurable -
Is this OK?
Esri UC2013 . Technical Workshop .
Automation and Quick Translations •
Use Quick Tools in ModelBuilder -
Get any format into a temporary File GDB Handle Multiple Formats In/Out
Esri UC2013 . Technical Workshop .
Demo
Quick Tools Bruce Harold
Esri UC2013 . Technical Workshop .
Demo Summary: Quick Import/Export •
“Quick” GP Tools -
Useful in ModelBuilder
-
Also Scriptable in Python
-
Easy interaction with others - Departments - Contractors - 3rd Party Software Users
Esri UC2013 . Technical Workshop .
Leveraging Formats Custom Formats
Esri UC2013 . Technical Workshop .
What if you need your own format? Save time by “canning” a custom format workflow
•
Commonly handled text-based sources -
CSV XML
•
Build format recognition workflow once
•
Re-use the persisted workflow any time
•
Lets you develop a file format for sharing
Esri UC2013 . Technical Workshop .
Demo
Custom Formats Bruce Harold
Esri UC2013 . Technical Workshop .
Demo Summary: Custom Formats •
Based on an existing format
•
May be shared .FDS file in profile directory for each format - C:\Users\<user>\My Documents\FME\Formats -
•
Editable later Get the custom Quake format by downloading the slides!
Esri UC2013 . Technical Workshop .
The Power of Transformation Spatial ETL
Esri UC2013 . Technical Workshop .
The Full Power of Spatial ETL •
Formats are only part of the story
•
Often the format does not change * -
•
Transformation is key -
•
But the schema does Many categories of transformer Feature by Feature or Dataset by Dataset Geometry or Attribute transformations
Many Inputs Many Outputs is OK *like shapefile to shapefile
Esri UC2013 . Technical Workshop .
Workbench Application •
Graphically Design Data Flows
READ
Esri UC2013 . Technical Workshop .
TRANSFORM
WRITE
It’s a Pipeline •
Readers send features into the workspace -
•
Transformers “do something’ -
•
Geometry – X:Y:Z:M Attributes – with a schema May operate “per feature” and pass it immediately May work on sets or all features together Some offer parallel processing
Writers send features to the destination DATA
Esri UC2013 . Technical Workshop .
Some Transformers Are All You Need •
This example derives Adds, Deletes and NoChanges in one step -
Esri Community Maps contributor data is processed this way
Esri UC2013 . Technical Workshop .
The Power of Transformation •
Using Spatial ETL •
100’s of Transformers -
Categorized
•
Browse
•
Search
•
Popup in Workbench
•
Navigate to in Workbench
•
Don’t see what you need? -
Esri UC2013 . Technical Workshop .
Ask for it!
Authoring Spatial ETL Tools Readers, Transformers, Visualizers, Writers
•
Add Spatial ETL tools into a ToolBox -
Right Click context menu
•
Wizard to stub out your Workbench
•
Refine the workspace
•
Publish Parameters -
•
Inputs are automatic
Run the workspace -
In Workbench As a GP Tool
Esri UC2013 . Technical Workshop .
Demo
ETL Bruce Harold
Esri UC2013 . Technical Workshop .
Demo Summary: Spatial ETL •
Complements Core Functionality
•
Analogous to ModelBuilder, except… Feature by Feature granularity - Feature geometry manipulable - Non-Esri & Web data sources are OK -
Esri UC2013 . Technical Workshop .
Wrapping Up More on use cases…
Esri UC2013 . Technical Workshop .
A Word on XML XML is a Document, not a Format
•
XMLTemplater -
•
Easily use an XML file to stub out XML Populate attributes with XQuery expressions
XQuery Extract, Update, Explode {fme:get-attribute("feederName")} {fme:get-attribute("feederNo")} <eaLoc>{fme:get-attribute("eaLoc")}
Esri UC2013 . Technical Workshop .
Organizational Interaction Regional / County National GIS State ETL
Federal
Esri UC2013 . Technical Workshop .
ArcGIS Server and Data Interoperability 64bit
•
Non-Native formats in Map Services
•
Publish ETL as GP services
•
Esri WFS services more flexible -
Non-cached streaming option Setting in Interop Connection
Esri UC2013 . Technical Workshop .
New in 10.2 See the Help too
•
Based on FME 2013 SP1 + -
•
New Formats – highlights only -
•
Improved performance Improved FME Workbench user experience Salesforce OData RIEGL Laser Scan SpatiaLite* ZIP Files
New or Updated Transformers -
Many condensed (e.g. AttributeCreator)
Esri UC2013 . Technical Workshop .
Resources: Learn More •
How do I try it? -
•
How do I learn it? -
•
Request software evaluation (60 days): http://www.esri.com/software/arcgis/extensio ns/datainteroperability/evaluate New course (another coming): http://training.esri.com/gateway/index.cfm?fa =catalog.webCourseDetail&courseid=2571
Who do I talk to? -
Esri UC2013 . Technical Workshop .
Visit the Analysis Island in the Esri Showcase or the SAFE Software booth (#1608) in the Expo.
Thank you… Please fill out the session evaluation
Wednesday Offering ID: 1343 Thursday Offering ID: 1430
Online – www.esri.com/ucsessionsurveys Paper – pick up and put in drop box Esri UC2013 . Technical Workshop .
Esri UC2013 . Technical Workshop . Data Interoperability - An Introduction
