ArcGIS GeoEvent Processor
Building Real-Time Web Applications Using ArcGIS GeoEvent Processor
Ryan Elliott |
Software Engineer ArcGIS GeoEvent Processor for Server [email protected]
James Cardona |
Web Developer DC Development Center [email protected]
ArcGIS GeoEvent Processor for Server Integrates and exploits real-time data •
Integrates real-time streaming data into ArcGIS
•
Performs continuous processing and real-time analytics
•
Sends updates and alerts to those who need it where they need it GeoEvent Services
GeoEvent Processor ws://
ArcGIS Server Inputs
Outputs
ArcGIS Web API for JavaScript •
Gives developers tools to add GIS functionality to web applications -
Interactive maps for visualizing data.
-
Widgets for finding addresses, editing data, making legends…
-
Analysis – Run a model and view results, enrich existing data with detailed demographic information
-
Embed into existing web page or make new focused application
Demo
Demonstration Field Worker Monitoring
Real-Time Data on the Map
•
SCREENSHOT
Special Handling of Alerts
•
SCREENSHOT
Selectively Display Historical Data
•
SCREENSHOT
Receiving Real-Time Data Ryan Elliott
Receiving real-time data Connectors
You can easily integrate real-time streaming data with ArcGIS by using an input connector. GeoEvent Processor Inputs
Outputs
GeoEvent Services
You can create your own connectors.
Twitter
GNIP
Receive text from a TCP Socket
Instagram
Geofeedia
Receive text from a UDP Socket
CAP
exactEarth
http:// Receive Features on a REST endpoint http:// Receive JSON on a REST endpoint ws:// Receive JSON on a Web Socket ws:// Receive JSON on external Web Socket Poll an ArcGIS Server for Features http:// Poll an external website for JSON
Cursor-on-Target VMF GeoMessage ActiveMQ RabbitMQ
Partner Gallery
Receive RSS
Esri Gallery
Out of the Box
•
ASDI (FAA) OSIsoft Valarm Harris CompassCom
NMEA
NetworkFleet Zonar
.csv
Watch a folder for new .csv files
TAIP (Trimble)
.json
Watch a folder for new .json files
RAP (Sierra Wireless)
Demo
Demonstration Connecting to Real-Time Data Feeds
Applying Real-Time Analytics Ryan Elliott
Applying real-time analytics GeoEvent Services •
A GeoEvent Service configures the flow of GeoEvents, the Filtering and GeoEvent Processing steps to perform, - what input(s) to apply them to, - and what outputs(s) to send the results to. -
Applying real-time analytics GeoEvent Processing
You can perform continuous analytics on geoevents as they are received using a processor. GeoEvent Processor GeoEvent Services
Track Idle Detector
Field Calculator Geotagger Field Mapper
Esri Gallery
Field Reducer
ETA Calculator Service Area Buffer Ellipse
Track Gap Detector
Range Fan
Incident Detector
Visibility Query Report
SDK
Inputs
Outputs
You can create your own processors.
Field Enricher
Out of the Box
•
Slope Calculator Volume Control
Demo
Demonstration Detecting GeoFence Violations
Sending Real-Time Data to the Map Ryan Elliott & James Cardona
Sending Real-time Events to Clients Patterns – pull and push •
•
Pull via feature services -
Must be backed by an enterprise geodatabase (EGDB)
-
Clients poll to get updates
Push via Web Socket output -
Low latency, high throughput
-
Clients subscribe to features of interest GeoEvent Processor
Your Applications
Broadcast Features
Subscribe (Push)
ws://
feature layers
ArcGIS Server
Polling (Pull)
Update a Feature Add a Feature
GeoEvent Services
ArcGIS Server Map Services Feature Services …
EGDB
Feature Layer Stream Layer
Demo
Demonstration Publishing Events through Web Sockets
Quick Tour of the GeoEvent Processor REST Interface Ryan Elliott
Administering GeoEvent Processor REST Admin API
A complete set of REST administrative endpoints enable you to manage GeoEvent Processor programmatically: Everything you can do in ‘GeoEvent Processor Manager’ you can also do via REST since it exclusively uses the REST Admin API https://localhost:6143/geoevent/admin GeoEvent Processor Manager
REST Admin API
JavaScript
GeoEvent Processor GeoEvent Services Outputs
-
Inputs
•
ArcGIS Server
Demo
Exploring the REST api Ryan Elliott
Displaying Real-Time Web Socket data on a map with the Stream Layer James Cardona
Stream Layer What is it?
•
-
•
esri/layers/GraphicsLayer
A new type of layer in the Javascript API
esri/layers/FeatureLayer
Introduced in version 3.6
esri/layers/StreamLayer
Draws data on map using client-side graphics
RESPONSE
Graphic § • •
Geometry Attributes Symbol
Stream Layer
Feature Layer
Graphics Layer
GET POST
HTTP Feature service
Web Socket
PUSH
GeoEvent Processor
ArcGIS Server ArcGIS Server
Stream Layer Advantages •
•
More efficient transfer of data. -
Only negotiate a connection once.
-
Messages sent without extra headers
Avoids need to poll for data updates Stream Layer Feature Layer
Request
Response
Close Establish Connection Connection
Data
GeoEvent Processor Feature service
ArcGIS Server
ArcGIS Server
Stream Layer Lifecycle
GeoEvent Processor
ArcGIS Server
Map
StreamLayer
Stream Layer (Constructor) new StreamLayer( featureCollection, options )
FeatureCollection: { layerDefinition: { geometryType: esriGeometryPoint, timeInfo: { startTimeField: “StartTime”, trackIdField: “Name” }, fields: [ … ] }, featureSet: null }
•
Options • webSocketUrl: ws://gep:6180/urlpath • purgeOptions: { displayCount: 500 }
Stream Layer Lifecycle
GeoEvent Processor
ArcGIS Server
{ geometry: { x:-77, y:42, spatialReference: { wkid:4326}, attributes: { name:”Buffy”}}
• Adds graphic to layer (Symbol generated using layer renderer) • Emits graphic-add event
Stream Layer (WebSocket.onmessage) • Parses message to generate geometry and attributes • Emits message event
• Checks if number of graphics is over threshold • Removes oldest graphic if necessary • Emits graphic-remove event if necessary
Stream Layer What is needed <meta http-equiv="X-UA-Compatible" content="IE=10" />
•
GeoEvent Processor Output Connector -
Feature JSON over Web Socket 10 +
•
•
Browser that supports Web Sockets http://caniuse.com/websockets Web Socket protocol allowed on network ws://, wss://
6+
14 + 6+
Demo
Demonstration Consuming Streams of Features with the Stream Layer
Road Ahead Stream Services Ryan Elliott & James Cardona
Stream Services – The Future of Streaming Data
•
Developer Productivity -
•
Customizable -
•
Provides all the metadata needed by clients to find and consume a stream of features. Individualized client connections provide filtering and projection.
Scalable -
Features published to a stream services are accessible from any machine in the cluster.
Stream Services – The Future of Streaming Data
GeoEvent Processor ws://
Subscribe
Stream Service
Subscription changes
ArcGIS Server
Stream Layer
GeoEvent Processor ws://
Subscribe
Stream Service
ArcGIS Server
Subscription changes
Stream Layer
Demo
Demonstration Stream Services
What We Covered Today
•
Consumed Live data from Sensors and a Web Service
•
Filtered and generated incidents from spatial behavior
•
Pushed events to a web app through feature services and web sockets
•
Used JavaScript API Stream Layer to receive messages pushed from server and display them on a map
•
Saw a preview of the Stream Service that will allow developers to easily receive data through a web socket and set filters that are processed on the server
ArcGIS GeoEvent Processor Additional Workshops
•
Use Cases for Applying Real-Time Analytics Using ArcGIS GeoEvent Processor Tuesday 4:00pm - 5:00pm Primrose B
•
ArcGIS GeoEvent Processor—An Introduction Wednesday 10:30am - 11:30am Primrose B
•
Extending ArcGIS GeoEvent Processor with New Connectors Wednesday 1:00pm - 2:00pm Pasadena/Ventura/Siearra
•
The Internet of Things (IoT) and ArcGIS GeoEvent Processor Wednesday 4:00pm - 5:00pm Primrose C/D
•
Extending ArcGIS GeoEvent Processor with New Processors Wednesday 5:30pm - 6:30pm Primrose C/D
•
Use Cases for Applying Real-Time Analytics Using ArcGIS GeoEvent Processor Thursday2:30pm - 3:30pm Primrose C/D
ArcGIS API for JavaScript Additional Workshops
•
ArcGIS API for JavaScript: What Have You Done for Me Lately? Tuesday 5:30 - 6:30 Oasis 4 and Thursday 2:30 – 3:30 Catalina/Madera
•
Transitioning to JavaScript: What to Expect and How to Quickly Come Up to Speed Tuesday 5:30 – 6:30 Primrose B
•
Introduction to ArcGIS API for JavaScript Wednesday 2:30 – 3:30 Primrose B
•
Tips and Tricks for Debugging Apps Built with ArcGIS API for JavaScript Wednesday 4:00 – 5:00 Catalina/Madera and Thursday 2:30 – 3:30 Primrose A
•
Dojo: The Good Parts Thursday 10:00 – 11:00 Smoketree F and Thursday 2:30 – 3:30 Smoketree F
•
Introduction to the ArcGIS WebApp Builder: JavaScript Apps Made Easy Thursday 1:00 – 2:00 Primrose B
Additional resources for GeoEvent Processor
•
Resource Center http://pro.arcgis.com/share/geoevent-processor
•
Forum http://forums.arcgis.com/forums/257-GeoEvent-Processor
Additional resources
•
ArcGIS API for JavaScript Resource Center https://developers.arcgis.com/javascript
•
Code Samples https://github.com/jcardonadcdev/DevSummit2014-realtime
•
GeoEvent Processor Tutorial for Web Sockets (Coming soon) In the GeoEvent Processor Gallery
Questions / Feedback? To learn more: https://developers.arcgis.com/javascript http://pro.arcgis.com/share/geoevent-processor
Ryan Elliott |
Software Engineer ArcGIS GeoEvent Processor for Server [email protected]
James Cardona |
Web Developer DC Development Center [email protected]
Ryan Elliott |
Software Engineer ArcGIS GeoEvent Processor for Server [email protected]
James Cardona |
Web Developer DC Development Center [email protected]
ArcGIS GeoEvent Processor for Server Integrates and exploits real-time data •
Integrates real-time streaming data into ArcGIS
•
Performs continuous processing and real-time analytics
•
Sends updates and alerts to those who need it where they need it GeoEvent Services
GeoEvent Processor ws://
ArcGIS Server Inputs
Outputs
ArcGIS Web API for JavaScript •
Gives developers tools to add GIS functionality to web applications -
Interactive maps for visualizing data.
-
Widgets for finding addresses, editing data, making legends…
-
Analysis – Run a model and view results, enrich existing data with detailed demographic information
-
Embed into existing web page or make new focused application
Demo
Demonstration Field Worker Monitoring
Real-Time Data on the Map
•
SCREENSHOT
Special Handling of Alerts
•
SCREENSHOT
Selectively Display Historical Data
•
SCREENSHOT
Receiving Real-Time Data Ryan Elliott
Receiving real-time data Connectors
You can easily integrate real-time streaming data with ArcGIS by using an input connector. GeoEvent Processor Inputs
Outputs
GeoEvent Services
You can create your own connectors.
GNIP
Receive text from a TCP Socket
Geofeedia
Receive text from a UDP Socket
CAP
exactEarth
http:// Receive Features on a REST endpoint http:// Receive JSON on a REST endpoint ws:// Receive JSON on a Web Socket ws:// Receive JSON on external Web Socket Poll an ArcGIS Server for Features http:// Poll an external website for JSON
Cursor-on-Target VMF GeoMessage ActiveMQ RabbitMQ
Partner Gallery
Receive RSS
Esri Gallery
Out of the Box
•
ASDI (FAA) OSIsoft Valarm Harris CompassCom
NMEA
NetworkFleet Zonar
.csv
Watch a folder for new .csv files
TAIP (Trimble)
.json
Watch a folder for new .json files
RAP (Sierra Wireless)
Demo
Demonstration Connecting to Real-Time Data Feeds
Applying Real-Time Analytics Ryan Elliott
Applying real-time analytics GeoEvent Services •
A GeoEvent Service configures the flow of GeoEvents, the Filtering and GeoEvent Processing steps to perform, - what input(s) to apply them to, - and what outputs(s) to send the results to. -
Applying real-time analytics GeoEvent Processing
You can perform continuous analytics on geoevents as they are received using a processor. GeoEvent Processor GeoEvent Services
Track Idle Detector
Field Calculator Geotagger Field Mapper
Esri Gallery
Field Reducer
ETA Calculator Service Area Buffer Ellipse
Track Gap Detector
Range Fan
Incident Detector
Visibility Query Report
SDK
Inputs
Outputs
You can create your own processors.
Field Enricher
Out of the Box
•
Slope Calculator Volume Control
Demo
Demonstration Detecting GeoFence Violations
Sending Real-Time Data to the Map Ryan Elliott & James Cardona
Sending Real-time Events to Clients Patterns – pull and push •
•
Pull via feature services -
Must be backed by an enterprise geodatabase (EGDB)
-
Clients poll to get updates
Push via Web Socket output -
Low latency, high throughput
-
Clients subscribe to features of interest GeoEvent Processor
Your Applications
Broadcast Features
Subscribe (Push)
ws://
feature layers
ArcGIS Server
Polling (Pull)
Update a Feature Add a Feature
GeoEvent Services
ArcGIS Server Map Services Feature Services …
EGDB
Feature Layer Stream Layer
Demo
Demonstration Publishing Events through Web Sockets
Quick Tour of the GeoEvent Processor REST Interface Ryan Elliott
Administering GeoEvent Processor REST Admin API
A complete set of REST administrative endpoints enable you to manage GeoEvent Processor programmatically: Everything you can do in ‘GeoEvent Processor Manager’ you can also do via REST since it exclusively uses the REST Admin API https://localhost:6143/geoevent/admin GeoEvent Processor Manager
REST Admin API
JavaScript
GeoEvent Processor GeoEvent Services Outputs
-
Inputs
•
ArcGIS Server
Demo
Exploring the REST api Ryan Elliott
Displaying Real-Time Web Socket data on a map with the Stream Layer James Cardona
Stream Layer What is it?
•
-
•
esri/layers/GraphicsLayer
A new type of layer in the Javascript API
esri/layers/FeatureLayer
Introduced in version 3.6
esri/layers/StreamLayer
Draws data on map using client-side graphics
RESPONSE
Graphic § • •
Geometry Attributes Symbol
Stream Layer
Feature Layer
Graphics Layer
GET POST
HTTP Feature service
Web Socket
PUSH
GeoEvent Processor
ArcGIS Server ArcGIS Server
Stream Layer Advantages •
•
More efficient transfer of data. -
Only negotiate a connection once.
-
Messages sent without extra headers
Avoids need to poll for data updates Stream Layer Feature Layer
Request
Response
Close Establish Connection Connection
Data
GeoEvent Processor Feature service
ArcGIS Server
ArcGIS Server
Stream Layer Lifecycle
GeoEvent Processor
ArcGIS Server
Map
StreamLayer
Stream Layer (Constructor) new StreamLayer( featureCollection, options )
FeatureCollection: { layerDefinition: { geometryType: esriGeometryPoint, timeInfo: { startTimeField: “StartTime”, trackIdField: “Name” }, fields: [ … ] }, featureSet: null }
•
Options • webSocketUrl: ws://gep:6180/urlpath • purgeOptions: { displayCount: 500 }
Stream Layer Lifecycle
GeoEvent Processor
ArcGIS Server
{ geometry: { x:-77, y:42, spatialReference: { wkid:4326}, attributes: { name:”Buffy”}}
• Adds graphic to layer (Symbol generated using layer renderer) • Emits graphic-add event
Stream Layer (WebSocket.onmessage) • Parses message to generate geometry and attributes • Emits message event
• Checks if number of graphics is over threshold • Removes oldest graphic if necessary • Emits graphic-remove event if necessary
Stream Layer What is needed <meta http-equiv="X-UA-Compatible" content="IE=10" />
•
GeoEvent Processor Output Connector -
Feature JSON over Web Socket 10 +
•
•
Browser that supports Web Sockets http://caniuse.com/websockets Web Socket protocol allowed on network ws://, wss://
6+
14 + 6+
Demo
Demonstration Consuming Streams of Features with the Stream Layer
Road Ahead Stream Services Ryan Elliott & James Cardona
Stream Services – The Future of Streaming Data
•
Developer Productivity -
•
Customizable -
•
Provides all the metadata needed by clients to find and consume a stream of features. Individualized client connections provide filtering and projection.
Scalable -
Features published to a stream services are accessible from any machine in the cluster.
Stream Services – The Future of Streaming Data
GeoEvent Processor ws://
Subscribe
Stream Service
Subscription changes
ArcGIS Server
Stream Layer
GeoEvent Processor ws://
Subscribe
Stream Service
ArcGIS Server
Subscription changes
Stream Layer
Demo
Demonstration Stream Services
What We Covered Today
•
Consumed Live data from Sensors and a Web Service
•
Filtered and generated incidents from spatial behavior
•
Pushed events to a web app through feature services and web sockets
•
Used JavaScript API Stream Layer to receive messages pushed from server and display them on a map
•
Saw a preview of the Stream Service that will allow developers to easily receive data through a web socket and set filters that are processed on the server
ArcGIS GeoEvent Processor Additional Workshops
•
Use Cases for Applying Real-Time Analytics Using ArcGIS GeoEvent Processor Tuesday 4:00pm - 5:00pm Primrose B
•
ArcGIS GeoEvent Processor—An Introduction Wednesday 10:30am - 11:30am Primrose B
•
Extending ArcGIS GeoEvent Processor with New Connectors Wednesday 1:00pm - 2:00pm Pasadena/Ventura/Siearra
•
The Internet of Things (IoT) and ArcGIS GeoEvent Processor Wednesday 4:00pm - 5:00pm Primrose C/D
•
Extending ArcGIS GeoEvent Processor with New Processors Wednesday 5:30pm - 6:30pm Primrose C/D
•
Use Cases for Applying Real-Time Analytics Using ArcGIS GeoEvent Processor Thursday2:30pm - 3:30pm Primrose C/D
ArcGIS API for JavaScript Additional Workshops
•
ArcGIS API for JavaScript: What Have You Done for Me Lately? Tuesday 5:30 - 6:30 Oasis 4 and Thursday 2:30 – 3:30 Catalina/Madera
•
Transitioning to JavaScript: What to Expect and How to Quickly Come Up to Speed Tuesday 5:30 – 6:30 Primrose B
•
Introduction to ArcGIS API for JavaScript Wednesday 2:30 – 3:30 Primrose B
•
Tips and Tricks for Debugging Apps Built with ArcGIS API for JavaScript Wednesday 4:00 – 5:00 Catalina/Madera and Thursday 2:30 – 3:30 Primrose A
•
Dojo: The Good Parts Thursday 10:00 – 11:00 Smoketree F and Thursday 2:30 – 3:30 Smoketree F
•
Introduction to the ArcGIS WebApp Builder: JavaScript Apps Made Easy Thursday 1:00 – 2:00 Primrose B
Additional resources for GeoEvent Processor
•
Resource Center http://pro.arcgis.com/share/geoevent-processor
•
Forum http://forums.arcgis.com/forums/257-GeoEvent-Processor
Additional resources
•
ArcGIS API for JavaScript Resource Center https://developers.arcgis.com/javascript
•
Code Samples https://github.com/jcardonadcdev/DevSummit2014-realtime
•
GeoEvent Processor Tutorial for Web Sockets (Coming soon) In the GeoEvent Processor Gallery
Questions / Feedback? To learn more: https://developers.arcgis.com/javascript http://pro.arcgis.com/share/geoevent-processor
Ryan Elliott |
Software Engineer ArcGIS GeoEvent Processor for Server [email protected]
James Cardona |
Web Developer DC Development Center [email protected]
