Son AFOSR PI Meeting Jan2016 Son Sent
A Hierarchical and Scalable Situation Awareness System for 3-D Border Surveillance Sponsor: Air Force Office of Scientific Research FA9550-12-1-0238 (DDDAS); 15RT1016 (New) Program Manager: Dr. Frederica Darema PIs: Young-Jun Son1, Jian Liu1, Jyh-Ming Lien2 Students: S. Minaeian1, Y. Yuan1, S. Lee1, and J. Han1 1Systems
and Industrial Engineering, University of Arizona 2Computer Science, George Mason University PI Contact: [email protected]; 1-520-626-9530 AFOSR DDDAS PI Meeting Jan. 2016
Agenda • Previous project • New project – New challenges – How DDDAS is addressed
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Overview of Previous Project Motivation: TUS 1- Project (23-mile long area of the border in Sasabe, AZ)
Problem: A highly complex, uncertain, dynamically changing border environment
Goal: Develop a simulation-based planning and control system for surveillance and crowd control via collaborative UAVs/UGVs
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
DDDAMS-based Planning and Control Framework
Khaleghi, A. M., Xu, D., Wang, Z., Li, M., Lobos, A., Liu, J., & Son, Y. (2013). A DDDAMS-based Planning and Control Framework for Surveillance and Crowd Control via UAVs and UGVs. Expert Systems with Applications, 40, 7168-7183. Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Multi-resolution Data
Challenge:
Aggregate multi-resolution data
Approach:
UAVs’ global perception and UGVs’ detailed perception Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Framework of Developed Methodology
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Detection Module: Simulated ODROID USB-CAM 720P HD (16:9): 1280x720p @ 30 fps FOV: 72
GoPro HERO 3- Tarot Gimbal Stabilizer HD (16:9): 1280x720p @ 120 ~ 25 fps FOV(x): 64.4 ; FOV(y): 37.2
EDD
: UGV’s Effective Detection Depth DR : Detection range for UGV FOV(y) FOV (x) A : Detection range for UAV DR FOV : Field of view : Distance h(G ) ( A) : Altitude h
G
h
DR(y)
G
G
DRmin 2 hmin tan FOV / 2 DR
G
G
G
2 hmax tan FOV / 2 DRmax
DR(x)
DR ( A) 2h( A) tan FOV / 2
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Detection Module (UAV & UGV): Actual
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Framework of Developed Methodology
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Crowd Tracking Module
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Case Study
Bayesian estimation: 75% Less computation time Comparable/Better prediction performance Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Framework of Developed Methodology
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Control Strategies associated with Motion Planning •
Given selected destination of UAV/UGV, find the path that optimizes a certain combination of criteria
(a) minimize travel distance
•
(b) minimize elevation penalty (fuel consumption)
(c) minimize the weighted average of (a) and (b)
Weighted average of the multiple objectives Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
System Implementation • Agent-based HIL simulation • UAVs and UGVs • Social force model and GIS
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Agent-based Hardware-in-the-loop Simulation Agent-based Simulation Repast Simphony with 3D GIS
Sensory Data (e.g. GPS) Assembled UAV (APM:Copter / Arducopter)
Hardware Interface: MAVproxy
Control Commands (MAVLink Messages) Assembled UGV (APM:Rover / Ardurover)
Wi-Fi / XBee PRO 900HP; APM one Khaleghi, A. M., Xu, D., Lobos, A., Minaeian, S., Son, Y. -J., & Liu, J. (2013). Agent- based hardware-in-the-loop simulation for modeling UAV/UGV surveillance and crowd control system. In Proceedings of the winter simulation conference 2013, Washington, DC, USA. Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Assembled UAV (Arducopter), AR.Drone, X8+ UAV, and UGV
Navigate GPS waypoints autonomously using APM autopilot set (Arduino-based Autopilot- APM 2.5) Microcontroller (ATMEGA2560) Low-power Atmel 8-bit AVR RISC-based 256KB ISP flash memory 8KB SRAM 4KB EPROM Throughput: 16 MIPS at 16MHz
Motion Processing Unit (MPU-6000) 3-Axis Gyro 3-Axis Accelerometer Barometric pressure sensor (SM5611)
Global Positioning System (GPS) GPS update rate: 5hz (5X per second) Using GPS unit, the UAV has an outdoor navigation accuracy of about +/- 5 meters Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Social Force Model for Crowd Motion Direction/angle and walking speed of humans has been modeled using 2 heuristics based on their visual data (Moussaïd et al., 2011) • Heuristic 1: Minimize the angle/direction of each individual by minimizing the distance to its destination 2 min d( ) dmax f 2 ( ) 2dmax f ( )cos( 0 )
• Heuristic 2: Change walking speed of human to avoid collisions v min(v0 ,
dh
)
human field of view: ( , ) (for e.g. 90 ,90) maximum range of view: dmax (for e.g. 10 m) human comfortable walking speed: v0 (for e.g. 1.5 m / s) distance to obstacle: dh
relaxation time time requires to adopt new behavior : (for e.g. 1 sec)
Khaleghi, A. M., Xu, D., Lobos, A., Minaeian, S., Son, Y. -J., & Liu, J. (2013). Agent- based hardware-in-the-loop simulation for modeling UAV/UGV surveillance and crowd control system. In Proceedings of the winter simulation conference 2013, Washington, DC, USA. Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Publications: Journals and Book Chapters •
•
•
•
•
S. Minaeian, J. Liu, Y. Son, Vision-based Target Detection and Localization via a Team of Cooperative UAV and UGVs, IEEE Transactions on Systems, Man, and Cybernetics: Systems (Special Issue on Biomedical Robotics and Bio-mechatronics Systems and Application), Accepted, 2015. A. Khaleghi, D. Xu, Z. Wang, M. Li, A. Lobos, J. Liu, Y. Son, A DDDAMS-based Planning and Control Framework for Surveillance and Crowd Control via UAVs and UGVs, Expert Systems with Applications, 40, 2013, 7168-7183. Yifei Yuan, Zhenrui Wang, Mingyang Li, Young-Jun Son, Jian Liu, DDDAS-based Information-Aggregation for Crowd Dynamics Modeling with UAVs and UGVs, Frontiers in Robotics and AI (Sensor Fusion and Machine Perception Section), 2:8, 2015, 1-10. A. Khaleghi, D. Xu, S. Minaeian, M. Li, Y. Yuan, C. Vo, A. Mousavian, J. Lien, J. Liu, and Y. Son, UAV/UGV Surveillance and Crowd Control via Hardware-in-the-loop DDDAMS System, Darema, F., Douglas, C (Eds.), Springer (under review) Online Collision Prediction Among 2D Polygonal and Articulated Obstacles, Yanyan Lu, Zhonghua Xi and Jyh-Ming Lien, International Journal of Robotics Research (IJRR), Accepted, 2015.
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Publications: Proceedings (1) •
•
•
•
• •
Minaeian, S., Yuan, Y., Liu, J., and Son, Y., 2015, “Human-in-the-Loop Agent-based Simulation for Improved Autonomous Surveillance using Unmanned Vehicles,” Proceedings of 2015 Winter Simulation Conference, Huntington Beach, CA (poster) Continuous Visibility Feature, Guilin Lu, Yotam Gingold, and Jyh-Ming Lien, in the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015, Boston, MA, USA Semantically Guided Location Recognition for Outdoors Scenes, Arsalan Mousavian, Jana Kosecka and Jyh-Ming Lien, Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), May 2015, Seattle, WA, USA Khaleghi, A., Xu, D., Minaeian, S., Yuan, Y., Liu, J., and Son, Y., 2015, “Analysis of UAV/UGV Control Strategies in a DDDAMS-based Surveillance System,” Proceedings of 2015 IIE Annual Meeting, Nashville, TN. Minaeian, S., Liu, J., and Son, Y., 2015, “Crowd Detection and Localization Using a Team of Cooperative UAV/UGVs,” Proceedings of 2015 IIE Annual Meeting, Nashville, TN. Khaleghi, A., Xu, D., Minaeian, S., Li, M., Yuan, Y., Liu, J., Son, Y., Vo, C., and Lien, J., 2014, “A DDDAMS-based UAV and UGV Team Formation Approach for Surveillance and Crowd Control,” Proceedings of 2014 Winter Simulation Conference, Savannah, GA.
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Publications: Proceedings (2) •
•
•
•
•
•
Khaleghi, A., Xu, D., Minaeian, S., Li, M., Yuan, Y., Liu, J., and Son, Y., 2014, “A Comparative Study of Control Architectures in UAV/UGV-based Surveillance System,” Proceedings of 2014 IIE Annual Meeting, Montreal, Canada. Wang, Z., Li, M., Khaleghi, A., Xu, D., Lobos, A., Vo, C., Lien, J., Liu, J., and Son, Y., 2013, “DDDAMS-based Crowd Control via UAVs and UGVs,” Procedia Computer Science 18, 2028–2035 (Proceedings of 2013 International Conference on Computational Science, Barcelona, Spain). Khaleghi, A., Xu, D., Lobos, A., Minaeian, S., Son, Y., and Liu, J., 2013, “Agent-based Hardware-in-the-Loop Simulation for Modeling UAV/UGV Surveillance and Crowd Control System,” Proceedings of 2013 Winter Simulation Conference, Washington DC. Wang, Z., Li, M., Khaleghi, A., Xu, D., Lobos, A., Vo, C., Lien, J., Liu, J., and Son, Y., 2013, “DDDAMS-based Crowd Control via UAVs and UGVs,” Procedia Computer Science 18, 2028–2035 (Proceedings of 2013 International Conference on Computational Science, Barcelona, Spain). Vo, C., McKay, S., Garg, N., and Lien, J., 2012, “Following a Group of Targets in Large Environments”, Proceedings of the Fifth International Conference on Motion in Games, Springer. Vo, C., and Lien, J., 2012, “Group Following in Monotonic Tracking Regions”, Proceedings of the 22nd Fall Workshop on Computational Geometry, 2012. Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
New Project • New challenges • How DDDAS is addressed
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
3-Level Surveillance Framework
High altitude level (HAL)
Low altitude level (LAL)
Surface level (SL)
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
New Major Problems •
3-D Surveillance System for aerial and ground targets
•
Latency in detection, recognition and identification of targets
•
Heterogeneous data from complex targets by 3 levels of sensors
•
Multi-level information aggregation
•
Active or pro-active surveillance strategies
•
Realistic scenarios and model validation based on data collection from our research partners (AFRL, Raytheon, University Partners …)
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
3-Level Measurement System in Border Surveillance 3-Levels
Types
Sensors
Measurement Data
ElectroOptical/Infrared (EO/IR)
High altitude level (HAL)
SAR Image
Synthetic Aperture Radar (SAR)
EO/IR Image Spectral Image
Remote Sensing
Low altitude level (LAL)
Surface level (SL)
Surveillance Camera
Lidar Image Thermal Images
Mobile Sensors
Fixed Sensors
Magnetic Data
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Agent Model of Sensors (Generic) A generic modeling framework of sensors for the surveillance application Surveillance Behavior Sub-Model
Availability SubModel
Target appears in the system
Adjust the Detection range; Set the threshold Receive signals / messages from the controller
Change location parameters of sensors in order for chasing foe targets
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Signal Processing Sub-Model
Previous EnhancedDDDAMS-based DDDAMS-basedFramework Framework
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
1.0 - Target DRI Detection, Recognition, and Identification of hostile targets (i.e. traffickers) and civilian targets. Discovery the presence of a person, object, or phenomenon*
• •
Sensing technologies: thermal technologies, radar, etc. Motion detection, optical flow, etc.
* Military, U. S. (2005). Dictionary of military and associated terms. US Department of Defense. Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
1.0 - Target DRI Detection, Recognition, and Identification of hostile targets (i.e. traffickers) and civilian targets.
Determination of the nature of a detected person, object or phenomenon, and its class or type *
• • • •
Image Feature Extraction Functional Discriminant Analysis for Time series sensor data Information-aggregation method for Multiple Sensor data Multivariate Classification Method
* Military, U. S. (2005). Dictionary of military and associated terms. US Department of Defense.
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
1.0 - Target DRI Detection, Recognition, and Identification of hostile targets (i.e. traffickers) and civilian targets.
Discrimination between recognizable objects as being friendly or enemy *
• •
Gaussian Mixture Model for Objective Identification BDI (Belief–Desire–Intention) framework
* Military, U. S. (2005). Dictionary of military and associated terms. US Department of Defense. Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Emerging Challenges - Target DRI Sensor Type
Vehicle
Armed People
Animals
New Observation
Image
Training Thermal Sensor
Information Aggregated Classifier
Magnetic Sensor
Predicted Class
Underground, … Radar, etc
…
…
Updating Real Class
Challenge: Feature Extraction Potential Method: Discriminant Analysis
Feature 2
Sample Image Data
V V V V V AV A V AA A AP AP A A AP A APAP AP Feature 1
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Challenge: Classification under DDDAS framework Potential Methods: SVM, KNN, etc.
Extended Belief-Desire-Intention Framework Bratman, 1987 Rao and Georgeff, 1998 Zhao and Son, 2008
Lee, S., Y.-J. Son, and J. Jin (2010), Integrated human decision making and planning model under extended belief-desire-intention framework, ACM Transactions on Modeling and Computer Simulation, 20(4), 23(1)~23(24). Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Model of Drug Traffickers based on BDI framework Developed behavior models of drug traffickers and ground patrol agents together with environmental conditions will provide rich scenarios Behavioral Models of Drug Traffickers Decision 1
Decision 2
Decision 3
Selection at Departure
Route Choice
En-Route Planning
Time
Fastest Way
Hiding/Stopping
Location
Rugged Road
Sudden Direction Change
Behavior Models of Border Patrols
Environment Conditions: weather, terrains
3-Level Sensor Networks
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Agent Models for Patrol Agent and Sensors
Patrol Agents
Sensors
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
2.0- Pattern Processing Recognition and Prediction of objective, behavior and route patterns of hostile targets
•
Spatial Optimization Method Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Emerging Challenges - Pattern Processing … … … …
…
0
0
1
0
…
0
1
0
0
…
2
0
0
0
…
0
0
3
0
…
… … … … …
3D Grid Matrix with Categorized Values e.g. 0: Unoccupied; 1: Hostile Crowd; 2: Friendly Crowd;3: Animals
Challenge: Computational complexity, modeling the interactions between targets.
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
3.0- Mission Control Resource allocation and Motion planning for controlling hostile targets to terminate/mitigate their activities
Predicting Target’s Behavior
Optimized Sensors Allocation
Persistent Surveillance
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Emerging Challenges - Mission Control Challenge: risk assessment and uncertainty quantification
Target Pattern Prediction
Target1
Target Risk Assessment
Target3
+
Target2 Updating
Target
Risk
1
0.75
2
0.97
3
0.23
…
…
Allocation Optimization
New Observation
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
DDDAS in Modules: Target DRI APPLICATION
ALGORITHMS
3-D spatial model
Multi-scale model with more distributions to model spatial, temporal and preference
Heterogeneous models for both civilian and hostile targets
Mixture of regressions analysis algorithm
Multi-scale model for dynamics and BDI for human decisions
Information fusion algorithm for target Identification
Model uncertainty for miss-classification
Dynamic model parameter updating for hostile targets
Sensors and vehicles are modeled as agents
Model uncertainty for miss-classification
Online camera motion strategies in 3D
Image and geometry registration algorithms
Fuse high-level motion command and dynamic environmental information
Online motion planning methods using estimated “earliest collision time” in 3D
MEASUREMENT
3-D data
SOFTWARE Bayesian updating software module for location, dynamic and sensor/vehicle status modeling
City-size midresolution dynamic data by Aerostat Heterogeneous model estimation software module Mid-range Highresolution dynamic data by UAVs
Adjacent highresolution dynamic data by UGVs
Status of sensors and vehicles measured
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Extension of Repast Simphony®-based HIL simulator with more sensors, a richer set of sensory and commands data
Detailed models of agents (e.g. sensors) in a Physics-based simulation software (e.g. Gazebo, MAK, AGI, etc.)
DDDAS in Modules: Pattern Processing APPLICATION
ALGORITHMS
3-D spatial model
Patterns extracted as statistics extracted from algorithm
3-D data
Assume hostile traffickers detected
Mixture of regressions analysis algorithm
City-size midresolution dynamic data by Aerostat
Heterogeneous models for location, dynamics and preference pattern recognition
Model uncertainty for pattern misrecognition
Impacts of sensors/vehicles status considered
Information fusion algorithm gives statistical patterns
Pattern extraction and recognition algorithm will be developed
Algorithm to update prior knowledge on patterns will be developed
MEASUREMENT
SOFTWARE Bayesian updating software module for location, dynamic and sensor/vehicle status modeling
Heterogeneous model estimation software module Mid-range Highresolution dynamic data by UAVs
Adjacent highresolution dynamic data by UGVs
Status of sensors and vehicles measured
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Extension of Repast Simphony®-based HIL simulator with more sensors, a richer set of sensory and commands data Detailed models of agents (e.g. sensors) in a Physics-based simulation software (e.g. Gazebo, MAK, AGI, etc.)
DDDAS in Modules: Mission Control APPLICATION
3-D pattern model
ALGORITHMS
Control index considered for 3-D optimality
Augmenting the Surface Level’s vision
3-D data
City-size midresolution dynamic data by Aerostat
Heterogeneous models
Controllability is estimated via repeated simulations
MEASUREMENT
Algorithm uncertainty caused by both model estimation uncertainty and sensor/vehicle uncertainty
Mid-range Highresolution dynamic data by UAVs
Adjacent highresolution dynamic data by UGVs
Algorithm updated considering agent dynamics, patternshifting and sensor/vehicle status
Status of sensors and vehicles measured
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
SOFTWARE Bayesian updating software module for location, dynamic and sensor/vehicle status modeling
Heterogeneous model estimation software module
Extension of Repast Simphony®-based HIL simulator with more sensors, a richer set of sensory and commands data
Detailed models of agents (e.g. sensors) in a Physics-based simulation software (e.g. Gazebo, MAK, AGI, etc.)
Acknowledgements Sponsor: Air Force Office of Scientific Research FA9550-12-1-0238 (DDDAS); 15RT1016 (New) Program Manager: Dr. Frederica Darema PIs: Young-Jun Son1, Jian Liu1, Jyh-Ming Lien2 Students (Previous Project): A. Khaleghi1, D. Xu1, S. Minaeian1, Y. Yuan1, M. Li1, and C. Vo2 Students (New Project): S. Minaeian1, Y. Yuan1, S. Lee1, and J. Han1 1Systems and Industrial Engineering, University of Arizona 2Computer Science, George Mason University PI Contacts: [email protected]; 1-520-626-9530 [email protected]; 1-520-621-6548 [email protected]; 1-703-993-9546 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
and Industrial Engineering, University of Arizona 2Computer Science, George Mason University PI Contact: [email protected]; 1-520-626-9530 AFOSR DDDAS PI Meeting Jan. 2016
Agenda • Previous project • New project – New challenges – How DDDAS is addressed
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Overview of Previous Project Motivation: TUS 1- Project (23-mile long area of the border in Sasabe, AZ)
Problem: A highly complex, uncertain, dynamically changing border environment
Goal: Develop a simulation-based planning and control system for surveillance and crowd control via collaborative UAVs/UGVs
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
DDDAMS-based Planning and Control Framework
Khaleghi, A. M., Xu, D., Wang, Z., Li, M., Lobos, A., Liu, J., & Son, Y. (2013). A DDDAMS-based Planning and Control Framework for Surveillance and Crowd Control via UAVs and UGVs. Expert Systems with Applications, 40, 7168-7183. Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Multi-resolution Data
Challenge:
Aggregate multi-resolution data
Approach:
UAVs’ global perception and UGVs’ detailed perception Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Framework of Developed Methodology
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Detection Module: Simulated ODROID USB-CAM 720P HD (16:9): 1280x720p @ 30 fps FOV: 72
GoPro HERO 3- Tarot Gimbal Stabilizer HD (16:9): 1280x720p @ 120 ~ 25 fps FOV(x): 64.4 ; FOV(y): 37.2
EDD
: UGV’s Effective Detection Depth DR : Detection range for UGV FOV(y) FOV (x) A : Detection range for UAV DR FOV : Field of view : Distance h(G ) ( A) : Altitude h
G
h
DR(y)
G
G
DRmin 2 hmin tan FOV / 2 DR
G
G
G
2 hmax tan FOV / 2 DRmax
DR(x)
DR ( A) 2h( A) tan FOV / 2
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Detection Module (UAV & UGV): Actual
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Framework of Developed Methodology
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Crowd Tracking Module
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Case Study
Bayesian estimation: 75% Less computation time Comparable/Better prediction performance Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Framework of Developed Methodology
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Control Strategies associated with Motion Planning •
Given selected destination of UAV/UGV, find the path that optimizes a certain combination of criteria
(a) minimize travel distance
•
(b) minimize elevation penalty (fuel consumption)
(c) minimize the weighted average of (a) and (b)
Weighted average of the multiple objectives Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
System Implementation • Agent-based HIL simulation • UAVs and UGVs • Social force model and GIS
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Agent-based Hardware-in-the-loop Simulation Agent-based Simulation Repast Simphony with 3D GIS
Sensory Data (e.g. GPS) Assembled UAV (APM:Copter / Arducopter)
Hardware Interface: MAVproxy
Control Commands (MAVLink Messages) Assembled UGV (APM:Rover / Ardurover)
Wi-Fi / XBee PRO 900HP; APM one Khaleghi, A. M., Xu, D., Lobos, A., Minaeian, S., Son, Y. -J., & Liu, J. (2013). Agent- based hardware-in-the-loop simulation for modeling UAV/UGV surveillance and crowd control system. In Proceedings of the winter simulation conference 2013, Washington, DC, USA. Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Assembled UAV (Arducopter), AR.Drone, X8+ UAV, and UGV
Navigate GPS waypoints autonomously using APM autopilot set (Arduino-based Autopilot- APM 2.5) Microcontroller (ATMEGA2560) Low-power Atmel 8-bit AVR RISC-based 256KB ISP flash memory 8KB SRAM 4KB EPROM Throughput: 16 MIPS at 16MHz
Motion Processing Unit (MPU-6000) 3-Axis Gyro 3-Axis Accelerometer Barometric pressure sensor (SM5611)
Global Positioning System (GPS) GPS update rate: 5hz (5X per second) Using GPS unit, the UAV has an outdoor navigation accuracy of about +/- 5 meters Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Social Force Model for Crowd Motion Direction/angle and walking speed of humans has been modeled using 2 heuristics based on their visual data (Moussaïd et al., 2011) • Heuristic 1: Minimize the angle/direction of each individual by minimizing the distance to its destination 2 min d( ) dmax f 2 ( ) 2dmax f ( )cos( 0 )
• Heuristic 2: Change walking speed of human to avoid collisions v min(v0 ,
dh
)
human field of view: ( , ) (for e.g. 90 ,90) maximum range of view: dmax (for e.g. 10 m) human comfortable walking speed: v0 (for e.g. 1.5 m / s) distance to obstacle: dh
relaxation time time requires to adopt new behavior : (for e.g. 1 sec)
Khaleghi, A. M., Xu, D., Lobos, A., Minaeian, S., Son, Y. -J., & Liu, J. (2013). Agent- based hardware-in-the-loop simulation for modeling UAV/UGV surveillance and crowd control system. In Proceedings of the winter simulation conference 2013, Washington, DC, USA. Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Publications: Journals and Book Chapters •
•
•
•
•
S. Minaeian, J. Liu, Y. Son, Vision-based Target Detection and Localization via a Team of Cooperative UAV and UGVs, IEEE Transactions on Systems, Man, and Cybernetics: Systems (Special Issue on Biomedical Robotics and Bio-mechatronics Systems and Application), Accepted, 2015. A. Khaleghi, D. Xu, Z. Wang, M. Li, A. Lobos, J. Liu, Y. Son, A DDDAMS-based Planning and Control Framework for Surveillance and Crowd Control via UAVs and UGVs, Expert Systems with Applications, 40, 2013, 7168-7183. Yifei Yuan, Zhenrui Wang, Mingyang Li, Young-Jun Son, Jian Liu, DDDAS-based Information-Aggregation for Crowd Dynamics Modeling with UAVs and UGVs, Frontiers in Robotics and AI (Sensor Fusion and Machine Perception Section), 2:8, 2015, 1-10. A. Khaleghi, D. Xu, S. Minaeian, M. Li, Y. Yuan, C. Vo, A. Mousavian, J. Lien, J. Liu, and Y. Son, UAV/UGV Surveillance and Crowd Control via Hardware-in-the-loop DDDAMS System, Darema, F., Douglas, C (Eds.), Springer (under review) Online Collision Prediction Among 2D Polygonal and Articulated Obstacles, Yanyan Lu, Zhonghua Xi and Jyh-Ming Lien, International Journal of Robotics Research (IJRR), Accepted, 2015.
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Publications: Proceedings (1) •
•
•
•
• •
Minaeian, S., Yuan, Y., Liu, J., and Son, Y., 2015, “Human-in-the-Loop Agent-based Simulation for Improved Autonomous Surveillance using Unmanned Vehicles,” Proceedings of 2015 Winter Simulation Conference, Huntington Beach, CA (poster) Continuous Visibility Feature, Guilin Lu, Yotam Gingold, and Jyh-Ming Lien, in the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015, Boston, MA, USA Semantically Guided Location Recognition for Outdoors Scenes, Arsalan Mousavian, Jana Kosecka and Jyh-Ming Lien, Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), May 2015, Seattle, WA, USA Khaleghi, A., Xu, D., Minaeian, S., Yuan, Y., Liu, J., and Son, Y., 2015, “Analysis of UAV/UGV Control Strategies in a DDDAMS-based Surveillance System,” Proceedings of 2015 IIE Annual Meeting, Nashville, TN. Minaeian, S., Liu, J., and Son, Y., 2015, “Crowd Detection and Localization Using a Team of Cooperative UAV/UGVs,” Proceedings of 2015 IIE Annual Meeting, Nashville, TN. Khaleghi, A., Xu, D., Minaeian, S., Li, M., Yuan, Y., Liu, J., Son, Y., Vo, C., and Lien, J., 2014, “A DDDAMS-based UAV and UGV Team Formation Approach for Surveillance and Crowd Control,” Proceedings of 2014 Winter Simulation Conference, Savannah, GA.
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Publications: Proceedings (2) •
•
•
•
•
•
Khaleghi, A., Xu, D., Minaeian, S., Li, M., Yuan, Y., Liu, J., and Son, Y., 2014, “A Comparative Study of Control Architectures in UAV/UGV-based Surveillance System,” Proceedings of 2014 IIE Annual Meeting, Montreal, Canada. Wang, Z., Li, M., Khaleghi, A., Xu, D., Lobos, A., Vo, C., Lien, J., Liu, J., and Son, Y., 2013, “DDDAMS-based Crowd Control via UAVs and UGVs,” Procedia Computer Science 18, 2028–2035 (Proceedings of 2013 International Conference on Computational Science, Barcelona, Spain). Khaleghi, A., Xu, D., Lobos, A., Minaeian, S., Son, Y., and Liu, J., 2013, “Agent-based Hardware-in-the-Loop Simulation for Modeling UAV/UGV Surveillance and Crowd Control System,” Proceedings of 2013 Winter Simulation Conference, Washington DC. Wang, Z., Li, M., Khaleghi, A., Xu, D., Lobos, A., Vo, C., Lien, J., Liu, J., and Son, Y., 2013, “DDDAMS-based Crowd Control via UAVs and UGVs,” Procedia Computer Science 18, 2028–2035 (Proceedings of 2013 International Conference on Computational Science, Barcelona, Spain). Vo, C., McKay, S., Garg, N., and Lien, J., 2012, “Following a Group of Targets in Large Environments”, Proceedings of the Fifth International Conference on Motion in Games, Springer. Vo, C., and Lien, J., 2012, “Group Following in Monotonic Tracking Regions”, Proceedings of the 22nd Fall Workshop on Computational Geometry, 2012. Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
New Project • New challenges • How DDDAS is addressed
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
3-Level Surveillance Framework
High altitude level (HAL)
Low altitude level (LAL)
Surface level (SL)
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
New Major Problems •
3-D Surveillance System for aerial and ground targets
•
Latency in detection, recognition and identification of targets
•
Heterogeneous data from complex targets by 3 levels of sensors
•
Multi-level information aggregation
•
Active or pro-active surveillance strategies
•
Realistic scenarios and model validation based on data collection from our research partners (AFRL, Raytheon, University Partners …)
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
3-Level Measurement System in Border Surveillance 3-Levels
Types
Sensors
Measurement Data
ElectroOptical/Infrared (EO/IR)
High altitude level (HAL)
SAR Image
Synthetic Aperture Radar (SAR)
EO/IR Image Spectral Image
Remote Sensing
Low altitude level (LAL)
Surface level (SL)
Surveillance Camera
Lidar Image Thermal Images
Mobile Sensors
Fixed Sensors
Magnetic Data
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Agent Model of Sensors (Generic) A generic modeling framework of sensors for the surveillance application Surveillance Behavior Sub-Model
Availability SubModel
Target appears in the system
Adjust the Detection range; Set the threshold Receive signals / messages from the controller
Change location parameters of sensors in order for chasing foe targets
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Signal Processing Sub-Model
Previous EnhancedDDDAMS-based DDDAMS-basedFramework Framework
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
1.0 - Target DRI Detection, Recognition, and Identification of hostile targets (i.e. traffickers) and civilian targets. Discovery the presence of a person, object, or phenomenon*
• •
Sensing technologies: thermal technologies, radar, etc. Motion detection, optical flow, etc.
* Military, U. S. (2005). Dictionary of military and associated terms. US Department of Defense. Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
1.0 - Target DRI Detection, Recognition, and Identification of hostile targets (i.e. traffickers) and civilian targets.
Determination of the nature of a detected person, object or phenomenon, and its class or type *
• • • •
Image Feature Extraction Functional Discriminant Analysis for Time series sensor data Information-aggregation method for Multiple Sensor data Multivariate Classification Method
* Military, U. S. (2005). Dictionary of military and associated terms. US Department of Defense.
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
1.0 - Target DRI Detection, Recognition, and Identification of hostile targets (i.e. traffickers) and civilian targets.
Discrimination between recognizable objects as being friendly or enemy *
• •
Gaussian Mixture Model for Objective Identification BDI (Belief–Desire–Intention) framework
* Military, U. S. (2005). Dictionary of military and associated terms. US Department of Defense. Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Emerging Challenges - Target DRI Sensor Type
Vehicle
Armed People
Animals
New Observation
Image
Training Thermal Sensor
Information Aggregated Classifier
Magnetic Sensor
Predicted Class
Underground, … Radar, etc
…
…
Updating Real Class
Challenge: Feature Extraction Potential Method: Discriminant Analysis
Feature 2
Sample Image Data
V V V V V AV A V AA A AP AP A A AP A APAP AP Feature 1
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Challenge: Classification under DDDAS framework Potential Methods: SVM, KNN, etc.
Extended Belief-Desire-Intention Framework Bratman, 1987 Rao and Georgeff, 1998 Zhao and Son, 2008
Lee, S., Y.-J. Son, and J. Jin (2010), Integrated human decision making and planning model under extended belief-desire-intention framework, ACM Transactions on Modeling and Computer Simulation, 20(4), 23(1)~23(24). Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Model of Drug Traffickers based on BDI framework Developed behavior models of drug traffickers and ground patrol agents together with environmental conditions will provide rich scenarios Behavioral Models of Drug Traffickers Decision 1
Decision 2
Decision 3
Selection at Departure
Route Choice
En-Route Planning
Time
Fastest Way
Hiding/Stopping
Location
Rugged Road
Sudden Direction Change
Behavior Models of Border Patrols
Environment Conditions: weather, terrains
3-Level Sensor Networks
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Agent Models for Patrol Agent and Sensors
Patrol Agents
Sensors
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
2.0- Pattern Processing Recognition and Prediction of objective, behavior and route patterns of hostile targets
•
Spatial Optimization Method Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Emerging Challenges - Pattern Processing … … … …
…
0
0
1
0
…
0
1
0
0
…
2
0
0
0
…
0
0
3
0
…
… … … … …
3D Grid Matrix with Categorized Values e.g. 0: Unoccupied; 1: Hostile Crowd; 2: Friendly Crowd;3: Animals
Challenge: Computational complexity, modeling the interactions between targets.
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
3.0- Mission Control Resource allocation and Motion planning for controlling hostile targets to terminate/mitigate their activities
Predicting Target’s Behavior
Optimized Sensors Allocation
Persistent Surveillance
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Emerging Challenges - Mission Control Challenge: risk assessment and uncertainty quantification
Target Pattern Prediction
Target1
Target Risk Assessment
Target3
+
Target2 Updating
Target
Risk
1
0.75
2
0.97
3
0.23
…
…
Allocation Optimization
New Observation
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
DDDAS in Modules: Target DRI APPLICATION
ALGORITHMS
3-D spatial model
Multi-scale model with more distributions to model spatial, temporal and preference
Heterogeneous models for both civilian and hostile targets
Mixture of regressions analysis algorithm
Multi-scale model for dynamics and BDI for human decisions
Information fusion algorithm for target Identification
Model uncertainty for miss-classification
Dynamic model parameter updating for hostile targets
Sensors and vehicles are modeled as agents
Model uncertainty for miss-classification
Online camera motion strategies in 3D
Image and geometry registration algorithms
Fuse high-level motion command and dynamic environmental information
Online motion planning methods using estimated “earliest collision time” in 3D
MEASUREMENT
3-D data
SOFTWARE Bayesian updating software module for location, dynamic and sensor/vehicle status modeling
City-size midresolution dynamic data by Aerostat Heterogeneous model estimation software module Mid-range Highresolution dynamic data by UAVs
Adjacent highresolution dynamic data by UGVs
Status of sensors and vehicles measured
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Extension of Repast Simphony®-based HIL simulator with more sensors, a richer set of sensory and commands data
Detailed models of agents (e.g. sensors) in a Physics-based simulation software (e.g. Gazebo, MAK, AGI, etc.)
DDDAS in Modules: Pattern Processing APPLICATION
ALGORITHMS
3-D spatial model
Patterns extracted as statistics extracted from algorithm
3-D data
Assume hostile traffickers detected
Mixture of regressions analysis algorithm
City-size midresolution dynamic data by Aerostat
Heterogeneous models for location, dynamics and preference pattern recognition
Model uncertainty for pattern misrecognition
Impacts of sensors/vehicles status considered
Information fusion algorithm gives statistical patterns
Pattern extraction and recognition algorithm will be developed
Algorithm to update prior knowledge on patterns will be developed
MEASUREMENT
SOFTWARE Bayesian updating software module for location, dynamic and sensor/vehicle status modeling
Heterogeneous model estimation software module Mid-range Highresolution dynamic data by UAVs
Adjacent highresolution dynamic data by UGVs
Status of sensors and vehicles measured
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Extension of Repast Simphony®-based HIL simulator with more sensors, a richer set of sensory and commands data Detailed models of agents (e.g. sensors) in a Physics-based simulation software (e.g. Gazebo, MAK, AGI, etc.)
DDDAS in Modules: Mission Control APPLICATION
3-D pattern model
ALGORITHMS
Control index considered for 3-D optimality
Augmenting the Surface Level’s vision
3-D data
City-size midresolution dynamic data by Aerostat
Heterogeneous models
Controllability is estimated via repeated simulations
MEASUREMENT
Algorithm uncertainty caused by both model estimation uncertainty and sensor/vehicle uncertainty
Mid-range Highresolution dynamic data by UAVs
Adjacent highresolution dynamic data by UGVs
Algorithm updated considering agent dynamics, patternshifting and sensor/vehicle status
Status of sensors and vehicles measured
Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
SOFTWARE Bayesian updating software module for location, dynamic and sensor/vehicle status modeling
Heterogeneous model estimation software module
Extension of Repast Simphony®-based HIL simulator with more sensors, a richer set of sensory and commands data
Detailed models of agents (e.g. sensors) in a Physics-based simulation software (e.g. Gazebo, MAK, AGI, etc.)
Acknowledgements Sponsor: Air Force Office of Scientific Research FA9550-12-1-0238 (DDDAS); 15RT1016 (New) Program Manager: Dr. Frederica Darema PIs: Young-Jun Son1, Jian Liu1, Jyh-Ming Lien2 Students (Previous Project): A. Khaleghi1, D. Xu1, S. Minaeian1, Y. Yuan1, M. Li1, and C. Vo2 Students (New Project): S. Minaeian1, Y. Yuan1, S. Lee1, and J. Han1 1Systems and Industrial Engineering, University of Arizona 2Computer Science, George Mason University PI Contacts: [email protected]; 1-520-626-9530 [email protected]; 1-520-621-6548 [email protected]; 1-703-993-9546 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson
