mitsubishi electric research laboratories
MITSUBISHI ELECTRIC RESEARCH LABORATORIES Cambridge, Massachusetts, USA
Using Multi-User Multi-Touch Tables to Visualize GIS Data
Alan Esenther, Kathy Ryall Technology Lab Mitsubishi Electric Research Labs, Cambridge, MA, USA Masakazu Furuichi Modeling & Simulation, Information Systems Dept Mitsubishi Electric Corp, Kamakura, Japan Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
DiamondTouch (DT): Novel Technology Platform Overcoming limitations of the mouse… – Multi-user • Simultaneous multiple users • Supports up to four users at a time
– Multi-touch • Each user can touch with multiple fingers at the same time • Enables rich gestures (unlike mouse/stylus)
– Toucher identification • It doesn’t just allow multiple touchers – it distinguishes them • Identifies which user is touching each point (unlike tables based on cameras, IR, or pressure)
– Direct-input • Input and Display spaces are superimposed • Allows simplicity of directly touching and interacting with content
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
DiamondTouch: Hardware Overview •
Basic Components – Table (touch surface) is a transmitter array – Chairs (or a conductive pad) are receivers
•
Receiver Transmitters
User capacitively couples a signal between the table and the receiver – Supports seated or standing interaction – Stylus-based interaction also possible – Transmitter signals are interpolated to 2736x2048 input resolution
•
DT most-often used as horizontal tabletop display – Front-projection from above (any projector will do) – Can be embedded in other forms (e.g. vertical, dashboard, etc)
•
Powered by 5 volt USB cable MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Multi-User? Multi-Touch? Toucher ID? Direct Input?
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Multi-Touch: Good for Rich Gestural Interactions • Supports all mouse functionality – including mouse-overs (vs mouse-drag), right/middle-drag, scroll-wheel, precision-input, etc – first/only direct touch surface to achieve this
• Can build new functionality on top of mouse – e.g. fist-swipe or fist-drag can be mapped to anything
• Flicking with 2 fingers • Simultaneously specify multiple points – e.g. piano chords - can’t do this with a mouse – e.g. Change the size and location of a region simultaneously
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Mouse Emulation Challenges + Current Solutions
• Specifying a particular pixel, finger obscuring content, moving the mouse without dragging: – “Precision-Hover” mode
• Timing and spacing (for double-tapping ,etc) – DiamondTouch settings independent of mouse settings (customizable)
• Right and middle mouse buttons – Tap with second finger to toggle right mouse button – Tap twice with second finger to toggle middle mouse button
• Desktop Incorporation – System tray icon with context menu; special gesture for muting; audio feedback MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Customer Usage • •
Modeling & Simulation, Information Systems Dept, Mitsubishi Elec Corp, Kamakura, Japan In Development: “Decision Support System for Disaster Response”
•
•
Used ArcGIS with maps and satellite images of December 2004 flooding in Indonesia Collaboratively analyzed on a DiamondTouch table MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Decision Support System for Disaster Response • Used before-after image processing to inform damage estimates
Tsunami images (c) 2005 DigitalGlobe Inc. Petroleum User Group 2007
MITSUBISHI ELECTRIC
Changes for the better
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Decision Support System for Disaster Response • Combined with damage reports from on-site personnel
• and inspection of satellite data
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Decision Support System for Disaster Response • To collaboratively determine usable routes and helicopter landing locations
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Emergency Response and Situational Awareness •
Shared situational awareness: – All participants perceive elements and comprehend meaning/implications – can only be achieved with effective communications.
• •
A shared tabletop environment allows for important face-to-face interactions without which subtleties of communication (gestures, expressions) could be lost. Direct Input – clearly communicates to other participants who is doing what. Facilitates building shared mental model.
•
Simultaneous users – In an emergency, you don’t want to take turns. • With customizations, everyone can do things at the same time. • If using the mouse, turn-taking is required, but a first-to-touch-wins policy avoids chaotic mouse cursor movement.
•
Multi-finger input – Key to rich gestures for advanced functionality MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Enhancing existing applications •
Customizing function – Drawing with different symbols (emergency unit deployments, routes) – Intelligent Undo – “Privileged” buttons (e.g. emergency Coordinator has final say)
•
Customizing content – Menus, bookmarks, tooltips and help tailored to the particular toucher’s professional background and to the current situation
•
Customizing appearance – Change font, color, language, size, thickness, orientation based on who is touching
•
Customizing group input – Simultaneous touches required to proceed – Modal input sequences: one person can annotate while another is erasing and another is browsing different map layers – Logging and audit trail creation
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Combining DiamondTouch with ESRI Software •
3 Directions 1 - Run existing ESRI software as is (convert touches to mouse input) 2 - Write extensions to existing ESRI software to add multi-user/multitouch capabilities 3 - Use ArcEngine or ArcWeb Services to create new multi-user/multi-touch utilities and applications
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
1. DiamondTouch & Unmodified ESRI applications: DTMouse Utility •
DiamondTouch Mouse Emulator – Allows you to run unmodified ESRI (or any) software on the table as is – Converts touch inputs to mouse inputs – Provides functionality of a 3-button mouse and mousewheel – Allows precision input – Coordinates multiple touchers • First to touch is the mouse – others are ignored
– One user at a time, but no need to explicitly hand off control – no physical device (i.e., mouse, keyboard) to pass back and forth
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
2 - DiamondTouch & Extending Existing ESRI Applications •
Example Prototype – DiamondTouch extension for ArcMap – Works in conjunction with DTMouse mouse emulator
– Provides special buttons for simultaneous operations (line and marker symbol creation) MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
3 - DiamondTouch & ESRI: New Application Development •
Use ArcEngine or ArcWeb Services to create new multi-user/multi-touch utilities and applications
• •
Example: Lensing to reveal different layers. Example: Simultaneously zooming subwindows into different regions
•
Future work: Exploit DiamondTouch capabilities within ESRI-based apps; e.g.: – – – –
Simultaneous input Audit and logging Permission/control on a per-user basis Visualizing parallel geoprocessing MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Challenges/Issues/Findings •
Mouse concepts are deeply embedded in modern operating systems – There is only one mouse focus – O.S. can’t handle 2 mouse-down events in a row, without a mouse-up (e.g. scrollbars)
•
Integration with O.S. is critical – Need mouse emulation often for existing buttons, menus, etc – Need to suppress the mouse when custom software is interpreting the touch
•
Rotating content to face different users – Especially text – COM-based software assumes normal orientation • clipping regions, windows, invalidation regions, etc are all rectangular
• • •
ArcGIS exposes a lot of useful APIs ArcGIS computation and rendering times are too slow to do in-place geo-processing The ideal tabletop is big enough for multiple people to sit at, but small enough so that everyone can reach anywhere on it MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Multi-surface GIS • Synchronized content
• Synchronized displays
• WebService-based synchronization of different applications (shown here on same surface)
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Summary •
DiamondTouch – Novel Direct-Touch development platform – Multi-user, Multi-touch, User-Differentiation
• • • •
Rich gestures require reliable multi-finger detection Careful integration of full-featured mouse emulation is critical Reliable multiple-user support requires ability to distinguish touchers Several ways to integrate with GIS applications
The computer mouse will continue to be the primary input mechanism, but be aware of it’s inherent limitations Technology should not get in the way: debris-tolerant, walk-up interaction style, portable/durable MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Questions? http://www.merl.com/projects/DiamondTouch/ http://diamondtouch.merl.com/stuff/ esenther @ merl.com
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
Using Multi-User Multi-Touch Tables to Visualize GIS Data
Alan Esenther, Kathy Ryall Technology Lab Mitsubishi Electric Research Labs, Cambridge, MA, USA Masakazu Furuichi Modeling & Simulation, Information Systems Dept Mitsubishi Electric Corp, Kamakura, Japan Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
DiamondTouch (DT): Novel Technology Platform Overcoming limitations of the mouse… – Multi-user • Simultaneous multiple users • Supports up to four users at a time
– Multi-touch • Each user can touch with multiple fingers at the same time • Enables rich gestures (unlike mouse/stylus)
– Toucher identification • It doesn’t just allow multiple touchers – it distinguishes them • Identifies which user is touching each point (unlike tables based on cameras, IR, or pressure)
– Direct-input • Input and Display spaces are superimposed • Allows simplicity of directly touching and interacting with content
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
DiamondTouch: Hardware Overview •
Basic Components – Table (touch surface) is a transmitter array – Chairs (or a conductive pad) are receivers
•
Receiver Transmitters
User capacitively couples a signal between the table and the receiver – Supports seated or standing interaction – Stylus-based interaction also possible – Transmitter signals are interpolated to 2736x2048 input resolution
•
DT most-often used as horizontal tabletop display – Front-projection from above (any projector will do) – Can be embedded in other forms (e.g. vertical, dashboard, etc)
•
Powered by 5 volt USB cable MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Multi-User? Multi-Touch? Toucher ID? Direct Input?
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Multi-Touch: Good for Rich Gestural Interactions • Supports all mouse functionality – including mouse-overs (vs mouse-drag), right/middle-drag, scroll-wheel, precision-input, etc – first/only direct touch surface to achieve this
• Can build new functionality on top of mouse – e.g. fist-swipe or fist-drag can be mapped to anything
• Flicking with 2 fingers • Simultaneously specify multiple points – e.g. piano chords - can’t do this with a mouse – e.g. Change the size and location of a region simultaneously
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Mouse Emulation Challenges + Current Solutions
• Specifying a particular pixel, finger obscuring content, moving the mouse without dragging: – “Precision-Hover” mode
• Timing and spacing (for double-tapping ,etc) – DiamondTouch settings independent of mouse settings (customizable)
• Right and middle mouse buttons – Tap with second finger to toggle right mouse button – Tap twice with second finger to toggle middle mouse button
• Desktop Incorporation – System tray icon with context menu; special gesture for muting; audio feedback MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Customer Usage • •
Modeling & Simulation, Information Systems Dept, Mitsubishi Elec Corp, Kamakura, Japan In Development: “Decision Support System for Disaster Response”
•
•
Used ArcGIS with maps and satellite images of December 2004 flooding in Indonesia Collaboratively analyzed on a DiamondTouch table MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Decision Support System for Disaster Response • Used before-after image processing to inform damage estimates
Tsunami images (c) 2005 DigitalGlobe Inc. Petroleum User Group 2007
MITSUBISHI ELECTRIC
Changes for the better
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Decision Support System for Disaster Response • Combined with damage reports from on-site personnel
• and inspection of satellite data
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Decision Support System for Disaster Response • To collaboratively determine usable routes and helicopter landing locations
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Emergency Response and Situational Awareness •
Shared situational awareness: – All participants perceive elements and comprehend meaning/implications – can only be achieved with effective communications.
• •
A shared tabletop environment allows for important face-to-face interactions without which subtleties of communication (gestures, expressions) could be lost. Direct Input – clearly communicates to other participants who is doing what. Facilitates building shared mental model.
•
Simultaneous users – In an emergency, you don’t want to take turns. • With customizations, everyone can do things at the same time. • If using the mouse, turn-taking is required, but a first-to-touch-wins policy avoids chaotic mouse cursor movement.
•
Multi-finger input – Key to rich gestures for advanced functionality MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Enhancing existing applications •
Customizing function – Drawing with different symbols (emergency unit deployments, routes) – Intelligent Undo – “Privileged” buttons (e.g. emergency Coordinator has final say)
•
Customizing content – Menus, bookmarks, tooltips and help tailored to the particular toucher’s professional background and to the current situation
•
Customizing appearance – Change font, color, language, size, thickness, orientation based on who is touching
•
Customizing group input – Simultaneous touches required to proceed – Modal input sequences: one person can annotate while another is erasing and another is browsing different map layers – Logging and audit trail creation
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Combining DiamondTouch with ESRI Software •
3 Directions 1 - Run existing ESRI software as is (convert touches to mouse input) 2 - Write extensions to existing ESRI software to add multi-user/multitouch capabilities 3 - Use ArcEngine or ArcWeb Services to create new multi-user/multi-touch utilities and applications
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
1. DiamondTouch & Unmodified ESRI applications: DTMouse Utility •
DiamondTouch Mouse Emulator – Allows you to run unmodified ESRI (or any) software on the table as is – Converts touch inputs to mouse inputs – Provides functionality of a 3-button mouse and mousewheel – Allows precision input – Coordinates multiple touchers • First to touch is the mouse – others are ignored
– One user at a time, but no need to explicitly hand off control – no physical device (i.e., mouse, keyboard) to pass back and forth
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
2 - DiamondTouch & Extending Existing ESRI Applications •
Example Prototype – DiamondTouch extension for ArcMap – Works in conjunction with DTMouse mouse emulator
– Provides special buttons for simultaneous operations (line and marker symbol creation) MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
3 - DiamondTouch & ESRI: New Application Development •
Use ArcEngine or ArcWeb Services to create new multi-user/multi-touch utilities and applications
• •
Example: Lensing to reveal different layers. Example: Simultaneously zooming subwindows into different regions
•
Future work: Exploit DiamondTouch capabilities within ESRI-based apps; e.g.: – – – –
Simultaneous input Audit and logging Permission/control on a per-user basis Visualizing parallel geoprocessing MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Challenges/Issues/Findings •
Mouse concepts are deeply embedded in modern operating systems – There is only one mouse focus – O.S. can’t handle 2 mouse-down events in a row, without a mouse-up (e.g. scrollbars)
•
Integration with O.S. is critical – Need mouse emulation often for existing buttons, menus, etc – Need to suppress the mouse when custom software is interpreting the touch
•
Rotating content to face different users – Especially text – COM-based software assumes normal orientation • clipping regions, windows, invalidation regions, etc are all rectangular
• • •
ArcGIS exposes a lot of useful APIs ArcGIS computation and rendering times are too slow to do in-place geo-processing The ideal tabletop is big enough for multiple people to sit at, but small enough so that everyone can reach anywhere on it MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Multi-surface GIS • Synchronized content
• Synchronized displays
• WebService-based synchronization of different applications (shown here on same surface)
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Summary •
DiamondTouch – Novel Direct-Touch development platform – Multi-user, Multi-touch, User-Differentiation
• • • •
Rich gestures require reliable multi-finger detection Careful integration of full-featured mouse emulation is critical Reliable multiple-user support requires ability to distinguish touchers Several ways to integrate with GIS applications
The computer mouse will continue to be the primary input mechanism, but be aware of it’s inherent limitations Technology should not get in the way: debris-tolerant, walk-up interaction style, portable/durable MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
MITSUBISHI ELECTRIC RESEARCH LABORATORIES
Questions? http://www.merl.com/projects/DiamondTouch/ http://diamondtouch.merl.com/stuff/ esenther @ merl.com
MITSUBISHI ELECTRIC
Changes for the better Petroleum User Group 2007
