Kevin Jin DDDAS PI meeting Jan2016
Dynamic Data-Driven and Real-Time Verifica4on for Industrial Control System Security @IIT Campus Microgrid
1
PI: Dong (Kevin) Jin Ph.D. Students: Christopher Hannon and Xin Liu Program Director: Dr. Frederica Darema DDDAS Program PI Mee4ng, January 2016
Industrial Control Systems (ICS) • Control many cri4cal infrastructures – e.g., weapons systems, aerospace, gas and oil distribu4on networks, wastewater treatment, transporta4on systems …
• Modern ICS increasingly adopt Internet technology to boost control efficiency, e.g., smart grid LOADS
SITES
DISTRIBUTION TRANSFORMER
DISTRIBUTION SUBSTATION TRANSMISSION
GENERATION
Next Genera4on of Power Grid 2
More Efficient or More Vulnerable? Communica4on Path Markets Retailer/ Wholesaler
WAMS
Enterprise Bus RTO SCADA
ISO/RTO Par4cipa4on
Transmission Ops
Enterprise Bus Transmission SCADA
Asset Mgmt
DMS
CIS
Demand Response
MDMS
Generators
Bulk Genera4on 3
Retail Energy Provider
CIS Billing
Home/Building Manager
Enterprise Bus
Aggregator Metering System
Distribu4on SCADA
Others
Internet / e-business Wide Area Network
Plant Control System
Third-Party Provider
Billing
Internet / e-business Market Services Interface
U5lity Provider
Distribu5on Ops
EMS
EMS
Energy Market Clearing hosue
Service Providers
Opera4ons RTO/ISO Ops
Aggregator
Network
Substa5on LANs
Data Collector
Substa4on Controller
Field Area Networks
Field Device
Electric Vehicle
Energy Services Interface
Distributed Genera4on
Premises Networks
Meter
Customer Equipment Substa4on Device
Electric Storage
Transmission
Distributed Genera4on
Appliances
Customer EMS
Distribu4on
Picture source: NIST Framework and Roadmap for Smart Grid Interoperability Standards
Electric Storage
Thermostat
Customer
Cyber Threats in Power Grids • 245 incidents, reported by ICS-CERT • 32% in energy sector • 80,000 residents in western Ukraine • 6 hours, lost power on Dec 23, 2015
Picture source: 1. Na4onal Cybersecurity and Communica4ons Integra4on Center (NCCIC). ICS-CERT Monitor Sep 2014 – Feb 2015 2. hep://dailysignal.com/2016/01/13/ukraine-goes-dark-russia-aeributed-hackers-take-down-power-grid/
4
Protec4on of Industrial Control Systems • Commercial of-the-shelf products – e.g., firewalls, an4virus sohware – fine-grained protec4on at single devices only
• How to check system-wide requirements – Security policy (e.g., access control) – Performance requirement (e.g., end-to-end delay)
• How to safely incorporate exis4ng networking technologies in control system infrastructures? – real-4me, large-scale, no interference with normal opera4ons … 5
Our Approach: DDDAS-based Real-Time System Verifica4on Policy Engine
ICS Applica5on Models System Framework
Dynamic Model Update/Selec3on
Verifica3on
Diagnosis • •
Vulnerabili*es Errors
Network Models topology
network-layer states (e.g., forwarding tables)
Dynamic Network Data (topology, forwarding tables … ) Dynamic Applica4on Data (control updates … ) User-specified Policy (security, performance …) 6
Verified System Updates
Network-Layer Verifica4on VeriFlow Operation Prior Work •
Network Controller
•
New rules
VeriFlow Generate equivalence classes
Generate forwarding graphs
• Run queries
•
FlowChecker [Al-Shaer et al.,SafeConfig2010] HeaderSpaceAnalysis [Kazemian et al.,NSDI2012] Anteater [Mai et al.,SIGCOMM2011] VeriFlow [Khurshid et al., NSDI2012]
Rules violating network invariant(s)
Good rules
Diagnosis report • Type of invariant violation • Affected set of packets 4/3/2013
Department of Computer Science, UIUC
7
11
Challenges — Timing Uncertainty Network devices are asynchronous and distributed in nature Controller'
Remove&rule&1&
Install'rule'2'
rule%1% Switch'A'
8
rule%2%
Switch'B'
Challenges — Timing Uncertainty
Controller'
Remove&rule&1& (delayed)&
Install'rule'2'
rule%1%
Packet' Switch'A'
rule%2%
Loop-freedom Viola4on 9
Switch'B'
Uncertainty-aware Modeling • Naively, represent every possible network state O(2^n) • Uncertain graph: represent all possible combina4ons
10
Update Synthesis via Verifica4on 2 1
3
4
A should reach B
Enforcing dynamic correctness with heuris4cally maximized parallelism 11
Wenxuan Zhou, Dong Jin, Jason Croh, Maehew Caesar, and P. Brighten Godfrey. “Enforcing Customizable Consistency Proper4es in Sohware-Defined Networks.” NSDI 2015.
OK, but…
Can the system “deadlock”?
• Proved classes of networks that never deadlock • Experimentally rare in prac4ce! • Last resort: heavyweight “fallback” like consistent updates [Reitblae et al, SIGCOMM 2012]
Number$of$Rules$ in$the$Network$
Is it fast? 25000$
6
12
25000
//$
//$
//$
20000 20000$ 15000 15000$ 10000 10000$
}
5000 5000$
6 6 6
8 0$0 0 8 2 8 7/22/2014$ 7/22/2014$ 8 22:00:00$ 22:00:02$
Immediate Update Immediate Update GCC ImmediateUpdates Update GCC Consistent GCC Consistent Updates End Immediate UpdateUpdates Consistent End Comple?on$ GCC CCG End Time$ Consistent Updates End 14 16 End 14 //$14 12 16 14 16 10 16 7/23/2014$ 7/23/2014$ 147/23/2014$ 16
//$
10
7/22/2014$ 23:00:00$ 4
10 //$ 6 10 7/22/2014$ 10
23:00:02$
12 12 7/23/2014$ 12
12
8
0:00:00$
Time$
0:00:02$
1:00:00$
1:00:02$
What’s next? • • • • •
Instability Loss of Load Synchroniza4on Failure Con4ngency Loss of Economics
Impact Virtualized U5lity Network 1 Frequency Control Cross-Layer Verifica5on Intrusion Detec5on
Power Control Applica5ons Demand Response
Frequency Control
State Es4ma4on
Topology Control
…
Rou4ng
…
Cyber Resources SCADA Servers
Field Devices
Communica4on Networks
Virtualized U5lity Network 2 Demand Response
Virtualized U5lity Network 3 State Es4ma4on
Control Center
Virtualized U5lity Network 4 Topology Control
Cyber AMacks Denial of Service
False Data Injec4on
Malware
Insider Aeack
…
(a) Current Power Grid: Poten4al Cyber Aeacks and Their Implica4ons
(b) Future SDN-enabled Power Grid: A Cyber-Aeack-Resilient Plauorm
• Detec4on => Mi4ga4on – Example, Self-healing PMU networks
• In-house research idea => Real system deployment – SDN-enabled IIT Microgrid
• Network layer => Applica4on layer, and Cross-layer verifica4on 13
Task 1: Self-Healing PMU Networks (Ongoing Work)
Video Demo
“Self-Healing Aeack-Resilient PMU Network for Power System Opera4on,” Submieed to IEEE Transac4on of Smart Grid, 2016 14
PMU – Phasor Measurement Unit
Task 2: Transi4on to an SDN-Enabled IIT Microgrid (Ongoing Work) • Real-4me reconfigura4on of power distribu4on assets • Real-4me islanding of cri4cal loads • Real-4me op4miza4on of power supply resources Solar PV Gas Generator Charging Sta4on Wind Turbine
Fisk Substa4on (12.47 kV)
15
ComEd ComEd
Pershing Substa4on (12.47 kV)
Communica4on Networks
Local SDN Controller 1 PMU Local SDN Controller 2 Building Control …
Control Center Grid Applica*ons Exis4ng Master Controller
SDN Master Controller
Local SDN Controller n
SDN Applica*ons
Solar PV Gas Generator Charging Sta4on Wind Turbine
Fisk Substa4on (12.47 kV)
16
ComEd ComEd
Pershing Substa4on (12.47 kV)
Task 2: Transi4on to an SDN-Enabled IIT Microgrid A Co-Simula4on Framework Legend
DSSnet
Configuration
Input or Import
TCP Socket
Processes/Elements
Named Pipe
Windows COM Port
Components
Windows
Linux
Power Coordinator ● ●
Setup Simulator Communicates Requests between Emulator and Simulator
Synchronization Events
Network Coordinator ●
zmq socket
●
Configure Network and Hosts Synchronize with Simulator
IED Configuration
Network & IED Configuration
Kernel Virtual Time System
COM Port
Power Element Configuration
Mininet
Elements Elements
CONTROLLER
Interface OpenDSS Circuit
Settings Monitors Monitors
HOSTS
SWITCHES
Controls
Figure 2: DSSnet system architecture diagram. Note that the power simulator runs on a Windows machine and the network emulator runs on a Linux machine.
“DSSnet: A Smart Grid Modeling Plauorm Combining Electrical Power Distribu4on System Simula4on and Sohware Defined Networking Emula4on,” to advance the simulation’s clock to the time stamp of the containers are running with one shared virtual clock; SimiSubmieed to ACM SIMSIG PADS, 2016 current event request and to solve the power flow at that larly, the container leverages the Linux process hierarchy to 17
time. Additionally, some elements of the power grid may be modeled in the power coordinator as a function of time, such as loads and generation. These elements are not necessarily represented in the communication network, but can still operate on DSSnet’s virtual clock.
guarantee that all the applications inside the container are using the same virtual clock. The two-layer consistency approach is well-suited to this work for pausing and resuming because: 1. All hosts should be paused or resumed when we stop
Task 3: Cross-layer Verifica4on Framework Power Control Applica4on layer
A network environment with desired proper4es (performance, security…)
Communica4on Network layer
18
Correct app behaviors
Emergency' Emergency' Occurs Detected
Condition Deteriorates
Time
Task 3: Cross-layer Verifica4on Framework Action'1
Action'2
...
Action'N
Maximum'Response'time Figure 4 Sequence of control actions by MPC
Emergency' Emergency' Occurs Detected
Emergency' Mitigated Time
Action'1 Action'2 Action'3 Action'4 (a) Desired sequence of control actions Condition' Emergency' Emergency' Deteriorates Occurs Detected
! System' Crashes Time
Action'1
Action'2
Action'3
Action'4
lost'or'delayed (b) Loss or delay of control actions Condition' Emergency' Emergency' Deteriorates Occurs Detected
! System' Crashes Time
Action'2
Action'1
Action'3
disordered (c) Disorder of control actions
Action'4
!
Model Predic4ve Control (MPC) Figure 5 Sequence of control actions Example: Incorrect Power Applica4on Control due to Network Temporal Uncertainty 19
Achievement Highlights • Journal Papers – 1 to appear (ACM TOMACS), 1 under review (IEEE Smart Grid)
• Conference Papers – 2 published, 1 under review (ACM SIMGSIM PADS, ACM SOSR)
• Awards – Best Paper Award (PADS’15) – Best Poster Award (PADS’15) – Student, Adnan Haider (co-advised with Dr. Xian-He Sun), named finalist for CRA Outstanding Undergraduate Researcher Award 20
DDDAS Workshop
in conjunc4on with the ACM SIGSIM PADS Conference • • • • •
21
When: May 16 – 17 noon, 2016 Where: Banff, Alberta, Canada Keynote speaker: Dr. Frederica Darema Co-chairs: Richard Fujimoto, Dong (Kevin) Jin Paper Submission: February 1, 2016
22
1
PI: Dong (Kevin) Jin Ph.D. Students: Christopher Hannon and Xin Liu Program Director: Dr. Frederica Darema DDDAS Program PI Mee4ng, January 2016
Industrial Control Systems (ICS) • Control many cri4cal infrastructures – e.g., weapons systems, aerospace, gas and oil distribu4on networks, wastewater treatment, transporta4on systems …
• Modern ICS increasingly adopt Internet technology to boost control efficiency, e.g., smart grid LOADS
SITES
DISTRIBUTION TRANSFORMER
DISTRIBUTION SUBSTATION TRANSMISSION
GENERATION
Next Genera4on of Power Grid 2
More Efficient or More Vulnerable? Communica4on Path Markets Retailer/ Wholesaler
WAMS
Enterprise Bus RTO SCADA
ISO/RTO Par4cipa4on
Transmission Ops
Enterprise Bus Transmission SCADA
Asset Mgmt
DMS
CIS
Demand Response
MDMS
Generators
Bulk Genera4on 3
Retail Energy Provider
CIS Billing
Home/Building Manager
Enterprise Bus
Aggregator Metering System
Distribu4on SCADA
Others
Internet / e-business Wide Area Network
Plant Control System
Third-Party Provider
Billing
Internet / e-business Market Services Interface
U5lity Provider
Distribu5on Ops
EMS
EMS
Energy Market Clearing hosue
Service Providers
Opera4ons RTO/ISO Ops
Aggregator
Network
Substa5on LANs
Data Collector
Substa4on Controller
Field Area Networks
Field Device
Electric Vehicle
Energy Services Interface
Distributed Genera4on
Premises Networks
Meter
Customer Equipment Substa4on Device
Electric Storage
Transmission
Distributed Genera4on
Appliances
Customer EMS
Distribu4on
Picture source: NIST Framework and Roadmap for Smart Grid Interoperability Standards
Electric Storage
Thermostat
Customer
Cyber Threats in Power Grids • 245 incidents, reported by ICS-CERT • 32% in energy sector • 80,000 residents in western Ukraine • 6 hours, lost power on Dec 23, 2015
Picture source: 1. Na4onal Cybersecurity and Communica4ons Integra4on Center (NCCIC). ICS-CERT Monitor Sep 2014 – Feb 2015 2. hep://dailysignal.com/2016/01/13/ukraine-goes-dark-russia-aeributed-hackers-take-down-power-grid/
4
Protec4on of Industrial Control Systems • Commercial of-the-shelf products – e.g., firewalls, an4virus sohware – fine-grained protec4on at single devices only
• How to check system-wide requirements – Security policy (e.g., access control) – Performance requirement (e.g., end-to-end delay)
• How to safely incorporate exis4ng networking technologies in control system infrastructures? – real-4me, large-scale, no interference with normal opera4ons … 5
Our Approach: DDDAS-based Real-Time System Verifica4on Policy Engine
ICS Applica5on Models System Framework
Dynamic Model Update/Selec3on
Verifica3on
Diagnosis • •
Vulnerabili*es Errors
Network Models topology
network-layer states (e.g., forwarding tables)
Dynamic Network Data (topology, forwarding tables … ) Dynamic Applica4on Data (control updates … ) User-specified Policy (security, performance …) 6
Verified System Updates
Network-Layer Verifica4on VeriFlow Operation Prior Work •
Network Controller
•
New rules
VeriFlow Generate equivalence classes
Generate forwarding graphs
• Run queries
•
FlowChecker [Al-Shaer et al.,SafeConfig2010] HeaderSpaceAnalysis [Kazemian et al.,NSDI2012] Anteater [Mai et al.,SIGCOMM2011] VeriFlow [Khurshid et al., NSDI2012]
Rules violating network invariant(s)
Good rules
Diagnosis report • Type of invariant violation • Affected set of packets 4/3/2013
Department of Computer Science, UIUC
7
11
Challenges — Timing Uncertainty Network devices are asynchronous and distributed in nature Controller'
Remove&rule&1&
Install'rule'2'
rule%1% Switch'A'
8
rule%2%
Switch'B'
Challenges — Timing Uncertainty
Controller'
Remove&rule&1& (delayed)&
Install'rule'2'
rule%1%
Packet' Switch'A'
rule%2%
Loop-freedom Viola4on 9
Switch'B'
Uncertainty-aware Modeling • Naively, represent every possible network state O(2^n) • Uncertain graph: represent all possible combina4ons
10
Update Synthesis via Verifica4on 2 1
3
4
A should reach B
Enforcing dynamic correctness with heuris4cally maximized parallelism 11
Wenxuan Zhou, Dong Jin, Jason Croh, Maehew Caesar, and P. Brighten Godfrey. “Enforcing Customizable Consistency Proper4es in Sohware-Defined Networks.” NSDI 2015.
OK, but…
Can the system “deadlock”?
• Proved classes of networks that never deadlock • Experimentally rare in prac4ce! • Last resort: heavyweight “fallback” like consistent updates [Reitblae et al, SIGCOMM 2012]
Number$of$Rules$ in$the$Network$
Is it fast? 25000$
6
12
25000
//$
//$
//$
20000 20000$ 15000 15000$ 10000 10000$
}
5000 5000$
6 6 6
8 0$0 0 8 2 8 7/22/2014$ 7/22/2014$ 8 22:00:00$ 22:00:02$
Immediate Update Immediate Update GCC ImmediateUpdates Update GCC Consistent GCC Consistent Updates End Immediate UpdateUpdates Consistent End Comple?on$ GCC CCG End Time$ Consistent Updates End 14 16 End 14 //$14 12 16 14 16 10 16 7/23/2014$ 7/23/2014$ 147/23/2014$ 16
//$
10
7/22/2014$ 23:00:00$ 4
10 //$ 6 10 7/22/2014$ 10
23:00:02$
12 12 7/23/2014$ 12
12
8
0:00:00$
Time$
0:00:02$
1:00:00$
1:00:02$
What’s next? • • • • •
Instability Loss of Load Synchroniza4on Failure Con4ngency Loss of Economics
Impact Virtualized U5lity Network 1 Frequency Control Cross-Layer Verifica5on Intrusion Detec5on
Power Control Applica5ons Demand Response
Frequency Control
State Es4ma4on
Topology Control
…
Rou4ng
…
Cyber Resources SCADA Servers
Field Devices
Communica4on Networks
Virtualized U5lity Network 2 Demand Response
Virtualized U5lity Network 3 State Es4ma4on
Control Center
Virtualized U5lity Network 4 Topology Control
Cyber AMacks Denial of Service
False Data Injec4on
Malware
Insider Aeack
…
(a) Current Power Grid: Poten4al Cyber Aeacks and Their Implica4ons
(b) Future SDN-enabled Power Grid: A Cyber-Aeack-Resilient Plauorm
• Detec4on => Mi4ga4on – Example, Self-healing PMU networks
• In-house research idea => Real system deployment – SDN-enabled IIT Microgrid
• Network layer => Applica4on layer, and Cross-layer verifica4on 13
Task 1: Self-Healing PMU Networks (Ongoing Work)
Video Demo
“Self-Healing Aeack-Resilient PMU Network for Power System Opera4on,” Submieed to IEEE Transac4on of Smart Grid, 2016 14
PMU – Phasor Measurement Unit
Task 2: Transi4on to an SDN-Enabled IIT Microgrid (Ongoing Work) • Real-4me reconfigura4on of power distribu4on assets • Real-4me islanding of cri4cal loads • Real-4me op4miza4on of power supply resources Solar PV Gas Generator Charging Sta4on Wind Turbine
Fisk Substa4on (12.47 kV)
15
ComEd ComEd
Pershing Substa4on (12.47 kV)
Communica4on Networks
Local SDN Controller 1 PMU Local SDN Controller 2 Building Control …
Control Center Grid Applica*ons Exis4ng Master Controller
SDN Master Controller
Local SDN Controller n
SDN Applica*ons
Solar PV Gas Generator Charging Sta4on Wind Turbine
Fisk Substa4on (12.47 kV)
16
ComEd ComEd
Pershing Substa4on (12.47 kV)
Task 2: Transi4on to an SDN-Enabled IIT Microgrid A Co-Simula4on Framework Legend
DSSnet
Configuration
Input or Import
TCP Socket
Processes/Elements
Named Pipe
Windows COM Port
Components
Windows
Linux
Power Coordinator ● ●
Setup Simulator Communicates Requests between Emulator and Simulator
Synchronization Events
Network Coordinator ●
zmq socket
●
Configure Network and Hosts Synchronize with Simulator
IED Configuration
Network & IED Configuration
Kernel Virtual Time System
COM Port
Power Element Configuration
Mininet
Elements Elements
CONTROLLER
Interface OpenDSS Circuit
Settings Monitors Monitors
HOSTS
SWITCHES
Controls
Figure 2: DSSnet system architecture diagram. Note that the power simulator runs on a Windows machine and the network emulator runs on a Linux machine.
“DSSnet: A Smart Grid Modeling Plauorm Combining Electrical Power Distribu4on System Simula4on and Sohware Defined Networking Emula4on,” to advance the simulation’s clock to the time stamp of the containers are running with one shared virtual clock; SimiSubmieed to ACM SIMSIG PADS, 2016 current event request and to solve the power flow at that larly, the container leverages the Linux process hierarchy to 17
time. Additionally, some elements of the power grid may be modeled in the power coordinator as a function of time, such as loads and generation. These elements are not necessarily represented in the communication network, but can still operate on DSSnet’s virtual clock.
guarantee that all the applications inside the container are using the same virtual clock. The two-layer consistency approach is well-suited to this work for pausing and resuming because: 1. All hosts should be paused or resumed when we stop
Task 3: Cross-layer Verifica4on Framework Power Control Applica4on layer
A network environment with desired proper4es (performance, security…)
Communica4on Network layer
18
Correct app behaviors
Emergency' Emergency' Occurs Detected
Condition Deteriorates
Time
Task 3: Cross-layer Verifica4on Framework Action'1
Action'2
...
Action'N
Maximum'Response'time Figure 4 Sequence of control actions by MPC
Emergency' Emergency' Occurs Detected
Emergency' Mitigated Time
Action'1 Action'2 Action'3 Action'4 (a) Desired sequence of control actions Condition' Emergency' Emergency' Deteriorates Occurs Detected
! System' Crashes Time
Action'1
Action'2
Action'3
Action'4
lost'or'delayed (b) Loss or delay of control actions Condition' Emergency' Emergency' Deteriorates Occurs Detected
! System' Crashes Time
Action'2
Action'1
Action'3
disordered (c) Disorder of control actions
Action'4
!
Model Predic4ve Control (MPC) Figure 5 Sequence of control actions Example: Incorrect Power Applica4on Control due to Network Temporal Uncertainty 19
Achievement Highlights • Journal Papers – 1 to appear (ACM TOMACS), 1 under review (IEEE Smart Grid)
• Conference Papers – 2 published, 1 under review (ACM SIMGSIM PADS, ACM SOSR)
• Awards – Best Paper Award (PADS’15) – Best Poster Award (PADS’15) – Student, Adnan Haider (co-advised with Dr. Xian-He Sun), named finalist for CRA Outstanding Undergraduate Researcher Award 20
DDDAS Workshop
in conjunc4on with the ACM SIGSIM PADS Conference • • • • •
21
When: May 16 – 17 noon, 2016 Where: Banff, Alberta, Canada Keynote speaker: Dr. Frederica Darema Co-chairs: Richard Fujimoto, Dong (Kevin) Jin Paper Submission: February 1, 2016
22
