KevinJin 2016 DDDAS PI Meeting
Dynamic Data-Driven and Real-Time Verification for Industrial Control System Security @IIT Campus Microgrid
1
PI: Dong (Kevin) Jin Ph.D. Students: Christopher Hannon and Xin Liu Program Director: Dr. Frederica Darema DDDAS Program PI Meeting, January 2016
Industrial Control Systems (ICS) • Control many critical infrastructures – e.g., weapons systems, aerospace, gas and oil distribution networks, wastewater treatment, transportation systems …
• Modern ICS increasingly adopt Internet technology to boost control efficiency, e.g., smart grid LOADS
SITES
DISTRIBUTION TRANSFORMER
DISTRIBUTION SUBSTATION TRANSMISSION
GENERATION
Next Generation of Power Grid 2
More Efficient or More Vulnerable? Communication Path Markets
Network Service Providers
Operations
Retailer/ Wholesaler
RTO/ISO Ops
Energy Market Clearing hosue
WAMS
Enterprise Bus RTO SCADA
ISO/RTO Participation
Enterprise Bus Transmission SCADA
DMS
Asset Mgmt
CIS
Demand Response
MDMS
Generators
Bulk Generation 3
Retail Energy Provider
CIS Billing
Home/Building Manager
Enterprise Bus
Aggregator Metering System
Distribution SCADA
Others
Internet / e-business Wide Area Network
Plant Control System
Third-Party Provider
Billing
Internet / e-business Market Services Interface
Utility Provider
Distribution Ops
EMS
EMS
Aggregator
Transmission Ops
Substation LANs Substation Device
Data Collector
Substation Controller
Electric Storage
Transmission
Field Area Networks Field Device
Distributed Generation
Electric Vehicle
Energy Services Interface
Distributed Generation
Premises Networks
Meter
Customer Equipment
Appliances
Customer EMS
Distribution
Picture source: NIST Framework and Roadmap for Smart Grid Interoperability Standards
Electric Storage
Thermostat
Customer
Cyber Threats in Power Grids • 245 incidents, reported by ICS-CERT • 32% in energy sector • 80,000 residents in western Ukraine • 6 hours, lost power on Dec 23, 2015
Picture source: 1. National Cybersecurity and Communications Integration Center (NCCIC). ICS-CERT Monitor Sep 2014 – Feb 2015 2. http://dailysignal.com/2016/01/13/ukraine-goes-dark-russia-attributed-hackers-take-down-power-grid/
4
Protection of Industrial Control Systems • Commercial of-the-shelf products – e.g., firewalls, antivirus software – fine-grained protection at single devices only
• How to check system-wide requirements – Security policy (e.g., access control) – Performance requirement (e.g., end-to-end delay)
• How to safely incorporate existing networking technologies in control system infrastructures? – real-time, large-scale, no interference with normal operations … 5
Our Approach: DDDAS-based Real-Time System Verification Policy Engine
ICS Application Models System Framework
Dynamic Model Update/Selection
Verification
Diagnosis • •
Vulnerabilities Errors
Network Models topology
network-layer states (e.g., forwarding tables)
Dynamic Network Data (topology, forwarding tables … ) Dynamic Application Data (control updates … ) User-specified Policy (security, performance …) 6
Verified System Updates
Network-Layer Verification VeriFlow Operation Prior Work •
Network Controller
•
New rules
VeriFlow
•
Generate equivalence classes
Generate forwarding graphs
Run queries
•
FlowChecker [Al-Shaer et al.,SafeConfig2010] HeaderSpaceAnalysis [Kazemian et al.,NSDI2012] Anteater [Mai et al.,SIGCOMM2011] VeriFlow [Khurshid et al., NSDI2012]
Rules violating network invariant(s)
Good rules
Diagnosis report • Type of invariant violation • Affected set of packets
2013
Department of Computer Science, UIUC
7
11
Challenges — Timing Uncertainty
Controller'
Remove&rule&1&
Install'rule'2'
rule%1% Switch'A'
8
rule%2%
Switch'B'
Challenges — Timing Uncertainty
Controller'
Remove&rule&1& (delayed)&
Install'rule'2'
rule%1%
Packet' Switch'A'
rule%2%
Loop-freedom Violation 9
Switch'B'
Uncertainty-aware Modeling • Naively, represent every possible network state O(2^n) • Uncertain graph: represent all possible combinations
10
Update Synthesis via Verification 2
1
3
4
A should reach B
Enforcing dynamic correctness with heuristically maximized parallelism 11
Wenxuan Zhou, Dong Jin, Jason Croft, Matthew Caesar, and P. Brighten Godfrey. “Enforcing Customizable Consistency Properties in Software-Defined Networks.” NSDI 2015.
OK, but…
Can the system “deadlock”?
• Proved classes of networks that never deadlock • Experimentally rare in practice! • Last resort: heavyweight “fallback” like consistent updates [Reitblatt et al, SIGCOMM 2012]
Number$of$Rules$ in$the$Network$
Is it fast?
6
12
25000 25000$
//$
//$
//$
20000 20000$ 15000 15000$ 10000 10000$
}
5000 5000$
6 6 6
8 0$0 0 8 2 8 7/22/2014$ 7/22/2014$ 8 22:00:00$ 22:00:02$
Immediate Update Immediate Update GCC ImmediateUpdates Update GCC Consistent GCC Consistent Updates End Immediate UpdateUpdates Consistent End Comple?on$ GCC CCG End Time$ Consistent Updates End 14 16 End 14 //$14 12 16 14 16 10 16 7/23/2014$ 7/23/2014$ 147/23/2014$ 16
//$
10
7/22/2014$ 23:00:00$ 4
10 //$ 6 10 7/22/2014$ 10
23:00:02$
12 12 7/23/2014$ 12
12
8
0:00:00$
Time$
0:00:02$
1:00:00$
1:00:02$
What’s next? • • • • •
Instability Impact Loss of Load Synchronization Failure Contingency Loss of Economics
Virtualized Utility Network 1 Frequency Control Cross-Layer Verification Intrusion Detection
Power Control Applications Demand Response
Frequency Control
State Estimation
Topology … Control
Cyber Resources SCADA Servers
Field Devices
Communication Networks
Routing
…
Virtualized Utility Network 2 Demand Response
Virtualized Utility Network 3 State Estimation
Control Center
Virtualized Utility Network 4 Topology Control
Cyber Attacks Denial of Service
False Data Injection
Malware
Insider Attack
…
(a) Current Power Grid: Potential Cyber Attacks and Their Implications
(b) Future SDN-enabled Power Grid: A Cyber-Attack-Resilient Platform
• Detection => Mitigation – Example, Self-healing PMU networks
• In-house research idea => Real system deployment – SDN-enabled IIT Microgrid
• Network layer => Application layer, and Cross-layer verification 13
Task 1: Self-Healing PMU Networks (Ongoing Work)
Video Demo “Self-Healing Attack-Resilient PMU Network for Power System Operation,” Submitted to IEEE Transaction of Smart Grid, 2016 14
PMU – Phasor Measurement Unit
Task 2: Transition to an SDN-Enabled IIT Microgrid (Ongoing Work) • Real-time reconfiguration of power distribution assets • Real-time islanding of critical loads • Real-time optimization of power supply resources Solar PV Gas Generator Charging Station Wind Turbine
Fisk Substation (12.47 kV)
15
ComEd ComEd
Pershing Substation (12.47 kV)
Communication Networks
Local SDN Controller 1 PMU Local SDN Controller 2 Building Control …
Control Center Grid Applications Existing Master Controller
SDN Master Controller
Local SDN Controller n
SDN Applications
Solar PV Gas Generator Charging Station Wind Turbine
Fisk Substation (12.47 kV)
16
ComEd ComEd
Pershing Substation (12.47 kV)
Task 2: Transition to an SDN-Enabled IIT Microgrid A Co-Simulation Framework Legend
DSSnet
Configuration
Input or Import
TCP Socket
Processes/Elements
Named Pipe
Windows COM Port
Components
Windows
Linux
Power Coordinator ● ●
Setup Simulator Communicates Requests between Emulator and Simulator
Synchronization Events
Network Coordinator ●
zmq socket
●
Configure Network and Hosts Synchronize with Simulator
IED Configuration
Network & IED Configuration
Kernel Virtual Time System
COM Port
Power Element Configuration
Mininet
Elements Elements
CONTROLLER
Interface OpenDSS Circuit
Settings Monitors Monitors
HOSTS
SWITCHES
Controls
Figure 2: DSSnet system architecture diagram. Note that the power simulator runs on a Windows machine and the network emulator runs on a Linux machine.
“DSSnet: A Smart Grid Modeling Platform Combining Electrical Power Distribution System Simulation and Software Defined Networking Emulation,” to advance the simulation’s clock to the time stamp of the containers are running with one shared virtual clock; SimiSubmitted to ACM SIMSIG PADS, 2016 current event request and to solve the power flow at that larly, the container leverages the Linux process hierarchy to 17
time. Additionally, some elements of the power grid may be modeled in the power coordinator as a function of time, such as loads and generation. These elements are not necessarily represented in the communication network, but can still operate on DSSnet’s virtual clock.
guarantee that all the applications inside the container are using the same virtual clock. The two-layer consistency approach is well-suited to this work for pausing and resuming because:
Task 3: Cross-layer Verification Framework Power Control Application layer
A network environment with desired properties (performance, security…) Communication Network layer
18
Correct app behaviors
Occurs
Detected
Deteriorates Time
Task 3: Cross-layer Verification Framework Action'1
Action'2
...
Action'N
Maximum'Response'time Figure 4 Sequence of control actions by MPC
Emergency' Emergency' Occurs Detected
Emergency' Mitigated Time
Action'1 Action'2 Action'3 Action'4 (a) Desired sequence of control actions Condition' Emergency' Emergency' Deteriorates Occurs Detected
! System' Crashes Time
Action'1
Action'2
Action'3
Action'4
lost'or'delayed (b) Loss or delay of control actions Condition' Emergency' Emergency' Deteriorates Occurs Detected
! System' Crashes Time
Action'2
Action'1
Action'3
disordered (c) Disorder of control actions
Action'4
!
Model Predictive Control (MPC) Figure 5 Sequence of control actions Example: Incorrect Power Application Control due to Network Temporal Uncertainty 19
Achievement Highlights • Journal Papers – 1 to appear (ACM TOMACS), 1 under review (IEEE Smart Grid)
• Conference Papers – 2 published, 1 under review (ACM SIMGSIM PADS, ACM SOSR)
• Awards – Best Paper Award (PADS’15) – Best Poster Award (PADS’15) – Student, Adnan Haider (co-advised with Dr. Xian-He Sun), named finalist for CRA Outstanding Undergraduate Researcher Award 20
DDDAS Workshop • • • • •
21
in conjunction with the ACM SIGSIM PADS Conference
When: May 16 – 17 noon, 2016 Where: Banff, Alberta, Canada Keynote speaker: Dr. Frederica Darema Co-chairs: Richard Fujimoto, Dong (Kevin) Jin Paper Submission: February 1, 2016
22
1
PI: Dong (Kevin) Jin Ph.D. Students: Christopher Hannon and Xin Liu Program Director: Dr. Frederica Darema DDDAS Program PI Meeting, January 2016
Industrial Control Systems (ICS) • Control many critical infrastructures – e.g., weapons systems, aerospace, gas and oil distribution networks, wastewater treatment, transportation systems …
• Modern ICS increasingly adopt Internet technology to boost control efficiency, e.g., smart grid LOADS
SITES
DISTRIBUTION TRANSFORMER
DISTRIBUTION SUBSTATION TRANSMISSION
GENERATION
Next Generation of Power Grid 2
More Efficient or More Vulnerable? Communication Path Markets
Network Service Providers
Operations
Retailer/ Wholesaler
RTO/ISO Ops
Energy Market Clearing hosue
WAMS
Enterprise Bus RTO SCADA
ISO/RTO Participation
Enterprise Bus Transmission SCADA
DMS
Asset Mgmt
CIS
Demand Response
MDMS
Generators
Bulk Generation 3
Retail Energy Provider
CIS Billing
Home/Building Manager
Enterprise Bus
Aggregator Metering System
Distribution SCADA
Others
Internet / e-business Wide Area Network
Plant Control System
Third-Party Provider
Billing
Internet / e-business Market Services Interface
Utility Provider
Distribution Ops
EMS
EMS
Aggregator
Transmission Ops
Substation LANs Substation Device
Data Collector
Substation Controller
Electric Storage
Transmission
Field Area Networks Field Device
Distributed Generation
Electric Vehicle
Energy Services Interface
Distributed Generation
Premises Networks
Meter
Customer Equipment
Appliances
Customer EMS
Distribution
Picture source: NIST Framework and Roadmap for Smart Grid Interoperability Standards
Electric Storage
Thermostat
Customer
Cyber Threats in Power Grids • 245 incidents, reported by ICS-CERT • 32% in energy sector • 80,000 residents in western Ukraine • 6 hours, lost power on Dec 23, 2015
Picture source: 1. National Cybersecurity and Communications Integration Center (NCCIC). ICS-CERT Monitor Sep 2014 – Feb 2015 2. http://dailysignal.com/2016/01/13/ukraine-goes-dark-russia-attributed-hackers-take-down-power-grid/
4
Protection of Industrial Control Systems • Commercial of-the-shelf products – e.g., firewalls, antivirus software – fine-grained protection at single devices only
• How to check system-wide requirements – Security policy (e.g., access control) – Performance requirement (e.g., end-to-end delay)
• How to safely incorporate existing networking technologies in control system infrastructures? – real-time, large-scale, no interference with normal operations … 5
Our Approach: DDDAS-based Real-Time System Verification Policy Engine
ICS Application Models System Framework
Dynamic Model Update/Selection
Verification
Diagnosis • •
Vulnerabilities Errors
Network Models topology
network-layer states (e.g., forwarding tables)
Dynamic Network Data (topology, forwarding tables … ) Dynamic Application Data (control updates … ) User-specified Policy (security, performance …) 6
Verified System Updates
Network-Layer Verification VeriFlow Operation Prior Work •
Network Controller
•
New rules
VeriFlow
•
Generate equivalence classes
Generate forwarding graphs
Run queries
•
FlowChecker [Al-Shaer et al.,SafeConfig2010] HeaderSpaceAnalysis [Kazemian et al.,NSDI2012] Anteater [Mai et al.,SIGCOMM2011] VeriFlow [Khurshid et al., NSDI2012]
Rules violating network invariant(s)
Good rules
Diagnosis report • Type of invariant violation • Affected set of packets
2013
Department of Computer Science, UIUC
7
11
Challenges — Timing Uncertainty
Controller'
Remove&rule&1&
Install'rule'2'
rule%1% Switch'A'
8
rule%2%
Switch'B'
Challenges — Timing Uncertainty
Controller'
Remove&rule&1& (delayed)&
Install'rule'2'
rule%1%
Packet' Switch'A'
rule%2%
Loop-freedom Violation 9
Switch'B'
Uncertainty-aware Modeling • Naively, represent every possible network state O(2^n) • Uncertain graph: represent all possible combinations
10
Update Synthesis via Verification 2
1
3
4
A should reach B
Enforcing dynamic correctness with heuristically maximized parallelism 11
Wenxuan Zhou, Dong Jin, Jason Croft, Matthew Caesar, and P. Brighten Godfrey. “Enforcing Customizable Consistency Properties in Software-Defined Networks.” NSDI 2015.
OK, but…
Can the system “deadlock”?
• Proved classes of networks that never deadlock • Experimentally rare in practice! • Last resort: heavyweight “fallback” like consistent updates [Reitblatt et al, SIGCOMM 2012]
Number$of$Rules$ in$the$Network$
Is it fast?
6
12
25000 25000$
//$
//$
//$
20000 20000$ 15000 15000$ 10000 10000$
}
5000 5000$
6 6 6
8 0$0 0 8 2 8 7/22/2014$ 7/22/2014$ 8 22:00:00$ 22:00:02$
Immediate Update Immediate Update GCC ImmediateUpdates Update GCC Consistent GCC Consistent Updates End Immediate UpdateUpdates Consistent End Comple?on$ GCC CCG End Time$ Consistent Updates End 14 16 End 14 //$14 12 16 14 16 10 16 7/23/2014$ 7/23/2014$ 147/23/2014$ 16
//$
10
7/22/2014$ 23:00:00$ 4
10 //$ 6 10 7/22/2014$ 10
23:00:02$
12 12 7/23/2014$ 12
12
8
0:00:00$
Time$
0:00:02$
1:00:00$
1:00:02$
What’s next? • • • • •
Instability Impact Loss of Load Synchronization Failure Contingency Loss of Economics
Virtualized Utility Network 1 Frequency Control Cross-Layer Verification Intrusion Detection
Power Control Applications Demand Response
Frequency Control
State Estimation
Topology … Control
Cyber Resources SCADA Servers
Field Devices
Communication Networks
Routing
…
Virtualized Utility Network 2 Demand Response
Virtualized Utility Network 3 State Estimation
Control Center
Virtualized Utility Network 4 Topology Control
Cyber Attacks Denial of Service
False Data Injection
Malware
Insider Attack
…
(a) Current Power Grid: Potential Cyber Attacks and Their Implications
(b) Future SDN-enabled Power Grid: A Cyber-Attack-Resilient Platform
• Detection => Mitigation – Example, Self-healing PMU networks
• In-house research idea => Real system deployment – SDN-enabled IIT Microgrid
• Network layer => Application layer, and Cross-layer verification 13
Task 1: Self-Healing PMU Networks (Ongoing Work)
Video Demo “Self-Healing Attack-Resilient PMU Network for Power System Operation,” Submitted to IEEE Transaction of Smart Grid, 2016 14
PMU – Phasor Measurement Unit
Task 2: Transition to an SDN-Enabled IIT Microgrid (Ongoing Work) • Real-time reconfiguration of power distribution assets • Real-time islanding of critical loads • Real-time optimization of power supply resources Solar PV Gas Generator Charging Station Wind Turbine
Fisk Substation (12.47 kV)
15
ComEd ComEd
Pershing Substation (12.47 kV)
Communication Networks
Local SDN Controller 1 PMU Local SDN Controller 2 Building Control …
Control Center Grid Applications Existing Master Controller
SDN Master Controller
Local SDN Controller n
SDN Applications
Solar PV Gas Generator Charging Station Wind Turbine
Fisk Substation (12.47 kV)
16
ComEd ComEd
Pershing Substation (12.47 kV)
Task 2: Transition to an SDN-Enabled IIT Microgrid A Co-Simulation Framework Legend
DSSnet
Configuration
Input or Import
TCP Socket
Processes/Elements
Named Pipe
Windows COM Port
Components
Windows
Linux
Power Coordinator ● ●
Setup Simulator Communicates Requests between Emulator and Simulator
Synchronization Events
Network Coordinator ●
zmq socket
●
Configure Network and Hosts Synchronize with Simulator
IED Configuration
Network & IED Configuration
Kernel Virtual Time System
COM Port
Power Element Configuration
Mininet
Elements Elements
CONTROLLER
Interface OpenDSS Circuit
Settings Monitors Monitors
HOSTS
SWITCHES
Controls
Figure 2: DSSnet system architecture diagram. Note that the power simulator runs on a Windows machine and the network emulator runs on a Linux machine.
“DSSnet: A Smart Grid Modeling Platform Combining Electrical Power Distribution System Simulation and Software Defined Networking Emulation,” to advance the simulation’s clock to the time stamp of the containers are running with one shared virtual clock; SimiSubmitted to ACM SIMSIG PADS, 2016 current event request and to solve the power flow at that larly, the container leverages the Linux process hierarchy to 17
time. Additionally, some elements of the power grid may be modeled in the power coordinator as a function of time, such as loads and generation. These elements are not necessarily represented in the communication network, but can still operate on DSSnet’s virtual clock.
guarantee that all the applications inside the container are using the same virtual clock. The two-layer consistency approach is well-suited to this work for pausing and resuming because:
Task 3: Cross-layer Verification Framework Power Control Application layer
A network environment with desired properties (performance, security…) Communication Network layer
18
Correct app behaviors
Occurs
Detected
Deteriorates Time
Task 3: Cross-layer Verification Framework Action'1
Action'2
...
Action'N
Maximum'Response'time Figure 4 Sequence of control actions by MPC
Emergency' Emergency' Occurs Detected
Emergency' Mitigated Time
Action'1 Action'2 Action'3 Action'4 (a) Desired sequence of control actions Condition' Emergency' Emergency' Deteriorates Occurs Detected
! System' Crashes Time
Action'1
Action'2
Action'3
Action'4
lost'or'delayed (b) Loss or delay of control actions Condition' Emergency' Emergency' Deteriorates Occurs Detected
! System' Crashes Time
Action'2
Action'1
Action'3
disordered (c) Disorder of control actions
Action'4
!
Model Predictive Control (MPC) Figure 5 Sequence of control actions Example: Incorrect Power Application Control due to Network Temporal Uncertainty 19
Achievement Highlights • Journal Papers – 1 to appear (ACM TOMACS), 1 under review (IEEE Smart Grid)
• Conference Papers – 2 published, 1 under review (ACM SIMGSIM PADS, ACM SOSR)
• Awards – Best Paper Award (PADS’15) – Best Poster Award (PADS’15) – Student, Adnan Haider (co-advised with Dr. Xian-He Sun), named finalist for CRA Outstanding Undergraduate Researcher Award 20
DDDAS Workshop • • • • •
21
in conjunction with the ACM SIGSIM PADS Conference
When: May 16 – 17 noon, 2016 Where: Banff, Alberta, Canada Keynote speaker: Dr. Frederica Darema Co-chairs: Richard Fujimoto, Dong (Kevin) Jin Paper Submission: February 1, 2016
22
