Maaz Mohiuddin

Real-time state estimation of the EPFL-campus medium-voltage grid by using PMUs

We describe the real-time monitoring infrastructure of the smart-grid pilot on the EPFL campus. We experimentally validate the concept of a real-time state-estimation for a 20 kV active distribution network. We designed and put into operation the whole infrastructure composed by the following main elements: (1) dedicated PMUs connected on the medium-voltage side of the network secondary substations by means of specific current/voltage transducers; (2) a dedicated communication network engineered to support stringent time limits and (3) an innovative state estimation process for real-time monitoring that incorporates phasor-data concentration and state estimation processes. Special care was taken to make the whole chain resilient to cyber-attacks, equipment failures and power outages. The achieved latency is within 65ms. The refresh rate of the estimated state is 20ms. The real-time visualization of the state estimator output is made publicly available, as well as the historical data (PMU measurements and estimated states). To the best of our knowledge, the work presented here is the first operational system that provides low-latency real-time state-estimation by using PMU measurements of a real active distribution network.

iPRP: Parallel redundancy protocol for IP networks

Reliable packet delivery within stringent delay constraints is of primal importance to industrial processes with hard real-time constraints, such as electrical grid monitoring. Because retransmission and coding techniques counteract the delay requirements, reliability is achieved through replication over multiple fail-independent paths. Existing solutions such as parallel redundancy protocol (PRP) replicate all packets at the MAC layer over parallel paths. PRP works best in local area networks, e.g., sub-station networks. However, it is not viable for IP layer wide area networks which are a part of emerging smart grids. Such a limitation on scalability, coupled with lack of security, and diagnostic inability, renders it unsuitable for reliable data delivery in smart grids. To address this issue, we present a transport-layer design: IP parallel redundancy protocol (iPRP). Designing iPRP poses non-trivial challenges in the form of selective packet replication, soft-state and multicast support. Besides unicast, iPRP supports multicast, which is widely using in smart grid networks. It duplicates only time-critical UDP traffic. iPRP only requires a simple software installation on the end-devices. No other modification to the existing monitoring application, end-device operating system or intermediate network devices is needed. iPRP has a set of diagnostic tools for network debugging. With our implementation of iPRP in Linux, we show that iPRP supports multiple flows with minimal processing and delay overhead. It is being installed in our campus smart grid network and is publicly available.

iPRP—The Parallel Redundancy Protocol for IP Networks: Protocol Design and Operation

Reliable packet delivery within stringent delay constraints is of paramount importance to mission-critical computer applications with hard real-time constraints. Because retransmission and coding techniques counteract the delay requirements, reliability may be achieved through replication over multiple fail-independent paths. The existing solutions, such as the parallel redundancy protocol (PRP), replicate all packets at the media access control layer over parallel paths. PRP works best in local area networks; however, it is not viable for IP networks that are a key element of emerging mission-critical systems. This limitation, coupled with diagnostic inability and lack of security, renders PRP unsuitable for reliable data delivery in these IP networks. To address this issue, we present a transport-layer solution: the IP parallel redundancy protocol (iPRP). Designing iPRP poses nontrivial challenges in the form of selective packet-replication, and soft-state and multicast support. iPRP replicates only time-critical unicast or multicast user datagram protocol traffic. iPRP requires no modifications to the existing monitoring application, end-device operating system, or to the intermediate network devices. It only requires a simple software installation on the end devices. iPRP has a set of diagnostic tools for network debugging. With our implementation of iPRP in Linux, we show that iPRP supports multiple flows with minimal processing-and-delay overhead. It is being installed in our campus smart-grid network and is publicly available.

Axo: Detection and Recovery for Delay and Crash Faults in Real-Time Control Systems

Real-time control systems use controllers that compute and issue setpoints within stringent delay constraints. Failure to do so, due to a crash or delay as a result of software and/or hardware faults, can cause failure of the controlled resources. Recently, Axo, a protocol for masking crash and delay faults by replicating the controller, was proposed. Axo provides safety by discarding delayed setpoints, and it relies on the presence of valid setpoints for providing availability. To ensure that enough valid setpoints are issued, faulty controller replicas need to be detected and recovered. We present a mechanism for detection and recovery of delay- and crash-faulty replicas under the Axo framework. These mechanisms were designed to be soft state (i.e., their state can be reconstructed from received messages) to enable seamless additions of new replicas. Besides presenting the design, we analytically characterize the time to detect and recover a faulty replica, and we validate them experimentally. We demonstrate the performance of Axo by using two case studies: the first provides a stability analysis of an inverted pendulum system with Axo, and the second shows the fault-tolerance performance of Axo through a deployment on a real-time control system that controls a CIGRÉ low-voltage benchmark microgrid.

T-RECS: A software testbed for multi-agent real-time control of electric grids

Multiple software agents can be used to perform the real-time control of electrical grids. The control performance of such solutions is influenced by software non-idealities such as crashes and delays of the software agents, and message losses and delays due to the underlying communication network. To study the effect of these non-idealities on control systems, we present an open-source software testbed, named T-RECS. It uses software containers to test existing software without modification. The communication network among the software containers is emulated using Mininet framework, which allows for real packets being exchanged. The electric resources in the grid are simulated using state-of-the-art models, whereas the grid itself is modeled in the phasor domain. As control agents are run as is and message exchanges are emulated, T-RECS accurately captures the real-world properties of the control framework. We demonstrate the working of T-RECS with the Commelec control framework and show the effect of network non-idealities on the control performance. We make a beta version available.

Axo: Masking delay faults in real-time control systems

We consider real-time control systems that consist of a controller that computes and sends setpoints to be implemented in physical processes through process agents. We focus on systems that use commercial off-the-shelf hardware and software components. Setpoints of these systems have strict real-time constraints: Implementing a setpoint after its deadline, or not receiving setpoints within a deadline, can cause failure. In this paper, we address delay faults: faults that cause setpoints to violate their real-time constraints. We present Axo, a fault-tolerance protocol that guarantees safety and improves availability for a class of such systems that exhibit two main properties: the setpoints must have a known validity horizon, and process agents must be capable of handling duplicate setpoints. To reason about delay faults, and consequently design Axo, we present an abstraction of a controller; the abstraction applies to a wide range of real-time control systems. We prove guarantees of safety and availability. Finally, we present an implementation of Axo and the results of the tests performed with Commelec, a real-time control system for electric grids.

T-RECS: A Virtual Commissioning Tool for Software-Based Control of Electric Grids: Design, Validation, and Operation

In real-time control of electric grids using multiple software agents, the control performance depends on (1) the proper functioning of the software agents, i.e., absence of software faults, and (2) the behavior of software agents in the presence of non-ideal communication networks such as message losses and delays. To evaluate the control performance of such systems, we propose T-RECS, a virtual commissioning tool. T-RECS enables testing the performance of software-based control in-silico (before the actual deployment of software agents in the grid), saving both time and money. Developers can run the binaries of their software agents in T-RECS where these binaries exchange real messages by using an emulated network and simulated models of the electric grid and resources. Consequently, the control of an entire microgrid can be tested on a standard computer. In this paper, we first describe the design and the open-source implementation of T-RECS. Second, we measure its CPU and memory usage and show that our implementation can accommodate eight software agents on a standard laptop computer. Third, we validate the simulated grid used in T-RECS by replaying data collected from experiments performed in a real low-voltage microgrid. We find that the average error is 0.037% and the 99th percentile of the error is less than 0.1%. Finally, we present some typical use-cases of T-RECS such as performance evaluation (1) under extreme grid conditions and (2) with non-ideal communication networks. The former, i.e., performance evaluation under extreme grid conditions, is difficult to test in the field due to safety concerns.

Ordering events based on intentionality in cyber-physical systems

We consider cyber-physical systems (CPSs) comprising a central controller that might be replicated for high-reliability, and one or more process agents. The controller receives measurements from process agents, causing it to compute and issue setpoints that are sent back to process agents. The implementation of these setpoints causes a change in the state of the controlled physical process, and the new state is communicated to the controllers through resulting measurements. To ensure correct operation, the process agents must implement only those setpoints that were caused by their most recent measurements. However, in the presence of replication of the controller, network or computation delays, setpoints and measurements do not necessarily succeed in causing the intended behavior. To capture the dependencies among events associated with measurements and setpoints, we introduce the intentionality relation among such events in a CPS and illustrate its differences with respect to the happened-before relation. We propose a mechanism, intentionality clocks, and the design of controllers and process agents that can be used to guarantee the strong clock-consistency condition under the intentionality relation. Moreover, we prove that our design ensures correct operation despite crash, delay, and network faults. We also demonstrate the practical application of our abstraction through an illustration with a real-world CPS for electrical vehicles.

Experimental validation of the suitability of virtualization-based replication for fault tolerance in real-time control of electric grids

Real-time control systems (RTCSs) perform complex control and require low response times. They typically use third-party software libraries and are deployed on generic hardware, which suffer from delay faults that can cause serious damage. To improve availability and latency, the controllers in RTCSs are replicated on physical nodes. As physical replication is expensive, we study the alternative of exploiting virtualization technology to run multiple virtual replicas on the same physical node. As virtual replicas share the same resources, the delay faults they experience might be correlated, which would make such a replication method unsuitable. We conduct several experiments with an RTCS for electric grids, with multiple virtual replicas of its controller. We find that although the delay of a virtual machine is higher than of a physical machine, the correlation between high delays among the virtual replicas is insignificant, causing an overall improved availability. We conclude that virtual replication is indeed applicable to certain RTCSs, as it can improve reliability without added cost.

Quarts: Quick agreement for real-time control systems

Real-time control systems (RTCSs) tolerate delay and crash faults by replicating the controller. Each replica computes and issues setpoints to actuators over a network that might drop or delay messages. Hence, the actuators might receive an inconsistent set of setpoints. Such inconsistency is avoided either by having a single primary replica compute and issue setpoints (in passive replication) or a consensus algorithm select one sending-replica (in active replication). However, due to the impossibility of a perfect failure-detector, passive-replication schemes can have multiple primaries, causing inconsistency, especially in the presence of intermittent delay faults. Furthermore, the impossibility of bounded-latency consensus causes both schemes to have poor real-time performance. We identified three properties of RTCSs that enable active-replication schemes to agree on the measurements before computing, instead of using traditional consensus. As all computing replicas compute with the same state, the resulting setpoints are guaranteed to be consistent. We present the design of Quarts, an agreement solution for active replication that guarantees consistency and bounded latency-overhead. We prove the guarantees and compare the performance of Quarts with existing solutions through simulation. We show that Quarts provides an availability higher than existing solutions, and that the availability improvement is up to 10x with two replicas.