Case Study: Implementing Redundant

Logic Controllers in Substations

Sagar Dayabhai
CONCO Energy Solutions

Brian Waldron and Marcel van Rensburg

Schweitzer Engineering Laboratories, Inc.

Presented at the
Power and Energy Automation Conference
Seattle, Washington
March 3–4, 2020
 1

Case Study: Implementing Redundant

Logic Controllers in Substations
Sagar Dayabhai, CONCO Energy Solutions
Brian Waldron and Marcel van Rensburg, Schweitzer Engineering Laboratories, Inc.

Abstract—The introduction of next generation smart grid assets in the same way, information availability and control
technologies and intelligent electronic devices (IEDs) has decisions have also become important in critical systems that
increased the level of integration and information that exists in a require redundancy. This has driven the need for replacing the
digital substation. This advent has propelled the use of central
logic controllers to allow system operators to safely monitor and traditional single data concentrator with redundant substation
operate the power system. This information is intelligently controllers. There are many technologies for communication
processed and analyzed by these controllers to improve power redundancy that keep information available to controllers so
system reliability and availability and facilitate operations and they can make communications-assisted decisions. While
maintenance. redundant communication methods are important, they provide
In addition to performing core supervisory control and data only a portion of the system that provides redundancy in control
acquisition (SCADA) functions and processing, modern central
algorithms. This paper examines the methods and techniques
logic controllers are equipped with advanced automation features
designed to perform logic, arithmetic, and complex algorithms. used to deliver information to substation logic controllers. It
These functions are employed to facilitate station-wide then describes how substation controllers manage duplicate
interlocking, measurement comparisons, power flow summations information sources, selects a primary controller to make
and load flow, power plant control functions, live busbar transfer control decisions, and explains how to send information back
routines, distribution automation sequences, and more. The into the system without duplicate information from the
criticality and dependency of the central logic controller in a
redundant controllers.
substation automation system often necessitate the
implementation of redundant controllers to enable a fault-tolerant Creating a robust redundant controller solution is not a one-
design. There has been significant conversation in the power size-fits-all approach. The redundancy solution is often heavily
system industry about communication redundancy protocols, dependent on the communications infrastructure available in
when redundant IEDs are needed, and cost effectiveness. There the system. The solution can be divided into several categories,
has, however, been comparatively little conversation about how but this paper focuses primarily on three logical communication
redundant controllers are implemented and how they coordinate
sections:
information collection and synchronize control logic between the
redundant controllers. This paper covers the efficient and reliable • Controller-to-IEDs
methods of collecting information for logic controllers, • Controller-to-controller
coordinating information and the decision-making processes • Controller-to-control center/supervisory control and
between controllers, and how that information is sent to the data acquisition (SCADA)
SCADA master station. The advantages and disadvantages of this
implementation experience are documented through a case study
The communication mediums and technologies used in these
that covers what worked well and the challenges encountered sections affect how the controllers can implement redundancy.
during the design, commissioning, and maintenance phase of the The technology selected for each of these communication paths
substation. does not necessarily need to be the same to create a reliable
solution. In practice, there are often significant differences
I. INTRODUCTION between these communication paths. Each path impacts how
The role of the data concentrator or remote terminal unit the redundant controllers manage information and their control
(RTU) in the substation has evolved substantially over the last decisions. This paper covers technologies and methods to create
three decades. What once simply collected field I/O and a robust, reliable redundant controller solution with lessons
transmitted the status and analog values back to the control learned from recent installations of redundant controllers.
center now manages data collection from a wide variety of
sources, alarm annunciation, engineering access, data II. IMPORTANCE OF REDUNDANCY
visualization, and other additional functions. Because these The systems used in mission-critical applications such as
devices have access to so much information from the system renewable energy facilities and transmission substations
around them, they become a natural location for making usually require more than 99.9 percent availability. In
decisions that require input from multiple sources. These transmission networks, it is unacceptable to have large network
devices have extensive logic written to control various power disturbances that can threaten the stability of the power system.
system assets and change operating modes of individual These networks operate several mission-critical applications
intelligent electronic devices (IEDs). Just as redundant including station-wide interlocking, use of IEC 61850 GOOSE
protection IEDs are important to protect valuable power system for protection and control, plant control, condition-based
 2

monitoring, substation automation, load-shedding schemes, etc. to communicate to a substation IED LAN with mission-critical
Furthermore, the information that these SCADA systems protocols such as IEC 61850 GOOSE. These applications
exchange with control centers through substation gateways and typically have the logic controller configured to control
logic controllers is considered mission-critical; it provides processes such as load shedding, remedial action schemes, etc.
awareness and visibility to system operators in real time. This makes it an essential component of the solution. The logic
In the case of a renewable energy facility, the highly variable controller can also be responsible for SCADA communications
nature of such resources as wind and solar power make it in such an application that requires it to be connected to a
essential for such facilities to operate on a maximum separate WAN.
availability factor of the plant. Unnecessary or preventable
B. Software-Defined Networking (SDN)
plant outages affect not only the ability of the facility to provide
power at its maximum capacity, but also directly impact the SDN was first used in the information technology (IT)
revenue-generating capacity of the plant. industry for applications such as data center networks and
Because of the critical nature of these facilities, it is common software-defined wide-area networking. However, the SDN
for substation designs to incorporate a high-availability architecture and its many features (e.g., network traffic
architecture. This includes redundancy in IEDs, logic programmability and network statistics) can provide innovative
controllers and substation gateways, SCADA infrastructure, solutions to networking challenges in other industries.
networking equipment, cabling, etc. In the context of logic Operational technology (OT) SDN does not change the basic
controllers, this redundancy is achieved by considering both architecture of SDN. It is rather a method of applying SDN to
hardware and software within a logic controller. solve unique challenges in automation networks composed of
To address hardware failures, it is common to introduce a devices like switches and programmable logic controllers [2].
second redundant logic controller to improve the reliability and SDN provides a solution for detecting network failures and
availability of a system. This redundancy accommodates measuring network path latency, which is essential for
failures in power supplies, communication ports, hard-disk monitoring mission-critical communications such as
drives, and various embedded ancillary components that may IEC 61850-9-2 solutions. In these solutions, it is critical for the
compromise system functionality in the event of a failure. In IEDs to publish and subscribe to the data streams with as short
the case of software, redundancy in a logic controller is a delay as possible to meet performance and high signaling
achieved by considering several aspects including the requirements. SDN provides a fully programmable solution for
following: using one network infrastructure to provide multiple paths for
• Synchronization of databases information exchange. Dedicated communication paths can be
designed and monitored based on latency, which ensures the
• Telecommunications paths
most efficient use of physical network infrastructure. It is also
• Redundancy management between the logic
a natively better cybersecure solution with a deny-by-default
controllers
design because the SDN requires configuration before it
• SCADA communication to end devices
operates [2].
The SDN is managed and configured on the Ethernet switch,
III. REDUNDANCY COMMUNICATION PATHS AND METHODS
so logic controllers can use SDN solutions without requiring
The primary purpose of this paper is not to discuss any special protocol support. This allows the logic controllers
communication redundancy, but delivering data to controllers to achieve redundancy without having to dedicate a specific
is an important part of the system. This section briefly covers Ethernet port for a single function. However, if the hardware
several commonly used technologies that offer redundancy in supports the usage of a second Ethernet port, this increases the
delivering data to controllers. reliability in the case of a port or cable failure.
A. Parallel Redundancy Protocol (IEC 62439-3 PRP) C. Rapid Spanning Tree Protocol (RSTP)
PRP is a protocol used to achieve communication path RSTP, defined in the IEEE 802.1D-2004 standard, is a
redundancy on a substation Ethernet local-area network (LAN) protocol used by Ethernet networking devices to detect, isolate,
for mission-critical communications. PRP solutions duplicate a and restore network paths between RSTP-supported devices
message on two independent networks on separate Ethernet without human intervention. RSTP requires the engineer to
ports on the sending device. The receiving end, that must be a design Ethernet networks with RSTP devices arranged in
PRP-compliant entity, accepts the message arriving first and multiple paths for information to be sent between IEDs [3].
discards the other [1]. The risk with using PRP is that individual Typically, IEDs do not partake in RSTP network topologies,
PRP paths are not monitored by the sending and receiving but they rather connect to RSTP-based Ethernet network
devices, and only after both paths have failed does the user switches, with the IEDs as edge devices.
become aware of a network failure. PRP is mostly applied on Considering applications where logic controllers are
substation LANs, but not as popular on wide-area networks connected to networks using RSTP, the least costly path time
(WANs) because of the duplicate network device requirements, for information exchange should be measured. This must be
which is costly. used for protocol-based time-out settings and must also be
Logic controllers support PRP in order to accommodate taken into consideration for critical solutions, such as load-
fault-tolerant independent communications networks to be able shedding schemes and interlocking, to ensure that coordination
 3

is maintained. The challenge with RSTP is that the engineer communication interface between the two controllers so that the
should design redundant paths with placement and connections primary controller passes the information to the backup
of supported Ethernet devices to reliably determine which path controller. This means that the backup controller may have
the Ethernet switches will use for each network failure location. reduced downtime when a switch between controllers occurs,
Failover time in RSTP networks is in the range of tens to because the backup controller already has the last current state
hundreds of milliseconds [4]. of all the information. While this alternative helps mitigate
some of the issues with the single communication interface, it
IV. CONTROLLER-TO-IEDS adds complexity to the redundancy configuration because the
When there are redundant controllers in the system there is controllers must identify which set of information should be
a desire to have both controllers run the exact same used in the transfer of data between the controllers. Depending
configuration and have seamless transfer between the two on the application and system requirements, this trade off may
controllers. However, this is not always possible depending on not be worth the benefits of mitigating the disadvantages of
how information is available to the two controllers. IEDs with single-path communication interfaces.
information to be collected, may only have a single serial port
available for controller communications, other devices such as V. CONTROLLER-TO-CONTROLLER
a human-machine interface (HMI) or other data collection A. Determining Which Logic Controller Should Be Active
services may also restrict the number of communication The two key aspects in a redundancy architecture is
sessions available to the redundant controllers. This section management of redundant data/events and state
discusses methods and previously implemented solutions for synchronization of the logic controller to ensure that
managing and collecting information from IEDs. information is reliably transported to and from the logic
A. Controllers Collecting Data Simultaneously controller without degrading the performance of the system
If system IEDs allow both controllers to collect information when a single failure occurs with any of the controllers. This
from the IEDs at the same time, redundancy configuration is setup is analogous to the setup of a cluster configuration
often simplified. No additional logic is necessary in the typically used between redundant firewalls.
controller to determine when information collection should be State synchronization information refers to internal system
active. This also means each controller has the latest data used in the redundancy scheme to allow the logic
information and latency in data collection and decision making controllers to synchronize the database and make the
should be minimal for a transfer of control between the appropriate decisions when managing the active, backup, and
controllers. maintenance states.
The redundant logic controllers must share a communication
B. One Controller Collects Data at a Time channel between the two controllers to keep track of the
Additional logic is necessary when only a single controller availability of each controller and determine which controller
can collect information from the source IEDs, but this typically should be active and issuing control commands. The
adds no significant complication to the logic of the system. The communication medium, protocol, and speed do not have
controllers must know when the system should collect significant impact when examining the logic of this decision-
information and turn off collections services. However, the making process. Each of these items has an impact on the speed
redundant controllers often already have logic that determines of failover and some affect the reliability of the communication
which controller is primary or active. This indication can be channel. They do not, however, affect how logic decisions are
used to tell the controller when information collection should made to determine which controller is active, so this section
occur. This often just results in a few additional lines of code to does not discuss protocol and connection method. In the initial
turn protocol collection on and off. One advantage of collecting examination of the logic, it appears that determination of which
information from only one controller at a time is that it controller should be active in either processing SCADA or
simplifies the connection to the SCADA master station and control algorithms is straightforward. Configure heartbeat logic
eliminates the need for event management. Only one controller between the two controllers by using either a Boolean indicator
has events to report to the SCADA master station at a time, and that changes on some interval or a counter value that increments
no logic or controller feature is necessary to manage events on that interval. When the backup/inactive controller no longer
previously sent. While this is an advantage, it may not be worth sees the heartbeat from the primary controller, it is time for the
the several disadvantages of this approach. Because there is backup controller to become the primary controller. However,
only one communication path to the IED, any interruption of the following factors can complicate this decision.
that communication path causes both controllers to be unable to
B. Lost Communications or Logic Controller Without Power
collect that information. The time to switch between controllers
increases because each protocol communication interface must A difficult aspect of communications between two
initialize and start collecting data. This interrupts data to controllers is how one controller loses detection of
SCADA and any control algorithms that are using those data. communications, while the other controller is still active and
An alternative to minimize the impact of a single processing data. Perhaps a detrimental event such as a hardware
communication interface to the IEDs is to support an additional problem or loss of power occurred to the other controller
 4

preventing it from performing its intended function. While the Topology 2, that utilizes both Ethernet and serial
issue may look a little different in each communication medium communications, is an attractive choice to synchronize data
(hardwire contact, serial, and Ethernet), it poses the same between logic controllers. It allows for one communication
fundamental challenge: Did something happen to the medium type to fail and still maintain communications. The
communication medium, or is the controller no longer sending serial communication path can have some limitations on the
the heartbeat? There are generally two approaches to amount of data or which applications can utilize that
determining the answer. First, consider implementing multiple communication channel, but if something specifically affects
communication channels between the two controllers. Fig. 1 the Ethernet communications channel it is less likely to affect
shows several different communication options. the serial communications and allow synchronization between
• Topology 1 uses two independent serial links to the two controllers to continue. Topologies 1 and 4 also allow
reliably transport state synchronization data and for one communication method to fail and still maintain
redundant information and events. Typically, both synchronization between controllers.
state synchronization information and redundant data Another advantage of using communication channels
are transmitted over both links for added reliability. between controllers separate from the interface that
• Topology 2 uses the Ethernet-based communications communicates with the SCADA or WAN interface is that it
network and a single serial communications link to allows the logic controllers to detect if the WAN interface or
transport both state synchronization information and SCADA communications are lost. This allows the controllers
redundant data. to switch the active state when both units are processing logic
• Topology 3 uses the Ethernet-based communications and communications between the logic controllers are good.
network to transport both state synchronization This increases the reliability for remote communication status
information and redundant data. This topology uses and controls.
two physical Ethernet ports to achieve separate The second approach involves a third device in a separate
communication paths and is reliant on the operability location that communicates with the two redundant logic
of the communications network. controllers. The entire purpose of this third device is to monitor
• Topology 4 uses two separate Ethernet-based the availability of the other two devices. If one device is
communications networks to transport both state physically powered down it provides an additional piece of
synchronization information and redundant data. This information to the other logic controller that lets it know that
topology uses two physically separated Ethernet ports both the logic controller and the third device do not see activity
and networking devices to achieve separate from the powered-down logic controller. This allows the
communication paths. backup device to make the appropriate decision. If
communications between the two logic controllers is lost, then
communication with the third logic controller informs the other
two logic controllers how to behave. While this method still
uses communications, it alleviates a single point of failure
between the two logic controllers.
In the unlikely case that either of the two approaches
discussed previously completely fail, it results in the possible
scenario where both logic controllers are powered on, and the
synchronization logic makes both controllers active at the same
time. This creates potential conflicts in the system depending
on other communications the controllers have to IEDs and
SCADA in the system. There are three possible approaches;
allow both controllers to be active, turn both controllers off, or
program only one controller to be active if communications are
lost.
1) Both Controllers Are Active
In this scenario the default synchronization logic wants to
turn both controllers on. It is unlikely that most system
operators prefer this behavior. Having both controllers active
can send conflicting controls to IEDs, send multiple data
responses to SCADA, or send conflicting data to other IEDs if
both units are not processing the same data.
2) Turn Both Controllers Off
Because it is difficult for the controller to detect if a loss of
communications is the result of a hardware failure or only a
Fig. 1. Communication Topologies Between Controllers
communication failure turning off, both controllers during a
 5

loss of communications defeats the purpose of redundant determined the other controller is unresponsive becomes active.
controllers in the case of hardware failure of a single controller. It is best to create a voting scheme to decide which controller
In this case, no application control logic would be sent to IEDs should be active if synchronization between controllers is lost.
or no communication to SCADA. A simple voting scheme to determine which controller should
be active is to determine which controller has been active for a
3) Default to a Single Controller
longer period of time. To do this, record how long each
Select a single controller to be active if communications
controller has been active and exchange that information
between devices is lost. In this mode, the system behavior is
between the two controllers. It is important that the amount of
desirable if both controllers are operating as normal, but lose all
active time is precise, perhaps to the millisecond if the
communication between controllers. Depending on system
controller is capable of this. This is important in a scenario
conditions, control decisions to IEDs are maintained, as well as
where power to both controllers is lost and then restored
SCADA communications. But, if a hardware failure occurs, it
simultaneously. Depending on the configuration and the
is unpredictable whether the controller selected to become
behavior of the controller, both controllers can begin processing
active fails. Either no controller is active, or a non-default
redundancy logic within the same second. It is unlikely that
controller has hardware failure by chance and the default
both units would start the redundancy logic within the exact
controller continues processing as normal.
same millisecond. This higher level of time accuracy prevents
In terms of total risk, choosing to default to a single
a tie in this voting scheme. This becomes a simple calculation
controller provides the most constant desirable behavior when
to determine which controller should be active and covers many
communications are lost. When hardware failure occurs, it may
scenarios that result in a complete loss of communications or a
provide the most desirable behavior depending on which
loss of synchronization between the controllers.
controller fails. In the option of both controllers becoming
active, it always provides the most desirable behavior in the E. Maintenance Mode
event of a controller hardware failure. However, it potentially Another desired behavior when using redundant logic
provides the least desirable option when only communications controllers is to implement a maintenance, or testing mode,
between controllers fail. Ultimately, system operators must where one controller can be removed from the redundancy
analyze their applications and systems to determine which scheme to allow testing of the controller, updating of the
trade-off is most acceptable to them. configuration, or updating of the hardware. This requires the
C. Primary-Primary vs. Primary-Backup two controllers to exchange an additional Boolean piece of
information to instruct that a selected controller should not
Use of two redundancy controllers typically requires the
participate in a redundancy scheme and to also inform the other
logical decision of several operation modes. In this paper,
controller that its partner is not participating in the redundancy
primary-primary refers to the functionality where one controller
scheme. This is especially helpful in the primary-backup mode,
processes the information, makes control decisions, and then
where the backup mode transfers the active mode back to the
issues those commands out of the box. Once the waiting
primary. If the controller is undergoing testing, it will still
controller detects that the decision-making box is unavailable,
maintain a communications channel to the remote controller to
it picks up the decision-making and control-issuing
confirm its behavior.
functionality. If the first controller becomes available again
during this time, it becomes the controller-in-waiting to make
VI. CONTROLLER-TO-SCADA
decisions and issue the commands. In the primary-backup
designations, it works similarly to primary-primary, but the When both controllers actively collect data from IEDs, often
backup controller returns control to the original controller once the logic controllers still provide data concentration for
the backup controller becomes active and recognizes that the communications to the SCADA master station. Often these
primary controller has returned. Which mode to select is based SCADA connections either use DNP3, IEC 60870-5-101/104,
on system operation preference. The primary-backup method Modbus, or IEC 61850 Manufacturing Message Specification
introduces a small and insignificant amount of complication (MMS). There are many other legacy or proprietary protocols
into the process of deciding which controller should be active. (e.g., LG8979) that were commonly used for SCADA
communications in the past. However, this is uncommon on
D. Regaining Synchronization After Communications newer redundant systems and is outside the scope of this paper.
Are Lost If the connection to the SCADA master station uses
An aspect to consider is which redundant controller should IEC 61850 MMS, there is no need for concern with the
be active after power is lost and then restored to a controller that redundant controllers sending duplicate events or data to
causes a loss-of-synchronization between the controllers. If the SCADA because the IEC 61850 standard makes the client
mode is primary-backup, the logic to determine that controller responsible for managing which information has already been
one should be active requires no modification. If the primary received by assigning the desired entry ID when connecting to
controller is available it will become active. However, if the a buffered report in the MMS server.
mode is primary-primary, the decision is less straightforward. The Modbus protocol has no provision for reported events
Primary-primary mode operates on the condition that if the or data with time stamp or quality. Modbus provides only
other controller is nonresponsive, then the controller that present status, so the client collects just the present value of
 6

each point in the active controller. While Modbus provides VIII. EXPERIENCES FROM THE FIELD
many advantages for its simplicity, it typically does not make a This paper has covered a variety of technologies,
good SCADA protocol because of its lack of time stamps with communication topologies, and logic control decision-making
data. discussions up to this point. When designing a redundant
This leaves two primary protocols for redundant controllers system for logic controllers, each of these areas must be
to address synchronizing data from two controllers: DNP3 and considered and design decisions selected. The following
IEC 60870-5-101/104. Both protocols support time-stamped section covers a recent redundant logic controller solution that
data and reporting data changes either unsolicited or in a polled was implemented at a utility. This case study discusses the
mechanism. When these protocols have sent their changes to functional and design requirements of the utility, what
the client, the client sends back an acknowledgment so that the decisions were selected from the redundancy design previously
servers know what data changes have been sent to the client. If discussed in this paper, and some challenges encountered
the server receives no acknowledgment, the protocols during the implementation.
retransmit the data. Each protocol defines factors that determine
interval and frequency of data collection. The ability of each A. Redundant Logic Controller Functionality Requirements
server to keep track of whether the client has already and System Design
acknowledged data is critical for managing the duplicate data A solution using redundant logic controllers has been
that each logic controller collects. This functionality, typically implemented at a utility for use in its transmission substations.
offered as a firmware feature in substation controllers, requires The design incorporates dual main protection and automation
only user configuration. Manufacturers use different methods schemes, lending itself to the concept of a segregated control
to manage this functionality. End users need only confirm the room for new transmission substations.
communications channel planned between the controllers to The segregated control room design includes areas
accommodate this functionality. designated for Main 1 (primary) equipment and Main 2
(backup) equipment. This design allows for the testing or
VII. CONTROLLER ETHERNET INTERFACE CONSIDERATIONS complete replacement of equipment in either of the main
Depending on the application requirements, the system may control rooms while the primary equipment remains in service
benefit from both logic controllers sharing a single Internet via the alternate backup equipment. The following
Protocol (IP) address. A single IP address allows both requirements must be achieved for the redundant logic
controllers to appear as a single controller inside the substation. controllers solution:
This shared IP address is typically used on the WAN interface • There must be a primary and logic controller that
on the controllers. Outside systems will therefore act as if they would automatically fail over to the backup controller
are interacting with only a single device in the substation. in the event of a failure.
Outside systems do not need to have failover detection logic or • The database of both logic controllers must be
any information about the redundant system inside the synchronized and maintained continuously.
substation. This provides a straightforward and simple • The logic controllers collate all information received
approach for SCADA master stations. and transmit this information to multiple SCADA
However, sharing a single IP address for the LAN inside the master stations and external HMI clients.
substation is unlikely to provide the same advantages as sharing • Control operations are managed between the logic
an IP address on the WAN interface. With the LAN it is very controllers to ensure that only a single control is
likely that both controllers must communicate with a variety of submitted for operation at a time.
devices in the substation simultaneously. Sharing a single IP • The redundancy scheme allows for seamless
address forces only one controller to communicate to other connection to the SCADA master stations in the event
devices at a time. It also makes it difficult to communicate to of a failover. The SCADA master station is unaware
each controller for engineering access unless there is a separate of the redundant logic controllers. The controllers
network for this functionality. A shared IP address could be manage the communication requests from the master
implemented on the LAN with additional unique IP aliases on station, and only the primary controller responds.
each controller. However, this begins to unnecessarily • Should the SCADA master station acknowledge
complicate the network. For most redundant controller events on the primary logic controller, there is no
implementations, it is best to have three network interfaces with reporting of that same event when the master station
which the controller communicates: begins communicating with the backup logic
• A WAN connection to communicate with SCADA controller. This prevents transmission of duplicate
and other systems outside the substation events to the same SCADA master station.
• A LAN connection to communicate with IEDs in the • Both controllers must connect to main and backup
substation, perform data collection, and send control protection IEDs in the substation and use the
signals out IEC 61850 MMS protocol to acquire data from all
• A connection directly between controllers to IEDs. Redundant signals are managed accordingly to
communicate all necessary information for a ensure that duplicate signals are not transmitted to the
redundant controller solution SCADA master stations.
 7

• The solution should support a maintenance mode 2) Communication Between Logic Controllers
embedded within the logic controllers to facilitate The redundant logic controllers share two communication
testing and maintenance. sessions between each other in this design: an Ethernet
connection and an EIA-232 serial connection. This was
B. Redundancy Design Choices Topology 2 from Fig. 1. Over these connections the logic
Fig. 2 shows the logical connections between the parts of the controllers share primary/backup status and coordinate what
system discussed in this paper, the relays to logic controller, information has already been sent to the SCADA system. This
between the logic controllers, and to the SCADA connection. topology allows for a failure of the network communications
while keeping logic controller synchronization active.
3) SCADA Connection
A unique aspect of this system compared to many others is
that the SCADA system sends data requests to each logic
controller simultaneously via a serial connection. The
controllers are wired in the same way traditional EIA-485
multidrop systems are, but both controllers share the same
protocol address. In addition to each logic controller keeping
track of which data has already been transmitted to the SCADA
system, the logic controllers must keep track of which logic
controller should respond to the requests that are sent to both
logic controllers. This type of topology is not common and was
not covered in the fundamental portion of this paper. Usually
when there are redundant logic controllers inside a substation
that wish to appear as a single device to SCADA control centers
the redundant logic controllers will share a single IP address.
C. Challenges During Implementation
We satisfied and implemented the previously listed
requirements but experienced several other challenges.
Following are a few of these challenges together with their
solutions.
1) Multiple Communication Failures Between
Logic Controllers
The solution incorporated a primary/backup logic controller
redundancy scheme. To increase the availability and reliability
of the system and mitigate a split-brain situation, we selected
Topology 2 from Fig. 1 as the communications scheme
between the logic controllers for primary/backup and data
Fig. 2. Logical Communication Topology synchronization with Ethernet and serial connections for
1) Logic Controller to IEDs inter-controller communication. The logic controllers use both
In this system each of the redundant controllers collected links to determine the primary, backup, and maintenance states.
data from the IEDs via the IEC 61850 MMS protocol In the unlikely event that both communication links fail
simultaneously. This allowed each logic controller to have the (Ethernet and serial communications), the primary logic
most recent data from the IEDs. In IEC 61850 MMS it is the controller would deactivate and the backup controller would
responsibility of the clients to keep track of what data has been assume the active state, and maintenance mode would be
collected from the IED. This relieves the relays from managing prohibited. This fail-safe scenario prevents the possibility of
which data has been already sent to the client. Because this is a both controllers assuming the active state at the same time
new installation and all intra-substation communications are (including at startup) and both controllers attempting to make
Ethernet with modern IEDs, this allows for simultaneous data control decisions or respond to SCADA.
collection to the redundant logic controllers and no data
collection services are required to be coordinated.
 8

2) SCADA Communications IX. CONCLUSION

A key challenge that needed to be addressed included When designing a substation or system for which you are
SCADA protocol redundancy and event synchronization considering redundant controllers, it is easy to think about
between the logic controllers. The solution required using the redundancy as a single feature in a substation controller. While
IEC 60870-5-101 serial-based SCADA protocol on only a substation controllers typically offer a number of features to
single controller to communicate to more than one SCADA implement redundant systems, many factors are still configured
master station (national control and regional control). The and managed by the end user. These are choices such as
SCADA master station would be unaware of the redundancy selecting how many communication interfaces each controller
scheme and would not be performing any failover if will have, how data are collected from IEDs, how duplicate data
communications to the primary controller failed. This required are sent to SCADA, selecting an operation scheme between
that each logic controller share the same address and manage controllers, and managing control logic. Each site has a variety
the failover. When the master station confirms the reception of of other design parameters that require a combination of
an event from the primary logic controller, a message is sent to functionalities that results in a customized solution for
the backup controller through use of the synchronization links. increased system reliability. There is no single feature or check
The backup controller uses this information to identify which box; the end user must instead make a series of choices to create
event the SCADA master station receives, acknowledges it, and a secure system.
recognizes the same event in the backup controller system
database. This prevents transmission of duplicate events to the X. REFERENCES
same master station if a transition between controllers occurs.
[1] IEC 62439-3:2016 Industrial Communication Networks – High
This type of SCADA connection was challenging because the Availability Automation Networks – Part 3: Parallel Redundancy
serial communications wiring did not physically allow for a Protocol (PRP) and High-Availability Seamless Redundancy (HSR),
mechanism for only one controller to recognize the SCADA 2016.
communications like an Ethernet system provides (both units [2] M. Hadley, D. Nicol, and R. Smith, “Software-Defined Networking
Redefines Performance for Ethernet Control Systems,” proceedings of
shared a single IP address). This forced additional logic in the
the Power and Energy Automation Conference, Spokane, WA,
logic controllers to determine when an IEC 60870-5-101 March 2017.
SCADA server should respond to a valid data request, and [3] IEEE 802.1D IEEE Standard for Local and Metropolitan Area
when it should ignore the data request based on when the logic Networks, 2004.
controller was the primary controller. [4] Q. Yang and R. Smith, “Improve Protection Communications Network
If any of the controllers were in maintenance mode, the Reliability Through Software-Defined Process Bus,” proceedings of the
Power and Energy Automation Conference, Spokane, WA, March 2019.
redundant controller would assume the role of primary
controller and communicate with the SCADA master stations.
XI. BIOGRAPHIES
The controller under maintenance would then suspend all
Sagar Dayabhai is currently pursuing his PhD at the University of
communications to the SCADA master stations. This Witwatersrand (Wits), South Africa in the field of smart grids. He received his
maintenance state was synchronized between the controllers BSc Eng. (electrical) degree from Wits in 2009 and earned his MSc Eng.
and allowed for project updates and testing on the logic (electrical) degree in 2014. He is a registered professional engineer with the
controller when it was in maintenance mode. Engineering Council of South Africa (ECSA). Sagar was one of the IEC Young
Professionals elected for South Africa in 2016 and is currently a member of
3) Managing Data From Redundant IEDs IEC TC57 WG10 and WG15. After working at Eskom in the field of
telecommunications and SCADA, he moved to Consolidated Power Projects
Since this system contained redundant protection IEDs, each (CONCO) as a Senior SCADA/Automation Engineer. Sagar now holds the
controller collected data from both redundant IEDs. This position of System Control Manager at CONCO Energy Solutions.
required each logic controller to subscribe to independent
datasets and reports in the redundant IEDs. The controller then Brian Waldron is a development lead automation engineer with Schweitzer
Engineering Laboratories, Inc. He has several years of experience in designing
had two copies of the information it needed to send to SCADA and troubleshooting automation systems and communications networks. He has
and use in its logic applications. The controller selected an authored several technical papers, application guides, and teaching
information source based on the quality and communication presentations focusing on integrating automation products. Brian graduated
from Gonzaga University with a B.S. degree in electrical engineering.
status of the IED. If quality was valid and IED communications
were good, each controller was programmed to have a preferred Marcel van Rensburg is currently pursuing his MS at Montana Tech in power
data source. systems. He received his BTech from Tshwane University of Technology,
South Africa in 2016. He is a member of IEEE and a secretary for his local
4) Managing Controls to Redundant IEDs Montana IEEE section. Marcel worked as an application engineer at Schweitzer
Since this system contained redundant IEDs for protection, Engineering Laboratories, Inc. for 6 years and is now a product engineer.
there was a primary and a backup IED. When the logic
controller needed to send a control operation it was individually
managed between the logic controllers and sent to the active
IED. To ensure that authorization of the bays was correctly
managed for safe operation during a failure of the active logic © 2020 by CONCO Energy Solutions and
controller, control authority information was synchronized Schweitzer Engineering Laboratories, Inc.
between the controllers. All rights reserved.
20200130 • TP6964-01