PAC Controller Redundancy

PAC Controller Redundancy

Applications that required high availability, advanced data handling, and superior communication capability are a good fit for PAC redundancy

Author: Advantech

E-mail:

1

Most PLC, and PAC controllers exhibit a high degree of reliability but, certain critical industrial automation applications simply cannot tolerate downtime. The preferred solution in these cases is redundancy, as this greatly increases availability.

Traditionally, redundant automation systems have been very expensive and complex, particularly as compared to standard automation systems. Upfront hardware costs have been very high and implementation has been difficult, requiring extensive software development effort. An example is the triple modular redundant (TMR) system, often employed in critical process and other automation applications.

But now, redundant PAC automation systems are available that greatly reduce both cost and complexity. These solutions simply require the purchase of a CPU and implementation just requires an extra configuration step to select the redundant option.

Most PAC redundant automation systems provide redundancy at the power supply and CPU levels, the two most critical areas. Redundancy at the I/O, communications and cable levels can also be provided depending on the needs of the application.

This White Paper will examine the benefits of PAC redundancy in industrial automation applications and will then show how to implement a redundant solution with a PAC automation system.

Benefits of Redundancy

For some applications, near 100% uptime is a must, and redundancy is thus required. For example, many discrete part manufacturing facilities must run 24/7 to keep up with demand, since any downtime will result in reduced sales or even lost customers.

Most discrete part production lines are a mixture of batch and continuous processes. Interruptions to batch processes can cause huge delays and other problems, particularly for batches with long execution times. If a batch takes eight hours to complete, then an interruption seven hours into the process will often require a restart of the entire process, causing significant delays. Continuous processes are generally more tolerant of interruptions, but can also require long restart times.

Environmental Facility Management Systems often require near 100 percent uptime, particularly if the environment directly affects the production process, as with a clean room. Failures in this area can not only result in lost production at the time of failure, but can also require significant restart times as the clean room is reconditioned and recertified for use.

Another example of an area where redundancy is required is control and monitoring of unmanned remote sites, as with a supervisory control and data acquisition (SCADA) system for an electrical distribution and generation system. In this case, a failure at a remote generating site such as a wind farm could cause significant problems including power outages, and could take a long time to address as the site might be hours away from the nearest service location.

In all of these cases and more, near 100% uptime is essential, and a redundant automation system is often required.

Options for Redundancy

Redundancy can be implemented with Programmable Logic Controllers (PLCs), Programmable Automation Controllers (PACs). Very large process plants often use a Distributed Control System (DCS) to provide redundant control but, this white paper is instead focused on factory automation, SCADA, and building automation applications areas where smaller scale control systems are a better fit.

Any consideration of how to create a redundant automation system must explore the technological options as well as the up front and subsequent costs. As ReliablePlant.com puts it, “Smart redundancy has to do with taking a critical look at your systems, looking at risks and searching for opportunities where a little redundancy may pay big dividends1.”

A PAC redundant automation system provides many benefits as compared to a traditional PLC- or PAC-based redundant automation system. These benefits are listed and are discussed in detail below.

Benefits of PAC Redundancy

1.  Reduced downtime

2.  Less maintenance

3.  Low cost as compared to traditional redundant automation systems

4.  Reduced complexity as compared to traditional redundant automation systems

5.  Provides benefits of PAC control along with redundancy

6.  Superior remote communications capabilities

The prime benefit of any redundant automation system is reduced downtime. With dual power supplies and CPUs, and near instantaneous switchover in the event of a failure, continuous operation is assured. Provided the failed component on the primary system is replaced in an expeditious manner while the secondary system is active, users can expect near 100% uptime from a redundant automation system.

Maintenance is reduced because it’s no longer necessary to closely monitor the operation of the power supply and the CPU in order to predict a failure, as a failure of either of these components won’t cause downtime.

Evaluating how redundancy can mitigate risk and provide a good return on investment requires a level of analysis and assumptions. “The true value of redundancy in a control system is shown when a critical failure occurs and there is no loss in production, damage to equipment or injury/death of humans”, points out Manufacturing.net2. They go on to liken redundancy analysis to insurance analysis, stating “what are the chances of a failure occurring and what are the consequences?” The end user must make these assessments and chose a technical approach that fits their operation.

ISA InTech advises that “The key is to be selective about where to invest in redundancy. By starting with the overall system, assessing which subsystems are required to be highly available, and considering the software environment as well as the hardware domain, organizations can achieve the high availability they need while driving down costs.3”

Traditionally, redundant automation systems for machine control have been implemented with PLCs or PACs. Although both solutions reduce downtime and maintenance, they are often very expensive and complex to implement.

And even after redundancy is implemented with a PLC or a PAC, the automation system will exhibit none of the benefits of PAC control in terms of greater data storage, superior data manipulation, and seamless communications.

To realize the benefits of PAC control and monitoring, it’s sometimes necessary to implement a two-level mix of PCs, and PACs or PLCs. This not only adds costs to the base system, but also adds significant complexity and design effort when implementing redundancy solutions.

Because redundant PCs have been in use for commercial applications for decades, manufacturers are now able to offer redundant PAC industrial automation systems at a reasonable cost and with relatively simple implementation from the end user’s perspective.

The resulting automation system provides the required redundancy, while still delivering the benefits of PAC control and monitoring.

Redundancy Implementation Details

At the PC level, redundancy options can range from pure software to pure hardware, or to some combination of the two. Software options available today can be applied on a standard hardware platform. But the bottom line is that without some level of hardware redundancy or multiple hardware platforms available in a pool, the redundant capabilities are limited.

Previous generations of PC redundancy technologies were quite specialized, and extremely expensive. They involved duplexed arrangements of unique hardware, processing the software in parallel lockstep. Any failed hardware component was immediately superseded by its mirrored counterpart, with a seamless transition that was transparent to the operating software.

The current generation of PC technologies can leverage more standardized components in order to provide a more cost effective redundant system. However, several subsystems must be considered in order to provide complete redundancy options. These areas can be summarized as follows:

1.  Power Supply

2.  Central Processing Unit (CPU)

3.  I/O

4.  Communications and Networking

5.  Cabling

A PAC platform that offers redundancy for each of these areas gives the user the power of a PC platform, but with far greater reliability. Even though each of the individual components is fairly reliable, combining them in a duplex manner extends normal reliability into the realm of high availability.

Figure 2 depicts a redundant PAC control system with dual CPUs and power supplies. In this type of configuration, communication between the CPUs and the power supplies allows uninterrupted operation in case of failure.

From a hardware standpoint, any PC that is actively controlling a process in a redundant fashion must use digital signal processors (DSP) designed to maintain data synchronization between the primary and secondary controllers. Active controllers will recognize each other and perform a “handshake”, and then they will ensure that all required data remains in sync.

In addition, the primary controller must send out a “heartbeat” or “keep alive” signal to the secondary controller so that it knows it’s still alive and does not need a backup. Redundant controller pairs follow rules allowing them to arbitrate for the one “primary” role. The non-primary controller then becomes the “secondary”. Any time a secondary controller loses the heartbeat from the primary, then the secondary takes over the entire operating role.

Conditions such as power supply failure, communications failure, or a failed program could cause the primary controller to stop sending the heartbeat signal. A time duration of just 10 milliseconds is enough for a secondary controller to detect a primary failure and take over operation. This quick response time is important to maintain continuous control.

Traditional consumer PCs must be shut down to repair or replace parts. However, redundant industrial PC controllers feature components that are hot-swappable. In fact, they are physically arranged to facilitate removal and replacement of components.

Role of Software in Redundancy

Software also plays a part when configuring an industrial automation control system. PAC systems used in industrial automation often use an operating system optimized for real-time control. A typical and well-proven candidate is Windows CE. “Real-time” operating systems are designed to favor reliable performance and response times instead of high throughput. The ability to almost deterministically process data as it is received makes such an operating system suited for real-time control.

Control of the actual process or machine is performed by software on top of the operating system. This type of software running on a PC is often called “softlogic”, and many times is based on one or more of the languages specified in IEC 61131-3 which is an open international standard for industrial controllers.

The PAC controller’s operating program is configured with a PAC software programming package, and then downloaded to the redundant pair of PCs. Modern software programming packages for redundant PAC control provide simple setup for redundancy through menu driven commands (Figure 3).

Redundancy in itself is not fully valuable without condition monitoring to verify the health and integrity of the platform. Therefore, a key feature of any redundant system is the ability to continuously monitor the status of duplexed components, and report any failures up to the operating system and the application software. Once the trouble information is available at the application layer, the system can locally alarm and indicate the problem.

More importantly, the issue can be communicated to the main control location, or even trigger other notification means such as an email, text message, or alarm phone dialer. Most PAC control systems also offer an option for a remote user to send commands, allowing for some problems to be completely resolved remotely.

The final goal of a redundant platform is not to ensure that the system will run indefinitely. Instead, the hardware redundancy ensures continued operation of the system even if a failure occurs. This buys enough time for maintenance personnel to be notified and to reach the site to initiate repairs.

This level of protection is just right for operations with widely distributed facilities that are not staffed with trained technicians, or are completely unmanned. Water and wastewater utilities, as well as electrical power generation and distribution operations, are two prime candidates for implementing PAC redundant systems.

These types of systems are typically staffed with trained personnel, but their operations are distributed across a wide geographic region. Redundancy will ensure continued operation of a remote site in the event of a failure, allowing time for maintenance personnel to reach the site and repair the problem.

Distributed Generation Requires Redundancy

Consider an electricity provider operating a number of regionally distributed power generation facilities, possibly utilizing renewable technologies such as solar or wind.

In contrast to a traditional centralized power plant, this type of operation is called a distributed energy resource (DER). DERs are comprised of comparatively small scale decentralized power storage and generation sites, typically less than 15MW per location. Power generated at DER sites meets on-site needs, and any excess is sold to the utility through the power distribution grid.

Due to their reduced size and local distribution, DERs are more likely to be owned by end users than utilities. A smaller end user would still have trained staff and/or contract employees, but could be expected to have fewer resources than a large established utility. This makes it all the more important to leverage limited talent in the most effective manner, which is often accomplished through various remote monitoring and control solutions, best implemented with redundant PAC control.

DERs by their very nature offer a type of geographical redundancy or at least resiliency because the sites are distributed and it is less likely that a failure at any one site would take down another site. However, that does not mean that an operator would want any facility to go offline unnecessarily.

For the mix of technologies in use at a typical DER site, it is probable that a PAC SCADA/control system would be very advantageous. Not only would basic control functionality be provided, but higher level functions such as monitoring, logging, trending, and alarming for power and other operating parameters could be achieved. This consolidated approach would provide high performance control, visibility, and communications in a single package in a cost-effective manner.