The DeltaV system supports redundant controllers. A redundant controller consists of a pair of standard controllers on separate 2-wide power/controller carriers connected together. The controllers in a pair must be the same model (for example MD Plus, SX, PK, and so on). Each controller requires a separate power supply mounted on its carrier to support redundancy.
One of the controllers in the pair is the active controller. The other controller is the standby controller. The standby controller contains the same configuration as the active controller. When an active controller fails, the standby controller takes over providing uninterrupted control operation without user intervention. The standby controller gets updates of certain parameter values in the active controller over the redundancy link, but does not execute control logic. When the switchover occurs, the new active controller reads back I/O data and the modules begin to execute. A limited amount of re-initialization occurs in order to resume control without disruption. For example, certain control and output function blocks begin executing in out-of-service mode and climb to their target mode in order to force handshaking with other blocks. In most cases control actions resume on the first scan after a switchover. For complex modules, one or a few scans of handshaking may be required. The apparent mode change on a switchover is expected and has no adverse effect on control.
Even though each controller in a redundant pair does have its own MAC address and node address, a redundant controller counts as a single node on the DeltaV control network in terms of network capacity.
A redundant controller requires a redundant controller license. Downloading a redundant controller without a license results in a description of "Redundancy Not Licensed" in Process History View. In addition, the RedEnb diagnostics parameter has a value of NO if no license is present.
Performance considerations for MD Plus and SD Plus controllers
Redundancy typically requires more controller CPU than the same configuration in a simplex controller. Additionally, larger configurations (more modules, more parameters) require more CPU time for redundancy processing than a smaller configuration. A majority of the redundancy processing time is allocated against the 35% of total controller CPU that is reserved for overhead processes. A minor amount of the redundancy processing will be allocated to the 65% of time reserved for user-configured control processing and decreases the FreTim parameter by several percent.
It is important to note that the FreTim parameter is the percentage of the available processor time for execution of the user configuration. Available processor time is approximately 65% of the total CPU that is reserved for user configuration tasks and the remaining 35% of controller processor time is reserved for other controller functions. The controller's idle time measures the entire CPU time available and is shown in the DeltaV Diagnostics' Time Utilization Chart.
It is possible that certain extraordinary configurations which perform properly with a simplex controller may not do so when a redundant controller is added. For example, additional CPU load from redundancy processing could cause the overhead processes to consume more than the allocated 35% of processor time. Additional load could be due to a higher than typical number of modules or parameters or other atypical usage (such as communications loading exceeding the recommended limits). When this occurs, overhead processing can consume some of the 65% of the CPU allocated to the user configuration. If adequate FreTim is available, this is not a problem. However, when FreTim is very low, the overhead system processes and the user configuration tasks compete for the CPU resources. In most cases this results in module slippage.
You can use the DeltaV Controller Load Estimator tool to assist in determining controller load by measuring the controllers' Free CPU in % (FreTim). However, the Controller Load Estimator might not adequately account for redundant CPU loading on systems with more than 500 modules if the redundancy creates additional overhead loading exceeding the 35% reserved for system functions. Monitoring the FreTim parameter and the Time Utilization Chart is also important. The Controller Load Estimator is provided on the DeltaV installation disk #1 in the _Support\Tools\LoadEstimator folder.
Performance considerations for MX, MQ, P-series and S-series controllers
- Control Performance Index: an indicator of controller module execution. Included in this metric is the amount of CPU time spent executing control for the high, medium, and low priority modules.
- Comms Performance Index: an indicator of the data that is transmitted and received by the controller. This metric takes into account bandwidth comparisons in terms of the average amount of data sent and received as compared to the maximum bandwidth that is supported for the controller, solicited and unsolicited data transfers, the amount of Remote I/O data sent and received, and the depth of the communications queue.
- System Performance Index: an all inclusive indicator of the controller processor's total free time.
- Free Memory Index: an all inclusive indicator of the controller's free memory.
The dialog also shows the overall performance index (Perf_Index) which is a comparison of the smallest values of the other indices. Refer to the DeltaV Diagnostics online help for more information on using the Controller Performance dialog.
Switchovers
A switchover from the active to the standby controller can occur for the following reasons:
-
Hardware failure within the active controller
-
Communications failure between the active controller and the I/O subsystem
-
Communications failure of both the primary and secondary network connections in the active controller
-
Removal of the active controller from the carrier
-
Manual switchover request
-
Loss of power supply for the active controller
-
Memory failure (the active controller detects a RAM or ROM failure)
-
Software watchdog timer trip
When a switchover occurs, the node status area on the Alarm Banner indicates the status change to the operator. The system regenerates active, unacknowledged and suppressed alarms. The system does not regenerate alarms that are both inactive and acknowledged. Alarm states are maintained during a switchover but General I/O Failure (IOF) Alarms may change state momentarily.
If a switchover occurs on an MD Plus or SD Plus controller while the ProfessionalPLUS workstation is offline, the controller Diagnostics parameters PAvail remains No and Status remains Standby not detected until the ProfessionalPLUS workstation is online.
The system stores a record of each switchover and the reason it occurred (if known). The switchover is logged in the Event Chronicle. After a switchover, the controller attempts to communicate with all of its I/O. The Event Chronicle description: Switchover I/O Readback Timeout means that after a redundancy switchover, the controller was unable to communicate with its I/O before the timeout period.
If the standby controller should fail, the software disables switchovers until you replace the failed unit.
Generally, there is no user impact related to a controller switchover. However, there are certain situations that the user should be aware of when configuring control. These relate to unit modules and phases, and HART digital variables, external references to other controllers, and forced debug parameter values.
Unit module and phase runtime data are not transferred to the standby controller before PAVAIL=YES and therefore one should wait at least 1 minute beyond PAVAIL=YES before doing a switchover so that the standby controller is ready to take control.
HART digital variables will have a value of zero with Bad status for six or more seconds after a controller switchover. If you access a HART digital variable in an expression (Calc, Condition, or Action function block; SFC transition or action), be sure to also consider the status of the variable in the expression. For example, use the value only when the status is Good. Accessing HART digital variables in AI function blocks or by external reference parameters should be done for monitoring, not control purposes.
A Condition function block that references a parameter in another controller has the potential to change its OUT_D parameter when the referenced controller switches over. You can prevent this by selecting the Abort on Read Errors option in ALGO_OPTS. This is recommended unless you want OUT_D.ST to be BadNoCommNUV during the switchover. In this case leave the Abort on Read Errors option deselected and change ERROR_OPT to Use Last.
Forced debug parameter values (that is, those set in Control Studio's debug mode) are not retained following a controller switchover. The value of the parameter is set to the default value as configured. If forced debug parameters exist, the system displays a warning when you start or terminate a debug session.
In general, the system regenerates process and device alarms during a controller switchover. This includes all active, unacknowledged and suppressed alarms. Inactive and acknowledged alarms are not regenerated. General I/O Failure (IOF) alarms may change state momentarily. The Time In values (displayed in the alarm list) and the alarm states are not affected by the regeneration. The Time Last values on the alarm list are updated. Alarm strings may not be maintained.
Manual Switchovers
Users with the Diagnostic Key can initiate a switchover from DeltaV Diagnostics. In Diagnostics, a redundant controller has a Redundancy subsystem. To perform a manual switchover, right-click the Redundancy subsystem, and then click Redundancy switchover.
Controllers have three parameters that determine whether a manual switchover can occur:
-
PExist -- indicates whether the standby controller is physically present
-
PAvail -- indicates whether the standby controller has received a download and is ready to take over
-
RedEnb -- indicates whether redundancy has been enabled for the pair. In order for redundancy to be enabled, the active controller must have received a download.
In order for a manual switchover to occur, all three of the above parameters must have a value of Yes. If not, the Status parameter provides additional details about why the controller cannot switch over.
Configuration Analysis tool
- Incorrect Function Block Order – First-time module execution after a switchover can be affected if function block wiring, feedback wiring and module execution order are not correct. Function block execution order determines the actual order. The tool locates instances where the function block wiring does not match the function block execution order. Feedback wiring, such as the wiring between function block BKCAL parameters, is expected and is not noted by the tool.
- SFC Confirm Expressions Hardcoded to True – SFC confirm expressions hardcoded as TRUE can cause unexpected results if the action expression does not happen immediately on switchover. Actions with confirm expressions set to TRUE are marked as complete immediately with no retry mechanism if the write fails (for example, if the write is being done to a controller at the instant it is undergoing a switchover). For this reason use of TRUE confirms is not recommended, readback of written parameters or confirmation of the write status with the .AWST parameter should be used. TRUE confirms in actions that reference only internal parameters are omitted from the check.
Review the results of the analysis report to determine if the items in the report are configured as intended.
To access the tool in Control Studio, open the module and click . You can add this command to the Quick Access toolbar. Click the down arrow for the Customize Quick Access Toolbar, then click More Commands and click Analyze Configuration.
To access the tool in DeltaV Explorer, select the module from any location in the hierarchy and right-click, Analyze Configuration. You can perform analysis on any of the following items in DeltaV Explorer:
- Area: Displays the analysis report for all modules (excluding SIS modules) inside the area.
-
Library Items: Displays the analysis report for individual modules or entire containers. This includes Composite Templates, Module Templates, Equipment Module Classes, Control Module Classes, and Phase Classes.
-
Individual Control Module: Displays the analysis report for modules inside the areas subsytem or the controller module assignments subsystem.
Creating a Controller Placeholder
To create a placeholder for a redundant controller in DeltaV Explorer, add a controller node, and then click the Redundant Controller option on the node's Properties dialog.
Installing a Redundant Controller
Installing a redundant controller is as simple as installing a single controller. The auto-sense feature in the DeltaV system recognizes two controllers on adjacent, connected carriers as a redundant controller.
Refer to the DeltaV hardware manuals for physical installation instructions.
After installation, drag the controller from the Decommissioned Controllers section to the control network or to a redundant placeholder. Assign an appropriate redundant license to the controller and download the controllers to enable redundancy.
Installing a Standby Controller
You can connect a second controller to an existing controller's carrier to introduce redundant control without interrupting your process. The system automatically commissions a standby controller when you install it. It is not necessary to remove or decommission the active controller. The active controller continues to operate without interruption. How the standby controller gets its configuration depends on the controller type:
-
MQ, MX, SQ, SX, and PK controllers — The standby controller receives its configuration from the active partner over the redundancy link.
-
MD Plus and SD Plus controllers — The system automatically assigns the standby with an address and downloads the standby controller with the latest download and with any online changes made to the active controller.
To install a standby controller follow these steps:
CAUTION!Follow the steps in the order shown. Do not install a second 2-wide carrier that already has a power supply and controller installed. Doing so will result in a loss of configuration data for the active controller.
-
Using the DeltaV Explorer, assign an appropriate redundant controller license to the controller node that you want to make redundant in DeltaV Explorer.
-
Install a second 2-wide or controller carrier to the left of the current controller's carrier.
-
Insert the appropriate Power Supply according to the controller hardware instructions (different controller models have different slots for the power supply).
-
Connect a controller to the carrier. The version of the controller you connect must match the existing controller's version.
-
DeltaV Explorer displays a redundant controller icon in place of the simplex icon.
-
If you are changing a simplex controller to a redundant pair, there is a one-time step required. In the DeltaV Explorer, right-click the controller, click Download, and then click Setup Data. This turns on redundancy for the pair but does not disrupt any existing process control.
Replacing a Failed Controller
When a switchover occurs, remove the failed controller and plug in a replacement. The system automatically commissions the replacement and makes it the standby controller.
Removing Controllers
The commissioning and decommissioning function in the DeltaV Explorer affects both controllers in the pair. You can remove one controller in a redundant pair without decommissioning it. For example, if you remove the standby controller and reinstall it, the address is reestablished, the standby receives a current download and is ready to accept a switchover. If you remove the active controller without decommissioning, the standby takes over control. When you reinstall the controller, the address is reestablished and the reinstalled controller becomes the standby controller. Anytime one controller is removed from a pair, the DeltaV Explorer continues to identify the controller as redundant. If you want to change a redundant controller to simplex, you must change the properties of the controller in DeltaV Explorer and download the controller's setup data.
Do not disconnect a carrier and its controller while the controller is powered and operating. If a controller and carrier must be removed, power down the controller before disconnecting the carrier and its controller.
Before removing both the controllers of a redundant pair, decommission the node through the DeltaV Explorer.
Diagnostics
The DeltaV Diagnostics displays a redundant controller icon in place of the simplex icon for redundant controllers and placeholders. The DeltaV Diagnostics enables you to view diagnostics, MAC addresses, and node (IP) addresses for both controllers in a pair. Diagnostics also enables you to flash the lights on either controller in the pair independently using the Identify Controller menu selection. All diagnostic and address information for the standby controller is accessed through the active controller at the Standby level (below the Redundancy Subsystem). Diagnostics requires a commissioned communicating controller to be able to flash the lights (unlike DeltaV Explorer, which can use the database MAC address when the controller is decommissioned). However, you can Telnet or perform a TCP/IP PING directly to the primary or secondary network connection node addresses on the standby.
Upgrading Controller Firmware
If the redundant controllers require a firmware upgrade (such as for a DeltaV version upgrade), the controllers contain flash ROM that allows you to update the firmware while your process is running. Simply upgrade the standby controller, then perform a manual switchover from Diagnostics. You can then upgrade controller that you recently switched to standby at your convenience. There is no downtime.