Introduction

In a previous blog, I introduced Operational Technology (OT) which focuses on the technologies used to monitor and control physical processes and industrial assets. That article also explored the key differences between OT and Information Technology (IT) environments.

From an operational perspective, all organisations should have plans in place to ensure Service Continuity in the event of disruptions that impact normal operations.

Operational disturbances can occur for many reasons and may interrupt the smooth running of an asset. If not managed correctly, these incidents can quickly escalate into larger operational or safety issues.

In some cases, the disruption may be severe enough to force a shutdown of operations. When this happens, organisations must follow structured processes or standard operating procedures (SOPs) to restart systems safely, securely, and in a controlled manner.

This is where Disaster Recovery begins.

 

What is Disaster Recovery?

Disaster Recovery (DR) refers to the structured process of restoring operational systems and services after a major disruption that has stopped production or plant operations.

Disaster Recovery is a component of a wider Disaster Management Cycle. While recovery focuses on restoring operations, organisations must also consider how they respond to incidents, prepare for disruption, and mitigate future risks.

Together, these capabilities form a continuous cycle of Operational Resilience.

Common Failure Points During OT System Restart

Restarting industrial systems after a disruption is one of the most critical and high-risk phases of Disaster Recovery.  Even when systems have been restored, failures during restart can lead to extended downtime, equipment damage, or safety incidents.

Some common failure points include:

• Control System Synchronisation Issues
  • Controllers, HMIs, and servers may be:
    • Out of sync
    • Running mismatched configurations
  • This can result in:
    • Incorrect process control behaviour
    • Alarm floods or system instability
• Loss or Corruption of Operational Data
  • Historian or configuration data may be:
    • Incomplete
    • Corrupted
  • This impacts:
    • Visibility
    • Decision-making during retstart
• Sequence of Startup Not Followed
  • Industrial processes require strict startup sequencing
  • Incorrect order can:
    • Damage equipment
    • Cause process imbalance
    • Trigger safety trips
• Field Device and Instrumentation Failures
  • Sensors, actuators, or PLC I/O may:
    • Fail to respond
    • Provide incorrect readings
  • This leads to:
    • Unsafe or unreliable operations

• Network and Communication Breakdowns
  • OT networks may not fully recover:
    • Roles may be unclear
    • Decisions may be rushed
  • Result:
    • Loss of communication between systems

• Human Factors and Role Confusion
  • During high-pressure recovery
    • Roles may be unclear
    • Decisions may be rushed
  • Leading to:
    • Errors in execution
    • Delayed Recovery
• Incomplete System Validation
  • Systems are started without:
    • Full functional checks
    • Safety Validation
  • This increases the risk of:
    • Secondary failures

 

What Does a Structured Recovery Approach Look Like in Practice?

A structured Disaster Recovery approach is not just about having a documented plan, it is about executing recovery in a controlled, sequenced, and validated manner.

In OT environments, recovery typically follows a defined progression that is made up of the following actions:
 

• Stabilisation and Safety Assurance
  • Confirm plant is in a safe state
  • Isolate affected systems
  • Ensure no ongoing hazards

• System Integrity Verification
  • Validate:
    • Control system configurations
    • Network availability
    • Data integrity

• Infrastructure and Network Recovery
  • Restore:
    • Servers
    • Networks
    • Communications

• Control System Restoration
  • Bring back:
    • PLCs / DCS / SCADA
  • Ensure:
    • Synchronisation
    • Correct configurations

• Field Device and Process Validation
  • Check:
    • Sensors
    • Actuators
    • Instrumention

• Controlled Process Restart
  • Follow defined statrtup sequences
  • Gradually re-introduce operations

• Monitoring and Stabilisation
  • Observe system behaviour
  • Validate performance
  • Address anomalies

• Post-Recovery Review
  • Capture lessons learned
  • Feed into mitigation and improvement

Many organisations have documented plans, but lack a clearly defined and tested recovery execution model. 
 

What are the causes of a Disaster Recovery event?

There are many potential causes of a Disaster Recovery event. These may originate from:

• Operational failures
• External threats
• Environmental or natural events

Some common examples include:

• Site-wide power failure
• Natural disasters such as flooding or extreme weather
• Cyberattacks (e.g. ransomware)
• Central control system failures
• Data historian corruption or loss
• Network infrastructure failures

In industrial environments, even a single failure in a critical system can disrupt production and require a structured recovery process.

How can organisations prepare for a such an event scenario?

Organisations can prepare for potential disruptions by developing a Disaster Recovery Plan (DRP).  A DRP is a living document that defines the procedures, roles, and responsibilities required to restore operations following a disaster scenario.

A well-developed DRP typically includes:

• Clearly defined roles and responsibilities
• Recovery procedures for different disaster scenarios
• Communication plans
• Defined recovery objectives and timelines

Developing a DRP is not a single task but a structured process.
 
 

Five Key Steps to Developing an effective Disaster Recovery Plan

 

Step 1. Discover

In the initial phase, organisations identify critical assets and potential threats to their OT systems and operational infrastructure.
This stage includes identifying possible disruption scenarios and understanding which systems are essential for maintaining operations.

 

Step 2. Analyse

During this phase, organisations perform a Business Impact Assessment (BIA) to understand how each disruption scenario could affect operations.

The BIA helps determine:

• Critical systems and assets
• Operational dependencies
• Acceptable downtime limits

Step 3. Design

In the design phase, organisations develop recovery strategies to restore operations and OT systems.
This may include:

• Backup strategies
• Redundant infrastructure
• Recovery procedures for control systems and networks

Step 4. Build

During the build phase, the recovery strategies are documented in detail.
This includes creating:

• Step-by-step recovery procedures
• Roles and responsibilities for incident response
• Communication and escalation plans

The resulting document becomes the organisation’s Disaster Recovery Plan (DRP).

 

Step 5. Validate

A Disaster Recovery plan is only effective if it has been tested and validated.
In this phase, organisations conduct:

• Tabletop exercises
• Simulated disaster scenarios
• Operational drills

These validation exercises help identify gaps in the plan and provide opportunities for continuous improvement.
 
 

Why is a Disaster Recovery Plan important?

 
A Disaster Recovery Plan (DRP) plays a critical role in restoring operations safely, efficiently, and with minimal disruption.

Much like emergency procedures used during safety incidents, a DRP provides clear instructions on:
 

• Managed and reliable operations recovery that consider process RTOs and RPOs
• What actions need to be taken
• Who is responsible for executing them
• How communication should be managed during the recovery process

Without a DRP, organisations risk confusion, delays, and potentially unsafe recovery actions during high-pressure situations.

RTO stands for Recovery Time Objective which defines the target time for restoring normal operations to prevent critical operational failures.

RPO stands for Recovery Point Objective which determines the frequency of backups, bridging the gap between the last valid backup and the disruption.

In many regulated industries, having a disaster recovery capability is also a requirement to meet compliance and regulatory obligations.
 

How Do You Know if Your Disaster Recovery Preparation is Effective?

The effectiveness of a Disaster Recovery Plan (DRP) can only be proven through regular testing, capturing lessons learned and continuous improvement.

Organisations should regularly perform:
 

• Simulation exercises
• Recovery drills
• Scenario-based testing

These activities allow teams to practise their response and ensure that recovery procedures work in real-world conditions.

Testing also helps identify weaknesses in systems, processes, or communication structures that can be improved before a real incident occurs.
 

Conclusion

Even short operational disruptions affecting critical industrial systems can result in significant production losses. Developing a structured recovery capability ensures that operational systems can be restored in a safe and controlled manner.

By developing a structured OT Disaster Recovery Plan, the facility will benefit from:

• Improved preparedness for operational disruptions
• Reduced risk of extended production downtime
• Clearly defined recovery procedures for critical systems
• Improved coordination during recovery events
• Increased confidence in the site’s ability to safely restore operations following a disruption

If your organisation has not recently tested its OT disaster recovery capability, now is the time to start. Understanding how quickly your systems can recover may make the difference between a minor disruption and a major operational outage.

If your organisation relies on industrial control systems, understanding your disaster recovery readiness is critical.

MHNK Associates offers independent Operational Technology Disaster Recovery Benchmarking to help organisations evaluate recovery capabilities and identify improvement opportunities.