LFCE 106 🐧 High Availability — Concepts - KandZ

⏱️ Estimated read: 10 min | |

High availability is the property of a system that remains accessible despite the failure of its individual components. It is not a single tool or a product; it is a design goal achieved by eliminating the single points of failure and by introducing the redundancy that allows the surviving components to take over the work of the failed ones. The previous chapters covered the services that the system provides; this chapter covers the concepts that determine whether those services survive a hardware fault, a network partition, or a software crash. The LFCE exam expects the candidate to deploy, configure, and maintain the high availability, the clustering, and the replication .

Quick How-To Guides & Fast FixesNeed a quick fix or step-by-step solution? Browse our fast answers on HowTo.KandZ.me:

Operating Systems CLI →Dev Environment Languages →Web Network Tools →Security Accounts →Security Access Tech →Device Phone Basics →Social Messaging →Digital Files Media →

The foundation of the high availability is the cluster: two or more computers, called nodes or members, that work together to provide the service . The cluster types are distinct. The high availability cluster focuses on the failover, and the load-balancing cluster focuses on the distribution of the requests. The storage cluster focuses on the shared file system, and the high performance cluster focuses on the parallel computation . The high availability cluster is the one that the LFCE engineer configures for the service continuity, and its defining property is that the node failures are not visible from the clients outside the cluster .

Key point: The high availability is not the absence of the failure. It is the ability to recover from the failure without the service interruption that the client can perceive. The failover is the mechanism, the quorum is the safeguard against the split-brain, and the fencing is the guarantee that the failed node cannot interfere with the surviving one.

Why high availability requires the redundancy and the quorum

The single-point-of-failure problem.

A single server is the simplest deployment, and it is also the most fragile. If the server fails, the entire service fails, and there is no recovery until the hardware is repaired . The high availability eliminates the single point of failure by adding the redundant units: the second server, the dual power supply, the multiple network paths . The redundancy is the mechanism that turns the failure of the one unit into the takeover by the other, and the takeover is the failover.

The split-brain problem.

A cluster of two nodes can divide into two groups when the network between them fails. Each group believes that the other is dead, and each tries to take over the service. The result is the split-brain: two nodes serving the same data, which corrupts the data and the service state . The quorum is the mechanism that prevents this. The cluster partition must have the majority of the votes to be allowed to proceed, and the minority partition stops . The quorum distinguishes exactly one partition, and the fencing is the prerequisite that ensures the quorum is unique .

The fencing problem.

The quorum decides which partition is allowed to run. The fencing is the mechanism that guarantees the failed node is actually stopped, so that it cannot write to the shared data or the network after the surviving node has taken over. The STONITH (Shoot The Other Node In The Head) is the fencing implementation: the surviving node powers off or resets the failed node through the power switch or the management interface . Without the fencing, the failed node might still be running and writing to the shared storage, which defeats the quorum.

a. The failover and the failover domain

The failover is the process where the resource or the node fails on one machine, and the affected resources move to another node . The failover is the automatic response to the failure. The switchover is the planned version of the same operation: the resources move to the other node for the maintenance or the upgrade .

The failover domain is the named subset of the cluster nodes that are eligible to run the resource in the event of the failure . The domain can be restricted or unrestricted, and ordered or unordered. A restricted domain limits the nodes that can run the service, which reduces the configuration work to the subset of the nodes that will actually run it . An ordered domain specifies the preference order: the first node is the most preferred, the second is the next, and so on . The unrestricted and unordered domain is the default, and it allows the service to run on any available node .

b. The active-passive and the active-active

The active-passive configuration has the server that owns the cluster resources and the other servers that are on standby until the resource owner is unavailable . The advantage is the simplicity: there is no service degradation, and the service restarts only when the active server no longer responds . The disadvantage is the inefficiency: the passive server does not provide any service while it is on standby, and the failover takes time .

The active-active configuration has the two or more servers that are active at the same time. The servers can operate on the separate data, in which case each is the passive failover partner for the other, or on the same data, which requires the cluster file system and the distributed lock manager . The active-active on the separate data is the easier configuration, and the active-active on the same data is the difficult one, because the cluster file system is the complex low-level code .

c. The heartbeat and the cluster communication

The heartbeat is the mechanism that the cluster nodes use to detect the failure of the other nodes. The nodes exchange the periodic messages, and when the messages stop, the surviving node concludes that the other is down and initiates the failover . The Corosync is the cluster engine that provides the reliable messaging, the membership, and the quorum information . The Pacemaker is the resource manager that decides which resource runs on which node, and it uses the Corosync membership and quorum to make the decision .

The QDevice and the QNetd are the mechanisms that improve the quorum in the two-node cluster. The QNetd is the arbitrator that runs outside the cluster, and the QDevice runs on each node and communicates with the QNetd to provide the additional votes . The additional vote allows the cluster to sustain the node failures beyond the standard quorum rules, which is the mechanism that makes the two-node cluster viable without the third node .

Complete Example Session

# ============================================
# PART 1: THE CLUSTER STATUS
# ============================================
pcs status
# Output shows the nodes, the resources, and the quorum

# ============================================
# PART 2: THE QUORUM CHECK
# ============================================
pcs quorum status
# Output shows the quorum votes and the partition

# ============================================
# PART 3: THE FAILOVER DOMAIN
# ============================================
pcs constraint location show
# Output shows the location constraints

# ============================================
# PART 4: THE RESOURCE STATUS
# ============================================
pcs resource status
# Output shows which node runs which resource

# ============================================
# PART 5: THE MANUAL FAILOVER
# ============================================
pcs resource move <resource> <node>
# The resource moves to the specified node

# ============================================
# PART 6: THE FENCING STATUS
# ============================================
pcs stonith status
# Output shows the fencing devices and their status

# ============================================
# PART 7: THE CLUSTER LOG
# ============================================
journalctl -u pacemaker -n 20
# Output shows the recent cluster events

The seven parts demonstrate the cluster concepts: the status, the quorum, the failover domain, the resource status, the manual failover, the fencing, and the log. The critical check is Part 2: the quorum confirms that the cluster has the majority of the votes, which is the prerequisite for the resource management.

Quick Reference

The Cluster Types

Type	Focus
High availability	The failover and the service continuity
Load balancing	The request distribution
Storage	The shared file system
High performance	The parallel computation

The Key Concepts

Concept	Meaning
Failover	The automatic resource move on failure
Switchover	The planned resource move
Failover domain	The eligible nodes for the resource
Quorum	The majority vote to proceed
Split-brain	The partitioned cluster with the conflicting state
Fencing (STONITH)	The forced stop of the failed node
Heartbeat	The periodic node communication

The Active-Passive vs Active-Active

Aspect	Active-Passive	Active-Active
Resource owner	One node	Multiple nodes
Standby	The passive node	No standby
Efficiency	Low	High
Complexity	Low	High (cluster file system)

Best Practices

✅ Do This:

# Check the quorum before the maintenance
pcs quorum status                                              # ✅

# Use the fencing for the production clusters
pcs stonith status                                             # ✅

# Use the restricted failover domain for the complex services
pcs constraint location ...                                    # ✅

# Use the QDevice for the two-node cluster
# The QNetd provides the additional vote                          # ✅

# Test the failover regularly
pcs resource move <resource> <node>                            # ✅

❌ Don’t Do This:

# Run the two-node cluster without the QDevice
# The split-brain risk is high                                   # ❌

# Skip the fencing
# The failed node may still write to the shared data             # ❌

# Assume the active-passive is the high availability
# It is the failover, not the load balancing                     # ❌

# Ignore the quorum
# The minority partition must stop                               # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
The split-brain	The two-node cluster without the quorum	Use the QDevice or the third node
The data corruption	The fencing disabled	Enable the STONITH
The resource does not move	The failover domain restricted	Check the location constraint
The cluster does not start	The Corosync config	Check the `corosync.conf`

Real-World Examples

1. The cluster status

pcs status

2. The quorum status

pcs quorum status

3. The manual failover

pcs resource move <resource> <node>

4. The fencing status

pcs stonith status

5. The resource status

pcs resource status

6. The location constraint

pcs constraint location <resource> prefers <node>

7. The cluster log

journalctl -u pacemaker -n 20

8. The Corosync config

cat /etc/corosync/corosync.conf

9. The QDevice status

pcs quorum device status

10. The cluster destroy

pcs cluster destroy

Visual

The Quorum and the Split-Brain

┌──────────────────────────────────────────────────────────────┐
│  THE QUORUM PREVENTS THE SPLIT-BRAIN                         │
│                                                              │
│  The 3-node cluster:                                         │
│  ├── Node A: 1 vote                                          │
│  ├── Node B: 1 vote                                          │
│  └── Node C: 1 vote                                          │
│                                                              │
│  The partition {A, B}: 2 votes → the majority → proceeds     │
│  The partition {C}: 1 vote → the minority → stops            │
│                                                              │
│  The two-node cluster needs the QDevice for the tie-breaker. │
└──────────────────────────────────────────────────────────────┘

The Active-Passive vs Active-Active

┌──────────────────────────────────────────────────────────────┐
│  ACTIVE-PASSIVE IS SIMPLER; ACTIVE-ACTIVE IS EFFICIENT       │
│                                                              │
│  Active-passive:                                             │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ Node A: the active resource                            │  │
│  │ Node B: the standby, no service                        │  │
│  │ The failover: B takes over when A fails                │  │
│  └────────────────────────────────────────────────────────┘  │
│                                                              │
│  Active-active:                                              │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ Node A: the resource 1                                 │  │
│  │ Node B: the resource 2                                 │  │
│  │ The failover: B takes over both when A fails           │  │
│  └────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────┘

The Fencing

┌──────────────────────────────────────────────────────────────┐
│  THE FENCING STOPS THE FAILED NODE                           │
│                                                              │
│  The quorum decides the surviving partition.                 │
│       │                                                      │
│       ▼                                                      │
│  The STONITH powers off the failed node.                     │
│       │                                                      │
│       ▼                                                      │
│  The failed node cannot write to the shared data.            │
│       │                                                      │
│       ▼                                                      │
│  The surviving node takes over safely.                       │
└──────────────────────────────────────────────────────────────┘

Summary

Item	Value
High availability	The service continuity despite the failures
Cluster	Two or more nodes working together
Failover	The automatic resource move on failure
Switchover	The planned resource move
Failover domain	The eligible nodes for the resource
Quorum	The majority vote to proceed
Split-brain	The partitioned cluster with the conflicting state
Fencing (STONITH)	The forced stop of the failed node
Heartbeat	The periodic node communication
Active-passive	One active node, one standby
Active-active	Multiple active nodes

Key takeaways:

High availability is the elimination of the single points of failure. The redundancy is the mechanism, and the failover is the response to the failure. The node failures are not visible to the clients outside the cluster .
The quorum prevents the split-brain. The cluster partition must have the majority of the votes to proceed. The minority partition stops, which ensures that the shared data is not corrupted by the two partitions writing to it .
The fencing is the guarantee that the quorum is unique. The STONITH powers off the failed node, so it cannot interfere with the surviving node after the takeover .
The failover domain is the subset of the nodes that can run the resource. The restricted domain limits the eligible nodes, and the ordered domain specifies the preference order .
The active-passive is the simpler configuration, and the active-active is the more efficient one. The active-passive has the standby that does not serve while the active node is running. The active-active has the multiple active nodes, and the same-data variant requires the cluster file system .
The heartbeat and the Corosync provide the node communication. The Pacemaker uses the Corosync membership and quorum to decide which resource runs on which node .

Remember: High availability is the design goal, not the product. The cluster is the foundation, the failover is the response, the quorum is the safeguard, and the fencing is the guarantee. The active-passive is the simpler configuration, and the active-active is the more efficient one. The result is the service that survives the failure of the individual component, which is the difference between the single server that fails and the cluster that continues.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!