LFCE 109 🐧 Pacemaker — Concepts - KandZ

⏱️ Estimated read: 11 min | |

Pacemaker is the high-availability cluster resource manager that runs on a set of nodes to preserve the integrity and minimize the downtime of the services those nodes provide . It is the “brain” of the cluster stack: it processes the events—nodes joining or leaving, resources failing, administrators changing the configuration—and computes the ideal state of the cluster and the path to achieve it . The previous chapter covered VRRP, which handles the virtual IP failover; this chapter covers the resource manager that handles the services, the storage, and the complex dependencies between them.

Quick How-To Guides & Fast FixesNeed a quick fix or step-by-step solution? Browse our fast answers on HowTo.KandZ.me:

Operating Systems CLI →Dev Environment Languages →Web Network Tools →Security Accounts →Security Access Tech →Device Phone Basics →Social Messaging →Digital Files Media →

Pacemaker does not work alone. It depends on a cluster membership layer, and the current standard is Corosync, which provides the reliable messaging, the membership information, and the quorum . The combination of Pacemaker and Corosync is what the Linux community usually calls the “cluster stack” . Pacemaker also depends on the resource agents—the scripts that actually start, stop, and monitor the services—and on the fence agents, which isolate the failed nodes .

Key point: Pacemaker manages resources, not just services. A resource can be an IP address, a filesystem, a database, or any scriptable component. The resource agents provide the uniform interface that lets Pacemaker manage services it knows nothing about . The fencing—also known as STONITH—is not optional for a production cluster: a cluster cannot safely recover from certain failure conditions, such as an unresponsive node, without it .

Why Pacemaker requires the cluster stack

The membership problem.

Pacemaker needs to know which nodes are alive and which are dead. That information comes from Corosync, which provides the membership and the quorum . If the membership layer is unreliable, Pacemaker may believe a node is dead when it is actually alive, and it may start the resources on the surviving node while the “dead” node is still running them. This is the split-brain, and the quorum is the safeguard against it . Pacemaker relies on the Corosync quorum to decide whether the cluster partition is allowed to proceed.

The resource-abstraction problem.

Pacemaker does not have the built-in knowledge of how to start a PostgreSQL database, mount an NFS share, or configure a virtual IP. It delegates those operations to the resource agents, which are scripts that implement the start, stop, and monitor actions for the specific resource type . The OCF (Open Cluster Framework) specification defines the standard interface, and the agents are shipped in the resource-agents package . This abstraction is what makes Pacemaker general-purpose: anything that can be scripted can be clustered.

The fencing problem.

When a node stops responding, Pacemaker cannot safely start its resources on another node until the unresponsive node is confirmed to be stopped. The fencing is the mechanism that guarantees this: it powers off the node or disables its access to the shared storage, so the resources cannot be running in two places at once . The fence agents implement the specific fencing methods—IPMI, power switches, cloud APIs, or the SBD (STONITH Block Device) . Without the fencing, the cluster cannot recover safely from the node failure.

a. The resources and the resource agents

A resource is the specific functionality that the cluster keeps available, such as an IP address, a filesystem, or a database . The resource agent is the script that translates the Pacemaker’s commands into the service-specific actions.

# The resource types
# ocf:heartbeat:IPaddr2   → a virtual IP address
# ocf:heartbeat:Filesystem → a mounted filesystem
# ocf:heartbeat:pgsql      → a PostgreSQL database
# systemd:nginx            → a systemd service

The resource agents follow the OCF specification, and they implement the start, stop, and monitor operations . The monitor operation is the health check that Pacemaker runs periodically to verify that the resource is still running. If the monitor fails, Pacemaker considers the resource failed and moves it to another node .

Pacemaker also supports the advanced resource types: the clones (resources that run on multiple nodes) and the promotable clones (clones that can run in one of two roles, such as a primary and a secondary database) . The clone is the mechanism for the active-active configuration, and the promotable clone is the mechanism for the primary/secondary replication.

b. The constraints and the resource relationships

Pacemaker can model the complex relationships between the resources. The constraints define the placement and the order of the operations .

Constraint	Purpose
Location	The preferred node for the resource
Colocation	The resources that must run on the same node
Ordering	The start and stop order of the resources

The location constraint tells Pacemaker which node should run the resource, and it can be used to restrict the resource to a specific node or to prefer one node over the others . The colocation constraint tells Pacemaker that two resources must run together—for example, the virtual IP and the web server that serves it . The ordering constraint tells Pacemaker that one resource must start before another—for example, the filesystem must be mounted before the database starts . The constraints are the mechanism that turns a set of independent resources into a coherent service.

c. The fencing and the STONITH

The fencing, also known as STONITH (Shoot The Other Node In The Head), is the mechanism that ensures the failed node is actually stopped before its resources are taken over . The fence devices are represented as a special class of resource in Pacemaker, and they are configured in the same CIB as the other resources .

# The SBD configuration
# /etc/default/sbd
SBD_DEVICE="/dev/sda1;/dev/sdb1;/dev/sdc1"
SBD_PACEMAKER="true"

The SBD is the STONITH Block Device, which uses the shared storage to exchange the fencing messages . The recommended configuration is three SBD devices, because the SBD will only self-fence if more than one device is lost, which makes the cluster resilient against the temporary single device outages . The SBD daemon starts before the Corosync and the Pacemaker, and it is terminated only after all the other cluster components have been shut down, which ensures that the resources are never activated without the SBD supervision .

The sbd daemon uses the watchdog to ensure that the node self-fences if the SBD process itself fails . The watchdog is a hardware timer that reboots the node if it is not “tickled” regularly. The SBD tickles the watchdog while it is healthy, and if the SBD stops tickling, the watchdog fires and reboots the node. This is the mechanism that guarantees the node cannot continue running after the cluster has decided it is dead.

Complete Example Session

# ============================================
# PART 1: THE CLUSTER STATUS
# ============================================
pcs status
# Output shows the nodes, the resources, and the quorum

# ============================================
# PART 2: THE RESOURCE AGENTS
# ============================================
pcs resource agents ocf:heartbeat
# Output lists the available OCF resource agents

# ============================================
# PART 3: THE RESOURCE CREATION
# ============================================
pcs resource create WebIP ocf:heartbeat:IPaddr2 ip=192.168.1.100 cidr_netmask=24
# The virtual IP is created as a cluster resource

# ============================================
# PART 4: THE COLOCATION CONSTRAINT
# ============================================
pcs constraint colocation add WebIP with WebServer INFINITY
# The IP and the web server must run on the same node

# ============================================
# PART 5: THE ORDERING CONSTRAINT
# ============================================
pcs constraint order start WebIP then WebServer
# The IP must start before the web server

# ============================================
# PART 6: THE LOCATION CONSTRAINT
# ============================================
pcs constraint location WebIP prefers node1
# The IP prefers node1

# ============================================
# PART 7: THE FENCING STATUS
# ============================================
pcs stonith status
# Output shows the fence devices and their status

The seven parts demonstrate the Pacemaker concepts: the status, the resource agents, the resource creation, the colocation, the ordering, the location, and the fencing. The critical distinction is between the constraints: the colocation says “run together,” the ordering says “start in this sequence,” and the location says “prefer this node.”

Quick Reference

The Cluster Stack Components

Component	Role
Pacemaker	The resource manager (the brain)
Corosync	The membership, the messaging, and the quorum
Resource agents	The scripts that start, stop, and monitor the resources
Fence agents	The scripts that isolate the failed nodes
libQB	The logging and the IPC library

The Resource Types

Type	Purpose
`ocf:heartbeat:IPaddr2`	The virtual IP
`ocf:heartbeat:Filesystem`	The mounted filesystem
`ocf:heartbeat:pgsql`	The PostgreSQL database
`systemd:nginx`	The systemd service
Clone	The resource on multiple nodes
Promotable clone	The resource with the primary/secondary roles

The Constraints

Constraint	Purpose
Location	The preferred node
Colocation	The resources on the same node
Ordering	The start/stop sequence

Best Practices

✅ Do This:

# Use the colocation for the dependent resources
pcs constraint colocation add WebIP with WebServer INFINITY # ✅

# Use the ordering for the startup sequence
pcs constraint order start WebIP then WebServer             # ✅

# Use the fencing for the production clusters
pcs stonith status                                          # ✅

# Use the three SBD devices for the resilience
SBD_DEVICE="/dev/sda1;/dev/sdb1;/dev/sdc1"                  # ✅

❌ Don’t Do This:

# Skip the fencing
# The cluster cannot safely recover from the node failure     # ❌

# Use the two SBD devices and expect the fencing to work
# The SBD cannot fence if one of the two is unavailable       # ❌

# Ignore the resource monitor
# The failed resource is not detected                         # ❌

# Use the location constraint as the only failover mechanism
# The resource does not move on the node failure              # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
The resource does not move	The location constraint restricts it	Check the `pcs constraint location`
The resource starts on the wrong node	The colocation conflict	Check the `pcs constraint colocation`
The cluster does not recover	The fencing is not configured	Configure the STONITH
The SBD does not start	The device is not accessible	Check the `SBD_DEVICE`

Real-World Examples

1. The cluster status

pcs status

2. The resource agents

pcs resource agents ocf:heartbeat

3. The resource creation

pcs resource create WebIP ocf:heartbeat:IPaddr2 ip=192.168.1.100

4. The colocation

pcs constraint colocation add WebIP with WebServer INFINITY

5. The ordering

pcs constraint order start WebIP then WebServer

6. The location

pcs constraint location WebIP prefers node1

7. The fencing status

pcs stonith status

8. The SBD configuration

SBD_DEVICE="/dev/sda1;/dev/sdb1;/dev/sdc1"

9. The clone resource

pcs resource create WebClone systemd:nginx clone

10. The promotable clone

pcs resource create DB ocf:heartbeat:pgsql promotable

Visual

The Cluster Stack

┌──────────────────────────────────────────────────────────────┐
│  PACEMAKER IS THE BRAIN; COROSYNC IS THE NERVOUS SYSTEM      │
│                                                              │
│  Pacemaker (the resource manager)                            │
│  ├── decides which resource runs on which node               │
│  ├── starts and stops the resources                          │
│  └── fences the failed nodes                                 │
│                                                              │
│  Corosync (the membership layer)                             │
│  ├── the reliable messaging                                  │
│  ├── the membership (which nodes are alive)                  │
│  └── the quorum (the majority vote)                          │
│                                                              │
│  Resource agents → the start/stop/monitor scripts            │
│  Fence agents    → the node isolation scripts               │
└──────────────────────────────────────────────────────────────┘

The Constraints

┌──────────────────────────────────────────────────────────────┐
│  THE CONSTRAINTS MODEL THE RESOURCE RELATIONSHIPS            │
│                                                              │
│  Location:                                                   │
│  └── "Run the resource on this node"                         │
│                                                              │
│  Colocation:                                                 │
│  └── "Run these resources on the same node"                  │
│                                                              │
│  Ordering:                                                   │
│  └── "Start this resource before that one"                   │
│                                                              │
│  The constraints turn the independent resources into the     │
│  coherent service.                                           │
└──────────────────────────────────────────────────────────────┘

The Fencing

┌──────────────────────────────────────────────────────────────┐
│  THE FENCING GUARANTEES THE FAILED NODE IS STOPPED           │
│                                                              │
│  A node stops responding                                     │
│       │                                                      │
│       ▼                                                      │
│  Pacemaker cannot start the resources on another node        │
│  until the failed node is confirmed to be stopped            │
│       │                                                      │
│       ▼                                                      │
│  The fence agent powers off the node (IPMI, power switch)    │
│       │                                                      │
│       ▼                                                      │
│  The resources are started on the surviving node             │
└──────────────────────────────────────────────────────────────┘

Summary

Item	Value
Pacemaker	The high-availability cluster resource manager
Corosync	The membership, the messaging, and the quorum
Resource	The cluster-managed functionality (IP, filesystem, database)
Resource agent	The script that starts, stops, and monitors the resource
Fence agent	The script that isolates the failed node
Constraint types	Location, colocation, ordering
Fencing (STONITH)	The forced stop of the failed node
SBD	The STONITH Block Device on the shared storage
Clone	The resource on multiple nodes
Promotable clone	The resource with the primary/secondary roles

Key takeaways:

Pacemaker is the resource manager, and Corosync is the membership layer. Pacemaker decides which resource runs on which node, and Corosync provides the membership, the messaging, and the quorum that Pacemaker relies on .
The resource agents abstract the services. Pacemaker does not know how to start the specific service; the resource agent does. The OCF specification defines the interface, and anything that can be scripted can be clustered .
The constraints model the relationships. The location constraint prefers a node, the colocation constraint keeps the resources together, and the ordering constraint defines the start/stop sequence .
The fencing is mandatory for the production clusters. The STONITH ensures the failed node is stopped before its resources are taken over. The cluster cannot safely recover from the node failure without it .
The SBD is the recommended fencing method. It uses the shared storage to exchange the fencing messages, and the three-device configuration is resilient against the temporary single device outages .
The clones and the promotable clones support the advanced configurations. The clones run on the multiple nodes, and the promotable clones have the primary and secondary roles, which is the mechanism for the active-active and the primary/secondary replication .

Remember: Pacemaker is the brain of the Linux high-availability cluster. It manages the resources, enforces the constraints, and fences the failed nodes. It depends on Corosync for the membership and the quorum, on the resource agents for the service control, and on the fence agents for the node isolation. The result is the cluster that keeps the services available across the node failures, which is the difference between the single server that fails and the cluster that continues.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!