LFCE 110 🐧 Corosync — Cluster Communication - KandZ

⏱️ Estimated read: 11 min | |

Corosync is the cluster engine that provides the reliable messaging, the membership, and the quorum that Pacemaker relies on. It implements the Totem Single-Ring Protocol, a token-passing protocol that guarantees the ordered delivery of messages to every node in the cluster . The previous chapter covered Pacemaker as the resource manager; this chapter covers the communication layer beneath it. Without Corosync, Pacemaker has no way to know which nodes are alive, no way to agree on the cluster state, and no way to detect the split-brain.

Quick How-To Guides & Fast FixesNeed a quick fix or step-by-step solution? Browse our fast answers on HowTo.KandZ.me:

Operating Systems CLI →Dev Environment Languages →Web Network Tools →Security Accounts →Security Access Tech →Device Phone Basics →Social Messaging →Digital Files Media →

The engine is not a single protocol; it is a stack of services. The CPG (Closed Process Group) provides the group messaging that the cluster applications use. The VoteQuorum service provides the quorum calculation that determines whether the cluster is allowed to operate. The KNET transport provides the encrypted, multi-link communication that carries the messages between the nodes . Each service is configured in the corosync.conf file, and the combination of the services is what makes the cluster work.

Key point: Corosync uses a token-passing protocol, not a broadcast protocol. A token circulates around the ring, and only the node holding the token may transmit. This is the mechanism that guarantees the total order of the messages: every node sees the messages in the same sequence . The token is the single point of serialization, and the ring is the topology that carries it.

Why Corosync requires the token and the quorum

The ordering problem.

A cluster application must see the events in the same order on every node. If node A sees the “resource started” event before the “resource stopped” event, and node B sees the opposite, the two nodes disagree about the cluster state, and the resource management becomes unpredictable. Corosync solves this with the token. Only one node can transmit at a time, so the messages are serialized, and the sequence number on the token ensures that every node delivers the messages in the same order . The total order is the foundation of the virtual synchrony property: the membership changes and the messages are delivered in a consistent order relative to each other .

The membership problem.

Corosync must know which nodes are alive and which are dead. The membership is not a static configuration; it changes when a node joins or leaves, or when the network partitions. The membership changes are delivered as the configuration events, with a unique view identifier and the list of the processors in the new configuration . The view identifier is the mechanism that distinguishes the different cluster states. A node that misses a view change is no longer a member of the current configuration, and it cannot participate in the quorum.

The quorum problem.

A cluster partition can divide the nodes into two groups, each believing the other is dead. If both groups proceed, the split-brain corrupts the state. The VoteQuorum service solves this by assigning votes to the nodes and requiring the majority to proceed . The expected_votes is the total number of the votes in the cluster, and the quorum is the majority (50% + 1). A partition with the majority is quorate and can operate; the minority partition is inquorate and must stop. The two_node option is the special case: it sets the quorum to 1 artificially, so the two-node cluster can operate when only one node is alive, at the cost of the split-brain risk .

a. The Totem single-ring protocol and the token

The Totem protocol is the core of the Corosync communication. It uses a logical ring topology, and a token circulates around the ring. The node holding the token is the only one allowed to transmit.

# The token rotation
Node A → Node B → Node C → Node A → ...

# The node with the token:
# 1. Empties its input buffer
# 2. Transmits the pending messages
# 3. Forwards the token to the next node

The token is the mechanism that enforces the total order. The messages are delivered in the sequence they were transmitted, and every node sees the same sequence . The flow control is built into the protocol: the node must empty its input buffer before it transmits, which prevents the buffer overflow when the messages arrive faster than they can be processed .

The ring is a logical topology, not a physical one. The nodes are connected through the Ethernet, and the ring is the sequence in which the token passes from one node to the next . The failure of a node is detected when the token does not return within the token timeout. The membership algorithm then forms the new ring without the failed node.

b. The CPG and the virtual synchrony

The CPG (Closed Process Group) is the API that the applications use to exchange the messages. It implements the virtual synchrony, which guarantees that the messages and the membership changes are delivered in the consistent order .

// The CPG initialization
cpg_callbacks_t callbacks = {
    .cpg_deliver_fn = my_deliver_fn,
    .cpg_confchg_fn = my_confchg_fn,
};
cpg_initialize(&handle, &callbacks);

The cpg_deliver_fn is called when the message is delivered, and the cpg_confchg_fn is called when the membership changes . The virtual synchrony ensures that the message is delivered to every node in the current configuration before the configuration changes. This is the mechanism that prevents the message loss during the membership transitions.

The cpg_join and the cpg_leave functions add and remove the process from the group . The group is the logical channel, and the nodes in the group receive the messages sent to the group. The membership of the group is the subset of the cluster nodes that participate in the specific application.

c. The VoteQuorum and the configuration

The VoteQuorum service calculates the quorum and provides the membership information to the applications. It is configured in the quorum block of the corosync.conf.

# /etc/corosync/corosync.conf
quorum {
    provider: corosync_votequorum
    expected_votes: 3
    two_node: 0
}

The expected_votes: 3 sets the total number of the votes, and the quorum is the majority (2) . The two_node: 0 disables the two-node mode. If the cluster has the two nodes and the two_node: 1, the quorum is set to 1, and the cluster can operate with one node alive .

The wait_for_all: 1 is the feature that delays the first quorate state until all nodes have been visible at least once. It avoids the startup race condition where the subset of the nodes becomes quorate and fences the nodes that have not yet started . The last_man_standing: 1 is the feature that dynamically recalculates the expected votes when the nodes leave, which allows the cluster to survive more failures than the static quorum calculation would allow.

Complete Example Session

# ============================================
# PART 1: THE COROSYNC CONFIGURATION
# ============================================
# /etc/corosync/corosync.conf
totem {
    version: 2
    cluster_name: mycluster
    crypto_cipher: aes256
    crypto_hash: sha256
    interface {
        ringnumber: 0
        bindnetaddr: 192.168.1.0
        mcastaddr: 239.255.1.1
        mcastport: 5405
    }
}
quorum {
    provider: corosync_votequorum
    expected_votes: 3
}
nodelist {
    node {
        ring0_addr: 192.168.1.1
    }
    node {
        ring0_addr: 192.168.1.2
    }
    node {
        ring0_addr: 192.168.1.3
    }
}

The crypto_cipher: aes256 and the crypto_hash: sha256 enable the encryption and the authentication of the messages . The interface block defines the ring 0 with the bind address and the multicast address. The nodelist block enumerates the nodes, and the VoteQuorum calculates the expected votes from the list .

# ============================================
# PART 2: THE AUTHKEY GENERATION
# ============================================
corosync-keygen
# The key is written to /etc/corosync/authkey

# ============================================
# PART 3: THE KEY DISTRIBUTION
# ============================================
# Copy the authkey to every node
scp /etc/corosync/authkey node2:/etc/corosync/authkey
scp /etc/corosync/authkey node3:/etc/corosync/authkey

The corosync-keygen generates the private key that authenticates the messages . The key must be identical on every node, or the nodes cannot join the same configuration. The Invalid digest error is the symptom of the mismatched keys .

# ============================================
# PART 4: THE SERVICE START
# ============================================
systemctl enable --now corosync
systemctl status corosync

# ============================================
# PART 5: THE QUORUM STATUS
# ============================================
corosync-quorumtool
# Output shows the quorum status and the votes

# ============================================
# PART 6: THE MEMBERSHIP CHECK
# ============================================
corosync-cmapctl | grep members
# Output shows the current members and their status

The corosync-quorumtool shows the quorum status, and the corosync-cmapctl shows the membership information. The corosync-cfgtool -s shows the ring status and the link status.

# ============================================
# PART 7: THE RING STATUS
# ============================================
corosync-cfgtool -s
# Output shows the ring 0 status

The corosync-cfgtool -s confirms that the ring is operational and the nodes are communicating . The output shows the link status and the node IDs.

Quick Reference

The Corosync Configuration Blocks

Block	Purpose
`totem`	The ring protocol, the token, the crypto
`quorum`	The VoteQuorum settings
`nodelist`	The node addresses
`logging`	The log destination and the level

The Key Parameters

Parameter	Purpose
`crypto_cipher`	The encryption (aes256, aes192, aes128, none)
`crypto_hash`	The authentication (sha256, sha1, md5, none)
`expected_votes`	The total votes for the quorum calculation
`two_node`	The two-node mode (quorum = 1)
`wait_for_all`	The WFA feature
`last_man_standing`	The LMS feature

The Commands

Command	Purpose
`corosync-keygen`	Generate the authkey
`corosync-quorumtool`	Show the quorum status
`corosync-cmapctl`	Show the membership and the runtime values
`corosync-cfgtool -s`	Show the ring status

The Transport

Transport	Features
KNET	Encryption, multiple links, IPv4/IPv6
UDPU	Single link, no crypto
UDP	Single link, multicast

Best Practices

✅ Do This:

# Generate the authkey and distribute it
corosync-keygen                                              # ✅

# Enable the crypto for the security
crypto_cipher: aes256
crypto_hash: sha256                                          # ✅

# Use the nodelist for the automatic vote calculation
nodelist { node { ring0_addr: ... } }                        # ✅

# Check the quorum after the configuration
corosync-quorumtool                                          # ✅

# Use the KNET transport for the multi-link support
transport: knet                                              # ✅

❌ Don’t Do This:

# Use the different authkeys on the nodes
# The nodes cannot join the same configuration                 # ❌

# Disable the quorum in the multi-node cluster
# The split-brain risk is high                                 # ❌

# Use the two_node in the cluster with more than two nodes
# The option is auto-disabled, but the intent is wrong        # ❌

# Ignore the ring status
# The communication failure is invisible                       # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
The nodes cannot join	The authkey mismatch	Copy the same authkey
The quorum is lost	The expected_votes is wrong	Use the nodelist or set the expected_votes
The split-brain	The quorum is not enforced	Enable the VoteQuorum
The ring is down	The network or the bind address	Check the `corosync-cfgtool -s`
The encryption fails	The crypto options mismatch	Use the same crypto on all nodes

Real-World Examples

1. The Totem configuration

totem {
    version: 2
    cluster_name: mycluster
    crypto_cipher: aes256
    crypto_hash: sha256
}

2. The Quorum configuration

quorum {
    provider: corosync_votequorum
    expected_votes: 3
}

3. The Nodelist

nodelist {
    node { ring0_addr: 192.168.1.1 }
    node { ring0_addr: 192.168.1.2 }
    node { ring0_addr: 192.168.1.3 }
}

4. The authkey generation

corosync-keygen

5. The quorum check

corosync-quorumtool

6. The ring status

corosync-cfgtool -s

7. The two-node mode

quorum {
    two_node: 1
}

8. The wait for all

quorum {
    wait_for_all: 1
}

9. The last man standing

quorum {
    last_man_standing: 1
}

10. The KNET transport

totem {
    transport: knet
}

Visual

The Token Ring

┌──────────────────────────────────────────────────────────────┐
│  THE TOKEN CIRCULATES AROUND THE LOGICAL RING                │
│                                                              │
│  Node A ──▶ Node B ──▶ Node C ──▶ Node A                     │
│                                                              │
│  The token holder is the only node that can transmit.        │
│  The messages are serialized by the token, so every node     │
│  delivers them in the same order.                            │
└──────────────────────────────────────────────────────────────┘

The Quorum

┌──────────────────────────────────────────────────────────────┐
│  THE MAJORITY IS REQUIRED TO PROCEED                         │
│                                                              │
│  3-node cluster, expected_votes: 3, quorum: 2                │
│                                                              │
│  Partition {A, B}: 2 votes → quorate → proceeds              │
│  Partition {C}: 1 vote → inquorate → stops                   │
│                                                              │
│  The minority partition cannot write to the shared state.    │
└──────────────────────────────────────────────────────────────┘

The Configuration Flow

┌──────────────────────────────────────────────────────────────┐
│  THE CONFIGURATION IS SHARED ACROSS THE NODES                │
│                                                              │
│  /etc/corosync/corosync.conf                                 │
│  ├── totem { ... }                                           │
│  ├── quorum { ... }                                          │
│  └── nodelist { ... }                                        │
│                                                              │
│  /etc/corosync/authkey (the same on every node)              │
│                                                              │
│  The corosync service reads the configuration and the key,   │
│  and joins the ring.                                         │
└──────────────────────────────────────────────────────────────┘

Summary

Item	Value
Corosync	The cluster engine (the messaging, the membership, the quorum)
Totem	The single-ring token-passing protocol
CPG	The closed process group messaging API
VoteQuorum	The quorum calculation service
Token	The serialization mechanism for the total order
Quorum	The majority vote required to proceed
`expected_votes`	The total votes in the cluster
`two_node`	The two-node mode (quorum = 1)
`wait_for_all`	The first quorate state requires all nodes
`last_man_standing`	The dynamic vote recalculation
authkey	The shared secret for the message authentication
KNET	The transport with the encryption and the multiple links

Key takeaways:

Corosync is the communication layer beneath Pacemaker. It provides the membership, the messaging, and the quorum that Pacemaker uses to manage the resources .
The Totem protocol uses a token-passing ring. The token serializes the messages, and the total order is the foundation of the virtual synchrony .
The VoteQuorum requires the majority to proceed. The expected_votes sets the total votes, and the quorum is the majority. The two_node option is the special case for the two-node cluster .
The authkey must be identical on every node. The corosync-keygen generates the key, and the mismatched keys prevent the nodes from joining the same configuration .
The KNET transport supports the encryption and the multiple links. The crypto_cipher and the crypto_hash enable the security, and the multiple interface blocks provide the redundancy .
The wait_for_all and the last_man_standing are the advanced quorum features. The wait_for_all avoids the startup race, and the last_man_standing allows the cluster to survive more failures by dynamically recalculating the expected votes .

Remember: Corosync is the nervous system of the cluster. It carries the messages, maintains the membership, and calculates the quorum. The Totem protocol is the token-passing ring that guarantees the order, and the VoteQuorum is the majority vote that prevents the split-brain. The authkey is the shared secret, and the KNET transport is the encrypted communication. The result is the cluster that agrees on its state, which is the prerequisite for the resource management that Pacemaker provides.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!