Docker 67 🐳 High Availability Homelab / Server Deployment with Docker Swarm and Portainer - KandZ

⏱️ Estimated read: 15 min | |

A homelab or small server deployment that runs on a single machine is not highly available. When that machine reboots, every service goes down. When a disk fails, every container is lost. The purpose of building a Swarm cluster for a homelab is not to match the scale of a cloud provider—it is to eliminate the single point of failure that a single Docker host represents, while keeping the operational complexity low enough that one person can maintain it. Docker Swarm is the right tool for this because it is built into Docker Engine, requires no additional control plane software, and its Raft consensus model is simple enough to reason about without a dedicated orchestration team.

Portainer is the management layer that makes the cluster practical for a homelab. Without it, every service deployment, stack update, and node inspection is a terminal command. With it, you get a web interface that shows the state of every node, every stack, and every container across the cluster. But Portainer introduces its own availability question: if the node running the Portainer server goes down, you lose the management interface even though your services keep running. The architecture of a high-availability homelab deployment must account for this, and the honest answer is that Portainer itself is not designed to run as a multi-instance, highly available service inside Swarm. The correct pattern is to pin Portainer to a specific manager node, back up its data, and accept that management access is temporarily lost if that node fails—while the workloads continue to run.

Key point: A high-availability homelab Swarm requires an odd number of manager nodes (three is the practical minimum) for Raft quorum; Portainer should be pinned to one manager with its data volume backed up, because Portainer does not support multiple server instances managing the same cluster, and losing the Portainer node does not stop your services—it only stops the management UI.

Why the manager count and Portainer placement determine availability

The Raft quorum problem. Swarm managers use the Raft consensus algorithm to maintain cluster state. Every change—adding a node, updating a service, creating a secret—must be acknowledged by a majority of managers before it is committed . A three-manager cluster has a quorum of two, which means it can tolerate one manager failure. A five-manager cluster has a quorum of three, tolerating two failures. The formula is quorum = ⌊n/2⌋ + 1, and fault tolerance = ⌊(n-1)/2⌋ . For a homelab, three managers is the correct starting point: it provides real fault tolerance without the coordination overhead of five.

The even-number trap. Running four managers provides no additional fault tolerance compared to three. Both configurations tolerate exactly one failure, because a four-manager cluster has a quorum of three, which means losing two managers leaves only two—below quorum . The additional manager adds coordination overhead without adding resilience. Always use an odd number.

The Portainer single-instance constraint. Portainer’s architecture consists of a server and agents. The agents are stateless and can run as a global service on every node. The server stores configuration, user accounts, and stack metadata in its /data volume, and Portainer explicitly does not support running multiple server instances managing the same cluster . If you run two Portainer servers against the same Swarm, they will conflict over state. This means Portainer is a single point of failure for management access, even though it is not a single point of failure for the workloads themselves.

The Portainer data persistence problem. If Portainer is pinned to a manager node and that node fails, Swarm will not automatically reschedule Portainer to another node with the same data, because the data volume is local to the failed node. The Portainer documentation recommends either running the server outside the Swarm on a standalone host and connecting via an agent, or ensuring that another node can access the Portainer data through a network share . For a homelab, the network share approach is practical if you already have NFS or GlusterFS, but it introduces the database-over-network-share risk that Portainer itself warns about. The simpler approach is to pin Portainer to a specific manager, back up the data volume regularly, and accept that management access requires bringing that node back online.

The workload continuity distinction. When a manager node fails in a three-manager cluster, the remaining two managers maintain quorum, and services continue running on all worker nodes . The failure only blocks administrative operations: you cannot add nodes, update services, or change cluster configuration until quorum is restored. Existing tasks keep running. This is the crucial distinction for a homelab: the services you actually use—your media server, your file sync, your home automation—do not stop when a manager fails. Only your ability to change them stops.

a. Designing the manager and worker layout

A homelab cluster should separate the control plane from the data plane. Managers run the Raft consensus and schedule services; workers run the containers. By default, manager nodes are also workers, which means the scheduler can place service tasks on them. For a small homelab, this is acceptable, but for anything that needs to stay responsive, draining the managers so they only handle control-plane duties is the cleaner design.

A three-node homelab with three managers and no dedicated workers is the minimum viable HA setup. It tolerates one node failure entirely: the remaining two nodes maintain quorum and can still run services. The downside is that each manager is also running workloads, so a failed node takes both a manager and its containers with it. Swarm reschedules the containers to the remaining nodes, but the cluster is now running with reduced capacity.

A more robust layout for a homelab with more hardware is three managers and two or more workers. The managers are drained of workloads and handle only cluster state. The workers run everything. If a worker fails, Swarm reschedules its tasks to the other workers. If a manager fails, quorum is maintained and no workloads are affected.

The placement of managers across physical failure domains matters. If all three managers are on the same physical host (three VMs on one machine), losing that host loses the entire cluster. The managers should be spread across separate physical machines if the goal is to survive hardware failure . For a homelab with limited hardware, this may mean running managers on three separate mini-PCs, or accepting that a single host failure takes down the cluster.

b. Deploying Portainer with the correct persistence and placement

Portainer is deployed as a stack that includes two services: the Portainer server (replicated, one instance) and the Portainer agent (global, one instance per node). The official manifest is retrieved from Portainer’s download server and deployed with docker stack deploy .

curl -L https://downloads.portainer.io/ce-sts/portainer-agent-stack.yml -o portainer-agent-stack.yml
docker stack deploy -c portainer-agent-stack.yml portainer

The default manifest uses a Docker-managed volume for Portainer’s data. For a homelab, a bind mount to a known host directory is easier to back up and inspect. The volume definition should be changed to a bind mount pointing to a directory that exists on the manager where Portainer will run:

volumes:
  portainer_data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/portainer_data

The /mnt/portainer_data directory must exist on the manager node before deployment. If Portainer is rescheduled to a different node, that node must also have the same directory with the same data, which is why a shared filesystem (NFS, GlusterFS) is necessary for automatic failover. Without shared storage, Portainer must be pinned to a specific manager using a placement constraint:

deploy:
  placement:
    constraints:
      - node.hostname == manager-1

This constraint ensures Portainer only runs on manager-1. If that node fails, Portainer is not rescheduled, and management access is lost until the node returns. The workloads on the other nodes continue running .

The Portainer agent runs as a global service on every node and communicates with the server over port 9001 . The agent is stateless, so it does not need persistent storage. If a node fails, its agent disappears, and when the node returns, the agent is redeployed automatically.

c. Verifying the deployment and understanding the failure modes

After deployment, verify that the manager quorum is healthy and that Portainer is running on the expected node.

docker node ls

The output shows each node’s manager status. One manager is Leader, the others are Reachable. If a manager shows Unreachable, the cluster has lost a manager but may still have quorum. If two of three managers show Unreachable, quorum is lost, and administrative operations will fail until a manager is restored .

docker service ps portainer_portainer

This shows which node is running the Portainer server. The placement constraint should ensure it is the expected manager. If the service is Pending, the constraint may not match any available node, or the bind mount directory may be missing.

The failure modes are distinct and worth understanding:

Worker failure: Swarm reschedules the worker’s tasks to other workers. No quorum impact. Services continue after a brief interruption.

Manager failure (one of three): Quorum is maintained by the remaining two managers. Services continue running on all workers. Administrative operations continue. The failed manager’s tasks, if it was also a worker, are rescheduled. When the manager returns, it rejoins automatically if its Docker state is intact .

Manager failure (two of three): Quorum is lost. No administrative operations are possible. Existing services continue running, but no new tasks can be scheduled, and no services can be updated. Recovery requires restoring at least one failed manager, or forcing a new cluster with docker swarm init --force-new-cluster on the surviving manager .

Portainer node failure: If Portainer is pinned to the failed node, the management UI is unavailable. The Swarm cluster and all services continue running. When the node returns, Portainer starts automatically. If the node cannot be recovered, Portainer must be redeployed on another manager with the backed-up data.

Complete Example Session

The following session builds a three-manager homelab Swarm, deploys Portainer with a pinned placement and bind-mounted data, and verifies the failure behavior.

# ============================================
# PART 1: INITIALIZE THE FIRST MANAGER
# ============================================
# On the first node (manager-1)
docker swarm init --advertise-addr 192.168.1.10

# Output includes the worker join token
# Get the manager join token
docker swarm join-token manager

# ============================================
# PART 2: JOIN THE OTHER MANAGERS
# ============================================
# On manager-2 and manager-3, run the manager join command
docker swarm join --token SWMTKN-1-... 192.168.1.10:2377

# ============================================
# PART 3: VERIFY QUORUM
# ============================================
docker node ls
# Expected: three managers, one Leader, two Reachable

# ============================================
# PART 4: CREATE THE PORTAINER DATA DIRECTORY
# ============================================
# On manager-1 only (where Portainer will run)
sudo mkdir -p /mnt/portainer_data
sudo chmod 700 /mnt/portainer_data

# ============================================
# PART 5: PREPARE THE PORTAINER STACK
# ============================================
curl -L https://downloads.portainer.io/ce-sts/portainer-agent-stack.yml -o portainer-agent-stack.yml

# Edit the stack to add the placement constraint and bind mount
# The portainer service should include:
#   volumes:
#     - /mnt/portainer_data:/data
#   deploy:
#     placement:
#       constraints:
#         - node.hostname == manager-1

# ============================================
# PART 6: DEPLOY THE STACK
# ============================================
docker stack deploy -c portainer-agent-stack.yml portainer

# ============================================
# PART 7: VERIFY THE DEPLOYMENT
# ============================================
docker service ls
# portainer_agent      global      3/3
# portainer_portainer  replicated  1/1

docker service ps portainer_portainer
# Expected: running on manager-1

# ============================================
# PART 8: ACCESS PORTAINER
# ============================================
# Open https://192.168.1.10:9443 in a browser
# Complete the setup wizard

# ============================================
# PART 9: TEST A MANAGER FAILURE
# ============================================
# On manager-3, stop Docker
sudo systemctl stop docker

# On manager-1, check quorum
docker node ls
# manager-3 shows Unreachable
# Quorum still maintained (2 of 3)

# ============================================
# PART 10: RESTORE THE MANAGER
# ============================================
# On manager-3, start Docker
sudo systemctl start docker

# Verify it rejoins
docker node ls
# manager-3 shows Reachable again

The ten parts cover initializing the first manager, joining the other managers, verifying quorum, creating the Portainer data directory, preparing the stack with placement constraints, deploying, verifying the deployment, accessing Portainer, testing a manager failure, and restoring the manager. What the session demonstrates is that the cluster survives a single manager failure, that Portainer remains accessible as long as its pinned node is healthy, and that the workloads continue running regardless of which manager fails.

Quick Reference

Quorum and Fault Tolerance

Managers	Quorum	Tolerated Failures
1	1	0
3	2	1
5	3	2
7	4	3

Portainer Deployment Choices

Approach	Availability	Complexity
Pinned to one manager, local volume	Management lost if node fails	Low
Pinned to one manager, shared volume	Management survives node failure	Medium
Portainer outside Swarm, agent inside	Management survives Swarm failure	Medium
Multiple Portainer servers	Not supported	—

Key Ports

Port	Purpose
2377	Swarm manager communication
7946	Node discovery (TCP/UDP)
4789	Overlay network (UDP)
9001	Portainer agent communication
9443	Portainer web UI

Failure Behavior

Failure	Quorum	Services	Portainer
Worker down	Unaffected	Rescheduled	Unaffected
One manager down (of 3)	Maintained	Continue	Affected only if on that node
Two managers down (of 3)	Lost	Continue running	Affected if on failed node
Portainer node down	Maintained	Continue	Unavailable

Best Practices

✅ Do This:

docker swarm init --advertise-addr 192.168.1.10          # ✅ stable IP
docker node update --availability drain manager-1         # ✅ drain managers
--constraint 'node.hostname == manager-1'                 # ✅ pin Portainer
tar czf backup.tgz /var/lib/docker/swarm                  # ✅ back up Swarm state
docker node promote worker-1                              # ✅ restore quorum

❌ Don’t Do This:

# run an even number of managers                        # ❌ no extra tolerance
# pin Portainer without backing up /data                # ❌ data loss on node failure
# run multiple Portainer servers in one Swarm           # ❌ not supported
# drain all managers without workers                   # ❌ no place to schedule

Common Pitfalls

Pitfall	Why It Happens	Fix
Quorum lost	Two of three managers down	Restore one, or `--force-new-cluster`
Portainer unreachable	Pinned node down	Bring node back, or redeploy on another manager
Portainer data lost	Local volume on failed node	Restore from backup, or use shared storage
Services not rescheduling	Managers drained, no workers	Add workers or un-drain a manager
Node cannot rejoin	Raft data copied	Remove and re-join with fresh state

Real-World Examples

1. Three-manager homelab

docker swarm init --advertise-addr 192.168.1.10
docker swarm join --token <manager-token> 192.168.1.10:2377

2. Drain managers of workloads

docker node update --availability drain manager-1

3. Pin Portainer to a manager

deploy:
  placement:
    constraints:
      - node.hostname == manager-1

4. Back up Swarm state

tar czf /backup/swarm.tgz /var/lib/docker/swarm

5. Force new cluster from survivor

docker swarm init --force-new-cluster

6. Deploy Portainer stack

docker stack deploy -c portainer-agent-stack.yml portainer

7. Check Portainer placement

docker service ps portainer_portainer

8. Restore quorum by promoting

docker node promote worker-1

9. Access Portainer

https://<manager-ip>:9443

10. Verify agent on all nodes

docker service ls | grep agent

Visual

Three-Manager Homelab Topology

┌──────────────────────────────────────────────┐
│  MANAGER-1          MANAGER-2       MANAGER-3│
│  Leader             Reachable       Reachable│
│  Portainer          (no workload)   (no workload)│
│  Agent              Agent           Agent    │
│         │                │                │   │
│         └────────────────┼────────────────┘   │
│                          │                    │
│              Raft quorum = 2                  │
│              Tolerates 1 failure              │
└──────────────────────────┼────────────────────┘
                           │
              ┌────────────┴────────────┐
              │                         │
              ▼                         ▼
┌──────────────────────┐   ┌──────────────────────┐
│  WORKER-1            │   │  WORKER-2            │
│  Application tasks   │   │  Application tasks   │
│  Agent               │   │  Agent               │
└──────────────────────┘   └──────────────────────┘

Failure Impact Matrix

┌──────────────────────────────────────────────┐
│  FAILURE: Worker down                        │
│  Quorum: unchanged                           │
│  Services: rescheduled to other workers      │
│  Portainer: unaffected                       │
│                                              │
│  FAILURE: One manager down (of three)        │
│  Quorum: maintained (2 of 3)                 │
│  Services: continue running                  │
│  Portainer: unavailable if on failed node    │
│                                              │
│  FAILURE: Two managers down (of three)       │
│  Quorum: lost                                │
│  Services: continue running                  │
│  Admin: blocked until quorum restored        │
└──────────────────────────────────────────────┘

Portainer Placement Decision

┌──────────────────────────────────────────────┐
│  Do you have shared storage for /data?       │
│  ├─ YES → Can reschedule Portainer           │
│  │         across managers                   │
│  └─ NO → Pin to one manager                  │
│            Back up /data regularly           │
│            Accept management downtime         │
└──────────────────────────────────────────────┘

Swarm State Backup

┌──────────────────────────────────────────────┐
│  /var/lib/docker/swarm on each manager       │
│         │                                    │
│         ▼                                    │
│  Contains: Raft log, service definitions,    │
│            secrets, configs, network state   │
│         │                                    │
│         ▼                                    │
│  Back up with Docker stopped:                │
│  systemctl stop docker                       │
│  tar czf swarm.tgz /var/lib/docker/swarm     │
│  systemctl start docker                      │
└──────────────────────────────────────────────┘

Summary

Item	Value
Minimum HA managers	3
Quorum (3 managers)	2
Tolerated failures (3 managers)	1
Portainer instances	1 per cluster
Portainer data	`/data` volume
Portainer pinning	`node.hostname` constraint
Swarm state backup	`/var/lib/docker/swarm`
Quorum recovery	Restore manager or `--force-new-cluster`

Key takeaways:

Three managers is the practical minimum for HA. It provides one-failure tolerance without the overhead of five. Always use an odd number; four managers provide no more tolerance than three .
Losing one manager does not stop your services. The remaining quorum maintains the cluster, and workloads continue running on all nodes. Only administrative operations are affected if quorum is lost .
Portainer is a single point of failure for management, not for workloads. Portainer does not support multiple server instances against one cluster . Pin it to a manager, back up its data, and accept that management access requires that node.
Shared storage enables Portainer failover. If /data is on a network share available to all managers, Portainer can be rescheduled. Without it, pin Portainer to one manager and rely on backups .
Drain managers in production. Managers should handle control-plane duties only. Workers run the containers. This keeps the Raft consensus responsive and isolates workloads from manager restarts.
Back up the Swarm state. The /var/lib/docker/swarm directory contains the Raft log, secrets, and configs. Without it, a total manager loss means rebuilding the cluster from scratch .

Remember: A high-availability homelab is not about running everything at once. It is about ensuring that the failure of any single machine does not take down the services you depend on. Three managers provide that guarantee for the cluster state. Portainer provides the management interface, but it is deliberately a single instance, and its availability is separate from the availability of your services. Pin it, back it up, and understand that when its node fails, your services keep running—you just cannot change them until the node returns or you redeploy Portainer with its data. That distinction is the difference between a homelab that is genuinely resilient and one that merely looks resilient.

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!