Disaster Recovery: rbd-mirror, Stretch Mode and Cluster Rescue

18-1 Before you start#

18-1-1 What questions does this chapter answer?#

"Disaster recovery" in Ceph is not a single feature but several layers of defence, each protecting against a particular kind of disaster. Three-way replication protects against disk and host failure, but it can do nothing against the accidental deletion of a pool, ransomware or the complete loss of a data centre. This chapter reviews the layers from inside a cluster to an independent backup, and explains the two main mechanisms down to the code level: rbd-mirror and stretch mode.

• What are the RPO and RTO of each layer, and which layer protects against which disaster?
• How does rbd-mirror work in its two modes, journal and snapshot?
• How do planned and emergency failover differ, and how is split-brain resolved?
• How does stretch mode keep one cluster across two sites with an RPO of zero?
• If all MONs are lost, how do we rebuild the maps from the OSDs?
• What must be backed up separately from Ceph?

18-2 Layers of defence#

18-3 rbd-mirror#

18-3-1 Two modes#

18-3-2 Setup (snapshot mode)#

# on both clusters: the rbd-mirror service (in cephadm)
ceph orch apply rbd-mirror --placement=1
# site A
rbd mirror pool enable --site-name site-a vmpool image
rbd mirror pool peer bootstrap create --site-name site-a vmpool > /tmp/token
# site B (transfer the token securely; it contains a cephx key)
rbd mirror pool enable --site-name site-b vmpool image
rbd mirror pool peer bootstrap import --site-name site-b --direction rx-tx vmpool /tmp/token
# enable for one image, and the schedule
rbd mirror image enable vmpool/db01 snapshot
rbd mirror snapshot schedule add --pool vmpool 15m
rbd mirror snapshot schedule status --pool vmpool
rbd mirror pool status vmpool --verbose

18-4 Inside the code: rbd-mirror#

18-4-1 Daemon structure#

rbd-mirror (src/tools/rbd_mirror/) is a standalone daemon that runs in the destination cluster and pulls from the source cluster. It builds one PoolReplayer per pool and one ImageReplayer per image. Depending on the image's mode, ImageReplayer uses one of two implementations: image_replayer/journal/Replayer or image_replayer/snapshot/Replayer. Several rbd-mirror daemons in a cluster coordinate with each other (through leader election on RADOS objects) and divide the images among themselves.

18-4-2 Snapshot mode: the algorithm#

Each mirror snapshot is recorded in the snapshot namespace with the MirrorSnapshotNamespace structure (src/cls/rbd/cls_rbd_types.h). The key fields:

The order of functions in snapshot/Replayer.cc follows exactly this path: load_local_image_meta → refresh_remote_image / refresh_local_image → scan_local_mirror_snapshots → scan_remote_mirror_snapshots → create_non_primary_snapshot → copy_image → apply_image_state → update_non_primary_snapshot(complete=true) → unlink_peer.

pseudocode: one Replayer round (this book's rewrite)
  local_last  = the last complete NON_PRIMARY snapshot at the destination
  remote_from = the source snapshot matching local_last.primary_snap_id
  remote_to   = the first PRIMARY snapshot after remote_from whose mirror_peer_uuids includes our peer
  if none: wait for remote image update notification; return
  if local has incomplete snapshot for remote_to: resume from last_copied_object_number
  else: create_non_primary_snapshot(remote_to, complete=false)
  deep_copy(remote, local, snap range = (remote_from, remote_to])   # only changed objects
  apply_image_state(size, metadata, snapshot names)
  mark local snapshot complete
  unlink_peer(remote_from)        # the source can prune the old snapshot

18-4-3 Journal mode: the algorithm#

In journal mode, librbd appends each write to the journal (the src/journal/ library, which stores events in RADOS objects of the same pool) before applying it to the image. At the destination, rbd-mirror connects to the source journal with init_remote_journaler, reads the events, prepares them with preprocess_entry (for example mapping snapshot ids) and replays them on the local image with process_entry. The replay position is recorded in the source journal with flush_commit_position so that the source can trim the consumed parts.

18-4-4 Failover and failback#

# planned
rbd --cluster site-a mirror image demote vmpool/db01
rbd --cluster site-b mirror image status vmpool/db01      # wait for up+replaying and sync
rbd --cluster site-b mirror image promote vmpool/db01
# emergency (site A is unavailable)
rbd --cluster site-b mirror image promote --force vmpool/db01
# after A returns: prevent two primaries
rbd --cluster site-a mirror image demote vmpool/db01
rbd --cluster site-a mirror image resync vmpool/db01

18-5 The lower layer: the journal on RADOS and the stretch state machine#

18-5-1 The journal format in journal-based mirroring#

The src/journal/ library is a general-purpose journal on RADOS, and librbd is only one of its consumers. Each image has a header object named journal.<image_id>, whose omap holds the order (object size), splay_width, the active set number, the list of clients (such as each peer's rbd-mirror) and each one's commit position. Data is written to objects named journal_data.<pool_id>.<image_id>.<n> (the constants JOURNAL_HEADER_PREFIX and JOURNAL_OBJECT_PREFIX in Journaler.cc).

In JournalRecorder.cc, two simple relations determine everything:

splay_offset  = entry_tid % splay_width
object_number = splay_offset + active_set * splay_width

With the defaults (rbd_journal_order=24, that is 16MiB, and rbd_journal_splay_width=4), the entry with tid=10 in active set 3 goes to object number 2+3×4=14. Spreading entries over four objects spreads the appends across four PGs (and usually four primary OSDs) so that one hot object does not become a bottleneck. Each set holds at most 4×16MiB=64MiB of data.

The bytes of an entry#

Entry::encode writes a fixed 25-byte header part (preamble, version, entry_tid, tag_tid) and a fixed 8-byte trailer (data length and crc). The preamble is the constant 0x3141592653589793 (the digits of π), which is checked on read-back to find entry boundaries and detect corrupt data. The crc is computed with crc32c(0) over everything except the crc itself.

rbd journal info --pool vmpool --image db01        # order, splay_width, object_pool
rbd journal status --pool vmpool --image db01      # active_set, minimum_set, clients' commit positions
rados -p vmpool ls | grep '^journal_data\.' | sort -t. -k4 -n | tail -4
rbd journal inspect --pool vmpool --image db01 --verbose | head   # check entry consistency

Options that matter in rbd-mirror#

18-5-2 The stretch mode state machine in the code#

In stretch mode the OSDMap has five fields: stretch_mode_enabled, stretch_bucket_count (the number of sites, currently 2), degraded_stretch_mode (0 or the number of live sites), recovering_stretch_mode and stretch_mode_bucket (the bucket type, for example datacenter). Each stretched pool also has four peering fields: peering_crush_bucket_count, peering_crush_bucket_target, peering_crush_bucket_barrier and peering_crush_mandatory_member.

The heart of the matter is the function pg_pool_t::stretch_set_can_peer. Before a PG can become active with an acting set, the OSDs' ancestors at the barrier level (datacenter) are counted:

pseudocode: stretch_set_can_peer (summary)
  if pool is not stretch: return true
  sites = { crush.parent_of_type(osd, barrier) for osd in want }
  if |sites| < peering_crush_bucket_count: return false     # all sites are required
  if mandatory_member != NONE and mandatory_member not in sites: return false
  return true

That is, in the healthy state no stretched PG becomes active with OSDs from only one site, even if it has min_size copies. This is the main barrier against split-brain: if the link between the sites breaks and the MONs have not yet marked the state degraded, the PGs stop rather than accept writes at one site. Only after the MONs decide (with the tiebreaker) and enter DEGRADED does the count drop to 1, and mandatory_member is set to the surviving site. Even if the dead site comes back, it cannot activate a PG on its own until recovery ends.

ceph osd dump | grep -E 'stretch_mode|degraded_stretch|recovering_stretch'
ceph osd pool ls detail | grep -E 'peering_crush|min_size'
ceph mon dump | grep -E 'tiebreaker|stretch|location'
ceph config get mon mon_stretch_cluster_recovery_ratio   # 0.6

18-6 Disaster recovery in numbers: MON connection scores, connectivity elections and image distribution in rbd-mirror#

The three algorithms in this section determine which MON becomes leader in a two-site cluster, and which rbd-mirror daemon replicates which image. References are to main as of November 2026. The scenarios are hypothetical, but all numbers were computed with a Python reimplementation of the same formulas.

18-6-1 The connection score of each MON pair#

pseudocode
h = mon_con_tracker_score_halflife (43200 s) ; each peer's initial score 1.0
ping every mon_elector_ping_timeout / divisor = 2 / 2 = 1 s ; after 2 s without a reply, the peer is "dead"
live reply after u seconds:  s = min(1, s × (1 − u/2h) + u/2h)
no reply for u seconds (every 2 s):     s = max(0, s × (1 − u/2h) − u/2h)
score(r) = Σ over the other MONs' reports: history[r], only if current[r] is alive

Numerical example. With α = u / 86400, a connection that dies from a score of 1 reaches 0.986 after 10 minutes, 0.918 after one hour, 0.558 after six hours, 0.213 after twelve hours, and zero after exactly 59888 seconds (86400 × ln 2, about 16.6 hours). Recovery is equally slow: going from 0 to 0.5 takes about 16.6 hours. This long memory is deliberate: a connection that keeps dropping and reconnecting should not be counted as "healthy" again after a few successful pings.

18-6-2 Elections with the connectivity strategy#

pseudocode
score(r) = −1 if r is in disallowed_leaders (e.g. the tiebreaker in stretch mode) ; otherwise get_total_connection_score(r)
on receiving PROPOSE from f:
    if I am the tiebreaker and f is marked down: ignore
    my_win = my ≥ 0 and (my > from or (my == from and my rank < f))
    if my_win: propose myself
    elif from is better than me and than the currently acked leader: ACK f
votes are not changed for an improvement smaller than mon_elector_ignore_propose_margin (0.0005)

Numerical example. We have five MONs: a and b in DC1, c and d in DC2, and e as the tiebreaker, which is not allowed to become leader. Suppose a's link to c and d has been down for an hour and everything else is healthy. a's score is summed only from the live reports of b and e and is 2.0. b's score is four (1 from each of a, c, d and e). c and d each have 3, because a's report about them is "dead" and is not counted. So b becomes leader. With the classic strategy, a would remain leader merely because of its lower rank, even though it cannot reach half the cluster. After the links reconnect, a's score is 1 + 0.918 + 0.918 + 1 = 3.837, still less than b's (4.0), so leadership does not change. The default mon_election_default_strategy is 1 (classic). stretch mode requires the connectivity strategy, which is set with ceph mon set election_strategy connectivity.

18-6-3 Distributing images among rbd-mirror daemons#

pseudocode
per = max(1, ⌊number of images / live instances⌋)
when an instance is added:
    for each instance with more than per images: cut = size − per
        walk its images in order: if being moved, cut−− ; elif can_shuffle: cut−− and mark it
can_shuffle = idle and (now − mapped_time ≥ rbd_mirror_image_policy_migration_throttle (300 s))
do_map(img) = the live instance with the fewest images
periodic rebalance only if rbd_mirror_image_policy_rebalance_timeout > 0 (default 0 = off)

Numerical example. 100 old images are on daemon A. B is added at t = 0: per = 50, so 50 images go to B (50/50). C is added at t = 200 seconds: per = 33. A must give up 17 images and does. B must also give up 17 images, but all of its images were mapped less than 300 seconds ago, so none of them move. The result is 33/50/17. Because periodic rebalancing is off by default, this imbalance remains until the next membership change. If rbd_mirror_image_policy_rebalance_timeout is, say, 300, the map is spread again once it settles, and the result is 34/33/33. If A dies, its 34 images go one by one to the least loaded instance, and the result is 50/50. Each move means releasing and re-acquiring the image and starting replay from the last snapshot. So bring rbd-mirror instances up together, not minutes apart.

18-7 stretch mode: one cluster across two sites#

18-7-1 The model#

stretch mode is for when you have two data centres with low latency (a few milliseconds) and want to tolerate the complete loss of one of them without data loss (RPO of zero) and with continued service. Each site keeps a full copy of the data, and a tiebreaker MON at a third site, which holds no data, decides which site stays active when the two sites lose contact.

18-7-2 Configuration in released versions (Squid, Tentacle, Umbrella)#

The general procedure in these versions' documentation:

1. 1. Create datacenter buckets in CRUSH and move the hosts into them.
2. 2. Set each MON's location with ceph mon set_location <mon> datacenter=<site>.
3. 3. Create a CRUSH rule that places two copies in each datacenter (in the documentation, by editing the CRUSH map).
4. 4. Change the MON election strategy to connectivity.
5. 5. Enable with ceph mon enable_stretch_mode <tiebreaker-mon> <rule> datacenter.

ceph osd crush add-bucket site1 datacenter
ceph osd crush add-bucket site2 datacenter
ceph osd crush move site1 root=default
ceph osd crush move site2 root=default
ceph osd crush move stor-a1 datacenter=site1      # and the other hosts
ceph mon set_location a datacenter=site1
ceph mon set_location b datacenter=site1
ceph mon set_location c datacenter=site2
ceph mon set_location d datacenter=site2
ceph mon set_location e datacenter=site3
ceph mon set election_strategy connectivity
ceph mon enable_stretch_mode e stretch_rule datacenter

18-7-3 Behaviour on failure#

When one site is lost, the cluster enters degraded stretch mode: the MONs form a quorum with the tiebreaker and, according to the Tentacle documentation, the pools' min_size drops to 1 (size does not change) so that the PGs stay active with the copies at the surviving site; that is, during this period you have only two copies at one site and the safety margin is small. After the site returns, the cluster goes into recovery and, once the PGs have synchronized, returns to normal.

18-8 When all MONs are lost#

18-8-1 Prevention#

• At least three, and preferably five, MONs on reliable disks with correct flush and write settings.
• Since Umbrella: periodic backups of the monitor's RocksDB with mon_backup_interval, manual backups with ceph tell mon.* backup and restoring with ceph-mon --restore-backup (chapter 8). This is the simplest and safest way to recover a MON.
• Separate backups of ceph config dump, ceph auth export and the service specifications from ceph orch ls --export somewhere outside the cluster (encrypted, because they include keys).

18-8-2 Rebuilding from the OSDs#

If no backup exists, the official documentation (Troubleshooting Monitors, section Monitor Store Failures) gives a procedure for rebuilding the monitor store from the maps that each OSD keeps. A summary in this book's words:

1. 1. Stop all OSDs.
2. 2. Create an empty store directory and take it to each OSD host in turn; on each host, for each OSD, run ceph-objectstore-tool --data-path <osd> --no-mon-config --op update-mon-db --mon-store-path <dir> to add that OSD's maps to the store; then bring the accumulated store back.
3. 3. Build a keyring containing the mon. and client.admin keys (and the mgr keys) with the necessary caps.
4. 4. Run ceph-monstore-tool <dir> rebuild -- --keyring <keyring> --mon-ids <ids> (the order of the ids matters if they are not sorted by IP).
5. 5. Make a copy of the broken store.db, replace it with the rebuilt store, fix the file ownership and start the MONs.

18-9 Rescuing data from an OSD#

On a stopped OSD, ceph-objectstore-tool can list, export and import PGs (chapter 17). Its DR use is when several OSDs have been lost at once and a PG is stuck incomplete or down, but the disk of one of the dead OSDs is still readable.

# inside cephadm shell for the stopped OSD
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12 --op list-pgs
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12 --pgid 2.5 --op info
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12 --pgid 2.5 --op export --file /backup/2.5.export
# on the destination OSD (stopped)
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-30 --op import --file /backup/2.5.export

18-10 Independent backup#

• RBD: rbd export-diff between snapshots and rbd import-diff into another cluster or store; or VM-level backup tools.
• RGW: bucket versioning together with S3 Object Lock in COMPLIANCE mode, so that even an administrator cannot delete before the period ends (chapter 4); synchronization to external storage.
• CephFS: directory snapshots and cephfs-mirror to another cluster (since Pacific; chapter 6).
• Configuration: the output of ceph orch ls --export, ceph config dump and the keyrings.

18-11 Supplementary and up to date#

• Snapshot-based mirroring has been available since Octopus and has become a suitable replacement for the journal for most uses.
• Umbrella added built-in backups of the monitor store (chapter 8).
• The documentation of the development branch describes the new stretch model with num_zones; until it is officially released in a version, use the enable_stretch_mode procedure.
• ceph-deploy, which old rebuilding procedures used, has been archived; all procedures today are done with cephadm.

18-12 Hands-on exercise#

1. 1. Build two small clusters, enable snapshot mirroring for one image and measure the sync lag with rbd mirror image status.
2. 2. Perform a planned failover and confirm that no data was lost (file checksums).
3. 3. Simulate an emergency failover by cutting site A's network; then run the resync procedure.
4. 4. In snapshot/Replayer.cc, find where copying resumes from last_copied_object_number.
5. 5. In a lab, delete one MON's store and run the full rebuild-from-OSDs procedure.
6. 6. Create a bucket with object lock in COMPLIANCE mode and try to delete from it.

18-13 Review questions#

1. 1. Why is replication not a substitute for backup?
2. 2. What is the difference between journal-based and snapshot-based mirroring in overhead and RPO?
3. 3. What role does the complete field in MirrorSnapshotNamespace play?
4. 4. In which cluster does rbd-mirror run, and why does it pull?
5. 5. What must be done after promote --force when the previous site returns?
6. 6. What is the role of the tiebreaker MON in stretch mode?
7. 7. What effect does stretch mode have on write latency?
8. 8. What information is lost when rebuilding the MONs from the OSDs?
9. 9. What tool has Umbrella added for MON backups?
10. 10. What is ceph-objectstore-tool used for in DR, and what is its prerequisite?

Answers to these questions are in Appendix E: Answers to Review Questions. Answers to this chapter's questions →

18-14 Sources for this chapter#

• RBD mirroring
  https://docs.ceph.com/en/latest/rbd/rbd-mirroring/
• Stretch mode (Tentacle
  https://docs.ceph.com/en/tentacle/rados/operations/stretch-mode/
• Stretch mode (Umbrella
  https://docs.ceph.com/en/umbrella/rados/operations/stretch-mode/
• Stretch mode (latest/dev
  https://docs.ceph.com/en/latest/rados/operations/stretch-mode/
• Troubleshooting monitors (Monitor Store Failures
  https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/
• CephFS mirroring
  https://docs.ceph.com/en/latest/cephfs/cephfs-mirroring/
• RGW S3 bucket operations (object lock config
  https://docs.ceph.com/en/latest/radosgw/s3/bucketops/
• Source: src/tools/rbd_mirror
  https://github.com/ceph/ceph/tree/main/src/tools/rbd_mirror
• Source: src/cls/rbd/cls_rbd_types.h
  https://github.com/ceph/ceph/blob/main/src/cls/rbd/cls_rbd_types.h
• Source: src/librbd/mirror/snapshot
  https://github.com/ceph/ceph/tree/main/src/librbd/mirror/snapshot