Specialist Part (9): RGW — Inside Multisite: Periods, Log Formats, Bucket Generations and the Sync Engine

42-1 Before we begin#

Chapter 5 covered multisite from the point of view of setup and operation: the realm, zonegroup and zone hierarchy, the three logs, the data sync state machine, choosing the datalog and mdlog shard, the coalescing window and trim. This last chapter of the specialist part opens up the inside of that same engine: exactly what checks committing a period involves and when it creates a new period, what fields the bilog, datalog and mdlog records have, which RADOS objects hold the sync status, how reshard copes with sync through log "generations", where errors are queued, and how a conflict from concurrent writes in two zones is resolved. Names have been matched against the main branch (November 2026).

42-1-1 What questions does this chapter answer?#

• Why does period update --commit sometimes raise the epoch and sometimes change the period id?
• What information do a bilog entry and a datalog entry carry?
• In which object, and under what name, is the sync status of a bucket shard stored?
• After a reshard in the source zone, how does the destination zone continue without losing changes?
• If syncing a bucket fails, where is it recorded and when is it retried?
• If an object is written in two zones at the same time, which version stays?

42-1-2 Prerequisites#

Chapter 5 (multisite), chapter 40 (the bucket index and bilog) and chapter 41 (the request path).

42-2 Periods and commit#

The multisite configuration is recorded in a period: an id, epoch, realm_epoch, predecessor_uuid, the master zonegroup and master zone, and the full map of zonegroups. Changes are first collected in the "staging" period (id <realm-id>:staging), and radosgw-admin period update --commit builds it on top of the current period.

pseudocode
commit_period(staging, current):
    require staging.master_zone == this zone                     # only the master zone commits
    require staging.predecessor_uuid == current.id
    require staging.realm_epoch == current.realm_epoch + 1
    if staging.master_zone != current.master_zone:               # failover / master change
        store current metadata sync status into the new period
        new period id ; epoch = FIRST_EPOCH ; realm_epoch advances
        set as realm's current period ; new mdlog prefix meta.log.<new-period>.
    else:
        require staging.epoch == current.epoch (else "period pull")
        keep current.id ; epoch = current.epoch + 1                # same period, newer epoch
    store period ; notify all zones (push to their /admin/realm/period)

The result: ordinary changes (adding a zone, an endpoint, a placement) only raise the epoch, and the mdlog continues. But changing the master zone creates a new period and therefore a new mdlog; the metadata sync status is stored in the previous period so that the zones can first read the old period's log to its end and then move to the new period. The message "does not come directly after current realm epoch. Use 'realm pull'" means this zone does not have the current period.

# example: viewing the current period
radosgw-admin period get-current
radosgw-admin period get | jq '{id, epoch, realm_epoch, predecessor_uuid, master_zone}'

example (output):
{ "current_period": "7c1f2b9e-5d1a-4c2e-9a0f-3b8d6e4f1a20" }
{ "id": "7c1f2b9e-5d1a-4c2e-9a0f-3b8d6e4f1a20", "epoch": 4, "realm_epoch": 2,
  "predecessor_uuid": "0e9a4d33-1b6c-4f7e-8a25-c5d2f8b7e911", "master_zone": "b3a9e1c4-...-zone-a" }

42-3 The format of log records#

zones_trace is the list of zones a change has passed through; when the destination zone fetches an object from the source and writes a bilog entry of its own, the source zone is in the trace, and the change is not sent back during reverse sync. Without it, in an active-active topology every write would bounce back and forth in an endless loop.

The gen field in the datalog (and the generation suffix in the names of status objects) only has meaning after reshards that have created a new generation; the encoding of rgw_data_change has compat 1 when gen is zero and 2 otherwise, so old RGWs can still read entries without a generation.

42-4 Sync status on RADOS#

All status objects are in the log pool of the destination zone (<zone>.rgw.log) and are named after the source zone's id (rgw_data_sync.cc and rgw_sync.cc):

Each datalog shard is processed by only one RGW in the destination zone at any moment. This is done with a lease (a cls lock with rgw_sync_lease_period of 2 minutes that is renewed) and, in new versions, with a bid mechanism that spreads the shards among the available RGWs; an RGW that loses a bid releases the shard.

42-5 The data sync engine#

pseudocode
for each datalog shard N owned by this RGW (lease or bid):
    state Init → BuildingFullSyncMaps: list all buckets of source into data.full-sync.index.*
    full sync: for each bucket shard in index → sync bucket shard (full) ; advance marker
    incremental: loop
        also wake on out-of-band notify from source (modified shards)
        entries = remote datalog list(N, after marker)
        for each entry key = bucket:instance:shard (gen):
            spawn RGWDataSyncSingleEntryCR (≤ 20 in flight, same key coalesced)
                ok   → advance marker when all earlier entries done (marker tracker)
                fail → write key to <status>.retry (error repo) ; marker still advances
        every pass: retry keys from error repo with backoff ; remove on success

The essential point is that the shard marker advances despite failures; otherwise one broken bucket would stall the whole shard. Failures stay in the error repo and are retried separately. radosgw-admin sync error list shows the recorded permanent failures, and data sync status --shard-id N --source-zone Z the details of one shard.

pseudocode
sync bucket (source bucket, dest bucket, gen):
    status = bucket.full-sync-status ; if Init → Full: list source bucket (versions) and fetch each
    init: incremental_gen = source latest_gen (full listing already covers older generations)
    Full done → Incremental
    incremental shard: bilog list(shard, gen, after inc_marker) from source
        squash entries of same object/instance ; skip entries whose zones_trace has this zone
        for each: ADD / LINK_OLH → fetch_remote_obj ; DEL / UNLINK_INSTANCE → remove locally
        advance inc_marker via marker tracker
    when every shard of gen is done (shards_done_with_gen) and remote has gen+1:
        incremental_gen = gen+1 (shard count may differ after reshard)
fetch_remote_obj(copy_if_newer):
    GET object from source zone (with sync manifest, attrs, versioned_epoch)
    compare obj_time_weight: mtime (seconds unless high precision), then zone_short_id, then pg_ver
    if destination copy is not older → skip ; else write locally, zones_trace += source

Generations and reshard. In the old model, a reshard in one zone destroyed the bilogs of the old shards, and the destination zone had to full sync that bucket from scratch. With the generational layout, a reshard creates generation gen+1 with the new number of shards and keeps the previous generation's bilogs until all peers have completed shards_done_with_gen for that generation. The destination reads the old generation to its end, then moves to the new generation; each shard's status is stored separately with the :<gen> suffix. Automatic trimming of the old generation's log (rgw_sync_log_trim_interval 20 minutes) happens after that.

A conflict example. Object k is written in zone A at 07:15:00.400 and in zone B at 07:15:00.900 (the same second). At one-second precision the mtimes are equal and the decision is made by zone_short_id: the version from the zone with the larger short id wins, and both zones end up with that same version. That is, multisite does not guarantee last writer wins in the strict temporal sense; applications that are sensitive to write order should write each key in only one zone or use versioning.

42-6 Metadata sync#

pseudocode
non-master zone, for each mdlog shard N of the current period (lease or bid):
    full sync: list all metadata keys on master (meta.full-sync.index.*) ; fetch each
    incremental: read meta.log.<period>.<N> after marker ; for each entry (section:key):
        GET /admin/metadata/<section>?key=K from master ; put locally (versioned with read/write version)
    on period change: finish old period's log to its end, then switch to new period's log
writes of metadata (create user, bucket) on non-master: forwarded to master zone first

That is why creating a bucket or a user in a non-master zone fails when the master is unavailable, while writing objects to existing buckets continues. (Failover and promoting the master are covered in chapter 5.)

42-7 Multisite command reference#

# example: a bucket that is in generation 1 after a reshard
radosgw-admin bucket sync status --bucket logs --source-zone zone-a

example (output):
          realm 2f61a0c4-... (prod)
      zonegroup 9b07d2e1-... (eu)
           zone 5c3e8a77-... (zone-b)
         bucket :logs[a1b2c3d4-...4123.9])
   current time 2026-11-15T07:20:11Z

    source zone b3a9e1c4-... (zone-a)
  source bucket :logs[a1b2c3d4-...4123.9])
                incremental sync on 23 shards
                bucket is behind on 2 shards
                behind shards: [4,17]

42-8 Tuning and troubleshooting#

• Lagging shards that do not advance: look at sync error list and data sync status --shard-id N; if one bucket keeps failing, bucket sync run for that bucket isolates the problem and makes it visible.
• Unbounded growth of the datalog or bilog: an old or removed peer is still defined in the zonegroup and trim is waiting for its marker (chapter 5); remove the dead zone from the zonegroup and commit the period.
• Slow sync with many small buckets: carefully increase rgw_data_sync_spawn_window and rgw_bucket_sync_spawn_window (both 20), and raise the number of syncing RGWs so the bids spread out. RGWs that serve users can be separated from sync with rgw_run_sync_thread = false.
• After a master change: if zones are stuck on the old period, period pull and an RGW restart are needed.
• Content differences despite "caught up": temporarily turn on rgw_sync_obj_etag_verify (default false) so that the ETag of each copied object is checked.

42-9 Supplementary and up to date#

• The generational bilog layout and sync after a reshard in multisite are supported in recent versions, and they require the zonegroup feature resharding; before resharding in multisite, check radosgw-admin zonegroup get and the enabled_features list.
• The bid mechanism for spreading shards is in the main branch; older versions had only the lease, and one RGW could take most of the shards.
• In the main branch the option rgw_default_bucket_bilog_type (default fifo) determines whether new buckets keep their bilog in the index (inindex) or in a separate FIFO; existing InIndex buckets migrate at their next reshard. This behavior does not exist in older released versions; check your version before relying on it.

42-10 Hands-on exercise#

1. 1. In a two-zone environment, add an endpoint to the zonegroup, commit, and see the epoch change. Then change the master and see the id and realm_epoch change.
2. 2. Do a PUT and follow its entry in bilog list, datalog list and then in bucket sync markers in the destination zone.
3. 3. Write one object in the same second in both zones and see which version wins; compare the result with the two zones' zone_short_id (in zonegroup get).
4. 4. Reshard a bucket in the source zone and follow the destination's bucket sync status until it moves to the new generation.

42-11 Review questions#

1. 1. What three conditions does committing a period check before anything else?
2. 2. When does a commit change the period id, and what is the consequence for the mdlog?
3. 3. What problem does zones_trace in the bilog solve?
4. 4. What does the key of a datalog entry identify?
5. 5. In which object and which pool is the sync status of a bucket shard?
6. 6. Why does the datalog shard marker advance despite a bucket's failure, and where is the failure recorded?
7. 7. What roles do the lease and the bid play in sync?
8. 8. After a reshard in the source zone, how does the destination zone continue without a full sync?
9. 9. In what order of criteria is a concurrent-write conflict resolved?
10. 10. Why does creating a bucket in a non-master zone depend on the master being available?

Answers to these questions are in Appendix E: Answers to Review Questions. Answers to this chapter's questions →

42-12 Sources for this chapter#

• rgw_zone.cc (commit_period
  https://github.com/ceph/ceph/blob/main/src/rgw/rgw_zone.cc
• rgw_data_sync.cc and rgw_data_sync.h
  https://github.com/ceph/ceph/blob/main/src/rgw/driver/rados/rgw_data_sync.cc
• rgw_sync.cc (metadata sync
  https://github.com/ceph/ceph/blob/main/src/rgw/driver/rados/rgw_sync.cc
• rgw_datalog.h
  https://github.com/ceph/ceph/blob/main/src/rgw/driver/rados/rgw_datalog.h
• cls_rgw_types.h (rgw_bi_log_entry
  https://github.com/ceph/ceph/blob/main/src/cls/rgw/cls_rgw_types.h
• Multisite documentation
  https://docs.ceph.com/en/latest/radosgw/multisite/
• Dynamic resharding documentation (including multisite
  https://docs.ceph.com/en/latest/radosgw/dynamicresharding/