Lower Layers (10): Scrub, Recovery, Backfill and Repair at Code Level

32-1 Chapter map#

We examine three data-correctness mechanisms of the OSD at the code level:

• scrub: periodically comparing the copies of a PG on different OSDs. shallow compares only metadata and size, and deep reads all the data and omap and builds digests.
• recovery and backfill: restoring missing copies. recovery is based on the pg log, and backfill on a full scan of the content.
• repair: a combination of the two. scrub finds the authoritative copy and recovery puts it in place of the damaged copy.

Since Quincy, the scrub code in src/osd/scrubber/ has been split into several components: PgScrubber (the per-PG logic), ScrubMachine (a boost::statechart state machine), ScrubBackend (comparing the maps), OsdScrub and ScrubQueue (scheduling at the OSD level) and ScrubResources and ScrubReservations (reservation). All references are to main. The defaults were read from osd.yaml.in in the same branch.

32-2 Scheduling scrub#

32-2-1 Targets and randomization#

Each PG has a ScrubJob with two targets (shallow_target and deep_target). Each target has scheduled_at (the desired time), not_before (the earliest allowed time) and an urgency. The urgencies range from periodic_regular to operator_requested and must_repair. Only periodic targets are randomized (requires_randomization).

pseudocode
shallow: target = last_scrub + I_s + I_s × ratio × U(0,1)          # I_s = osd_scrub_min_interval
         not_before = max(last_scrub, target)
deep:    sd = I_d × cv                                              # cv = osd_deep_scrub_interval_cv
         delay = clamp(Normal(I_d, sd), I_d − 2sd, I_d + 2sd)
         target = not_before = last_deep + delay
initiate_scrub (on every OSD tick):
   if random() < osd_scrub_backoff_ratio: skip           # coin toss: 0.66
   if loadavg/ncpu ≥ osd_scrub_load_threshold: only mandatory scrubs
   take the earliest eligible target (not_before ≤ now) from the queue and reserve resources

Numerical example. A PG's last deep scrub was on Monday at 10:00. sd = 7×0.2 = 1.4 days. With a probability of about 68% the next one is between 5.6 and 8.4 days later, that is from about 22:00 on Saturday to about 20:00 on the following Tuesday. It is never earlier than 4.2 days (about 15:00 on Friday) or later than 9.8 days, unless the OSD queue is behind. The mon raises the PG_NOT_DEEP_SCRUBBED warning when more than I_d × (1 + mon_warn_pg_not_deep_scrubbed_ratio) = 7 × 1.75 = 12.25 days have passed since the last deep scrub (PGMap.cc). So the gap from 9.8 to 12.25 days is the margin for a lagging queue. The similar warning for shallow is computed with osd_scrub_max_interval × 1.5 = 10.5 days.

The coin toss and load. OsdScrub::scrub_random_backoff cancels the whole attempt of that tick with probability osd_scrub_backoff_ratio=0.66. If the tick is every second, the OSD tries on average once every 3 seconds (1/(1−0.66) ≈ 2.9). This reduces the cost of walking the queue and staggers the start of scrubs across OSDs. osd_scrub_load_threshold=10.0 is compared with loadavg divided by the number of CPUs. On a machine with 32 cores, loadavg must exceed 320 for normal scrubs to stop. That is why this option rarely has an effect in practice.

32-2-2 Reserving resources#

Before starting, PgScrubber reserves one slot of osd_max_scrubs (default 3) on its own OSD (inc_scrubs_local) and then on each replica (the ReservingReplicas state). If one replica refuses, the whole reservation is released and the PG is requeued with a delay of osd_scrub_retry_delay (30 seconds). Since Squid, replicas queue reservation requests instead of refusing them immediately. osd_scrub_disable_reservation_queuing=true restores the pre-Squid behaviour (an immediate grant or refusal). According to the option's description, it is only for mixed-version clusters and debugging and will be removed in the next release.

32-3 The scrub state machine#

Why WaitPushes and WaitLastUpdate? Before building its own map, the primary must be sure that in-flight recovery pushes have finished (WaitPushes) and that all writes accepted within the chunk's range have been applied to its own disk (WaitLastUpdate, until last_update_applied catches up). The replica likewise stays in ReplicaWaitUpdates before building its map, until its in-flight pushes drop to zero. The scrub_to field in MOSDRepScrub is, according to the code's own comment, no longer used. Because the range is locked before sending, no new write is accepted in it. Without these waits, the replica and the primary would compare different pictures of one moment, and false inconsistencies would be reported.

32-3-1 Choosing the chunk and preemption#

pseudocode
min_sz = max(3, deep ? osd_scrub_chunk_min : osd_shallow_scrub_chunk_min)
max_sz = max(min_sz, (deep ? osd_scrub_chunk_max : osd_shallow_scrub_chunk_max) / divisor)
objects, end = objects_list_partial(start, min_sz, max_sz)
while end.is_head() and end == objects.back().get_head():   # do not separate a head and its clones
    end = objects.pop_back()
if end.is_head(): end = end.get_object_boundary()
if not range_available_for_scrub(start, end): return blocked   # RangeBlocked
on preemption (a client write in the range): if left > 0: left -= 1; divisor *= 2

Numerical example. A deep scrub with the defaults: min=max(3,5)=5 and max=15. After each preemption, the divisor doubles: the maximum chunk is 15, 7, 5, 5, ... in turn (because 15/4=3 is less than 5). After osd_scrub_max_preemptions=5 times, no further preemption is allowed and client writes in that range wait until the end of the chunk. For shallow: min=50 and max=100, and after the first preemption all chunks are 50 objects. A PG with 3000 objects in a deep scrub without preemption has at least 3000/15 = 200 chunks. If osd_scrub_sleep=0.1, at least 20 seconds are just sleep between chunks.

32-3-2 Comparing and choosing the authoritative copy#

When the maps have arrived, ScrubBackend::scrub_compare_maps calls select_auth_object for each object in the union of the maps and then compares each shard with the authoritative copy (compare_obj_in_maps and compare_obj_details): size, object_info, snapset, attrs, data_digest and omap_digest.

pseudocode
shards = [primary] + others
for s in shards:
    r = possible_auth_shard(obj, s)          # read/stat/ec/oi/snapset errors
    if r.unusable: record error; continue
    if auth is none or r.oi.version > auth.version
       or (r.oi.version == auth.version and dcount(r.oi) > dcount(auth.oi)):
         auth = s
    if digests of s and auth differ: digest_match = false
dcount(oi) = [data_digest present] + [omap_digest present]

Numerical example. The object rbd_data.abc.0000000000000012 on three shards. The primary (osd.4): oi.version = 812'5530 and both digests recorded. osd.9: the same version, only data_digest recorded. osd.11: a read error (EIO from BlueStore because of a checksum mismatch). The result: osd.11 is excluded and marked with read_error in shard_info. Between osd.4 and osd.9 the version is equal, but dcount(osd.4)=2 > dcount(osd.9)=1, so osd.4 is authoritative. If osd.9's data digest differs from osd.4's, a data_digest_mismatch inconsistency is also recorded for osd.9. The output of rados list-inconsistent-obj shows this same structure.

32-4 recovery and backfill#

32-4-1 From peering to reservation#

After peering (chapter 17), the primary knows which shards have missing objects (peer_missing) and which shards need the whole content (backfill_targets, because their log does not overlap the authoritative log). In Active, the PG enters the sub-machine shown in the figure below.

32-4-2 Reservation priority#

pseudocode
backfill:
  if FORCED_BACKFILL: return 254
  below = num_acting_below_min_size(pool, acting)
  if below:            base = 220; p = 220 + below
  elif undersized:     base = 140; p = 140 + (size − |actingset|)
  elif degraded:       base = p = 140
  else:                base = p = 100
recovery:
  if FORCED_RECOVERY: return 255
  base = p = 180;  if degraded and below(avail_no_missing): base = 220; p = 220 + below
clamp(p, pool_prio, max_for_base):
  pool_prio = clamp(pool_prio, −10, +10) + 10      # shifted to 0..20
  return clamp(p + pool_prio, 0, max_for_base)

Numerical example. Consider a replicated pool with size=3, min_size=2 and the default recovery_priority of zero:

The subtle point is that even with recovery_priority=0, the +10 shift is always applied, because the range −10 to +10 is moved to 0 to 20. The ceilings of each base guarantee that even the highest pool priority cannot push a normal backfill (at most 139) ahead of a backfill with missing replicas (at least 140).

32-4-3 AsyncReserver: reservation with preemption#

Each OSD has two reservers: local_reserver (for PGs whose primary is this OSD) and remote_reserver (for PGs for which this OSD is a target). The limit of both is osd_max_backfills (default 1). With the mClock scheduler, changing this option only has an effect when osd_mclock_override_recovery_settings=true (OSD::maybe_override_options_for_qos). Otherwise the change is removed from the configuration database.

pseudocode
queues: map<prio, list<Reservation>>; in_progress; preempt_by_prio: set<(prio, item)>
do_queues():
  while preempt_by_prio not empty and (|in_progress| > max_allowed or min(preempt_by_prio).prio < min_priority):
      preempt_one()                                   # the lowest-priority preemptible one
  while queues not empty:
      (prio, list) = highest
      if prio < min_priority: break
      if |in_progress| ≥ max_allowed and preempt_by_prio not empty and prio > min(preempt_by_prio).prio:
          preempt_one()
      if |in_progress| ≥ max_allowed: break
      grant(list.pop_front()); if preemptible: preempt_by_prio.insert(prio, item)

Numerical example. osd.7 is the backfill target for a PG with priority 151 and has taken the only slot of remote_reserver (max=1). Another PG that has only one healthy shard (priority 231) asks osd.7 for a reservation. In do_queues, the highest queue is 231, in_progress is full and 231 > 151, so preempt_one takes back the 151 reservation. The primary of that PG receives a REVOKE message and goes back to WaitLocalBackfillReserved. The 231 reservation is granted immediately. This is the mechanism that lets inactive PGs recover without waiting for long backfills.

32-4-4 Log-based recovery and async recovery#

In recovery, the primary has each shard's missing list and recovers the objects in version order: first it pulls its own missing objects (recover_primary), then it pushes to the replicas (recover_replicas). The number of concurrent operations is limited by osd_recovery_max_active_hdd=3 and osd_recovery_max_active_ssd=10. Each push carries at most osd_recovery_max_chunk=8M of data. In async recovery, a shard that is far behind is left out of acting so that writes do not wait for it, and it is recovered in the background.

pseudocode
for osd in want (up and not stray):
    cost = |auth.last_update.version − osd.last_update.version| + osd.stats.num_objects_missing
    if cost > osd_async_recovery_min_cost (100): candidates.add(cost, osd)
for (cost, osd) in candidates from the highest cost:
    w = want − {osd}
    if acting_set_writeable(pool, w): want = w; async_recovery.add(osd)

Numerical example. want=[1, 2, 3], size=3, min_size=2. The authoritative last_update.version is 5000. osd.2 is at version 4990 and has no missing objects, so its cost is 10 and it is not a candidate. osd.3 is at version 4800 with 30 historically missing objects, so its cost is 230 > 100. [1, 2] is still writeable, so osd.3 becomes an async recovery target and acting is [1, 2]. Client writes are acknowledged with two copies, and after recovery finishes osd.3 returns to acting. If osd.2 also had a cost above 100, removing the second one would give [1], which is below min_size, so it would not be done.

32-4-5 backfill: merging two sorted lists#

pseudocode
while ops < max:
    refill backfill_info / peer intervals if exhausted (MOSDPGScan for replicas)   # one op
    check = min(backfill_info.begin, min over targets of pbi.begin)
    if check < backfill_info.begin:                 # only on the target(s)
        for t with pbi.begin == check: to_remove.add(check, t); pbi.pop_front()
    else:                                           # a primary object
        for t in targets:
            if pbi.begin == check: (version differs ? need_ver : keep_ver).add(t)
            elif check > peer_info[t].last_backfill: missing.add(t)
        if need_ver or missing: prep_backfill_object_push(check, need_ver ∪ missing); ops++
        pop check from backfill_info and matching pbi
    last_backfill_started = check
send MOSDPGBackfill(progress, last_backfill) periodically

Time estimate. Consider a PG with 50 GiB of data in 4 MiB objects (12800 objects) on an HDD target that writes each object push at about 100 MB/s. Each object takes about 42 milliseconds. With one concurrent PG (osd_max_backfills=1), the whole backfill takes about 12800 × 0.042 ≈ 537 seconds, that is about 9 minutes. If osd_recovery_sleep_hdd=0.1 is applied between ops (with the wpq scheduler; with mClock, the scheduler controls the cost), another 12800 × 0.1 = 1280 seconds are added. The total is about 30 minutes. This shows why, with HDDs, backfilling a full OSD with hundreds of PGs takes hours.

32-5 repair#

The key point is that repair has no separate mechanism for copying data. It is the same recovery, fed with an artificial missing list. So all the limits of recovery (reservation, max_active and priority) also apply to repair. If there is no usable shard (for example all have read errors), repair can do nothing and the object stays "unfound". You have to decide with ceph pg PGID mark_unfound_lost revert|delete.

32-6 Commands and debugging#

ceph pg deep-scrub 2.1f                    # an operator request (urgency operator_requested)
ceph tell osd.4 dump_scrubs                # this OSD's scrub queue: targets, not_before and urgency
ceph tell osd.4 dump_scrub_reservations    # local and remote reservations
rados list-inconsistent-obj 2.1f --format=json-pretty
ceph pg 2.1f query | jq '.recovery_state'  # the current state of the peering/recovery machine
ceph tell osd.4 dump_recovery_reservations # local_reserver and remote_reserver
ceph config set osd debug_osd 20/20        # temporary; select_range and do_queues messages

32-7 Supplementary and up to date#

• In Squid the scrub code was reworked: a replica reservation queue, two separate targets for shallow and deep in ScrubJob, and the cv option for deep. The dump_scrubs commands had a different output structure in older versions.
• In main, ScrubBackend has extended the EC stripe check with ec_stripe_mismatches and find_corrupt_zone_copies, which relates to FastEC (chapter 27). Check your own version before relying on the details of the EC inconsistency output.
• Crimson (chapter 30) implements backfill with a separate state machine (BackfillState in src/crimson/osd/backfill_state.cc). The merge logic is the same.

32-8 Hands-on exercise#

1. 1. With vstart, create a pool with size=3, write an object and damage it on one replica with ceph-objectstore-tool. Look at the deep scrub and the list-inconsistent-obj output and explain why that shard was not chosen.
2. 2. Set osd_scrub_sleep=0.5 and predict and then measure the deep-scrub time of a PG with a thousand objects.
3. 3. With ceph osd out on one OSD, read the backfill priority of the undersized PGs from pg query and compare it with this chapter's table.
4. 4. With debug_osd=20, find the do_queues and preempt_one messages while an inactive PG and a normal backfill run at the same time.

32-9 Review questions#

1. 1. In what range is the next shallow and deep scrub time with the defaults, and why does the deep distribution have peaks at the edges?
2. 2. After exactly how long is the PG_NOT_DEEP_SCRUBBED warning raised?
3. 3. Why must a scrub chunk not separate a head from its clones?
4. 4. After three preemptions in a deep scrub, what is the maximum chunk size?
5. 5. In select_auth_object, if the versions are equal, what criterion decides the winner?
6. 6. What is the backfill priority of an undersized PG with size=3 and two acting members?
7. 7. Why does the priority rise by 10 units even with recovery_priority=0?
8. 8. Under what condition does AsyncReserver take back an in-progress reservation?
9. 9. Which target does async recovery leave out of acting, and what condition limits it?
10. 10. Why does repair have the same limits as recovery?

Answers to these questions are in Appendix E: Answers to Review Questions. Answers to this chapter's questions →

32-10 Sources for this chapter#

• scrub in the Ceph documentation
  https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#scrubbing
• scrub developer documentation
  https://docs.ceph.com/en/latest/dev/osd_internals/scrub/
• backfill reservation and priority
  https://docs.ceph.com/en/latest/dev/osd_internals/backfill_reservation/
• recovery in the developer documentation
  https://docs.ceph.com/en/latest/dev/osd_internals/recovery_reservation/
• scrubber code
  https://github.com/ceph/ceph/tree/main/src/osd/scrubber
• PeeringState
  https://github.com/ceph/ceph/blob/main/src/osd/PeeringState.cc
• AsyncReserver
  https://github.com/ceph/ceph/blob/main/src/common/AsyncReserver.h
• PrimaryLogPG
  https://github.com/ceph/ceph/blob/main/src/osd/PrimaryLogPG.cc
• OSD options
  https://github.com/ceph/ceph/blob/main/src/common/options/osd.yaml.in
• Troubleshooting PGs and repairing inconsistencies
  https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/