The Comprehensive Ceph Guide
فارسیGet started
Interactive lab

Peering and recovery stepper

In the book: Chapter 35

Lab

Peering and recovery stepper

Step through three stories of one PG with three replicas and watch the PeeringState machine, the up and acting sets and the PG logs.

pg 1.7 · peering · epoch 45
up[0,1,2]
acting[0,1,2]

6/8 · merge_log rewinds the divergent entry

merge_log sees that 40'4 is not in the authoritative history: rewind_divergent_log drops it and _merge_divergent_entries decides the object's fate. D did not exist before, so it is removed; E and B become missing on osd.0.

Follows: PGLog::merge_log, rewind_divergent_log, _merge_divergent_entries

  1. Reset
  2. Started
  3. Primary
  4. Peering
  5. GetInfo
  6. GetLog
  7. GetMissing
  8. WaitUpThru
  9. WaitActingChange
  10. Active
  11. Activating
  12. WaitLocalRecoveryReserved
  13. WaitRemoteRecoveryReserved
  14. Recovering
  15. WaitLocalBackfillReserved
  16. WaitRemoteBackfillReserved
  17. Backfilling
  18. Recovered
  19. Clean
  20. ReplicaActive
  21. Stray
  22. ToDelete

osd.0 primary

last_update 41'5
  1. 5'1 A
  2. 5'2 B
  3. 5'3 C
  4. 41'4 E to recover
  5. 41'5 B to recover
missing: {E, B}
removed by the log merge: D

osd.1 replica

last_update 41'5
  1. 5'1 A
  2. 5'2 B
  3. 5'3 C
  4. 41'4 E
  5. 41'5 B

osd.2 replica

last_update 41'5
  1. 5'1 A
  2. 5'2 B
  3. 5'3 C
  4. 41'4 E
  5. 41'5 B
What happened so far
  1. e40 The primary logs a write
  2. e41 osd.0 is gone; osd.1 leads
  3. e41 New writes in the new interval
  4. e45 osd.0 returns as primary
  5. e45 The log of osd.1 wins
  6. e45 merge_log rewinds the divergent entry

Follows:

  • src/osd/PeeringState.h :: struct Started, Primary, Peering, GetInfo, GetLog, GetMissing, Active, … the state machine and its nesting
  • src/osd/PeeringState.cc :: start_peering_interval, should_restart_peering, build_prior when peering restarts and which past OSDs to ask
  • src/osd/PeeringState.cc :: find_best_info, choose_acting, calc_replicated_acting the authoritative log and the acting set
  • src/osd/PeeringState.cc :: proc_master_log, proc_replica_log, activate merging logs and activating the replicas
  • src/osd/PGLog.cc :: merge_log, rewind_divergent_log, _merge_divergent_entries, proc_replica_log merging and rewinding divergent entries
  • src/osd/PrimaryLogPG.cc :: start_recovery_ops, recover_primary, recover_replicas, recover_backfill log-based recovery and backfill
A fixed snapshot of the simulator; the interactive version is in the web edition's lab. Interactive lab: Peering and recovery stepper