Day-to-Day Operations: Upgrades, Disk Replacement, Autoscaler and Balancer

19-1 Before you start#

19-1-1 What questions does this chapter answer?#

Most of a Ceph cluster's life consists of routine work: upgrading versions, replacing disks, adding and removing hosts, adjusting the number of PGs and balancing data. Each of these tasks is uneventful if done right, and turns into an incident if not. This chapter reviews today's procedures with cephadm and then goes inside two important mgr modules: pg_autoscaler and balancer.

• In what order does a cephadm upgrade proceed, and what is a staggered upgrade?
• How do we replace a failed disk while keeping the OSD id?
• How do we take a host out for maintenance, or remove it permanently?
• With what formula does pg_autoscaler compute the number of PGs, and what changed in Umbrella?
• In upmap mode, how does the balancer decide which PG to move?
• Which flags and settings are needed for maintenance work?

19-2 Upgrading#

cephadm performs the upgrade as a rolling upgrade: first the mgrs (so that the new cephadm module carries on with the work), then the MONs one by one while keeping quorum, then the OSDs, checking that stopping them does not make PGs inactive, and finally the other services.

ceph -s
ceph versions
ceph orch upgrade start --ceph-version 20.2.5          # or --image quay.io/ceph/ceph:v20.2.5
ceph orch upgrade status
ceph -W cephadm                                        # live follow-up
ceph orch upgrade pause | resume | stop

19-2-1 Staggered upgrades#

cephadm can upgrade a subset of daemons (a staggered upgrade); for example only mgr and mon on a few hosts, or only specific RGW services with a limit on the count:

ceph orch upgrade start --image quay.io/ceph/ceph:v20.2.5 --daemon-types mgr,mon --hosts mon1,mon2
ceph orch upgrade start --image quay.io/ceph/ceph:v20.2.5 --services rgw.s3main --limit 2

19-3 Replacing a disk#

ceph osd tree | grep -E "down|destroyed"
ceph orch osd rm 17 --replace          # the id is kept and the OSD stays in CRUSH
ceph orch osd rm status
# after replacing the disk
ceph orch device ls stor-b3 --refresh
ceph orch device zap stor-b3 /dev/sdk --force      # only if the disk was used before
ceph orch ps --daemon-type osd | grep osd.17

19-4 Host maintenance and removal#

ceph orch host maintenance enter stor-b3        # stop all Ceph daemons on the host
# ... kernel update, hardware repair, reboot ...
ceph orch host maintenance exit stor-b3
# permanent removal
ceph orch host drain stor-b3                    # the _no_schedule and _no_conf_keyring labels; daemons leave
ceph orch osd rm status
ceph orch host rm stor-b3

19-5 Inside the code: pg_autoscaler#

19-5-1 The problem#

The number of PGs in each pool has two opposing effects: few PGs mean an uneven data distribution and little parallelism; many PGs mean more memory and CPU use in the OSDs and heavier peering. The pg_autoscaler module in the mgr (src/pybind/mgr/pg_autoscaler/module.py) automates this balance. Its goal is for each OSD to host about mon_target_pg_per_osd PG replicas.

19-5-2 The algorithm#

pseudocode: pg_autoscaler (this book's rewrite)
  for root in crush_roots:
      budget[root] = osds_in(root) * mon_target_pg_per_osd
  for pool:
      root      = root_of(pool.crush_rule)
      ratio     = max(actual_used / root_capacity,
                      target_size_bytes / root_capacity,
                      normalized target_size_ratio)          # ratios are normalized
      pg_target = ratio * budget[root] * pool.bias / pool.size
      if pool.bulk:     start near full share
      else:             start small, grow with usage
      pg_final  = round_to_power_of_two(pg_target)            # Umbrella: optimal (knapsack) rounding
      pg_final  = clamp(pg_final, pg_num_min, pg_num_max)
      threshold = dynamic_threshold(pg_final) if scaling up else threshold   # default 3.0
      if pg_final > pg_num_target * threshold or pg_final < pg_num_target / threshold:
          if pool.mode == 'on':   set pg_num_target = pg_final
          elif pool.mode == 'warn': raise POOL_TOO_FEW_PGS / POOL_TOO_MANY_PGS

The threshold prevents oscillation: the autoscaler only changes pg_num when the ideal value is at least three times (default threshold=3.0) larger or smaller than the current value. The minimum allowed threshold is 1.0 (ceph osd pool set threshold).

In current code (and Umbrella), the threshold for increases is dynamic at a few common sizes:

# from ceph/ceph, LGPL — src/pybind/mgr/pg_autoscaler/module.py (excerpt)
def get_dynamic_threshold(self, final_pg_num: int, default_threshold: float) -> float:
    if final_pg_num in (512, 1024):
        return 1.0
    elif final_pg_num == 2048:
        return 2.0
    return default_threshold

That is, reaching 512 or 1024 PGs does not require a threefold difference (any increase is enough), and for 2048 a twofold difference is enough. For decreases, the default threshold applies. In Umbrella, rounding pg_target to a power of 2 is also no longer done separately for each pool: the function _find_optimal_pg_distribution uses dynamic-programming knapsack to decide which pools to round up and which down so that the total rounding error is minimized and the root's PG budget is respected (this function does not exist at tag v20.2.0).

ceph osd pool autoscale-status
ceph osd pool set vm-fast target_size_ratio 0.4       # predict the future share
ceph osd pool set archive bulk true                   # a large pool: full share from the start
ceph osd pool set archive pg_num_min 256
ceph osd pool get threshold
ceph osd pool set noautoscale                          # temporary global stop
ceph osd pool unset noautoscale

19-6 Inside the code: balancer#

19-6-1 Modes#

19-6-2 The upmap algorithm#

Module.do_upmap() in src/pybind/mgr/balancer/module.py, with the options upmap_max_optimizations (default 10, the maximum number of changes per round) and upmap_max_deviation (default 5 PGs), hands the work to OSDMap::calc_pg_upmaps() in C++. Its logic:

pseudocode: calc_pg_upmaps (this book's summary of OSDMap.cc)
  for osd: target[osd] = total_pgs * weight[osd] / total_weight      # weighted share
           deviation[osd] = pgs[osd] - target[osd]
  repeat up to max_optimizations:
      over  = OSDs with deviation >  max_deviation (fullest first)
      under = OSDs with deviation < -max_deviation
      if none: break
      for pg on over[0]:
          candidate = an OSD in under that keeps CRUSH rule valid
                      (same failure domain, no repeats in the up set)
          if found: add pg_upmap_items[pg] += (over[0] → candidate); update counts; break
      also: remove existing upmap items that no longer help
  return pending increment

ceph balancer status
ceph balancer mode upmap
ceph balancer on
ceph config set mgr mgr/balancer/upmap_max_deviation 1     # a more precise distribution in a large cluster
ceph balancer eval
ceph osd df tree | sort -nk17 | tail                      # the VAR and PGS columns

19-7 Three maintenance algorithms in numbers#

The previous sections explained the autoscaler and the balancer at the conceptual level. This section opens both down to the level of loops and numbers. Because the autoscaler's rounding algorithm changed in Umbrella, we give both versions separately.

19-7-1 Rounding to a power of two and the 3x threshold#

pseudocode
budget[root] = Σ_osd mon_target_pg_per_osd / (number of roots of that OSD) ; pg_left = budget
pass 1 (non-bulk pools): used = capacity_ratio × pg_left ; pg_left −= used ; target = used / size × bias
pass 2 (bulk with ratio > even = 1/(pools − used_pools)): ratio = max(used, even) ; same calculation
pass 3 (remaining bulk): ratio = 1/(remaining) ; target = ratio × pg_left / size × bias
final = max(pg_num_min=32, nearest_power_of_two(target)) ; final = min(final, pg_num_max)
adjust = (final > current × 3) or (final < current / 3)       # default threshold 3.0

Numerical example. Ten OSDs in one root with mon_target_pg_per_osd = 100 (the v20.2.0 default) give a budget of 1000 PG replicas. A non-bulk pool named rbd with size 3 and capacity_ratio = 0.3 has used = 300 and target = 300/3 = 100. The function nearest_power_of_two(100) builds the bounds 64 and 128 (by ORing shifts: 99 → 127 → 128). Because 128 − 100 = 28 < 100 − 64 = 36, the result is 128. If the current pg_num is 32, 128 > 96 and the change is applied. If the current pg_num were 64, 128 < 192, and the autoscaler would only warn and change nothing. This hysteresis prevents back-to-back splits and merges. The rounding decision point is 1.5 times each power of two.

19-7-2 Global rounding with knapsack (Umbrella)#

pseudocode
for each pool group (same target, same size, same bias): down = pow2_down(t)×size , up = pow2_up(t)×size
    cost_down = |down − t×size| , cost_up = |up − t×size| , extra = up − down
base = Σ down ; budget = pg_left − base
parent[0] = {0: (Σ cost_down)}
for group i: for each sum s in parent[i]:
    s' = s + extra_i ; c' = c − cost_down_i + cost_up_i
    if s' ≤ budget and (s' is new or c' is lower): parent[i+1][s'] = (c', s)
opt = lowest cost; on a tie, the most PGs ; then backtrack to find the "up" groups
dynamic threshold for increases: final ∈ {512, 1024} ⇒ 1.0 , final = 2048 ⇒ 2.0 , otherwise threshold

Numerical example. pg_left = 1000. Pool A has a target of 100 with size 3, and pool B a target of 180 with size 3.

• A: down = 64×3 = 192, up = 128×3 = 384, costs 108 and 84, extra = 192.
• B: down = 128×3 = 384, up = 256×3 = 768, costs 156 and 228, extra = 384.

base = 576 and budget = 424. There are four options:

• Both down: cost 264.
• Only B up: 336.
• Only A up: 240.
• Both up: 576 extra PGs, more than the budget.

So A = 128 and B = 128 are chosen. In this example the result is the same as local rounding. The difference appears when local rounding would round both up and break the root's budget. The three-pass version could not see that. The dynamic threshold also allows a change from 256 to 512 without the 3x condition.

19-7-3 Deviation and the balance score in upmap#

pseudocode
pgs_per_weight = total_pg_replicas / Σ weight
for osd: target = weight × pgs_per_weight ; dev = pgs − target
         bal_score += (dev/target)^2 × weight ; max_dev = max(|dev|)
if max_dev ≤ max_deviation (5): return 0
loop (max times): overfull = {dev > 5} , underfull = {dev < −5}
    if both empty: break ; if overfull is empty: use more_overfull (0 < dev ≤ 5)
    from the fullest OSD: find a PG whose move to underfull does not break the CRUSH rule ⇒ pg_upmap_items
    recompute the deviations ; accept only if bal_score improved

Numerical example. We have six OSDs with weights 1, 1, 1, 1, 2 and 2 (total 8) and 384 PG replicas (128 PGs × size 3), so pgs_per_weight = 48. The weight-1 OSDs have a target of 48 and the weight-2 ones 96. The current counts are 55, 41, 50, 46, 100 and 92, so the deviations are +7, −7, +2, −2, +4 and −4.

• bal_score = $2 \times (7/48)^2 + 2 \times (2/48)^2 + 4 \times (4/96)^2 = 0.0530$.
• First move, one PG from osd.0 to osd.1: deviation ±6 and score 0.0417.
• Second move: ±5 and score 0.0321.

Now no OSD deviates by more than 5, and the loop stops. In practice, the choice of PG is constrained by the CRUSH rule. If osd.0 and osd.1 are on the same host and the rule wants one copy per host, a direct move is impossible and the algorithm turns to another underloaded OSD. The code deliberately accumulates bal_score in a double, because in a float the change from one PG in large clusters fell below the number's precision and the balancer stalled.

19-8 The balancer and splits on a real mapping#

The two experiments in this section were run on the same 120-OSD map (10 hosts × 12, straw2, chooseleaf host, size = 3) that we built in chapter 1. The tools were crushtool and osdmaptool from Ceph 18.2.7.

19-8-1 The upmap balancer on 1024 and 8192 PGs#

pseudocode
for each OSD: target = pg_num × size × (OSD weight / total pool weight)
deviation(o) = PGs on o − target(o)
while max |deviation| > upmap_max_deviation and the number of changes < upmap_max:
    o_over = the largest positive deviation ; o_under = the largest negative deviation, with the same allowed failure domain
    take a PG from o_over that does not violate the CRUSH rule and move it to o_under
    ← a pg-upmap-items entry: pg  o_over  o_under

Measurement. The command osdmaptool om --mark-up-in --upmap out.txt --upmap-deviation 1 --upmap-max 10000 --upmap-pool rbd --save was run on two osdmaps.

With 8192 PGs, CRUSH alone gives each OSD between 172 and 234 PGs (a standard deviation of 13.24 against a mean of 204.8). After 618 pg-upmap-items entries (630 OSD pairs, that is 2.6% of the replicas), the range narrowed to 204 to 206 and the standard deviation to 0.54.

With 1024 PGs, the range went from 13 to 42 to 25 to 26. This needed 226 entries and 246 pairs, which is 8.0% of the replicas. That is, the share of movement needed to even things out is larger at a low pg_num.

Because CRUSH is more uniform at a high pg_num, the balancer also has less work. But in both cases removing the deviation moves a few percent of the data, and the balancer spreads it out in instalments under the same 5% cap (target_max_misplaced_ratio).

19-8-2 Doubling pg_num: half the data moves#

pseudocode
if all of the pool's PGs are active and pg_num − pgp_num < 128:
    pg_num += min(128 − (pg_num − pgp_num), pg_num_target − pg_num)      # split, still no movement
target = min(pg_num_pending, pgp_num_target)
if inactive or degraded or unknown > 0  or  misplaced > 0.05: wait
if the pool is nearly empty (objects ≤ pgp_num_target): pgp_num = target in one go
otherwise: room = min(0.05 − misplaced, 0.025) ; estmax = max(⌊pg_num × room⌋, 1)
       pgp_num = clamp(target, pgp_num − estmax, pgp_num + estmax)       # the real movement is here

Measurement. For each doubling, the mapping of child PG ps + n was compared with the mapping of parent PG ps in the previous osdmap (pgp_num equal to pg_num). In every step from 256 to 512 up to 32768 to 65536, about 92% of the children landed on three new OSDs and the rest almost all on two new OSDs. In total, 48.6 to 49.2 percent of all data moved. No parent PG changed its location.

This number agrees with a simple estimate. Half the data goes to the children. The probability that a child replica happens to land on one of the parent's OSDs is about 3/120. So 0.5 × (1 − 3/120) ≈ 48.75%.

Until pgp_num is raised, the child is mapped with ceph_stable_mod(ps, pgp_num, mask) to the same place as the parent, and the split is metadata only. The real movement happens when pgp_num goes up.

The controller model shows that the mgr spreads this movement in steps of about 2.5% of pg_num. A simulation of 1024 to 2048, assuming backfill finishes before each round, gives 27 steps, the first of which is 1024 to 1052. So increasing pg_num on a full pool is equivalent to moving half of the pool's data, and depending on the volume takes from a few hours to a few days.

19-9 Configuration management#

Since Mimic, the central configuration is kept in the MON database, and ceph.conf on the hosts is only needed to find the MONs. This means configuration changes are traceable, reversible and identical across all daemons.

ceph config dump | head
ceph config get osd.12 osd_memory_target
ceph config show osd.12 | grep -E "osd_memory_target|osd_max_backfills"   # the effective value and its source
ceph config set osd/class:hdd osd_memory_target 6G                       # a mask by device class
ceph config set osd.12 debug_osd 10/10                                   # temporary, for troubleshooting
ceph config rm osd.12 debug_osd
ceph config log 20                                                        # change history
ceph config reset 1234                                                    # revert to a specific version
ceph config help osd_op_queue

19-10 Maintenance flags#

ceph osd set-group noout stor-b3          # only one host's OSDs
ceph osd unset-group noout stor-b3
ceph osd set norebalance && ceph osd unset norebalance

19-11 Periodic tasks#

19-12 Supplementary and up to date#

• OSD upgrades by CRUSH bucket and the ceph osd ok-to-upgrade command appear in the Umbrella documentation.
• In Umbrella, the default mon_target_pg_per_osd has reached 200 and PG rounding is done with global optimization.
• ceph-deploy has been archived and all procedures are done with cephadm; ceph-ansible is still maintained "for now", and migrating to cephadm is recommended (chapter 11).
• The read balancer has been available since Reef for balancing primaries.

19-13 Hands-on exercise#

1. 1. Perform a point upgrade in a test cluster and record the daemon upgrade order from ceph -W cephadm.
2. 2. Remove and recreate an OSD with --replace; compare the amount of backfilled data with a removal without --replace.
3. 3. Put a host into maintenance mode and look at the group noout flag in ceph osd dump.
4. 4. Compare the output of ceph osd pool autoscale-status before and after setting bulk and target_size_ratio.
5. 5. Lower upmap_max_deviation to 1 and follow the change in the PGS column of ceph osd df.
6. 6. In pg_autoscaler/module.py, find where pg_num_min and pg_num_max are applied.

19-14 Review questions#

1. 1. What is the daemon upgrade order in cephadm, and why is the mgr first?
2. 2. What is the advantage of upgrading OSDs by CRUSH bucket?
3. 3. Why does --replace reduce data movement?
4. 4. What does maintenance enter do with the noout flag?
5. 5. What is the basic pg_target formula in the autoscaler?
6. 6. What problem does the autoscaler threshold solve, and when is the dynamic threshold applied?
7. 7. What are the consequences of the change in the default mon_target_pg_per_osd in Umbrella?
8. 8. How does the upmap balancer choose a PG to move?
9. 9. Why is upmap safer than changing weights by hand?
10. 10. What is the difference between ceph config set and injectargs?

Answers to these questions are in Appendix E: Answers to Review Questions. Answers to this chapter's questions →

19-15 Sources for this chapter#

• cephadm upgrade
  https://docs.ceph.com/en/latest/cephadm/upgrade/
• cephadm upgrade (Umbrella
  https://docs.ceph.com/en/umbrella/cephadm/upgrade/
• cephadm host management
  https://docs.ceph.com/en/latest/cephadm/host-management/
• cephadm OSD service
  https://docs.ceph.com/en/latest/cephadm/services/osd/
• Placement groups and autoscaling
  https://docs.ceph.com/en/latest/rados/operations/placement-groups/
• Balancer
  https://docs.ceph.com/en/latest/rados/operations/balancer/
• Configuring Ceph (central config
  https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/
• Source: src/mon/ConfigMonitor.cc (config log / reset
  https://github.com/ceph/ceph/blob/main/src/mon/ConfigMonitor.cc
• Device management
  https://docs.ceph.com/en/latest/rados/operations/devices/
• Ceph releases
  https://docs.ceph.com/en/latest/releases/
• Source: src/pybind/mgr/pg_autoscaler/module.py
  https://github.com/ceph/ceph/blob/main/src/pybind/mgr/pg_autoscaler/module.py
• Source: src/pybind/mgr/balancer/module.py
  https://github.com/ceph/ceph/blob/main/src/pybind/mgr/balancer/module.py
• Source: src/osd/OSDMap.cc (calc_pg_upmaps
  https://github.com/ceph/ceph/blob/main/src/osd/OSDMap.cc