The Comprehensive Ceph Guide
فارسیGet started
Interactive lab

CRUSH playground

In the book: Chapter 8, Chapter 28

Lab

CRUSH playground

Build a root, hosts and OSDs, choose a rule, and watch straw2 place every PG; mark an OSD out or down or change a weight and see which PGs move.

64 × 3PGs × replicas
9.9%data moved by the last change
19PGs whose up set changed
0PGs with fewer copies than size
0empty slots (NONE)

Last change: osd.4 marked out. Moved shards are highlighted in the list.

osd.0: 29 (≈24)029osd.1: 14 (≈24)114osd.2: 21 (≈24)221osd.3: 0 (≈0)30osd.4: 38 (≈24)438osd.5: 26 (≈24)526osd.6: 27 (≈24)627osd.7: 14 (≈24)714osd.8: 23 (≈24)823
PG shards on each OSD (bars) against the share its weight would give (line). Red: out; amber: down.
$ ceph osd map pool1 rbd_data.10226b8b4567.0000000000000000ceph_str_hash_rjenkins → 0x87bd7b9a · ceph_stable_mod(·, 64, 63) → pg 1.1a · pps 0x8d2145b9 · up [4,2,8] p4
PG → up set (64 PGs; primary underlined)
  1. 1.0[0,6,5]
  2. 1.1[7,4,0]
  3. 1.2[4,7,0]
  4. 1.3[4,8,2]
  5. 1.4[1,8,5]
  6. 1.5[4,8,0]
  7. 1.6[6,4,2]
  8. 1.7[1,7,5]
  9. 1.8[6,0,4]
  10. 1.9[8,4,2]
  11. 1.a[1,5,7]
  12. 1.b[1,8,5]
  13. 1.c[6,5,2]
  14. 1.d[7,4,2]
  15. 1.e[1,8,4]
  16. 1.f[6,2,4]
  17. 1.10[5,2,8]
  18. 1.11[4,8,0]
  19. 1.12[0,6,5]
  20. 1.13[4,6,1]
  21. 1.14[5,8,1]
  22. 1.15[8,5,2]
  23. 1.16[5,0,6]
  24. 1.17[8,0,5]
  25. 1.18[6,2,4]
  26. 1.19[6,0,4]
  27. 1.1a[4,2,8]
  28. 1.1b[8,1,4]
  29. 1.1c[8,4,1]
  30. 1.1d[6,0,5]
  31. 1.1e[8,4,0]
  32. 1.1f[8,5,2]
  33. 1.20[2,6,5]
  34. 1.21[6,0,4]
  35. 1.22[2,4,6]
  36. 1.23[0,5,7]
  37. 1.24[0,4,6]
  38. 1.25[8,5,0]
  39. 1.26[1,4,6]
  40. 1.27[1,7,5]
  41. 1.28[5,7,0]
  42. 1.29[8,2,4]
  43. 1.2a[5,2,6]
  44. 1.2b[8,0,4]
  45. 1.2c[7,0,4]
  46. 1.2d[2,7,4]
  47. 1.2e[6,2,4]
  48. 1.2f[0,4,6]
  49. 1.30[6,5,0]
  50. 1.31[0,6,4]
  51. 1.32[0,6,4]
  52. 1.33[4,6,2]
  53. 1.34[8,0,5]
  54. 1.35[8,2,4]
  55. 1.36[7,0,4]
  56. 1.37[4,7,0]
  57. 1.38[6,4,2]
  58. 1.39[0,5,6]
  59. 1.3a[0,8,4]
  60. 1.3b[4,1,6]
  61. 1.3c[5,6,0]
  62. 1.3d[5,8,2]
  63. 1.3e[5,1,7]
  64. 1.3f[7,4,1]

Follows:

  • src/crush/mapper.c :: crush_do_rule runs the rule steps (take, chooseleaf, emit)
  • src/crush/mapper.c :: crush_choose_firstn, crush_choose_indep the replicated (firstn) and EC (indep) choice loops with retries
  • src/crush/mapper.c :: bucket_straw2_choose, generate_exponential_distribution, crush_ln straw2: the largest ln(hash)/weight wins
  • src/crush/mapper.c :: is_out the reweight check that rejects an out OSD
  • src/crush/hash.c :: crush_hash32_rjenkins1_2, _3 the Jenkins hash that CRUSH uses
  • src/osd/osd_types.cc :: pg_pool_t::raw_pg_to_pps, hash_key the PG's seed that enters CRUSH
  • src/include/rados.h :: ceph_stable_mod object hash to PG while pg_num is not a power of two
  • src/common/ceph_hash.cc :: ceph_str_hash_rjenkins the object-name hash
  • src/osd/OSDMap.cc :: _pg_to_raw_osds, _raw_to_up_osds raw set to up set (down OSDs are dropped or become NONE)
  • src/crush/CrushWrapper.cc :: add_simple_rule_at, insert_item; CrushWrapper.h :: set_tunables_jewel the rule ceph osd pool create makes and the jewel tunables
A fixed snapshot of the simulator; the interactive version is in the web edition's lab. Interactive lab: CRUSH playground