The Comprehensive Ceph Guide
فارسیGet started

Contents

Every part, chapter, appendix and list of the book, with the approximate reading time of each.

Front matter

  1. Preface8 min

Chapters

  1. 1Ceph: A New Era — Architecture, Planning and Your First Cluster1-1 Before you start · 1-2 Ceph: the beginning of a new era · 1-3 The end of the RAID era · 1-4 The Ceph architecture at a glance · 1-5 Planning a Ceph deployment · 1-6 Building a virtual lab · 1-7 Installing and configu49 min
  2. 2Working with the Ceph Block Device (RBD)2-1 Before you start · 2-2 RBD from the inside · 2-3 Configuring a Ceph client · 2-4 Creating, mapping and resizing · 2-5 Snapshots and clones · 2-6 Disaster recovery with RBD mirroring · 2-7 From the code: RBD object la34 min
  3. 3Ceph and OpenStack3-1 Before you start · 3-2 Ceph: OpenStack's best companion · 3-3 Setting up a test OpenStack · 3-4 Configuring OpenStack as a Ceph client · 3-5 Configuring Glance with a Ceph backend · 3-6 Configuring Cinder · 3-7 Confi37 min
  4. 4Object Storage with the RADOS Gateway4-1 Before you start · 4-2 Getting to know object storage in Ceph · 4-3 Deploying and configuring the RADOS Gateway · 4-4 Users, keys and tenants · 4-5 Access with the S3-compatible API · 4-6 Access with the Swift API · 36 min
  5. 5Multi-Site Object Storage with RGW Multi-Site5-1 Before you start · 5-2 Multisite concepts · 5-3 Multisite prerequisites · 5-4 Configuring multisite by hand · 5-5 Automated configuration with the rgw module · 5-6 Checking the sync status · 5-7 Testing the sync of u31 min
  6. 6The Ceph File System (CephFS)6-1 Before you start · 6-2 CephFS and the MDS · 6-3 Client access · 6-4 Exporting with NFS and SMB · 6-5 From the code: capabilities, the journal and dirfrags in the MDS · 6-6 CephFS by the numbers: object names, dirfrag30 min
  7. 7Operating and Managing a Ceph Cluster7-1 Before you start · 7-2 Managing services · 7-3 Configuration · 7-4 Growing and shrinking the cluster · 7-5 Replacing a failed disk · 7-6 Upgrading · 7-7 Capacity and balance · 7-8 Inside the code: MON, Paxos and the 31 min
  8. 8Ceph Under the Hood: CRUSH, Maps, Authentication and PGs8-1 Before you start · 8-2 Scaling without a central table · 8-3 The CRUSH mechanism · 8-4 Tunables · 8-5 Cluster maps and MONs · 8-6 Inside the code: placement, authentication and messaging · 8-7 Data movement by the nu35 min
  9. 9From VSM to the Ceph Dashboard: Graphical Cluster Management9-1 Before you start · 9-2 History: from separate tools to the Dashboard · 9-3 Ceph Dashboard architecture · 9-4 Enabling and security · 9-5 Users, roles and SSO · 9-6 Tentacle: mgmt-gateway and oauth2-proxy · 9-7 Day-to28 min
  10. 10Benchmarking and Expert Tools for Ceph10-1 Before you start · 10-2 Layered measurement · 10-3 Hardware baseline · 10-4 Measuring RADOS · 10-5 Measuring RBD · 10-6 The operator's toolbox · 10-7 Measuring RGW and CephFS · 10-8 Benchmark report · 10-9 Inside th28 min
  11. 11Deploying Ceph: Lab, cephadm, Ansible and Rook11-1 Before you start · 11-2 A reproducible lab · 11-3 The evolution of deployment tools · 11-4 Declarative deployment with cephadm · 11-5 Ansible alongside cephadm · 11-6 From lab to production · 11-7 Inside the code: t33 min
  12. 12BlueStore: The OSD Storage Engine12-1 Before you start · 12-2 Why BlueStore? · 12-3 How does BlueStore work? · 12-4 Using BlueStore · 12-5 Migrating and maintaining OSDs · 12-6 BlueStore by the numbers: the compression acceptance test, the effect of min27 min
  13. 13Erasure Coding for Storage Efficiency13-1 Before you start · 13-2 What is EC? · 13-3 EC inside Ceph · 13-4 Plugins and profiles · 13-5 Where EC and where replication? · 13-6 FastEC in Tentacle · 13-7 Troubleshooting: the 2147483647 error · 13-8 Inside the E27 min
  14. 14Programming with librados14-1 Before you start · 14-2 What is librados? · 14-3 Getting started · 14-4 Atomic operations · 14-5 watch and notify · 14-6 Beneath librados: the MOSDOp message, sessions and backoff · 14-7 backoff: the OSD says "wait"26 min
  15. 15RADOS Classes: Computing Next to the Data15-1 Before you start · 15-2 Concept: computing next to the data · 15-3 Calling a class from the client · 15-4 Writing our own class · 15-5 cls_lua: scripts inside the OSD · 15-6 Three algorithms from the class path · 1529 min
  16. 16Storage Tiering: From Cache Tiering to Device Classes and Storage Classes16-1 Before you start · 16-2 Cache Tiering: architecture · 16-3 Inside the code: the tier agent · 16-4 Migration: safely removing a cache tier · 16-5 Tiering at the byte level: storage classes and lifecycle in RGW · 16-631 min
  17. 17Troubleshooting Ceph: From HEALTH_WARN to the Peering State Machine17-1 Before you start · 17-2 The troubleshooting method · 17-3 MON problems · 17-4 OSD problems · 17-5 PG states · 17-6 Inside the code: the Peering state machine · 17-7 Three troubleshooting algorithms in numbers · 17-826 min
  18. 18Disaster Recovery: rbd-mirror, Stretch Mode and Cluster Rescue18-1 Before you start · 18-2 Layers of defence · 18-3 rbd-mirror · 18-4 Inside the code: rbd-mirror · 18-5 The lower layer: the journal on RADOS and the stretch state machine · 18-6 Disaster recovery in numbers: MON conn27 min
  19. 19Day-to-Day Operations: Upgrades, Disk Replacement, Autoscaler and Balancer19-1 Before you start · 19-2 Upgrading · 19-3 Replacing a disk · 19-4 Host maintenance and removal · 19-5 Inside the code: pg_autoscaler · 19-6 Inside the code: balancer · 19-7 Three maintenance algorithms in numbers · 125 min
  20. 20Monitoring Ceph: Prometheus, ceph-exporter, Alerts and Observability20-1 Before you start · 20-2 Architecture of the monitoring stack · 20-3 From inside the code: perf counters · 20-4 Key signals · 20-5 Designing alert rules · 20-6 The lower layer: from an atomic variable to a /metrics l29 min
  21. 21Ceph Performance Tuning: From mClock to the Kernel and Network21-1 Before you start · 21-2 Method and layers · 21-3 From inside the code: shards and threads · 21-4 From inside the code: mClock · 21-5 OSD memory · 21-6 The lower layer: the memory control loop and throttle in BlueSto28 min
  22. 22Ceph Code Architecture and the Request Path from the Inside22-1 Repository map · 22-2 Build and development environment · 22-3 Main classes and relationships · 22-4 Thread model · 22-5 Encoding and version compatibility · 22-6 Reading and debugging the code in practice · 22-7 Th19 min
  23. Part III: Lower Layers (Chapters 23–33)
  24. 23Lower Layers (1): Block Device, bufferlist, mempool and crc32c23-1 Where this part fits · 23-2 KernelDevice: BlueStore's window onto the disk · 23-3 bufferlist: Ceph's core data structure · 23-4 mempool: memory accounting · 23-5 crc32c: from a CPU instruction to a block checksum · 23 min
  25. 24Lower Layers (2): BlueStore and BlueFS On-Disk Format24-1 Introduction: what does BlueStore write on disk? · 24-2 The device label: bluestore_bdev_label_t · 24-3 Compact encodings: varint, lowz and lba · 24-4 BlueFS: a file system for RocksDB · 24-5 The full map of KV pref29 min
  26. 25Lower Layers (3): RocksDB Inside BlueStore25-1 Introduction: why RocksDB? · 25-2 From a BlueStore transaction to a WriteBatch · 25-3 The WAL: record format · 25-4 memtable and flush · 25-5 SST: file format · 25-6 Compaction and levels · 25-7 Column family shardi25 min
  27. 26Lower Layers (4): msgr2 at the Byte Level, AsyncConnection and EventCenter26-1 Introduction: the wire between two daemons · 26-2 The banner and feature negotiation · 26-3 The frame at the byte level · 26-4 secure mode: AES-128-GCM on the wire · 26-5 Session, reconnection and reset · 26-6 Async22 min
  28. 27Lower Layers (5): Erasure-Coding Math, from GF(2⁸) to FastEC27-1 Introduction: from XOR to Reed-Solomon · 27-2 GF(2⁸): arithmetic on bytes · 27-3 Systematic Reed-Solomon · 27-4 Stripes, chunks and layout on the OSDs · 27-5 read-modify-write in the old EC · 27-6 Codes with lower r27 min
  29. 28Lower Layers (6): CRUSH at the Bit Level, from rjenkins to upmap28-1 Introduction: how does CRUSH "compute"? · 28-2 From the object name to x · 28-3 straw2: an exponential draw · 28-4 crush_do_rule step by step · 28-5 choose_args, weight-set and upmap · 28-6 is_out and reweight: a nu29 min
  30. 29Lower Layers (7): Linux Kernel Clients — libceph, krbd and ceph.ko29-1 Introduction: Ceph inside the kernel · 29-2 libceph: structure · 29-3 krbd: from blk-mq to the object · 29-4 krbd in practice · 29-5 ceph.ko and caps · 29-6 Kernel versus user space · 29-7 A lower layer: from file o25 min
  31. 30Lower Layers (8): Crimson and Seastar — Shard-per-Core OSD and SeaStore30-1 Introduction: why a new OSD? · 30-2 Seastar: future, continuation and reactor · 30-3 The PG-to-core mapping algorithm · 30-4 The operation pipeline · 30-5 Storage backends in Crimson · 30-6 SeaStore: architecture · 20 min
  32. 31Lower Layers (9): SPDK and NVMe — from Hardware Queues to the NVMe-oF Gateway31-1 Why this chapter? · 31-2 NVMe from the driver's point of view · 31-3 NVMEDevice: SPDK in BlueStore, function by function · 31-4 The NVMe-oF gateway · 31-5 Practical commands · 31-6 Supplementary and up to date · 31-21 min
  33. 32Lower Layers (10): Scrub, Recovery, Backfill and Repair at Code Level32-1 Chapter map · 32-2 Scheduling scrub · 32-3 The scrub state machine · 32-4 recovery and backfill · 32-5 repair · 32-6 Commands and debugging · 32-7 Supplementary and up to date · 32-8 Hands-on exercise · 32-9 Review 21 min
  34. 33Lower Layers (11): Memory and CPU Inside the OSD — from mempool to NUMA and Profiling33-1 Why memory and CPU? · 33-2 The threads of a classic OSD · 33-3 mempool: lock-free accounting · 33-4 tcmalloc and "mapped" memory · 33-5 PriorityCache: dividing the memory budget · 33-6 NUMA · 33-7 Profiling · 33-8 m19 min
  35. Part IV: Specialist Part on RADOS, RBD and RGW (Chapters 34–42)
  36. 34Specialist Part (1): RADOS — Object Identity, Metadata and the Op Path Inside the OSD34-1 Before we start · 34-2 The identity of a RADOS object · 34-3 Per-object metadata · 34-4 Op queueing: shard and thread · 34-5 The gates of do_op · 34-6 Executing the op and numbering versions · 34-7 The pg log and pg20 min
  37. 35Specialist Part (2): RADOS — The PG Log, Intervals, Missing Sets, and How Recovery and Backfill Work35-1 Before we begin · 35-2 The info and history of a PG · 35-3 Past intervals and the prior set · 35-4 Merging logs · 35-5 Log length and trimming · 35-6 Recovery and backfill in practice · 35-7 Commands and troubleshoo19 min
  38. 36Specialist Part (3): RADOS — Snapshots and Clones, Watch/Notify, OSD-Side cls, and the rados and pool Reference36-1 Before we begin · 36-2 Two kinds of snapshot · 36-3 SnapSet and clones · 36-4 snaptrim and the SnapMapper · 36-5 Watch and notify in the OSD · 36-6 OSD-side cls calls · 36-7 rados reference · 36-8 ceph osd pool refe17 min
  39. 37Specialist Part (4): RBD — Layout on RADOS, Striping, Object Map and Fast-Diff, and the rbd Reference37-1 Before we begin · 37-2 The objects of an image · 37-3 Striping · 37-4 The object map · 37-5 rbd reference · 37-6 Tuning and troubleshooting · 37-7 Supplementary and up to date · 37-8 Hands-on exercise · 37-9 Review 17 min
  40. 38Specialist Part (5): RBD — The librbd I/O Path, Layering and Copyup, Exclusive Lock, Caches and QoS38-1 Before we begin · 38-2 The dispatch layers · 38-3 Layering: clone, copyup and flatten · 38-4 exclusive-lock · 38-5 Caches · 38-6 QoS · 38-7 krbd, librbd and rbd-nbd · 38-8 Tuning and troubleshooting · 38-9 Supplemen15 min
  41. 39Specialist Part (6): RBD — Inside rbd-mirror: Journal versus Snapshot39-1 Before we begin · 39-2 The journal on RADOS · 39-3 Replay in journal mode · 39-4 Snapshot mode · 39-5 Daemons: leader, instances and dividing the images · 39-6 Command reference · 39-7 Tuning and troubleshooting · 313 min
  42. 40Specialist Part (7): RGW — The Data Model on RADOS: Metadata, Bucket Index, Versioning, Manifest and Multipart40-1 Before we begin · 40-2 Metadata: pools and namespaces · 40-3 The bucket index · 40-4 Versioning and the OLH · 40-5 Manifest, placement and storage classes · 40-6 Multipart · 40-7 radosgw-admin reference · 40-8 Tunin15 min
  43. 41Specialist Part (8): RGW — The Request Path, SigV4, Quotas, Rate Limits, GC and Lifecycle41-1 Before we begin · 41-2 From socket to op · 41-3 The SigV4 signature · 41-4 PUT and GET in motion · 41-5 Quotas · 41-6 Rate limits · 41-7 garbage collection · 41-8 lifecycle · 41-9 Commands with S3 clients · 41-10 Tu14 min
  44. 42Specialist Part (9): RGW — Inside Multisite: Periods, Log Formats, Bucket Generations and the Sync Engine42-1 Before we begin · 42-2 Periods and commit · 42-3 The format of log records · 42-4 Sync status on RADOS · 42-5 The data sync engine · 42-6 Metadata sync · 42-7 Multisite command reference · 42-8 Tuning and troublesho13 min

Appendices

  1. AGlossary10 min
  2. BCommand Cheat Sheet35 min
  3. CTroubleshooting Guide3 min
  4. DRelease Table1 min
  5. EAnswers1 h 13 min
  6. FBibliography11 min
  7. GList of Algorithms14 min
  8. HRelease Changes6 min

Lists and index

  1. List of Figures
  2. Index