| 1-1 | Ceph layers: the three block, object and file interfaces are all built on a common RADOS layer |
| 1-2 | Many-to-one rebuild in RAID versus many-to-many recovery in Ceph: in Ceph every healthy OSD takes part in the repair |
| 1-3 | Daemon roles: the client gets the maps from the MONs and then works directly with the OSDs |
| 1-4 | The placement path: the object name maps to a PG, and CRUSH takes that PG to a set of OSDs in different failure domains |
| 1-5 | The planning decision path: workload type, failure domain, network and capacity |
| 1-6 | Lab topology: three nodes to start with, a fourth node for expansion, and a client machine |
| 1-7 | The steps of bootstrapping and adding hosts with cephadm |
| 1-8 | The three layers of an object's path. In the logical layer, the object name is turned into a 32-bit number by the hash function called rjenkins (hobject_t keeps this hash together with the namespace and snap), and ceph_stable_mod maps it to a PG of the pool. In the mapping layer, CRUSH builds the list of OSDs from the placement seed (pps) and that pool's rule; then the exceptions recorded in the OSDMap, namely upmap and pg_temp/primary_temp, are applied to it, giving the up set and the acting set. The first member of acting is the primary. In the physical layer, each PG on each OSD is a collection in BlueStore (for example 3.1a_head). The metadata of each object is an onode in RocksDB, and the data points to raw ranges on the block device through the chain of extent, blob and pextent |
| 1-9 | Key prefixes in the RocksDB inside BlueStore, defined in BlueStore.cc. S is for the superblock, T for statistics, C for collections, O for onodes, M for omap, L for deferred transactions, B for the freelist and X for shared blobs. The bottom row shows the main fields of pg_log_entry_t, which is stored in the omap of each PG's special pgmeta object. These fields are the operation, the object id, the version and the previous version, the reqid used to detect a duplicate request after a client reconnects, and mod_desc, which allows rollback in EC |
| 1-10 | Result of the exact computation on the real CRUSH mapping for 120 OSDs on 10 hosts and a three-replica pool (crushtool and osdmaptool, Ceph 18.2.7). Left: the probability that three simultaneous failures completely destroy at least one PG, as a function of pg_num, together with the upper bound pg_num/C(120,3). Right: the number of OSDs that share a PG with one OSD (mean and range), and the number of PGs that land on the busiest target OSD after osd.0 is marked out |
| 1-11 | Time to restore redundancy after the failure of a full 16 TB disk. Red is RAID with one spare disk and green is Ceph, where the number of PGs on the failed disk and on the busiest target come from the real CRUSH mapping (osdmaptool --mark-out 0). The vertical axis is logarithmic, and only the rate of 0.15 TB/h per disk is assumed |
| 2-1 | The structure of an image in RADOS: a header object for metadata and 4 MB data objects that are created only after they are written |
| 2-2 | Ways of accessing RBD: librbd in user space (QEMU, rbd-nbd, the NVMe-oF gateway) and krbd in the kernel |
| 2-3 | The chain of parent image, snapshot and clone; flatten cuts the clone's dependency on the parent |
| 2-4 | The two mirroring modes: in journal mode every write is recorded and replayed; in snapshot mode the difference between periodic snapshots is transferred |
| 2-5 | Mirroring topology: rbd-mirror in site-b reads the changes from site-a; failback also needs rbd-mirror in site-a |
| 2-6 | The layout of a format 2 image in RADOS. The top row shows the pool-level lists: in the omap of the rbd_directory object, the name_ and id_ keys map names and ids to each other. The middle row is each image's metadata objects: rbd_id.<name> holds the id, and rbd_header.<id> stores the size, order, features, object_prefix, snapshots and parent in its omap; watchers register on this same object. The object map has two state bits for each data object. The bottom row is the data objects, whose names are built from object_prefix and a sixteen-digit hex number; only objects that have actually been written are created (thin provisioning) |
| 2-7 | Handing the exclusive lock over between two clients. B first reads the current owner and, if the lock is held, sends a lock request through watch/notify on the header. If A is healthy, it flushes its cache and releases the lock according to the policy (default: automatic release). If A no longer has a watch, B breaks the lock and, with rbd_blocklist_on_break_lock=true (the default), blocklists A's address so that A's late writes are not accepted. B then takes the lock and, in PostAcquire, opens the object map and the journal |
| 2-8 | The chain of objects of a format 2 image. The first row is building the id from the instance id and a random number. The second row is the name-to-id mapping in rbd_id and rbd_directory, and the image header. The third row is the object map, the data objects created by the first write, and the trash entry on deletion. The sample id is symbolic |
| 2-9 | Five discard requests on three 4 MB objects. The grey bar above each row is the requested range. Red is remove, orange is truncate, blue is zero, and hatching is "skipped". Rounding to 64 KiB causes discards smaller than one full unit to be dropped |
| 2-10 | The discard decision path in librbd: conversion to object extents, pruning with a 64 KiB unit (unless skip_partial_discard is off or there is a dirty cache), and the choice of remove, truncate or zero. For a clone with a parent, a full removal becomes a truncate so that the parent's data does not reappear |
| 2-11 | USED measured on a Ceph 18.2.7 test cluster for four workloads. Grey is the bytes actually written, blue is rbd du --exact, and orange is the default rbd du (counting whole objects from the object map). The vertical axis is logarithmic. In the random workload D, even --exact shows about two hundred times the data written, because it counts each object from byte 0 to the highest byte written |
| 2-12 | The order of steps in a switchover, a forced failover and a failback |
| 3-1 | Mapping OpenStack services to Ceph components: each service has its own pool or interface |
| 3-2 | Distribution of users and keyrings: the control services each have their own user, and nova-compute reaches the client.cinder key through a secret in libvirt |
| 3-3 | End-to-end copy-on-write: both the Cinder volume and the Nova ephemeral disk are clones of the same snapshot in Glance |
| 3-4 | The boot-from-volume path: Cinder creates the volume with a clone, and nova-compute hands it to QEMU through libvirt and librbd |
| 3-5 | The common OpenStack pattern on RBD. The top row shows that Glance stores the image with a snapshot. In clone v1 this snapshot must be protected; in clone v2 (default rbd_default_clone_format=auto, provided min_compat_client is at least mimic) this is not needed, and the parent snapshot can even be deleted while it has children, in which case it goes to the trash. The middle row shows hundreds of volumes created from one snapshot without copying data. The bottom row is the operational consequences: reads of unwritten regions are served from the parent, the first write to each object causes a copyup the size of the whole object, and flatten cuts the dependency but consumes the full space |
| 3-6 | The write path on a cloned image. If the child object exists, the write goes straight through. Otherwise CopyupRequest reads the whole object range from the parent snapshot, copies the data to the child's snapshots as well if deep-flatten and child snapshots exist (deep_copy), updates the object map, and finally sends a single atomic operation to the OSD. This operation contains the cls method rbd.copyup or sparse_copyup, which writes the parent's data only if the object does not exist, followed by the user's own write. The atomicity of this combination guarantees that no reader sees a half-copied object |
| 3-7 | Mapping a 1 MB write at offset 10.5 MiB to RADOS objects in two layouts. Top: stripe_count = 1 (the default), where the whole write lands in object number 2 from offset 0x280000. Bottom: stripe_unit = 64K and stripe_count = 4, where the same write is divided into 16 pieces of 64K and spread in turn across four objects. The buffer piece numbers are written on each object. All pieces are in "object set" zero, because 42 to 45 is less than the 64 stripes per object |
| 3-8 | Object map size as a function of image size, for three object sizes, on logarithmic axes. Each object takes two bits, so the map size is linear in the image size. The vertical dashed lines are the largest image compatible with the object map for each object size, that is where the number of objects reaches 256 million. The black dot is a typical one-terabyte volume with 4 MiB objects, which has only a 64 KiB map |
| 3-9 | The state machine of one object map cell. Cells start as NONEXISTENT. Before the first write they move to EXISTS, because librbd updates the map before the data so that after a client crash an existing object is never marked "nonexistent". When a snapshot is created, every EXISTS cell in the current map (HEAD) becomes EXISTS_CLEAN. Every later write makes the cell EXISTS again. Removal has two steps: first PENDING, then NONEXISTENT after a successful removal |
| 3-10 | The diff state table for six sample objects from snap1 to HEAD. Each cell shows the object's state in that object map (orange EXISTS, blue EXISTS_CLEAN, grey NONEXISTENT), and under it the diff state after processing that map. The last column is the result: red means changed data that must be transferred, purple means a region that must be reported as zero, green and white mean unchanged. o2 shows why HOLE does not change in an intermediate snapshot: the result depends only on the state in the last map |
| 3-11 | Left: the number of IOs accepted per second for a saturated client, in three configurations. Without burst the rate is a constant 100. With burst 300 and burst_seconds 2, the rate stays at 300 for about three seconds and then returns to 100. If the client was idle beforehand for those three seconds, the first second gets 585 IOs. Right: the tokens added per tick for three values of avg. Distributing avg across ticks with integers keeps each second's total exact. These numbers come from a Python re-implementation of TokenBucketThrottle assuming continuous demand, not from measurements on a cluster |
| 4-1 | RGW architecture: stateless instances behind a load balancer, and separate pools for the index, data and metadata |
| 4-2 | A 20 MB S3 object in RADOS: a head object with metadata and the manifest, tail objects for the rest of the data, and an entry in the relevant index shard |
| 4-3 | Two multi-tenancy mechanisms: the tenant for separating bucket namespaces, and the account (since Squid) for IAM-style self-service management of users and policies |
| 4-4 | The Swift authentication flow with Keystone: RGW validates the token with Keystone and automatically creates a user for each project |
| 4-5 | The PUT path of an object. radosgw computes the shard number from the hash of the object name. First the cls method bucket_prepare_op records an entry with state PENDING_MODIFY and a unique tag in the shard. Then the head (containing at most the first 4MiB of data and the manifest, ACL and etag xattrs) and the tail stripes are written. Finally bucket_complete_op moves the entry to the COMPLETE state with the size, etag and mtime and creates a bilog record for multisite sync. If RGW fails between these two phases, the pending entry stays behind and is later fixed by dir_suggest_changes (during list) or radosgw-admin bucket check |
| 4-6 | The physical layout. In the index pool each bucket is divided into N .dir.<marker>.<shard> objects, and the entries are kept in their omap. In the data pool the head is stored with the name <marker>_<key> and the tails with the __shadow_ prefix. In multipart each part is a separate __multipart_ object, and CompleteMultipartUpload only builds the manifest and does not copy data. The tails of deleted or overwritten objects are removed asynchronously from the GC queue (the rgw_gc class) |
| 4-7 | Left: the spread of 100000 consecutive keys across 11 index shards. The dashed line is the mean, and every shard is within about one percent of it. Right: three sample keys step by step from hash to shard number, for both 11 and 23 shards. The XOR step moves the hash's low byte to the high byte, because the remainder modulo a small prime is more sensitive to the low bits |
| 4-8 | Number of index shards as a function of the number of objects in the bucket, on logarithmic axes. Blue is the path of one zone with factor 2, and red is the multisite path with factor 8. Each step is a reshard, and the number next to it is the new shard count (a prime). The dotted line is the cap of 100000 objects per shard. Each time the bucket crosses it, a reshard is queued. The cap of 1999 is reached at different object counts on the two paths |
| 4-9 | The total number of index entries requested to build one page of an ordered listing, as a function of the number of shards, for max-keys of 1000 and 100. The dotted line of the same colour is the actual number of keys on the page. With few shards the over-read is small. From the point where n / s drops below 8, the minimum of 8 dominates, the two curves coincide, and the cost rises linearly with the number of shards. The numbers show only the first attempt, before the factor expands |
| 4-10 | Top: a simple 10 MB PUT. The head (yellow) holds the first four megabytes and the metadata, and two stripes in the shadow namespace (blue) with a 32-character random prefix hold the rest. Bottom: a multipart upload with two 10 MB parts. The head has no data and holds only the manifest. Each part has one object in the multipart namespace (red) for its first 4 megabytes and two stripes in the shadow namespace. The names are taken from get_implicit_location. The marker and upload id are symbolic |
| 4-11 | Placement and storage classes: new data lands in the STANDARD class and, after a while, lifecycle moves it to a cold EC class or to external cloud storage |
| 5-1 | The multisite hierarchy: a realm contains zonegroups, and each zonegroup contains zones, each of which lives in a separate Ceph cluster |
| 5-2 | From federated in Hammer to multisite version 2: sync moved from an external agent into radosgw itself, and all zones became writable |
| 5-3 | Setup steps: create the realm, zonegroup and master zone in the primary cluster, then pull the realm and create the secondary zone in the second cluster |
| 5-4 | The sync mechanism: the destination zone reads the source's datalog and bilog and pulls changed objects with GET from the source endpoint; metadata comes from the metadata log in the master zone |
| 5-5 | The full multisite flow. In the source zone each PUT or DELETE creates a bilog entry in the index shard, and a datalog entry announces that "that bucket shard has changed". The 30-second datalog window merges successive changes to one shard so that the log does not explode. Creating a bucket or user is recorded in the mdlog, which only the master zone produces. In the destination zone three layers of coroutines work: meta sync, data sync for each datalog shard, and bucket sync for each bucket shard, which reads the bilog. The final layer pulls the object from the source with fetch_remote_obj, writes it locally and stores the marker in a status object |
| 5-6 | The data sync state machine. First the status objects are created (Init), then all the source's buckets are listed in the full-sync map (BuildingFullSyncMaps), and after that each datalog shard, independently of the others, runs a full sync first and then an incremental sync with a twenty-second poll. Errors are recorded in the error repo and retried later. If there are several RGWs in the destination zone, each shard is assigned to only one of them with a two-minute lease. Trimming of the source's logs happens every 20 minutes and only up to the lowest marker that every peer has reached |
| 5-7 | Top: the share of datalog shards from 11000 bucket shards (1000 buckets with numbered names and 11 shards each). Only 88 of the 128 shards get work, in groups of 11 starting at 8, 24, …, 120. Bottom: the same number of buckets with random names, covering all 128 shards. The dashed line is the ideal mean, about 86 bucket shards per datalog shard. Our simulation, based on a re-implementation of choose_oid |
| 5-8 | Top: a busy bucket shard with 12000 writes in 120 seconds produces only six datalog entries. The first is recorded at the moment of the first write and the rest at the 22-second renew ticks. Bottom: with scattered writes, a renew happens only when a write was left in the previous window. The write at second 100 creates an entry immediately because the window has expired |
| 5-9 | The add_entry decision tree in main. The first row shows the safe path: if the watch on the sem-set is broken, the entry is written immediately. The second row is merging within the window and waiting for an in-flight push. The third row is the actual push and the renew cycle, which extends the window and decrements the counters |
| 5-10 | Left: the mdlog shard for 1000 bucket keys with numbered names, which occupy only four shards. Right: 1000 random names covering all 64 shards, with 7 to 31 keys per shard. Computed with a re-implementation of rgw_shard_name |
| 5-11 | The datalog trim cycle. Every 20 minutes the sync status is fetched from all peers. If one does not respond, that round of trimming is skipped. For each shard the lowest stable marker is computed (in full sync, next_step_marker), and a trim happens only when it is ahead of the previous trim. The bottom row is a symbolic example |
| 5-12 | Deciding when the master zone is lost: wait during a short outage; make another zone the master during a long one |
| 6-1 | CephFS architecture: the client gets metadata from the MDS and data directly from the OSDs; the MDS keeps its journal and inodes in the metadata pool |
| 6-2 | Capability recall: when a second client opens the file, the MDS asks the first client to flush its buffered data and release its permission |
| 6-3 | Multiple active MDSs: subtrees are divided between ranks with an explicit pin or a distributed pin |
| 6-4 | Four ways to access CephFS: the kernel client, ceph-fuse, NFS through Ganesha and SMB through Samba |
| 6-5 | Decoding a cap string. The first row shows the prefixes: p for pin, A for Auth (ownership and permissions), L for Link, X for Xattr and F for File (content and size). The second row is the generic bits applied to each prefix: s for shared read, x for exclusive, c for cache, r and w for read and write, b for write buffering, a for extending EOF and l for lazyio. The third row shows three cases: an ordinary reader, a sole writer with full cache and buffering, and two concurrent writers on one file from which the MDS takes back c and b. In the last case all I/O goes synchronously to the OSD; this is the main reason applications that write to one file from several nodes at once are slow |
| 6-6 | The life cycle of a metadata operation. In the first row the client request takes the necessary locks in the MDS (SimpleLock, ScatterLock and so on), updates the cache and records the event in the MDLog. In the second row events are collected into segments of 1024 events, and the MDS tries to keep at most 128 segments. When a segment expires, dirfrags are written to omap. If trimming falls behind, the MDS_TRIM_BEHIND warning appears. The third row is the sequence of states after a rank fails: replay, resolve (only with several active ranks), reconnect (clients declare their caps), rejoin, clientreplay and finally active. standby-replay shortens replay time by following the journal continuously |
| 6-7 | The layout of a 10 MB file. Left: the default layout, with each 4 MB stored contiguously in one object. Right: su of 1 MiB and sc of 4, which spreads successive megabytes round-robin across four objects. The object name is the ino in hex and the object number as eight hex digits. The arrow shows where byte 9 MiB lands in both cases |
| 6-8 | The hash space of names in one directory. The top row is the root frag, the middle row is the eight frags after a three-bit split with _enc and the number of names in each, and the bottom row is a further split of frag 010 into eight six-bit frags. The coloured lines show where the hashes of three sample names fall. The horizontal axis is the low 24 bits of the hash |
| 6-9 | The number of dirfrags (blue, stepped) and the size of the largest frag (red) while creating 100000 entries in one directory, assuming immediate splitting. The dashed line is the 10000 threshold and the dotted line the fast-split threshold of 15000. After the first split, the size of the largest frag drops to about 1250 and climbs back towards 10000. Our simulation with rjenkins and the frag_t logic |
| 6-10 | The IWR counter curve of a dirfrag under steady write load, with a 5-second half-life. Each curve tends towards the steady value r × 7.21. The dashed line is the mds_bal_split_wr threshold, and the dots show the moment it is crossed. The 1000 writes per second curve never reaches the threshold |
| 6-11 | The dirfrag split and merge decision tree. The first row is the two split triggers: size and heat. The second row is normal queueing with a 5-second delay, or the fast path above 1.5 times the threshold. The third row shows the split into 8 children, the merge of frags under 50 entries and the hard cap of 100000 |
| 6-12 | The volume, subvolumegroup and subvolume hierarchy; the ceph-csi driver and Manila map each PVC or share to a subvolume |
| 7-1 | The daemon management layers in cephadm: from the ceph orch command to the container on the host |
| 7-2 | Configuration sources and their order of precedence; see each daemon's effective value with ceph config show |
| 7-3 | Two ways to add capacity: more disks in the same nodes, or more nodes |
| 7-4 | The safe disk replacement procedure: recovery first, then the safe-to-destroy check, then removal keeping the id, and the physical replacement |
| 7-5 | The upgrade order in cephadm: MGR and MON first, then OSD and MDS, and the gateways last |
| 7-6 | The sequence of Paxos messages in the MON: after an election the leader announces a new proposal number with OP_COLLECT and gathers unfinished values with OP_LAST; then, for each change, OP_BEGIN is sent with the next version, OP_ACCEPT is received from every quorum member, OP_COMMIT is announced, and OP_LEASE grants the peons permission to serve reads for mon_lease |
| 7-7 | The path of a command in a Paxos service such as OSDMonitor: read-only commands are answered in preprocess_query from the current map; modifying commands are applied in prepare_update to the pending incremental, batched with a paxos_propose_interval delay and proposed to Paxos with propose_pending; after commit the map is loaded, the incremental is published to subscribers and the issuer gets a reply |
| 7-8 | The timeline of a powered-off OSD with the defaults. After 20 seconds without replies, peers on two hosts report it and the MON marks it down. PGs become degraded, but no data moves. After 600 seconds (plus the laggy term) the OSD is marked out and backfill starts. The purple line shows the exceptions: the noout flag and a whole rack being down (mon_osd_down_out_subtree_limit), which reset the counter |
| 7-9 | Left: growth of laggy_probability and laggy_interval with each wrong down (a live OSD that returns 40 seconds later), with weight 0.3. Right: the effective grace of the next failure, i.e. the 20-second base plus p × I. The peer term is assumed to be zero in this example. If the next down is more than 48 hours later, the history is reset |
| 7-10 | Left: the usage percentage of twelve OSDs, with one 4 TB OSD 80% full (red). Right: for each OSD, the raw data that can be written before that OSD reaches full_ratio. Their minimum (red line) is the basis of MAX AVAIL and is divided by size or (k + m) / k |
| 7-11 | The ladder of fullness ratios with the defaults: the NEARFULL warning from 85%, backfill to the OSD stops from 90%, the FULL flag and client writes stop from 95%, and the OSD's own failsafe from 97%. The first three ratios are kept in the OSDMap and changed with set-*-ratio. The failsafe is an OSD configuration option |
| 8-1 | The two placement stages: the object name is mapped to a PG by a hash, and CRUSH maps that PG to a set of OSDs |
| 8-2 | A sample CRUSH hierarchy: root, datacenter, rack, host and OSDs with device classes |
| 8-3 | Cluster maps are kept in the MONs and versioned by epoch; clients and OSDs get the new map from the MON or from each other |
| 8-4 | The three stages of mapping an object to an OSD: in the client the object name is hashed with ceph_str_hash and turned into a PG number with ceph_stable_mod; then the PG number is combined with the pool id and given to crush_do_rule as input x; finally the OSDMap removes non-existent OSDs and applies upmap and pg_temp to obtain the acting set and the primary |
| 8-5 | The straw2 algorithm in three stages: for each bucket item a sixteen-bit pseudo-random number is built from the input hash, the item id and the attempt number, and its natural logarithm is computed with a table and divided by the weight; the item with the largest value is chosen, which is mathematically equivalent to picking the minimum of exponential variables with rate equal to the weight and makes each item's selection probability exactly proportional to its weight; the consequence is minimal data movement when weights change |
| 8-6 | The cephx authentication flow: by proving knowledge of the shared key, the client gets a session key and an auth ticket from the MON, then obtains a service ticket for the OSD service encrypted with that service's rotating key; the OSD, which has received the rotating keys from the MON, decrypts the ticket, checks its expiry and caps, and replies so that mutual authentication is complete |
| 8-7 | The frame structure in the msgr2 protocol: a fixed-size preamble containing the frame type tag, the number of segments up to four, the length and alignment of each segment and a separate CRC; then up to four segments containing the header, front, middle and data; and finally the epilogue, which in crc mode holds each segment's CRC and in secure mode the AES-GCM encryption and authentication of the whole frame |
| 8-8 | Adding an eleventh host (12 OSDs) to a 10 × 12 cluster with straw2, a three-replica pool and 8192 PGs, measured with osdmaptool. Grey is the ideal share, green the replicas that went to the new host, and orange the movement between old OSDs caused by the position's r changing in chooseleaf |
| 8-9 | Changing the CRUSH weight of one OSD (osd.5 on host0) from 1.0 to five values and classifying the moved placements: blue is osd.5 itself, yellow the other OSDs of host0, which gain or lose share as the host's weight changes, and red unrelated hosts. Left straw2 and right old straw, with 8192 PGs, measured with osdmaptool |
| 8-10 | The cephx flow: the client proves its identity to the MON, gets a ticket and uses it to work directly with the OSD |
| 8-11 | A write to a replicated pool: the client talks only to the primary; the primary replicates to the secondaries in parallel and acks after all replicas have committed |
| 8-12 | The main PG states and the common transitions between them |
| 9-1 | Two generations of graphical Ceph management tools: standalone tools (Calamari, VSM, openATTIC) and the Dashboard inside the MGR |
| 9-2 | The Dashboard's components: the MGR module, Grafana and Prometheus for charts, and in Tentacle the nginx gateway and oauth2-proxy as a single entry point |
| 9-3 | Role-based access control in the Dashboard: each user has one or more roles, and each role is a set of permissions on specific scopes |
| 9-4 | The Dashboard's role in the cluster life cycle: deployment, service configuration and the daily work of monitoring and maintenance |
| 9-5 | The ceph-mgr architecture. The top row is the C++ part. MgrStandby sends beacons to the MON, and when the MON declares it active, Mgr subscribes to the maps and maintains ClusterState. DaemonServer receives periodic reports (MMgrReport) and perf counters from all daemons. The middle row is the Python hosting: PyModuleRegistry discovers and loads the modules, ActivePyModules runs each module in a separate sub-interpreter with a dedicated serve() thread, and BaseMgrModule is the C++-to-Python bridge for get, mon_command and set_store. The bottom row shows the structure of a module: a Module class inheriting from MgrModule, with options, the notification types it needs, CLI commands via decorators and the serve loop |
| 9-6 | The complexity credits of four sample passwords. Each cell is one character; its colour shows the character category and its width its credit. The dashed line is the minimum of 10. The third password is rejected despite having enough credit because of three consecutive letters. The fourth shows that each non-ASCII letter is worth five credits. The question marks stand for Persian letters |
| 9-7 | The two stages of handling a REST request in the Dashboard. The top row is JWT authentication: the signature, the expiry time and the token id not being in the list of revoked tokens. The bottom row is authorisation: the controller's scope and the HTTP method's permission must both be present in one of the user's roles. Combining the permissions of two different roles is not enough for one request |
| 9-8 | The timeline of eleven login attempts with ACCOUNT_LOCKOUT_ATTEMPTS = 10. Each failed attempt increments the counter. The lockout condition is checked before the password is verified. So the eleventh attempt, with any password, disables the account, and only an administrator can unlock it with ac-user-enable |
| 9-9 | The time from writing an object until pg dump shows it, for four values of mgr_stats_period. Left: the cumulative distribution of 15 trials. Right: a box plot with the lines y = p and y = 2p. Measured on Ceph 18.2.7 |
| 9-10 | The JSON response size (left) and median time (right) for a full pg dump, pg dump pgs_brief and status in clusters with 32 to 1024 PGs. Measured on Ceph 18.2.7 |
| 10-1 | Layer-by-layer measurement: measure each layer only after you know the ceiling of the layer below |
| 10-2 | Three ways to run fio on Ceph: the rbd engine directly with librbd, a mapped block device, and a file on CephFS |
| 10-3 | Two paths to talk to a daemon: ceph tell over the network and ceph daemon through the local admin socket |
| 10-4 | Three measurement mechanisms in the Ceph code. In the top row, osd bench in OSD::run_osd_bench_test first checks the limits. The block size must not exceed 64MiB. For small blocks the total volume is limited to 100 IOPS times 30 seconds, and for large blocks to 100MiB/s times 30 seconds, so that the test does not disable the OSD in production. Then the OSDBenchTest class (defined in OSD.h) runs write transactions directly on the ObjectStore in the meta collection; no network, PG or replication is involved. In the middle row, rados bench uses the ObjBencher class in common/obj_bencher.cc: it writes the objects benchmark_data_<host>_<pid>_objectN with a default of 16 concurrent operations, stores the run's metadata in benchmark_last_metadata, and status_printer prints the instantaneous bandwidth and latency every second. In the bottom row, at boot (unless the benchmark is disabled) the OSD runs a short 4K test, compares the result with reasonable thresholds for the device type and, if it is valid, records osd_mclock_max_capacity_iops_hdd or _ssd in the MON config |
| 10-5 | The accepted count cap for osd bench as a function of block size, both axes logarithmic. Below one megabyte the cap grows linearly, assuming 100 IOPS for 30 seconds. From one megabyte on it stays constant, assuming 100 MB per second. The star is the mClock boot test's parameters, which lie exactly on the cap. Above 64 megabytes any block size is rejected |
| 10-6 | The acceptance range of the boot test result for HDD and SSD on a logarithmic axis. The green area is the range between the lower and upper thresholds and the blue line is the default value. The green points are stored. The red points, such as an HDD with a write cache or a very fast NVMe, are rejected and the previous value remains |
| 10-7 | The reservation (solid bar) and limit (light bar) of the three service classes in mClock's three built-in profiles, in small ops per second for an HDD with the defaults. Each class's weight is its share of the capacity left after the reservations. A "max" limit means no limit other than the device's capacity. The numbers come from multiplying the profile ratios by 150 megabytes per second and dividing by the cost per IO |
| 10-8 | Left: IOPS and mean latency of 4 KiB writes as a function of the number of operations in flight (rados bench -t) on the test cluster. Right: the product of measured IOPS and mean latency, which reconstructs T at every point (Little's law for a closed-loop system) |
| 10-9 | Write bandwidth (orange) and IOPS (blue) as a function of object size with 16 operations in flight on the same test cluster, together with the mean latency of each point. Below about 64 KiB the IOPS ceiling rules, and above about 256 KiB the bandwidth ceiling |
| 10-10 | A rescue with ceph-objectstore-tool: stopped OSD, content inspection, export of one PG and its import on another OSD |
| 11-1 | The suggested lab topology: three Ceph nodes and one client on a KVM host |
| 11-2 | Deployment tools of yesterday and today |
| 11-3 | The reconciliation loop in cephadm: the spec sets the desired state and the orchestrator fixes differences from the actual state |
| 11-4 | One round of CephadmServe.serve() in the order in which it is called in the code. The first row refreshes the host and daemon cache (gather-facts, the daemon list, the device inventory with ceph-volume), followed by checking for stray daemons and running requested asynchronous actions. The second row is convergence: _apply_all_services calls _apply_service for each spec, and HostAssignment.place() in schedule.py decides, based on the placement (count, label, host_pattern, hosts), which daemons should be added or removed. The third row is maintenance: checking daemon and certificate health, cleaning up removed services, detecting moved OSDs and retrying failed operations. At the end the loop sleeps, but every orch command wakes it immediately with _kick_serve_loop |
| 11-5 | The deterministic host order for five sample services in a six-host cluster. Each row is the output of get_candidates: the sorted host list shuffled with a random generator seeded with the sha1 of the service name modulo 2³². Cells with a bold border are the hosts chosen for an empty cluster with the given count. crash, with no count, goes to every host. Each colour is a fixed host, so you can see the services spread across different hosts |
| 11-6 | The three stages of HostAssignment.place. The first stage builds the candidates from the placement and shuffles them with the service name's seed. The second stage expands the slots and assigns each existing daemon to the first compatible slot. Active and ranked daemons take their places first so that they do not move. The third stage makes up the shortfall or surplus: when scaling down, daemons not next to a related service are removed first (for example an ingress without a backend). When scaling up, hosts of the related service are chosen first and then the rest in shuffle order |
| 11-7 | Automatic division of NVMe between OSD DBs on two sample hosts. Left: 12 HDDs and 2 NVMes, six equal slots on each NVMe. Right: 10 HDDs and 3 NVMes, four slots of 372.5 GB on each NVMe. The third NVMe has only two filled slots, and two hatched slots stay empty, because batch stops once the OSD count is reached. The 1490 GiB VG size is hypothetical |
| 11-8 | The order of daemon types in a cephadm upgrade, from CEPH_UPGRADE_ORDER in the main code. The first and second rows are the types that use Ceph's own image. mgr comes before mon because the orchestrator itself runs in the mgr and must know the new release. OSDs are upgraded in batches after the ok-to-stop check. The third row is the gateways and ingress, and the last row the monitoring stack and mgmt-gateway, which have their own separate images |
| 11-9 | The growth of the set of stoppable OSDs in a simulation with 4 hosts and 3 OSDs per host (replicated size 3, min_size 2). The algorithm starts from osd.0 and first tries its host-mates (green: accepted). At the root level, adding any OSD from another host takes some PGs below min_size (red, with the number of unsafe PGs under each OSD) and is rejected. Because there was at least one rejection, the algorithm does not go up a level and returns that one host |
| 11-10 | Rook: the operator turns custom resources (CRDs) into Ceph daemons, and ceph-csi makes volumes available to pods |
| 11-11 | The suggested change cycle: every spec change goes through review, dry-run and a change window, and is checked after it is applied |
| 12-1 | Comparing the FileStore and BlueStore stacks: BlueStore removes the intermediate file system and the separate journal |
| 12-2 | BlueStore's internal components: the allocator for data, RocksDB for metadata and BlueFS as RocksDB's small file system |
| 12-3 | The two write paths in BlueStore: small writes are first recorded in the WAL and applied later; large writes go directly to new space |
| 12-4 | A common hybrid node layout: each NVMe provides several LVs for the block.db of several HDD OSDs |
| 12-5 | Migrating OSD by OSD to a new layout: drain or replace, wipe, recreate and backfill |
| 12-6 | The space allocated for a 64 KiB blob as a function of its compressed length. Left: with min_alloc_size 4K, the steps give real savings up to 56 KiB, and above that the blob is stored uncompressed. Right: with 64K, 64 KiB is always allocated and compression has no effect. The red dashed line is the acceptance threshold |
| 12-7 | The compression path in a BlueStore write: compressing the blob, computing want, the two-step test (before and after adding the header), padding to min_alloc and recording in statfs, from which the USED COMPR and UNDER COMPR columns of ceph df are built |
| 12-8 | The ratio of allocated space to the logical object size for three values of min_alloc_size. For objects larger than a few tens of kilobytes the difference is negligible, but for objects of a few kilobytes the usage grows by tens of times |
| 12-9 | The target sizes of levels L1 to L4 in RocksDB with BlueStore's default options (bars) and their cumulative sum (red line) on a logarithmic axis. Each cumulative step is a block.db size that fits the next level completely. L0 and the WAL each add about 1 GiB |
| 12-10 | BlueStore's key prefixes in RocksDB and the default column families. Each kind of metadata is separated from the others by a one-letter prefix; the bluestore_rocksdb_cfs setting places some of these prefixes in separate, sharded column families |
| 12-11 | The metadata structure of an object: the Onode keeps the logical mapping in the ExtentMap; each Extent points to a Blob that holds the checksum, compression and the physical location (pextent); Blobs shared between clones are managed with SharedBlob |
| 12-12 | The TransContext state machine in BlueStore. First row: writing data with aio; second row: committing metadata in RocksDB by kv_sync_thread, where the OSD receives the commit confirmation; third row: applying deferred writes to the device and deleting their records |
| 12-13 | HybridAllocator: free extents are first kept in an AVL tree, which searches fast; when the tree's allowed memory fills up, the extra extents spill over to the BitmapAllocator, which uses a small, fixed amount of memory |
| 13-1 | 4+2 coding: a sixteen-kilobyte stripe is split into four four-kilobyte data shards, and two parity shards are computed from them; each shard is placed on an OSD in a separate failure domain |
| 13-2 | Stripe layout: the object's data is split into stripes of length k×stripe_unit; chunk i of each stripe goes to shard i, and each stripe's parity is computed separately |
| 13-3 | A degraded read: when shard 1 is unavailable, minimum_to_decode chooses a set of k available shards, and decode rebuilds the missing data |
| 13-4 | Comparing a small write in old EC and FastEC: the old implementation reads and rewrites the whole stripe; FastEC updates only the changed shard and the parities |
| 13-5 | EC plugins in Ceph: all implement the common ErasureCodeInterface and are loaded as dynamic libraries |
| 13-6 | A write in an EC pool. The primary (which is not always shard zero, but is assumed to be shard 0 in this figure) breaks the transaction into stripes and decides whether an RMW is needed. If needed, it reads the old chunks, computes the parity, and sends each shard an ECSubWrite containing that shard's chunk and the pg log entry. Each log entry has an ObjectModDesc that describes how this change can be undone. The client gets a reply only after all acting shards have committed |
| 13-7 | Reads and recovery in EC. On reads, the OSD uses minimum_to_decode to determine the smallest set of needed shards. With fast_read=false only those are read. With fast_read=true all shards are read, and decoding happens when the first k replies arrive (lower tail latency, more traffic). In recovery, the primary builds the list of missing objects for each shard from the pg log, reads from k healthy shards (or fewer with LRC and CLAY), decodes only the missing chunk and pushes it to the new OSD. The bottom of the figure shows the default min_size formula |
| 13-8 | Reed-Solomon encoding of one byte column. Left: the 6×4 matrix built with gf_gen_rs_matrix, containing the identity for the data and two parity rows. Right: computing P0 and P1 for "ABCD" and decoding after losing d1 and d3. The multiplications are in GF(2⁸) with the polynomial 0x11d |
| 13-9 | The number of patterns of losing up to m shards that cannot be decoded with the Vandermonde matrix. Green means MDS and red means it has failures. The full check was done with our script based on gf_gen_rs_matrix |
| 13-10 | The steps of a parity-delta update for changing one byte in d2: XOR of old and new, multiplication by that shard's column coefficient, and XOR into each parity. The last row shows the match with a full re-encode. The purple line compares the number of shards read on the two paths for three profiles |
| 13-11 | The data read to rebuild one lost chunk with RS, LRC and CLAY in two configurations, together with each scheme's raw overhead. LRC reduces reads with more space, and CLAY with the same space as RS |
| 13-12 | The code comment's example for an optimized 4+2 pool: shard 0 is lost; pg_temp is ordered so that the parity shards (which are always up to date) come first and osd.9 becomes the primary |
| 14-1 | Where librados sits: upstream libraries and your application work with librados through an IoCtx; the Objecter computes each operation's target and the MonClient manages maps and authentication |
| 14-2 | Structure of a compound operation: guards, modifications and a class call run in one atomic transaction on a single object |
| 14-3 | Optimistic concurrency control with assert_version: client B, which saw the old version, gets an error and retries after reading again |
| 14-4 | watch/notify: watchers register on an object; a notify reaches all of them, and the notifier gets an aggregated reply after the acks arrive or a timeout |
| 14-5 | Field order in the MOSDOp message payload with HEAD_VERSION=9 (COMPAT_VERSION=3). First come the pgid and object hash, the map epoch and flags, the request id (reqid), client_inc and mtime, then the object_locator and object name, the number and structure of each op, and finally the snapshot information, retry_attempt and features. In msgr2 all of these are in the front segment, and the input data of all ops is placed back to back in the data segment |
| 14-6 | The backoff mechanism. When a PG is peering or an object is unfound, instead of holding the op in its own queue (which consumes memory and threads), the OSD sends the client an MOSDBackoff with the BLOCK action for a range of hobjects. The client acknowledges and holds the ops for that range on its side. Once the PG becomes active, the OSD announces the release with UNBLOCK, and the client resends the ops |
| 14-7 | Left: the number of concurrent ops allowed as a function of op size. Up to 100 KiB the cap of 1024 governs, and above that ⌊100 MiB / s⌋. Above 100 MiB only one op is in flight. Right: the throughput cap of one Objecter instance by Little's law, for three sample latencies. The latencies are hypothetical; the formula comes from the code |
| 14-8 | Timeline of watch liveness. Pings are sent every 5 seconds. After the network drops at second 12, no more pings arrive. The OSD removes the watch around second 40 (30 seconds after the last ping). The Objecter, earlier, after 10 seconds, considers the session laggy and requests a fresh map |
| 14-9 | The work of each Objecter tick: counting laggy ops, pinging registered watches and, when there is a homeless op or a suspect session, requesting a fresh osdmap from the MON and sending MPing. The OSD reply side has a lossy policy, so the ping is needed to discover a session reset |
| 14-10 | Effective notify timeout as a function of the input timeout_ms. Zero leads to client_notify_timeout (10 seconds). 1 to 999 milliseconds become zero after integer division, and the OSD applies its 30-second default. From 1000 upward, whole seconds are applied |
| 14-11 | The steps of _calc_target: pool existence, mapping the object to a PG, computing up/acting with CRUSH, pg_temp and upmap, and deciding to resend if the interval changed |
| 15-1 | Comparing two ways of incrementing a counter: in the client-side method, two clients read the value 10 and both write 11, and one update is lost; with the class method numops.add, the OSD runs the requests serially inside the PG and the final value is 13 |
| 15-2 | Map of the consumers of the built-in classes: librbd relies on cls_rbd for the image header and object map, RGW on cls_rgw for the bucket index and on cls_rgw_gc, cls_2pc_queue and cls_fifo for GC and queues; cls_lock, cls_version and cls_refcount are general purpose |
| 15-3 | The three stages of validating a CALL. The first row runs when the message arrives and finds the class and method. The second row converts the method's registered flags into op flags so that OSDCap can check permission and the PG knows whether the op reads or writes. The third row compares the read and write counters after the method runs. A method that behaves contrary to its flags is rejected with EIO |
| 15-4 | Timeline of six lock requests from two clients on one object. Green points succeed and red ones fail. Shaded bars are the lock holding intervals. A's renewal at second 25 moves the expiry to 55. Because nobody touches the lock before second 60, A's entry stays in the xattr until then and is only cleared by B's request |
| 15-5 | Three consecutive cls_cxx_map_get_vals calls over 2500 keys with the prefix u_. Each call continues from the last key of the previous call (UPPER_BOUND, that is, the key after start_after). The red line is the cap of 1024 keys per request, applied to max_to_get. The more flag is false only on the last page |
| 15-6 | Left: the counter's final value after 300 increments per thread, with cls_numops (green) and with reading the omap, adding one and writing from the client side (red). The black line is the expected value. Right: the median time per increment on the Ceph 18.2.7 test cluster |
| 15-7 | Left: median time to count lines in objects of 64 KiB to 16 MiB, with cls_lua inside the OSD (purple) and with reading the whole object on the client (blue) over loopback. The dashed lines show the same client time plus the transfer time over 10 and 1 gigabit (computed). Right: the break-even bandwidth below which the Lua run finishes first |
| 15-8 | The full path of a class call in three rows: the client sends a CEPH_OSD_OP_CALL request to the primary through the Objecter; PrimaryLogPG::do_osd_ops decodes the class and method names and opens the class with ClassHandler; then the method runs, the read and write counts are compared with the declared flags, and the changes are recorded in the PG transaction |
| 15-9 | The two stages of loading a class in ClassHandler: top, the first call, with the osd_class_load_list check, dlopen of libcls_X.so from osd_class_dir and running __cls_init; bottom, registering methods with names and flags and setting the allowed state from osd_class_default_list, which affects how the class-read and class-write caps are interpreted |
| 16-1 | Cache tiering architecture in three rows: because of the overlay, the client sends requests for the base pool to the cache pool; the cache OSD checks in maybe_handle_cache_detail whether the object is present or should be promoted according to the HitSet; in the background the tier agent flushes dirty objects and evicts cold clean objects |
| 16-2 | The tier agent's state machine: three flush modes (IDLE, LOW, HIGH) based on the ratio of dirty objects to the target, and three evict modes (IDLE, SOME, FULL) based on fullness; hysteresis with osd_agent_slop prevents rapid oscillation between modes, because the entry threshold is slightly above the target and the exit threshold slightly below it |
| 16-3 | The procedure for removing a writeback cache tier: switch to proxy to stop promotion, drain with cache-flush-evict-all and repeat until the cache pool is empty, then remove the overlay and detach the tier |
| 16-4 | A ten-megabyte S3 object in RADOS. The HEAD object holds the metadata (manifest, ACL, etag and other xattrs) and, in the default storage class, the first piece of data (up to rgw_max_chunk_size=4M) inline. The rest of the data is in tail objects of size rgw_obj_stripe_size=4M. After a transition to another storage class, the HEAD stays in the default pool, but its manifest points to new tails in the new class's data_pool, and the old data is handed to garbage collection. The bottom of the figure shows the pools of one placement target |
| 16-5 | Running lifecycle in RGW. The list of buckets with rules is kept in rgw_lc_max_objs=32 RADOS objects (the lc shards). Work is done only in the rgw_lifecycle_work_time window (default 00:00 to 06:00 local time). Each shard is taken with a lock lasting rgw_lc_lock_max_time=90 seconds. rgw_lc_max_worker=3 parallel workers, each with 128 coroutines (rgw_lc_max_wp_worker), walk the bucket index and perform expire, transition or abort multipart |
| 16-6 | Size of a HitSet's Bloom filter as a function of the number of unique objects in the previous set, for two values of hit_set_period. The target size scales by the ratio of the new period to the actual duration of the previous set and is clipped between the OSD's floor and cap. The dashed lines are these two limits. Each cache PG builds one filter per period and archives it as an object in a separate namespace of the same pool |
| 16-7 | The evict condition in SOME mode for the example's ten-sample histogram. The horizontal axis is the power-of-two temperature bin, and the bars are the number of samples in each bin. The left side is main's decision, with the condition 10⁶ − upper ≥ effort for keeping: only the cold bin zero stays (green) and all the hot bins are evicted (red). The right side is the decision with the fix proposed on ceph-devel in 2017, with the condition lower ≥ effort for keeping: the cold are evicted and the hot stay. In both halves the dashed line is the decision boundary on the cumulative-share axis: in main's code the boundary is upper = 0.6 (that is, 1 − 0.4), and in the proposed fix it is lower = 0.4 |
| 16-8 | Left: the grade table of archived HitSets for two decay rates: with decay 20 the weight of the oldest set is still 41%, and with decay 50 only 6%. Right: the temperature of the four sample objects: presence in the current set always earns one million units, and with search_last_n = 1 only the first archived set in which the object is found adds its share |
| 16-9 | evict_effort as a function of the fullness ratio for the two hysteresis states. The blue curve is when the agent has been idle and the entry threshold is 0.816. The red curve is when the agent is evicting and the threshold drops to 0.784. The steps come from the 0.1 quantization, and above 100% fullness the effort jumps straight to 1.0 |
| 16-10 | The path of a lifecycle rule from registration to execution. The bucket key is mapped to one of 32 lc.N objects with a simple Linux hash. In the work window (default 00:00 to 06:00 in the server's local time), each worker locks the shards in random order and takes the next bucket. It then examines the bucket's objects with a pool of coroutines and applies expiration or transition to each version |
| 16-11 | Expiry timeline of an object with Days = 30. The object was written in the middle of day 0. obj_has_expired rounds the base time to midnight, so the run on day 30 sees a gap of 29.4 days and keeps the object. The first run that deletes it is in the window of day 31 |
| 16-12 | The DB space that the default use_some_extra policy allows for files of the higher RocksDB levels, as a function of the DB device size. The curve is saw-toothed: after each level boundary, the formula reserves twice the size of the current level and the extra space drops to zero. The red points are the text's four example sizes. The dashed line is the device's 95 percent cap |
| 16-13 | Three levels of tiering in Ceph today: at the pool level with device classes and separate CRUSH rules for nvme and hdd, inside the OSD by placing block.db and the WAL on NVMe next to HDD data, and at the object level with storage classes and lifecycle rules in RGW that move objects to a capacity pool or cloud storage after a while |
| 17-1 | The layered troubleshooting method: start from the ceph -s summary and ceph health detail, identify the layer involved (MON, OSD, PG or performance), gather evidence with that layer's commands, then proceed with a cycle of hypothesis, test and small fix, and record everything in the incident log |
| 17-2 | The Peering state machine in three rows: the main path GetInfo, GetLog, GetMissing and WaitUpThru under Started/Primary/Peering; the failure branches Down, Incomplete and WaitActingChange; and after activation, the Active path, including reservation and Recovering, reservation and Backfilling, and finally Recovered and Clean |
| 17-3 | Running find_best_info on four sample infos. The red row is dropped because of an older last_epoch_started. Of the two rows with the newest last_update, the one with the smaller log_tail, and therefore the longer log (green), wins. Preferring the current primary only applies when all previous criteria are equal. In an EC pool the direction of the last_update comparison is reversed, because newer versions must be rolled back |
| 17-4 | Timeline of an op stuck for 400 seconds. From second 30 onward it counts among the slow ops. But a "slow request" log line is written only when the op's age passes 30 × multiplier, and the multiplier doubles after each log. So the gaps between logs are 30, 60 and 120 seconds, and the OSD log is not flooded during a long incident |
| 17-5 | Distribution of the next scrub time with the defaults, from a simulation of the same formulas as the code. The left side is the shallow scrub: a uniform distribution over one to one and a half days. The right side is the deep scrub: a normal distribution around seven days with a standard deviation of 1.4 days, clipped at ±2σ. The two tall bars at the edges are the clipped samples. The yellow area is the range 0.8 to 1.2 times |
| 17-6 | The state of 32 PGs in pool t17 (size 2) on a three-OSD BlueStore test cluster with Ceph 18.2.7, sampled about every second. The dashed lines mark the events: osd.0 out and in, and osd.1 stopped and restarted. Green is active+clean, blue recovery, yellow peering, grey stale and red undersized+degraded |
| 17-7 | Left: the percentage of degraded, inactive and copyless PGs when 1, 2 or 3 of 10 hosts fail, with an exact count over all combinations from the osdmaptool mapping (the bar is the mean and the lines the minimum to maximum). Right: the probability that a client touching N objects hits an inactive PG with two failed hosts |
| 17-8 | The life cycle of an operation in the OSD in three rows: arrival, being placed in the shard queue and reaching the PG; common waits, recorded with "waiting for" events; and execution, sending sub-operations to replicas, receiving acks and sending the reply. The last recorded event of a slow operation shows where it is stuck |
| 17-9 | The scrub state machine in three parts: after reserving the replicas (limited by osd_max_scrubs), the primary selects objects chunk by chunk, waits for in-flight writes, builds a local map and receives the replicas' maps; a replica builds and sends its own map; scrub_backend chooses the authoritative copy and records inconsistencies in the ScrubStore |
| 18-1 | Four layers of defence against disaster: inside the cluster with replication and automatic recovery, between two sites with one stretched cluster (stretch mode) and an RPO of zero, between independent clusters with asynchronous replication via rbd-mirror, RGW multisite and cephfs-mirror, and finally an independent backup with object lock and a configuration backup |
| 18-2 | One round of snapshot-based synchronization: site A creates a primary snapshot S2 according to the schedule; rbd-mirror at site B scans both sides' snapshots, creates an incomplete snapshot S2' at the destination, transfers the difference from S1 to S2 with deep-copy, applies the image state and marks the snapshot complete, and finally removes the peer link from S1 so that it can be deleted |
| 18-3 | Two failover scenarios: in the planned case the application is stopped, the image is demoted in A, promoted in B after syncing, and the application is started in B, with no data lost; in the emergency case B is activated with promote --force, and after A returns, A's image is demoted and resynchronized from B with resync |
| 18-4 | The journal layout of an RBD image. The header is in the omap of the journal.<id> object. Data is in sets of splay_width objects, and each entry is appended to one of the active set's objects according to entry_tid % splay_width. When one of the objects reaches 2order bytes, the whole set is closed and the next set becomes active. Sets that all clients have passed are deleted |
| 18-5 | The three stretch mode states and the pg_pool_t values in each (the logic of the functions trigger_degraded_stretch_mode, trigger_recovery_stretch_mode and trigger_healthy_stretch_mode in OSDMonitor.cc in Squid). The min_size values are for the default stretch pool with size=4 |
| 18-6 | A peer's score in ConnectionTracker. Red: a connection that drops from a full score and reaches zero after 16.6 hours. Green: recovery from zero. The slope of both is set by the 43200-second half-life. The points are values computed with 2-second steps |
| 18-7 | Election scores of five MONs in the scenario where a's link to DC2 is down. Left: during the outage, b wins with a score of 4. Right: after reconnecting, a's score history is still damaged, and b, which has a lower rank than c and d in a tie, remains leader. e, as the tiebreaker, has a score of −1 |
| 18-8 | The proposal handling logic in the connectivity strategy. The first row is the tiebreaker ignoring a proposal from a down MON, and computing three scores. The second row is the decision between proposing oneself and ACKing another. The third row shows two side rules: a score of −1 for disallowed leaders, and a margin of 0.0005 to prevent oscillating votes |
| 18-9 | Distribution of 100 images among rbd-mirror daemons with SimplePolicy across five events. The third column shows the effect of migration_throttle: B's freshly moved images cannot be moved for 300 seconds, and C gets only 17 images. The fourth column happens only when rebalance_timeout is non-zero. Our simulation, based on SimplePolicy.cc |
| 18-10 | stretch mode topology: two data centres, each with two MONs and OSDs that hold two copies of every PG, and a data-less tiebreaker MON at a third site; every write is recorded at both sites before the ack, and if the link between the two sites breaks, the tiebreaker determines the winning site |
| 18-11 | The procedure for rebuilding the monitor store from the OSDs in two stages: first, on each OSD host with the OSDs stopped, ceph-objectstore-tool with the update-mon-db operation collects the maps into a shared store directory, and this directory is carried from host to host; then ceph-monstore-tool rebuilds the store with the admin key and the MON ids, and store.db is replaced on the MONs |
| 19-1 | The upgrade procedure in three stages: preparation, including a healthy cluster, reading the release notes and making sure the hosts can reach the image; the cephadm order, from mgr and mon to the OSDs with ok-to-stop checks and then the other daemons; and the end, with identical versions and a check of require_osd_release |
| 19-2 | Replacing a failed disk while keeping the OSD id, in two stages: removal with orch osd rm --replace, which drains the PGs and marks the OSD destroyed, and replacement, in which after the physical swap and, if needed, a zap, the OSD service specification picks up the new disk and the OSD is created with the same id and backfilled |
| 19-3 | The steps of the pg_autoscaler calculation: first each CRUSH root's PG budget from the number of OSDs times mon_target_pg_per_osd, then each pool's share from actual usage or a target ratio, converted to pg_target taking bias and size into account, and finally rounding to a power of 2 and changing only when the difference exceeds the threshold |
| 19-4 | How the balancer works in upmap mode: the mgr module runs do_upmap every sleep_interval (default 60 seconds) if the cluster is quiet; OSDMap::calc_pg_upmaps compares each OSD's PG count with the weighted target value and, for OSDs deviating by more than max_deviation, creates pg_upmap_items entries that respect the CRUSH rule; the result is backfill only for the selected PGs |
| 19-5 | Left: the step function nearest_power_of_two: each pool's target is rounded to the nearest power of two, and the decision boundary (orange dotted line) is 1.5 times the lower power. The red point is the text's example. Right: the threshold rule with threshold = 3 and a current pg_num of 32: only the green values, that is above 96 or below 10.7, cause a real change |
| 19-6 | The four possible choices for rounding the two pools in the text's example. The height of each bar is the total distance of the rounded values from the target. The grey option exceeds the budget of 424 extra PG replicas and is excluded. The green option has the lowest cost and is chosen. Instead of trying all 2ⁿ combinations, dynamic programming keeps only the possible sums of extra PGs |
| 19-7 | Each OSD's PG count deviation from its weighted target in the text's example, before upmap and after each upmap. The dashed lines are the upmap_max_deviation = 5 boundary. Each upmap takes one unit from the fullest OSD and adds it to the least loaded. The bal_score, the weighted sum of squared relative deviations, decreases with each move. The weight-2 OSDs with deviation ±4 remain untouched, because they are within the boundary |
| 19-8 | Distribution of PG counts over 120 OSDs before upmap (orange, CRUSH only) and after upmap (blue), for 1024 and 8192 PGs, with osdmaptool --upmap and an allowed deviation of 1. Each chart's title shows the number of pg-upmap-items entries and the share of replicas moved |
| 19-9 | Left: after doubling pg_num and pgp_num, what percentage of child PGs land on 0, 1, 2 or 3 new OSDs, and the share of all data moved (about 48.7%), measured with osdmaptool on 120 OSDs. Right: a model of the adjust_pgs controller in the mgr for 1024 to 2048, with a cap of 128 on the gap between pg_num and pgp_num and a pgp step of pg_num × min(0.05 − misplaced, 0.025), assuming backfill finishes before each round |
| 20-1 | The monitoring stack architecture in three rows: on each host, ceph-exporter reads the daemons' performance counters from the admin socket and node-exporter publishes operating system metrics; the active mgr with the prometheus module provides cluster-level metrics and the service discovery endpoint; Prometheus scrapes everything and evaluates alert rules, Alertmanager routes alerts and Grafana displays dashboards |
| 20-2 | The path of a performance counter from production to consumption: the OSD registers a counter such as op_w_latency with PerfCountersBuilder, the admin socket exposes it through the counter dump and counter schema commands, ceph-exporter converts it to Prometheus format with the ceph_daemon label, and Prometheus stores it and uses it with PromQL in alert rules and panels |
| 20-3 | The alert cycle in three stages: signals from ceph_health_detail and the state and latency metrics; the rule in Prometheus with expr and a persistence duration for, a severity label and a description with a runbook link; and delivery in Alertmanager with grouping by alertname and cluster, inhibit to silence secondary alerts, and sending to on-call |
| 20-4 | The admin socket protocol. One UNIX connection is opened per command. The request is a JSON string terminated by 0x00 or 0x0A. The response comes as a 4-byte big-endian length followed by the payload. If the first byte of the request is 0x00, the old 4-byte protocol is assumed (code 1 means perfcounters_dump and code 2 means perfcounters_schema) |
| 20-5 | Two paths by which a counter reaches Prometheus. The mgr path uses MgrClient and the MMgrReport message (every mgr_stats_period=5 seconds), and the prometheus module publishes it on port 9283. The ceph-exporter path on each host reads directly from each daemon's admin socket (counter schema and counter dump every exporter_stats_period=5 seconds) and publishes on port 9926. LONGRUNAVG counters become two series, _sum and _count |
| 20-6 | Left: a view of the two-dimensional write histogram for a synthetic load with three request sizes (4 KiB, 64 KiB and 4 MiB) and a log-normal latency distribution. Colour is the logarithm of the count. Each row is a request size and each column a latency range. Right: the ranges of the latency axis buckets on a logarithmic axis. Each bucket is twice the previous one, and three sample values (150 µs, 1.5 ms and 30 ms) are marked on it. The numbers on the left do not come from measurement and only show the shape of the data |
| 20-7 | Timeline of the prometheus module's cache with a scrape_interval of 15 seconds. The top row is the duration of each collect(). The orange collections took more than 15 seconds. The two lower rows show the responses to Prometheus scrapes under the two policies. In fail (which the default value log is also turned into), scrapes whose last completed collection was slow get 503. In return, the same scrapes get stale data with code 200. The durations are hypothetical and were chosen to show the logic |
| 20-8 | Top: ceph_osd_up samples every 15 seconds for an OSD that goes down and up seven times in the last five minutes. The yellow area is the 5-minute evaluation window at second 585. Bottom: the value of rate(ceph_osd_up[5m]) × 60 at each evaluation, with an approximate implementation of Prometheus's extrapolation. The curve rises one step with each flap and crosses the dashed threshold line of 1 after the seventh flap |
| 20-9 | PG counts of ten OSDs in one device class, with an average of 105 (black line) and a ±30% band (green). The percentage above each bar is that OSD's deviation from the average. Only osd.9, at 34.3%, is outside the band (red), and if it stays that way for five minutes the CephPGImbalance alert fires. The numbers are an example |
| 20-10 | Synthetic ceph_pool_percent_used samples over 48 hours (blue dots), with linear growth from 0.70 to 0.78 and some noise. The red dashed line is the least squares line over the same window, extended 5 days forward. The prediction at that point is 0.987, which crosses the 0.95 line, so the alert fires. The vertical line marks the evaluation point |
| 20-11 | mgmt-gateway as the single entry: the user's browser connects to the nginx-based gateway with TLS and a virtual address, which, when authentication is enabled, passes the user through oauth2-proxy and the external IdP; behind the gateway sit the Dashboard, Grafana, Prometheus and Alertmanager, which are no longer directly exposed to the network |
| 21-1 | The performance tuning cycle: define a measurable goal, take a baseline with benchmarking tools, apply only one change, measure again with the same load and the same duration, and record the change if it improved things or revert with ceph config reset otherwise |
| 21-2 | Four performance layers from bottom to top: hardware and network, the operating system, OSD settings including memory, shards and mClock, and at the top pool design and client settings; the biggest effect usually comes from the lower layers |
| 21-3 | The request path inside the OSD: the Messenger receives the message and one of the shards is chosen based on the hash of the PG ID; each shard has its own mClock queue, and that shard's threads execute the request by taking the PG lock in PrimaryLogPG::do_op and hand the transaction to BlueStore |
| 21-4 | The dmClock algorithm in three parts: each request gets three tags R, P and L based on its class's parameters; the scheduler first, in the reservation phase, runs a request whose R tag is due, and otherwise picks the lowest P tag among the requests that have not exceeded their limit; immediate and high-priority requests such as peering are processed before the mClock queue |
| 21-5 | The OSD memory control loop with default values. The min and max bounds are computed from target, fragmentation and base. tune_memory increases or decreases the budget every second based on how far mapped memory is from target. The resulting budget is split every 5 seconds by priority between the RocksDB, onode and data caches |
| 21-6 | Transaction cost computation and BlueStore concurrency capacity for 4K writes. On HDD about 50 direct transactions or 100 deferred transactions fit in the queue, and on SSD more than 5500. A short queue in BlueStore makes requests wait longer in the mClock queue, where the scheduler can still prioritize between client and recovery |
| 21-7 | Left: the number of PGs of a sample SSD OSD in each of the eight queue shards, with 100 PGs and random ps. The dashed line is the average of 12.5. Right: the distribution of the ratio of the fullest shard to the average across 2000 simulated OSDs. Even when every PG has equal load, one shard usually has 30 to 70 percent more PGs than average. These numbers are a simulation and assume each PG's ps is random and independent |
| 21-8 | Splitting the memory of a 256-gigabyte host with MemoryAutotuner. The top bar is the total memory. The hatched 30 percent is set aside for the operating system and RSS overhead. The bottom bar shows the share of non-OSD daemons (with the minimums from the min_size_by_type table or current usage, whichever is larger) and then 20 equal OSD shares. The mon and RGW usage in this example is hypothetical |
| 21-9 | Five sample writes on an HDD OSD and how _do_write_data splits each one. The grey grid is 4 KiB blocks. Yellow is the part that goes to the small path. Red is the part of the big path that is deferred because it is less than 64 KiB. Blue is a direct write. The last row shows the aligned 4 KiB edge case, which despite lying in one block goes to the big path because of the length ≠ min_alloc_size condition |
| 21-10 | dmClock simulation output for three clients with different reservations, weights and limits. Left, capacity of 1000 requests per second: blue is service in the reservation phase and orange is service in the weight phase. client gets only its reservation, recovery reaches its limit of 200 and scrub takes the rest. Right, capacity of 350 requests per second, which is below the sum of the reservations: the reservations shrink in a 3 to 1 ratio and scrub gets almost zero. Each request's cost is assumed to be 1 |
| 21-11 | The two-phase logic of dmClock. When each request reaches the head of the client's queue, three tags R, P and L are built from that client's previous tag and the arrival time. On each do_next_request call, due reservations are served first. If no reservation was due, the minimum P is chosen among clients that have not exceeded their limit, and that client's R tag is moved back. The OSD is built with AtLimit::Wait, meaning that if all have reached their limit, no request comes out |
| 22-1 | Map of the main src/ directories on the main branch. The first row is the shared base code: basic types and encoding in include, the config, log, perf counter and thread infrastructure in common, the network layer in msg/async, cephx authentication in auth and the CRUSH algorithm in crush. The second row is the client libraries: Objecter and Striper in osdc, librados, librbd, the CephFS client in client and OSD-side classes in cls and objclass. The third row is the daemons: mon, osd, os (ObjectStore and BlueStore), mds and mgr. The fourth row is the rest: rgw, crimson (the next-generation OSD on Seastar), the erasure-code plugins, command-line tools in tools and Python code in pybind and python-common |
| 22-2 | Relationships of the main classes on the data path. On the client side, RadosClient owns an Objecter. Objecter keeps an OSDSession for each OSD and establishes connections (AsyncConnection with ProtocolV2) through AsyncMessenger. On the OSD side, the OSD class, which is the message Dispatcher, has N OSDShard objects, each with its own scheduler (default mClockScheduler). Each PG is mapped to a shard by hash. PrimaryLogPG extends PG and has PeeringState and PGLog, and performs replication through PGBackend (the ReplicatedBackend or ECBackend implementations). In the storage layer, each change is an ObjectStore::Transaction that BlueStore executes with Collection, Onode and TransContext on top of RocksDB, BlueFS and the Allocator |
| 22-3 | The thread model in the classic OSD. The first row is networking: there are ms_async_op_threads (default 3) Workers, each running an epoll-based EventCenter. Data messages are delivered in the same network thread through ms_fast_dispatch to reduce queue and context switch cost. The second row is op processing: an op is queued in the scheduler of one of the OSDShards based on the hash of the PG ID, and the ShardedThreadPool has threads_per_shard threads per shard that pick up the op, take the PG lock and run do_request. The third row is helpers: Finisher for running callbacks in a separate thread, SafeTimer for timed events such as tick and heartbeat, and BlueStore's two main threads, kv_sync_thread and kv_finalize_thread |
| 22-4 | Sequence diagram of a replicated write from librados to the ack. Objecter computes the target and sends MOSDOp over msgr2 to the primary. On the primary, the message is received in ms_fast_dispatch and queued in the PG's shard. A worker thread picks it up and brings it through do_request and do_op to execute_ctx, where prepare_transaction with do_osd_ops turns the operations into an ObjectStore::Transaction. issue_repop sends the transaction to the replicas through the backend and at the same time hands the local transaction to BlueStore. When all acting members have committed, eval_repop sends the reply to the client |
| 22-5 | The life cycle of a TransContext in BlueStore. queue_transactions builds a TransContext for each transaction, and _txc_add_transaction executes each operation; writes go from _write to _do_write. Depending on alignment and size, either _do_write_small or _do_write_big is chosen, and _do_alloc_write gets space from the Allocator and applies checksums and compression. The data is either written directly with aio (AIO_WAIT then IO_DONE), or, if smaller than prefer_deferred_size (64K on HDD, and zero, meaning never, on SSD), stored in the WAL together with the metadata. Then kv_sync_thread commits several transactions in one RocksDB batch with one fsync, and on_commit is called. On the deferred path, the actual write to the device happens later in the background |
| 22-6 | The RBD write path in librbd/io. The request enters from api::Io into the ImageDispatcher, which runs layers such as QoS, exclusive lock, journal and cache in order. AbstractImageWriteRequest::send_request splits the image range into RADOS objects with Striper. Each object passes through ObjectDispatcher::send_dispatch and ObjectDispatch::write, and AbstractObjectWriteRequest in turn updates the object map, writes the object, and runs copyup if the object does not exist in a clone (chapter 3) |
| 22-7 | The PUT path in RGW. The beast-based frontend hands the HTTP request to process_request, and after authentication rgw_process_authenticated runs verify_permission, pre_exec and execute for the op in order. RGWPutObj::execute hands the data through the Writer in the rgw::sal abstraction layer to the AtomicObjectProcessor in the rados driver, which writes the head and the tail stripes with RadosWriter::process. Finally _do_write_meta writes the metadata and updates the bucket index with UpdateIndex::prepare and complete (cls_rgw methods, chapter 4) |
| 22-8 | The CephFS path. In the first row, open is a metadata request: Client::_open sends an MClientRequest, the MDS in Server::handle_client_open takes the needed locks with Locker::acquire_locks and issues suitable caps with the reply. In the second row, write is the data path: Client::_write with get_caps needs at least Fw and prefers Fb. If it has Fb, the data is buffered in the ObjectCacher; otherwise it is written synchronously. The data goes directly to the OSDs with Striper and Objecter, and the MDS is not on the data path |
| 22-9 | Median (dark blue) and 90th percentile (light blue) of the time between consecutive OpTracker events for 300 4 KiB writes on a Ceph 18.2.7 test cluster, on a symlog axis. Left: osd_op on the primary, dominant order only. Right: osd_repop on the replica |
| 22-10 | Median timeline of a four-kilobyte replicated write. The top is the op on the primary, and the bottom is the op on the replica, shifted to the moment the subop was sent on the primary. The replica finishes its work about 1170 microseconds after the start, but the primary sees its confirmation at about 1600 microseconds. The difference is the cost of the return path in the messenger and queue |
| 23-1 | The path of a write from BlueStore to the disk. In user space, BlueStore builds a block-aligned bufferlist and collects a list of aio_t (offset, iovec and write hint) through an IOContext. KernelDevice::aio_submit hands them in a batch to the aio queue. At the syscall boundary, libaio uses io_submit and io_uring uses io_uring_prep_writev and io_uring_submit, on an fd opened with O_RDWR and O_DIRECT (a separate fd for each write hint). In the kernel, the bio becomes a request in blk-mq, and the NVMe or SCSI driver transfers the data with DMA directly from user memory pages. Completion reaches _aio_thread through io_getevents or a CQE, and it calls aio_callback |
| 23-2 | The sequence of an I/O batch. The transaction thread only records aio_write in the IOContext, and then aio_submit sends them all in a batch (up to a depth of 1024). The independent _aio_thread, in a loop with a 250-millisecond timeout, reaps at most 16 completions and decrements each IOContext's num_running counter. When this counter reaches zero, aio_callback is called with that transaction's priv, and BlueStore moves it from AIO_WAIT to IO_DONE and then KV_QUEUED |
| 23-3 | The three layers of bufferlist. At the top, list keeps a linked list of ptr_nodes in _buffers, along with _carriage (pointing to the last ptr, on which fast appends are done), _len (total bytes) and _num (number of ptrs). In the middle, each ptr is only a window: a pointer to a raw along with _off and _len. At the bottom, raw owns the actual memory and holds data, len, the atomic reference counter nref, the mempool ID and a cache of the last crc. Several ptrs in several lists can point to one raw; copying a bufferlist only increments nref |
| 23-4 | The crc32c implementation selection chain and where it is used. The check order is: on x86 first AVX512 with VPCLMULQDQ, then SSE4.2 (with or without PCLMUL for processing three streams in parallel), on aarch64 the CRC32 instructions, and on ppc, s390x and riscv dedicated versions. If none is available, the software table-driven sctp implementation is used. This function is used in BlueStore for each data chunk, in msgr2 crc mode for each segment and preamble, and in bufferlist with the crc cache on the raw |
| 23-5 | The sixty-four-byte iocb structure that aio_t in Ceph fills for each I/O. The example is a pwritev with three iovecs, 12 KiB in total, at offset 0x5A000. aio_data is the pointer to the aio_t itself and comes back in the io_event. With this pointer, the reaping thread knows without searching which IOContext to complete. In vectored commands, aio_nbytes is the number of iovecs, not the byte length |
| 23-6 | The wait before each retry (bars, logarithmic scale) and the cumulative wait (red line) when io_submit keeps returning EAGAIN. After 8 attempts only 32 milliseconds have passed, but the last 8 attempts take more than 8 seconds. So a short saturation is almost invisible, and only a long saturation turns into a slow op |
| 23-7 | The io_uring submission queue entry that init_sqe in Ceph builds for the same pwritev. The opcode is IORING_OP_WRITEV (2). The IOSQE_FIXED_FILE flag means the fd is an index into the table of registered files, and the kernel does not pay the fget/fput cost on each I/O. user_data is the same aio_t pointer and comes back in the sixteen-byte CQE |
| 23-8 | The full io_uring path in KernelDevice. Submission is done under sq_mutex, because several threads (transaction, deferred, BlueFS) submit I/O concurrently. Reaping is in the aio_thread under cq_mutex. When no CQE is ready, instead of blocking in io_uring_enter, it waits on ring_fd with epoll, which preserves the same bdev_aio_poll_ms timing pattern as libaio |
| 23-9 | The sizes of the ten raw_combined buffers that refill_append_space creates when appending one megabyte of data in small pieces. The sizes double from 4K until they reach the 256K cap. S is the size of the raw_combined header, which is subtracted from each allocation. The unused space at the end of the last buffer is the reason for the difference between the 1276 KiB allocated and the 1024 KiB of data |
| 23-10 | Three uses of crc32c jump matrices in Ceph. The top row is the ceph_crc32c_zeros algorithm, which breaks the length into powers of two. The middle row is the example of 4096 zero bytes, solved with one matrix multiplication. The bottom row is the adjustment of a cached crc in bufferlist: because CRC is linear, the difference between two seeds equals the crc of a zero region with a seed equal to the XOR of the two |
| 24-1 | The layout of the start of BlueStore devices. On a block device with a co-located DB, block 0 is the label, block 1 the BlueFS superblock, and the rest shared space for data and BlueFS. The label is also repeated at the 1, 10, 100 and 1000 gigabyte positions (only if the device is that large). On a separate block.db device, BlueFS extents start at 8K (SUPER_RESERVED) |
| 24-2 | The structure of BlueFS. The superblock is in block 1 and points to the fnode of the log file (ino 1). The log is a sequence of transactions whose ops (directory creation, link, full or incremental fnode update, removal, seq jump) build the entire metadata. Each fnode has a list of extents {bdev, offset, length}. When the log exceeds 16 megabytes and 5 times the estimated size, compaction starts |
| 24-3 | The structure of an onode. The onode key is encoded in an order such that its lexicographic sort matches the ghobject_t order: shard + 0x80, then pool + 2⁶³, the bit-reversed hash, the namespace and escaped name, snap, generation, and finally the character 'o'. The value is bluestore_onode_t. lextents point to blobs, and blobs to physical pextents. Large extent maps are broken into separate shards with the key "onode + u32 offset + x" |
| 24-4 | Allocators and the freelist. The default allocator is hybrid: free space is kept in an AVL (an offset tree and a size tree), and if its memory exceeds bluestore_hybrid_alloc_mem_cap (64M), small pieces overflow into a Bitmap. The persistent freelist, if used, is under prefix b: each key covers 128 allocation units with 128 bits, and changes are applied with a bitwise_xor merge operator. In allocation-from-file mode (the default), the allocator is saved to a BlueFS file on a clean shutdown and rebuilt from the onodes after a crash |
| 24-5 | The BlueStore cache. The buffer cache uses the 2Q algorithm by default: fresh data goes to warm_in. When evicted, only its header remains in warm_out. If the same range is read again, it moves to hot. This prevents a sequential scan from evicting frequently used data. Every 5 seconds autotune splits memory between kv, meta and data based on osd_memory_target |
| 24-6 | Three views of a 3-megabyte region of the device. Green is a free range and hatching is the allocated part. In the middle view, near-fit with the cursor for the 64K size walks forward from the start of the region, skips the 32K range and takes the start of the 128K range. In the bottom view, best-fit places the 32K request exactly in the smallest sufficient range, so the large ranges remain for large requests |
| 24-7 | The fragmentation score for one gigabyte of free space divided into equal-sized pieces. The horizontal axis is logarithmic. The curve falls quickly between 1 megabyte and 128 megabytes, because the value factor for each doubling of length decreases from 1.2 to 1 in this range. The dashed blue line is the text's sample mix, which has a score of 0.26 despite thousands of small pieces |
| 24-8 | The two paths of the hybrid allocator. The top row is inserting a free range: up to the memory cap the range is inserted into the two AVL trees, and after that the smallest ranges are pushed to the bitmap. The bottom row is allocation: requests smaller than the smallest AVL range are answered from the bitmap first. This keeps the OSD's memory bounded without losing large ranges |
| 24-9 | The first 588 bytes of the real label of osd.0 on the Ceph 18.2.7 test cluster, coloured by field: the magic text and text uuid (60 bytes), the ENCODE_START header with version and length, the binary uuid, size, btime, description, the fourteen-pair meta and the final crc32c, which was verified with an independent computation |
| 24-10 | The real free space map of osd.0 at three moments. Left: the number of free extents in each size range (logarithmic axis). Right: the total free bytes in each range. Tens of thousands of 4 KiB holes have almost no share of the bytes, which is why fragmentation_rating stays low |
| 25-1 | The write and read paths in RocksDB. On the write path, the WriteBatch is first recorded in the WAL (on the WAL or DB device through BlueFS) and then in each column family's active memtable. In the background, full memtables become immutable, once at least 6 have accumulated they are merged and flushed to L0, and compaction moves them to lower levels. A read checks, in order, the memtable, the immutables, all L0 files, and then at most one file from each other level (with the help of the bloom filter) |
| 25-2 | The structure of a block-based SST file. Data is stored in blocks of about 4K (rocksdb_block_size) with key prefix compression and restart points. After the data come the filter block (bloom with 20 bits per key in Ceph), the index (the last key of each block and its BlockHandle), properties and the metaindex. The footer at the end has the BlockHandles of the metaindex and index, the format_version and the magic number 0x88e241b785f4cff7. Each block has a five-byte trailer containing the compression type and the checksum |
| 25-3 | The target size of the levels with Ceph's defaults and their placement on devices. L1 is 1G and each level is 8 times the previous one, so L2 is about 8G, L3 about 64G and L4 about 512G. The use_some_extra policy places the levels that fit completely on block.db, uses the extra space for part of the next level, and sends only the rest to the slow device |
| 25-4 | Column family sharding in BlueStore. The default on main is: "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32". Each listed prefix has its own separate CF(s). To choose the shard, the specified byte range of the key is hashed with ceph_str_hash_rjenkins and the remainder modulo the number of shards is taken. Unlisted prefixes stay in the default CF |
| 25-5 | A skiplist with ten keys and random heights, and the path of Seek(38) in red. The search starts at head's highest level, moves forward while the next key is smaller, and moves down a level when it is larger or null. If an equal key is found at any level, it returns there. In InlineSkipList the upper-level pointers sit before the node itself in the same arena allocation, so a node of height h has exactly h eight-byte pointers followed by the key and value in one contiguous chunk |
| 25-6 | The group commit sequence in WriteThread. The first writer becomes the leader with a CAS on newest_writer_. The others wait. The leader merges batches into one WAL record up to the size cap and as long as the flags are compatible. It Syncs once, inserts each batch into the memtable with consecutive sequence numbers, and then wakes everyone with STATE_COMPLETED |
| 25-7 | How log::Writer::AddRecord writes a seventy-thousand-byte batch into four 32 KiB blocks. Each fragment has a 7-byte header: a four-byte crc32c, a two-byte length and a one-byte type. Blocks are independent: if a reader hits a corrupt block during recovery, it can resynchronize from the start of the next block. If fewer than 7 bytes remain to the end of a block, they are filled with zeros |
| 25-8 | Byte-by-byte layout of an SST data block with two sample entries. The first entry is a restart and holds the full 18-byte key (10 bytes of user key and 8 bytes of trailer containing seq and type). The second entry has only 9 non-shared bytes. The end of the block is the array of restart offsets and a u32 footer whose bit 31 indicates the in-block index type and whose remaining bits give the number of restarts |
| 25-9 | False positive rate versus bits per key for a standard Bloom and a cache-line-local Bloom, with the number of probes each implementation chooses. The red point is Ceph's setting (20 bits, 11 probes, about 0.017%). At 10 bits the two curves differ little, and at 20 bits by about 2.5 times. But the local Bloom always touches one 64-byte line, and this is decisive for onode Gets, most of which look in L0 to L2 for a key that is not there |
| 25-10 | The actual and target size of each level with level_compaction_dynamic_level_bytes=true. The targets are obtained from the last level upward by dividing by 8. L1 and L2 are below the base level and stay empty. No score is computed for the last level, because MaxInputLevel is num_levels − 2 |
| 26-1 | The full msgr2 handshake. After the banner, each side sends HELLO (the entity type and the address at which it sees the other side). Then cephx authentication runs with AUTH_REQUEST, MORE round trips and AUTH_DONE (which sets the global_id and the connection mode, crc or secure). From this point on, frames are in the chosen mode. AUTH_SIGNATURE (HMAC-SHA256 over the bytes from before authentication) prevents tampering with the unencrypted phase. CLIENT_IDENT and SERVER_IDENT establish the session with a cookie and global_seq |
| 26-2 | The frame structure in msgr2.1. preamble_block_t is exactly 32 bytes and is designed to be a multiple of the AES block (16). In crc mode the first segment comes right after the preamble together with its own crc, and if there are other segments, the epilogue holds late_status and the crc of the remaining segments. In secure mode the preamble is encrypted together with the first 48 bytes of segment zero in one go and gets a GCM tag. Messages have four segments: header2, front, middle and data (page-aligned for zero copy) |
| 26-3 | The same KEEPALIVE2 frame as above, this time as a byte grid: the thirty-two-byte preamble with the tag, the segment count and the table of four segments (the last three zero), then the preamble's own crc, then the eight-byte utime_t segment and finally that segment's crc. The total is 44 bytes, and since there is only one segment there is no epilogue. This is the smallest frequent frame on an idle connection; for example, MonClient sends one to the mon every mon_client_ping_interval seconds (send_keepalive) |
| 26-4 | The ProtocolV2 state machine. The connect side goes from START_CONNECT through banner, HELLO, auth and compression to SESSION_CONNECTING (a new session) or SESSION_RECONNECTING (a previous cookie) and finally READY. The accept side follows a similar path. In READY, receiving each message passes three throttlers (message count, policy bytes and the dispatch queue). An error on a lossless connection leads to STANDBY or reconnection with backoff (0.2 to 15 seconds), and on a lossy connection to CLOSED |
| 26-5 | The threading model in AsyncMessenger. NetworkStack (Posix by default) creates ms_async_op_threads=3 Workers, and each Worker has an EventCenter. Each connection is assigned to one of the workers when it is created and stays there for its whole life. Each worker's loop first runs the expired timers, then waits in epoll_wait (edge-triggered) for fd events, and finally runs the external events that other threads have queued. Other threads wake the worker by writing to an eventfd (or a pipe on systems without it) |
| 26-6 | Comparison of the on-wire bytes for a message with 4 KiB of data in crc and secure modes. In crc mode, segment 0 has its own four-byte crc and the thirteen-byte epilogue holds the crcs of the other three segments. In secure mode the preamble has room for 48 inline bytes and the sixteen-byte GCM tag, so the message header is encrypted without a separate frame. front is rounded up to a multiple of 16 and the epilogue is one 16-byte block plus the tag. The data bar is cut at 1000 bytes for readability |
| 26-7 | Layout of the first 40 bytes of the connection_secret that cephx gives both sides at the end of authentication: 16 bytes of shared AES-128 key, then a twelve-byte receive nonce and a twelve-byte send nonce. Each nonce has a fixed four-byte part and an eight-byte counter. The server side swaps the rx and tx roles with the crossed flag so that each direction has its own nonce |
| 26-8 | The structure of process_events in EventCenter. The wait timeout comes from the nearest time event and has a floor of 1 millisecond. If an external event is waiting, it becomes zero. After epoll_wait, the fd callbacks run first, then the timers and finally the external events. The bottom row shows the wake-up path from another thread: a queue under a lock, an atomic counter and an eight-byte write on the eventfd |
| 26-9 | A trace of get_worker for ten connections and one close. The columns on the right show each Worker's references counter after every event. The choice is always the lowest counter, and on a tie the first Worker wins. Closing the fifth connection opens a slot on Worker 1 that the next connection fills immediately |
| 26-10 | The reconnect sequence of a lossless connection. After the drop the client moves the unacknowledged messages back to the head of the queue and, after authenticating again, sends SESSION_RECONNECT. The OSD announces the last seq it received in RECONNECT_OK, and the client discards everything up to that number. Lossy connections (such as client to OSD in the Objecter) do not take this path; they are reset instead, and the Objecter resends the ops itself |
| 27-1 | Three layers of computation. At the top, GF(2⁸) arithmetic: addition is XOR, and multiplication is done with shifts and an XOR with 0x1D on overflow, or with log/exp tables. In the middle, ISA-L's systematic generator matrix for k=4 and m=2 with the Vandermonde method: the first four rows are the identity (the data stays untouched), row P0 is all ones (plain XOR) and row P1 is the powers of 2. At the bottom, the SIMD implementation: ec_init_tables builds two 16-entry tables (low and high nibble) for each coefficient, and the PSHUFB instruction uses these tables to multiply 32 or 64 bytes in one instruction |
| 27-2 | Splitting a 64K object in a pool with k=4, m=2 and stripe_unit=4K. stripe_width is 16K, so the object has four stripes. In each stripe, four consecutive 4K regions go to data shards 0 to 3, and two parities are computed and written to shards 4 and 5. Each OSD keeps one "shard object" with its own shard_id and a size of 16K |
| 27-3 | A partial 4K write compared in the two backends. In the old EC, the primary reads the whole stripe from k shards, merges the new data, encodes again and rewrites all the shards. In optimized EC (Tentacle with allow_ec_optimizations), only the same old 4K and the parities are read, the delta = old ⊕ new is computed, and each parity is updated with its coefficient times the delta. A small read is also served directly from one data shard instead of a decode from k shards |
| 27-4 | Rebuild traffic for one shard of size S. RS with k=4 needs 4S. LRC with l=3 reads only from the local group (3 chunks). CLAY with d=5 reads from 5 helpers, each a fraction 1/(d−k+1) = 1/2 of a chunk, that is 2.5S. SHEC builds parities over overlapping subsets of the data so that a single failure is repaired by reading fewer than k |
| 27-5 | The 32-byte table that gf_vect_mul_init_base builds for the coefficient c=0x02. The first 16 entries are the products of c with the low nibble (0 to F), and the second 16 entries the products of c with the high nibble (0x00 to 0xF0). Each byte is multiplied with two lookups and one XOR, because multiplication in GF(2⁸) is linear over XOR: c·(hi ⊕ lo) = c·hi ⊕ c·lo. The yellow entries are the two entries read for the byte 0xB7 |
| 27-6 | The path for multiplying 32 bytes of data by the constant c=2 with AVX2. The data is split into two nibbles. Each nibble is looked up with VPSHUFB in its 16-entry table and the two results are XORed. The bottom row shows that this multiplication is repeated for each data column and accumulated in the parity. The gf_Nvect_dot_prod versions build several parities in one pass over the data, so the data is read from memory only once |
| 27-7 | The two ISA-L encoding matrices for k=4, m=2 and the resulting parity for the sample byte D = [0x37, 0x5C, 0xA1, 0x0F]. The top four rows are the identity and carry the data unchanged to shards 0 to 3. The bottom two rows are the parity coefficients. In Vandermonde, row P is all 1, so P is XOR and the fast xor_gen path is used for m=1. In Cauchy all the parity coefficients are non-trivial |
| 27-8 | A 4K write inside shard D1 in a pool with k=4, m=2 on the parity delta path. The primary reads only that extent from D1 and the two parities. It computes Δ with encode_delta, and with apply_delta XORs the column-1 coefficient times Δ into each parity. It then sends the three writes in one distributed transaction. Shards D0, D2 and D3 are not involved |
| 27-9 | How parse_kml turns the three parameters k=4, m=2 and l=3 into the mapping and layers strings. In mapping, the letter D marks data and the underscore a coded chunk. In each layer, c means a chunk that layer produces. The global layer builds parities 2 and 6 with RS, and each local layer an XOR-like parity for the three chunks of its group. With crush-locality=rack, the CRUSH rule chooses two racks and four hosts in each rack, so that local repair does not leave the rack |
| 27-10 | The Clay sub-chunk grid with k=4, m=2 and d=5 for three different lost nodes. The rows are the nodes with coordinates (x, y) in the q×t grid, and the columns are the sub-chunk indexes. The red row is the lost node, and the blue cells are the sub-chunks each helper sends. The pattern comes from get_repair_subchunks: the larger y is, the shorter and more scattered the extents. So the reads on disk become more fragmented, and this is the hidden cost of Clay on HDDs |
| 28-1 | The complete mapping pipeline. The object name is hashed with rjenkins to 0xD1895615. With pg_num=128, ceph_stable_mod gives 0x15, that is PG 3.15. With the HASHPSPOOL flag, the pool number is also mixed into the hash so that the PGs of different pools do not overlap, and x = 0x36D15C32 is the CRUSH input. crush_do_rule returns the raw list of OSDs, then non-existent OSDs are removed and upmaps applied, and finally pg_temp determines the acting set |
| 28-2 | A real straw2 run for x=0x36D15C32 in a root bucket with three hosts: -3 and -4 each with weight 10 and -5 with weight 20. At r=0, host -4 wins with the largest draw (−8.9M). At r=1, host -5 wins. For the third replica, r=2 to r=5 land on repeated hosts and ftotal is increased each time, until r=6 reaches host -3. The retry limit is choose_total_tries=50 |
| 28-3 | Executing replicated_rule. TAKE puts the root into the working vector w. CHOOSELEAF_FIRSTN with numrep=0 (that is, equal to size) chooses a host for each replica and then, recursively, an OSD inside it. EMIT appends the result to the output. In each choice r = rep + ftotal. If the result is a repeat or the OSD is rejected by is_out, ftotal is increased and the attempt is repeated. For a reweight between 0 and 1, is_out makes a probabilistic decision with another hash of (x, osd) |
| 28-4 | The three tools for correcting the distribution and the order in which they apply. reweight works through a probabilistic is_out and causes widespread movement. choose_args (weight-set) provides substitute weights for straw2 that can even differ per position (first replica, second, and so on), and the balancer uses it in crush-compat mode. upmap explicitly replaces the result after CRUSH. In the OSDMap the order of application is: crush_do_rule with choose_args, removal of non-existent OSDs, pg_upmap, pg_upmap_items and pg_upmap_primary |
| 28-5 | A real run of bucket_perm_choose for bucket -5 with four items and x=0x1234ABCD. Three Fisher–Yates steps build the permutation [3 2 0 1]. So r=0 reaches item 13, r=1 reaches 12, r=2 reaches 10 and r=3 reaches 11. The permutation is kept in crush_work so that later attempts with the same x are not recomputed |
| 28-6 | A real run of bucket_list_choose for bucket -6 with weights 1, 1 and 2. For r=0 the scaled value of item 22 is 3.90, which is more than its weight 2, so it is rejected. item 21 is chosen with 0.36 < 1. For r=1 and r=2 both of the last items are rejected and item 20 is chosen |
| 28-7 | A real run of bucket_tree_choose for four items with weights 1, 2, 3 and 4. The root n4 has weight 10. t=4.22 is greater than the weight of the left subtree (3), so the path goes right (n6, weight 7). There t=0.78 < 3, so leaf n5, that is item 32, is chosen. The same path was taken for r=1. The distribution over 400 thousand inputs was about 1:2:3:4 |
| 28-8 | The percentage of inputs that got a different destination when the bucket changed, over 200 thousand values of x with the real hash. Blue bar: adding a fifth item to four equal-weight items (theoretical minimum 20%). Orange bar: removing the second item and rebuilding the bucket (theoretical minimum 25%). Only straw2 reaches the theoretical minimum in both cases |
| 28-9 | The structure of OSDMap::calc_pg_upmaps. After computing the PGs of each OSD and each OSD's deviation from its weight target, the overfull and underfull sets are built in every round. Then, starting from the fullest OSD, first removing earlier upmaps and then adding a new pair is tried. Each change is accepted only if the balance score does not get worse |
| 28-10 | An MSR rule for EC 4+2 on three hosts. The six-slot output is divided into three strides of two, and the hosts of each stride are kept apart from the other strides. When osd.7 is out, index 3 stays empty in this round, and in the next round the descent is repeated from the root with a new try number. The replacement can be in the same host or in a host not used in another stride |
| 29-1 | The path of a write in krbd. The program writes to /dev/rbd0 or to a filesystem on it. The block layer passes requests to rbd_queue_rq through the blk-mq tag_set (with depth queue_depth, default BLKDEV_DEFAULT_RQ). rbd.c turns each request into an img_request and splits it into obj_requests according to the object size. Each obj_request is a ceph_osd_request that osd_client sends to the primary OSD using in-kernel CRUSH |
| 29-2 | The state machines in rbd.c. If exclusive-lock is enabled, the img_request first acquires the lock, then runs all obj_requests and waits. Each object write first updates the object-map if it is enabled, then writes to the OSD. In a clone, if the object does not exist yet, copyup sends the parent's data along with the write, and at the end the object-map is finalized. A read that meets -ENOENT in a clone goes to the parent |
| 29-3 | Caps coordination in kernel CephFS. Client A opens a file for writing and, because it is the only writer, gets full caps, including Fc (read cache) and Fb (write buffer). So the writes stay in the page cache. When client B opens the same file, the MDS asks A to give back Fc and Fb. A writes its dirty pages directly to the OSDs and reports the size and mtime to the MDS. Then B gets only Fr and performs every read directly and synchronously from the OSD |
| 29-4 | Comparing kernel and user-space clients. krbd and ceph.ko are built on libceph, which is an independent implementation in C whose features are limited by the kernel version. librbd and ceph-fuse are built on librados, Objecter and AsyncMessenger, have all the features (journaling, mirroring, QoS, cache) and are upgraded without a reboot |
| 29-5 | Striping computation in the kernel. The offset is converted to a block number (su), a stripe number and an object set. In the example, a 3MiB read from offset 10.5MiB with a 1MiB/4/4MiB layout breaks into four pieces in four different objects. The bottom of the figure shows the object-name format in RBD v2, RBD v1 and CephFS |
| 29-6 | Choosing the OSD in libceph. calc_target builds the raw seed from the hash of the object name, maps it to pg_num with ceph_stable_mod, builds pps with crush_hash32_2 and the pool id, and applies CRUSH and then upmap and primary_temp. With every new osdmap, scan_requests re-evaluates in-flight requests and resends them if the target changed. A request with no reachable OSD waits in homeless_osd |
| 29-7 | Mapping a 256 KB write on a krbd device with custom striping. The top row shows the pieces of the request in the image space and the destination object of each. The bottom row shows the same pieces inside the four objects. The two pieces of object 0, which are 224K apart in the image space, fall next to each other inside the object, and ceph_file_to_extents turns them into one extent. So one obj_request with two bio_vecs is built |
| 29-8 | The decision branches of calc_target in libceph. The top row reads the inputs from the new map: the existence of the pool, the force-resend counter and the new PG mapping. The middle row evaluates three kinds of change. The bottom row is the result: every meaningful change detaches the request from the previous session and attaches it to the new OSD's session. Unrelated changes cost nothing |
| 29-9 | The handle_timeout timeline in osd_client. The blue dashed lines are the five-second rounds. Request A finished on time. Request B is counted as slow from the tenth second on, and a keepalive is sent to its OSD. The idle session of osd.3 is kept until lru_ttl passes and is then closed in the next handle_osds_timeout round |
| 29-10 | The two branches of lock acquisition in krbd. If the current owner has a live watcher on rbd_header, krbd asks it with a notify to release the lock and waits. If there is no watcher, the owner is assumed dead: its address is blocklisted in the mon so that it cannot finish a pending write, and then the lock is broken and acquired. The order of blocklist before break_lock is vital to prevent two hosts writing at the same time |
| 29-11 | The layout of the cap bits in an integer, and the example 0x3F55. Bit 0 is PIN. AUTH, LINK and XATTR each have two bits (shared and exclusive). FILE has eight bits from bit 8 on: s, x, c, r, w, b, a and l. The highlighted cells are the active bits of the example. ceph_cap_string turns these bits into the string seen in debugfs and the MDS logs |
| 30-1 | Comparing the execution models of the classic OSD and Crimson. In the classic OSD each request passes through several thread pools and several locks. In Crimson each core has an independent reactor that runs both the network and its own PGs. Communication between cores happens only through explicit messages (submit_to and invoke_on), and the result returns to the originating core as a future |
| 30-2 | The path of a request on one reactor. No step waits. Every asynchronous operation returns a future, and the continuation is queued on the same core once the result is ready. Because scheduling is cooperative, each task must finish quickly. Crimson sets each task's time slice with crimson_reactor_task_quota_ms (default 0.5 milliseconds in main) |
| 30-3 | Mapping PGs to cores in Crimson. If a core does not find the PG in its local map, it asks core 0. For a new PG, core 0 chooses the least-loaded core with min_element over the PG count of each core, increments the counter and broadcasts the map to all cores. From then on all the work of that PG runs only on that core |
| 30-4 | The phases of a ClientRequest in Crimson. The connection level (waiting to become active, getting the map and finding the PG's core), the shard level (creating or waiting for the PG), the PG level (PG ready, OSDMap version gate, PG active and getting the object context) and the object level (processing, waiting for scrub, waiting for replication to the replicas and sending the reply). Exclusive phases guarantee order, and concurrent ones make overlap possible |
| 30-5 | Crimson's storage backends. SeaStore and CyanStore are native and run directly on the reactor. AlienStore runs the classic BlueStore or MemStore in "alien" threads (outside Seastar). The request is sent from the reactor to a thread pool, and the result returns to the originating core with seastar::alien::submit_to |
| 30-6 | The layers of SeaStore. The logical managers (onode with the FLTree, omap and xattr with a BTree, object data and collections) work on the logical address laddr. The TransactionManager gathers the changes in a Transaction and commits them as one atomic record. The LBAManager keeps the laddr-to-paddr mapping in a B-tree, and the BackrefManager keeps the reverse mapping for GC. The Journal, ExtentPlacementManager, Cache and AsyncCleaner work on segmented, RBM or ZBD devices |
| 30-7 | The 64-bit structure of paddr_t in seastore_types.h. The top eight bits are the device_id (DEVICE_ID_BITS = digits(uint8_t) = 8). On a segmented device, the next 24 bits are the segment number within the device and the low 32 bits are the offset within the segment. On RBM, the low 56 bits are an absolute offset. The top seven device_id values are reserved for symbolic addresses such as an offset relative to a record or a delayed address |
| 30-8 | The structure of a journal segment and a record group. Each record group is committed with one disk write and contains a record_group_header_t (number of records, metadata and data length, segment nonce, committed_to and the data crc), then the header of each record (transaction type, number of deltas and extents, modification time), the deltas and the inline extents. RecordSubmitter gathers up to 16 records in a batch and keeps at most 5 concurrent writes on the journal (the main defaults) |
| 30-9 | The extent state machine in SeaStore's Cache. A new extent is created in INITIAL_WRITE_PENDING and becomes CLEAN after commit. Modifying an existing extent creates a version in MUTATION_PENDING that becomes DIRTY after commit, with the delta recorded in the journal (the in-memory content differs from disk and is pinned in memory). Extents whose metadata has not been committed yet are EXIST_CLEAN and EXIST_MUTATION_PENDING. DIRTY extents become CLEAN again when they are rewritten by the cleaner or by the journal trim |
| 31-1 | Comparing the two paths from BlueStore to the NVMe controller. In the kernel path, the request passes through a syscall, the blk-mq layer and the nvme-pci driver, and completion is reported with an MSI-X interrupt. In the SPDK path all this work is done in user space: the SPDK driver writes the SQE into the queue itself, updates the doorbell through the device's mapped BAR and finds the completion by polling. Both paths meet at the same hardware structure (SQ and CQ in host memory) |
| 31-2 | The 64-byte structure of a Read/Write command in the submission queue, with the values of a 4 KiB read from offset 1 GiB. opcode=0x02 means Read (0x01 is Write). command_id is an identifier the driver chooses and the controller returns in the CQE. slba is the starting logical address in blocks, and length is the number of blocks, zero-based. prp1 and prp2 are the physical addresses (or IOVAs behind the IOMMU) of the data pages |
| 31-3 | The 16-byte structure of a completion queue entry. sq_head tells the host how far into the SQ the controller has read. sq_id and command_id together identify the command uniquely. Bit zero of the status field is the Phase Tag: the controller writes it as 1 on the first pass through the queue, 0 on the second, and so on |
| 31-4 | The full sequence of a command: writing the SQE into slot sq_tail, updating the SQ Tail Doorbell, the controller fetching and executing the command, writing the CQE, an interrupt (or polling), walking the CQ and updating the CQ Head Doorbell. The doorbell offsets are derived from the CAP.DSTRD register |
| 31-5 | The NVMEDevice code path in three stages. On open, the transport ID string is read from the file and parsed with spdk_nvme_transport_id_parse. NVMEManager creates a separate thread for the DPDK environment and the probe. On write, the data is split into 128 KiB pieces and collected in a linked list of Tasks in the IOContext. aio_submit sends the commands on a qpair dedicated to that thread (thread_local). The completion is found by polling on the same thread, and the DMA buffers return to the pool |
| 31-6 | The components of the NVMe-oF gateway. The SPDK target (nvmf_tgt) receives NVMe/TCP commands from the host and hands each namespace to a bdev of type rbd. This bdev works with asynchronous librbd (rbd_aio_readv and rbd_aio_writev). The gateway's gRPC service keeps its configuration in an omap object in RADOS. NVMeofGwMonitorClient talks to the mon and gets the ANA state of each group from NVMeofGwMap |
| 31-7 | The bytes of NVMe/TCP. Each PDU starts with an 8-byte common header (type, flags, hlen, pdo, plen). The Command Capsule carries the same 64-byte SQE, and if the data is smaller than in_capsule_data_size, the data also comes inside the capsule. For larger data, the target grants permission to send with an R2T and the host sends the data in H2CData |
| 31-8 | The state machine of each gateway for each ANA group in gw_states_per_group_t (NVMeofGwTypes.h): IDLE, STANDBY, ACTIVE, OWNER_WAIT_FAILBACK_PREPARED, WAIT_FAILBACK_PREPARED and WAIT_BLOCKLIST_CMPL. The state exported to the host has only two values: OPTIMIZED for ACTIVE and WAIT_BLOCKLIST_CMPL, and INACCESSIBLE for the rest |
| 31-9 | The failover sequence. After mon_nvmeofgw_beacon_grace (default 7 seconds) without a beacon, gateway A is declared unavailable. find_failover_candidate finds the least-loaded gateway, A is blocklisted and B is put in WAIT_BLOCKLIST_CMPL with a 30-second timer. When the new OSDMap reaches B, the group becomes active for B and the host's ANA is updated |
| 32-1 | The distribution of the next scrub time with the main defaults. Left: shallow with osd_scrub_min_interval=1 day and ratio=0.5, uniform between 24 and 36 hours after the previous scrub. Right: deep with osd_deep_scrub_interval=7 days and cv=0.2, normal with a mean of 7 days and a deviation of 1.4 days, truncated to [4.2, 9.8] days. The truncation creates two small peaks at the edges, each with a probability of about 2.3%. The dashed line is the PG_NOT_DEEP_SCRUBBED warning threshold |
| 32-2 | The ScrubMachine state machine in scrub_machine.h. The primary side goes from NotActive to PrimaryActive, takes the replica reservations in Session and runs the chunk cycle in ActiveScrubbing: PendingTimer (sleep between chunks), NewChunk (choosing the range), WaitPushes and WaitLastUpdate (waiting for pushes and in-flight writes), BuildMap, WaitReplicas and the comparison. The PG finishes with WaitDigestUpdate. The replica side has the states ReplicaIdle, ReplicaWaitUpdates and ReplicaBuildingMap |
| 32-3 | The sequence of one chunk. The primary gets the list of objects in the range from the PGBackend, adjusts the boundary so that a head and its clones stay in one chunk, locks the range and sends MOSDRepScrub to the replicas. Each side builds its own map (in deep, reading the data in 4 MiB steps). Then the primary compares the maps. A client write in the locked range preempts the chunk |
| 32-4 | The logic for choosing the authoritative copy in ScrubBackend::select_auth_object. The candidates start with the primary itself. Any shard with a read error, a stat error, an EC hash or size mismatch, or a damaged object_info or snapset is rejected. Among the usable shards, the larger object_info version wins. On a tie, the shard that recorded more digests wins, and on a complete tie, the first one (the primary) |
| 32-5 | The recovery and backfill states inside Active in PeeringState.h. Both paths first take a local reservation (on the primary) and then a remote reservation (on each target). If the reservation is refused or revoked, the PG goes to NotRecovering or NotBackfilling. If the target is full, it gets the toofull state and retries after osd_recovery_retry_interval or osd_backfill_retry_interval (both 30 seconds) |
| 32-6 | The priority ranges in osd_types.h. Each base has a ceiling (get_max_prio_for_base) so that a pool's priority cannot lift a PG into a higher class: normal backfill 100 to 139, backfill with missing replicas 140 to 179, recovery 180 to 219, and an inactive PG (below min_size) 220 to 253. The values 254 and 255 are for force-backfill and force-recovery. The PG-deletion priorities (179, 219, 255) also compete in this same queue |
| 32-7 | The backfill reservation sequence. The primary first takes a slot of its own local_reserver and then, with MBackfillReserve REQUEST, a slot of the target's remote_reserver. Both are limited to osd_max_backfills. A higher-priority request on the target takes back a lower-priority preemptible reservation, and the primary receives a REVOKE message |
| 32-8 | The merge logic in recover_backfill. The primary keeps one interval of its own objects (backfill_info) and one interval from each target (peer_backfill_info), scanned 64 to 512 objects at a time (osd_backfill_scan_min/max). Both are sorted in hobject order. An object that is only on the target is deleted. An object whose version differs is pushed. An object whose version is equal is kept. Objects after the target's last_backfill are pushed as missing |
| 32-9 | The flow of ceph pg repair. The scrub target's urgency goes to must_repair and a full deep scrub runs. For each inconsistent object, ScrubBackend::repair_object uses force_object_missing to add the damaged shards to the missing list and makes only the healthy shards the source in missing_loc. Then the normal recovery mechanism pushes the authoritative copy |
| 33-1 | The most important ceph-osd threads, with the names seen in top -H or perf. msgr-worker-N (from Stack.h) read and decode messages. tp_osd_tp (the OSD::osd_op_tp thread pool) runs ops from ShardedOpWQ. bstore_aio receives I/O completions from the kernel. bstore_kv_sync commits transactions to RocksDB in batches. bstore_kv_final and cfin run callbacks. bstore_mempool tunes memory, rocksdb:low/high do compaction and flush, and osd_srv_heartbt sends heartbeats |
| 33-2 | An example of aliasing in the PG-to-shard mapping. PGs whose ps values are equal modulo 8 (3, 11, 19, 27) all land in shard 3. If these PGs are busy, the two threads of shard 3 saturate while the other 14 threads are idle |
| 33-3 | The structure of a shard_t in mempool.h. Each shard is an array of stats_t for all pools and starts with aligned(128) so that two shards never share a cache line (the code comment takes 128 bytes as the largest cache-line size on known processors). The total of a pool is the sum over all shards |
| 33-4 | The levels of tcmalloc. A small allocation is served from the same thread's cache without a lock. A shortfall is refilled from the transfer cache and the central free list, and these get spans from the page heap. Free page-heap memory is given back to the operating system with madvise and shows up in the pageheap_unmapped_bytes counter. PriorityCache computes "mapped" as heap_size minus this unmapped amount |
| 33-5 | A simulation of eight tune_memory rounds with the defaults and hypothetical input values for mapped. From 1500 MiB, because mapped=3000 is below the target, the budget rises to 1824.7. In the next round mapped=4500 is above the target and the budget drops to 1672.4. The budget never goes above max=2713.6 or below min=128 |
| 33-6 | The result of the balance example after PRI0 and two rounds of PRI1. kv has reached 500 MiB, meta 723.3, data 96.4 and kv_onode 50 MB. Because the meta and data caches still have more fresh data than their budget, their older bins (the lower priorities) are evicted at the next commit |
| 33-7 | The NUMA affinity decision in OSD::set_numa_affinity. The OSD finds the NUMA node of the storage device (get_numa_node) and the node of the public and cluster network interfaces (get_iface_numa_node). If all three are the same and osd_numa_auto_affinity=true, all threads are restricted to the CPUs of that node. osd_numa_node, if set explicitly, takes precedence over the automatic decision |
| 33-8 | The tools by layer. Inside Ceph: dump_mempools, heap stats, perf dump and dump_historic_ops. CPU sampling with perf and flame graphs. Event tracing and off-CPU time with bpftrace. Function-by-function tracing with uftrace. And the stacks of all threads with gdb or eu-stack for a stuck state |
| 33-9 | Left: the maximum and average pg log length over the 32 PGs of pool m33 against the number of writes, with each OSD's budget as a dashed line (30000 divided by the number of PGs). Right: the size of the osd_pglog mempool (solid line) and bluestore_cache_onode (dashed line) for each OSD. The dotted line marks the boundary between overwriting 1000 objects and creating new objects |
| 33-10 | The real cost per item (bytes divided by the number of items) in the main mempools of osd.0 on the Ceph 18.2.7 test cluster, together with the number of items and the total size of each mempool |
| 34-1 | The bitwise key and the PG ranges. The low five bits of the hash determine the PG number at pg_num=32. After reversing the bits, those five bits move to the top of the key, and each PG becomes a contiguous range in hobject_t order. When pg_num doubles, each range is split exactly in half |
| 34-2 | The op path inside the OSD from the messenger to commit. ms_fast_dispatch hands the message to the shard without a global lock; the shard's thread takes the PG lock and do_op checks the gates. If a condition does not hold, the op stays in the corresponding wait queue. Otherwise execute_ctx builds the transaction, issue_repop sends it to the replicas, and the reply is sent after all members of acting have committed |
| 34-3 | The sequence of a three-copy replicated write. The primary commits the transaction locally and at the same time sends it to the two replicas. Each member replies ONDISK after committing to the BlueStore WAL. The reply to the client is sent only after all replies have arrived; min_size only determines with how many members the PG accepts writes at all |
| 34-4 | The key layout of a pg log entry in the omap of the pgmeta object, byte by byte. Ten digits of epoch with leading zeros, a dot and twenty digits of version. The dup key has the same format with the four-byte prefix dup_ (the note below the figure). The fixed length makes string order the same as numeric order |
| 35-1 | The timeline of a PG's intervals. Every change in up, acting or min_size creates a new interval. Only intervals in which the primary obtained up_thru or the PG went clean are maybe_went_rw and count in the prior set. An interval whose acting set was smaller than min_size is ignored |
| 35-2 | Merging logs with a divergent entry. Member A wrote up to 1462'95, but the authoritative log of member B goes up to 1462'93 and then has 1470'94 and 1470'95 in the new epoch. The entries 1462'94 and 1462'95 on A are divergent; they are rolled back or removed, and the corresponding object goes into the missing set at the prior version |
| 35-3 | Backfill as the merge of two ordered scans. The primary's interval and the target's interval advance together. An object present only on the target is removed, an object present only on the primary or with a different version is pushed, and last_backfill advances after each round |
| 36-1 | The evolution of an object's SnapSet in three steps. Each first write after one or more new snapshots creates a clone named after the largest new snap. clone_snaps shows which snaps each clone represents, and clone_overlap shows the parts still shared with the newer version |
| 36-2 | The flow of deleting a snapshot. The mon adds the number to removed_snaps_queue, each PG finds and trims that snap's clones using the SnapMapper, and finally the mon counts it as purged |
| 37-1 | The layout of an image with a separate data pool. The replicated metadata pool holds rbd_id, rbd_directory, the header and the object maps, and the EC data pool holds the rbd_data objects prefixed with the metadata pool's id. Clients watch the header, and the exclusive lock is on an xattr of that same header |
| 37-2 | Striping with stripe_unit=64K and stripe_count=4. Each row is a stripe and the four columns are the four objects of an object set. Blocks 15 to 19, which make up the example write, are colored |
| 37-3 | The object map encoding of a ten-gigabyte image with 4 MiB objects (2560 objects), byte by byte. The eighteen-byte header contains the length, struct_v and compat, and the element count; then 640 bytes of data, each byte holding four objects with two bits each, and finally the footer with the crcs |
| 38-1 | The dispatch layers in librbd. An image request passes through nine image layers and is split into object extents by the Striper in CORE; each extent then passes through six object layers until it reaches RADOS as an op. Inactive layers (for example journal or crypto) are not in the chain |
| 38-2 | Copyup for the first write to object 1280 of a clone. librbd learns from the object map that the object does not exist, fetches the parent object with a sparse read, moves the object map to EXISTS, and sends the copyup and the user's write to the clone object in one atomic op |
| 38-3 | Acquiring exclusive-lock. If the lock is held, client B checks whether the owner still has a live watch. If it is alive, B sends a release request with notify; otherwise it first blocklists the owner and then breaks the lock |
| 39-1 | A journal event, byte by byte. A fixed eight-byte preamble for recovering event boundaries, the version, the entry and tag ids, the data length, the encoded librbd EventEntry data, and a crc32c over everything before it |
| 39-2 | The two mirroring modes side by side. In journal mode each write is first recorded in the source journal and rbd-mirror continuously replays the events on the destination. In snapshot mode the scheduler creates mirror snapshots and rbd-mirror copies only the objects changed between two snapshots, using fast-diff |
| 40-1 | The special keys in the omap of an index shard. The instance key is built from 0x80, the prefix 1000_, the name, a zero byte, the letter i and the version id; the OLH key has only the prefix 1001_ and the name; the bilog key has an eleven-digit zero-padded index_ver so that lexical order equals numeric order |
| 40-2 | The sequence of the two-phase update: prepare puts the operation tag in pending_map, the data is written, and complete removes the tag and updates the entry, the statistics and the bilog. If complete never arrives, the next listing checks the head's state and fixes the entry with dir_suggest_changes |
| 40-3 | The layout of a 10 MiB object in two cases. In STANDARD, the head holds the first four megabytes and two tails the rest. In COLD, the head keeps only the manifest and xattrs in the STANDARD pool, and all the data is in three tails in the COLD pool |
| 41-1 | The pipeline of a request in RGW: Beast, choosing the handler and op, authentication, reading permissions, checks, rate limiting, execution and completion. An error at any stage goes straight to the HTTP response |
| 41-2 | Deriving the SigV4 key: four chained HMACs from the secret key with the date, region, service and aws4_request, and a fifth HMAC over the string to sign, which itself contains the hash of the canonical request |
| 41-3 | The scheduling of gc and lifecycle. Deleting an object puts the tail chain in the queue of a gc.N object with a two-hour expiry, and the gc thread clears it after locking the shard. The lifecycle thread locks the lc.N shards in the daily work window and runs the rules for each bucket |
| 42-1 | The chain of three logs for a PUT in the source zone. complete in the index writes a bilog entry, RGW records the bucket shard key in the datalog, and the destination zone reads the datalog, fetches the bilog of that shard from its marker onward and copies the object with fetch_remote_obj. Metadata changes are read separately from the mdlog of the current period |
| 42-2 | The transition between bilog generations after a reshard: generation 0 with 11 shards and generation 1 with 23 shards. The destination zone syncs all shards of generation 0 to their end and marks them in shards_done_with_gen, then moves incremental_gen to 1; the generation 0 log is trimmed only after all peers have finished |