Appendix A
English Glossary
Key terms used in the book, in English alphabetical order, each with a short definition. Use the index at the end of the book to find where each term is used.
| Term | Definition |
|---|---|
| Acting set | The set of OSDs that actually serve a PG; it may temporarily differ from the up set |
| Admin socket | Each daemon's local management socket for direct commands (ceph daemon) |
| AES-GCM | Authenticated encryption in msgr2 secure mode (16-byte key, 12-byte IV and 16-byte tag) |
| Allocator | The raw-space allocator in BlueStore (hybrid by default) |
| ANA | Asymmetric Namespace Access state in NVMe-oF, which tells the host which path is optimal |
| Backfill | Full rebuild of a PG on a new OSD by walking all of its objects |
| Balancer | The mgr module that evens out PG distribution (upmap mode) |
| Banner | The opening string of an msgr2 connection ("ceph v2") with the supported and required features |
| Bilog (bucket index log) | The per-shard change log of a bucket index, read by multisite for incremental sync; kept in the index (InIndex) or in a separate FIFO |
| Bilog generation | The generation of a bucket's log layout after a reshard; peers finish the old generation before moving to the new one |
| Blob | The unit of checksumming and compression in BlueStore, which points to pextents |
| Blocklist | The OSDMap list of blocked clients, used to stop late writes |
| Bloom filter | A probabilistic membership structure with false positives but no false negatives; used in RocksDB and HitSet |
| BlueFS | BlueStore's small internal file system for RocksDB |
| BlueStore | The default OSD storage backend, which writes directly to the raw device |
| Bucket index | The list of a bucket's objects in RGW, kept in the omap of the index shards |
| bufferlist | A chain of bufferptrs over reference-counted buffers; the main data unit in Ceph code |
| Cache tiering | A cache layer between two pools; deprecated since Reef |
| Capability (cap) | A caching and mutation permission that the MDS grants a CephFS client for each inode |
| ceph-volume | The tool that prepares disks and creates OSDs (LVM) |
| cephadm | A tool and mgr module that deploys clusters and manages their life cycle with containers |
| Cephx | Ceph's authentication protocol, based on shared keys and tickets |
| Clone | A writable RBD image created from a snapshot of a parent image |
| cls_lock | A RADOS class for advisory exclusive or shared locks with expiry, stored in an xattr |
| Collection | The group of objects of one PG in the ObjectStore |
| Copy-on-write | Parent data is copied only when it is first modified |
| Copyup | Copying the whole parent object to the child on the first write to a clone |
| crc32c | A checksum with the Castagnoli polynomial, used by BlueStore, msgr2 and journals |
| Crimson | A rewrite of the OSD on Seastar, with one shard per core and no shared locks |
| CRUSH | The pseudo-random, deterministic algorithm that places data on OSDs according to a hierarchy |
| CRUSH rule | The rule that selects OSDs for a pool (failure domain, device class) |
| Datalog | The change log of bucket shards used for multisite synchronisation |
| Deep scrub | A full data check that reads the data and compares checksums |
| Deferred write | A small write that is first recorded in the WAL and applied to the device later |
| Delete marker | A data-less version created by a delete without versionId in a versioned bucket, which becomes the current version |
| Dirfrag | A fragment of a CephFS directory; the unit of splitting and migration between MDSs |
| Divergent entry | A PG log entry on a replica that is not in the authoritative log and must be rolled back or removed during merge_log |
| Dynamic resharding | Automatic increase of the index shard count of a large bucket in RGW |
| Envelope mode (BlueFS) | A new on-disk record format for BlueFS files, present in Tentacle and later |
| Erasure coding | Splitting data into k data chunks and m parity chunks |
| Error repo (sync) | The omap of the .retry object beside each datalog shard status, holding failed bucket shards for retry |
| Exclusive lock | A cooperative RBD lock that ensures a single active writer |
| Failure domain | A level of the hierarchy (host, rack) across which replicas are spread |
| Fast-diff | An RBD feature that computes differences quickly using the object map |
| Flatten | Detaching a clone from its parent by copying all data |
| Frame preamble | The 32-byte header of every msgr2 frame, with tag, segment lengths and crc |
| Full ratio | The OSD fullness threshold above which writes stop |
| Garbage collection (RGW) | The queue of tails of deleted or overwritten objects in gc.N objects, processed after rgw_gc_obj_min_wait |
| GF(2⁸) | The 256-element finite field in which Reed-Solomon codes multiply and add (XOR) |
| GFNI | Galois-field instructions in x86 processors that perform GF(2⁸) multiplication in hardware |
| HitSet | The set of objects accessed in a time interval, used to judge object temperature in cache tiering |
| hobject_t | The full object identifier in the OSD (pool, namespace, key, name, snap, hash) with the bitwise order that listing and backfill rely on |
| Incremental map | The difference between two consecutive map epochs, sent to subscribers |
| io_uring | The Linux kernel asynchronous I/O interface; an optional replacement for libaio in BlueStore |
| Journal (RBD) | A sequential record of image changes for journal-based mirroring |
| Knapsack | The knapsack problem; the Umbrella autoscaler uses it to decide which pool is rounded up to the next power of two |
| krbd | The kernel RBD driver (rbd.ko) that exposes an image as the block device /dev/rbdN |
| Label (BlueStore) | The 4096-byte block at the start of the device with the uuid, size and OSD metadata |
| Lease | A time-limited permission that the MON leader grants peons for reads |
| libaio | The older Linux AIO interface (io_submit and io_getevents), the default for KernelDevice in BlueStore |
| libceph | The shared Linux kernel library for the messenger, monclient, osd_client and CRUSH |
| librados | The native RADOS client library |
| librbd | The user-space client library for RBD block devices |
| Lifecycle (RGW) | Expiration and transition rules for objects, run by LCWorkers in a time window |
| Linger request | A persistent libceph or Objecter request for watch and notify, re-evaluated with every map |
| LSM tree | The log-structured merge tree on which RocksDB is built |
| Manifest | The map of the parts of an RGW object (head and tail) |
| mClock | The default OSD QoS scheduler, based on reservation, weight and limit |
| Mdlog (metadata log) | The RGW metadata change log (users, buckets), one per period, in meta.log.<period>.<N> objects |
| MDS | The CephFS metadata service |
| mempool | Ceph's per-data-type memory accounting; visible with dump_mempools |
| memtable | RocksDB's in-memory table before it is flushed to an SST |
| Mgr | The manager daemon that runs Python modules such as dashboard and prometheus |
| min_size | The minimum number of available replicas for a pool to accept I/O |
| Mirror snapshot | A special RBD snapshot in the mirror namespace on which snapshot-based mirroring copies the difference between two points |
| Mirroring | Asynchronous replication of RBD images or CephFS to another cluster |
| MMgrReport | The periodic message from daemons to the mgr carrying perf counters |
| MON | Monitor; keeps the cluster maps with Paxos consensus |
| MOSDOp | The message carrying a client's operation request to an OSD |
| msgr2 | Version 2 of the Ceph network protocol, with optional encryption (port 3300) |
| MSR | Multi-step retry CRUSH rule for placing EC data across several failure domains, from Squid |
| Multipart upload | Uploading an object in independent parts that complete joins with one manifest and no data copy; ETag carries a -N suffix |
| Multisite | The realm, zonegroup and zone structure for replicating RGW between sites |
| NVMe-oF | Access to NVMe namespaces over the network (TCP or RDMA); the Ceph NVMe-oF gateway exports RBD images this way |
| O_DIRECT | Opening the device without the page cache; BlueStore writes data this way |
| Object map | A bitmap of which objects of an RBD image exist |
| Objecter | The client component that sends an op to the right OSD and retries it |
| ObjectStore | The abstract local storage interface of the OSD |
| OLH (Object Logical Head) | The pointer to the current version of an object in a versioned RGW bucket; a 1001_ index entry plus a data-less head with user.rgw.olh.* xattrs |
| omap | Key-value storage attached to each RADOS object |
| Onode | The metadata structure of one object in BlueStore |
| OSD | Object storage daemon; usually one per disk |
| OSDMap | The map of the state of OSDs, pools and placement exceptions |
| PastIntervals | A summary of intervals in which a PG's acting set was stable; peering uses it to find the OSDs it must query |
| Paxos | The consensus algorithm used by the MONs |
| Peering | The process by which the OSDs of a PG agree on its state and history |
| Perf counter | A performance counter inside a daemon (u64, time, avg or histogram) |
| Period | A committed version of the multisite configuration with an id, epoch and realm_epoch; changing the master zone creates a new period |
| PG (Placement Group) | Placement group; the logical unit of object distribution and recovery |
| PG log | The recent history of changes of each PG, used for fast recovery |
| pg_autoscaler | The mgr module that automatically tunes the PG count of each pool |
| pg_temp | A temporary acting-set mapping in the OSDMap during backfill |
| Placement target | A zonegroup rule that maps a bucket to the index pool, data_extra_pool and per-storage-class data pools in a zone |
| Pool | A logical space with its own replication policy, CRUSH rule and PG count |
| Primary affinity | An OSD's affinity factor for becoming primary |
| Primary OSD | The first OSD of the acting set, which coordinates client requests |
| PriorityCache | The memory budget manager that tunes the BlueStore and RocksDB caches to osd_memory_target |
| PSHUFB | The SSSE3 byte-shuffle instruction that ISA-L uses with 16-entry tables for GF(2⁸) multiplication |
| PWL (persistent write log) | librbd's persistent write-back cache on SSD or PMEM that logs writes in order and flushes them to the cluster later |
| Quorum | The majority of available MONs that can make decisions |
| RADOS | Ceph's foundation layer: distributed, self-healing object storage |
| Rate limit (RGW) | An in-memory token bucket per RGW for accounts, users and buckets that rejects excess requests with 503 SlowDown |
| RBD | The Ceph block device |
| Recovery | Restoring missing object replicas based on the PG log |
| Reed-Solomon | An MDS erasure code that tolerates any m failures with k data chunks and m parity chunks |
| Replication | Full copies of data in several replicas |
| Reweight | A temporary factor (0 to 1) that reduces the data on an OSD |
| RGW | The object gateway, compatible with S3 and Swift |
| Rjenkins hash | Bob Jenkins' hash function, used for object names and CRUSH input |
| Rook | A Kubernetes operator for deploying and managing Ceph |
| Scrub | Periodic consistency check of the metadata of a PG's replicas |
| Seastar | An asynchronous C++ framework with future/promise and one thread per core, on which Crimson is built |
| SeaStore | Crimson's new ObjectStore, designed for fast devices and the one-core-per-shard model |
| Shard | One part of partitioned data, e.g. an index shard, an EC shard or an OSD queue shard |
| SigV4 | AWS Signature Version 4; HMAC-SHA256 over the string to sign with a key derived from the secret, date, region and service |
| Skiplist | A probabilistic multi-level linked list; the default memtable structure in RocksDB |
| SnapMapper | The omap mapping between snapshots and objects (SNA_ and OBJ_ keys) that snap trim uses to find clones |
| SnapSet | The snapshot metadata of a RADOS object on its head (seq, clones, clone_overlap, clone_size) used to resolve snapshot reads |
| Snapshot | A read-only image of a point in time |
| SPDK | Storage Performance Development Kit; a user-space, polling NVMe driver that BlueStore and Crimson can use instead of the kernel |
| SST | RocksDB's immutable sorted table file |
| stable_mod | The seed-to-PG mapping that moves only part of the objects when pg_num grows |
| Storage class | An RGW storage class, mapped to a separate data_pool in the placement target |
| straw2 | The CRUSH bucket selection algorithm that draws a "straw" as ln(hash)/weight |
| Stretch mode | A cluster stretched between two data centres, with a tiebreaker MON |
| Striping | Splitting contiguous data across several objects |
| Superblock (BlueFS) | The BlueFS header at offset 4096 of the device, with log_fnode and version |
| Throttle (BlueStore) | A limit on the cost of in-flight transactions, in equivalent bytes |
| Tiebreaker | The arbiter MON in stretch mode that breaks quorum ties |
| Up set | The set of OSDs computed by CRUSH before pg_temp is applied |
| Upmap | An explicit PG placement exception in the OSDMap, used for balancing |
| Volume selector | The BlueFS policy that chooses the WAL, DB or slow device for each RocksDB file |
| vstart.sh | A script that starts a local development cluster from the build tree |
| WAL | The RocksDB write-ahead log in BlueStore |
| Watch/Notify | The RADOS mechanism where clients register a watch on an object and a notify delivers a message to all and waits for acks |
| WriteBatch | An atomic batch of Put and Delete operations in RocksDB |
| Zone | One RGW deployment with its own pools in multisite |
| Zonegroup | A group of zones that replicate data among themselves |