Appendix E

Answers to Review Questions

1 h 13 min read

The answers below are short and indicative; for full explanations go back to the corresponding chapter.

Chapter 1: Ceph: A New Era — Architecture, Planning and Your First Cluster#

1. What are the three fundamental limitations of traditional storage arrays that software-defined storage removes?
Vertical scaling limited to a pair of controllers, dependence on one vendor's hardware and software, and data islands (a separate product for block, file and object). Ceph removes them by scaling horizontally on ordinary servers and serving all three interfaces on one shared RADOS.
2. Why does RAID rebuild time get worse as disks get bigger, while Ceph recovery time gets better as the cluster gets bigger?
A RAID rebuild is limited by the write speed of one spare disk, so its duration grows linearly as disks get bigger. In Ceph the data of a failed OSD is spread over hundreds of PGs, and all healthy OSDs read and write in parallel; the more OSDs there are, the smaller each one's share and the faster the repair.
3. Compare the roles of the MON and the OSD in the write path of an object. Does data pass through the MON?
The MON only gives the cluster map to the client and takes part in the consensus on map changes. Data does not pass through the MON; the client writes directly to the primary OSD, and the primary OSD coordinates replication to the secondaries.
4. What problem does the PG layer solve, and what would happen if it did not exist?
The PG aggregates billions of objects into a few thousand groups, so that peering, recovery and scrub happen at the group level. Without it, OSDs would have to track the state of each object separately, and the memory and messaging overhead would become unbearable.
5. Why is the number of MONs chosen to be odd? How many failures do four MONs tolerate?
A majority (quorum) is needed to make decisions; an odd number gives the highest failure tolerance for the fewest members. Four MONs need a majority of three and tolerate only one failure, the same as three MONs.
6. What is the difference between size and min_size, and why is min_size=1 dangerous?
size is the desired number of copies, and min_size is the minimum number of available copies needed to accept I/O. With min_size=1 a write may be acknowledged on only one OSD, and if that OSD fails, acknowledged data is lost.
7. When is a separate cluster network useful, and what risk does it carry?
Separation is useful when replication and recovery traffic is heavy enough to compete with client traffic. The risk is that if the cluster network partly fails, OSDs report each other down while the MON sees them as alive, and flapping follows.
8. Why is an SSD without PLP unsuitable for Ceph?
Ceph flushes on every durable write; an SSD without PLP must really write its cache to flash and becomes very slow, and on a power loss it may lose acknowledged data.
9. Which files does cephadm create in /etc/ceph during bootstrap, and which one is the most sensitive?
A minimal ceph.conf, ceph.client.admin.keyring and ceph.pub. The most sensitive file is the admin keyring, which gives full access to the cluster.
10. Which two Umbrella default changes can alter the cluster's behaviour after an upgrade?
The increase of mon_target_pg_per_osd from 100 to 200 and of mon_max_pg_per_osd from 250 to 500, which can make pools grow their PG count after the upgrade. (The change to the mClock scheduler's shard settings for HDD is another example.)

Chapter 2: Working with the Ceph Block Device (RBD)#

1. How many RADOS objects does a newly created 100 GB image that has not been written to yet have? Why?
Almost no data objects; only metadata objects (the header, the id and the directory entry, and the object map if it is enabled). RBD is thin, and a data object is created only after the first write to its corresponding range.
2. What is the difference between krbd, librbd and rbd-nbd, and where is each one appropriate?
krbd is the kernel driver and creates a /dev/rbdX device; it is fast and suits physical hosts and Kubernetes, but its features depend on the kernel version. librbd is the complete user-space library that QEMU and OpenStack use. rbd-nbd builds a kernel block device on top of librbd; use it when you need a feature krbd lacks.
3. Why does the object-map feature depend on exclusive-lock?
object-map keeps the existence state of objects in the client's memory and is valid only when a single client is the only writer; exclusive-lock guarantees exactly that exclusivity.
4. How does a crash-consistent snapshot differ from an application-consistent one, and how do we get the latter?
crash-consistent is like a power cut: whatever was on disk is recorded, but buffers and half-finished transactions are not. application-consistent is achieved by freezing the file system (fsfreeze or the guest agent) or by the application's own mechanism (a database's backup mode) before the snapshot.
5. What changed in clone format version 2 compared with version 1?
In version 2 snap protect is not needed, and a parent snapshot that has children can be "deleted"; the snapshot is moved to the trash until the last child is flattened or deleted. It requires at least Mimic-compatible clients.
6. Compare the journal and snapshot mirroring modes in terms of RPO, overhead and compatibility with krbd.
journal: RPO of a few seconds, roughly double the writes, librbd only. snapshot: RPO equal to the schedule interval, low overhead, also works with krbd.
7. What is the difference between --direction rx-only and rx-tx in bootstrap import?
rx-only means the local cluster only receives (one-way). rx-tx means both clusters can be source and target (two-way), and no reconfiguration is needed after a failover.
8. Why is a forced promote without fencing dangerous?
If the primary site is still alive and its clients are writing, two primaries with different histories come into existence (split-brain), and one of the two sets of changes has to be thrown away.
9. What are the failback steps after the primary site comes back?
Demote the image in the primary site, resync it from the current primary, wait for the up+replaying state, then demote in the backup site and promote in the primary site.
10. For a VMware host that has no Ceph client, which way of using RBD is recommended today?
The NVMe-oF gateway over TCP, with at least two gateways in a gateway group for HA; iSCSI is in maintenance mode.

Chapter 3: Ceph and OpenStack#

1. What are the three main reasons that made Ceph a popular backend for OpenStack?
One backend for all services (block, images, backup, file and object), end-to-end copy-on-write that makes VM creation instant, and shared storage that makes live migration and evacuate possible without moving disks; plus horizontal scaling and being open source.
2. Why does client.cinder have only read permission on the images pool?
Cinder and Nova only need to create clones from image snapshots; writing or deleting images is Glance's job. The principle of least privilege protects the images against mistakes or a compromise in cinder.
3. Which two conditions are needed for copy-on-write between Glance and Cinder?
The image must be stored in Glance in raw format, and show_image_direct_url = True must be set so that Cinder sees the RBD address; and naturally both services must be connected to the same Ceph cluster.
4. What role does the secret in libvirt play, and why must its UUID be the same on all compute nodes?
QEMU gets the Ceph key from libvirt, and the domain XML only refers to the secret's UUID. During live migration the same XML goes to the destination host, so a secret with the same UUID must exist there too.
5. What does the images_type = rbd option change in Nova, and what advantage does it bring for evacuate?
The VM's root disk becomes an image in the vms pool instead of a file on the local disk. If the compute node fails, the VM can be evacuated to another compute node because its disk is safe on Ceph.
6. Which trade-off do rbd_flatten_volume_from_snapshot and rbd_max_clone_depth control?
The speed of volume creation (no flatten, deep clones) against chain dependencies and read performance. Flatten cuts the chain but takes more time and space.
7. Why is a backup on the same Ceph cluster not enough?
Because it gives no protection against a failure or mistake in that same cluster (accidental deletion, a bug, an attacker with the admin key). Backups must be in another cluster or site.
8. How do the two Manila drivers for CephFS differ?
The native driver uses the CephFS client directly in the guest (fast, but the guest needs network access to Ceph); the NFS driver exports shares through NFS-Ganesha (better isolation, one extra layer).
9. Why is disk_cachemodes = "network=writeback" safe?
Because librbd honours the guest's flushes, and with rbd cache writethrough until flush the cache stays in writethrough mode until the guest has sent a flush.
10. Why is it better to manage Ceph independently of the OpenStack deployment tool?
So that the life cycles and upgrades of the two systems are independent, a problem in one does not spread to the other, and Ceph can serve other consumers (Kubernetes, S3) at the same time.

Chapter 4: Object Storage with the RADOS Gateway#

1. Why do we call RGW "stateless", and what effect does this have on high-availability design?
Because no data or persistent state is kept in the radosgw process itself; everything is in RADOS. So any number of instances can be put behind a load balancer, and the failure of one instance affects only its in-flight requests.
2. Which objects does a 20 MB S3 object become in RADOS?
One head object (about the first 4 megabytes of data together with the metadata and manifest), several tail objects for the rest of the data, and an omap entry in one of the bucket index shards.
3. Why should the index pool not be EC or on HDD?
The index is kept in omap, which EC does not support; and every PUT and DELETE is an index update, so HDD latency directly becomes the latency of every operation.
4. What problem does dynamic resharding solve, and what is its default threshold?
It prevents index shards from growing too large (giant omap objects); when the number of objects per shard reaches rgw_max_objs_per_shard (default 100,000), the shards are increased automatically.
5. What is the difference between a tenant and an account?
A tenant is a namespace for buckets and users; an account (since Squid) is an AWS-style unit of ownership whose root user manages its own users, groups, roles and policies with the IAM API.
6. Which configuration is needed for virtual-hosted-style access?
A wildcard DNS record for the domain, setting rgw_dns_name (or hostnames in the zonegroup), and a wildcard TLS certificate.
7. How do the GOVERNANCE and COMPLIANCE modes of object lock differ?
In GOVERNANCE a user with the bypass permission can get around the lock; in COMPLIANCE no S3 user can delete the version or shorten the period until it ends.
8. Why is cluster space not freed immediately after a mass deletion of objects?
Only the head object is removed immediately; the tails are handed to the garbage collector and removed after rgw_gc_obj_min_wait.
9. What does the rgw_s3_auth_use_keystone option make possible?
It lets an S3 client authenticate with EC2 credentials created in Keystone, with RGW validating them against Keystone.
10. Why do S3A committers matter for Spark on RGW?
Because Hadoop's traditional commit works by rename, which in S3 means copying and deleting every object; the S3A committers remove this cost with multipart uploads that are completed only at the final commit, and write the output safely and quickly.

Chapter 5: Multi-Site Object Storage with RGW Multi-Site#

1. How do the roles of the zonegroup and the realm differ in replicating data and metadata?
The zonegroup is the boundary of data replication: bucket data is replicated only between the zones of one zonegroup. The realm is the boundary of the namespace and metadata: users and buckets are unique and in sync across the whole realm.
2. What is a period, and why must it be committed after structural changes?
A period is a named, epoch-numbered version of the realm configuration. Committing it publishes the new configuration to every zone so that they all agree on the structure and the master.
3. What are the three main differences between multisite version 2 and the federated configuration in Hammer?
Sync inside radosgw instead of the external radosgw-agent; active-active instead of a read-only slave; and the realm and period concepts for versioned configuration and failover.
4. Why does creating a bucket in a non-master zone fail when the master is lost, while writing an object does not?
Creating a bucket is a metadata operation that must be recorded in the master zone, and the request is forwarded to the master; writing an object in an existing bucket is local and is synced later.
5. What role does the system user play, and why must it be strongly protected?
Zones use its key to read each other's logs and objects; it has full access to the realm's data and metadata, so exposing it means exposing all the data.
6. What information does the phrase "oldest incremental change not applied" give in the output of sync status?
The time of the oldest change that has not yet been applied in this zone; its difference from the current time is an estimate of the real RPO.
7. Write down the steps of a failover and of returning the old master.
Failover: in the healthy zone zone modify --master --default, period update --commit and an RGW restart, then a DNS change. Return: in the old zone realm pull and period pull, check that it has caught up, and if desired make it master again with a period commit.
8. What is the difference between the enabled, allowed and forbidden states in sync policy?
enabled means sync happens; allowed means it is permitted but a lower level must enable it; forbidden means it is prohibited and a lower level cannot enable it either.
9. What problem does the archive zone solve?
Protection against accidental and malicious deletion or overwriting, by keeping every version in a separate zone, without requiring versioning in the user zones.
10. Why was resharding problematic in multisite before Reef, and how is it solved today?
A reshard changed the number and layout of index shards, and sync, which is based on each shard's bilog, was disrupted; since Reef, with the zone feature named resharding, each zone reshards independently without interrupting replication.

Chapter 6: The Ceph File System (CephFS)#

1. Why is the MDS not on the file data path, and what advantage does this have?
The MDS only manages metadata, and after granting a capability the client exchanges data directly with the OSDs; so data bandwidth scales with the number of OSDs and the MDS does not become a bottleneck.
2. What is a capability, and when does the MDS evict a client?
A capability is a permission the MDS grants a client on each inode (read, write, cache, buffer) and takes back when necessary. If the client does not respond within the set timeout, or its session expires, the MDS evicts it and puts it on the blocklist.
3. What is the difference between standby and standby-replay?
A standby is idle and must replay the whole journal when it takes over; a standby-replay continuously follows the journal of a specific rank and has a warm cache, so failover is faster.
4. Why should the default CephFS data pool not be EC?
Because the backtrace metadata of every file is written to the default data pool, and these small operations are inefficient on EC; add the EC pool as a second pool and use it with a layout.
5. When are multiple active MDSs needed, and why is explicit pinning recommended?
When the metadata load (number of files, clients or operations per second) exceeds what one MDS can handle. The automatic balancer has a history of unstable migrations, and explicit pinning gives predictable behaviour.
6. Compare the kernel client and ceph-fuse in terms of performance and currency.
The kernel client is faster but depends on the kernel version; ceph-fuse is slower but matches the Ceph version, is easier to debug, and its failures are limited to one process.
7. Why does NFS-Ganesha keep export configuration in RADOS?
So that all Ganesha daemons have the same centralised configuration, and adding or moving a daemon needs no file synchronisation.
8. Why are quotas in CephFS called "cooperative"?
Because the client itself tracks usage and stops writing; with a delay it may write slightly beyond the cap, and a malicious client can ignore it.
9. Why does changing a directory's layout not move existing files?
The layout is recorded on a file when it is created and does not change where existing objects are; a rename only changes metadata. To move data, the file must be copied.
10. What is the usual replacement for "Hadoop on CephFS" today?
Running Spark, Trino or Hive on RGW through the S3A interface; for POSIX workloads, mounting CephFS directly.

Chapter 7: Operating and Managing a Ceph Cluster#

1. Why is stopping a daemon with systemctl not recommended in cephadm?
Because the orchestrator does not know about the stop and may restart the daemon or report an error; for planned work, use ceph orch or maintenance mode.
2. What is the precedence order of configuration sources, and how do you see the effective value?
From lowest to highest: the compiled-in default, the MON configuration database, the local ceph.conf, command-line arguments and run-time changes. The effective value is shown by ceph config show <daemon>.
3. What is the difference between ceph tell and ceph daemon?
ceph tell reaches the daemon over the network through the MON/messaging path; ceph daemon uses the local admin socket and works only on that daemon's host.
4. Compare the advantages and disadvantages of scale-up and scale-out in terms of failure domain.
Scale-up lowers the cost per terabyte but has a larger failure domain and recovery volume; scale-out has a smaller failure domain and more linear performance but needs more rack space and network.
5. Why are the norebalance and nobackfill flags useful when adding many OSDs?
So that PGs move once, after all the OSDs have been created, rather than after each OSD.
6. What is the difference between ok-to-stop and safe-to-destroy?
ok-to-stop asks whether a temporary stop would make a PG unavailable; safe-to-destroy asks whether permanent removal is possible without losing data.
7. What advantage does the --replace option give when removing an OSD?
The OSD's id and position in CRUSH are kept and the new OSD is created with the same id; there are fewer map changes and less data movement.
8. What does cephadm's maintenance mode do?
It stops all daemons on the host and sets noout for that host so that no unnecessary recovery starts.
9. Why does setting osd_max_backfills directly have no effect under mClock, and what is the right way?
Because mClock manages these settings itself; the right way is to change osd_mclock_profile, or, if needed, enable osd_mclock_override_recovery_settings.
10. What risk does upgrading OSDs in CRUSH bucket order in Umbrella reduce?
It limits simultaneous upgrades of OSDs that hold replicas of the same PG, and with ok-to-upgrade it makes sure data availability is preserved.

Chapter 8: Ceph Under the Hood: CRUSH, Maps, Authentication and PGs#

1. What are the two stages of mapping an object to an OSD, and why is the PG needed?
The object is mapped to a PG by a hash of its name, and CRUSH maps that PG to a set of OSDs. The PG is needed because tracking replication, recovery and scrub state separately for each object is impossible at a scale of billions.
2. What is the difference between the up set and the acting set?
The up set is CRUSH's pure output; the acting set is the set that actually serves I/O, and it may differ from the up set because of pg_temp or upmap.
3. What is the advantage of straw2 over the earlier algorithms?
When one item's weight changes, data moves only between that item and the rest, and no needless movement happens between other items.
4. Why does changing tunables cause data movement, and what must be checked beforehand?
Because the result of the CRUSH computation changes for many PGs. Beforehand, check the compatibility of all clients (ceph features) and the cluster's capacity for the movement.
5. Why do four MONs not tolerate more failures than three?
Because the majority of four MONs is three, so only one failure is tolerated, the same as with three MONs.
6. Where do OSDs receive the new map from?
From the MON or from their peers (OSDs and clients), who spread the newer epoch in their messages.
7. Why does cephx use tickets?
So that OSDs can check a client's validity without consulting the MON for every request and the MON does not become a bottleneck; the secret key is also never sent over the network.
8. Why does write latency depend on the slowest OSD in the acting set?
Because the primary acks only after all replicas have committed.
9. How do the incomplete and inconsistent states differ?
Incomplete means there are not enough valid copies for the PG to become active and I/O is stopped; inconsistent means the PG is active but scrub has found an inconsistency between replicas.
10. What is the recommended way to place a pool on specific OSDs?
Define a device class (default or custom), build a CRUSH rule with that class and assign the rule to the pool.

Chapter 9: From VSM to the Ceph Dashboard: Graphical Cluster Management#

1. Why is VSM not a suitable option today?
Its repository is archived, its last release is from 2016, it worked only with old Ceph releases, and it has no maintenance or security updates.
2. Which of VSM's ideas have continued in today's Ceph, and in what form?
Storage groups became device classes and CRUSH rules, the hardware manifest became cephadm specs, and the web interface became the Ceph Dashboard.
3. Why is the Dashboard served only by the active MGR, and what consequence does this have?
Because it is an MGR module, and only the active MGR's modules run; when the MGR moves, the address changes, unless a redirect, load balancer or mgmt-gateway is used.
4. What is the orchestrator's role in the Dashboard?
The Dashboard performs deployment operations (hosts, OSDs, services) through the orchestrator (cephadm or Rook); without it these sections are read-only.
5. Why do we read the password from a file with -i?
So that the password does not show up in the shell history or the process list.
6. What is the difference between a role and a scope in Dashboard RBAC?
A scope is an area of resources (such as pool or rbd-image), and a role is a set of permissions (read, create, update, delete) on several scopes.
7. Why is SSO alone not enough for access?
Because SSO only confirms identity; the user must be defined in the Dashboard so that their role and permissions are known.
8. What problem does mgmt-gateway solve?
It provides a single TLS entry point for the Dashboard and the monitoring tools and removes the address's dependence on the active MGR; with oauth2-proxy, centralised authentication also becomes possible.
9. After the removal of the restful module, what replacement is there for automation?
The Dashboard's versioned REST API for operations, and Prometheus for metrics.
10. Why should the cluster's main monitoring not depend only on the Dashboard?
Because the Dashboard depends on the MGR and the MONs and is the first thing to become unavailable in serious crises; monitoring and alerting must have an independent path.

Chapter 10: Benchmarking and Expert Tools for Ceph#

1. Why should performance measurement be done layer by layer?
Because without knowing the ceilings of the lower layers (disk and network) you cannot tell in which layer the performance shortfall arises.
2. Why is fio preferred over dd for measuring disks?
fio controls queue depth, random/sequential pattern, direct I/O and sync, and reports latency percentiles; dd measures only a simple sequential pattern.
3. Why is the sync write test at queue depth 1 important when choosing an SSD?
Because the WAL and DB in BlueStore work with small sync writes, and SSDs without power-loss protection are very slow in this pattern.
4. What symptoms does a mismatched MTU cause in Ceph?
Small heartbeats get through and OSDs look up, but large messages are lost and operations become slow or hang.
5. In the theoretical ceiling example, why did the network become the bottleneck?
Because every incoming byte must travel twice more over the network to the secondary OSDs, and each node's network bandwidth was far lower than the combined throughput of its disks.
6. What is the difference between ceph tell osd.N bench and rados bench?
ceph tell osd.N bench measures only one OSD's BlueStore, without network or replication; rados bench measures the full RADOS path, including network and replication, on a pool.
7. Why should an RBD image be filled before measuring random writes?
Because the first write to each part creates a new RADOS object and is slower than an overwrite; without filling, the result is mixed and misleading.
8. What is the difference between the admin socket and ceph tell, and when is each more useful?
The admin socket is local and works without the network or the MON (useful when a daemon does not respond to network messages); ceph tell is available from anywhere in the cluster.
9. In the output of heap stats, what does freelist memory mean?
It is memory the program has freed but tcmalloc has not yet returned to the OS; heap release returns it.
10. What steps are needed before using ceph-objectstore-tool?
Stop the OSD, take a backup export of the PG, use dry-run where possible, and make sure the normal recovery procedures have failed.

Chapter 11: Deploying Ceph: Lab, cephadm, Ansible and Rook#

1. Why is ceph-deploy no longer a suitable option?
Its repository is archived, and it does not work with recent Ceph releases or the container model.
2. What is the philosophical difference between ceph-ansible and cephadm?
ceph-ansible is imperative and corrects the state only when the playbook runs; cephadm is declarative and continuous, keeps the desired state inside the cluster and constantly reconciles it with the actual state.
3. Why must the virtual CPU model be checked for an Umbrella lab?
Because Umbrella images require a CPU at the x86-64-v3 level, and generic virtual CPU models may not advertise it.
4. What does the --single-host-defaults option do, and what is the limitation of a single-node cluster?
It applies settings suited to one host (for example an OSD failure domain); a single-node cluster has no protection against host failure and is not suitable for practising failures.
5. Why is --dry-run important before applying an OSD spec?
Because it shows which OSDs will be created on which disks and prevents disks from becoming OSDs unintentionally.
6. Why can the rotational filter be misleading in a virtual environment?
Because virtual disks may present themselves as rotational or non-rotational regardless of the real hardware.
7. What is the suggested division of labour between Ansible and cephadm?
Ansible for the OS, packages, network, time and keys; cephadm for all Ceph daemons, specs and upgrades.
8. What is the migration path from ceph-ansible to cephadm?
Using cephadm adopt (or the cephadm-adopt.yml playbook) to take over the daemons one by one, after practising in a lab and with a healthy cluster.
9. What is the risk of running Ceph inside the same Kubernetes cluster as the applications, and how can it be reduced?
A circular dependency between Kubernetes and storage; it is reduced by separating the storage nodes, or by deploying Ceph outside Kubernetes and using external cluster mode.
10. What is the benefit of drift detection?
It reveals unrecorded manual or automatic changes so that they are either brought back into Git or fixed.

Chapter 12: BlueStore: The OSD Storage Engine#

1. What were FileStore's three main limitations?
Double writes because of the separate journal, extra layers of a POSIX file system with unpredictable behaviour, and no end-to-end checksums (plus slow enumeration).
2. Why does BlueStore not need a separate journal for large writes?
Because it writes the data to new space and points to it only after the metadata commits in RocksDB; until that moment the previous version is untouched.
3. What is BlueFS, and why does Ceph not use XFS for RocksDB?
A small file system dedicated to RocksDB's files that runs on the raw device and avoids the overhead and complexity of a general-purpose file system.
4. What is BlueFS spillover, and what effect does it have?
When block.db fills up, RocksDB files are written to the slow block device and metadata performance drops; the BLUEFS_SPILLOVER warning.
5. What is the condition for a write to be deferred in _defer_or_allocate?
The size needed must not exceed the space released by the same write, and that space must be smaller than prefer_deferred_size.
6. Why is it safe to ack the client after KV_DONE?
Because the deferred data became durable together with the L record in the RocksDB WAL and is applied again with _deferred_replay at mount.
7. What are the three generations of omap in BlueStore's key space, and why is per-PG better?
M (shared), m (per-pool) and p (per-PG); per-PG makes deleting and moving a PG possible with one contiguous key range and makes the statistics accurate.
8. Why was min_alloc_size lowered from 64 KB to 4 KB?
To reduce wasted space with small objects, especially in RGW.
9. What trade-off does HybridAllocator strike between AVL and bitmap?
AVL searches free extents fast, but its memory grows with fragmentation; past the memory cap, extents spill over to a bitmap with fixed memory.
10. What effect does the failure of an NVMe that backs db devices have on the cluster?
All the OSDs whose db is on that NVMe are lost at the same time, and large-scale recovery starts.

Chapter 13: Erasure Coding for Storage Efficiency#

1. What are the overhead and fault tolerance of EC 8+3 compared with three-way replication?
An overhead of 1.375 times against 3 times, tolerating three failures against two; but at least eleven failure domains are needed.
2. Why must every k×k submatrix of the generator matrix be invertible?
Because any k available shards must be enough to rebuild, and that is possible only when the corresponding submatrix has an inverse.
3. What is the difference between a shard in EC and a replica in replication?
Each shard keeps only chunk i of every stripe, and its position has meaning; in replication every replica holds the whole object and is interchangeable.
4. Why were small writes expensive in old EC?
Because overwriting part of a stripe required reading the rest of the stripe, encoding again and writing all the shards.
5. What mathematical property does parity delta rely on?
The linearity of the code over the Galois field: the new parity equals the old parity plus the coefficient times the difference between the old and new data.
6. What are the limitations on enabling FastEC?
All MONs and OSDs on Tentacle+, only ISA-L or Jerasure reed_sol_van, stripe_unit a multiple of 4K, and it cannot be turned off once enabled.
7. Why is the allow_ec_optimizations flag irreversible?
Because it changes the data storage format, and the old implementation (ECBackendL) cannot read it.
8. How do CLAY and LRC reduce recovery traffic?
LRC limits a rebuild to a small group with local parity; CLAY uses sub-chunks to read only part of the helpers' data.
9. What does the number 2147483647 in a PG mapping mean, and what are its common causes?
It is CRUSH_ITEM_NONE: CRUSH found no OSD for that shard position; causes: too few failure domains, OSDs out, a wrong rule or too few CRUSH tries.
10. Why is the primary not always shard 0 in an optimized pool?
Because in a partial write some data shards are not updated and cannot have the PG's complete log; pgtemp_primaryfirst moves the allowed shards to the front.

Chapter 14: Programming with librados#

1. What are the three independent parts of a RADOS object?
Data (bytes with offsets), xattrs and the omap (sorted key-value).
2. How does a namespace differ from a pool, and when is it useful?
A namespace is a name subspace inside a pool and has no separate CRUSH rule or PGs; it is useful for separating tenants without creating a new pool, with a capability limited to the namespace.
3. Why is the omap not supported in EC pools, and what does that imply for design?
Because the EC implementation over shards is designed for data and does not cover key-value omap; metadata and indexes must live in a replicated pool.
4. What does a compound operation guarantee, and what does it not guarantee?
Atomic execution of all operations on one object (all or nothing); it does not guarantee transactions across several objects.
5. In which cases does assert_version return ERANGE and EOVERFLOW?
ERANGE when the given version is lower than the object's current version (stale); EOVERFLOW when it is higher.
6. Why is an advisory lock alone not enough for safety?
Because only cooperating clients respect it, and a slow client may carry on unaware that its lock has expired; it must be combined with a guard or blocklisting.
7. How is a watch preserved in the Objecter after a map change?
As a linger operation whose target the Objecter recomputes on every new map or connection drop, resending it to the new primary.
8. Why is resending an Op after a map change safe?
Because each Op has a unique identifier, and the OSD recognizes duplicate requests from the PG log and does not execute them again.
9. When should rados_osd_op_timeout be set?
When the application must reply or fail within a bounded time, because the default is no timeout and the Op waits until the PG becomes active.
10. What pattern is suitable for storing multi-gigabyte data on RADOS?
Splitting the data into objects of a few megabytes (striping), for example with libradosstriper, as RBD and CephFS do.

Chapter 15: RADOS Classes: Computing Next to the Data#

1. What are the three main benefits of running logic in a class instead of the client?
Fewer network round trips, atomic execution inside the PG transaction, and less data transfer by processing next to the data.
2. Why can a class method not modify another object?
Because the other object may be in a different PG and OSD, and the method runs only in the transaction context of its own object.
3. What is the difference between osd_class_load_list and osd_class_default_list?
load_list decides which classes are loaded at all; default_list decides which are permitted with the generic caps (x, class-read, class-write).
4. What happens if a method writes without the WR flag, and why?
After execution the OSD checks the num_write counter and returns -EIO, because a write operation must be processed, recorded and replicated as a write.
5. What exactly does the cap "allow class numops add" permit?
Only calling the add method of the numops class (within the specified match scope, for example the pool).
6. Why is cls_lua not loaded by default?
Because it makes running arbitrary code inside the OSD possible, and without a sandbox or timeout it can block the PG.
7. How do you deploy a custom class in cephadm?
By building a custom image on top of the official same-version image that has libcls_*.so in osd_class_dir, upgrading to it or setting container_image to it, and then adding the class to the lists.
8. What changed in the method registration interface in Umbrella?
Registration changed from cls_register_cxx_method with explicit flags to ClassRegistrar and a typed ClsMethod with RD/WR/PROMOTE Tags.
9. What effect does a slow method have on the PG's other operations?
Because it runs under the PG lock in the processing thread, the PG's other operations wait and slow ops appear.
10. Which classes do librbd and RGW use?
librbd uses cls_rbd (and cls_lock, cls_journal); RGW uses cls_rgw, cls_rgw_gc, cls_2pc_queue, cls_fifo, cls_version, cls_refcount and cls_log.

Chapter 16: Storage Tiering: From Cache Tiering to Device Classes and Storage Classes#

1. What is the official status of cache tiering in current releases?
It has been deprecated since Reef, has no maintainer and may be removed without notice; no new tier should be built, and existing ones should migrate.
2. How does the overlay route a client request to the cache pool?
set-overlay points the base pool's read_tier and write_tier fields in the OSDMap at the cache pool, and the Objecter sends the request to that pool.
3. What advantage does a bloom-type HitSet have, and what does min_read_recency_for_promote do?
Low memory with a controlled false-positive rate; min_read_recency_for_promote is the number of recent HitSets in which an object must have been seen to be promoted on a read (0 means always).
4. Why did cache tiering reduce performance for most RBD workloads?
Because accesses are scattered and every promote and evict costs extra reads and writes; the cache never warms up and I/O increases.
5. How does the tier agent turn the whole-pool target into each PG's share?
By dividing the target by get_pg_num_divisor, which returns the number of PGs taking an unfinished split into account.
6. What is the role of osd_agent_slop?
Hysteresis: the threshold for entering a higher mode is above the target and the exit threshold below it, to prevent oscillating between modes (default 0.02).
7. How does eviction in SOME mode choose a victim?
The object's temperature is estimated from weighted HitSets and placed in a histogram; if its position is in the part hotter than evict_effort it is kept, otherwise evicted.
8. Write down the steps for removing a writeback cache tier.
cache-mode proxy, rados cache-flush-evict-all until empty, tier remove-overlay on the base pool, tier remove.
9. What removed the historical reason for a cache tier in front of an EC pool?
Support for overwrites in EC pools (allow_ec_overwrites since Luminous) and the FastEC optimizations in Tentacle.
10. What alternative is recommended for object data with a known lifetime?
Storage classes and lifecycle rules in RGW (and cloud transition for external archiving).

Chapter 17: Troubleshooting Ceph: From HEALTH_WARN to the Peering State Machine#

1. What are the first two commands in any incident, and why?
ceph -s for the overview and ceph health detail for each warning's exact code; these two show the layer involved.
2. What is the difference between degraded, undersized and remapped?
degraded: some objects do not have enough copies; undersized: there are fewer acting OSDs than size; remapped: acting differs from up (because of pg_temp or backfill).
3. What problem does the WaitUpThru state solve?
It guarantees that the MON has recorded that the OSD was alive up to this epoch; so in later peerings it can be determined whether that interval could have accepted writes.
4. Why does a PG go down, and what does the down_osds_we_would_probe field say?
Because there is an interval in which writes may have happened and none of its OSDs is available; the field shows the OSDs whose return makes peering possible.
5. How does find_best_info differ between replicated and EC pools, and why?
In replicated pools the largest last_update is the reference; in EC the smallest last_update that all shards can roll back to, because a partial EC write cannot be reconstructed.
6. What does a last event of sub_op_sent in a slow op mean?
The operation has been sent to the replicas but not all acks have arrived; the replica OSD or its network is slow.
7. How does scrub coordinate with in-flight writes?
Scrub proceeds chunk by chunk and, for each range, waits until in-flight writes finish and last_update catches up (RangeBlocked, WaitPushes, WaitLastUpdate).
8. What should you check before ceph pg repair?
The output of rados list-inconsistent-obj, to know which shard or OSD the error is on and whether the authoritative copy is clear; and the health of the guilty disk.
9. What is the risk of ceph osd lost?
Acknowledged writes that existed only on that OSD are lost forever.
10. Why must nodown and noout be temporary?
nodown on a dead OSD keeps clients waiting, and noout prevents automatic recovery; both deprive the cluster of its self-healing behaviour.

Chapter 18: Disaster Recovery: rbd-mirror, Stretch Mode and Cluster Rescue#

1. Why is replication not a substitute for backup?
Because accidental deletion, ransomware and logical corruption are replicated too; a backup must be out of reach of the same credentials and operations.
2. What is the difference between journal-based and snapshot-based mirroring in overhead and RPO?
The journal performs each write twice and has an RPO of a few seconds; snapshot mode has little overhead, and its RPO equals the schedule interval.
3. What role does the complete field in MirrorSnapshotNamespace play?
It shows that copying the difference for that destination snapshot has finished; an incomplete snapshot means the image may be inconsistent, and the last consistent point is the previous complete snapshot.
4. In which cluster does rbd-mirror run, and why does it pull?
In the destination cluster; it pulls from the source so that the source does not need to know about or have access to the destination, and the daemon runs next to the copy it writes.
5. What must be done after promote --force when the previous site returns?
Demote the previous site's image and then resync it so that it synchronizes again from the current primary; writes made on that site after the disconnection are lost.
6. What is the role of the tiebreaker MON in stretch mode?
A data-less MON at a third site that, when the two sites lose contact, determines through the quorum which site stays active.
7. What effect does stretch mode have on write latency?
The latency of each write increases by the round trip between the two sites, because a write must be recorded at both sites before the ack.
8. What information is lost when rebuilding the MONs from the OSDs?
The MDS keyrings and most other keyrings, the state of pools that were being created, and some MON-only settings.
9. What tool has Umbrella added for MON backups?
Periodic backups of the monitor's RocksDB (mon_backup_interval), the command ceph tell mon.* backup, and restoring with ceph-mon --restore-backup.
10. What is ceph-objectstore-tool used for in DR, and what is its prerequisite?
Exporting a PG from a dead but readable OSD and importing it into another OSD, to rescue an incomplete or down PG; prerequisite: both the source and destination OSDs must be stopped.

Chapter 19: Day-to-Day Operations: Upgrades, Disk Replacement, Autoscaler and Balancer#

1. What is the daemon upgrade order in cephadm, and why is the mgr first?
mgr, then the mons one by one, then the OSDs and finally the other services; the mgr is first because the cephadm module that drives the upgrade runs in the mgr and must be the new version.
2. What is the advantage of upgrading OSDs by CRUSH bucket?
All OSDs of one failure domain are upgraded together after ok-to-upgrade confirms it; the upgrade time of a large cluster drops sharply without breaking availability.
3. Why does --replace reduce data movement?
The OSD stays in the destroyed state with the same ID and CRUSH position; so only that OSD's PGs are refilled and no cluster-wide redistribution happens.
4. What does maintenance enter do with the noout flag?
With osd set-group it sets noout on the host's CRUSH subtree, and removes it on exit.
5. What is the basic pg_target formula in the autoscaler?
pg_target = the pool's capacity ratio × (number of OSDs in the root × mon_target_pg_per_osd) × bias / size, then rounding to a power of 2.
6. What problem does the autoscaler threshold solve, and when is the dynamic threshold applied?
It prevents small, frequent pg_num changes (a change only on a threefold difference); the dynamic threshold applies to increases to 512 and 1024 (threshold 1) and 2048 (threshold 2).
7. What are the consequences of the change in the default mon_target_pg_per_osd in Umbrella?
The value went from 100 to 200; after the upgrade the autoscaler may split PGs and data moves.
8. How does the upmap balancer choose a PG to move?
It finds OSDs whose deviation from their weighted share exceeds max_deviation and moves a PG from a full OSD to an underloaded one, provided the CRUSH rule is not violated.
9. Why is upmap safer than changing weights by hand?
It is applied on top of the CRUSH output and is ignored and removed if it violates the rule; a weight change alters the whole mapping and is not predictable.
10. What is the difference between ceph config set and injectargs?
ceph config set persists in the MON database and is reversible; injectargs only temporarily changes the value in the daemon's memory.

Chapter 20: Monitoring Ceph: Prometheus, ceph-exporter, Alerts and Observability#

1. Why were perf counters moved from the prometheus module to ceph-exporter?
Collecting every counter of every daemon from one mgr reduced mgr performance in large clusters; ceph-exporter on each host reads only the local daemons from the admin socket.
2. Why should all mgrs or service discovery be defined for scraping?
Only the active mgr publishes cluster metrics; after a failover, a fixed target pointing at the previous mgr creates a blind spot.
3. How are a counter's priority and exporter_prio_limit related?
ceph-exporter only publishes counters whose priority is at least exporter_prio_limit (default 5).
4. How is the average latency of an interval computed from a time_avg counter?
The ratio of the increase in sum to the increase in avgcount over that interval; in PromQL, the ratio of the rates of the two series _sum and _count.
5. What is the ceph_health_detail metric used for?
One series per active health code with a name label; it allows precise alerting on codes such as SLOW_OPS or MON_DOWN.
6. What is the role of for in an alert rule?
The condition must hold for the whole for duration before the alert fires; it prevents alerts for transient events.
7. What problem does inhibit in Alertmanager solve?
When a root alert (for example a host down) is active, it silences the secondary alerts it causes (the down OSDs of that host).
8. What are the advantages of mgmt-gateway?
A single entry point with TLS and a virtual address for all interfaces, centralized authentication with oauth2-proxy, and secured monitoring endpoints.
9. Why should Prometheus not be directly exposed to users?
Prometheus's security model assumes data is visible to anyone with access; its API reveals all collected (meta)data.
10. Name a few suitable SLIs for an RBD service.
The percentage of time all of the pool's PGs are active, the 99th percentile of write latency, and for mirrored images the RPO (timestamp difference).

Chapter 21: Ceph Performance Tuning: From mClock to the Kernel and Network#

1. Why should performance tuning be done with a baseline and one change at a time?
Because without a baseline the effect cannot be measured, and with several simultaneous changes you cannot tell which one helped or did harm.
2. How is each PG's shard determined, and why are all operations of a PG in one shard?
With pgid.hash_to_shard(num_shards); so that the ordering of a PG's operations is preserved and no coordination between shards is needed.
3. What change did Squid 19.2.1 make to the default HDD shards, and why is it sensible?
From 5 shards with 1 thread to 1 shard with 5 threads; with one shard, mClock has a complete view of all of the OSD's requests and enforces the shares more precisely.
4. Define the three mClock parameters.
reservation is the guaranteed minimum share, weight the relative share of spare capacity, limit the share ceiling.
5. How is the R tag computed in dmclock, and what role does max(now, ...) play?
R = max(now, R_prev + increment × (δ + cost)) with increment = 1/reservation; max(now, ...) prevents an idle class from accumulating credit.
6. How does the high_client_ops profile differ from balanced?
A client reservation of 60% with weight 2 versus 50% with weight 1, a recovery reservation of 40%, and a best-effort limit of 70% instead of 90%.
7. Why does changing osd_max_backfills have no effect in the built-in profiles?
Because mClock's built-in profiles manage these values, and they can only be changed with osd_mclock_override_recovery_settings=true.
8. How does the OSD determine its IOPS capacity, and when does it discard the result?
At startup, if the capacity value is the default and the benchmark is not skipped, it runs an internal benchmark; if the result is outside the range of the low and high thresholds, it uses the default value.
9. Name three operating system or network settings with the greatest effect.
The CPU profile (tuned and C-states), a consistent MTU and an adequate cluster network, and a suitable I/O scheduler (none for NVMe).
10. Why should percentiles be reported instead of the average?
Because the average hides the tail of the distribution; user experience and SLOs are determined by the high percentiles.

Chapter 22: Ceph Code Architecture and the Request Path from the Inside#

1. Why does the OSD not write directly to disk, and why does it use ObjectStore::Transaction?
Because a Transaction is atomic: the object data, xattrs, omap and the pg log entry become durable together or not at all. The backend (BlueStore, MemStore, SeaStore) also stays swappable.
2. What inputs does _calc_target in Objecter have, and when is an op resent?
The inputs are the object name, pool, namespace, snap and the current OSDMap, and the output is the pg, acting and primary. If the primary or acting set changes with a new map, the op is resent to the right OSD.
3. Why does each PG's op always go to one OSDShard?
Because the shard is derived from the hash of the PG ID. This preserves the order of a PG's ops without coordination between shards.
4. What is the role of ms_fast_dispatch, and what must not be done in it?
Fast delivery of the message in the same network thread and queueing it in the shard. No blocking work or long lock may be done in it.
5. How does ENCODE_START(v, compat, bl) make rolling upgrades possible?
v is the current version and compat the lowest compatible decoder version. The block length is stored so that an old decoder skips the new fields.
6. What is the difference between the direct and deferred paths in BlueStore, and why is the SSD default zero?
On the direct path, data is written with aio to fresh space. With deferred, small data is first recorded in the RocksDB WAL and written later. On SSD a seek is not expensive, and the double write only adds write amplification.
7. When does the primary reply to the client in replication?
When all acting members (and the primary itself) have committed the transaction; eval_repop checks this.
8. In RGW, which function moves an index entry from pending to complete?
The cls method named bucket_complete_op, called through UpdateIndex::complete.
9. In CephFS, why is the MDS not on the data path, and what is the effect of not having the Fb cap?
Because the client writes directly to the OSDs with Striper and Objecter. Without Fb, the client cannot buffer and every write goes synchronously to the OSD.
10. Where do you look to find the real default of an option in a specific version?
The src/common/options/*.yaml.in file at that version's tag (for example v20.2.0) in the ceph/ceph repository.

Chapter 23: Lower Layers (1): Block Device, bufferlist, mempool and crc32c#

1. Why does KernelDevice open a separate fd for each write hint?
So that, on kernels that support F_SET_FILE_RW_HINT, the data lifetime hint (for SSDs with streams) is set once on each fixed fd, and every write goes through the fd that matches its hint.
2. What limit does O_DIRECT alignment place on BlueStore's smallest write?
The offset, length and buffer address must be aligned to the block (4K). So the smallest write is 4K, and smaller writes become either RMW or deferred.
3. What are the roles of bdev_aio_max_queue_depth and bdev_aio_reap_max?
The first sets the maximum number of I/Os in flight in the queue, and the second the maximum number of completions reaped in each round.
4. When does BlueStore force a separate flush of the data device?
When data and DB are on separate devices, or when there is nothing to commit in RocksDB; in that case the BlueFS log flush does not cover the data device.
5. Why is discard off by default?
Because TRIM is synchronous and slow on many devices and raises latency; today's NVMe devices work well without it.
6. What is the difference between buffer::ptr and buffer::raw?
raw owns the memory and has a reference count; ptr is only a window (offset and length) onto one raw.
7. Why can bufferlist::c_str() be expensive?
Because if the list is fragmented, rebuild copies all the data into a new contiguous buffer.
8. How does mempool help memory autotuning?
It records the usage of each part precisely and in shards, and autotune compares that with osd_memory_target to adjust the share of the caches.
9. Why does Ceph's crc32c of "123456789" differ from the standard reference value?
Because ceph_crc32c does not apply the final XOR; 0x1CF96D7C XORed with 0xFFFFFFFF gives 0xE3069283.
10. What is the checksum overhead with a 4K chunk, and what is the trade-off of making the chunk larger?
About 0.1% (4 bytes per 4096 bytes). A larger chunk lowers the overhead but forces a small read to read the whole chunk.

Chapter 24: Lower Layers (2): BlueStore and BlueFS On-Disk Format#

1. What do the label's first 60 bytes contain, and why does the decoder skip them?
The string "bluestore block device\n", the OSD's uuid as text and a \n; this part is only for human readability, and the encoded data starts after it.
2. Why is the BlueFS superblock always at offset 4096?
Because the first block is reserved for the label and BlueFS deliberately makes the same assumption (the second block); this is fixed in bluestore_common.h.
3. How many bytes does denc_lba use for a 4K-aligned offset, and why?
Usually 4 bytes; 12 trailing zero bits are removed with one type bit, and 30 data bits cover addressing up to about 4 terabytes.
4. What condition starts BlueFS log compaction?
The log size must be at least bluefs_log_compact_min_size (16M), and its ratio to the estimated size at least bluefs_log_compact_min_ratio (5).
5. Why is the hash in the onode key bit-reversed?
So that the objects of a PG, which share the low bits of the hash, form a contiguous range in the key space, and list and scrub can be done with a range scan.
6. What savings do the BLOBID flags in the extent map create?
Repeated logical_offset, blob_offset or length are not written; a typical lextent becomes 1 to 2 bytes.
7. Why is the checksum the largest part of an encoded blob?
With crc32c and a 4K chunk, each 64K blob has 64 bytes of checksum, while the pextents are a few bytes.
8. How does the hybrid allocator limit memory use?
With the bluestore_hybrid_alloc_mem_cap (64M) cap for the AVL; extra pieces overflow into a fixed-memory Bitmap.
9. Why does the freelist use merge with XOR?
Because XOR is commutative and self-inverse; allocation and release become a merge without reading the previous value.
10. What problem of LRU does the 2Q algorithm solve?
A one-off sequential scan does not empty the cache of frequently used data, because data only moves to hot when it is seen again.

Chapter 25: Lower Layers (3): RocksDB Inside BlueStore#

1. How does group commit in _kv_sync_thread reduce the number of fsyncs?
Several ready transactions are recorded one after another in one WriteBatch, and only the last of them is sync. One fsync on the WAL makes them all durable.
2. How many bytes is a WAL record header, and what fields does it have?
7 bytes: masked crc32c (4), length (2) and type (1). In the recyclable format a 4-byte log number is added as well.
3. Why is min_write_buffer_number_to_merge=6 important for Ceph?
Because short-lived keys (deferred and pg log) are deleted while memtables are merged and never reach an SST, which greatly reduces write amplification.
4. Why is this number 32 for the L and P CFs?
Because deferred (L) and pgmeta (P) keys live very briefly, and more merging means more deletions before the flush.
5. What role does an SST's filter block play on the read path?
Without reading a data block, it determines that the key is probably not in this file, and it removes most of the useless reads from L0 to Ln.
6. With the defaults, how large is L3?
About 64G (1G × 8 × 8).
7. How does use_some_extra differ from rocksdb_original?
rocksdb_original puts only whole levels on the DB. use_some_extra uses the extra space for part of the next level.
8. How is the CF of a per-pg omap key chosen?
Bytes 0 to 12 of the key are hashed with ceph_str_hash_rjenkins, and the remainder modulo 3 gives the CF number.
9. Why should a PG's omap stay in one shard?
So that walking the omap of an object or PG happens in one CF with one range scan.
10. Why is changing the whole bluestore_rocksdb_options string dangerous, and what is the alternative?
Because important defaults (such as memtable merging and LZ4) are removed unintentionally. bluestore_rocksdb_options_annex adds only the changed keys.

Chapter 26: Lower Layers (4): msgr2 at the Byte Level, AsyncConnection and EventCenter#

1. What is the structure of the msgr2 banner, and which features exist today?
"ceph v2\n", a u16 length (16) and two u64s for the supported and required features; the current features are REVISION_1 and COMPRESSION.
2. Why is the preamble exactly 32 bytes?
So that it is a multiple of the AES block (16) and can be encrypted in secure mode without extra padding.
3. With what seeds are the preamble crc and the segment crc computed?
The preamble with seed zero over the first 28 bytes; the segments with seed 0xFFFFFFFF (−1).
4. Why does the data segment have an alignment of 4096?
So that the receiver can allocate a page-aligned buffer and hand the data to O_DIRECT without a copy.
5. What is the benefit of inlining 48 bytes in secure mode?
The message header and small control frames are read with one decryption and one tag, and the length of the rest of the frame is known at once.
6. What attack does AUTH_SIGNATURE prevent?
A downgrade attack, or tampering with the unencrypted bytes exchanged before auth, such as changing preferred_modes.
7. When are SESSION_RESET and SESSION_RETRY_GLOBAL sent?
RESET when the session cookie is unknown to the server (for example after a restart); RETRY_GLOBAL when the client's global_seq is older than what the server has seen.
8. What are the three throttlers on message receive, and how do they create backpressure?
The message count, the policy bytes and the dispatch queue; when any of them fills up, reading from the socket stops and the TCP window slows the sender down.
9. Why must handlers in EventCenter read until EAGAIN?
Because epoll is in edge-triggered mode, and unread data does not create a new event.
10. How does another thread wake a worker to send a message?
It queues the event in external_events and wakes that worker's epoll_wait by writing to the eventfd (or pipe).

Chapter 27: Lower Layers (5): Erasure-Coding Math, from GF(2⁸) to FastEC#

1. Why are addition and subtraction in $\mathrm{GF}(2^8)$ both XOR?
Because the coefficients are in GF(2) and 1 + 1 = 0; so bitwise addition is XOR and every element is its own negative.
2. How is 0x56 × 0x04 computed in $\mathrm{GF}(2^8)$ with 0x11D?
0x56 is shifted twice, giving 0x158; because bit 8 is set, 0x58 ⊕ 0x1D = 0x45.
3. What is an MDS code, and why is ISA-L's simple Vandermonde not always MDS?
A code in which any k of the k+m chunks are enough for reconstruction; in the simple Vandermonde some k×k submatrices are not invertible in $\mathrm{GF}(2^8)$ .
4. Under what conditions does Ceph switch to cauchy automatically?
When m > 4, or m = 4 and k > 21.
5. In this chapter's example, how were P0 and P1 obtained?
P0 is the XOR of the four bytes (0x08) and P1 is 12 ⊕ 2·34 ⊕ 4·56 ⊕ 8·78 = 0xD8.
6. Why are the decode coefficients cached?
Because inverting the matrix for each failure pattern is expensive, and the same pattern repeats many times during recovery.
7. How does PSHUFB make multiplication by a constant fast?
The multiplication becomes two 16-entry lookups for the nibbles, and PSHUFB performs dozens of lookups in one instruction.
8. What is the parity delta, and why is it correct?
Updating the parity with $P' = P \oplus g \cdot (D \oplus D')$ ; because the code is linear, changing one data chunk has a separate linear effect on each parity.
9. What is the default stripe_unit in an optimized pool, and why?
16K; because with optimized partial writes a larger stripe has no RMW cost, and sequential reads and metadata become more efficient.
10. What is the CLAY repair bandwidth with k=8, m=4 and d=11, and what does it cost?
2.75S versus 8S; the cost is 64 sub-chunks per chunk and tiny, scattered reads.

Chapter 28: Lower Layers (6): CRUSH at the Bit Level, from rjenkins to upmap#

1. How does ceph_stable_mod limit data movement when pg_num is increased?
Only the PGs that have the new mask bit and whose number is less than pg_num move (split); the rest keep the same number.
2. Why is HASHPSPOOL needed?
So that PGs with the same number in different pools do not reach the same x and pile up on the same OSDs.
3. Why does CRUSH use only integer operations?
So that the result is bit-for-bit identical on all architectures, in the kernel and in user space.
4. In straw2, how is the draw computed and why does the largest draw win?
draw = $(\mathrm{crush\_ln}(u) - 2^{48}) / w$ ; because ln is negative, a larger weight gives a larger (less negative) draw. This is equivalent to the minimum of exponential variables.
5. What does crush_ln return, and with what precision?
It returns $2^{44} \cdot \log_2(x+1)$ , computed with two tables and integer operations, with enough precision for a 16-bit distribution.
6. Why does straw2 have minimal movement, and why did the old straw not?
Because each item's draw depends only on its own weight; in straw the straw lengths were relative and depended on all the weights.
7. How is a collision resolved in firstn, and what is its limit?
ftotal is increased and r = rep + ftotal; up to choose_total_tries=50.
8. What is the difference between firstn and indep for EC?
In firstn the positions shift; in indep the position of each shard is fixed and only the lost shard is chosen again.
9. What problem does the weight-set in choose_args solve?
It corrects the deviation of the second and third replicas' distribution from the weights with a separate weight per position, without changing the main weights.
10. What is the order in which upmaps are applied, and what does the mon do with an invalid upmap?
pg_upmap, then pg_upmap_items, then pg_upmap_primary; the mon removes upmaps that violate the rule.

Chapter 29: Lower Layers (7): Linux Kernel Clients — libceph, krbd and ceph.ko#

1. What parts make up libceph, and why is CRUSH in the kernel too?
messenger, mon_client, osd_client, osdmap and CRUSH, cephx and striper; CRUSH is in the kernel so that the client can compute PG→OSD itself, with no intermediary.
2. How is a blk-mq request split into obj_requests?
The img_request uses the striper and the object size to break the byte range into per-object ranges, and builds one obj_request for each.
3. When does krbd use WRITEFULL, ZERO, TRUNCATE or DELETE instead of WRITE?
WRITEFULL to cover a whole object; ZERO to zero the middle; TRUNCATE to zero the end; DELETE to discard a whole object that has no parent.
4. When does copyup happen in krbd?
When, in a clone, a write lands on an object that does not exist yet and the parent has data in that range.
5. Why does the kernel client get stuck on a full pool, and how can this be bounded?
Because the requests are held by PAUSEWR or pool_full until the next map; it can be bounded with osd_request_timeout.
6. Where do the io_opt and io_min values in krbd come from?
io_opt comes from object_size × stripe_count and io_min from alloc_size (default 64K).
7. Interpret a caps string of the form pAsLsXsFrwcb.
pin; Auth shared; Link shared; Xattr shared; File read, write, cache and buffer.
8. Why does the second reader of a file being written not get Fc?
Because with a concurrent writer, cached data may be stale; the MDS gives only Fr and reads go directly to the OSD.
9. What does the MDS_HEALTH_CLIENT_LATE_RELEASE warning mean?
A client has not answered a caps revoke request and the other clients are waiting.
10. Name two advantages and two disadvantages of krbd compared with librbd.
Advantages: low overhead and no need for a user process; disadvantages: limited features (no journaling) and the need for a new kernel to upgrade.

Chapter 30: Lower Layers (8): Crimson and Seastar — Shard-per-Core OSD and SeaStore#

1. Why is the thread-pool model of the classic OSD expensive on NVMe?
Each request moves several times between threads and locks, and the cost of context switches and cache misses is comparable to NVMe's latency of a few tens of microseconds.
2. How do futures and continuations in Seastar replace blocking?
An asynchronous function immediately returns a future; the continuation is attached with then and, after set_value, runs in the queue of the same reactor.
3. What does crimson_reactor_task_quota_ms limit, and what is the effect of a long continuation?
The maximum time the reactor runs tasks before polling again; a long continuation stops all the PGs of that core (a stall).
4. How is a new PG assigned to a core, and what is the weakness of this method?
Core 0 chooses the core least loaded by PG count with min_element; the load of the PGs is not considered and there is no work-stealing.
5. Explain the difference between exclusive and concurrent phases with the WaitRepop example.
exclusive: one operation at a time, in arrival order; concurrent: several operations at once, leaving in order; WaitRepop lets several writes wait for the replicas at the same time.
6. Why is AlienStore needed, and what does it cost?
To run the classic BlueStore and MemStore outside the reactor; the cost is two crossings of the reactor/thread boundary and CPU contention.
7. How do laddr and paddr differ in SeaStore, and why does this separation reduce the cost of GC?
laddr is logical and stable and paddr is physical; when a block moves only the LBA tree is updated.
8. Describe the 64-bit structure of paddr_t for segments and for RBM.
8 bits of device_id; segment: 24 bits of segment number and 32 bits of offset; RBM: a 56-bit absolute offset.
9. Write the cost-benefit formula and explain autotune with an example.
$(1-u) \cdot \mathit{age} / (2u)$ ; if greedy's free fraction is at least ratio times the free fraction of the formula's choice, the greedy choice replaces it.
10. Which three flags are needed to enable Crimson?
The crimson experimental feature, the allow_crimson flag in the OSDMap (set-allow-crimson) and osd_pool_default_crimson for pools.

Chapter 31: Lower Layers (9): SPDK and NVMe — from Hardware Queues to the NVMe-oF Gateway#

1. Where are the SQ and CQ located, and who writes into each?
Both are in host memory; the host writes commands into the SQ and the controller writes results into the CQ.
2. How does the Phase Tag identify a new CQE without reading a device register?
The controller writes P inverted on each pass through the queue; the host treats an entry as new if its P equals the current cq_phase.
3. Why does the kernel defer the doorbell write until the end of a batch?
Each doorbell is an expensive MMIO write across PCIe; one write for the whole batch is enough.
4. Why is the length field of a Read command zero-based, and what is its value for 8 blocks?
Because zero blocks is meaningless and the maximum of 65536 blocks fits in 16 bits; for 8 blocks the value is 7.
5. Why does NVMEDevice copy data into DMA buffers?
The controller needs pinned memory with a known IOVA (a DPDK hugepage), and a bufferlist is not like that.
6. What are the benefit and the cost of the thread_local SharedDriverQueueData?
Benefit: a lock-free qpair and buffer pool; cost: every new thread consumes 8 MiB of DMA buffers and one qpair.
7. How does aio_submit in NVMEDevice differ from KernelDevice in terms of blocking?
NVMEDevice polls on the same thread until all the I/Os of the IOContext complete; KernelDevice only submits.
8. Describe the path of a write from the host's nvme_tcp to the OSD, using the names of the bdev_rbd functions.
Command Capsule ⇒ nvmf_tgt ⇒ bdev_rbd ⇒ rbd_aio_writev ⇒ librbd/Objecter ⇒ bdev_rbd_finish_aiocb ⇒ spdk_thread_send_msg to submit_td ⇒ Response Capsule.
9. Why must the replacement gateway wait for the blocklist before becoming ACTIVE?
So that the previous gateway, if it is still alive, cannot write, and two concurrent writers on one namespace do not arise (fencing).
10. When does the replacement-gateway algorithm return -EBUSY, and when -ENOENT?
-EBUSY if there is no candidate but at least one gateway is in transition; -ENOENT if there is no available gateway in scope.

Chapter 32: Lower Layers (10): Scrub, Recovery, Backfill and Repair at Code Level#

1. In what range is the next shallow and deep scrub time with the defaults, and why does the deep distribution have peaks at the edges?
shallow is uniform in [24, 36) hours; deep is normal with a mean of 7 days and a deviation of 1.4 days, truncated to [4.2, 9.8] days. The truncation piles the probability of the tails onto the edges.
2. After exactly how long is the PG_NOT_DEEP_SCRUBBED warning raised?
When more than deep_interval × (1 + 0.75), that is 12.25 days, have passed since the last deep scrub.
3. Why must a scrub chunk not separate a head from its clones?
A new clone is always created right next to its head and could go unseen between two chunks.
4. After three preemptions in a deep scrub, what is the maximum chunk size?
The divisor is 8 and 15/8=1, but the minimum is 5, so 5.
5. In select_auth_object, if the versions are equal, what criterion decides the winner?
The number of recorded digests (dcount), and on a complete tie, the order of the candidates, which starts with the primary.
6. What is the backfill priority of an undersized PG with size=3 and two acting members?
140 + 1 + 10 = 151.
7. Why does the priority rise by 10 units even with recovery_priority=0?
clamp_recovery_priority shifts the range −10 to +10 to 0 to 20.
8. Under what condition does AsyncReserver take back an in-progress reservation?
When the slots are full and a queued request has a higher priority than the lowest-priority preemptible reservation, or min_priority has risen or max has dropped.
9. Which target does async recovery leave out of acting, and what condition limits it?
A member whose cost (version gap + missing objects) exceeds 100, provided the remaining acting set is still writeable.
10. Why does repair have the same limits as recovery?
Because repair only marks the damaged shards as missing and leaves the copying to normal recovery.

Chapter 33: Lower Layers (11): Memory and CPU Inside the OSD — from mempool to NUMA and Profiling#

1. Why does mempool use shards instead of one shared atomic counter, and how is the shard chosen?
So that the counter's cache line does not bounce between cores; with sched_getcpu() & (num_shards−1), or a fixed round-robin per thread.
2. Why is the size of shard_t a multiple of 128?
aligned(128), so that two shards never share a cache line.
3. Compute, with the defaults, the two numbers derived from osd_memory_target as the max and min of the cache budget.
max = 0.85 × 4096 − 768 = 2713.6 MiB and min = osd_memory_cache_min = 128 MiB.
4. How does tune_memory behave differently when increasing and when decreasing?
The increase is proportional to (1 − mapped/target) × the distance to max, and the decrease to (1 − target/mapped) × the distance to min.
5. Why is osd_memory_target not a hard limit?
Because it only shrinks the caches; unaccounted memory can by itself exceed the target.
6. In balance, what happens to a cache that asks for less than its share, and who gets the remainder?
It gets that amount and leaves the later rounds; the remainder is divided by ratio among the caches still unsatisfied.
7. What is get_chunk for a tuned_mem of 1672 MiB?
chunk = 8 MiB; get_chunk(1) = 72 MiB.
8. Why does increasing threads_per_shard not help a hot PG?
Because the PG lock limits parallel execution of one PG's ops, and all of that PG's ops go to one shard.
9. Under what condition does the OSD set NUMA affinity automatically?
When the node of the storage device and the node of the public and cluster interfaces are the same and osd_numa_auto_affinity=true; or with an explicit osd_numa_node.
10. How do you compute the "CPU budget per op", and what do you conclude from it?
Cores used divided by ops per second; then multiply each layer's share from the flame graph by it to find the maximum saving of each tuning.

Chapter 34: Specialist Part (1): RADOS — Object Identity, Metadata and the Op Path Inside the OSD#

1. In what order is hobject_t compared?
max, pool, the bitwise key (the reversed hash), nspace, the effective key (if either of the two keys is non-empty), oid and finally snap.
2. Why does the bitwise key reverse the hash?
So that the low bits of the hash, which determine the PG, move to the top of the key and all objects of a PG form one contiguous range in the sort order.
3. After pg_num doubles, how is the bitwise-key range of PG 0x1f split?
Into two contiguous halves: PG 0x1f gets the range [0xf8000000, 0xfbffffff] and the new PG 0x3f gets [0xfc000000, 0xffffffff].
4. What do the _ and snapset xattrs hold?
_ is the object_info_t (version, size, flags, digests, watchers) and snapset is the SnapSet of the head object.
5. How is the shard of an op chosen, and what are the defaults on SSD?
shard = ps % num_shards; on SSD the default is 8 shards and 2 threads per shard.
6. If the object is missing on the primary, in which queue does the op wait?
waiting_for_unreadable_object, with the text "waiting for missing object".
7. How is a duplicate request detected, and what role do dups play?
By looking up the reqid in caller_ops, extra_caller_ops and dup_index; dups keep the result of requests whose log entry has been trimmed.
8. How is user_version computed, and why may it not be contiguous for one object?
max(the PG's last_user_version, the object's user_version) + 1 (and at least at_version.version); because the counter increases at the PG level, not at the object level.
9. When is the reply to a replicated write sent to the client?
When all members of acting (including the primary itself) have reported the commit.
10. What format does the key of a pg log entry have, and why is its length fixed?
%010u.%020llu from epoch and version (with the prefix dup_ for a dup); the fixed length makes string order the same as numeric order.

Chapter 35: Specialist Part (2): RADOS — The PG Log, Intervals, Missing Sets, and How Recovery and Backfill Work#

1. What changes in the OSDMap create a new interval? Name three other than up and acting.
Besides up, acting and the primaries: a change in size or min_size, a split or merge, sort_bitwise, recovery_deletes, the CRUSH stretch parameters, and EC optimizations being turned on.
2. Under what conditions is maybe_went_rw true?
When acting was writeable (at least min_size) and either the primary's up_thru covered the start of the interval or last_epoch_clean fell inside that interval.
3. Why is up_thru necessary for the safety of peering?
Because the primary goes active only after up_thru is recorded in the mon; so an interval in which up_thru was not recorded had no writes, and there is no need to wait for its members.
4. When does a PG go down, and whom does blocked_by point to?
When, for an interval with maybe_went_rw, there are not enough up or lost members (replicated: none, EC: fewer than k) and at least one of its members is down; blocked_by lists those down OSDs.
5. What does merge_log do when the authoritative log is ahead?
It appends the entries after lower_bound, updates the missing set for each of them, treats the local entries after lower_bound as divergent and sets last_update to the head of the authoritative log.
6. What is a divergent entry, and why is rolling it back safe?
An entry that is not in the authoritative log; since the client reply is given only after all of acting has committed, the client never saw that write succeed.
7. How do cases 4 and 5 of divergent handling differ?
In 4 all entries have rollback information and the local object is rolled back; in 5 the object is removed and put in the missing set with need=prior, to be recovered from another member.
8. What is the target log length for an OSD with 600 PGs under the defaults?
300000/600 = 500, which lies between 250 and 10000; so 500.
9. What condition determines that a member is backfilled?
If our log_tail is ahead of that member's last_update, or its last_backfill is empty (a new member).
10. How does backfill determine that an object must be removed from the target?
An object that is in the target's interval but smaller than the smallest remaining object in the primary's interval, meaning it does not exist on the primary.

Chapter 36: Specialist Part (3): RADOS — Snapshots and Clones, Watch/Notify, OSD-Side cls, and the rados and pool Reference#

1. Why can a pool not have both pool snapshots and self-managed snapshots?
Because both take their numbers from one snap_seq, and the SnapContext is determined by the OSD in one case and by the client in the other; combining them makes the ownership of the numbers ambiguous, so the mon rejects it.
2. What fields does a SnapContext have, and who sends it in RBD?
seq and the descending list of snaps; librbd reads it from the image header and sends it with every write to the data objects.
3. What is the condition for creating a clone in make_writeable?
The head exists and is not a whiteout, the SnapContext has at least one snap, and snaps[0] is greater than snapset.seq.
4. What number is a clone named with?
With snapc.seq, that is, the largest new snap the clone represents.
5. In the chapter's example, what are the results of reading at snap 5 and at snap 3?
Snap 5 reaches clone @6 because 5 is in clone_snaps[6]; snap 3 reaches @4 but 3 is not in [4], so ENOENT.
6. What is a whiteout, and when is it deleted?
A deleted head that remains to keep the SnapSet; when snaptrim deletes the last clone, the whiteout is deleted too.
7. What structure does the SNA_ key in the SnapMapper have, and why is it shaped this way?
SNA_<pool>_<snap as 16-digit hex>_ followed by the shard prefix and hobject.to_str(); so that all clones of a snap are found with one range scan.
8. When a clone no longer has any snaps, what happens to clone_overlap?
If there is an older clone, its overlap is intersected with the overlap of the deleted clone; then the clone's entries are removed from clones, clone_overlap, clone_size and clone_snaps.
9. How is an expired watch removed, and why may it linger for a while?
After osd_client_watch_timeout without a ping, it is removed from object_info_t by a write on the object; if the PG does not accept writes, it stays until the PG does.
10. Why is a cls call atomic with the rest of the ops in a message?
Because the method runs on the same OpContext and all the cls_cxx_* calls are added to the same transaction and the same log entry.

Chapter 37: Specialist Part (4): RBD — Layout on RADOS, Striping, Object Map and Fast-Diff, and the rbd Reference#

1. Which header keys are needed to build the names of the data objects?
object_prefix and order (and, with custom striping, stripe_unit and stripe_count); each object's name is object_prefix followed by the object number as 16-digit hex.
2. With a separate data pool, how do the data object names change, and why?
The prefix becomes rbd_data.<metadata pool id>.<id>, so that objects of images from several pools that share one data pool do not collide.
3. What two kinds of key does rbd_directory have?
name_<name>, which points to the id, and id_<id>, which points to the name.
4. In the striping example, which object and which offset hold the byte at offset 1100000?
Block 16, stripe 4, stripepos 0; so object 0 at offset 313568.
5. How many bits does the object map keep per object, and in which bits of the first byte is object 0?
Two bits; object 0 is in the top two bits of the first byte (shift = 6).
6. Why does the object map footer have a separate crc for each 4 KiB block?
So that updating one object rewrites only that 4 KiB block and its crc, without needing the whole map.
7. When is EXISTS_CLEAN set and when is it cleared?
When a snapshot is taken, all EXISTS entries of the head become EXISTS_CLEAN; the next first write to that object turns it back to EXISTS.
8. In fast-diff, if an object is DATA at from_snap and EXISTS in the head, what is the result?
DATA_UPDATED, because the EXISTS state (not EXISTS_CLEAN) means the object was written after the snapshot.
9. Why does the object map depend on exclusive-lock?
Because only one writer should change the map, so that the in-memory and RADOS states stay consistent; the lock owner is that writer.
10. How is a namespace implemented in RBD, and what advantage does it have?
It is the same as a namespace in RADOS, and all the listing objects (rbd_directory and the others) are created separately in each namespace; a client can be restricted to one namespace with a cap without creating a separate pool.

Chapter 38: Specialist Part (5): RBD — The librbd I/O Path, Layering and Copyup, Exclusive Lock, Caches and QoS#

1. Which layers does the EXCLUSIVE_LOCK layer run before, and why?
Before REFRESH, MIGRATION, JOURNAL, WRITE_BLOCK, WRITEBACK_CACHE and CORE; because the journal, object map and cache are valid only for the lock owner, and a write must wait for the lock before it reaches them.
2. When is copyup done without sending the first write?
When the object map says the object does not exist in the clone and the extent is within the parent overlap; librbd copies up directly and attaches the write to the same op.
3. Why does the cls copyup method do nothing if the object exists?
Because if another client has already copied up, the parent's data must not overwrite that other client's newer writes; the accompanying write in the same op is applied anyway.
4. What does deep-flatten add?
Copyup also reaches the clone's snapshots (a separate copyup with an empty SnapContext and an update of the snapshots' object maps), so the clone's snapshots also become independent of the parent.
5. What structure does the cookie of the rbd_lock lock have, and what is it used for?
"auto <watch handle>" with the tag internal; by comparing the address and handle with list_watchers, one can tell whether the lock owner still has a live watch.
6. Why is breaking the lock without a blocklist dangerous?
Because the old owner may only be cut off from the network, and its in-flight writes could later land on the new owner's data.
7. When does writearound not acknowledge a write immediately?
When max_dirty is zero or the FUA flag is given, when the extent overlaps an in-flight write, or when the in-flight bytes have reached rbd_cache_max_dirty.
8. What does rbd_cache_writethrough_until_flush protect against?
Against a guest that never sends a flush; until the first flush, all writes are synchronous.
9. With rbd_qos_iops_limit = 1000 and a 50 ms tick, how many tokens are added per tick?
1000 × 0.05 = 50 tokens.
10. Name two reasons to use rbd-nbd instead of krbd.
Needing features that krbd lacks (journaling, the librbd cache or QoS, LUKS in librbd), and an old kernel that does not support the image's features.

Chapter 39: Specialist Part (6): RBD — Inside rbd-mirror: Journal versus Snapshot#

1. What are the names of the journal's header and data objects?
The header is in journal.<image-id> and the data in journal_data.<pool-id>.<image-id>.<n>.
2. In which object is the event with entry_tid 1234 written, in a journal with splay_width 4 and active_set 9?
splay_offset = 1234 % 4 = 2; object 9 × 4 + 2 = 38, that is journal_data.<pool>.<id>.38.
3. How many bytes of fixed overhead does each journal event have, and which fields make it up?
33 bytes: preamble 8, version 1, entry tid 8, tag tid 8, data length 4 and crc32c 4.
4. Why must the data write wait for the event to become SAFE?
Otherwise, if the client failed after writing the data and before recording the event, the destination (and local replay) would see a change that is not in the journal; the "event first" order guarantees that every data change is in the journal.
5. What prevents the journal from being trimmed?
The commit position of the slowest registered client; sets before the lowest commit set are not removed.
6. Why is the destination always crash-consistent in journal mode?
Because the events are replayed in the source's own order and flush points, so the destination always holds the source's state at some past moment.
7. What are the four mirror snapshot states?
PRIMARY, PRIMARY_DEMOTED, NON_PRIMARY and NON_PRIMARY_DEMOTED.
8. How does rbd-mirror find the changed objects in snapshot mode?
With an object map diff (fast-diff) between two consecutive mirror snapshots on the source, copying only the DATA_UPDATED or HOLE_UPDATED objects.
9. When does snapshot mode use more bandwidth than journal mode?
When writes are small and scattered over many objects; each changed object is transferred whole (up to 4 MiB).
10. How is the leader chosen in rbd-mirror, and after how long is it replaced?
With an exclusive cls_lock on rbd_mirror_leader and a heartbeat every 5 seconds; after two missed heartbeats, the others break the lock and a new leader is elected.

Chapter 40: Specialist Part (7): RGW — The Data Model on RADOS: Metadata, Bucket Index, Versioning, Manifest and Multipart#

1. How do a bucket's entrypoint and instance differ, and which one does a reshard change?
The entrypoint, named after the bucket, only points to the current instance; the instance (.bucket.meta.<bucket>:<id>) has all the properties and the index layout. A reshard creates a new instance with a new bucket_id and moves the entrypoint to it; the marker stays the same.
2. What structure does the key of a bilog entry in the index omap have?
In InIndex mode: the byte 0x80, the prefix 0_, then index_ver in the format %011llu, a dot, the OSD version, and a dot and the subop number.
3. Why do the special keys start with the byte 0x80?
To separate them from ordinary names: in UTF-8 no name starts with 0x80, so the first byte tells the key type. Listing must skip the 0x80 range because non-ASCII names sort after it.
4. How do size and accounted_size in an index entry differ?
size is the actual stored size (after compression or encryption) and accounted_size is the size the user sent and saw; quotas and statistics are based on the latter.
5. If RGW fails after prepare and the data write but before complete, how is the entry repaired?
The tag stays in pending_map. When the tag is older than rgw_pending_bucket_index_op_expiration (120 seconds), the next listing reads the head with check_disk_state and updates or removes the entry with dir_suggest_changes. bucket check --fix does the same for all entries.
6. What is the OLH, and where is it stored?
The object logical head, the pointer to the current version: a 1001_<name> entry in the index (with a change log) and a data-less head object with the original name that has the user.rgw.olh.* xattrs.
7. What does a delete without a versionId do in a versioned bucket?
It creates a new instance of type delete marker and makes it the current version with link_olh; no data is deleted, and a GET without a versionId returns 404.
8. In which pool is the head of a COLD object, and what does it hold?
In the STANDARD pool of the bucket's placement; with no data, only the manifest and xattrs. All the data is in tails in the COLD pool.
9. How is the ETag of a three-part upload built?
The md5 of the concatenation of the parts' binary ETags in order, as hex, plus - and the number of parts.
10. What does the 2~ prefix in the upload_id indicate?
The upload was created in the new format (v2), and the parts' omap keys are part.%08d and therefore sorted in numerical order.

Chapter 41: Specialist Part (8): RGW — The Request Path, SigV4, Quotas, Rate Limits, GC and Lifecycle#

1. What are the main stages of rgw_process_authenticated, in order?
init permissions and reading permissions, init op, verifying the op mask, verifying CORS, verifying op permissions, verifying params, pre-exec, checking the rate limit, execute and complete.
2. How much clock skew does RGW allow for a SigV4 request, and what error does it return?
RGW_AUTH_GRACE, that is 15 minutes; beyond that the request is rejected with RequestTimeTooSkewed.
3. How is the SigV4 signing key derived from the secret key?
Four chained HMAC-SHA256s: the first key is "AWS4" plus the secret over the date, then the region, then the service name s3, and finally the string aws4_request.
4. What limits the throughput of a PUT stream, and which option tunes it?
The amount of data in flight divided by the RADOS write latency; it is tuned with rgw_put_obj_min_window_size (16 MiB) (for GET, rgw_get_obj_window_size).
5. In the default mode, how much does the size quota count for a 100-byte object?
4096 bytes, because DefApplier rounds the size up to a multiple of 4 KiB (unless check_on_raw is on).
6. Why is the quota in RGW "soft"?
Each RGW reads the statistics from its own cache, and the cache is refreshed after rgw_bucket_quota_ttl; several RGWs at the same time can go slightly over the cap.
7. In practice, what is a rate limit of 600 read ops per minute behind three RGWs?
Up to about 1800 per minute, because each RGW has its own token bucket.
8. How is a chain's gc shard chosen, and why is the tail not deleted immediately?
XXH64 of the tag with a fixed seed, modulo rgw_gc_max_objs (32). The tail is kept for at least rgw_gc_obj_min_wait (2 hours) so that in-flight GETs holding the old manifest are not left incomplete.
9. How does lifecycle map a bucket to a shard, and in what time window does it work?
The key "tenant:bucket:marker" goes to an lc.N object via ceph_str_hash_linux modulo 7877 and then modulo rgw_lc_max_objs; the workers only work within rgw_lifecycle_work_time (00:00 to 06:00).
10. Why does bucket_lc_process check the marker?
If the bucket was deleted and recreated with the same name, the marker differs, and the stale entry must not run against the new bucket.

Chapter 42: Specialist Part (9): RGW — Inside Multisite: Periods, Log Formats, Bucket Generations and the Sync Engine#

1. What three conditions does committing a period check before anything else?
The commit happens only on the master zone, predecessor_uuid must equal the current period, and realm_epoch must be exactly one more than the current period's.
2. When does a commit change the period id, and what is the consequence for the mdlog?
When the master zone changes; a new period is created with the first epoch and the mdlog starts with the new prefix meta.log.<period>., so the zones must read the old period's log to its end and then move on to the new log.
3. What problem does zones_trace in the bilog solve?
It prevents changes from bouncing back and forth endlessly in active-active: the zones a change has passed through are recorded, and it is not sent to them again.
4. What does the key of a datalog entry identify?
A bucket shard (bucket:instance:shard) and, when needed, its generation; that is, "something changed in this shard, read its bilog".
5. In which object and which pool is the sync status of a bucket shard?
In bucket.sync-status.<src-zone>:<bucket>:<shard> (with the generation suffix if there is one), in the log pool of the destination zone.
6. Why does the datalog shard marker advance despite a bucket's failure, and where is the failure recorded?
So that one broken bucket does not stall the whole shard. The failed key is recorded in the error repo, that is the omap of the object datalog.sync-status.shard.<zone>.<N>.retry, and retried separately.
7. What roles do the lease and the bid play in sync?
They ensure each shard has only one processor at any moment (the lease, with rgw_sync_lease_period), and the bid spreads the shards among the RGWs.
8. After a reshard in the source zone, how does the destination zone continue without a full sync?
The source keeps the previous generation's bilogs; the destination finishes all shards of the old generation (shards_done_with_gen), then moves incremental_gen to the new generation with its new number of shards.
9. In what order of criteria is a concurrent-write conflict resolved?
mtime (at one-second precision unless high precision), then zone_short_id, and finally pg_ver.
10. Why does creating a bucket in a non-master zone depend on the master being available?
Metadata (users, buckets) is written only on the master; other zones forward the request to the master and receive the result through metadata sync.