Specialist Part (4): RBD — Layout on RADOS, Striping, Object Map and Fast-Diff, and the rbd Reference

37-1 Before we begin#

Chapter 2 showed which objects a format 2 image has and how its id is generated (algorithm 2-1). This chapter opens that same layout key by key: exactly which omap keys an image header has, what the pool-level objects (rbd_directory, rbd_trash, rbd_mirroring, rbd_namespace and others) hold, how striping with stripe_unit and stripe_count maps an offset to an object, how the object map packs and stores its bits, and how fast-diff finds the changes between two snapshots without reading data. The chapter ends with a complete reference for the rbd command and rbd-nbd. Names have been checked against the main branch of the ceph/ceph repository (November 2026).

37-1-1 What questions does this chapter answer?#

• What keys does an image header have, and who writes each of them?
• How does the data object name differ with a separate data pool?
• With stripe_unit=64K and stripe_count=4, which object holds offset 1000000?
• How many bits does the object map keep per object, and in what order are the bytes on disk?
• What does EXISTS_CLEAN mean, and when is it set?
• Which states does fast-diff use to detect a change?
• How are pools, namespaces, the trash and groups indexed in RADOS?

37-1-2 Prerequisites#

Chapter 2 (RBD), chapter 15 (cls), chapters 34 and 36 (omap, SnapContext and clones).

37-2 The objects of an image#

37-2-1 The header#

All the metadata of a format 2 image is in the omap of the object rbd_header.<id> and changes only through the methods of the rbd class (src/cls/rbd/cls_rbd.cc):

Besides the omap, the xattr lock.rbd_lock (the lock class) holds the advisory lock that ManagedLock uses for exclusive-lock (chapter 38). This same header object is the watch target of all clients that have the image open.

The default sum (rbd_default_features) is 1+4+8+16+32 = 61.

37-2-2 Data objects and the object map#

Data objects are named <object_prefix>.<objectno:016x>. If the image is created with --data-pool, the header and metadata stay in the main pool (which must be replicated) and the data is written to the second pool (often EC with allow_ec_overwrites). In this case the prefix includes the main pool's id: rbd_data.<metadata-pool-id>.<image-id>, so that objects of images from several different pools do not collide in a shared data pool.

The object map is stored in the object rbd_object_map.<id> for the head and rbd_object_map.<id>.<snapid:016x> for each snapshot.

37-2-3 Pool-level objects#

A namespace in RBD is the same as a namespace in RADOS (chapter 34). All the objects listed above are created separately in each namespace. With the cap profile rbd pool=P namespace=N a client can be restricted to one namespace without creating a separate pool for each tenant.

# example: reading the header directly (for inspection only; never change it with rados)
rados -p rbd listomapkeys rbd_header.5f3a2b8c1d4e
rados -p rbd getomapval rbd_header.5f3a2b8c1d4e object_prefix

example (output):
access_timestamp
create_timestamp
features
flags
modify_timestamp
object_prefix
order
size
snap_seq
snapshot_0000000000000004
value (25 bytes) :
00000000  15 00 00 00 72 62 64 5f  64 61 74 61 2e 35 66 33  |....rbd_data.5f3|
00000010  61 32 62 38 63 31 64 34  65                       |a2b8c1d4e|

The first four bytes (15 00 00 00, that is 21) are the string length in little-endian; all strings in Ceph's encoding have this format.

37-3 Striping#

By default stripe_unit equals the object size and stripe_count is 1, so the image is simply divided into consecutive 4 MiB pieces. With custom striping, data is spread round-robin across stripe_count objects, so that small sequential writes go to several OSDs at once.

pseudocode
su = stripe_unit ; sc = stripe_count ; os = object_size
if sc == 1: su = os
stripes_per_object = os / su
while left > 0:
    blockno     = cur / su
    stripeno    = blockno / sc
    stripepos   = blockno % sc
    objectsetno = stripeno / stripes_per_object
    objectno    = objectsetno * sc + stripepos
    x_offset    = (stripeno % stripes_per_object) * su + cur % su
    x_len       = min(su - cur % su, left)
    emit (objectno, x_offset, x_len) ; cur += x_len ; left -= x_len

Numerical example. object_size = 4 MiB, stripe_unit = 64 KiB, stripe_count = 4, a write of 300000 bytes from offset 1000000. Each object has 64 stripes and one "object set" of 4 objects covers 16 MiB.

The 300 KB write reaches all four objects. With default striping, the same write would touch only object 0 (offsets 1000000 to 1300000). Striping helps a single-stream sequential load with a low queue depth, but for a random 4K load it only increases the number of objects involved. krbd supports custom striping from kernel 4.17 (chapter 29).

rbd create pool/img --size 10G --object-size 4M --stripe-unit 64K --stripe-count 4

37-4 The object map#

37-4-1 States and bit packing#

The object map has 2 bits per object (BitVector<2>):

Each byte holds four objects, and the object with the lower number is in the higher bits: shift = (3 − objectno % 4) × 2. The on-disk encoding has three parts: a header (length, version and element count), the raw data, and a footer containing the crc32c of the header and one crc32c for each 4096-byte block of data. So updating one object rewrites only one 4 KiB block and its crc (the object_map_update method in cls_rbd).

37-4-2 Who changes the state?#

• A write to an object that is NONEXISTENT or EXISTS_CLEAN updates the object map to EXISTS before the write is sent (first in memory, then with object_map_update on RADOS). If the object is already EXISTS, it costs nothing.
• A discard or deletion first sets the state to PENDING, deletes the object, and then sets NONEXISTENT.
• Taking a snapshot copies the head map to rbd_object_map.<id>.<snapid> and then turns all EXISTS entries of the head into EXISTS_CLEAN (object_map_snap_add in cls_rbd).
• Deleting a snapshot merges the EXISTS bits of that snapshot's map into the map of the next snapshot (or the head), so the "changed" information is not lost (object_map_snap_remove).

The object map is valid only with exclusive-lock, because only the lock owner may change it. If the client fails in the middle of an update, the OBJECT_MAP_INVALID flag is set in flags and rbd object-map rebuild must be run.

37-4-3 fast-diff#

pseudocode
# for each snapshot from from_snap to end_snap (and head), its map is loaded and combined with the previous state
first map (from_snap):  state[o] = HOLE if NONEXISTENT else DATA (PENDING → DATA_UPDATED)
each later map m:
    HOLE:         if m[o] != NONEXISTENT → DATA_UPDATED     (diff-iterate: only in the final map)
    DATA:         if m[o] == NONEXISTENT → HOLE_UPDATED
                  elif m[o] != EXISTS_CLEAN → DATA_UPDATED
    HOLE_UPDATED: if m[o] != NONEXISTENT → DATA_UPDATED
    DATA_UPDATED: if m[o] == NONEXISTENT → HOLE_UPDATED
objects beyond the previous size (image grew) → HOLE or DATA_UPDATED

The key point is EXISTS_CLEAN: if an object existed at from_snap and stayed EXISTS_CLEAN in all later maps, it had no writes after from_snap and does not appear in the diff. rbd diff --whole-object, rbd du and rbd export-diff (and snapshot mode in rbd-mirror) all use this calculation. If FAST_DIFF_INVALID is set, librbd falls back to list_snaps on each individual object, which is much slower.

Numerical example. A 10 GiB image with 2560 objects. After snap1, twenty objects have been written and three have been discarded. With fast-diff, rbd export-diff --from-snap snap1 reads only two 640-byte maps (plus header and footer) and finds the 23 objects. Without it, 2560 list_snaps calls would be needed.

37-5 rbd reference#

37-5-1 Images and snapshots#

37-5-2 Mapping#

# example: creating an image with striping and a data pool, and inspecting it
rbd create rbd/vm01 --size 10G --stripe-unit 64K --stripe-count 4 --data-pool rbd-ec
rbd info rbd/vm01

example (output):
rbd image 'vm01':
	size 10 GiB in 2560 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 5f3a2b8c1d4e
	data_pool: rbd-ec
	block_name_prefix: rbd_data.3.5f3a2b8c1d4e
	format: 2
	features: layering, striping, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool
	op_features:
	flags:
	create_timestamp: Sun Nov 15 09:41:07 2026
	access_timestamp: Sun Nov 15 09:41:07 2026
	modify_timestamp: Sun Nov 15 09:41:07 2026
	stripe unit: 64 KiB
	stripe count: 4

block_name_prefix is the object_prefix, and the number 3 in it is the id of the rbd pool, because we have a separate data pool.

37-6 Tuning and troubleshooting#

• Slow first writes on a new image: each first write to an object also involves an object_map_update. Once the image is filled, this cost disappears. For performance testing, fill the image first.
• rbd du or export-diff is slow: look at rbd info; if it shows flags: object map invalid, fast diff invalid, run rbd object-map rebuild (separately on the snapshots too).
• An image with --data-pool on EC cannot be created: check allow_ec_overwrites on the EC pool and that rbd pool init has been run on both pools.
• Deleting a large image is slow: deletion depends on the number of existing objects; with the object map only existing objects are deleted. For long jobs use rbd trash mv followed by rbd trash rm or ceph rbd task add remove.
• Never change the header with rados setomapval: cls_rbd checks for inconsistencies, but open clients are not told about the change (no notify is sent).

37-7 Supplementary and up to date#

• Format 2 clones (the default rbd_default_clone_format: auto chooses format 2 if all clients are Mimic or newer) do not need protect and keep the children in snap_children_* of the parent's own header.
• The op_features value in new versions has bits for snapshots in the trash and for groups; an old client that sees an unknown op_feature does not open the image.
• In the main branch, rbd-nbd with netlink and --try-netlink is more stable than the ioctl method; ubbd support depends on the distribution's packaging.

37-8 Hands-on exercise#

1. 1. Create an image and read all the header keys with rados listomapkeys and getomapval. Then take a snapshot and find the new key.
2. 2. Create an image with --stripe-unit 64K --stripe-count 4, write 300000 bytes at offset 1000000 (rbd bench or dd on rbd-nbd) and see with rados ls | grep which objects were created. Compare with the example table.
3. 3. Take rados get rbd_object_map.<id> /tmp/om and with xxd match the first four data bytes (after the eighteen-byte header) against the states of objects 0 to 15.
4. 4. After a snapshot, write a few objects and time rbd diff --from-snap with and without fast-diff (after rbd feature disable ... fast-diff).

37-9 Review questions#

1. 1. Which header keys are needed to build the names of the data objects?
2. 2. With a separate data pool, how do the data object names change, and why?
3. 3. What two kinds of key does rbd_directory have?
4. 4. In the striping example, which object and which offset hold the byte at offset 1100000?
5. 5. How many bits does the object map keep per object, and in which bits of the first byte is object 0?
6. 6. Why does the object map footer have a separate crc for each 4 KiB block?
7. 7. When is EXISTS_CLEAN set and when is it cleared?
8. 8. In fast-diff, if an object is DATA at from_snap and EXISTS in the head, what is the result?
9. 9. Why does the object map depend on exclusive-lock?
10. 10. How is a namespace implemented in RBD, and what advantage does it have?

Answers to these questions are in Appendix E: Answers to Review Questions. Answers to this chapter's questions →

37-10 Sources for this chapter#

• cls_rbd.cc
  https://github.com/ceph/ceph/blob/main/src/cls/rbd/cls_rbd.cc
• rbd_types.h and features.h
  https://github.com/ceph/ceph/blob/main/src/include/rbd/features.h
• Striper.cc
  https://github.com/ceph/ceph/blob/main/src/osdc/Striper.cc
• bit_vector.hpp
  https://github.com/ceph/ceph/blob/main/src/common/bit_vector.hpp
• librbd/object_map/DiffRequest.cc
  https://github.com/ceph/ceph/blob/main/src/librbd/object_map/DiffRequest.cc
• RBD layering and striping documentation
  https://docs.ceph.com/en/latest/dev/rbd-layering/
• rbd command reference
  https://docs.ceph.com/en/latest/man/8/rbd/
• rbd-nbd reference
  https://docs.ceph.com/en/latest/man/8/rbd-nbd/