Specialist Part (7): RGW — The Data Model on RADOS: Metadata, Bucket Index, Versioning, Manifest and Multipart

40-1 Before we begin#

Chapter 4 showed that RGW keeps each S3 object in two places (the bucket index and the data) and presented the algorithms for choosing the shard, resharding, listing, and naming the head and tail objects. This chapter goes one layer down: exactly which pool and namespace hold user and bucket metadata, what fields an index entry has and how its omap keys are built, how the two-phase index update copes with failure, how versioning is implemented with the OLH, how the manifest lays out the head and tail across placement and storage classes, and how multipart completes without copying data. The chapter ends with a radosgw-admin reference. Names and defaults have been checked against the main branch of the ceph/ceph repository (November 2026).

40-1-1 What questions does this chapter answer?#

• In which RADOS objects are the users, keys and buckets?
• What fields does an object's index entry have, and what are the 0x80 keys in the index omap?
• What happens if RGW fails between writing the data and updating the index?
• How is the "current" version of an object chosen in a versioned bucket?
• In which pool is an object's head written if its storage class is COLD?
• How is the ETag of a multipart object computed?

40-1-2 Prerequisites#

Chapter 4 (RGW), chapter 15 (cls), chapter 34 (omap and xattrs) and chapter 36 (the atomicity of cls).

40-2 Metadata: pools and namespaces#

Each zone has a set of pools, most of which are implemented as namespaces inside a few shared pools (rgw::init_zone_pool_names in rgw_zone.cc). The full name is built with the zone prefix; for the zone default:

A bucket has two metadata objects. The entrypoint, named after the bucket itself (or <tenant>/<bucket>) in root, only points to the current instance. The instance, named .bucket.meta.<bucket>:<bucket-id>, has all the properties: placement, the number of shards and the index layout, the owner, the versioning and object lock flags, and xattrs such as policy, lifecycle and CORS. Resharding creates a new instance with a new bucket-id and only moves the entrypoint.

# example: viewing a bucket's and a user's metadata
radosgw-admin metadata get bucket:photos
radosgw-admin metadata get bucket.instance:photos:a1b2c3d4-0000-4000-8000-1234567890ab.4123.7
rados -p default.rgw.meta -N users.keys ls | head -3

example (shortened output):
{ "key": "bucket:photos",
  "data": { "bucket": { "name": "photos", "marker": "a1b2c3d4-...4123.7", "bucket_id": "a1b2c3d4-...4123.7" },
            "owner": "alice", "creation_time": "2026-11-02T08:14:33.120Z" } }
AKIAEXAMPLE7Q2WZ3L
AKIAEXAMPLEM9K1D0B
AKIAEXAMPLEP5T8H4N

The marker is the prefix from which the names of the data and index objects are built (chapter 4), and it does not change after a reshard, whereas bucket_id does. The users.keys list holds only the access key ids; the secret key is inside the user record and must never appear in a log or report.

40-3 The bucket index#

40-3-1 Objects and keys#

Each bucket's index is in the objects .dir.<marker>.<shard> (or, for generations after a reshard, .dir.<marker>.<gen>.<shard>) in the index pool. Each index object has only omap. The omap header keeps the shard's statistics (count, size and actual size per category) and the index version. The omap keys have several key spaces (cls_rgw.cc):

Since in UTF-8 no name starts with the byte 0x80 (that byte is only a continuation byte of a multi-byte character), the bi_entry_type function in cls rgw tells ordinary keys from special ones by looking at the first byte. Special keys sort after all ASCII names and before names that start with non-ASCII characters (first byte 0xC2 and up); hence listing (get_obj_vals) must explicitly skip the 0x80 range, and "a scan up to 0x80" alone is not enough.

40-3-2 The index entry#

rgw_bucket_dir_entry has these fields: key (name and instance), ver (pool and epoch, that is the RADOS version of the head object), locator, exists, meta (category, size, mtime, etag, owner, owner_display_name, content_type, accounted_size, user_data, storage_class, appendable and the restore status), pending_map (unfinished operations), index_ver, tag, flags (VER, CURRENT, DELETE_MARKER, VER_MARKER) and versioned_epoch.

accounted_size is the size the user saw (before compression or encryption) and size is the actual stored size; quotas and bucket statistics are computed from accounted_size.

40-3-3 The two-phase update#

Writing the data and updating the index happen in two separate objects (and usually on two separate OSDs), and RADOS has no cross-object transactions. RGW coordinates the two with a two-phase protocol.

pseudocode
PUT key:
    tag = random op tag
    cls rgw.bucket_prepare_op(shard, key, op=ADD, tag)
        entry = read or init(exists=false) ; entry.pending_map[tag] = {PENDING_MODIFY, now, op}
    write head (and tails) to data pool ; get head RADOS version (epoch)
    cls rgw.bucket_complete_op(shard, key, op=ADD, tag, ver=epoch, meta)
        erase pending_map[tag]
        if ver older than entry.ver: skip (stale completion)
        unaccount old entry stats ; entry.meta = meta ; entry.exists = true ; account new
        append bilog entry (if logging enabled)
    on write failure: complete_op(op=CANCEL) → just erase tag (no bilog)
crash between prepare and complete:
    entry stays with stale pending_map ; listing sees pending older than rgw_pending_bucket_index_op_expiration (120 s)
    → RGW checks head object's real state (check_disk_state)
    → cls rgw.dir_suggest_changes(UPDATE or REMOVE) fixes the entry

The practical consequence: the index may be inconsistent with the data for a short time (the object is written but not yet listed, or the other way round), but listing repairs itself. radosgw-admin bucket check --fix --check-objects performs the same check for all entries.

40-4 Versioning and the OLH#

In a versioned bucket each version has a separate head object with a random instance (for example 3sL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY). The current version is identified by an OLH (object logical head): a 1001_ entry in the index and a data-less head object with the original name, which points to the current instance in the xattrs user.rgw.olh.info, user.rgw.olh.ver, user.rgw.olh.idtag and user.rgw.olh.pending.*.

pseudocode
PUT on versioned bucket:
    write instance head object <name> with instance id ; index prepare/complete on 1000_<name>\0i<instance>
    cls rgw.bucket_link_olh(key, olh_tag, delete_marker = false, olh_epoch = 0 or remote epoch):
        candidate_epoch = olh_epoch from multisite, else now in ns   # main
        if olh.start_modify(candidate_epoch):
            promote = newer epoch, or same epoch and instance sorts after current
            if promote: old current loses FLAG_CURRENT ; this instance gets FLAG_CURRENT
                        plain entry <name> now mirrors this instance
            olh.update_log(LINK_OLH, op_tag, key, delete_marker)
            olh.update(key, delete_marker)
        bilog entry LINK_OLH (or LINK_OLH_DM for delete marker)
    RGW then reads OLH log and applies it to the OLH head xattrs (apply_olh_log), then trims the log
DELETE without versionId: link_olh with delete_marker = true (new instance, no data)
DELETE with versionId: unlink_instance ; if it was current → promote next newest

Example. The object report.pdf was uploaded three times and then deleted without a versionId. The index has four instance entries (three versions and one delete marker with the flags DELETE_MARKER|CURRENT), one OLH entry and one ordinary entry. GET report.pdf looks at the OLH, sees that the current version is a delete marker and returns 404. GET report.pdf?versionId=<v2> reads the head of the second version directly.

40-5 Manifest, placement and storage classes#

40-5-1 Layout rules#

Each bucket has a placement target (from the zonegroup), which the zone maps to three pools: index_pool, data_extra_pool and, for each storage class, a data_pool (with optional compression). The head is always in the pool of the default storage class (STANDARD) of the bucket's placement; if the object has a different storage class, the head holds only metadata (the manifest and xattrs) and all the data is in tails in that class's pool.

pseudocode
head_pool = data pool of (bucket placement, STANDARD)
max_head_chunk = rgw_max_chunk_size (4 MiB), aligned to pool alignment (EC)
if object storage class != STANDARD:
    tail_pool = data pool of that class ; head_max_size = 0 ; chunk = tail pool max chunk
elif placement.inline_data:
    head_max_size = max_head_chunk ; chunk = max_head_chunk
else: head_max_size = 0
stripe = rgw_obj_stripe_size (4 MiB), aligned
manifest.set_trivial_rule(head_max_size, stripe)
data: first head_max_size bytes → head ; rest → tail objects of `stripe` bytes, written in chunks
xattrs on head: user.rgw.manifest, user.rgw.etag, user.rgw.acl, user.rgw.idtag, user.rgw.tail_tag, user.rgw.storage_class, ...

Numerical example. A 10 MiB PUT in the STANDARD class: the head holds 4 MiB of data and two tails of 4 MiB and 2 MiB are created (three RADOS objects). The same PUT with x-amz-storage-class: COLD: a data-less head in the STANDARD pool and three tails of 4, 4 and 2 MiB in the COLD pool. So even for a cold class, the STANDARD pool has one small head per object; this matters when sizing an SSD pool. A lifecycle transition (chapter 41) produces the same pattern: new tails in the destination pool and a new manifest on the same head.

40-6 Multipart#

pseudocode
init(key): upload_id = "2~" + random ; create meta object "<key>.<upload_id>.meta"
           (namespace multipart, data_extra_pool) ; index entry in multipart namespace
upload part n (size >= rgw_multipart_min_part_size 5 MiB, except last):
    write part head "<marker>__multipart_<key>.<upload_id>.<n>" + shadow stripes
    omap set on meta object: key "part." + %08d(n) → RGWUploadPartInfo{num, size, etag, manifest}
complete(list of (n, etag)):
    read part infos from meta omap (sorted thanks to %08d) ; verify each etag and order
    manifest = concatenation of part manifests (no data copy)
    etag = hex(md5(concat(binary md5 of each part))) + "-" + P
    write final head <key> with manifest and attrs ; index prepare/complete
    remove meta object and its index entry ; parts not listed → gc

Numerical example. Three parts of 5, 5 and 2 MiB, filled with A, B and C respectively, have the ETags b8fc857a25e7958868c2f003d5e0952d, ba8c3fac0e224c9b79a8e74bebd54654 and bf52955b454974e73c3e2694bd7207c4. The final ETag is the md5 of the concatenation of their 48 binary bytes: eacd2737fa049d51213f10cdc5732ece-3. This value is not the md5 of the whole file; a tool that compares the ETag with the file's md5 must recognize the -N suffix.

The 2~ prefix in the upload_id shows that the part keys are stored sorted with %08d. Old uploads (prefix 2/) stored parts without leading zeros, and complete had to sort them numerically.

40-7 radosgw-admin reference#

# example: the manifest of a 10 MiB object in STANDARD
radosgw-admin object stat --bucket photos --object big.bin | jq '.manifest | {obj_size, head_size, max_head_size, rules}'

example (output):
{
  "obj_size": 10485760,
  "head_size": 4194304,
  "max_head_size": 4194304,
  "rules": [ { "key": 0, "val": { "start_part_num": 0, "start_ofs": 4194304, "part_size": 0,
               "stripe_max_size": 4194304, "override_prefix": "" } } ]
}

40-8 Tuning and troubleshooting#

• Slow listing and wrong bucket stats: there are many old pending entries. Run radosgw-admin bucket check --check-objects --fix at a quiet time.
• Large omap in the index pool: a shard has more than rgw_max_objs_per_shard (100000) entries, or (in InIndex mode) the bilog has not been trimmed. Look at bucket limit check and bilog list (chapter 42).
• The STANDARD pool fills up despite a cold class: each object has one head in STANDARD; for billions of small objects this alone is a lot of space and objects.
• Space in the data pool is not freed: deleted tails are waiting for gc (by default at least 2 hours; chapter 41). Look at gc list --include-all.
• Abandoned multipart uploads: only lifecycle with AbortIncompleteMultipartUpload or radosgw-admin bucket rm --purge-objects removes them; rados ls on data_extra_pool shows how many there are.

40-9 Supplementary and up to date#

• In the main branch, the epoch of versions in link_olh is taken from the current time in nanoseconds (unless it comes from multisite). Older versions behaved differently, and this matters when versions are compared across zones.
• The inline_data option in the placement (on by default) determines whether the head holds data; turning it off helps EC pools that store small heads poorly.
• In the main branch, new buckets keep the bilog by default in a separate FIFO instead of the index omap (rgw_default_bucket_bilog_type = fifo); 0_ keys appear only in inindex mode and in old buckets.
• The :accounts, :groups and :vector pools and dedup_pool have been added in new versions; check your version before relying on them.

40-10 Hands-on exercise#

1. 1. Create a bucket and find its entrypoint and instance with rados -p default.rgw.meta -N root ls. Look again after bucket reshard.
2. 2. With rados -p default.rgw.buckets.index listomapkeys .dir.<marker>.0 | xxd | head, look at the 0x80 keys in a versioned bucket.
3. 3. Upload a 10 MiB object once in STANDARD and once in COLD, and compare where the head and tails are with object stat and rados ls.
4. 4. Do a three-part upload and match the final ETag against a manual calculation (the md5 of the concatenated md5s).

40-11 Review questions#

1. 1. How do a bucket's entrypoint and instance differ, and which one does a reshard change?
2. 2. What structure does the key of a bilog entry in the index omap have?
3. 3. Why do the special keys start with the byte 0x80?
4. 4. How do size and accounted_size in an index entry differ?
5. 5. If RGW fails after prepare and the data write but before complete, how is the entry repaired?
6. 6. What is the OLH, and where is it stored?
7. 7. What does a delete without a versionId do in a versioned bucket?
8. 8. In which pool is the head of a COLD object, and what does it hold?
9. 9. How is the ETag of a three-part upload built?
10. 10. What does the 2~ prefix in the upload_id indicate?

Answers to these questions are in Appendix E: Answers to Review Questions. Answers to this chapter's questions →

40-12 Sources for this chapter#

• cls_rgw.cc and cls_rgw_types.h
  https://github.com/ceph/ceph/blob/main/src/cls/rgw/cls_rgw.cc
• rgw_zone.cc (pool names
  https://github.com/ceph/ceph/blob/main/src/rgw/rgw_zone.cc
• rgw_putobj_processor.cc
  https://github.com/ceph/ceph/blob/main/src/rgw/driver/rados/rgw_putobj_processor.cc
• rgw_sal_rados.cc (multipart
  https://github.com/ceph/ceph/blob/main/src/rgw/driver/rados/rgw_sal_rados.cc
• Bucket index and layout documentation
  https://docs.ceph.com/en/latest/dev/radosgw/bucket_index/
• Placement and storage class documentation
  https://docs.ceph.com/en/latest/radosgw/placement/
• radosgw-admin
  https://docs.ceph.com/en/latest/man/8/radosgw-admin/