Ceph Code Architecture and the Request Path from the Inside

22-1 Repository map#

The github.com/ceph/ceph repository is a large C++ monorepo with Python parts. On the main branch, the src/ directory contains, in addition to Ceph's own code, several submodules and external libraries (rocksdb, seastar, spdk, isa-l, fmt, googletest, s3select, arrow, zstd and so on). Knowing this boundary is the first step in reading the code: whatever is inside these submodules is not Ceph code.

22-2 Build and development environment#

The build is done with CMake and Ninja. The do_cmake.sh script at the repository root creates the build directory (its name can be changed with the BUILD_DIR variable) and runs CMake with -GNinja. Extra flags can be passed with the ARGS variable. The target named vstart in src/CMakeLists.txt builds only what is needed for a local development cluster.

git clone https://github.com/ceph/ceph.git && cd ceph
git submodule update --init --recursive
./install-deps.sh
ARGS="-DCMAKE_BUILD_TYPE=Debug -DWITH_MGR_DASHBOARD_FRONTEND=OFF" ./do_cmake.sh
cd build && ninja vstart            # only the components needed for vstart
# local cluster: 1 MON, 3 OSDs, no MDS
MON=1 OSD=3 MDS=0 MGR=1 ../src/vstart.sh -n -d -x --bluestore
./bin/ceph -s
../src/stop.sh

The vstart.sh options, from the script's own help text: -n (new cluster), -d (debug), -x/-X (enable/disable cephx), -l (localhost) and -b (BlueStore, which is the default). The number of daemons is set with the environment variables MON, OSD, MDS, MGR, RGW and so on.

22-3 Main classes and relationships#

22-3-1 The ObjectStore interface#

src/os/ObjectStore.h is the boundary between PG logic and storage. The OSD never writes directly to disk. The OSD builds an ObjectStore::Transaction (defined in src/os/Transaction.h) containing a list of operations such as write, zero, truncate, remove, setattrs, omap_setkeys and omap_rmkeys on collections, and hands it to the backend with queue_transactions. The backend guarantees that the whole transaction is atomic. This property is why the data object and the pg log entry become durable at the same time. The existing implementations are BlueStore (the default), MemStore (testing) and, in crimson, SeaStore and CyanStore.

22-4 Thread model#

22-5 Encoding and version compatibility#

Every structure that goes over the network or to disk is serialized with the include/encoding.h infrastructure or its faster version include/denc.h. The main pattern is versioning each struct with two numbers: struct_v (the current version) and compat (the lowest version whose decoder can read this data). The block length is also stored so that an old decoder can skip new trailing fields. The macros are ENCODE_START(v, compat, bl), ENCODE_FINISH(bl), DECODE_START(v, bl), DECODE_FINISH(bl) and WRITE_CLASS_ENCODER(cl). In denc, the equivalents are DENC_START(v, compat, p) and WRITE_CLASS_DENC(T).

// this guide's own example (not from Ceph code): adding a field while keeping compatibility
struct demo_info_t {
  uint64_t size = 0;
  std::string owner;          // added in version 2
  void encode(ceph::buffer::list& bl) const {
    ENCODE_START(2, 1, bl);   // v=2, a version 1 decoder can also read it
    encode(size, bl);
    encode(owner, bl);
    ENCODE_FINISH(bl);
  }
  void decode(ceph::buffer::list::const_iterator& p) {
    DECODE_START(2, p);
    decode(size, p);
    if (struct_v >= 2) decode(owner, p);
    DECODE_FINISH(p);         // unknown fields from later versions are skipped
  }
};
WRITE_CLASS_ENCODER(demo_info_t)

22-6 Reading and debugging the code in practice#

# reading an op's path in the log: follow the tid and reqid
ceph config set osd.3 debug_osd 20 ; ceph config set osd.3 debug_bluestore 20
rados -p test put obj1 /etc/hosts
grep -E 'do_op|execute_ctx|issue_repop|_txc_state_proc' /var/log/ceph/ceph-osd.3.log | tail
ceph config rm osd.3 debug_osd ; ceph config rm osd.3 debug_bluestore
ceph-dencoder list_types | grep -i -E 'osdmap|pg_log'

22-7 The full path of a RADOS write, function by function#

For EC, step 8 becomes ECBackend::submit_transaction and the read-modify-write pipeline in ECCommon::RMWPipeline::start_rmw. On this path, for a partial write the existing shards are read first, parity is computed, and then all shards are written.

pseudocode: do_op on the primary (this book's summary of PrimaryLogPG.cc)
  do_op(op):
     if pg not active or op.epoch > osdmap: wait / requeue
     if client blocklisted or cap denies: reply -EPERM/-EBLOCKLISTED
     obc = get_object_context(oid)            # may wait for recovery of this object
     if object degraded/missing: wait_for_degraded_object
     ctx = OpContext(op, obc) ; execute_ctx(ctx)
  execute_ctx(ctx):
     r = prepare_transaction(ctx)             # do_osd_ops: each OSDOp → Transaction operations
     if read-only: reply now
     ctx.log += pg_log_entry_t(version = ++last_update, reqid, prior_version)
     repop = new RepGather(ctx) ; issue_repop(repop, ctx)

22-8 Inside BlueStore: a TransContext#

22-9 The RBD write path#

22-10 The PUT path in RGW#

22-11 The open and write path in CephFS#

22-12 The write path with real OpTracker timings#

The function-by-function path of the previous section can be timed with the OSD's own tool. Each OpRequest records timestamped points in OpTracker. Some are fixed flags: mark_queued_for_pg, mark_reached_pg, mark_started and mark_commit_sent in src/osd/OpRequest.h. Some are text events: "op_commit", "sub_op_commit_rec" and "waiting for subops from …" in src/osd/ReplicatedBackend.cc.

On a three-OSD BlueStore test cluster with Ceph 18.2.7, we set osd_op_history_size to 1000. Then we wrote 300 4 KiB objects one after another into pool p22 with size = 2 and read ceph daemon osd.N dump_historic_ops from all three OSDs.

pseudocode
for each op in ops (all three OSDs):
    if description starts with "osd_op(" and pool = 4 ← primary ; if "osd_repop(" ← replica
    ev = [(event, time)] in recorded order
    for each consecutive pair (a, b): stage["a → b"].append(t_b − t_a)
median and 90th percentile of each stage ; count the order of op_commit and sub_op_commit_rec

Result (median / 90th percentile, microseconds). The whole write on the primary took about 1637 microseconds. Most of this time was spent waiting for commits. In 268 of the 300 ops, the local commit (op_commit) arrived before the replica's confirmation. The interval from waiting for subops to op_commit had a median of 1006 and a 90th percentile of 1407 microseconds. After that, it took another 356 microseconds until sub_op_commit_rec. In the remaining 32 ops the order was reversed.

The path before the PG is cheap. Its largest share was the sharded op queue: queued_for_pg to reached_pg with a median of 54 and a 90th percentile of 233. After that comes reached_pg to started at 86 microseconds, which is the time for the PG lock and preparing the op context.

On the replica, the started to commit_sent part (the BlueStore transaction and kv sync), with a median of 809 microseconds, is the main part. When the two paths are put on one axis, the replica finishes its work about 1170 microseconds after the start. But the primary sees the confirmation at about 1600 microseconds. This gap of several hundred microseconds is the cost of the return messenger path and the reply being queued, and it grows on a real network.

Practical lesson: when analysing a slow op (chapter 17), first see which two events have the largest gap. A large gap before reached_pg means the queue or the PG lock. A large gap between waiting for subops and sub_op_commit_rec means the replica or the network. A large gap up to op_commit means the local device or BlueStore.

22-13 Supplementary and up to date#

• crimson is still maturing in Tentacle and Umbrella. The classic OSD (src/osd/) and crimson (src/crimson/osd/) have separate op path code but share encoding, OSDMap and CRUSH. Before operational use, check the release notes of that version.
• The cls class method registration API on main has changed to a typed form (ClassRegistrar) (chapter 15). Build out-of-tree class code against the target version's tag.
• neorados (in src/neorados/) is a modern, asynchronous asio-based C++ interface, and RGW and librbd are gradually adopting it.

22-14 Hands-on exercise#

1. 1. Bring up a vstart cluster with MON=1 OSD=3 MDS=0, enable debug_osd=20 on one of the OSDs and run a rados put. Find the ten functions of the "full path of a write" table in the log and note their order.
2. 2. Decode the map with ceph osd getmap -o om and ceph-dencoder type OSDMap import om decode dump_json | jq '.epoch, .pools[0]'.
3. 3. Compile the demo_info_t structure in a small unit test. Encode data with version 2 and decode it with a decoder that only knows the size field.
4. 4. On a stopped OSD, find the last pg log entry with ceph-objectstore-tool --op log and compare the version, prior_version and reqid fields with the table in chapter 1.
5. 5. During a rados bench test, run thread apply all bt on an OSD with gdb -p and identify the tp_osd_tp, msgr-worker and bstore_kv_sync threads.

22-15 Review questions#

1. 1. Why does the OSD not write directly to disk, and why does it use ObjectStore::Transaction?
2. 2. What inputs does _calc_target in Objecter have, and when is an op resent?
3. 3. Why does each PG's op always go to one OSDShard?
4. 4. What is the role of ms_fast_dispatch, and what must not be done in it?
5. 5. How does ENCODE_START(v, compat, bl) make rolling upgrades possible?
6. 6. What is the difference between the direct and deferred paths in BlueStore, and why is the SSD default zero?
7. 7. When does the primary reply to the client in replication?
8. 8. In RGW, which function moves an index entry from pending to complete?
9. 9. In CephFS, why is the MDS not on the data path, and what is the effect of not having the Fb cap?
10. 10. Where do you look to find the real default of an option in a specific version?

Answers to these questions are in Appendix E: Answers to Review Questions. Answers to this chapter's questions →

22-16 Sources for this chapter#

• The Ceph repository
  https://github.com/ceph/ceph
• Developer guide
  https://docs.ceph.com/en/latest/dev/developer_guide/
• Running vstart
  https://docs.ceph.com/en/latest/dev/dev_cluster_deployment/
• OSD internals documentation
  https://docs.ceph.com/en/latest/dev/osd_internals/
• encoding
  https://docs.ceph.com/en/latest/dev/encoding/
• BlueStore
  https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/
• crimson
  https://docs.ceph.com/en/latest/dev/crimson/
• Weil et al., "Ceph: A Scalable, High-Performance Distributed File System", OSDI 2006
  https://www.usenix.org/legacy/event/osdi06/tech/full_papers/weil/weil.pdf