The Comprehensive Ceph Guide
فارسیGet started
Interactive lab

BlueStore write path

In the book: Chapter 12

Lab

BlueStore write path

Choose the device, offset and length of a write and watch BlueStore split it and decide what goes through the WAL (deferred) and what is written directly.

1pieces
4 KiBbytes through the WAL
–bytes written directly
64 KiBprefer_deferred_size
big 4 KiB0 B4 KiBmin_alloc_size = 4 KiB
The write over 4 KiB allocation units: green is written directly, amber goes through the WAL.
PieceRangePathHowRead firstAllocated
big0 B + 4 KiBbig overwrite in existing blobsdeferred (WAL)––
OSDBlueStoreallocatorblock deviceRocksDB / WAL1. queue_transactions2. split at min_alloc3. deferred op4. KV commit with data5. ack6. deferred write7. cleanup
  1. queue_transactions The OSD hands BlueStore a transaction; a TransContext is created in state PREPARE. BlueStore::queue_transactions, _txc_create
  2. split at min_alloc _do_write_data cuts the write into 1 piece(s) and sends each to _do_write_small or _do_write_big. BlueStore::_do_write_data, _do_write_small, _do_write_big
  3. deferred op The bytes to be written in place are put into a bluestore_deferred_op_t inside the same RocksDB transaction. bluestore_deferred_op_t, BlueStore::_txc_add_transaction
  4. KV commit with data _kv_sync_thread commits the metadata and the deferred data to RocksDB with a sync; the WAL write is the only synchronous disk write. BlueStore::_kv_sync_thread, _txc_apply_kv
  5. ack After the commit the write is durable; _txc_committed_kv completes the OSD's callbacks and the client can get its ack. BlueStore::_txc_committed_kv, _txc_state_proc
  6. deferred write Later, in batches (bluestore_deferred_batch_ops), the deferred bytes are written to their place on the block device. BlueStore::_deferred_queue, _deferred_submit_unlock
  7. cleanup When that write is on disk, _deferred_aio_finish removes the deferred record from RocksDB. BlueStore::_deferred_aio_finish
new data: deferred ⇔ data_size < prefer_deferred_size (64 KiB) · small overwrite: always deferred · big overwrite: deferred ⇔ len ≤ 2 × prefer_deferred_size

Follows:

  • src/os/bluestore/BlueStore.cc :: _do_write, _do_write_data splitting a write into head, middle and tail
  • BlueStore.cc :: _do_write_small, _do_write_big the small and big write paths
  • BlueStore.cc :: _do_alloc_write (prefer_deferred_size) allocation and the deferred-or-direct decision
  • BlueStore.cc :: _txc_state_proc, _kv_sync_thread, _deferred_queue the transaction's state machine, the KV commit and the deferred queue
  • src/os/bluestore/BlueStore_components.h :: TransContext::state_t the TransContext states
  • src/common/options/global.yaml.in :: bluestore_min_alloc_size_hdd/ssd (4K), bluestore_prefer_deferred_size_hdd (64K) / _ssd (0) the default sizes used here
A fixed snapshot of the simulator; the interactive version is in the web edition's lab. Interactive lab: BlueStore write path