Lower Layers (4): msgr2 at the Byte Level, AsyncConnection and EventCenter

26-1 Introduction: the wire between two daemons#

Every message exchanged in Ceph, whether a client's write request to an OSD, a heartbeat between OSDs or a map that the mon publishes, passes through the messenger layer. Since Nautilus the default protocol is msgr2 (port 3300 for the mon). This chapter opens it up at the byte level: the opening banner, the frame format with its preamble and segments, the two modes crc and secure (AES-GCM), the authentication handshake, the session and reconnection, and finally the ProtocolV2 state machine and the EventCenter event loop that runs all of this with a small number of threads. The source of the structures is src/msg/async/frames_v2.h, ProtocolV2.cc, crypto_onwire.cc and src/include/msgr.h on the main branch.

26-2 The banner and feature negotiation#

The first bytes sent after the TCP connection is established are not encrypted. ProtocolV2::_banner_exchange builds this structure: the eight-byte string CEPH_BANNER_V2_PREFIX = "ceph v2\n", a u16 for the payload length, and then the payload of two u64s, namely CEPH_MSGR2_SUPPORTED_FEATURES and CEPH_MSGR2_REQUIRED_FEATURES.

26-3 The frame at the byte level#

26-3-1 A complete example: one KEEPALIVE2 in crc mode#

KEEPALIVE2 has one segment that is just a utime_t (sec and nsec, each a u32). For the time 1700000000.5 seconds, the complete frame is 44 bytes. We computed these values with an independent encoder:

Because there is only one segment, no epilogue is sent (in msgr2.1). The server replies with KEEPALIVE2_ACK and the same timestamp. The client uses this reply to measure the connection's "last alive time" (mon_client_ping_interval and heartbeats).

26-3-2 A message: four segments#

MESSAGE always has four segments (SegmentIndex::Msg): HEADER (0), which is ceph_msg_header2 (seq, tid, type, priority, version, data_pre_padding_len, data_off, ack_seq, flags, compat_version); FRONT (1), which is the main encoded body of the message (for example the fields of MOSDOp); MIDDLE (2), which is rarely used; and DATA (3), which is the raw data of a write or read. The data segment's alignment is PAGE_SIZE_ALIGNMENT = 4096, so that the receiver can allocate a page-aligned buffer and take the data without copying all the way to O_DIRECT in BlueStore (chapter 23). sizeof(ceph_msg_header2) is less than 48, so in secure mode the whole header fits in the inline part and the receiver can learn the length of the rest with one decryption.

26-4 secure mode: AES-128-GCM on the wire#

When con_mode is CEPH_CON_MODE_SECURE (0x2), crypto_onwire.cc works with OpenSSL (EVP_aes_128_gcm): AESGCM_KEY_LEN=16, AESGCM_IV_LEN=12 and AESGCM_TAG_LEN=16. The nonce is a twelve-byte structure: a fixed 32-bit part and a 64-bit counter that is incremented for each frame. The key and the initial nonce are derived, separately for each direction, from the cephx session key and the connection_secret exchanged in AUTH_DONE.

pseudocode: sending one frame in secure mode (msgr2.1, this book's summary)
  plain_head = preamble(32) || segment0[0:48] (padded with zeros if shorter)
  out  = AES_GCM_encrypt(key_tx, nonce_tx, plain_head) || tag(16); nonce_tx.counter++
  if len(segment0) > 48 or num_segments > 1:
      rest = segment0[48:] || segment1 || segment2 || segment3   # each padded to 16
      out += AES_GCM_encrypt(key_tx, nonce_tx, rest || epilogue) || tag(16); nonce_tx.counter++
  send(out)

Overhead: for a small control message, secure mode uses about 96 bytes (32 + 48 + 16) versus 44 bytes in crc mode. For a 4M write the overhead is negligible, but the CPU cost of AES-GCM (with AES-NI, around a few gigabytes per second per core) is measurable on busy NVMe OSDs. This is why the default for data traffic is crc secure (crc preferred) and for the mon secure crc.

26-5 Session, reconnection and reset#

Lossless connections (for example OSD↔OSD, or client↔mon under some policies) must continue after a TCP drop without losing or repeating messages. For this msgr2 keeps the following state:

Reconnection scenario (this book's summary)
  client: SESSION_RECONNECT(addrs, client_cookie, server_cookie, global_seq, connect_seq, msg_seq)
  server:
    unknown cookie (the server restarted)     → SESSION_RESET(full=true)  → new session, queues empty
    global_seq older than what I have seen    → SESSION_RETRY_GLOBAL(gs)  → again with a larger gs
    inconsistent connect_seq                  → SESSION_RETRY(cs)
    race with a connection I am establishing  → WAIT
    everything correct                        → SESSION_RECONNECT_OK(msg_seq)
                                               resend the messages above msg_seq

26-6 AsyncMessenger, Worker and EventCenter#

EventCenter::process_events does the following in each round:

1. 1. It computes when the nearest time event is due, to set the wait duration (at least ms_time_events_min_wait_interval).
2. 2. If no poller is active and no external event is queued, it blocks in event_wait (that is epoll_wait in EventEpoll.cc) until that time.
3. 3. For each ready fd, it calls the registered read_cb or write_cb. For connections these are AsyncConnection::process and handle_write.
4. 4. It runs the expired timers (for example keepalive, connection timeout and reconnection backoff).
5. 5. It empties and runs the external_events queue. AsyncConnection::send_message, from another thread (for example a PG thread in the OSD), puts the message in out_q and schedules the write on the right worker with dispatch_event_external.

Why only three threads? Protocol processing (encryption or crc, frame parsing and throttling) is light, and the main work happens in the OSD's thread pools (osd_op_num_threads_per_shard). Three workers are usually enough for tens of thousands of connections, but with secure mode on busy NVMe OSDs these workers may become a CPU bottleneck. In that case top -H -p <pid> shows threads named msgr-worker-N near 100%.

top -H -p $(pidof -s ceph-osd)            # msgr-worker-0..2
ceph daemon osd.4 perf dump AsyncMessenger::Worker-0 | jq '.[] | {msgr_recv_messages, msgr_send_messages, msgr_running_total_time}'
ceph daemon osd.4 messenger dump        # without arguments, the list of messengers; with one's name, its connections
ss -tinp | grep ceph-osd | head            # cwnd, rtt and retrans for each connection
tcpdump -i any -s 128 -w /tmp/msgr.pcap port 3300   # the banner and preamble are visible in crc mode

26-7 Exact on-wire frame size#

FrameAssembler in frames_v2.h has a separate function for each part of the frame: get_preamble_onwire_len, get_segment_onwire_len and get_epilogue_onwire_len. With these three functions the exact overhead of each message can be computed before it is sent.

pseudocode
preamble = secure ? 32 + 48 (inline) + 16 (tag) : 32
seg(i):
    if secure:
        p = round_up(len_i, 16)
        if i == 0: return p > 48 ? p + 16 − 48 : 0       # up to 48 bytes fit inside the preamble
        return p
    return (i == 0 and len_0 > 0) ? len_0 + 4 : len_i        # separate crc for segment 0
epilogue = (only one segment) ? 0 : secure ? 16 + 16 : 1 + 3×4
total = preamble + Σ seg(i) + epilogue

Numerical example. Take a MESSAGE with segment 0 = ceph_msg_header2 (packed, 41 bytes), front = 200 bytes, middle = 0 and data = 4096 bytes. The logical data is 4337 bytes.

• crc mode: 32 + (41 + 4) + 200 + 0 + 4096 + 13 = 4386 bytes, that is 49 bytes of overhead (1.1%).
• secure mode: 96 + 0 (the 41-byte header is rounded up to 48 and fits entirely inside the preamble) + 208 + 0 + 4096 + 32 = 4432 bytes, that is 95 bytes of overhead (2.2%).

For KEEPALIVE2 in crc mode the figure is the same 44 bytes as in the table above, because it has only one segment and no epilogue. In secure mode the eight-byte segment is rounded up to 16 and fits inside the preamble, so the whole frame is 96 bytes.

26-8 Nonces in AES-GCM and preventing reuse#

pseudocode
create_handler_pair(secret, crossed):
    key = secret[0:16] ; rx = secret[16:28] ; tx = secret[28:40]
    Tx = AESGCM(key, crossed ? rx : tx) ; Rx = AESGCM(key, crossed ? tx : rx)
reset_tx_handler(segments):
    if nonce == initial_nonce:
        if used_initial_nonce: throw "nonce reuse"     # the counter has wrapped around
        used_initial_nonce = true
    EVP_EncryptInit_ex(..., iv = nonce)
    if msgr2.1: nonce.counter += 1  else: nonce.fixed += 1   # msgr2.0: wraps after 2^32

Numerical example. An OSD sending 10000 frames per second on one connection would, in msgr2.0, return to the initial nonce after $2^{32} / 10^4 \approx 429497$ seconds (about 5 days), and the connection had to be reset. In msgr2.1 the 64-bit counter lasts $2^{64} / 10^4 \approx 5.8 \times 10^7$ years. GCM also allows at most about 64 GiB of data per nonce ($2^{32}$ blocks of 16 bytes), and msgr frames, with their segment limit, are far smaller than that.

26-9 The EventCenter loop and waking up across threads#

pseudocode
loop (each Worker, msgr-worker-N): process_events(30 s)
process_events(timeout):
    if first time event < now + timeout: timeout = max(that gap, 1000 µs) ; trigger_time = true
    if pollers or external_num_events > 0: timeout = 0
    n = epoll_wait(timeout)
    for each ready fd: read_cb and/or write_cb
    if trigger_time: process_time_events()
    if external_num_events: lock ; swap(external_events) ; unlock ; do_request on all of them
dispatch_event_external(e) (from another thread):
    lock ; external_events.push(e) ; external_num_events++ ; unlock
    if the current thread is not that Worker: wakeup() ⇒ write(eventfd, 1)

Numerical example and latency. A thread of an OSD op shard prepares the reply to an op and calls send_message. The connection belongs to msgr-worker-1, so the send is queued as an external event and the eventfd is woken. The cost of this path is one write on the eventfd, one epoll_wait wake-up (usually a few microseconds to a few tens of microseconds, depending on load and the C-state settings) and then a writev on the socket. If the Worker is busy processing other fds at that moment, the event waits until the end of that round. This is why a Worker that has hundreds of busy connections at once raises tail latency. The 1000-microsecond floor of ms_time_events_min_wait_interval keeps very close timers from turning the loop into a busy loop.

26-10 Choosing a Worker for each connection#

pseudocode
get_worker():
    lock(pool_spin)
    best = argmin_w w.references       # tie ⇒ the first in the vector
    unlock ; best.references++ ; return best
release_worker(): references.fetch_sub(1)    # in AsyncConnection::_stop when the connection closes

Numerical example. With three Workers and ten connections in a row, the distribution is round-robin: 0, 1, 2, 0, 1, 2. If the fifth connection (on Worker 1) is closed, the counters become [2, 1, 2] and the seventh connection goes to Worker 1. At the end the counters are [3, 3, 3]. This balance is based only on the number of connections, not on their traffic. An OSD with 2 busy connections to two neighbouring OSDs and 200 idle connections to clients may end up with both busy connections on one Worker. Raising ms_async_op_threads (which needs a restart) makes this coincidence less likely. Each Worker's msgr_recv_bytes and msgr_send_bytes counters in ceph daemon osd.N perf dump AsyncMessenger::Worker-N show whether the real balance holds.

26-11 Reconnection: which messages are sent again?#

pseudocode
on fault (lossless connection): requeue_sent():
    out_seq −= |sent|
    for the sent messages from last to first: clear_payload ; push_front into out_queue[HIGHEST]
handle_reconnect_ok(msg_seq from the peer):
    out_seq = discard_requeued_up_to(out_seq, msg_seq)
discard_requeued_up_to(out_seq, seq):
    count = out_seq
    while the HIGHEST queue is not empty and front.seq ≠ 0 and front.seq ≤ seq: pop_front ; count++
    return count

Numerical example. A client has sent up to seq 120, but acks have arrived only up to 117. So sent = {118, 119, 120}. When TCP drops, requeue_sent sets out_seq back to 117 and moves the three messages to the head of the queue. The OSD had received 118 before the drop (in_seq = 118), so it returns msg_seq = 118 in SESSION_RECONNECT_OK. discard_requeued_up_to(117, 118) discards message 118 and out_seq becomes 118. Only 119 and 120 are sent again, and no message reaches dispatch twice. The payload of each message is cleared (clear_payload), because the features of the new connection may differ and a fresh encode may be needed.

26-12 Supplementary and up to date#

• msgr2.1 (the REVISION_1 bit) exists in Nautilus and later releases. It changed the epilogue format and the inlining of the first segment in secure mode, and it removed the epilogue from single-segment frames. msgr2.0 is kept only for compatibility with early Nautilus releases.
• On-wire compression (COMPRESSION_REQUEST and COMPRESSION_DONE, the ms_osd_compress_mode option) is negotiated after auth. Its default is none.
• The Linux kernel has supported msgr2.1 since 5.11, with the ms_mode option (legacy, crc, secure, prefer-crc, prefer-secure) (chapter 29).
• crimson-osd uses a Seastar-based messenger instead of AsyncMessenger, but it has the same msgr2 format on the wire.

26-13 Hands-on exercise#

1. 1. Capture the first 26 bytes of a connection to port 3300 with tcpdump and compare the banner and features with this chapter's table.
2. 2. Write a Python encoder for KEEPALIVE2 and check the preamble crc against this chapter's calculation (0x99F5D5D8).
3. 3. On a test cluster set ms_client_mode=secure and ms_cluster_mode=secure, and compare the CPU used by the msgr-workers with rados bench and top -H.
4. 4. During a rados bench, take one OSD's network card down for 5 seconds and find the SESSION_RECONNECT and RECONNECT_OK messages in the log (debug_ms=1).

26-14 Review questions#

1. 1. What is the structure of the msgr2 banner, and which features exist today?
2. 2. Why is the preamble exactly 32 bytes?
3. 3. With what seeds are the preamble crc and the segment crc computed?
4. 4. Why does the data segment have an alignment of 4096?
5. 5. What is the benefit of inlining 48 bytes in secure mode?
6. 6. What attack does AUTH_SIGNATURE prevent?
7. 7. When are SESSION_RESET and SESSION_RETRY_GLOBAL sent?
8. 8. What are the three throttlers on message receive, and how do they create backpressure?
9. 9. Why must handlers in EventCenter read until EAGAIN?
10. 10. How does another thread wake a worker to send a message?

Answers to these questions are in Appendix E: Answers to Review Questions. Answers to this chapter's questions →

26-15 Sources for this chapter#

• msgr2 design document
  https://docs.ceph.com/en/latest/dev/msgr2/
• frames_v2.h
  https://github.com/ceph/ceph/blob/main/src/msg/async/frames_v2.h
• ProtocolV2.cc
  https://github.com/ceph/ceph/blob/main/src/msg/async/ProtocolV2.cc
• crypto_onwire.cc
  https://github.com/ceph/ceph/blob/main/src/msg/async/crypto_onwire.cc
• Messenger configuration
  https://docs.ceph.com/en/latest/rados/configuration/msgr2/
• The ms_mode option in the kernel
  https://docs.kernel.org/filesystems/ceph.html
• NIST SP 800-38D (GCM
  https://csrc.nist.gov/pubs/sp/800/38/d/final
• epoll(7
  https://man7.org/linux/man-pages/man7/epoll.7.html