A native Go communication layer for Ceph Tentacle: msgr2.1, CephX, raw MON/MGR commands and subscriptions, explicitly targeted OSD requests, and bounded MDS/CephFS wire primitives.
The product requires Go 1.25 or later and no CGO, go-ceph, librados, Ceph CLI,
or operating-system Ceph installation. Snappy and zstd codecs use the pure Go
github.com/klauspost/compress dependency. Native development tools belong to
the separate integration/fixture module.
The protocol reference is Ceph v20.2.4, commit
7f793731f1b39eb4f465e960113d2363c311b964. Tentacle (20.2) is the minimum
release family. A successful connection does not certify every patch,
configuration, platform or higher-level storage operation.
| Reader | Start here |
|---|---|
| Package user | Which side Go implements, Installation, Quick start, Representative operations and Limitations |
| Protocol implementer | Tracing one feature: public contract, pinned wire reference, implementation status, scenario, observed native result and remaining limits |
| Fixture or CI contributor | Fixture and CI entry points, the scenario catalog guide and the native path coverage map |
DOCUMENTATION.md names the authoritative document for each kind of information. This README summarizes usage and feature boundaries; when a summary here and an owning contract differ, the contract is correct.
The product is always the connecting CLIENT entity (client.<id>, CephX
entity type CLIENT). It opens outbound msgr2.1 connections to MON, MGR, OSD
and MDS daemons that already exist. It has no listening Messenger, accepts no
connection, and implements no daemon role: it cannot act as a MON, MGR, OSD or
MDS, and it sends none of the daemon-to-daemon messages (quorum, heartbeat,
peering, replication, recovery, metadata migration).
| Peer | What the Go client does | What it never does |
|---|---|---|
| MON | Authenticates, runs raw commands and Tell, receives maps and subscriptions | Join quorum or serve maps |
| MGR | Runs raw commands and Tell on the active MGR; optionally registers a service daemon as a CLIENT | Become an MGR or a module |
| OSD | Sends object requests and raw commands to a target the caller names | Peer, replicate, recover, store objects or choose placement |
| MDS | Runs raw commands and the opt-in CephFS session, metadata and capability messages | Hold a rank, journal or migrate metadata |
MGR service registration is the one place where the client presents a service name to a daemon. It is still a CLIENT session: the registration announces caller-supplied metadata and status to the active MGR and makes nothing in this repository a Ceph daemon.
The native test environment supplies every daemon. Tests start disposable
clusters from the pinned official Ceph image: MON, MGR, OSD and MDS daemons,
and in selected recipes an RGW gateway or an rbd-mirror daemon, together
with native clients (librados, librbd, libcephfs) that act as independent
oracles. Daemon-to-daemon paths are therefore exercised between real Ceph
daemons, never by Go; see the daemon-to-daemon rows of the
coverage map.
The fixture and its dependencies live in the separate integration/fixture
module and are not product dependencies.
| Communication layer | Application responsibility |
|---|---|
| Framing, CephX, ticket renewal, sessions and recovery | Credential selection, keyring/configuration loading and operational retry policy |
| Raw command JSON, bulk input, original output and server codes | Command construction, schemas, response interpretation and typed management APIs |
| Wire maps, logs, config, digest and raw OSDMap delivery | Applying configuration, interpreting maps and using advertised service URIs |
| Explicit OSD connections, operation vectors and control events | CRUSH, placement, target selection, object APIs and explicit resubmission |
| MDS sessions, metadata, capabilities and recovery messages | Filesystem routing, inode/path/cache state, layout, writeback and consistency |
This is not a complete RADOS, RBD or CephFS client. Test-only consumers exercise those underlying paths to verify communication. Requirements and completion criteria are in SPEC.md. APIs follow the current specification without compatibility wrappers for superseded APIs.
go get github.com/JSYoo5B/ceph-msgr-go/cephmsgrThe import path is github.com/JSYoo5B/ceph-msgr-go/cephmsgr; the package name
is cephmsgr. There are no published module version tags, so Go resolves a
pseudo-version of the default branch; record the commit you verified. The
module needs Go 1.25 or later and builds with CGO_ENABLED=0.
The first connection needs MON addresses, an exact CephX identity and its
key. Dial authenticates to a MON and returns a client that can already run
MON and MGR commands.
key, err := cephmsgr.ParseKey(encodedKey)
if err != nil {
return err
}
bootstrap, stop := context.WithTimeout(context.Background(), 10*time.Second)
client, err := cephmsgr.Dial(bootstrap, cephmsgr.Options{
Monitors: []string{"v2:192.0.2.10:3300/0", "v2:192.0.2.11:3300/0"},
Identity: "client.management",
Key: key,
ExpectedFSID: clusterFSID,
})
stop()
if err != nil {
return err
}
defer client.Close()
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
result, err := client.MonCommand(ctx, cephmsgr.Command{
JSON: []byte(`{"prefix":"status","format":"json"}`),
})
if err != nil {
return err
}
fmt.Printf("%s\n", result.Data)ParseKey accepts the selected base64 CephX key, not a whole keyring.
The caller supplies the exact identity. A successful client's lifetime is
independent of the bootstrap context passed to Dial.
Options.ConnectionMode defaults to SecureMode. Select CRCMode explicitly
for CRC framing after authentication. Both modes use CephX and verify the
authentication transcript. CRC does not encrypt subsequent traffic; CRC32C
detects corruption. Connections offer only the selected mode, without fallback.
Every operation below is a communication primitive with raw input and output.
The Opt-in column names the Options fields each row needs. A row marked
none works on any client returned by Dial; the OSD, MDS, MDS map and MGR
service rows are refused unless their option was set at Dial.
| Task | Entry point | Opt-in | Section |
|---|---|---|---|
| Run a MON or MGR command | MonCommand, MgrCommand, MonTell, MgrTell, MonTellTo, MgrTellTo |
none | Raw commands |
| Follow cluster state | Snapshot, WaitMonMap, WaitMgrMap, WatchLogs, WatchConfig, WatchDigest |
none | State snapshots, logs, configuration, digest |
| Receive raw OSDMaps from MON | RequestOSDMaps, WatchOSDMaps |
none | MON OSDMap requests |
| Send an object request to one OSD | OpenOSD, then OSDConnection.Request |
EnableOSD, optionally OSDProfile and EnableOSDCachePools |
OSD object requests |
| Send a raw OSD or PG command | OpenOSD, then OSDConnection.Command or OSDConnection.PGCommand |
EnableOSD |
OSD and PG commands |
| Receive MDS maps | WatchFSMapUser, WatchMDSMaps |
EnableMDSMaps |
MDS maps |
| Send a raw MDS command | OpenMDS, then MDSConnection.Command |
EnableMDS |
MDS commands |
| Open a CephFS session and send metadata requests | MDSConnection.OpenSession, MDSSession.Submit |
EnableMDS, EnableCephFS for metadata and capabilities |
CephFS sessions, metadata |
| Register a service daemon with the active MGR | OpenMgrService |
EnableMgrService |
MGR service registration |
The Go documentation of each name states its exact inputs, bounds and outcomes. The contracts linked from each section define the wire subset.
- The client never becomes a daemon and never accepts connections; see Which side of the connection Go implements.
- It is not a RADOS, RBD or CephFS client. Callers supply OSD targets, placement and map epochs; there is no CRUSH computation, automatic routing, redirect following or automatic resend of an object request.
- Commands are raw. The product builds no command JSON, parses no command output and loads no keyring or configuration file.
- Outcome rules differ by API; the package documentation in
api.go owns them.
- A daemon command or an OSD request that fails after its transmission
started returns an
OutcomeUnknownError: the daemon may have executed it, and the client never sends it again on its own. - A CephFS request is different.
MDSSession.Submitreturns once the request is queued. AWaitorWaitSafecontext that expires returns the bare context error while the call stays pending, so the caller can wait on the same call again instead of discarding it. On a resumable MDS connection, Messenger resume retransmits unacknowledged frames with their original sequence and transaction ID; that is delivery of the same request, not a new one. Only a session that ends before the reply resolves the call with anOutcomeUnknownError. Sending a request again (MDSSession.ResubmitForwarded) is always an explicit caller action. - Methods with other outcome rules, such as
MDSSession.SendCapsandOpenMgrService, document them. - Canceling a local wait never establishes remote rollback.
- A daemon command or an OSD request that fails after its transmission
started returns an
- Admission requires an authenticated MonMap with
min_mon_release >= 20(Tentacle, 20.2). A cluster that retains an older minimum, including one that is being upgraded, is rejected, and earlier release families have no compatibility path. - Only implemented features are advertised. OSD opcodes outside the supported set are refused; see unsupported opcodes. Withheld CephFS client feature bits are listed in the client extension decisions.
- Live compressed frames, secure-mode compression and zlib are unverified or deferred; see compression.
- Native verification covers selected profiles only. Each contract and validation record names the profile and source revision it observed; a result on one profile is not a claim about another. Completion criteria are in SPEC.md.
MonCommand and MgrCommand transport caller-built JSON and optional
Command.Input. The product checks structure and bounds without interpreting
command-specific schemas. MgrCommand authenticates to the active MGR.
Result.Data is binary-capable original output, Message is the status string,
and Code is the original server code. Negative codes return *CommandError
while preserving the result. Codes are not translated into host OS errno values.
Explicit server authentication refusal returns *AuthenticationError.
MonTell and MgrTell address the current MON or active MGR. MonTellTo
binds an exact bare MON name through a separate authenticated connection and
checks FSID, minimum release, name and original address. It preserves the main
connection, identity, subscriptions and tickets. Another MON is never substituted.
MgrTellTo binds the exact available active MGR name, GID, original address
vector and client identity. It shares the existing session and command slots.
Unknown, standby, unavailable and changed targets are rejected explicitly,
without waiting for promotion. See the contract.
A context ends the local wait, not remote execution or rollback. Failures after
transmission starts may return OutcomeUnknownError. The client never
re-executes an uncertain command automatically. Messenger ACK is not completion.
RequestOSDMaps(ctx, OSDMapRequest{}) requests the latest full map. Inclusive
full/incremental epoch ranges are also available. A range is disabled by two
zeros or satisfies 0 < First <= Last. No OSD ticket or EnableOSD is required.
Batches retain FSID, trim/latest epochs, ordered blobs and original front bytes.
Embedded maps remain opaque. Native limits can produce partial or empty batches;
NewestMap is not a completion cursor. Independent authenticated connections
provide request isolation because the protocol has no correlation ID. They
share MaxInFlight without replacing the main MON session.
WatchOSDMaps allows one continuous watch per client. Zero requests a current
full baseline; positive start epochs request inclusive history. Its FIFO holds
at most 64 batches and a conservative retained-byte budget. The default budget
is MaxFrameSize, with a range of 1 KiB to 1 GiB. Overflow terminates the watch
rather than silently dropping a map. Local registration is not a capability ACK.
Consumers own baseline application, continuity, trim handling, placement and additional requests. See range requests, MON watches and OSD delivery.
The opt-in MDS path receives filesystem, rank and daemon identities, epochs, addresses and raw wire metadata through MON. Reception is not filesystem selection, mount success or routing. See the map contract and native observations.
The MDS service path authenticates to an explicitly selected daemon and transports raw commands with bounded input/output, server errors and uncertain outcomes. It does not expand wildcards or silently select another rank. Identity and map-generation changes stay visible. See the contract and validation.
The opt-in profile provides OPEN, renewal, CLOSE, selected reclaim messages, Messenger resume and explicit new-incarnation reconnect state. It is not a mount. Consumers supply capabilities, realms and pending request/flush state. Same-TID resubmission is explicit; unsafe requests retain REPLAY/target-inode semantics, and unanswered requests retain retry identity. Recovery must not repeat completed OSD writes or infer success from a new session alone.
See session semantics, fresh connections, session validation and recovery.
Bounded codecs expose capabilities, leases, forwarding, unsafe/safe completion, snapshots and reconnect messages. Consumers own path/inode/cache state, writeback and explicit acknowledgements. A transport ACK, metadata reply and completed writeback have different meanings.
Test-only consumers compare selected metadata and striped object I/O paths with libcephfs, including failover, forwarding, capability handoff and cap-snapshot recovery. These profiles do not establish complete POSIX/CephFS, arbitrary dirty state or all-rank recovery. See the public contract, wire details, file data, explicit resubmission and flush deduplication.
Test-only consumers exercise class payloads, snapshots, sparse reads, watches, notifications, image metadata, writes, locks, object maps, clones and journal replay against librbd. The product transports bytes; image layout, striping, parent/copy-up, lock policy, groups, mirroring and migration belong to the consumer. See base primitives, writes, locks, clones, journaling, a journal append with a lost reply, the same append across a primary change, the loss of a lock owner and that loss with requests still unanswered.
Callers supply the address/ID, pool, locator, raw hash, map epoch and required PG/shard placement. The product does not compute CRUSH, apply maps, choose targets or move a request after a redirect. Requests preserve identity, incarnation, attempt, epochs, per-op results, object versions, flags and raw data.
Support depends on request/reply version, negotiated features and configuration:
| Area | Contracts |
|---|---|
| Core operations and versions | core, request v8, request v9, reply v8 |
| Completion and recovery | compound operations, resubmission, fullness |
| Metadata and conditions | metadata, OMAP, allocation hints |
| Object data | checksums, sparse reads, MAPEXT, CALL, COPY_FROM, WRITESAME |
| Snapshots and cache flags | snapshots, manifest, dirty state, cache pools |
| Enumeration and events | PG listing, v8 listing, watchers, watch/notify, backoff, events |
| Placement profiles | replica reads, erasure coding |
Native recipes compare bytes, versions, metadata and completion flags, including selected replicated-primary failures and EC configurations. They do not prove power-loss durability, every CRUSH rule, all EC profiles or every platform. Compound vectors have only the selected Ceph contract's ordering/completion semantics.
EC requests require explicit shard information. The narrow CRUSH_V2 admission bit does not claim CRUSH computation. MSR and unsupported required features are rejected. Replica reads and optimized EC shard numbering have separate contracts.
Raw commands bind an explicit target and preserve binary input/output, server codes and uncertainty. PG interpretation, discovery and typed administration belong to the consumer. See command validation.
Options.EnableMgrService permits OpenMgrService, an explicit service-daemon
registration (MMgrOpen/Update/Report/Close) on a dedicated connection to the
active MGR, with caller-supplied raw service/daemon names, metadata and status.
It changes no ordinary MON/MGR command behavior or feature mask. The handle
fails on connection loss or active-MGR change and never re-registers by itself;
Close reports an unconfirmed acknowledgement explicitly. The registration is
a CLIENT session and does not make the caller a Ceph daemon.
Evidence is owned by the contract: source-derived vectors and synthetic unit/race tests, retained fresh native codec vectors for the five messages, and native observation on selected profiles of registration, status, task status, applied metadata, close, ticket renewal and an active-MGR change. Each linked validation record names its own tested source and profile. Nonempty daemon extensions, older configure versions, further profiles, further failure configurations and further cold-bootstrap configurations are not verified.
Command catalogs remain raw management/daemon responses. The product does not construct typed methods or interpret schemas. Native test oracles may normalize known formatter differences without changing the original product output.
WatchLogs registers one local watch per client without consuming a command
slot. LogStream.Next(ctx) returns owned batches with cursor metadata and the
first terminal cause. Registration/SubscribeAck does not prove authorization
or lossless history. The queue holds up to 64 batches and a conservative byte
budget: default MaxFrameSize, range 1 KiB to 1 GiB, not an exact heap limit.
Overflow ends only the watch. A live Next receives accepted batches before the
terminal cause; canceled waits do not consume queued data. Close releases the
slot and waits for its worker without replacing an earlier recorded cause.
Configuration watches expose owned raw map[string]string snapshots without
parsing or applying settings. A newer unread map replaces the older one. Keys,
newlines, Unicode, NUL and whitespace remain raw. Local subscription success is
not an authorization result.
Digest watches preserve health-detail and MON-status bytes together, replacing an older unread pair. They do not interpret health policy or guarantee a full history. The other MON streams' lifetime, ownership and first-cause rules apply.
Snapshot() copies state without network I/O. MonMap members retain names/ranks;
MgrMap retains active/standby identities, module metadata and schemas, activation
policies and raw service URIs. Metadata is not readiness, permission or an
instruction to contact a service.
WaitMonReady/WaitMgrReady wait for existing preparation without commands or
command slots. WaitMonMap(ctx, afterEpoch)/WaitMgrMap wait for strictly newer
authenticated maps; zero accepts an existing nonzero map. They promise neither
intermediate epochs nor subsequent command success.
MaxFrameSizedefaults to 16 MiB;MaxInFlightdefaults to 64. Bulk input is copied after command admission. Fixed codec/transcript limits also apply.ErrLimitExceededcovers byte/collection bounds. A valid oversized response is not automatically malformed; structural errors may preserve both categories.ErrMalformedMessagedistinguishes complete invalid messages from interrupted network reads while preserving underlying decoder causes.- Per-operation cancellation must not set a deadline on a shared connection. Late replies are consumed safely; partial writes must not corrupt live framing.
- A watch context owns the watch; a
Nextcontext owns only that wait. Accepted data and a recorded first cause survive later cancellation or Close. Closestops owned sessions/workers and releases pending callers. It does not convert uncertain execution into confirmed failure or rollback.- Ticket renewal first re-authenticates to the MON that currently admits the client, keeping the GlobalID. Other seeds and learned MonMap addresses are fallbacks. Initial setup and recovery after a lost MON session try seeds first. See the CephX ledger.
- MON/MGR/OSD have no cookie-based resume here. The opt-in MDS profile has an explicit resume/reconnect contract.
Seeds accept host:port or v2:host:port/nonce, with bracketed IPv6. Local IPv6
routing zones are rejected rather than guessed as remote wire ScopeID.
Authenticated family, nonce, scope and flow remain intact. Options.Hostname
is caller-supplied and immutable; the product does not infer the execution host's name.
ExpectedFSID pins identity; otherwise the first authenticated MonMap does.
Admission requires min_mon_release >= 20, not a particular daemon patch.
Unsupported features are rejected explicitly.
CGO_ENABLED=0 go build ./...
CGO_ENABLED=0 go test ./...
CGO_ENABLED=0 go vet ./...
CGO_ENABLED=1 go test -race ./...
go -C integration/fixture test -mod=readonly ./...
go -C integration/fixture vet -mod=readonly ./...
python3 -m unittest discover -s tools -p 'test_*.py'
CGO_ENABLED=0 python3 tools/product_dependencies.pyCI and the Windows interoperability verifier share this dependency check. It
checks the root module's build graph, permits only the documented compression
module externally, and rejects native sources, module replacements and subprocess
dependencies. Fixture-only dependencies remain in integration/fixture/.
Basic CI also runs, with the race detector, exactly the cluster-free
integration-package tests, and tools/cluster_free_tests.py requires each of
them to report PASS. The tests are the ones go test -list reports for
./integration (tools/go_test_list.py), so Go's own rules decide what a test
is. A top-level TestCeph...Integration test is native and stays opt-in, and
every other listed test there is cluster-free, so adding one needs no list edit.
A cluster-free name that Go lists twice (declared in both the internal and the
external test package) is refused before the log is read, because one result
line cannot stand for two tests.
integration/fixture/ contains bounded native setup/control/oracle recipes.
Run Docker profiles on Linux with their documented selectors. Native vectors,
unit/race/fuzz tests, live interoperability and full CI are separate evidence.
Skipped/canceled jobs and other-source results do not prove the current commit.
Selected profiles do not establish full RADOS/RBD/CephFS or long-duration coverage.
Snappy/zstd codec and frame round trips are unit-tested. The live CRC test shows only that the pinned MON and OSD report no negotiated compression and commands still succeed. The test does not capture the peer compression bit or negotiation frames, so it cannot distinguish a declined offer from skipped negotiation. Live compressed frames remain unverified; secure-mode compression and zlib are deferred. See compression, fixture validation and completion criteria.
Wire semantics are independently implemented in Go from the pinned Ceph source and Tentacle Messenger documentation. Ceph native sources/libraries are development-oracle inputs, not product dependencies. See THIRD_PARTY_NOTICES.md, NOTICE, LICENSE and native-vector provenance records. Those records distinguish native encoding from live ProtocolV2 sessions and independent server execution.