# Changes by Version

Release Notes.

## 0.11.0

### Features
- **Breaking Change**: Upgrade the API version to 0.11 for the schema revision fields and `SchemaBarrierService`
  operations added in this release. External clients and internal node communication must use API 0.11; mixed BanyanDB
  0.10 and 0.11 clusters are not supported.
- Add batch-write phase metrics for client-held-open time, post-seal admission time, and admission deadlines, plus per-part liaison-wqueue-to-data-node file-sync latency for measure, stream, and trace.
- Add bounded trace plugin overhead metrics: chain decision batches by result, per-plugin execution count and duration, batch trace-count distribution, and per-plugin fail-open bypasses by reason. The configured `plugin_name` distinguishes links in a multi-plugin chain; segment, shard, and schema labels remain absent. The catalog works with `pipeline_traces_evaluated` for per-plugin time/calls per trace, latency percentiles, timeout/circuit-open rates, and link-health monitoring. The Phase 10 merge benchmark mirrors chain batches and named-link timing into JSON and HTML and rejects missing, failed, bypassed, timed-out, or circuit-open execution records.
- Add trace **finalization sampling** (`PIPELINE_EVENT_FINALIZE`), a best-effort background backstop for the in-merge trace retention filter. A single node-wide, concurrency-1 scanner periodically sweeps cooled segments (`segEnd < now − finalize_grace`, reopening idle-closed segments via `SelectSegments(reopenClosed=true)`) and force-merges each shard's un-finalized parts through the group's registered sampler chain, reusing the hot merge path (`mergeParts` + per-sidx `Merge` + the shard's introducer loop) so it never touches the hot merge semaphore. Selection uses a per-part `finalizeGen` stamp (min-propagated on ordinary merges, written to disk before the part metadata so a crash cannot double-sample on replay); per-shard state (`finalize.json`) holds the generation, cooldown clock, round count, terminal flag, and an arrival-based unsampled-bytes counter incremented O(1) on the flush path. Re-rounds are bounded by an absolute FLOOR (8 MiB), a RATIO (0.10 of segment size), a cooldown (= `finalize_grace`), and a hard `max_finalize_rounds` (8). Coverage is best-effort — misses (e.g. a segment TTL-deleted before finalization) are accepted, not guaranteed. Enabled per group via `enabled_events` containing `PIPELINE_EVENT_FINALIZE`; the settling window is `finalize_grace` with a fixed 5m engine fallback. Finalize rounds emit the merge metrics under `type="finalize"`/`lane="finalize"`.
- Bound the trace merge **drop-set** so a large sampled merge cannot exhaust the heap. The merge-wide dropped-trace-ID collector was unbounded: because a shard's first finalize round selects every cooled part into one merge, an 18M-drop set reaches ~1.3 GiB live and ~2.6 GiB reserved heap (append doubling plus a single contiguous lookup index), inside a process also serving queries. The bound is applied to the sampling **decision**, not to the pruning predicate: once a merge's drop set is full every further proposed drop is retained instead of recorded, so the set stays complete with respect to the drops actually performed and each sidx still contains exactly its part's retained traces — no orphan entries, none missing. The fragment guard is skipped for a trace that will be retained, since it exists only to confirm drops. Every merge is charged the same ceiling, finalize included, resolved from the memory protector as `limit/(16*CPUs)` so the aggregate across concurrent merges stays near `limit/16`; bytes-per-entry is derived from the observed trace-ID length rather than a constant, so a deployment with long service-prefixed IDs cannot overshoot the budget. A node whose protector is disabled (no `--allowed-bytes` and no valid cgroup limit) gets a flat fallback ceiling regardless of RAM — there the lever is the protector limit, not the machine's memory. The accepted cost is bounded deletion: a capped finalize round leaves traces undeleted and they are only revisited by the existing re-round path, which is capped at `max_finalize_rounds` (8) per shard for its lifetime, so a very large shard can retain traces the sampler wanted dropped. Finalize I/O is unchanged, because the round count is unchanged. New group-level metrics make the loss visible without segment or shard cardinality: `pipeline_drop_set_budget_bytes`, a lane-split `pipeline_drop_set_entries` histogram emitted whether or not a merge caps, `pipeline_traces_retained_by_ceiling`, `pipeline_merges_ceiling_reached{lane}`, and scan-level `finalize_rounds`/`finalize_terminal` worst-case gauges. Design: `docs/design/trace-drop-set-bounding.md`.
- Redesign queue (`queue_pub`/`queue_sub`) metrics around a uniform model with per-message and per-batch-stream catalogs. Use `operation` and `group` labels, add `error_type` and remote-endpoint labels, and propagate sender identity and group on the wire. [Breaking Change] Previous queue metric families, chunk-ordering metrics, and the `topic` label are removed; update dashboards and alerts.
- Stamp the lifecycle's tier-migration publisher's identity onto the wire so the receiving data node records a non-empty `remote_node`/`remote_role`/`remote_tier` on its `banyandb_queue_sub_total_finished` series. The lifecycle's `parseGroup` resolves the lifecycle's self identity by matching the lifecycle pod's hostname (POD_NAME via the K8s downward API, falling back to `os.Hostname()` — same precedence as `nativeNodeContext` at `banyand/backup/lifecycle/service.go`) against the data-node registry's `GrpcAddress` (host-portion match with loopback-alias and IP-literal normalization, via `hostMatches` at `banyand/backup/lifecycle/steps.go`) — `Metadata.Name` becomes `remote_node`, `Labels["type"]` becomes `remote_tier` — and calls `SetSelfNode(senderNode, "lifecycle", senderTier)` on the migration publisher. The previous `--grpc-addr` address-match (Pass 1) and `--node-labels` superset-match (Pass 2-3) fallbacks are replaced by this single host-based match because they failed on the production cluster where the data pod's `GrpcAddress` is a headless-service FQDN but the lifecycle's `--grpc-addr` is the loopback. Mirrors the liaison's existing `SetSelfNode(node.NodeID, "liaison", liaisonTier)` call in `pkg/cmdsetup/liaison.go:170-171`; no new CLI flags are introduced.
- Add labeled lifecycle health metrics (`banyandb_lifecycle_cycles_total`, `banyandb_lifecycle_last_run_timestamp_seconds`, `banyandb_lifecycle_last_run_success`) keyed by `remote_node`, `remote_role`, `remote_tier`, and `group`; stamp the last-run gauges atomically at cycle end and remove their previous label tuple. [Breaking Change] Update dashboards and alerts for the labeled series.
- Remove `banyandb_lifecycle_self_identity_resolution_total`. The regression-detection role moves to the now-labeled `banyandb_lifecycle_cycles_total{remote_node!=""}` (an empty `remote_node` series means the registry match failed for every group, the bug the old counter caught), plus the existing receiver-side count of empty `remote_node` on lifecycle `banyandb_queue_sub_total_finished` series. The wire-level `cluster.v1.SendRequest` sender-identity fields are unchanged.
- Vectorized measure query path is now enabled by default. The columnar pipeline replaces per-row protobuf serialization in `NewMIterator`, cutting allocations and ns/op for scan-heavy measure queries; gRPC wire format (`*measurev1.InternalDataPoint`) is byte-identical. Single-node coverage is complete: scan, GroupBy+Agg via `BatchAggregation`, scalar reduce (`Agg` without `GroupBy`), raw `GroupBy` (without `Agg`), implicit projection coverage for GroupBy/Agg fields, `TopN`/`BottomN`, `order_by` (via `logical.ParseOrderBy`, mirroring the row path's `PushDownOrder` rule), queries with hidden criteria tags, and boundary-error parity (nil time range, unknown projection, empty result) all resolve through the vec dispatch with row-path-equivalent semantics (SUM/COUNT/MIN/MAX/MEAN type semantics, first-seen carry-forward of non-key projected tags, canonical validation errors). Validated by a 6h production soak (byte-identical parity, zero divergences) and the per-workload bench gates. Distributed Map-mode partial aggregation (multi-node GroupBy+Agg / TopN), multi-measure (multi-group) requests, and non-vectorized backends continue to flow through the row path pending the distributed vectorized query work. Rollback: pass `--measure-vectorized-enabled=false` on the standalone or data-node command line and restart; the row path resumes immediately.
- Enable vectorized stream and trace queries by default. Stream uses the columnar pipeline and native liaison-to-data frame for supported timestamp-, index-, criteria-, and multi-group queries; unsupported shapes and traced queries fall back to row or protobuf execution. Roll back with `--stream-vectorized-enabled=false` or `--trace-vectorized-enabled=false`.
- Add `banyandb_vec_frame_encoded{engine}` and `banyandb_vec_frame_decoded{engine}` gauges (engine: `measure`/`stream`/`trace`), reporting how many query responses this process has put on the wire as a native columnar frame and how many it has decoded from one. Which wire format a node is actually speaking was otherwise invisible from outside the process, and the 0.11 upgrade-ordering rule depends on exactly that: a data node emits frames only when it is distributed AND its engine flag is on, so a standalone server reports zero encodes however the flag is set. Both series are cumulative and, being labelled, appear on `/metrics` only after the first collection tick. A run whose encode count stays flat on the data nodes is falling back to protobuf.
- [Breaking Change] Upgrade liaison nodes before data nodes when rolling out vectorized query paths. A 0.11 liaison accepts both columnar frames and protobuf, but an older liaison cannot decode frames emitted by an upgraded data node. Standalone deployments are unaffected.
- Add validation to ensure Measure's ShardingKey contains all Entity tags to guarantee entity locality.
- Organize access logs under a dedicated "accesslog" subdirectory to improve log organization and separation from other application data.
- Collect BanyanDB data on e2e test failure for CI debugging.
- Add log query e2e test.
- Sync lifecycle e2e test from SkyWalking stages test.
- Add `noDuplicates` verification to all e2e expected files to detect duplicate data in query results.
- Add a program-generated trace query integration-test framework under `test/cases/trace/cmd/{generate,capture}`: layered case generation (criteria leaves, AND/OR trees, and feature-pairwise across the traceID-lookup and order-based query modes), env-gated golden capture, and a shared `SeedAll` seeder — mirroring the measure test-case framework.
- Persist segment end time in per-segment metadata so boundaries don't shift across restarts or config changes.
- Introduce fair fast/slow lane scheduling for trace part merges to prevent short merges from being blocked by long-running merges; expose queue wait time as `total_merge_queue_latency`.
- [Breaking Change] Remove etcd components. The property-based schema registry is now the only supported mode.
  - All `--etcd-*` CLI flags have been removed.
  - The `--namespace` CLI flag has been removed (it previously configured the etcd key prefix).
  - The `--node-discovery-mode` flag no longer accepts `etcd` (supported values: `none`, `dns`, `file`).
  - The `--schema-registry-mode` flag only accepts `property`.
- Implement panic diagnostics and the FODC crash reporting pipeline.
- Schema consistency (Phase 1): introduce client-observable revision and propagation primitives. All gates are opt-in and zero-valued requests preserve prior behavior.
  - Add `mod_revision` to Group / IndexRule / IndexRuleBinding / TopNAggregation Create and Update responses.
  - Add `delete_time` to all `*ServiceDeleteResponse` messages so clients can observe tombstones.
  - Add `created_at` to Stream / Measure / Trace / Property / IndexRule / IndexRuleBinding / TopNAggregation / Group; preserved across updates.
  - Add `STATUS_SCHEMA_NOT_APPLIED` (10) for writes/queries whose `mod_revision` is ahead of the server cache.
  - Add three-way write-path `ModRevision` gate on Stream and Measure write RPCs (`<` expired, `==` succeed, `>` not-applied, `0` skipped).
  - Add per-group query-path gate via `QueryRequest.group_mod_revisions` and `QueryResponse.group_statuses` for Stream / Measure / Trace queries.
  - Add automatic time-range clamp `max(time_range.start, schema.created_at)`; multi-group uses the maximum across queried groups; nil for pre-upgrade schemas is a no-op.
  - Add `SchemaBarrierService` with `AwaitRevisionApplied`, `AwaitSchemaApplied`, `AwaitSchemaDeleted` (10 000-key cap, timeout-bounded, returns laggards on expiry).
  - Add tombstone retention/GC (default 7 days, configurable via `--schema-server-tombstone-retention`) with a per-cache count cap to bound memory under bulk deletes.
  - Reject `Create` with `updated_at <= tombstone.delete_time` to prevent replayed creates from overwriting newer deletes.
  - Guard `pkg/schema/cache` against out-of-order `EventDelete` events; expose monotonic `LatestModRevision` watermark.
- Schema consistency (Phase 2): add cluster-wide barriers that fan out `AwaitRevisionApplied`, `AwaitSchemaApplied`, and `AwaitSchemaDeleted` across liaison and data nodes through `NodeSchemaStatusService`, while safely handling mixed-version, membership, and timeout cases.
- Align schema barriers and write/query revision gates with each node's executor cache, including correct watermark advancement and no-op update revisions.
- Add schema-barrier metrics, structured access logs, integration coverage, and a load-test harness.
- Introduce a migration tool with `copy`, `verify`, and `analyze` subcommands for measure and stream data.
- Support displaying a measure's indexed tags in the dump tool, resolved per part so peak memory is bounded by the part rather than a segment-wide series map.
- Make segment lifecycle safe for snapshots, backups, and inspection: segments retain a dormant state while idle, and these operations no longer reopen closed segments or race idle cleanup.
- Add opt-in vectorized measure query tracing over raw-frame distributed queries, including a trace envelope and fixed trace-label vocabulary.
- Speed up GCS backup uploads: write each object and its checksum metadata in one request, dropping the per-object `Update` round-trip.
- Lifecycle migration now archives rows whose measure/stream schema was deleted from the registry, instead of aborting the group.
- Canopy M3 metadata CRUD: add IndexRule + IndexRuleBinding create/edit/delete forms with admin-only edit/delete, pure `validate*()` module covering 8 input groups (group, stream, measure, trace, index rule, index rule binding, interval, lifecycle stages), `useFocusTrap` + `useDirtyGuard` modal hooks, live `m3-indexrule` e2e (CRUD round-trip + MF2 server-authority negative), and opt-in `m3-handoff` e2e + zero-dep static server for the design-bundle comparison.
- Prevent lifecycle migration receiver OOMs with memory-efficient external series-index deduplication, receiver load shedding and bounded memory waits, and sender retries for transient failures; incomplete parts resume in the next cycle.
- Add a storage-node in-merge trace-retention filter (`PIPELINE_EVENT_MERGE`) that evaluates per-group sampler chains and safely drops non-retained traces from core and secondary-index parts. Configure samplers dynamically through each group's pipeline, with bounded staging, fail-open error handling, and runtime register, update, and removal support.
- Add FODC memory-pressure pprof capture: agent grabs heap/goroutine profiles when RSS nears the cgroup limit, served via the proxy.
- Add fuzzy resource search to the From row: type a name and jump to any measure / stream / trace / Top-N across every group, with contiguous-run + word-boundary scoring, highlighted matches, arrow/Enter/Escape navigation, and click-outside dismiss.
- Add bounded plugin telemetry (meter + logger) to the trace-pipeline sampler SDK. A sampler plugin opts in by implementing `sdk.HostAware`; the engine calls `UseHost(h)` once per group before the first `Decide`, delivering a scoped `sdk.Host` the plugin may cache and use in `Decide`. The host enforces all resource limits: metric names are forced under the `banyandb_trace_pipeline_plugin_` prefix; `{group, plugin_name}` labels are always present; per-plugin cardinality is capped at 100 series (overflow counted under `_overflow`); log throughput is capped at 50 lines/s burst 100, messages at 1 KiB, and 16 structured fields per call. Per-group delivery means a `HostAware` plugin receives a distinct `Host` per group so attribution is correct by construction. Series are reclaimed on pipeline teardown via `Delete`. No ABI bump — delivery is via type assertion (`sdk.ABIVersion` remains 1). Reference plugins `_telemetrysampler` (well-behaved) and `_faultysampler` (adversarial bounds verification) are in `test/plugins/`; build with `make build-trace-pipeline-telemetry-plugins`.
- Add the first-party trace-retention sampler plugins `sw-trace-sampler.so` and `zipkin-trace-sampler.so` (`plugins/skywalking/`), implementing the Scenario 6.1/6.2 keep logic on SkyWalking's two trace schemas. Both compile the shared `plugins/skywalking/internal/tracesampler` engine and differ only in a `Schema` value naming where each schema keeps the rule inputs: the flattened searchable-tag array (`tags` for segments, `query` for Zipkin), the error signal (a real `is_error` column vs. Zipkin's conventional `error` key inside `query`), and the duration columns (`start_time`/`latency` in ms vs. `timestamp_millis`/`duration` in µs, normalized via `Schema.DurationTagNanosPerUnit`). Config keys are `durationThresholdMs`, `keepErrors` (with an `errorTag` override), `keepTagRules`, and `healthySampleRate`: a trace is kept when any sure-keep rule matches, otherwise a deterministic FNV-1a hash of the trace ID admits `healthySampleRate` of the remainder — stable across the merge and finalize passes, so a trace is never half-kept. `durationThresholdMs` tests the end-to-end envelope `max(start + duration) − min(start)` over the trace's rows, catching traces slow only through sequential spans, rather than any single span's duration or the intrinsic `MaxTS − MinTS` (which is the spread of per-row start timestamps, and 0 for a single-row trace). Because OAP flattens every searchable tag into one string-array column, every `keepTagRules` entry resolves there; a rule naming a first-class column could never match and is rejected at admission, with the array-column workaround named in the error. `keepTagRules` also accepts a compact `key=value,key=~regex,key` string so a rule set fits in an environment variable. Unknown and empty configs are rejected rather than ignored — every option is a keep rule, so a key that silently missed would drop the whole group — and any predicate that cannot be evaluated keeps the trace. Built by `make build-plugins` and shipped in the `-plugins-carrier` image; see `plugins/README.md` for the config reference.
- Remove the `latencystatussampler` first-party plugin. It projected `duration` and `status`: `status` exists in neither shipped trace schema and `duration` only in Zipkin (in µs, so a `thresholdMs` there was off by 1000x), so it could never match live data and — failing open — kept every trace it was given. It shipped in the `-plugins-carrier` image beside the two real samplers, leaving three first-party plugins of which one silently retained everything. The near-identical copy under `test/plugins/_latencystatussampler` is unaffected and still backs the trace-pipeline integration suites, the plugin-sidecar test and `sdktest`'s `LoadSO` coverage; the worked-example references in `docs/operation/plugins*.md` now point at `sw-trace-sampler`.
- Add a Claude/Codex plugin packaging the BanyanDB MCP server with a parse-only `validate_bydbql` tool (backed by the `mcp/tools/bydbql-parse` Go validator) and a BydbQL skill for read-only natural-language-to-BydbQL generation over STREAM/MEASURE/TRACE/PROPERTY resources.
- Introduce positional parameter binding (`?` placeholders) into BydbQL to eliminate QL injection.
- Add reusable BydbQL binding and prepared-statement caching on the gRPC query path, with bounded cache and top-K retention, cache and slow-query observability, and redacted bound parameters in the latest slow-query log.
- Add an FODC schema consistency check comparing registry, cache, and runtime fingerprints across all nodes via lifecycle inspection.

### Bug Fixes

- Prevent long-lived client write streams from failing whole liaison batches when the write timeout expires; start the liaison admission timeout
  only after the batch is sealed and raise its default to two minutes. (apache/skywalking#13982)
- Stabilize measure, stream, and trace snapshot tests by waiting for persisted manifests and all flushed memory parts rather than directory creation or the latest snapshot creator, eliminating asynchronous-flush and fallback races.
- Prevent merge-time trace sampling from dropping fragments when the same trace ID may remain in unselected parts. Provisional drops are now checked against time bounds and
  trace-ID Bloom filters in candidate outside parts whose time bounds intersect the trace's max-fragment-gap-expanded range, revalidated before publication, and retained on
  uncertainty; the default merge grace is two hours.
- FODC: namespace dynamic node labels as `node_*`, continuously resolve node roles and labels, and prevent oversized metric uploads and ghost `ROLE_UNSPECIFIED` series.
- Close BanyanDB merge write-path durability gap that allowed torn parts to be created by a crash between data write and metadata commit. Metadata files (`metadata.json` for trace/measure/stream, `manifest.json` for sidx, plus `traceID.filter` and `tag.type`) now go through a new `WriteAtomic` (write-tmp + fsync + rename + fsync-dir) sequence; data writers (`seqWriter.Close`, `localFileSystem.Write`) now propagate fdatasync errors instead of silently dropping them. `mustOpenFilePart` / `mustOpenPart` in each engine cleans up safe post-rename `.tmp` leftovers on open. (#13862, root cause for #13861)
- Fix bydbctl command tests using global stdout capture, which caused race-enabled runs to corrupt captured command output.
- Fix flaky trace query filtering caused by non-deterministic sidx tag ordering and add consistency checks for integration query cases.
- Fix index-mode measure queries returning documents outside the requested time range when a widened segment overlaps the query window.
- MCP: Add validation for properties and harden the mcp server.
- Make property-schema client connections recover after data-node restarts and add periodic health checks.
- Fix flaky on-disk integration tests caused by Ginkgo v2 random container shuffling closing gRPC connections prematurely.
- Fix snapshot error when there is no data in a segment.
- ui: fix query editor refresh/reset behavior and BydbQL keyword highlighting.
- Disable the rotation task on warm and cold nodes to prevent incorrect segment boundaries during lifecycle migration.
- Prevent epoch-dated segment directories (seg-19700101) from being created by zero timestamps in distributed sync paths.
- Fix SIDX streaming sync sending SegmentID as MinTimestamp instead of the actual timestamp, causing sync failures on the receiving node.
- Fix handoff controller TOCTOU race allowing disk size limit bypass, and populate sidx MinTimestamp/MaxTimestamp during replay to prevent corrupt segment creation on recovered nodes.
- Delete orphaned parts when no snapshot references them during tsTable initialization.
- Extract shared LocateAll on NodeRegistry to ensure resolveAssignments and syncer GetNodes always produce identical node lists, preventing liaison from enqueuing parts to online/healthy data nodes.
- Add validation for MATCH and IN conditions in inverted index query builder, and handle nil OR branch when all entities are specific.
- Fix wrong backup path of schema property.
- Fix lifecycle migration failure when the target stage has `close: true`.
- Fix stale sync request blocking watch session channel, causing repeated "channel full, skipping session" errors when a watch stream is in backoff.
- Fix nil pointer panic in disk monitor when group schema is not yet initialized during early startup, and ensure monitor loop survives recovered panics.
- Fix `FileSystemError` not satisfying `errors.Is(err, io/fs.ErrNotExist)`, which prevented the segment controller from cleaning up half-born segment directories and left groups in a permanent zombie state after a crash or partial sync.
- Fix lifecycle migration panic when a stream shard's snapshot has no element index (`idx/`) directory.
- Avoid FODC lifecycle inspection failing on busy data nodes by raising the per-broadcast `CollectDataInfo` / `CollectLiaisonInfo` deadline from 5s to 30s and parallelizing per-group inspection in the cluster-internal `InspectAll`.
- Fix deadlock when fodc-agent reconnects to fodc-proxy after a pod rotation.
- Fix flaky `TestCollectWithPartialClosedSegments` by raising `SegmentIdleTimeout` so wall-clock variance on slow CI does not mark still-open segments as idle.
- Fix FODC lifecycle cache poisoning where transient `InspectAll` failures were cached for 10 minutes and masked liaison recovery; raise FODC agent and proxy timeouts from 10s to 40s.
- Fix FODC `/cluster/lifecycle` dropping zero-valued group fields (e.g. `replicas=0`, `close=false`) under `encoding/json` + `omitempty`; switch to `protojson` so all fields are emitted (nil nested messages serialize as `null`).
- Fix trace `block_writer` panic on out-of-order timestamps within the same traceID, which dropped one trace-write batch per panic in multi-agent SkyWalking deployments. Spans of a single trace originate from independently-clocked services, and trace storage is organized by traceID rather than timestamp, so per-traceID timestamp monotonicity is not a writer invariant.
- Add `GroupLifecycleInfo.errors` to surface per-group collection failures from FODC `InspectAll` instead of silently dropping the affected node entry.
- Fix `CollectDataInfo` and `CollectLiaisonInfo` not handling `CATALOG_PROPERTY` groups.
- Fix lifecycle migration where the receiving node could create segments shorter than the configured `SegmentInterval`.
- Fail fast on incompatible storage version at boot. Previously the server would start in a degraded `SERVING` state with affected groups un-loaded because the property schema-registry retry loop swallowed the version-incompatibility panic. Compatible versions are listed in `banyand/internal/storage/versions.yml`.
- Release bluge index writers on segment rotation so `analysisWorker` pools sized from `GOMAXPROCS` don't accumulate across rotations. Two layered defects kept the existing idle-segment reclaim path from running: `segmentIdleTimeout` defaulted to `0` (which disabled the 10-minute reclaim ticker), and `incRef` refreshed `lastAccessed` on every rotation tick so `closeIdleSegments` never observed an idle segment. Defaults to `time.Hour`, moves the `lastAccessed` bump to real read/write call sites, and rewrites `closeIdleSegments` to take its own CAS-bumped snapshot so a concurrent reopen cannot have its only ref dropped under the reclaimer (apache/skywalking#13874).
- Fix incorrect counts and missing trace fields in the lifecycle migration report.
- Fix lifecycle migration placing data in the wrong target segment when the source segment interval is not a multiple of the target stage's interval, by row-level replaying parts that straddle a target-segment boundary instead of chunk-copying them into a single segment.
- Fix trace query identity-tag projection: when `trace_id`/`span_id` are explicitly projected, reconstruct them from span identity at response build time instead of requesting them as stored tags, and preserve tag order with null-filled per-span value alignment in the distributed trace result iterator.
- Fix measure, stream, and trace queries returning data from segments already expired by the TTL. Retention removes a segment only on its next scheduled run, so a fully expired segment can linger on disk and keep serving TTL-expired data; queries now skip segments whose whole time range is past the retention deadline, matching retention's own removal condition.
- Trace storage metrics now expose the `storage` sub-scope, matching the `stream_storage_*` naming. The `StorageMetricsFactory` for trace switched from the root `trace` scope to `trace.storage`, so per-segment inverted-index metrics (`inverted_index_total_updates`, `inverted_index_total_doc_count`, `inverted_index_total_term_searchers_started`) are now emitted as `banyandb_trace_storage_*` instead of `banyandb_trace_*`, aligning the dashboard query names. Other trace metrics (`trace_tst_*`, `trace_scheduler_*`) are unchanged.
- Fix FODC proxy corrupting Prometheus metric types. The agent dropped the `# TYPE` line while parsing banyandb `/metrics`, the `StreamMetrics` proto carried no type field, and the proxy guessed the type from a name-suffix heuristic — downgrading counters to gauge, mislabeling `_count`-suffixed counters as histograms, and splitting summaries into two conflicting `# TYPE` lines. Capture the type with the Prometheus `expfmt` parser, store it in the flight recorder, thread it through a new `Metric.type` enum over gRPC, and emit the real type from the proxy; pre-upgrade (untyped) samples fold into the matching typed family so a mixed-version rollout never emits two conflicting `# TYPE` lines for one metric.
- Fix lifecycle row-replay OOM on large measure parts by streaming the dump reader, pooling size-classed marshal buffers, and bounding in-flight batch bytes (default 32 MiB); peak heap drops ~1.5 GB→~296 MB.
- Consolidate lifecycle migration report errors into a single flat list of structured, stage-aware entries.
- Fix backup container OOM from overlapping scheduled runs; serialize runs and upload small snapshot files concurrently.
- Fix block metadata reset before unmarshal for stream and trace.
- Lifecycle only handles stream/measure/trace snapshots, skipping other catalogs.
- Align code-mode editor with the handoff toolbar + footer (move "← Builder" back button into the toolbar, switch the footer to `qb-foot`/`qb-foot-row`, restore the ⌘/Ctrl+↵ kbd hint on Run, switch markup to the handoff class names, and drop the syntax-highlight `<pre>` that double-rendered side-by-side with the textarea).
- Fix liaison goroutine leak from uncanceled gRPC SyncPart stream context in write-queue part sync.
- Fix group update overwriting a warm/cold node's stage-resolved segment interval, TTL and shard count with the group default.
- Re-index bound resources when an index rule or binding is deleted, clearing stale index config.
- Deleting one TopN aggregation no longer tears down sibling aggregations on the same source measure.
- Purge a deleted group's resource, index-rule and binding cache entries to avoid dangling references.
- Clear a trace subject's index when its last index rule or binding is removed.
- Allow `*` as a non-initial character in BydbQL identifiers, so resources named with `*` become queryable, not just writable.
- Enable periodic health checks on the queue client (`--<prefix>-client-health-check-interval`, default `10s`), evicting dead data nodes proactively.
- Build the plugin `.so` with `-tags slim` to match the host binary's ABI. Without this, `pkg/pool` has build-tag-dependent code (`tracker.go` vs `tracker_stub.go`) and the plugin-sidecar E2E failed with "plugin was built with a different version of package `pkg/pool`".
- Make the trace merge loop survive a persistently failing merge instead of destroying the node. A production cluster wedged when a few unreadable parts made every merge selection fail: each failed attempt leaked its partially-written output directory (the block writer creates it before the merge runs), which exhausted the volume's inodes at 8% byte usage; the resulting disk-full panic then killed the merge lane workers, which are recovered but never respawned, so the dispatcher blocked on the lane channel forever and the parts stayed pinned in flight. Four layered fixes: (1) `mergeParts` removes the output directory on any error return or panic, in trace, stream, measure, and sidx; (2) merge-read failures are wrapped in a typed error carrying the failing part's identity, and a part accumulating three consecutive attributable failures is excluded from both the hot merge selector and the finalize-round selector — it stays in the snapshot (queryable, TTL-deleted as usual) and the registry is swept when parts leave the snapshot; (3) merge dispatch backs off exponentially from 1s to a 60s cap after consecutive failures, reset on any success, with wave mode (operator-driven controlled merges) bypassing it; (4) merge-execution panics are converted to ordinary merge errors at the lane worker and the dispatch cycle, so the semaphore release, in-flight unpin, and failure accounting always run and the goroutine survives. New metrics: `total_merge_part_quarantined`, `merge_quarantined_parts`, `total_merge_backoff_seconds`, and `total_merge_panic_recovered`.
- Report panics recovered by the gRPC servers through `panicdiag`, so they reach `banyandb_panic_total{component}`, the crash reporters and the artifact writer. Each server (`queue/sub`, `liaison/grpc`, `property/gossip`, `metadata/schema/schemaserver`) hand-rolled a recovery handler that only logged and returned `codes.Internal`, bypassing every panicdiag signal: during a production incident one data node logged 737 recovered request-path panics while `banyandb_panic_total` stayed at 4 (the only goroutine panics, which route through `run.Go`). The panics most worth alerting on were the ones nothing counted. The four copies are replaced by `panicdiag.GRPCRecoveryHandler(logger, component)`, which reports via the new exported `panicdiag.RecoverExternal` -- the entry point for a panic the caller recovered itself -- and is labelled per server (`grpc.queue-sub`, `grpc.liaison`, `grpc.property-gossip`, `grpc.schema-server`). The stack is captured inside the interceptor's recovering defer, so artifacts still show the panic site rather than the recovery machinery.
- Fix the measure schema-change merge test, which was time-of-day dependent rather than flaky. It wrote its two batches at `now-2h` and `now-1h` and waited for the part count to drop, but `sw_metric` uses a 1-day segment aligned to local midnight and a merge only ever combines parts within one segment, so any run starting within two hours of midnight split the batches across yesterday's and today's segments and the count could never drop. The budget had been raised twice (10x, then 20x) blaming merger starvation, which no budget could have cured; both offsets are now compressed into the elapsed part of the current segment.
- Fix the flaky distributed schema clamp test: it queried once immediately after `AwaitRevision`/`AwaitApplied`, which confirm the schema cache but not that the data node has loaded the group's query topology, so the query could still fail with "group not found". It now retries until the query resolves, the same treatment the multi-group spec in the same file already had.
- Stop the TopN test overriding the configured eventually timeout. It passed `flags.EventuallyTimeout` and then chained `WithTimeout(10s)`, which takes precedence, so the spec always ran on a 10s budget even though CI builds the integration suites with `-X ...flags.eventuallyTimeout=30s` to give slow runners room. It was the only case in `test/cases` opting out.
- Await SchemaBarrier applied state after standalone schema preload so measure/stream/trace writes no longer race local caches.
- Fix wrong log level in trace lifecycle migration.

### Document

- Add a code-accurate, API-first "Storage & File Format" doc and correct stale storage/format descriptions: fix the on-disk hierarchy to `group → segment → shard → part` (in `tsdb.md`, `data-model.md`, `clustering.md`, `disk-management.md`, including the `dump` CLI path examples), correct the measure field-values file name (`fv.bin`, not `fields.bin`), clarify that the `GORILLA`/`ZSTD` enums are schema hints (the engine uses delta/dictionary + size-thresholded zstd), document the measure `index_mode` two-engine split and the trace span-store/sidx layout, and fix the property repair Merkle-tree SHA/snapshot-state descriptions. Replace the file-structure diagrams with inline mermaid.
- Refresh observability for FODC-proxy scraping: add two dashboards built for the deployment where Prometheus scrapes the FODC proxy `/metrics` as the single target, split by aggregation dimension — `grafana-fodc-nodes.json` for node/pod-level health and resources aggregated by `pod_name` (fleet overview, per-node health table, a "Topology: Pod-to-Pod Flows" table joining the publisher's and subscriber's views of each directed `source → target` flow into one row, resources, disk-by-path, Go runtime) and `grafana-fodc-workload.json` for business/data-level throughput and latency aggregated by `group` (cluster workload summary, liaison ingestion/query/publish + write-queue backlog, data storage/inverted-index, and the internal queue with per-operation throughput & p99 by group) — and remove the stale direct-scrape `grafana-cluster.json`. Audit and validate the observability prose/PromQL against a live cluster, replace the out-of-date self-observability-write image with an inline mermaid write-flow diagram, add a "Key Signals to Watch" section, and split the oversized `observability.md` into an `observability/` folder (`overview`, `logging`, `metrics`, `providers`, `profiling`, `tracing`). Reconcile the FODC overview's HTTP API surface (`/metrics`, `/metrics-windows`, `/cluster/topology`, `/cluster/lifecycle`, `/diagnostics`) with the proxy code and `apis.md`.
- Add a "Cluster Topology Rendering" doc (`docs/operation/fodc/topology.md`) that joins the FODC `/cluster/topology` node inventory with cluster signals to render a directed topology with two edge layers — the **request pipeline** (liaison→data, weighted from `banyandb_queue_pub_*` metrics) and the **lifecycle tier migration** (hot→warm→cold; path from the lifecycle service's entries in `/cluster/topology` `calls`, weight from the `banyandb_lifecycle_migration_*` family). A data node and its lifecycle sidecar render inside one pod boundary with the containers labeled `data` / `lifecycle`. Includes the per-edge PromQL, three captured sample inputs (topology, queue metrics, and the migration metrics taken while the daily run was in flight), and a stdlib-only join script (`render_topology.py`) that emits Graphviz DOT and Mermaid; cross-linked from the FODC overview and the metrics queue reference.

### Chores

- Upgrade Go, npm, OpenTelemetry, AWS SDK, and Google Cloud dependencies.
- Regenerate expired TLS test certificates with 100-year validity.
- Set Ginkgo `--repeat` to 0 in the flaky-test workflow so the hourly run completes within the 50-minute timeout.
- Refactor the dump tool into a reusable `banyand/dump` parser library.
- Strip macOS AppleDouble (`._*`) and `__MACOSX/` metadata from every release tarball (src, banyand, bydbctl, fodc-agent, fodc-proxy) so downstream users running `make generate` from a downloaded source tarball no longer hit "invalid control character" errors when `buf generate` walks the resource-fork files; export `COPYFILE_DISABLE=1` and filter `._*` files at the source.
- Bump Go and ui/mcp dependencies to clear Dependabot advisories: `golang.org/x/net` v0.52.0→v0.56.0 (CVE-2026-25680), `opencontainers/runc` v1.3.3→v1.3.6 (CVE-2026-41579). Refresh mcp/ui lockfiles and license attribution.
- Optimize the SkyWalking and Zipkin trace samplers and shared var-array decoder through deferred decoding, zero-copy string and tag handling, early tag rejection, direct scalar reads, cached rule prefixes, and escape-free decoding paths.
