# Lifecycle Management in BanyanDB

BanyanDB uses a hot-warm-cold data tiering strategy to optimize storage costs and query performance. The lifecycle management feature automatically migrates data between storage tiers based on configured policies.

## Overview

A dedicated lifecycle agent (an independent Go process) is responsible for:

- Identifying data eligible for migration based on lifecycle stages.
- Extracting data from the source node.
- Migrating data to designated target nodes.
- Removing the original data after a successful migration.

This process ensures data flows correctly through hot, warm, and cold stages as defined in your group configurations.

> **Prerequisite: node discovery must be configured.** The lifecycle agent resolves the target nodes of every stage by listing all data nodes through the cluster's node registry and filtering them by the stage's `node_selector`. In the default `--node-discovery-mode=none` setting the registry is empty, so the agent cannot find any destination to migrate data to and the run will be a no-op. Configure `dns` or `file` mode on the lifecycle agent using the same source of truth as the rest of the cluster; see the [node discovery documentation](node-discovery.md) for the full list of `--node-discovery-*` flags.

## Data Migration Process

The lifecycle agent performs the following steps:

1. **Identify Groups for Migration:**
   - Connect to the metadata service and list all groups.
   - Filter groups configured with multiple lifecycle stages.
   - Skip groups already processed (using progress tracking).

2. **Take Snapshots:**
   - Request a snapshot from the source data node.
   - Retrieve the stream, measure and trace data directory paths.

3. **Setup Query Services:**
   - Create read-only services for streams, measures and traces using the snapshot paths.

4. **Process Each Group:**
   - Determine the current stage based on node labels.
   - Identify target nodes for the next lifecycle stage.
   - For each stream, measure or trace:
     - Query data from the current stage.
     - Transform data into write requests.
     - Send write requests to target nodes.
   - Delete the source data after a successful migration.

5. **Progress Tracking:**
   - Track migration progress in a persistent progress file.
   - Enable crash recovery by resuming from the last saved state.
   - Clean up the progress file once the migration is complete.

## Configuration

Lifecycle stages are defined within the group configuration using the `LifecycleStage` structure.

| Field              | Description                                                            |
| ------------------ | ---------------------------------------------------------------------- |
| `name`             | Stage name (e.g., "hot", "warm", "cold")                               |
| `shard_num`        | Number of shards allocated for this stage                              |
| `segment_interval` | Time interval for data segmentation (uses `IntervalRule`)              |
| `ttl`              | Time-to-live before data moves to the next stage (uses `IntervalRule`) |
| `node_selector`    | Label selector to identify target nodes for this stage                 |
| `close`            | Indicates whether to close segments that are no longer live            |

### Example Configuration

```yaml
metadata:
  name: example-group
catalog: CATALOG_MEASURE
resource_opts:
  shard_num: 4
  segment_interval:
    unit: UNIT_DAY
    num: 1
  ttl:
    unit: UNIT_DAY
    num: 7
  stages:
    - name: warm
      shard_num: 2
      segment_interval:
        unit: UNIT_DAY
        num: 7
      ttl:
        unit: UNIT_DAY
        num: 30
      node_selector: "type=warm"
      close: true
    - name: cold
      shard_num: 1
      segment_interval:
        unit: UNIT_DAY
        num: 30
      ttl:
        unit: UNIT_DAY
        num: 365
      node_selector: "type=cold"
      close: true
```

In this example:

- Data starts in the default stage with 4 shards and daily segments.
- After 7 days, data moves to the "warm" stage with 2 shards and weekly segments.
- After 30 days in the warm stage, data transitions to the "cold" stage with 1 shard and monthly segments.
- Data is purged after 365 days in the cold stage.

## Command-Line Usage

The lifecycle command offers options to customize data migration:

```bash
lifecycle \
  --grpc-addr 127.0.0.1:17912 \
  --stream-root-path /path/to/stream \
  --measure-root-path /path/to/measure \
  --trace-root-path /path/to/trace \
  --node-labels type=hot \
  --progress-file /path/to/progress.json \
  --schedule @daily \
  --node-discovery-mode file \
  --node-discovery-file-path /etc/banyandb/nodes.yaml
```

### Command-Line Parameters

| Parameter                           | Description                                                                                              | Default Value                  |
|-------------------------------------|----------------------------------------------------------------------------------------------------------| ------------------------------ |
| `--node-labels`                     | Labels of the current node (e.g., `type=hot,region=us-west`)                                             | `nil`                          |
| `--grpc-addr`                       | gRPC address of the source data node to snapshot and read from                                           | `127.0.0.1:17912`              |
| `--enable-tls`                      | Enable TLS for gRPC connection                                                                           | `false`                        |
| `--insecure`                        | Skip server certificate verification                                                                     | `false`                        |
| `--cert`                            | Path to the gRPC server certificate                                                                      | `""`                           |
| `--stream-root-path`                | Root directory for stream catalog snapshots                                                              | `/tmp`                         |
| `--measure-root-path`               | Root directory for measure catalog snapshots                                                             | `/tmp`                         |
| `--trace-root-path`                 | Root directory for trace catalog snapshots                                                               | `/tmp`                         |
| `--progress-file`                   | File path used for progress tracking and crash recovery                                                  | `/tmp/lifecycle-progress.json` |
| `--schedule`                        | Schedule for periodic backup (e.g., @yearly, @monthly, @weekly, @daily, etc.)                            | `""`                           |
| `--migration-orphan-policy`         | What to do with rows whose measure/stream schema was deleted from the registry: `archive` or `discard`   | `archive`                      |
| `--migration-orphan-archive-subdir` | Relative subdirectory, under each catalog's root path, for archived orphan rows when policy is `archive` | `archive`                      |
| `--lifecycle-send-retry-timeout`    | Maximum total time to retry streaming a migration part to a target node on transient failures (target node restarting, disconnect, receiver busy) | `15m`                          |

## Handling Orphan (Deleted-Schema) Data

A source segment can hold data for a measure or stream whose schema was later deleted from the registry (for example, when the upstream metric definition is renamed or removed). Such rows can no longer be written to a target — their schema no longer exists — so the migration treats them as **orphan** rows.

Orphan handling applies to the **measure** and **stream** catalogs only. Trace migration is not wired for per-series orphan handling: a trace group has a single schema, so a deleted trace schema is a whole-group concern rather than a per-series orphan within a surviving group.

On the row-replay path (used when a source segment spans multiple target segments), the migration detects orphan rows, skips them so the rest of the group still migrates, and then — unlike rows skipped because of a series-index gap — lets the source segment be deleted normally. What happens to the orphan rows themselves is controlled by `--migration-orphan-policy`:

- `archive` (default): each orphan row is written as one JSON line so an operator can recover it. The archive lives in the `--migration-orphan-archive-subdir` subdirectory **under each catalog's own root path** (so measure orphans land under `<measure-root-path>/<subdir>/…` and stream orphans under `<stream-root-path>/<subdir>/…`); the catalog is therefore not a path level (it is recorded inside every record and manifest instead). The per-part JSONL is **gzip-compressed** on disk (the rows are highly repetitive, so this is ~37× smaller — e.g. 269 MB of raw JSONL becomes ~7 MB). Files are laid out as:

  ```
  <catalog-root-path>/<subdir>/<group>/seg-<segment-suffix>/shard-<id>/part-<part-id>.jsonl.gz
  <catalog-root-path>/<subdir>/<group>/seg-<segment-suffix>/manifest.json
  ```

  Each line is self-describing (decoded from the part's own column types, with no registry schema needed): it carries the group, catalog, the measure/stream name (always under the JSON key `measure`, regardless of catalog — the `catalog` field disambiguates), source location, series id, entity, timestamp (both RFC3339 `timestamp` and epoch-nanos `timestamp_unix_nano`), tags, and — for measures — `version`, `indexed_tags`, and `fields`; streams carry `element_id` instead of those three and omit `version` entirely (rather than emitting a misleading zero). The per-segment `manifest.json` (plain JSON) indexes which deleted measures/streams were archived and their row counts. A part with no orphan rows writes no file; re-running a part rewrites its file idempotently.
- `discard`: orphan rows are dropped; nothing is written to disk.

Orphan handling is **not** a migration error — the source segment is migrated and deleted normally. Instead of appearing in the report's `errors` buckets, it is summarized under an `orphans` section that records, per catalog → group → deleted measure/stream, how many rows were archived or discarded:

```json
"orphans": {
  "policy": "archive",
  "measure": { "sw_metricsHour": { "meter_..._hour": 1234 } },
  "stream":  { "<group>": { "<stream>": 56 } }
}
```

(The counts accumulate across resume cycles. If the process is hard-killed mid-group before the group finishes, that run's not-yet-flushed counts are lost from the report — the archive files and their `manifest.json` are unaffected.)

### Reading the archive

The `manifest.json` files are plain text — read them directly. The per-part data is gzip-compressed JSON Lines:

```bash
# inspect one part's rows (Linux: zcat; macOS: gzcat)
gunzip -c <measure-root-path>/<subdir>/<group>/seg-20260601/shard-0/part-000000000000003b.jsonl.gz | jq .

# project a few fields
gunzip -c part-*.jsonl.gz | jq -c '{measure, ts:.timestamp, v:.fields.value.value}'

# search without fully decompressing (Linux)
zgrep "meter_banyandb_instance_disk_usage" part-*.jsonl.gz

# what was archived for a segment (counts per deleted measure)
jq '{total_rows, total_series, measures:[.measures[]|{measure,rows}]}' .../seg-20260601/manifest.json
```

**Limitations and operations notes:**

- The archive lives under each catalog's own root path (`--measure-root-path` / `--stream-root-path`), so it shares the durability of the volume that already holds the catalog data — no separate path to provision. `--migration-orphan-archive-subdir` only changes the subdirectory name within that root.
- Detection only happens on the row-replay path. On the chunk-sync path (a source segment that maps to a single target segment), whole part files are copied without per-row decoding, so orphan rows pass through to the target. They then occupy target-stage disk until that stage's TTL expires (e.g. up to the warm-stage TTL), and whether the copy succeeds depends on the receiver's part-acceptance behavior for a now-unregistered schema.
- The archive directory is **not** cleaned up automatically. Prune it periodically once the data is no longer needed.

## Automatic Behavior on Warm and Cold Nodes

When a data node matches a lifecycle stage (i.e., its labels match a stage's `node_selector`), two behaviors are automatically configured:

- **Rotation is disabled**: The rotation task (which pre-creates segments on hot nodes) is not started on warm or cold nodes. Segments are created on demand by the lifecycle migration process, ensuring correct segment boundaries for multi-day intervals.
- **Retention is disabled on non-last stages**: TTL-based segment deletion is disabled on all stages except the last one. Data removal on intermediate stages is managed by the lifecycle migration process, not by the built-in retention mechanism.

If a node's labels do not match any stage, both rotation and retention are disabled as a safety measure.

The lifecycle agent also accepts the full suite of `--node-discovery-*` flags so it can enumerate every candidate target node in the cluster. At a minimum you need `--node-discovery-mode` and either `--node-discovery-dns-srv-addresses` (DNS mode) or `--node-discovery-file-path` (file mode). The set of nodes visible to the lifecycle agent must include every warm/cold data node that any stage's `node_selector` could match — use the same discovery configuration that the liaison nodes use. See the [node discovery documentation](node-discovery.md) for the complete reference.

## Best Practices

1. **Node Labeling:**
   - Use clear, consistent labels that reflect node capabilities and roles.
2. **Sizing Considerations:**
   - Allocate more shards to hot nodes for higher write performance.
   - Optimize storage by reducing shard count in warm and cold stages.
3. **TTL Planning:**
   - Set appropriate TTLs based on data access patterns.
   - Consider access frequency when defining stage transitions.
4. **Migration Scheduling:**
   - Run migrations during off-peak periods.
   - Monitor system resource usage during migrations.
5. **Progress Tracking:**
   - Use a persistent location for the progress file to aid in recovery.

## Example Workflow

1. **Setup nodes with appropriate labels:**

   Every node (liaison, hot, warm, cold, and the lifecycle agent itself) must point at the same node-discovery source so that the lifecycle agent and liaison nodes see the identical topology. The example below uses file mode with a shared `/etc/banyandb/nodes.yaml`; see the [node discovery documentation](node-discovery.md) for DNS mode.

   ```bash
   # Liaison node
   banyand liaison \
     --node-discovery-mode=file \
     --node-discovery-file-path=/etc/banyandb/nodes.yaml

   # Hot data node
   banyand data \
     --node-labels type=hot \
     --grpc-port 17912 \
     --node-discovery-mode=file \
     --node-discovery-file-path=/etc/banyandb/nodes.yaml

   # Warm data node
   banyand data \
     --node-labels type=warm \
     --grpc-port 18912 \
     --node-discovery-mode=file \
     --node-discovery-file-path=/etc/banyandb/nodes.yaml
   ```

2. **Create a group with lifecycle stages.**

`bydbctl` connects the `liaison` node to create a group with lifecycle stages:

```bash
   bydbctl group create -f - <<EOF
metadata:
   name: example-group
catalog: CATALOG_MEASURE
resource_opts:
   shard_num: 4
   segment_interval:
      unit: UNIT_DAY
      num: 1
   ttl:
      unit: UNIT_DAY
      num: 7
   stages:
      - name: warm
      shard_num: 2
      segment_interval:
         unit: UNIT_DAY
         num: 7
      ttl:
         unit: UNIT_DAY
         num: 30
      node_selector: "type=warm"
      close: true
EOF
```

3. **Run the lifecycle agent:**

`lifecycle` connects to the `hot` node (as its snapshot source via `--grpc-addr`) and uses node discovery to locate the warm/cold target nodes:

   ```bash
   lifecycle \
     --node-labels type=hot \
     --grpc-addr 127.0.0.1:17912 \
     --stream-root-path /data/stream \
     --measure-root-path /data/measure \
     --trace-root-path /data/trace \
     --progress-file /data/progress.json \
     --node-discovery-mode=file \
     --node-discovery-file-path=/etc/banyandb/nodes.yaml
   ```

4. **Verify migration:**
   - Check logs for progress details.
   - Query both hot and warm nodes to ensure data availability.

This process automatically migrates data according to the defined lifecycle stages, optimizing your storage usage while maintaining data availability.
