# `EKV`

Eventually consistent durable KV store with opt-in per-key linearizable CAS.

EKV supports three runtime roles:

- **member mode** — the default. Stores data on disk, replicates to other
  members, serves eventual reads locally, and participates in CAS quorum.
- **observer mode** — stores data on disk, replicates to other durable
  replicas, serves eventual reads and eventual writes locally, but routes CAS
  operations to voting members.
- **client mode** — stateless. Uses the same public API, but forwards
  operations to a selected voting member based on configured region
  preference.

In member and observer mode, EKV stores data on disk (survives restarts and
node death) and replicates across all connected durable replicas
automatically. There is no leader: every durable replica serves eventual
reads and writes at all times, including during network partitions.
CAS writes (`if_vsn:`, `consistent: true`, `update/4`) still require quorum
and may fail when quorum is unavailable. Voting members execute CAS locally;
observers route CAS to voting members and apply the committed result locally
before replying on success. When connectivity is restored, durable replicas
converge to the same state.

## Quick Start

    # In your supervision tree
    {EKV, name: :my_kv, data_dir: "/var/data/ekv"}

    # Basic operations
    EKV.put(:my_kv, "user/1", %{name: "Alice"})
    EKV.get(:my_kv, "user/1")          #=> %{name: "Alice"}
    EKV.delete(:my_kv, "user/1")
    EKV.get(:my_kv, "user/1")          #=> nil

    # Prefix scans
    EKV.scan(:my_kv, "user/") |> Enum.to_list()
    #=> [{"user/1", %{name: "Alice"}, {ts, origin_node}}]

    EKV.keys(:my_kv, "user/") |> Enum.to_list() #=> [{"user/1", {ts, origin_node}}]

    # TTL — entry expires after 30 minutes
    EKV.put(:my_kv, "session/abc", token, ttl: :timer.minutes(30))

    # Subscribe to changes
    EKV.subscribe(:my_kv, "rooms/")
    EKV.put(:my_kv, "rooms/1", %{title: "Elixir"})
    # receive
    # => {:ekv, [%EKV.Event{type: :put, key: "rooms/1", value: %{title: ...}}], %{name: :my_kv}}

    # CAS setup (requires cluster_size; node_id auto-generates if omitted)
    {EKV,
     name: :my_kv_cas,
     data_dir: "/var/data/ekv_cas",
     cluster_size: 3,
     node_id: 1,
     wait_for_quorum: :timer.seconds(30)}

    # Client mode — stateless EKV API that routes to voters by region order
    {EKV,
     name: :my_kv_client,
     mode: :client,
     region: "ord",
     region_routing: ["iad", "dfw", "lhr"],
     wait_for_route: :timer.seconds(10),
     wait_for_quorum: :timer.seconds(10),
     shutdown_barrier: :timer.seconds(5)}

    # Observer mode — local eventual reads/writes, CAS routed to voters
    {EKV,
     name: :my_kv_observer,
     mode: :observer,
     data_dir: "/var/data/ekv_observer",
     cluster_size: 3,
     region: "lhr",
     region_routing: ["iad", "dfw", "lhr"],
     wait_for_route: :timer.seconds(10),
     wait_for_quorum: :timer.seconds(10)}

    # CAS put via lookup + if_vsn
    case EKV.lookup(:my_kv_cas, "lock/job-123") do
      nil ->
        EKV.put(:my_kv_cas, "lock/job-123", %{owner: "node-a"}, if_vsn: nil)

      {_value, vsn} ->
        EKV.put(:my_kv_cas, "lock/job-123", %{owner: "node-a"}, if_vsn: vsn)
    end

    # Handling ambiguous CAS result explicitly
    case EKV.put(:my_kv_cas, "lock/job-123", %{owner: "node-a"}, if_vsn: nil) do
      {:ok, vsn} ->
        {:locked, vsn}

      {:error, :conflict} ->
        :already_locked

      {:error, :unconfirmed} ->
        # Resolve by reading through the CAS barrier
        EKV.get(:my_kv_cas, "lock/job-123", consistent: true)
    end

    # Or opt in to internal ambiguity resolution for this call
    EKV.put(:my_kv_cas, "lock/job-123", %{owner: "node-a"},
      if_vsn: nil,
      resolve_unconfirmed: true
    )

Values can be any Erlang/Elixir term. They are stored as `:erlang.term_to_binary/1`
internally and deserialized with `:erlang.binary_to_term/1` on read.
*Note*: Avoid storing structs or anonymous functions within values.
See `Value Serialization Caveats` below for more information.

## Consistency Guarantees

EKV provides **eventual consistency**: all nodes will converge to the same
state given sufficient time and connectivity. It does **not** provide:

- **Read-your-writes across nodes.** A write on node A is not immediately
  visible on node B. Replication is asynchronous. On the *local* node, writes
  are immediately visible.

- **Causal ordering.** If node A writes key "x" then key "y", another node
  may see "y" before "x" (or "y" without "x" during a partition).

- **Multi-key transactions.** There is no way to atomically write multiple
  keys as a single unit. Per-key atomic read-modify-write is available via
  `update/4` (CAS mode).

### Conflict Resolution: Last-Writer-Wins (LWW)

For eventual-mode writes (`put/4` and `delete/3` without CAS options), EKV
keeps the write with the **highest nanosecond timestamp**. If timestamps are
identical (unlikely but possible with clock sync), the write from the
**lexicographically greater node name** wins as a deterministic tiebreaker.

This means:

- The "latest" write always wins, where "latest" is determined by the
  writer's local wall-clock timestamp. EKV uses `System.system_time(:nanosecond)`
  as the base and ensures same-node local writes on a shard do not reuse a
  timestamp.

- **Clock skew matters.** If node A's clock is 5 seconds ahead of node B,
  then A's writes will beat B's writes for the same key even if B wrote
  "after" A in wall-clock time. For best results, run NTP or a similar time
  sync service on all nodes. Clock skew of a few milliseconds (typical for
  NTP) is fine — conflicts at that granularity are rare. Clock skew of
  seconds or more will cause surprising results.

- **Deletes are writes too.** A delete is a timestamped tombstone. If you
  delete a key on node A at time T1 and write the same key on node B at time
  T2 > T1, the write wins and the key reappears. This is by design — the
  higher timestamp always wins regardless of whether the operation was a put
  or a delete.

CAS-mode writes do not use LWW ordering; they are ordered by CASPaxos ballots.

### Reads During Partitions

During a network partition, each side of the partition continues to serve
reads and accept eventual writes independently. CAS writes continue only on
partitions that can reach quorum; minority partitions return CAS errors such
as `{:error, :no_quorum}` or `{:error, :quorum_timeout}`. When the partition
heals, nodes sync and converge. For eventual writes, LWW determines the
surviving value for conflicting writes on the same key.

### Consistency Modes Per Key (Important)

EKV supports two write modes:

- **Eventual/LWW mode** — default `put/4` and `delete/3` without CAS options.
- **CAS mode** — `put/4` with `if_vsn:` or `consistent: true`, `delete/3`
  with `if_vsn:`, and `update/4`.

For predictable semantics, treat mode as **key ownership**:

- Different keys may use different modes in the same EKV instance.
- A key may start in eventual/LWW mode and later transition to CAS mode
  (`LWW -> CAS` is supported).
- Once a key is CAS-managed, eventual writes on that key are rejected
  (`CAS -> LWW` is not supported for writes).
- Keys managed via CAS should continue to use CAS **write** APIs.
- Reads for CAS-managed keys may be eventual (`get/2`, `lookup/2`) when
  staleness is acceptable, or consistent (`get/3, consistent: true`) when
  fresh linearizable reads are required.
- After transition to CAS mode, do not issue eventual writes on that key.

### CAS Outcomes and `:unconfirmed`

CAS write APIs (`put/4` with `if_vsn:` or `consistent: true`, `delete/3`
with `if_vsn:`, and `update/4`) can return:

- `{:ok, ...}` — write committed; returns the committed VSN.
- `{:error, :conflict}` — definite non-application for this attempt
  (for example stale `if_vsn`).
- `{:error, :unconfirmed}`: write entered accept phase, but the caller
  could not confirm final outcome. The write may already be committed,
  or it may have lost to a competing ballot and never committed.

On `:unconfirmed`, callers can issue `get(name, key, consistent: true)` to
resolve current committed state before taking follow-up actions.

For convenience, pass `resolve_unconfirmed: true` on CAS writes to make EKV
perform one internal barrier read and map ambiguous outcomes to current-state
results: `{:ok, ...}` / `{:error, :conflict}` when resolvable, or
`{:error, :unavailable}` if the resolution read cannot complete.

## Configuration

All options are passed when starting EKV.

Member mode (default):

    {EKV,
      name: :my_kv,
      data_dir: "/var/data/ekv",
      region: "iad",
      shards: 8,
      tombstone_ttl: :timer.hours(24 * 7),
      gc_interval: :timer.minutes(5),
      log: :info}

Observer mode:

    {EKV,
      name: :my_kv_observer,
      mode: :observer,
      data_dir: "/var/data/ekv_observer",
      cluster_size: 3,
      region: "lhr",
      region_routing: ["iad", "dfw", "lhr"],
      log: :info}

Client mode:

    {EKV,
      name: :my_kv_client,
      mode: :client,
      region: "ord",
      region_routing: ["iad", "dfw", "lhr"],
      log: false}

| Option | Default | Description |
|--------|---------|-------------|
| `:name` | *required* | Atom identifying this EKV instance. Used to register processes and as the first argument to all API functions. |
| `:mode` | `:member` | Runtime role. `:member` stores/replicates data and votes in CAS quorum. `:observer` stores/replicates data and routes CAS to voters. `:client` is stateless and routes operations to voters. |
| `:region` | `"default"` | Region label for this EKV instance. Durable replicas expose it for routing. Clients may set it for observability. |
| `:region_routing` | `nil` | Observer and client mode only. Ordered list of preferred voter regions, e.g. `["iad", "dfw", "lhr"]`. |
| `:wait_for_route` | `false` | Observer and client mode only. Optional startup gate. Blocks `EKV.start_link/1` until the first reachable voter in `:region_routing` order is selected, or fails startup on timeout. Voters advertise routes only after all their shards have initialized; this does not guarantee quorum. |
| `:data_dir` | *required in `:member` and `:observer`* | Directory where SQLite database files are stored. Created automatically if it doesn't exist. Each shard gets its own file (`shard_0.db`, `shard_1.db`, etc.). |
| `:shards` | `8` | Member and observer mode only. Number of shards. See "Choosing a Shard Count" below. |
| `:reader_connections` | `:auto` | Member and observer mode only. Number of SQLite reader connections per shard. `:auto` uses `min(System.schedulers_online(), 16)`. Use `:schedulers` to preserve one reader per scheduler, or a positive integer to tune explicitly. |
| `:cluster_size` | `nil` | Member and observer mode only. Logical voting cluster size for CAS quorum math. Required for CAS-capable durable replica deployments. |
| `:node_id` | auto-generated+persistent | Member and observer mode only. Stable logical durable-replica identity used by CAS ballots, persisted replay origins, member-progress retention, and blue-green overlap. Auto-generated on first boot if omitted, then reused from disk. |
| `:wait_for_quorum` | `false` | Optional startup gate. In member mode, blocks startup until this EKV member can reach CAS quorum. In observer and client mode, blocks startup until the selected backend voter reports CAS quorum reachable. |
| `:anti_entropy_interval` | `30_000` (30 sec) | Member and observer mode only. Periodic background repair for already-connected durable replicas. Re-runs the normal HWM-driven delta/full sync path to heal missed replication without waiting for reconnect. Must be a positive timeout in ms. |
| `:sync_chunk_size` | `500` | Member and observer mode only. Max entries per delta/full sync chunk during anti-entropy and catch-up. |
| `:sync_chunk_max_bytes` | `:replication_batch_max_bytes` | Member and observer mode only. Approximate uncompressed byte cap for delta/full sync chunks. Count and byte limits are both enforced; one oversized entry may exceed this cap so sync can make progress. |
| `:delta_sync_log_min_entries` | `8` | Member and observer mode only. Suppresses per-delta `info` logs for successful terminal delta syncs smaller than this many entries. `:verbose` logging still prints all deltas. |
| `:delta_sync_storm_window` | `60_000` (60 sec) | Member and observer mode only. Rolling per-shard window used to aggregate delta sync activity for storm detection. |
| `:delta_sync_storm_threshold` | `100` | Member and observer mode only. When a shard sends at least this many delta syncs inside one storm window, EKV emits a single aggregated warning for that window. `false`/`nil` disables storm warnings. |
| `:wire_compression_threshold` | `262_144` (256 KB) | Optional byte threshold for member-to-member wire compression of large replicated value payloads. `false`/`nil` disables it. Large live replication batch entry values, CAS accept, and full-payload CAS commit messages compress values on the wire only; values remain uncompressed on disk and on reads. |
| `:transport` | `nil` | Optional data-plane transport adapter for member shard sends and routed client RPC. `nil` uses Erlang distribution. Custom transports are externally supervised and configured as `{Module, opts}`. |
| `:local_write_batch_max_entries` | `32` | Member and observer mode only. Max adjacent non-CAS local LWW writes the shard will opportunistically drain into one SQLite batch before replying. Also currently sets the bounded post-replication control/CAS priority turn budget; there is no separate fairness knob yet. |
| `:local_write_batch_max_bytes` | `262_144` (256 KB) | Member and observer mode only. Max encoded byte size of one opportunistic non-CAS local LWW batch before the shard stops draining more local writes. |
| `:wal_checkpoint_interval` | `1_000` (1 sec) | Member and observer mode only. Target interval between passive checkpoints of the same shard. One background process visits independent shard databases round-robin so checkpoint I/O does not overlap across shard files. |
| `:wal_size_limit` | `67_108_864` (64 MB) | Member and observer mode only. SQLite journal size limit retained after a completed WAL reset. A pinned read transaction can temporarily exceed this value; EKV reports that checkpoint-starvation state. |
| `:replication_batch_flush_ms` | `3` | Member and observer mode only. Max time one live LWW replication batch may stay queued per destination shard before EKV flushes it. |
| `:replication_batch_max_entries` | `64` | Member and observer mode only. Max live LWW replication operations EKV queues per destination shard before flushing immediately. |
| `:replication_batch_max_bytes` | `262_144` (256 KB) | Member and observer mode only. Max encoded byte size of one live LWW replication batch per destination shard before flushing immediately. Replication turn-taking itself is not separately configurable today. |
| `:shutdown_barrier` | `false` | Optional graceful-shutdown barrier. Keeps EKV serving during coordinated shutdown for up to the configured timeout so members can finish final writes and replication. |
| `:allow_stale_startup` | `false` | Member and observer mode only. Dangerous recovery override. If `true`, EKV trusts on-disk data even when stale-db detection would normally refuse startup. Intended only for explicit disaster recovery / full cold-cluster restore cases. |
| `:tombstone_ttl` | `604_800_000` (7 days) | Member and observer mode only. How long tombstones (deleted entries) are kept before being permanently purged, in milliseconds. See "Tombstone Lifetime" below. |
| `:gc_interval` | `300_000` (5 min) | Member and observer mode only. How often garbage collection runs, in milliseconds. GC emits `:expired` events for TTL expiry, tombstones expired LWW rows, lazily purges expired CAS rows, and truncates the replication oplog. |
| `:oplog_gc_batch_size` | `1_000` | Member and observer mode only. Maximum replay-log rows deleted in one serialized SQLite transaction. EKV yields between batches and recomputes retained-member floors before every continuation. |
| `:log` | `:info` | Logging level. `:info` logs cluster events (connects, syncs). `false` disables logging. `:verbose` logs per-shard detail. |
| `:partition_ttl_policy` | `:quarantine` | Member and observer mode only. Policy for reconnects after downtime longer than `tombstone_ttl`. `:quarantine` blocks replication with that member identity until operator rebuild. `:ignore` disables that quarantine and allows reconnect/sync anyway. |
| `:blue_green` | `false` | Member and observer mode only. Enable blue-green deployment mode. See "Blue-Green Deployment" below. |

### Transport Adapters

By default EKV sends member shard traffic and routed client RPC over Erlang
distribution. `:transport` lets applications move that data plane onto an
externally supervised volatile ordered lane:

    {EKV,
      name: :my_kv,
      data_dir: "/var/data/ekv",
      transport: {MyApp.EKVTransport, name: :my_transport}}

The adapter contract is `EKV.Transport`. In `{Module, opts}`, EKV validates
`Module` and passes `opts` directly to `Module.init/1`; option keys such as
`:name` are adapter-owned and not interpreted by EKV. EKV initializes
adapter state per replica shard process for member traffic, and initializes
a lightweight adapter handle for each routed client RPC. Adapter `init/1`
must be cheap: validate opts, fetch/build a handle to an externally
supervised transport, and return. It must not start per-call transport
workers. EKV does not start or supervise the external transport.

### Observer Mode

Observer mode is for nodes that should keep a full local durable replica and
serve low-latency eventual reads locally, but should not expand the CAS voter
set.

- Observers start SQLite, replication, GC, subscriptions, and anti-entropy.
- Eventual reads and eventual writes are local on the observer.
- CAS reads/writes route to a selected voting member based on
  `:region_routing`.
- Successful observer CAS calls apply the committed result locally before
  replying, so immediate local eventual read-your-CAS-writes is preserved.
- Observers do not count toward CAS quorum and clients do not route to them.
- `{:error, :unconfirmed}` on an observer means EKV could not confirm the
  committed state was safely visible for the caller's current path; resolve
  with `get(name, key, consistent: true)` before trusting local eventual
  state.

### Client Mode

Client mode is intended for application nodes that need the EKV API but
should not expand the durable replica set or CAS quorum size.

- Clients do not start SQLite, replication, GC, or blue-green machinery.
- Eventual reads are no longer local SQLite reads; they are routed to the
  selected voter.
- `wait_for_route` can delay startup until a voter route is selected.
- `wait_for_quorum` can additionally delay startup until that selected voter
  reports CAS quorum reachable.
- Members run periodic anti-entropy by default so a connected member that
  missed a prior update eventually replays the normal sync path and heals.
- `scan/2` and `keys/2` still return Elixir streams, but are backed by paged
  RPC to the selected voter.
- `subscribe/2` works in client mode; client subscribers are delivered
  cluster-wide via `:pg`.
- If no backend is reachable, read APIs raise and write APIs return
  `{:error, :unavailable}`.
- After backend failover, eventual reads may observe an older replica view.
  Use `consistent: true` when freshness matters.

For deployment, scaling, blue-green rollout, quorum startup, and shutdown
guidance, see `OPERATORS.md` in the repository.

### Shutdown Barrier

`shutdown_barrier: timeout_ms` is an opt-in graceful-shutdown aid.

- In coordinated shutdowns, members can stay alive briefly while other members and
  clients enter terminal state, which reduces final-write `:no_quorum`
  failures and allows more replication to complete before exit.
- Blue-green outgoing members skip the barrier after successful handoff.
- This is best-effort only: crashes, `:kill`, and VM death bypass it.
- It does not replace correct supervision ordering; processes that must flush
  state should still shut down before EKV fully exits.

### Startup Quorum Gate

CAS-enabled EKV instances can optionally block startup until quorum is
reachable:

    {EKV,
     name: :my_kv_cas,
     data_dir: "/var/data/ekv_cas",
     cluster_size: 3,
     node_id: 1,
     wait_for_quorum: :timer.seconds(30)}

This is useful when downstream children perform CAS reads or writes during
their own `init/1` or startup callbacks. With `:wait_for_quorum` set, EKV
does not finish starting until quorum is reachable (or the timeout expires).

The gate is opt-in and only applies to CAS-configured instances. Eventual
reads/writes never require quorum and are unaffected.

### Choosing a Shard Count

The shard count controls write parallelism. Each shard is an independent
SQLite database with its own writer process. Writes to different shards
execute in parallel; writes to the same shard are serialized.

- **8 shards** (default) — good for most workloads. Provides 8-way write
  parallelism, which saturates typical NVMe drives.

- **1–2 shards** — appropriate for low-write-volume use cases (configuration
  stores, feature flags) where simplicity matters more than throughput.

- **16–32 shards** — consider this for write-heavy workloads (>10k writes/sec)
  on fast storage, or when you have many cores and want to reduce lock
  contention.

- **>32 shards** — rarely needed. Each shard opens multiple SQLite
  connections and file descriptors. More shards means more memory and more
  files.

**The shard count is permanent.** It is persisted in each database file on
first open. If you later start EKV with a different shard count against the
same `data_dir`, it will raise an `ArgumentError`. To change the shard
count, you must delete the existing data directory and start fresh. All
replicas in the cluster must use the same shard count — a node with a
mismatched count will have its replication connections rejected by members.

### Tombstone Lifetime

When you delete a key, EKV doesn't immediately erase it. Instead, it writes
a timestamped tombstone that replicates to all members, ensuring every node
learns about the delete. Tombstones are permanently purged after
`tombstone_ttl` (default 7 days).

If a node is offline for longer than the tombstone TTL, it may miss deletes
that have already been purged from other nodes. EKV handles this with
**stale database detection**: on startup, if the database's last activity
timestamp is older than the tombstone TTL safety window, EKV refuses startup
by default instead of trusting that on-disk state. This prevents "zombie"
keys from reappearing. Operators can then either wipe that node's data dir
so it rebuilds from members, or explicitly set `allow_stale_startup: true`
when they intentionally want to trust the old on-disk cluster state.

Reduce `tombstone_ttl` if storage is tight and your nodes are rarely offline
for long. Increase it if nodes may be offline for extended maintenance
windows.

### Long Live-Partition Protection

Startup stale-db detection covers nodes that were down and restarted. A
separate edge case is a very long network partition where nodes stay up past
`tombstone_ttl`.

By default (`partition_ttl_policy: :quarantine`), EKV detects reconnects
after downtime longer than `tombstone_ttl` and quarantines that member pair
instead of syncing potentially unsafe state. Replication remains blocked for
that member until an operator rebuilds one side.

Down-since markers are persisted in `kv_meta`, keyed by `node_id` when
available (fallback: node name). This preserves quarantine history across
restarts and prevents node-name churn from bypassing quarantine when
`node_id` is stable.

Anti-entropy also retries `member_connect` to current `EKV.MemberPresence`
members missing from `remote_shards`. That lets transient false down-markers
usually self-heal while they are still within `tombstone_ttl`.

Presence alone does not bypass overdue quarantine. If a persisted down-marker
has already aged past `tombstone_ttl`, EKV still quarantines that reconnect
by default until an operator rebuilds one side or explicitly widens the
safety window.

Fallback name-based markers are bounded: EKV periodically prunes very old
entries and caps retained fallback markers per shard.

## Blue-Green Deployment

When deploying with a blue-green strategy on a single machine, two BEAM VMs
run side by side briefly — the old release and the new one. If both VMs open
the same SQLite files simultaneously, the result is corruption. The
`:blue_green` option solves this with a synchronized handoff — the new VM
coordinates with the old VM to drain operations, flush WAL, and close the
writer before opening the same database files.

    {EKV, name: :my_kv, data_dir: "/var/data/ekv", blue_green: true}

**Startup behavior:**

- **First boot** (no marker file) — writes the marker and operates normally.

- **Same node restart** (marker node matches `node()`) — no handoff needed.
  This is a normal EKV restart within the same VM.

- **Different node** (marker node does not match `node()`) — synchronized
  handoff. The new VM sends a handoff request to each shard on the old VM.
  The old VM drains pending CAS operations, persists its ballot counter,
  checkpoints the WAL, and closes its writer connection. Only then does the
  new VM open the database files. The old VM enters proxy mode, forwarding
  any remaining write requests to the new VM.

If the marker points at a dead or unreachable old VM, EKV skips handoff and
opens the files directly. SQLite's WAL recovery handles any incomplete
writes.

**Requirements:**

- Each deploy must use a **different node name** (e.g. include a timestamp
  or release version in the name). Both VMs must share the same `node_id`.

- The `data_dir` must be on a **shared filesystem** accessible to both VMs.

**Disk space:** No additional storage — both VMs use the same database files
(sequentially, never simultaneously).

## Multiple Instances

You can run multiple independent EKV instances in the same BEAM by giving
each a different `:name`. Each instance has its own SQLite databases, its
own replication mesh, and its own configuration. They do not interact.

    children = [
      {EKV, name: :users, data_dir: "/data/users"},
      {EKV, name: :sessions, data_dir: "/data/sessions"}
    ]

## Replication

Replication is automatic and requires no configuration beyond Erlang
distribution. When a new node connects (via `Node.connect/1` or a cluster
manager like `DNSCluster`), EKV discovers member shards and syncs:

- **Delta sync** — if the nodes were recently connected and the replication
  log hasn't been truncated, only the missed entries are sent.

- **Full sync** — if this is the first connection or the node was away long
  enough for the oplog to be truncated, a full state transfer is performed
  (live entries + recent tombstones; expired rows are omitted).

After the initial sync, every local write is replicated to all connected
members in real time (fire-and-forget, async). Consistency is maintained by
LWW, not by delivery order.

## TTL

Entries can be given a time-to-live:

    EKV.put(:my_kv, "session/abc", token, ttl: :timer.minutes(30))

Expired entries are not returned by `get/2`, `scan/2`, or `keys/2`.
Periodic GC emits `:expired` events for subscribers. For eventual/LWW keys,
GC also writes a tombstone that replicates to other members. For CAS-managed
keys, expiry stays local/lazy and long-expired rows are purged later instead
of becoming replicated deletes.

### Value Serialization Caveats

Because values are persisted to disk and may be deserialized by a different
version of your application, **avoid storing terms that are tied to the
running code**:

- **Structs** — a struct is a map with a `__struct__` key pointing to a
  module atom. If the module is renamed, removed, or its fields change,
  deserialization will produce a bare map or a struct with missing/extra
  keys. Prefer plain maps (e.g. `%{type: "user", name: "Alice"}`) for
  durable storage.

- **Anonymous functions** — an anonymous function captures a reference to
  the module and function clause that created it. After a code deploy, that
  reference is invalid and `:erlang.binary_to_term/1` will fail. Never
  store anonymous functions within values.

- **PIDs, ports, references** — these are ephemeral identifiers that are
  meaningless after a restart.

If you need to evolve your value schema over time, **version-stamp your
values**:

    # Write
    EKV.put(:my_kv, "user/1", {2, %{name: "Alice", email: "a@example.com"}})

    # Read with migration
    case EKV.get(:my_kv, "user/1") do
      {2, data} -> data
      {1, data} -> Map.put(data, :email, nil)   # migrate v1 → v2
      nil       -> nil
    end

This way old values written before a schema change are migrated on read
without needing to backfill every key.

# `await_quorum`

Block until CAS quorum is reachable, or returns an error on timeout.

In member mode, this checks the same member-reachability predicate that CAS
writes use for early `:no_quorum` rejection.

In observer and client mode, this first waits for a backend route and then
asks the selected voter to perform the quorum readiness check.

It is primarily useful for startup orchestration and other cases where callers
want a bounded wait before issuing CAS traffic.

Returns:

- `:ok` when quorum is reachable
- `{:error, :timeout}` when quorum was not reached before `timeout_ms`
- `{:error, :cluster_overflow}` when more distinct `node_id`s are visible
  than `cluster_size` allows

Raises `ArgumentError` if the target member is not configured for CAS.

# `backup`

Create a backup of all shards to `dest_dir`.

Uses SQLite's online backup API — safe to call while EKV is running.
Returns `:ok` on success or `{:error, reason}` on failure.

Member and observer mode only.

# `child_spec`

# `delete`

Delete a key.

Writes a tombstone that replicates to all members.

## Options

- `:if_vsn` — compare-and-swap delete. Only succeeds if the key's current
  version matches.
  Returns `{:error, :conflict}` if the expected version does not match and
  the delete was not applied.
  Returns `{:error, :unconfirmed}` when the delete entered accept phase but the
  caller could not confirm final outcome; issue `get(name, key, consistent: true)`
  to resolve.
  Requires `cluster_size` config. Member mode auto-generates/persists
  `node_id` if omitted.
  For strict behavior, keep the key in CAS mode after transition and do not
  mix with eventual `put`/`delete` on the same key (`CAS -> LWW` writes are
  rejected).
  Eventual deletes on CAS-managed keys are rejected with
  `{:error, :cas_managed_key}`.
- `:resolve_unconfirmed` — boolean (default `false`). When `true`, if a CAS
  delete enters accept phase but cannot confirm final outcome, EKV performs
  one internal barrier read to resolve current state and returns
  `{:ok, vsn}`/`{:error, :conflict}` when possible, or
  `{:error, :unavailable}` if that resolution cannot complete.

## Returns

- Eventual delete (`delete` without CAS options): `:ok` or
  `{:error, :cas_managed_key}` when the key is CAS-managed
- CAS delete (`if_vsn:`): `{:ok, vsn}` where `vsn` is the version tuple
  `{timestamp, origin_node}` where `origin_node` is the persisted origin
  string (normally the stable member `node_id`)
- With `resolve_unconfirmed: true`, CAS delete may also return
  `{:error, :unavailable}` if ambiguity resolution cannot complete.

# `get`

Get a value by key. Returns `nil` for missing, expired, or deleted entries.

By default, reads directly from SQLite via a per-shard reader connection pool
(eventually consistent, no GenServer hop).

## Options

- `:consistent` — when `true`, performs a CASPaxos consensus read
  (barrier/linearizable for CAS-managed keys). This read always goes through
  CAS accept+commit to resolve any in-flight accepted value before replying.
  Requires `cluster_size` config. Member mode auto-generates/persists
  `node_id` if omitted.
- `:retries` — non-negative integer max CAS retries (default 5). Only for
  `consistent: true`.
- `:backoff` — `{min_ms, max_ms}` backoff range in ms where both are
  non-negative integers and `min_ms <= max_ms` (default `{10, 60}`).
- `:timeout` — positive integer call timeout in ms (or `:infinity`, default
  10_000).

# `info`

Return cluster status information.

Member mode returns node_id, cluster_size, shards, data_dir, connected members,
and region metadata.

Client mode returns region metadata, routing preferences, and the currently
selected backend (if any).

## Examples

    iex> EKV.info(:my_kv)
    %{
      name: :my_kv,
      mode: :member,
      region: "iad",
      node_id: "1",
      cluster_size: 3,
      shards: 8,
      data_dir: "/data/ekv",
      connected_members: [
        %{node: :"ekv2@10.0.0.2", node_id: "2"},
        %{node: :"ekv3@10.0.0.3", node_id: "3"}
      ]
    }

    iex> EKV.info(:my_kv_client)
    %{
      name: :my_kv_client,
      mode: :client,
      region: "ord",
      region_routing: ["ord", "iad", "dfw"],
      current_backend: :"ekv1@10.0.0.1"
    }

# `keys`

List keys (with versions) matching a prefix. Scans all shards using
cursor-based streaming.

Returns a `Stream` of `{key, vsn}` tuples where `vsn` is
the version tuple `{timestamp, origin_node}` where `origin_node` is the
persisted origin string.

This avoids decoding values while still allowing CAS workflows to chain
`if_vsn:` operations from scan output.

Results are sorted by key within each shard but not globally sorted across
shards.

In client mode, the returned stream is still local to the caller, but each
chunk is fetched from the selected voter via paged RPC.

# `lookup`

Look up a key's value and version.

Returns `{value, vsn}` where `vsn` is the version tuple
`{timestamp, origin_node}`,
or `nil` for missing, deleted, or expired keys.

The vsn can be passed to `put/4` with `if_vsn:` for compare-and-swap.
This is a local read (eventually consistent, no GenServer hop).

# `put`

Put a key-value pair.

## Options

- `:ttl` — positive integer time-to-live in milliseconds. Entry expires
  after this duration.
- `:if_vsn` — compare-and-swap. Only succeeds if the key's current version
  matches. Use `nil` for insert-if-absent.
  Returns `{:error, :conflict}` if the expected version does not match and
  the write was not applied.
  Returns `{:error, :unconfirmed}` when the write entered accept phase but the
  caller could not confirm final outcome; in this case, issue
  `get(name, key, consistent: true)` to resolve the committed value.
  Requires `cluster_size` config. Member mode auto-generates/persists
  `node_id` if omitted.
  Can be used on an existing LWW key to transition it to CAS-managed
  (`LWW -> CAS`).
- `:consistent` — when `true`, uses CASPaxos consensus for the write
  (quorum-backed CAS write). Mutually exclusive with `:if_vsn`. Requires
  `cluster_size` config. Member mode auto-generates/persists `node_id` if omitted.
  For strict behavior, keep the key in CAS mode after transition and do not
  mix with eventual `put`/`delete` on the same key (`CAS -> LWW` writes are
  rejected).
  Eventual writes to CAS-managed keys are rejected with
  `{:error, :cas_managed_key}`.
- `:retries` — non-negative integer max CAS retries on conflict (default 5).
  Only for `:consistent` and `:if_vsn` paths.
- `:backoff` — `{min_ms, max_ms}` backoff range in ms where both are
  non-negative integers and `min_ms <= max_ms` (default `{10, 60}`).
- `:timeout` — positive integer call timeout in ms (or `:infinity`, default
  10_000).
- `:resolve_unconfirmed` — boolean (default `false`). When `true`, if a CAS
  write enters accept phase but cannot confirm final outcome, EKV performs one
  internal barrier read to resolve current state and returns
  `{:ok, vsn}`/`{:error, :conflict}` when possible, or
  `{:error, :unavailable}` if that resolution cannot complete.

## Returns

- Eventual put (`put` without CAS options): `:ok` or
  `{:error, :cas_managed_key}` when the key is CAS-managed
- CAS put (`if_vsn:` or `consistent: true`): `{:ok, vsn}` where
  `vsn` is the version tuple `{timestamp, origin_node}` where `origin_node`
  is the persisted origin string (normally the stable member `node_id`)
- With `resolve_unconfirmed: true`, CAS put may also return
  `{:error, :unavailable}` if ambiguity resolution cannot complete.

# `read_node_id`

Returns the stored node_id from the data dir.

# `scan`

Scan key-value pairs matching a prefix.

Scans all shards using cursor-based streaming. Returns a `Stream` of
`{key, value, vsn}` tuples where `vsn` is the version tuple
`{timestamp, origin_node}` and `origin_node` is the persisted origin string.

Results are sorted by key within each shard but not globally sorted
across shards.

In client mode, the returned stream is still local to the caller, but each
chunk is fetched from the selected voter via paged RPC.

# `start_link`

# `subscribe`

Subscribe the calling process to change events for keys matching `prefix`.

The subscriber receives messages of the form:

    {:ekv, [%EKV.Event{type: :put | :delete | :expired, key: key, value: value}], %{name: name}}

- `:put` events contain the new value (decoded Elixir term).
- `:delete` events contain the previous value before deletion (or `nil`).
- `:expired` events contain the last local value observed before TTL expiry.

## Prefix Matching

Prefixes match at `"/"` boundaries:

- `""` — matches **all** keys (wildcard). This is the default.
- `"user/"` — matches `"user/1"`, `"user/abc/xyz"`, etc.
- `"user/1"` — matches **exactly** `"user/1"` (no trailing `/` = exact key match).

A subscription to `"foo"` does **not** match `"foobar"`. To match all
keys under a namespace, use a trailing slash: `"foo/"`.

## Delivery

Events are dispatched asynchronously — the write returns to the caller
before subscribers are notified. Under load, multiple writes may be
batched into a single message to each subscriber.

A process subscribed to overlapping prefixes (e.g. both `""` and
`"user/"`) receives each event exactly once.

Delivery is best-effort. Events may be lost if a dispatcher process
crashes between receiving the dispatch and sending to subscribers.

In client mode, subscriptions are distributed via `:pg`, so member writes can
deliver events directly to client subscribers without backend affinity.

# `unsubscribe`

Unsubscribe the calling process from events for the given prefix.

# `update`

Atomic read-modify-write.

Reads the current value, applies `fun`, and writes the result using CASPaxos.
Auto-retries on conflict (up to 5 times with random backoff).

Returns `{:ok, new_value, vsn}` on success where
`vsn` is the version tuple `{timestamp, origin_node}` for the committed
value.
Returns `{:error, :conflict}` when retries are exhausted before entering an
accept phase that could decide the write.
Returns `{:error, :unconfirmed}` when accept phase started but the caller could
not confirm final outcome; issue `get(name, key, consistent: true)` to
resolve.

Requires `cluster_size` config. Member mode auto-generates/persists
`node_id` if omitted.
Can be used to move a key from LWW to CAS-managed mode. After transition,
do not mix with eventual writes on the same key (`CAS -> LWW` writes are
rejected).

## Options

- `:ttl` — positive integer time-to-live in milliseconds for the new value.
- `:retries` — non-negative integer max CAS retries on conflict (default 5).
- `:backoff` — `{min_ms, max_ms}` backoff range in ms where both are
  non-negative integers and `min_ms <= max_ms` (default `{10, 60}`).
- `:timeout` — positive integer call timeout in ms (or `:infinity`, default
  10_000).
- `:resolve_unconfirmed` — boolean (default `false`). When `true`, if an
  update enters accept phase but cannot confirm final outcome, EKV performs
  one internal barrier read to resolve current state and returns
  `{:ok, new_value, vsn}`/`{:error, :conflict}` when possible, or
  `{:error, :unavailable}` if that resolution cannot complete.

The update callback may be either:

- a `fun/1` which receives the current value
- an MFA tuple `{Mod, fun, extra_args}` which is invoked as
  `apply(Mod, fun, [current_value | extra_args])`

In observer and client mode, the callback is executed on the selected voter,
so prefer a named function capture like `&MyModule.bump/1` or an MFA tuple.

---

*Consult [api-reference.md](api-reference.md) for complete listing*