> For the complete documentation index, see [llms.txt](https://docs.fast.poker/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.fast.poker/for-operators-dealer-service/heartbeat-metrics.md).

# Heartbeat and metrics

The Dealer Service writes runtime state under `data/` beside the packaged binary, unless `DEALER_HOME` points to another state directory. The dashboard and CLI read the same files.

## Files

| File                        | Cadence                      | Format                          |
| --------------------------- | ---------------------------- | ------------------------------- |
| `data/crank-heartbeat.json` | Every 5 seconds              | Single JSON object, overwritten |
| `data/crank-metrics.json`   | Batched after metric changes | Single JSON object, overwritten |
| `data/.crank.pid`           | On crank startup             | Plain text PID                  |

## Heartbeat

```json
{
  "pid": 12345,
  "startedAt": 1747652479000,
  "heartbeat": 1747654321000,
  "status": "running",
  "tablesTracked": 12,
  "tablesProcessing": 3,
  "uptime": "0h 30m",
  "recentErrors": [
    "settle_hand: InvalidWritableAccount"
  ],
  "crankMetrics": {
    "totals": {
      "totalCranks": 42,
      "totalSuccess": 41,
      "totalFailed": 1
    }
  }
}
```

### Field reference

| Field                               | Meaning                                                                                               |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `pid`                               | OS process ID                                                                                         |
| `startedAt`                         | Unix ms when the crank started                                                                        |
| `heartbeat`                         | Unix ms when this file was last written                                                               |
| `status`                            | `"running"` or `"stopped"`                                                                            |
| `tablesTracked`                     | Tables the dealer is watching                                                                         |
| `tablesProcessing`                  | Tables with an in-flight TX right now                                                                 |
| `uptime`                            | Human-readable uptime                                                                                 |
| `recentErrors`                      | Recent crank errors                                                                                   |
| `rpc`                               | L1 and TEE RPC health snapshots                                                                       |
| `config`                            | Current config with sensitive values redacted                                                         |
| `teePayer`, `feeAccounts`, `funder` | Fee-payer health and balances                                                                         |
| `cashRewardSnapshots`               | Cash reward snapshot counters                                                                         |
| `governor`                          | Per-channel RPC governor state (`l1` / `tee`): `enabled`, `inflight`, `waiting`, `rejected`, `tokens` |
| `altReclaim`                        | Dead-ALT rent-reclaim sweep counters and reclaimable SOL                                              |
| `crankMetrics`                      | Same aggregate metrics exposed in `data/crank-metrics.json`                                           |
| `laserstream`                       | Streaming state when LaserStream is enabled                                                           |

### Liveness probe

A dealer is healthy if `heartbeat` is within 30 seconds of now. Beyond that, treat it as stale and consider restarting.

Minimal bash check:

```bash
LAST=$(jq -r '.heartbeat' data/crank-heartbeat.json)
NOW=$(date +%s%3N)
AGE=$(( (NOW - LAST) / 1000 ))
[ "$AGE" -lt 30 ] && echo "OK ($AGE s)" || echo "STALE ($AGE s)"
```

## Metrics

`data/crank-metrics.json` is an aggregate JSON object:

```json
{
  "updatedAt": 1747654321000,
  "totals": {
    "erCranks": 30,
    "l1Cranks": 12,
    "totalCranks": 42,
    "erCostLamports": 0,
    "l1CostLamports": 60000,
    "totalCostLamports": 60000,
    "totalSuccess": 41,
    "totalFailed": 1,
    "simSaved": 2
  },
  "byLabel": {
    "settle_hand": {
      "chain": "ER",
      "count": 10,
      "costLamports": 0,
      "successCount": 10,
      "failCount": 0
    }
  }
}
```

### Categories tracked

| Category               | Labels                                                                                                                                                                             |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Game actions           | `start_game`, `tee_deal`, `tee_reveal`, `settle_hand`, `handle_timeout`                                                                                                            |
| SNG duels              | `maybe_start_sng_duel`, `sng_duel_action`, `sng_duel_timeout`, `resolve_sng_duel`                                                                                                  |
| Delegation             | `delegate_*`, `commit_state`                                                                                                                                                       |
| Undelegation           | `commit_and_undelegate_table`, `cleanup_seat`, `cleanup_seat_cards`, `cleanup_deposit_proof`                                                                                       |
| SNG records and prizes | `init_settlement_record`, `delegate_settlement_record`, `settle_sng_jackpots`, `distribute_prizes_from_record`, `assert_table_clean`, `reset_sng_table`, `close_settlement_record` |
| Rake and rewards       | `settle_table_rewards`, `update_acc_reward_per_weight`, `finalize_operator_reward_epoch`, `claim_operator_rewards`, `claim_operator_token_rewards`                                 |
| Cashouts               | `process_cashout_v2`, `clear_leaving_seat`                                                                                                                                         |
| Admin                  | `crank_remove_player`, `crank_kick_inactive`                                                                                                                                       |

### Querying

The file is a single JSON object, so query it with `jq`:

```bash
# Overall failure rate
jq '.totals |
    {total: .totalCranks,
     failed: .totalFailed,
     rate: (if .totalCranks == 0 then 0 else .totalFailed / .totalCranks end)}' \
  data/crank-metrics.json

# Highest-count labels
jq '.byLabel |
    to_entries |
    map({label: .key, count: .value.count, failures: .value.failCount}) |
    sort_by(.count) |
    reverse |
    .[0:5]' \
  data/crank-metrics.json
```

### Retention

The metrics file is overwritten with current aggregate counters. To archive a snapshot, move or copy `data/crank-metrics.json`; the dealer will recreate it.

Operators who need longer retention should archive snapshots or ship dashboard status responses to their own logging stack.

## Metrics export

The public Dealer Service monitoring surface is the heartbeat file, aggregate metrics file, CLI status, and dashboard status API. Operators who need Prometheus-style metrics can publish their own exporter from those sources.