# BenchCode operations runbook (HA, degradation, backups)

> **Canonical doc tree:** In the Laravel app repository, open **`docs/README.md`** (markdown cross-links target that tree). You are viewing the **public mirror** intended for `/docs/runbooks/` in the browser.

## High availability overview

BenchCode is composed of **Laravel (control plane)**, your **database**, and the **Python inference service** (`PYTHON_SERVICE_URL`). Clients see streaming failures if any hop is unhealthy. Treat each layer independently in monitoring.

### What to monitor

- Laravel `/up` route (load balancers / k8s probes).
- Database connectivity and disk headroom.
- `GET {PYTHON_SERVICE_URL}/health` latency and non-200 ratios.
- OAuth dependencies (GitHub availability) for PR workflows.

### Graceful degradation (product feature)

1. Open **Admin → Settings → Governance / Ops → Upstream degraded mode**.
2. Populate **substring allow-list** with cheap / local model slugs (`llama3.2`, `qwen2.5-coder:1.5b`, …).
3. Operators should pair this with **routing rules** (**Admin → Governance**) so “Explore” traffic maps to those models automatically.

When degraded mode is enabled, any model request that does not match the comma-separated allow-list receives a deterministic error rather than pinning your GPU cluster.

Scheduled probes refresh the diagnostics cache via `AiUpstreamHealth::refreshProbeCache` (every five minutes when `schedule:run` is active).

## Metadata backups

Backups intentionally focus on **governance tables** plus operational metadata—not raw Git objects.

Generate an encrypted artifact (requires `BENCHCODE_BACKUP_KEY`):

```bash
php artisan benchcode:backup-metadata --with-messages
```

Restore (**destructive** — only staging / empty clones):

```bash
php artisan benchcode:restore-metadata metadata/your-file.json.enc --force
```

Test restores quarterly: import into SQLite locally, smoke sign-in, validation only.

### Key management

Generate a fresh 256-bit secret and base64-wrap it:

```bash
openssl rand -base64 32
```

Store it in vault automation and mirror into `.env` as `BENCHCODE_BACKUP_KEY`.

## Directory sync

SCIM is guarded by bearer token auth. Configure:

```
SCIM_ENABLED=true
SCIM_BEARER_TOKEN=...
SCIM_AUTO_APPROVE=true # optional — auto-approve IdP births
```

Base URL your IdP should call:

```
https://<benchcode-host>/scim/v2/
```