jerapah-flow/README.md

# JerapahFlow

![JerapahFlow](packages/web/src/theme/brand/wordmark.png)

Workflow runner with a sandboxed script engine, SQLite run history, and an admin UI.

## Packages

- `@jerapah-flow/server` (`packages/server`) — Fastify runner, HTTP/cron triggers, admin REST API
- `@jerapah-flow/web` (`packages/web`) — React admin UI (Vite, DaisyUI, React Query)

## Setup

```bash
pnpm install
# Redis required for the workflow queue
pnpm dev
```

- UI (dev): http://localhost:8500
- API: http://localhost:8700

The first account created becomes **admin**. Later accounts are created from Users.

### Process modes

| Command | Processes | Ports |
|---|---|---|
| `pnpm dev` | Monolith (`runner.js` = API + worker) + Vite | UI **8500**, API **8700** |
| `pnpm dev:pm2` | Control + PM2 HTTP + PM2 workers + Vite | UI **8500**, control **8600**, API **8700** |

`pnpm dev:pm2` is the mode for Ops (start/stop HTTP, scale workers, drain restart). Control owns SQLite migrations; HTTP/workers do not migrate.

## Scripts (core vs plugins)

| Kind | Name in YAML | Editable | Location |
|---|---|---|---|
| **Core** | `fetch-http.js`, `s3.js`, … | No (fork only) | `packages/server/scripts/` |
| **User plugin** | `plugin/<id>` | Yes | `plugins/<id>/` |
| **Example source** | install → `plugin/<id>` | After install | `examples/plugins/<id>/` |

- App version is **`0.1.0`** (root `package.json`). Plugin manifests declare `jerapah: ">=0.1.0 <1.0.0"`.
- Install plugins via admin API: zip (base64), HTTPS git URL, example, or fork a core script.
- Install/update/uninstall sets **restart-needed** — drain-restart HTTP + workers under `pnpm dev:pm2`.
- Shipped example source (install from UI/API): `examples/plugins/get-current-time` → runtime `plugin/get-current-time`.
- `plugins/joplin-api` and `plugins/send-sms` are **personal/user plugins** appropriate for a fork — not shipped examples. See **AGENTS.md** for creating user plugins under `plugins/<id>/`.

## Workflows (instance data vs examples)

| Kind | Loaded by runner? | Location |
|---|---|---|
| **Live workflows** | Yes | `packages/server/data/workflows/<owner>/` (gitignored) |
| **Example presets** | No | `examples/workflows/*.yaml` — offered when creating a new workflow |

- Live YAML is **instance data**, same as SQLite and secrets — not product source. Prefer owner `local` for personal workflows.
- On first start, if the instance store is empty and a legacy `packages/server/workflows/` tree still exists, it is copied into `data/workflows/`.
- New workflow editor starts empty; optional presets copy example YAML into the editor (nothing is saved until Save).
- Override the live store in tests with `JFLOW_WORKFLOWS_DIR`.

```bash
# Smoke
JFLOW_PLUGINS_DIR=packages/server/data/plugins-smoke-test \
JFLOW_DB_PATH=packages/server/data/plugins-smoke.db \
node packages/server/test/plugins-smoke.js
```


## Script contract

Each script is `async function main(ctx)` and **must** return:

```js
{ output, context?, skipRemaining? }
```

| Field | Meaning |
|---|---|
| `ctx.data` | This step’s input (trigger payload, previous `output`, or DAG `needs`) |
| `ctx.context` | Run clipboard (plain object, default `{}`) |
| `ctx.config` | This step’s YAML config |
| `output` | Becomes the **next** step’s `data` |
| `context` | Next snapshot of the bag. Omitted → keep incoming |
| `skipRemaining` | Stop later steps. Sibling of `output`/`context`, not inside `output` |

Returning the full `ctx` is an error. Mutating `ctx.data` or `ctx.context` does not persist unless returned.

YAML **SET** evaluates JSONata against the full `ctx`; the result is `output` (the next step’s data). `jsonata.js` does the same.

DAG `needs` assemble this step’s `data` from upstream **outputs**. Independent steps in the same wave share a context snapshot; sibling writes to the same context key fail the run.

Optional `script.meta.reads = "ctx"` documents expression hosts. `meta.input` / `meta.output` / `meta.context` describe `data`, the return pipe, and clipboard keys.

## Scripts

| Command | Description |
|---|---|
| `pnpm dev` | Monolith server + Vite (no PM2) |
| `pnpm dev:pm2` | Control + PM2 HTTP/workers + Vite (Ops UI) |
| `pnpm dev:server` | Monolith API/runner only |
| `pnpm dev:web` | UI only (proxies `/api` → :8700, `/ops` → :8600) |
| `pnpm build` | Production UI build |
| `pnpm start` | Monolith: API + worker + built UI |
| `pnpm start:control` | Control plane only (migrates, manages PM2 children) |
| `pnpm start:api` | HTTP API + cron enqueue (`JFLOW_ROLE=api`) |
| `pnpm start:worker` | BullMQ worker only |
| `pnpm migrate` | Apply SQLite migrations |

## Ops (control plane)

Admin UI route **Ops** (`/ops`) talks to the control process.

| Action | Behavior |
|---|---|
| Pause / resume | BullMQ `queue.pause()` / `resume()` — cron/HTTP still enqueue |
| Reload workflows | Redis pub/sub → all live HTTP/worker processes re-read YAML |
| Scale workers | PM2 scale; scale-down drains active jobs unless `force` |
| Drain restart | Pause → wait active=0 → stop children → migrate → recreate → resume |
| Force restart | Same without waiting (interrupts active runs; orphans marked `worker_lost`) |

Desired state is stored in `packages/server/data/control-state.json` (generation, worker count, restart-needed). Plugin installs (later) bump generation and set restart-needed; you apply with Drain restart.

## Environment

| Variable | Default | Notes |
|---|---|---|
| `JFLOW_JWT_SECRET` | `jflow-dev-secret` (dev only) | **Required in production**. |
| `JFLOW_SECRETS_KEY` | `jflow-dev-secrets-key` (dev only) | Master key for named secrets. **Required in production**. Changing it makes existing secrets unreadable. 64 hex chars are used as a raw AES-256 key; any other string is derived with scrypt. |
| `JFLOW_DB_PATH` | `packages/server/data/jerapah-flow.db` | SQLite file. |
| `JFLOW_WORKFLOWS_DIR` | `packages/server/data/workflows` | Live workflow YAML (instance data). |
| `REDIS_URL` | `redis://127.0.0.1:6379` | Redis for BullMQ workflow queue. **Required** — the server will not start if Redis is unreachable. |
| `REDIS_PASS` | — | Optional Redis AUTH password (sent via ioredis `password`). Prefer this over embedding credentials in `REDIS_URL` so logs stay clean. |
| `JFLOW_QUEUE_NAME` | `jerapah-workflows` | BullMQ queue name. |
| `JFLOW_WORKER_CONCURRENCY` | `5` | Max parallel workflow jobs per worker process. |
| `JFLOW_ROLE` | `all` | `all` (HTTP + cron + worker), `api`, or `worker`. Prefer `pnpm start:api` / `start:worker` under control. |
| `JFLOW_CONFIG_GENERATION` | `1` | Set by control/PM2 so children report config generation in heartbeats. |
| `JFLOW_CONTROL_PORT` | `8600` | Control ops API port. |
| `JFLOW_LOG_LEVEL` | `debug` | Pino level |
| `JFLOW_RETENTION_DAYS` | `30` | Run history prune |
| `JFLOW_CORS_ORIGIN` | `http://localhost:8500` | Vite origin in dev |
| `PORT` | `8700` | HTTP API port |
| `NODE_ENV` | — | Set `production` for secure cookies |

Workflow runs are **queued** via BullMQ. HTTP and manual triggers return `202 { runId, status: "queued" }` immediately; poll `GET /api/runs/:id` for progress (`queued` → `running` → `success` \| `failed`). Cron remains an in-process producer that enqueues jobs on each tick.

## Production

```bash
pnpm install
pnpm build
# Redis must be reachable at REDIS_URL (set REDIS_PASS if Redis requires AUTH)
# Recommended: run control (migrates + manages PM2 HTTP/workers)
JFLOW_JWT_SECRET=... JFLOW_SECRETS_KEY=... REDIS_URL=redis://127.0.0.1:6379 REDIS_PASS=... NODE_ENV=production pnpm start:control
# Or monolith (dev-style):
# ... pnpm start
```

With control, serve the built UI from Vite preview, a reverse proxy, or set `JFLOW_SERVE_UI=1` on the HTTP process.