Compare commits
15
Commits
main
..
f1cdb7ac68
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f1cdb7ac68 | ||
|
|
3f0774813d | ||
|
|
5615d989fa | ||
|
|
e2e70a5aad | ||
|
|
dd98421b08 | ||
|
|
69312c9080 | ||
|
|
c8697532f9 | ||
|
|
be8122f9e5 | ||
|
|
0a8463d8f4 | ||
|
|
721f15e900 | ||
|
|
877ea9e3f8 | ||
|
|
93b51dcaaa | ||
|
|
eb7c318c13 | ||
|
|
864427b45c | ||
|
|
46aa8ca327 |
@@ -0,0 +1,33 @@
|
||||
name: Deploy to Raspberry Pi
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [dev]
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: home # must match a label on your act_runner
|
||||
steps:
|
||||
- name: Deploy
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
# act_runner uses bash --noprofile --norc; load nvm/pnpm explicitly
|
||||
export NVM_DIR="/home/nsrb/.nvm"
|
||||
export PNPM_HOME="/home/nsrb/.local/share/pnpm"
|
||||
# shellcheck disable=SC1091
|
||||
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
|
||||
export PATH="$PNPM_HOME/bin:$PATH"
|
||||
|
||||
APP_DIR=/home/nsrb/apps/jerapah-flow
|
||||
cd "$APP_DIR"
|
||||
|
||||
git fetch origin dev
|
||||
git checkout dev
|
||||
git pull --ff-only origin dev
|
||||
|
||||
pnpm install --frozen-lockfile
|
||||
pnpm build
|
||||
|
||||
pm2 startOrReload ecosystem.config.cjs --update-env
|
||||
pm2 save
|
||||
+3
-1
@@ -1,6 +1,8 @@
|
||||
node_modules/
|
||||
.pnpm-store/
|
||||
# Instance data (SQLite, live workflows, control-state, backups)
|
||||
data/
|
||||
# Process logs
|
||||
logs/
|
||||
*.db
|
||||
*.db-*
|
||||
@@ -10,7 +12,7 @@ packages/web/dist
|
||||
# Personal/local scripts and workflows (not for the repo)
|
||||
debug-*.js
|
||||
|
||||
# Legacy live workflow tree (migrated to packages/server/data/workflows/)
|
||||
# Legacy live workflow trees
|
||||
packages/server/workflows/
|
||||
|
||||
# Plugin install staging and per-plugin deps
|
||||
|
||||
@@ -4,14 +4,14 @@ This file tells agents how to add a **user plugin**. Do not put personal or site
|
||||
|
||||
## Workflows (instance data)
|
||||
|
||||
Live workflows are **not** product source. They live under `packages/server/data/workflows/<owner>/` (gitignored; override with `JFLOW_WORKFLOWS_DIR`).
|
||||
Live workflows are **not** product source. They live under `data/workflows/<owner>/` (gitignored; override with `JFLOW_WORKFLOWS_DIR`).
|
||||
|
||||
| Kind | In git? | Path |
|
||||
|---|---|---|
|
||||
| Live / personal YAML | No | `packages/server/data/workflows/<owner>/` |
|
||||
| Live / personal YAML | No | `data/workflows/<owner>/` |
|
||||
| Example presets | Yes | `examples/workflows/*.yaml` (copy into editor only; runner does not load them) |
|
||||
|
||||
Do **not** add personal YAML under `packages/server/`, `examples/workflows/`, or the legacy `packages/server/workflows/` tree. Prefer owner `local`. Example presets must use **core** scripts only (no `plugin/…` that requires install).
|
||||
Do **not** add personal YAML under `packages/server/`, `examples/workflows/`, or `data/workflows/`. Prefer owner `local`. Example presets must use **core** scripts only (no `plugin/…` that requires install).
|
||||
|
||||
## Where things live
|
||||
|
||||
@@ -120,7 +120,7 @@ Return `{ output, context?, skipRemaining? }`. Do not return `ctx`. Mutations of
|
||||
|---|---|
|
||||
| `ctx.data` | Step input (trigger payload, previous `output`, or DAG `needs`) |
|
||||
| `ctx.context` | Run clipboard (plain object) |
|
||||
| `ctx.config` | YAML `config` (secrets already unwrapped from `$SECRET_name`) |
|
||||
| `ctx.config` | YAML `config` (mustache refs like `{{ secrets.name }}` already resolved) |
|
||||
| `output` | Next step’s `data` |
|
||||
| `context` | Next clipboard. Omit to keep incoming |
|
||||
|
||||
@@ -142,7 +142,7 @@ scripts:
|
||||
script: plugin/my-plugin
|
||||
config:
|
||||
url: http://10.8.0.6:3030/notes
|
||||
token: $SECRET_joplin_api_token
|
||||
token: "{{ secrets.joplin_api_token }}"
|
||||
```
|
||||
|
||||
Optional `name` is the display title in the editor and graph (falls back to the script filename). Canonical ref is `plugin/<id>` (`.js` suffix is optional).
|
||||
@@ -153,4 +153,4 @@ Optional `name` is the display title in the editor and graph (falls back to the
|
||||
- Use an id that matches a core script file (`ntfy`, `jsonata`, …).
|
||||
- Mismatch folder name and `jerapah-plugin.json` `id` (plugin is disabled).
|
||||
- Commit `plugins/.staging-*` or `plugins/*/node_modules/`.
|
||||
- Put secrets in `script.js`; use YAML `$SECRET_name` / `$VAR_name`.
|
||||
- Put secrets in `script.js`; use YAML `{{ secrets.name }}` / `{{ vars.name }}` (quote if the value starts with `{`).
|
||||
|
||||
@@ -55,16 +55,16 @@ pnpm --dir packages/server reset-admin -- --username admin --password 'your-pass
|
||||
|
||||
| Kind | Loaded by runner? | Location |
|
||||
|---|---|---|
|
||||
| **Live workflows** | Yes | `packages/server/data/workflows/<owner>/` (gitignored) |
|
||||
| **Live workflows** | Yes | `data/workflows/<owner>/` (gitignored) |
|
||||
| **Example presets** | No | `examples/workflows/*.yaml` — offered when creating a new workflow |
|
||||
|
||||
- Live YAML is **instance data**, same as SQLite and secrets — not product source. New resources use owner `local` (owner remains in storage/URLs for a possible future multi-tenant mode; the UI hides it).
|
||||
- On first start, if the instance store is empty and a legacy `packages/server/workflows/` tree still exists, it is copied into `data/workflows/`.
|
||||
- New workflow editor starts empty; optional presets copy example YAML into the editor (nothing is saved until Save).
|
||||
- Override the live store in tests with `JFLOW_WORKFLOWS_DIR`.
|
||||
|
||||
```bash
|
||||
# Smoke
|
||||
# Smoke (isolated under packages/server/data — not the live instance tree)
|
||||
JFLOW_DATA_DIR=packages/server/data \
|
||||
JFLOW_PLUGINS_DIR=packages/server/data/plugins-smoke-test \
|
||||
JFLOW_DB_PATH=packages/server/data/plugins-smoke.db \
|
||||
node packages/server/test/plugins-smoke.js
|
||||
@@ -83,13 +83,34 @@ Each script is `async function main(ctx)` and **must** return:
|
||||
|---|---|
|
||||
| `ctx.data` | This step’s input (trigger payload, previous `output`, or DAG `needs`) |
|
||||
| `ctx.context` | Run clipboard (plain object, default `{}`) |
|
||||
| `ctx.config` | This step’s YAML config |
|
||||
| `ctx.config` | This step’s YAML config (mustache refs already resolved) |
|
||||
| `output` | Becomes the **next** step’s `data` |
|
||||
| `context` | Next snapshot of the bag. Omitted → keep incoming |
|
||||
| `skipRemaining` | Stop later steps. Sibling of `output`/`context`, not inside `output` |
|
||||
|
||||
Returning the full `ctx` is an error. Mutating `ctx.data` or `ctx.context` does not persist unless returned.
|
||||
|
||||
### Config interpolation
|
||||
|
||||
YAML `config` strings may use mustache paths. Quote values that start with `{`.
|
||||
|
||||
```yaml
|
||||
url: "{{ vars.ntfy_channel }}"
|
||||
token: "{{ secrets.joplin_api_token }}"
|
||||
id: "{{ context.user.id }}"
|
||||
title: "{{ data.httpResponse.data.date }}"
|
||||
topic: "{{ vars.ntfy_prefix }}/{{ data.channel }}"
|
||||
```
|
||||
|
||||
| Root | Meaning |
|
||||
|---|---|
|
||||
| `vars` | Owner variable; remaining segments are the flat name (`{{ vars.foo.bar }}` → variable `foo.bar`) |
|
||||
| `secrets` | Same for secrets |
|
||||
| `context` | Run clipboard (nested) |
|
||||
| `data` | This step’s input (nested; numeric segments index arrays) |
|
||||
|
||||
A string that is exactly one `{{ path }}` keeps the native type (object/array/number/boolean). Mixed strings concatenate as text. Bare name fields such as `passwordSecret: gmail_app_password` stay names for `$secrets.get` — do not wrap them in `{{ secrets.… }}`. Script APIs `$vars.get` / `$secrets.get` are unchanged.
|
||||
|
||||
YAML **SET** evaluates JSONata against the full `ctx`; the result is `output` (the next step’s data). `jsonata.js` does the same.
|
||||
|
||||
DAG `needs` assemble this step’s `data` from upstream **outputs**. Independent steps in the same wave share a context snapshot; sibling writes to the same context key fail the run.
|
||||
@@ -105,8 +126,9 @@ Optional `script.meta.reads = "ctx"` documents expression hosts. `meta.input` /
|
||||
| `pnpm dev:server` | Monolith API/runner only |
|
||||
| `pnpm dev:web` | UI only (proxies `/api` → :8700, `/ops` → :8600) |
|
||||
| `pnpm build` | Production UI build |
|
||||
| `pnpm start` | Monolith: API + worker + built UI |
|
||||
| `pnpm start` | Monolith: API + worker + built UI (serves `dist` on :8700) |
|
||||
| `pnpm start:control` | Control plane only (migrates, manages PM2 children) |
|
||||
| `pnpm start:web` | Production UI on :8500 (`dist` + proxies to control/HTTP) |
|
||||
| `pnpm start:api` | HTTP API + cron enqueue (`JFLOW_ROLE=api`) |
|
||||
| `pnpm start:worker` | BullMQ worker only |
|
||||
| `pnpm migrate` | Apply SQLite migrations |
|
||||
@@ -123,7 +145,7 @@ Admin UI route **Ops** (`/ops`) talks to the control process.
|
||||
| Drain restart | Pause → wait active=0 → stop children → migrate → recreate → resume |
|
||||
| Force restart | Same without waiting (interrupts active runs; orphans marked `worker_lost`) |
|
||||
|
||||
Desired state is stored in `packages/server/data/control-state.json` (generation, worker count, restart-needed). Plugin installs (later) bump generation and set restart-needed; you apply with Drain restart.
|
||||
Desired state is stored in `data/control-state.json` (generation, worker count, restart-needed). Plugin installs (later) bump generation and set restart-needed; you apply with Drain restart.
|
||||
|
||||
## Environment
|
||||
|
||||
@@ -131,8 +153,10 @@ Desired state is stored in `packages/server/data/control-state.json` (generation
|
||||
|---|---|---|
|
||||
| `JFLOW_JWT_SECRET` | `jflow-dev-secret` (dev only) | **Required in production**. |
|
||||
| `JFLOW_SECRETS_KEY` | `jflow-dev-secrets-key` (dev only) | Master key for named secrets. **Required in production**. Changing it makes existing secrets unreadable. 64 hex chars are used as a raw AES-256 key; any other string is derived with scrypt. |
|
||||
| `JFLOW_DB_PATH` | `packages/server/data/jerapah-flow.db` | SQLite file. |
|
||||
| `JFLOW_WORKFLOWS_DIR` | `packages/server/data/workflows` | Live workflow YAML (instance data). |
|
||||
| `JFLOW_DATA_DIR` | `data/` | Instance data root (SQLite, workflows, control-state, backups, trash). Falls back to `packages/server/data` if that tree still has the db or workflows. |
|
||||
| `JFLOW_DB_PATH` | `data/jerapah-flow.db` | SQLite file. |
|
||||
| `JFLOW_WORKFLOWS_DIR` | `data/workflows` | Live workflow YAML (instance data). |
|
||||
| `JFLOW_LOGS_DIR` | `logs/` | Rolling process logs. |
|
||||
| `REDIS_URL` | `redis://127.0.0.1:6379` | Redis for BullMQ workflow queue. **Required** — the server will not start if Redis is unreachable. |
|
||||
| `REDIS_PASS` | — | Optional Redis AUTH password (sent via ioredis `password`). Prefer this over embedding credentials in `REDIS_URL` so logs stay clean. |
|
||||
| `JFLOW_QUEUE_NAME` | `jerapah-workflows` | BullMQ queue name. |
|
||||
@@ -140,24 +164,45 @@ Desired state is stored in `packages/server/data/control-state.json` (generation
|
||||
| `JFLOW_ROLE` | `all` | `all` (HTTP + cron + worker), `api`, or `worker`. Prefer `pnpm start:api` / `start:worker` under control. |
|
||||
| `JFLOW_CONFIG_GENERATION` | `1` | Set by control/PM2 so children report config generation in heartbeats. |
|
||||
| `JFLOW_CONTROL_PORT` | `8600` | Control ops API port. |
|
||||
| `JFLOW_UI_PORT` | `8500` | Production UI server (`web-server.js`) port. |
|
||||
| `JFLOW_HTTP_PORT` | `8700` | HTTP API port (PM2 children / UI proxy target). |
|
||||
| `JFLOW_LOG_LEVEL` | `debug` | Pino level |
|
||||
| `JFLOW_RETENTION_DAYS` | `30` | Run history prune |
|
||||
| `JFLOW_CORS_ORIGIN` | `http://localhost:8500` | Vite origin in dev |
|
||||
| `PORT` | `8700` | HTTP API port |
|
||||
| `NODE_ENV` | — | Set `production` for secure cookies |
|
||||
| `JFLOW_CORS_ORIGIN` | `http://localhost:8500` | Browser origin (Vite in dev, UI server in prod) |
|
||||
| `PORT` | `8700` | HTTP API port (alias; prefer `JFLOW_HTTP_PORT` under control) |
|
||||
| `NODE_ENV` | — | Set `production` for secure cookies (unless overridden) |
|
||||
| `COOKIE_SECURE` | (from `NODE_ENV`) | `true`/`false` — force Secure cookie flag. Use `false` for plain HTTP LAN access (`http://192.168.x.x`) |
|
||||
|
||||
Workflow runs are **queued** via BullMQ. HTTP and manual triggers return `202 { runId, status: "queued" }` immediately; poll `GET /api/runs/:id` for progress (`queued` → `running` → `success` \| `failed`). Cron remains an in-process producer that enqueues jobs on each tick.
|
||||
|
||||
## Production
|
||||
|
||||
Control-plane topology (same ports as `pnpm dev:pm2`):
|
||||
|
||||
| Process | Port | Role |
|
||||
|---|---|---|
|
||||
| `jflow-web` | **8500** | Built UI + proxies `/api` → :8700, `/ops` + `/api/auth` → :8600 |
|
||||
| `jflow-control` | **8600** | Migrations, Ops API, starts/stops PM2 HTTP + workers |
|
||||
| `jflow-http` | **8700** | API + cron enqueue (managed by control) |
|
||||
| `jflow-worker` | — | BullMQ workers (managed by control) |
|
||||
|
||||
```bash
|
||||
pnpm install
|
||||
pnpm build
|
||||
# Redis must be reachable at REDIS_URL (set REDIS_PASS if Redis requires AUTH)
|
||||
# Recommended: run control (migrates + manages PM2 HTTP/workers)
|
||||
JFLOW_JWT_SECRET=... JFLOW_SECRETS_KEY=... REDIS_URL=redis://127.0.0.1:6379 REDIS_PASS=... NODE_ENV=production pnpm start:control
|
||||
# Or monolith (dev-style):
|
||||
# ... pnpm start
|
||||
# Put secrets in .env (JFLOW_JWT_SECRET, JFLOW_SECRETS_KEY, REDIS_URL, …)
|
||||
# Use in-tree PM2 6.x (same module control.js requires). A global `pm2` 7.x
|
||||
# against a 6.x daemon pegs CPU even when ls shows only 2 fork instances.
|
||||
pnpm start:pm2
|
||||
# UI: http://localhost:8500
|
||||
# If you already mixed versions: pnpm pm2 -- kill && pnpm start:pm2
|
||||
```
|
||||
|
||||
With control, serve the built UI from Vite preview, a reverse proxy, or set `JFLOW_SERVE_UI=1` on the HTTP process.
|
||||
Or without the ecosystem file:
|
||||
|
||||
```bash
|
||||
NODE_ENV=production pnpm start:control # :8600 + PM2 children
|
||||
NODE_ENV=production pnpm start:web # :8500
|
||||
```
|
||||
|
||||
Monolith (no Ops stop/scale): `pnpm build && pnpm start` serves the UI from the API process on :8700. Optional `JFLOW_SERVE_UI=1` on `start:api` does the same when you run HTTP alone — do **not** use that under control-plane mode (stopping HTTP would take down the UI).
|
||||
|
||||
+36
-9
@@ -1,7 +1,12 @@
|
||||
/**
|
||||
* Production PM2 ecosystem (monolith runner).
|
||||
* Use for deployed/single-process starts. For local multi-process Ops UI, use
|
||||
* `pnpm dev:pm2` → packages/server/ecosystem.dev.cjs / control.js instead.
|
||||
* Production PM2 ecosystem (control plane + UI).
|
||||
* Starts always-on processes only; HTTP (:8700) and workers are owned by
|
||||
* control.js via PM2 (same as `pnpm dev:pm2`).
|
||||
*
|
||||
* Use `pnpm start:pm2` (in-tree PM2 6.x). Do not use a global `pm2` 7.x —
|
||||
* a CLI/daemon version mismatch pegs CPU even with instances: 1.
|
||||
*
|
||||
* Prerequisites: `pnpm build` (packages/web/dist), Redis, .env secrets.
|
||||
*/
|
||||
const fs = require("fs");
|
||||
const path = require("path");
|
||||
@@ -28,21 +33,43 @@ function loadEnv(file) {
|
||||
}
|
||||
|
||||
const root = __dirname;
|
||||
const env = {
|
||||
NODE_ENV: "production",
|
||||
...loadEnv(path.join(root, ".env")),
|
||||
};
|
||||
|
||||
module.exports = {
|
||||
apps: [
|
||||
{
|
||||
name: "jerapah-flow",
|
||||
name: "jflow-control",
|
||||
cwd: root,
|
||||
script: "packages/server/runner.js",
|
||||
interpreter: "node",
|
||||
script: "packages/server/control.js",
|
||||
interpreter: process.execPath,
|
||||
instances: 1,
|
||||
exec_mode: "fork",
|
||||
autorestart: true,
|
||||
max_restarts: 8,
|
||||
restart_delay: 2000,
|
||||
env: {
|
||||
...env,
|
||||
JFLOW_CONTROL_PORT: env.JFLOW_CONTROL_PORT ?? "8600",
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "jflow-web",
|
||||
cwd: root,
|
||||
script: "packages/server/web-server.js",
|
||||
interpreter: process.execPath,
|
||||
instances: 1,
|
||||
exec_mode: "fork",
|
||||
autorestart: true,
|
||||
max_restarts: 20,
|
||||
env: {
|
||||
NODE_ENV: "production",
|
||||
...loadEnv(path.join(root, ".env")),
|
||||
...env,
|
||||
JFLOW_UI_PORT: env.JFLOW_UI_PORT ?? "8500",
|
||||
JFLOW_CONTROL_PORT: env.JFLOW_CONTROL_PORT ?? "8600",
|
||||
JFLOW_HTTP_PORT: env.JFLOW_HTTP_PORT ?? "8700",
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
};
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
name: Comic - monkeyuser to ntfy
|
||||
description: |
|
||||
Scrape the latest MonkeyUser comic and send it to ntfy (requires $VAR_ntfy_channel).
|
||||
Scrape the latest MonkeyUser comic and send it to ntfy (requires {{ vars.ntfy_channel }}).
|
||||
scripts:
|
||||
- script: fetch-html.js
|
||||
config:
|
||||
@@ -20,7 +20,7 @@ scripts:
|
||||
- script: fetch-binary.js
|
||||
- script: ntfy.js
|
||||
config:
|
||||
url: $VAR_ntfy_channel
|
||||
url: "{{ vars.ntfy_channel }}"
|
||||
triggers:
|
||||
- type: HTTP
|
||||
method: POST
|
||||
|
||||
@@ -14,6 +14,9 @@
|
||||
"start:api": "pnpm --filter @jerapah-flow/server start:api",
|
||||
"start:worker": "pnpm --filter @jerapah-flow/server start:worker",
|
||||
"start:control": "pnpm --filter @jerapah-flow/server start:control",
|
||||
"start:web": "pnpm --filter @jerapah-flow/server start:web",
|
||||
"pm2": "node scripts/pm2.mjs",
|
||||
"start:pm2": "node scripts/pm2.mjs start ecosystem.config.cjs",
|
||||
"migrate": "pnpm --filter @jerapah-flow/server migrate",
|
||||
"test": "pnpm --filter @jerapah-flow/shared test && pnpm --filter @jerapah-flow/web test",
|
||||
"lint": "pnpm --filter @jerapah-flow/web lint"
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
import fs from "fs";
|
||||
import path from "path";
|
||||
import { SERVER_ROOT } from "./paths.js";
|
||||
import { REPO_ROOT } from "./paths.js";
|
||||
|
||||
const ROOT_PKG = path.resolve(SERVER_ROOT, "../../package.json");
|
||||
const ROOT_PKG = path.join(REPO_ROOT, "package.json");
|
||||
|
||||
/**
|
||||
* JerapahFlow app version from the monorepo root package.json.
|
||||
|
||||
+156
-72
@@ -3,36 +3,24 @@ import { assertSecretName, getSecretPlaintext } from "./secrets-store.js";
|
||||
import { isSecret } from "./secret-value.js";
|
||||
import { assertVariableName, getVariablePlain } from "./variables-store.js";
|
||||
|
||||
const PREFIXES = [
|
||||
{ kind: "context", prefix: "$CONTEXT_" },
|
||||
{ kind: "secret", prefix: "$SECRET_" },
|
||||
{ kind: "var", prefix: "$VAR_" },
|
||||
];
|
||||
const FORBIDDEN_SEGMENTS = new Set(["__proto__", "constructor", "prototype"]);
|
||||
const MUSTACHE_TOKEN_RE =
|
||||
/\{\{\s*([A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z0-9_]+)*)\s*\}\}/g;
|
||||
const WHOLE_MUSTACHE_RE =
|
||||
/^\{\{\s*([A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z0-9_]+)*)\s*\}\}$/;
|
||||
|
||||
/**
|
||||
* @typedef {{ kind: "secret" | "context" | "var", name: string, raw: string }} ConfigRef
|
||||
* @typedef {{ owner: string, workflowKey: string, context?: unknown }} ConfigRefCtx
|
||||
* @typedef {{
|
||||
* owner: string,
|
||||
* workflowKey?: string,
|
||||
* context?: unknown,
|
||||
* data?: unknown,
|
||||
* }} ConfigRefCtx
|
||||
*/
|
||||
|
||||
/**
|
||||
* Parse a whole-value config placeholder. Returns null for literals.
|
||||
* @param {unknown} value
|
||||
* @returns {ConfigRef | null}
|
||||
*/
|
||||
export function parseConfigRef(value) {
|
||||
if (typeof value !== "string") return null;
|
||||
const trimmed = value.trim();
|
||||
for (const { kind, prefix } of PREFIXES) {
|
||||
if (trimmed.startsWith(prefix)) {
|
||||
return { kind, name: trimmed.slice(prefix.length), raw: trimmed };
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Walk config (objects/arrays) and replace whole-value `$SECRET_` / `$CONTEXT_` / `$VAR_`
|
||||
* strings. Does not walk trigger data.
|
||||
* Walk config (objects/arrays) and interpolate `{{ path }}` strings.
|
||||
* Does not walk trigger data.
|
||||
*
|
||||
* @param {unknown} value
|
||||
* @param {ConfigRefCtx} ctx
|
||||
@@ -68,83 +56,179 @@ export async function resolveConfigRefs(value, ctx, seen = new WeakSet()) {
|
||||
/**
|
||||
* @param {string} value
|
||||
* @param {ConfigRefCtx} ctx
|
||||
* @returns {Promise<string | number | boolean>}
|
||||
* @returns {Promise<unknown>}
|
||||
*/
|
||||
async function resolveStringRef(value, ctx) {
|
||||
const ref = parseConfigRef(value);
|
||||
if (!ref) return value;
|
||||
const whole = WHOLE_MUSTACHE_RE.exec(value);
|
||||
if (whole && whole[0] === value) {
|
||||
return resolvePath(whole[1], ctx, { raw: value, allowObject: true });
|
||||
}
|
||||
|
||||
if (ref.kind === "secret") {
|
||||
return resolveSecretRef(ref, ctx);
|
||||
if (!value.includes("{{")) {
|
||||
return value;
|
||||
}
|
||||
if (ref.kind === "var") {
|
||||
return resolveVarRef(ref, ctx);
|
||||
|
||||
MUSTACHE_TOKEN_RE.lastIndex = 0;
|
||||
let out = "";
|
||||
let lastIndex = 0;
|
||||
let match;
|
||||
while ((match = MUSTACHE_TOKEN_RE.exec(value)) != null) {
|
||||
out += value.slice(lastIndex, match.index);
|
||||
const resolved = await resolvePath(match[1], ctx, {
|
||||
raw: match[0],
|
||||
allowObject: false,
|
||||
});
|
||||
out += stringifyScalar(resolved, match[0]);
|
||||
lastIndex = match.index + match[0].length;
|
||||
}
|
||||
return resolveContextRef(ref, ctx);
|
||||
out += value.slice(lastIndex);
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {ConfigRef} ref
|
||||
* @param {string} pathExpr
|
||||
* @param {ConfigRefCtx} ctx
|
||||
* @returns {Promise<string>}
|
||||
* @param {{ raw: string, allowObject: boolean }} opts
|
||||
*/
|
||||
async function resolveSecretRef(ref, ctx) {
|
||||
try {
|
||||
assertSecretName(ref.name);
|
||||
} catch {
|
||||
throw new Error(`config ref ${ref.raw}: invalid secret name`);
|
||||
async function resolvePath(pathExpr, ctx, opts) {
|
||||
const segments = pathExpr.split(".");
|
||||
if (segments.length === 0 || segments.some((s) => !s)) {
|
||||
throw new Error(`config ref ${opts.raw}: empty path`);
|
||||
}
|
||||
const plaintext = await getSecretPlaintext(ctx.owner, ref.name);
|
||||
if (plaintext == null) {
|
||||
throw new Error(`config ref ${ref.raw}: secret "${ref.name}" not found`);
|
||||
for (const seg of segments) {
|
||||
if (FORBIDDEN_SEGMENTS.has(seg)) {
|
||||
throw new Error(`config ref ${opts.raw}: forbidden path segment "${seg}"`);
|
||||
}
|
||||
}
|
||||
return plaintext;
|
||||
|
||||
const root = segments[0];
|
||||
const rest = segments.slice(1);
|
||||
|
||||
if (root === "vars") {
|
||||
return resolveNamedStore("var", rest, ctx, opts);
|
||||
}
|
||||
if (root === "secrets") {
|
||||
return resolveNamedStore("secret", rest, ctx, opts);
|
||||
}
|
||||
if (root === "context") {
|
||||
return walkObject(ctx.context, rest, opts);
|
||||
}
|
||||
if (root === "data") {
|
||||
return walkObject(ctx.data, rest, opts);
|
||||
}
|
||||
throw new Error(
|
||||
`config ref ${opts.raw}: unknown root "${root}" (use vars, secrets, context, or data)`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {ConfigRef} ref
|
||||
* @param {"var" | "secret"} kind
|
||||
* @param {string[]} rest
|
||||
* @param {ConfigRefCtx} ctx
|
||||
* @returns {Promise<string | number | boolean>}
|
||||
* @param {{ raw: string, allowObject: boolean }} opts
|
||||
*/
|
||||
async function resolveVarRef(ref, ctx) {
|
||||
if (ref.name.length === 0) {
|
||||
throw new Error(`config ref ${ref.raw}: empty variable name`);
|
||||
async function resolveNamedStore(kind, rest, ctx, opts) {
|
||||
if (rest.length === 0) {
|
||||
throw new Error(`config ref ${opts.raw}: empty ${kind} name`);
|
||||
}
|
||||
const name = rest.join(".");
|
||||
try {
|
||||
assertVariableName(ref.name);
|
||||
if (kind === "secret") assertSecretName(name);
|
||||
else assertVariableName(name);
|
||||
} catch {
|
||||
throw new Error(`config ref ${ref.raw}: invalid variable name`);
|
||||
throw new Error(`config ref ${opts.raw}: invalid ${kind} name`);
|
||||
}
|
||||
const value = await getVariablePlain(ctx.owner, ref.name);
|
||||
|
||||
if (kind === "secret") {
|
||||
const plaintext = await getSecretPlaintext(ctx.owner, name);
|
||||
if (plaintext == null) {
|
||||
throw new Error(`config ref ${opts.raw}: secret "${name}" not found`);
|
||||
}
|
||||
return plaintext;
|
||||
}
|
||||
|
||||
const value = await getVariablePlain(ctx.owner, name);
|
||||
if (value == null) {
|
||||
throw new Error(`config ref ${ref.raw}: variable "${ref.name}" not found`);
|
||||
throw new Error(`config ref ${opts.raw}: variable "${name}" not found`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {ConfigRef} ref
|
||||
* @param {ConfigRefCtx} ctx
|
||||
* @returns {string}
|
||||
* @param {unknown} root
|
||||
* @param {string[]} rest
|
||||
* @param {{ raw: string, allowObject: boolean }} opts
|
||||
*/
|
||||
function resolveContextRef(ref, ctx) {
|
||||
if (ref.name.length === 0) {
|
||||
throw new Error(`config ref ${ref.raw}: empty context key`);
|
||||
function walkObject(root, rest, opts) {
|
||||
if (rest.length === 0) {
|
||||
return unwrapValue(root, opts);
|
||||
}
|
||||
const bag =
|
||||
ctx.context != null && typeof ctx.context === "object" && !Array.isArray(ctx.context)
|
||||
? /** @type {Record<string, unknown>} */ (ctx.context)
|
||||
: {};
|
||||
if (!Object.prototype.hasOwnProperty.call(bag, ref.name)) {
|
||||
throw new Error(`config ref ${ref.raw}: context "${ref.name}" not found`);
|
||||
|
||||
let cur = root;
|
||||
for (const seg of rest) {
|
||||
if (cur == null || typeof cur !== "object") {
|
||||
throw new Error(`config ref ${opts.raw}: path not found`);
|
||||
}
|
||||
if (Array.isArray(cur)) {
|
||||
if (!/^\d+$/.test(seg)) {
|
||||
throw new Error(`config ref ${opts.raw}: path not found`);
|
||||
}
|
||||
const idx = Number(seg);
|
||||
if (!Number.isInteger(idx) || idx < 0 || idx >= cur.length) {
|
||||
throw new Error(`config ref ${opts.raw}: path not found`);
|
||||
}
|
||||
cur = cur[idx];
|
||||
continue;
|
||||
}
|
||||
const bag = /** @type {Record<string, unknown>} */ (cur);
|
||||
if (!Object.prototype.hasOwnProperty.call(bag, seg)) {
|
||||
throw new Error(`config ref ${opts.raw}: path not found`);
|
||||
}
|
||||
cur = bag[seg];
|
||||
}
|
||||
const raw = bag[ref.name];
|
||||
if (isSecret(raw)) {
|
||||
return raw.reveal();
|
||||
return unwrapValue(cur, opts);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {unknown} value
|
||||
* @param {{ raw: string, allowObject: boolean }} opts
|
||||
*/
|
||||
function unwrapValue(value, opts) {
|
||||
if (isSecret(value)) {
|
||||
return value.reveal();
|
||||
}
|
||||
const coerced = coerceCredentialString(raw);
|
||||
if (coerced == null) {
|
||||
throw new Error(`config ref ${ref.raw}: context "${ref.name}" is not a scalar`);
|
||||
if (!opts.allowObject && value != null && typeof value === "object") {
|
||||
throw new Error(`config ref ${opts.raw}: value is not a scalar`);
|
||||
}
|
||||
return coerced;
|
||||
// Whole-value context/data may be any JSON type; mixed strings need scalars only.
|
||||
if (opts.allowObject) {
|
||||
if (value != null && typeof value === "object") return value;
|
||||
if (
|
||||
typeof value === "string" ||
|
||||
typeof value === "number" ||
|
||||
typeof value === "boolean"
|
||||
) {
|
||||
return value;
|
||||
}
|
||||
// Prefer credential coercion for odd primitives (e.g. bigint) when whole-value.
|
||||
const coerced = coerceCredentialString(value);
|
||||
if (coerced != null) return coerced;
|
||||
return value;
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {unknown} value
|
||||
* @param {string} raw
|
||||
*/
|
||||
function stringifyScalar(value, raw) {
|
||||
if (value == null) {
|
||||
throw new Error(`config ref ${raw}: value is null`);
|
||||
}
|
||||
if (typeof value === "string") return value;
|
||||
if (typeof value === "number" || typeof value === "boolean") {
|
||||
return String(value);
|
||||
}
|
||||
throw new Error(`config ref ${raw}: value is not a scalar`);
|
||||
}
|
||||
|
||||
+21
-18
@@ -64,13 +64,6 @@ try {
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
try {
|
||||
await connectPm2();
|
||||
} catch (err) {
|
||||
log.error({ err }, "failed to connect to PM2 — is pm2 installed?");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
async function applyDesiredState() {
|
||||
const state = readControlState();
|
||||
await ensureHttp({
|
||||
@@ -98,8 +91,6 @@ async function applyDesiredState() {
|
||||
);
|
||||
}
|
||||
|
||||
await applyDesiredState();
|
||||
|
||||
const server = fastify({ loggerInstance: log });
|
||||
await server.register(cookie);
|
||||
await server.register(jwt, {
|
||||
@@ -484,12 +475,24 @@ process.on("SIGINT", shutdown);
|
||||
process.on("SIGTERM", shutdown);
|
||||
|
||||
const port = Number(process.env.JFLOW_CONTROL_PORT ?? process.env.PORT ?? 8600);
|
||||
server
|
||||
.listen({ host: "0.0.0.0", port })
|
||||
.then(() => {
|
||||
log.info(`Control is running on port ${port}`);
|
||||
})
|
||||
.catch((err) => {
|
||||
log.error({ err }, "failed to start control");
|
||||
process.exit(1);
|
||||
});
|
||||
try {
|
||||
await server.listen({ host: "0.0.0.0", port });
|
||||
log.info(`Control is running on port ${port}`);
|
||||
} catch (err) {
|
||||
log.error({ err }, "failed to start control");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
try {
|
||||
await connectPm2();
|
||||
} catch (err) {
|
||||
log.error({ err }, "failed to connect to PM2 — is pm2 installed?");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
try {
|
||||
await applyDesiredState();
|
||||
} catch (err) {
|
||||
log.error({ err }, "failed to apply desired state");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
@@ -1,24 +0,0 @@
|
||||
/**
|
||||
* @param {import("knex").Knex} knex
|
||||
*/
|
||||
export async function up(knex) {
|
||||
await knex.schema.alterTable("workflow_runs", (t) => {
|
||||
t.text("job_id");
|
||||
t.text("queued_at");
|
||||
});
|
||||
|
||||
await knex.schema.raw(
|
||||
"CREATE INDEX workflow_runs_job_id_idx ON workflow_runs (job_id)",
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {import("knex").Knex} knex
|
||||
*/
|
||||
export async function down(knex) {
|
||||
await knex.schema.raw("DROP INDEX IF EXISTS workflow_runs_job_id_idx");
|
||||
await knex.schema.alterTable("workflow_runs", (t) => {
|
||||
t.dropColumn("job_id");
|
||||
t.dropColumn("queued_at");
|
||||
});
|
||||
}
|
||||
@@ -1,21 +0,0 @@
|
||||
/**
|
||||
* @param {import("knex").Knex} knex
|
||||
*/
|
||||
export async function up(knex) {
|
||||
await knex.schema.alterTable("workflow_runs", (t) => {
|
||||
t.integer("workflow_revision");
|
||||
});
|
||||
await knex.schema.raw(
|
||||
"CREATE INDEX workflow_runs_workflow_revision_idx ON workflow_runs (workflow, workflow_revision)",
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {import("knex").Knex} knex
|
||||
*/
|
||||
export async function down(knex) {
|
||||
await knex.schema.raw("DROP INDEX IF EXISTS workflow_runs_workflow_revision_idx");
|
||||
await knex.schema.alterTable("workflow_runs", (t) => {
|
||||
t.dropColumn("workflow_revision");
|
||||
});
|
||||
}
|
||||
+5
-38
@@ -17,6 +17,9 @@ export async function up(knex) {
|
||||
t.text("output");
|
||||
t.text("error");
|
||||
t.text("parent_run_id").references("id").inTable("workflow_runs");
|
||||
t.text("job_id");
|
||||
t.text("queued_at");
|
||||
t.integer("workflow_revision");
|
||||
});
|
||||
|
||||
await knex.schema.raw(
|
||||
@@ -28,45 +31,11 @@ export async function up(knex) {
|
||||
await knex.schema.raw(
|
||||
"CREATE INDEX workflow_runs_status_started_at_idx ON workflow_runs (status, started_at DESC)",
|
||||
);
|
||||
|
||||
await knex.schema.createTable("step_runs", (t) => {
|
||||
t.text("id").primary();
|
||||
t.text("run_id")
|
||||
.notNullable()
|
||||
.references("id")
|
||||
.inTable("workflow_runs")
|
||||
.onDelete("CASCADE");
|
||||
t.integer("step_index").notNullable();
|
||||
t.text("script").notNullable();
|
||||
t.text("config");
|
||||
t.text("status").notNullable();
|
||||
t.text("started_at").notNullable();
|
||||
t.text("finished_at");
|
||||
t.integer("duration_ms");
|
||||
t.text("output");
|
||||
t.text("error");
|
||||
});
|
||||
|
||||
await knex.schema.raw(
|
||||
"CREATE INDEX step_runs_run_id_step_index_idx ON step_runs (run_id, step_index)",
|
||||
"CREATE INDEX workflow_runs_job_id_idx ON workflow_runs (job_id)",
|
||||
);
|
||||
|
||||
await knex.schema.createTable("logs", (t) => {
|
||||
t.increments("id").primary();
|
||||
t.text("run_id")
|
||||
.notNullable()
|
||||
.references("id")
|
||||
.inTable("workflow_runs")
|
||||
.onDelete("CASCADE");
|
||||
t.text("step_id");
|
||||
t.text("ts").notNullable();
|
||||
t.integer("level").notNullable();
|
||||
t.text("msg");
|
||||
t.text("payload");
|
||||
});
|
||||
|
||||
await knex.schema.raw(
|
||||
"CREATE INDEX logs_run_id_ts_idx ON logs (run_id, ts)",
|
||||
"CREATE INDEX workflow_runs_workflow_revision_idx ON workflow_runs (workflow, workflow_revision)",
|
||||
);
|
||||
}
|
||||
|
||||
@@ -74,7 +43,5 @@ export async function up(knex) {
|
||||
* @param {import("knex").Knex} knex
|
||||
*/
|
||||
export async function down(knex) {
|
||||
await knex.schema.dropTableIfExists("logs");
|
||||
await knex.schema.dropTableIfExists("step_runs");
|
||||
await knex.schema.dropTableIfExists("workflow_runs");
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
/**
|
||||
* @param {import("knex").Knex} knex
|
||||
*/
|
||||
export async function up(knex) {
|
||||
await knex.schema.createTable("step_runs", (t) => {
|
||||
t.text("id").primary();
|
||||
t.text("run_id")
|
||||
.notNullable()
|
||||
.references("id")
|
||||
.inTable("workflow_runs")
|
||||
.onDelete("CASCADE");
|
||||
t.integer("step_index").notNullable();
|
||||
t.text("script").notNullable();
|
||||
t.text("config");
|
||||
t.text("status").notNullable();
|
||||
t.text("started_at").notNullable();
|
||||
t.text("finished_at");
|
||||
t.integer("duration_ms");
|
||||
t.text("output");
|
||||
t.text("error");
|
||||
});
|
||||
|
||||
await knex.schema.raw(
|
||||
"CREATE INDEX step_runs_run_id_step_index_idx ON step_runs (run_id, step_index)",
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {import("knex").Knex} knex
|
||||
*/
|
||||
export async function down(knex) {
|
||||
await knex.schema.dropTableIfExists("step_runs");
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
/**
|
||||
* @param {import("knex").Knex} knex
|
||||
*/
|
||||
export async function up(knex) {
|
||||
await knex.schema.createTable("logs", (t) => {
|
||||
t.increments("id").primary();
|
||||
t.text("run_id")
|
||||
.notNullable()
|
||||
.references("id")
|
||||
.inTable("workflow_runs")
|
||||
.onDelete("CASCADE");
|
||||
t.text("step_id");
|
||||
t.text("ts").notNullable();
|
||||
t.integer("level").notNullable();
|
||||
t.text("msg");
|
||||
t.text("payload");
|
||||
});
|
||||
|
||||
await knex.schema.raw(
|
||||
"CREATE INDEX logs_run_id_ts_idx ON logs (run_id, ts)",
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {import("knex").Knex} knex
|
||||
*/
|
||||
export async function down(knex) {
|
||||
await knex.schema.dropTableIfExists("logs");
|
||||
}
|
||||
+20
-20
@@ -29,35 +29,35 @@ const DEFAULT_EMAIL_TEMPLATE = `<!DOCTYPE html>
|
||||
* @param {import("knex").Knex} knex
|
||||
*/
|
||||
export async function up(knex) {
|
||||
await knex.schema.alterTable("http_pages", (t) => {
|
||||
await knex.schema.createTable("http_pages", (t) => {
|
||||
t.text("id").primary();
|
||||
t.text("name").notNullable().unique();
|
||||
t.text("content").notNullable();
|
||||
t.text("mime").notNullable();
|
||||
t.integer("status").notNullable().defaultTo(200);
|
||||
t.text("kind").notNullable().defaultTo("response");
|
||||
t.integer("system").notNullable().defaultTo(0);
|
||||
t.text("created_at").notNullable();
|
||||
t.text("updated_at").notNullable();
|
||||
});
|
||||
|
||||
const now = new Date().toISOString();
|
||||
const existing = await knex("http_pages").where({ name: "email-default" }).first();
|
||||
if (!existing) {
|
||||
await knex("http_pages").insert({
|
||||
id: "00000000-0000-4000-8000-000000000001",
|
||||
name: "email-default",
|
||||
content: DEFAULT_EMAIL_TEMPLATE,
|
||||
mime: "html",
|
||||
status: 200,
|
||||
kind: "template",
|
||||
system: 1,
|
||||
created_at: now,
|
||||
updated_at: now,
|
||||
});
|
||||
}
|
||||
await knex("http_pages").insert({
|
||||
id: "00000000-0000-4000-8000-000000000001",
|
||||
name: "email-default",
|
||||
content: DEFAULT_EMAIL_TEMPLATE,
|
||||
mime: "html",
|
||||
status: 200,
|
||||
kind: "template",
|
||||
system: 1,
|
||||
created_at: now,
|
||||
updated_at: now,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {import("knex").Knex} knex
|
||||
*/
|
||||
export async function down(knex) {
|
||||
await knex("http_pages").where({ name: "email-default", system: 1 }).del();
|
||||
await knex.schema.alterTable("http_pages", (t) => {
|
||||
t.dropColumn("kind");
|
||||
t.dropColumn("system");
|
||||
});
|
||||
await knex.schema.dropTableIfExists("http_pages");
|
||||
}
|
||||
+2
-13
@@ -2,21 +2,11 @@
|
||||
* @param {import("knex").Knex} knex
|
||||
*/
|
||||
export async function up(knex) {
|
||||
await knex.schema.createTable("http_pages", (t) => {
|
||||
t.text("id").primary();
|
||||
t.text("name").notNullable().unique();
|
||||
t.text("content").notNullable();
|
||||
t.text("mime").notNullable(); // html | json
|
||||
t.integer("status").notNullable().defaultTo(200);
|
||||
t.text("created_at").notNullable();
|
||||
t.text("updated_at").notNullable();
|
||||
});
|
||||
|
||||
await knex.schema.createTable("http_auths", (t) => {
|
||||
t.text("id").primary();
|
||||
t.text("name").notNullable().unique();
|
||||
t.text("type").notNullable(); // bearer | basic | header
|
||||
t.text("config").notNullable(); // JSON
|
||||
t.text("type").notNullable();
|
||||
t.text("config").notNullable();
|
||||
t.integer("unauthorized_status").nullable();
|
||||
t.text("unauthorized_response").nullable();
|
||||
t.text("created_at").notNullable();
|
||||
@@ -29,5 +19,4 @@ export async function up(knex) {
|
||||
*/
|
||||
export async function down(knex) {
|
||||
await knex.schema.dropTableIfExists("http_auths");
|
||||
await knex.schema.dropTableIfExists("http_pages");
|
||||
}
|
||||
-18
@@ -21,29 +21,11 @@ export async function up(knex) {
|
||||
await knex.schema.raw(
|
||||
"CREATE INDEX workflow_revisions_workflow_id_created_at_idx ON workflow_revisions (workflow_id, created_at DESC)",
|
||||
);
|
||||
|
||||
await knex.schema.createTable("workflow_trash", (t) => {
|
||||
t.text("id").primary();
|
||||
t.text("workflow_id").notNullable();
|
||||
t.text("owner").notNullable();
|
||||
t.text("file").notNullable();
|
||||
t.text("name");
|
||||
t.text("deleted_at").notNullable();
|
||||
t.text("trash_path").notNullable();
|
||||
});
|
||||
|
||||
await knex.schema.raw(
|
||||
"CREATE INDEX workflow_trash_deleted_at_idx ON workflow_trash (deleted_at ASC)",
|
||||
);
|
||||
await knex.schema.raw(
|
||||
"CREATE UNIQUE INDEX workflow_trash_owner_file_idx ON workflow_trash (owner, file)",
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {import("knex").Knex} knex
|
||||
*/
|
||||
export async function down(knex) {
|
||||
await knex.schema.dropTableIfExists("workflow_trash");
|
||||
await knex.schema.dropTableIfExists("workflow_revisions");
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
/**
|
||||
* @param {import("knex").Knex} knex
|
||||
*/
|
||||
export async function up(knex) {
|
||||
await knex.schema.createTable("workflow_trash", (t) => {
|
||||
t.text("id").primary();
|
||||
t.text("workflow_id").notNullable();
|
||||
t.text("owner").notNullable();
|
||||
t.text("file").notNullable();
|
||||
t.text("name");
|
||||
t.text("deleted_at").notNullable();
|
||||
t.text("trash_path").notNullable();
|
||||
});
|
||||
|
||||
await knex.schema.raw(
|
||||
"CREATE INDEX workflow_trash_deleted_at_idx ON workflow_trash (deleted_at ASC)",
|
||||
);
|
||||
await knex.schema.raw(
|
||||
"CREATE UNIQUE INDEX workflow_trash_owner_file_idx ON workflow_trash (owner, file)",
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {import("knex").Knex} knex
|
||||
*/
|
||||
export async function down(knex) {
|
||||
await knex.schema.dropTableIfExists("workflow_trash");
|
||||
}
|
||||
+3
-1
@@ -14,7 +14,9 @@ export async function up(knex) {
|
||||
t.unique(["owner", "name"]);
|
||||
});
|
||||
|
||||
await knex.schema.raw("CREATE INDEX profiles_owner_name_idx ON profiles (owner, name)");
|
||||
await knex.schema.raw(
|
||||
"CREATE INDEX profiles_owner_name_idx ON profiles (owner, name)",
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -0,0 +1,97 @@
|
||||
const HOP_BY_HOP = new Set([
|
||||
"connection",
|
||||
"keep-alive",
|
||||
"proxy-authenticate",
|
||||
"proxy-authorization",
|
||||
"te",
|
||||
"trailer",
|
||||
"transfer-encoding",
|
||||
"upgrade",
|
||||
"host",
|
||||
"content-length",
|
||||
]);
|
||||
|
||||
const REPLY_SKIP = new Set([
|
||||
"connection",
|
||||
"keep-alive",
|
||||
"transfer-encoding",
|
||||
"content-encoding",
|
||||
"content-length",
|
||||
]);
|
||||
|
||||
/**
|
||||
* Control origin used when the UI or HTTP process proxies to control.
|
||||
* @returns {string}
|
||||
*/
|
||||
export function controlOrigin() {
|
||||
const explicit = process.env.JFLOW_CONTROL_URL?.trim();
|
||||
if (explicit) return explicit.replace(/\/$/, "");
|
||||
const port = Number(process.env.JFLOW_CONTROL_PORT ?? 8600);
|
||||
return `http://127.0.0.1:${port}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* HTTP API origin (workflow triggers + REST).
|
||||
* @returns {string}
|
||||
*/
|
||||
export function httpOrigin() {
|
||||
const explicit = process.env.JFLOW_HTTP_URL?.trim();
|
||||
if (explicit) return explicit.replace(/\/$/, "");
|
||||
const port = Number(process.env.JFLOW_HTTP_PORT ?? process.env.PORT ?? 8700);
|
||||
return `http://127.0.0.1:${port}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Forward the incoming request to `origin`, preserving path + query.
|
||||
* @param {import("fastify").FastifyRequest} req
|
||||
* @param {import("fastify").FastifyReply} reply
|
||||
* @param {string} origin
|
||||
* @param {{ unreachableMessage?: string }} [opts]
|
||||
*/
|
||||
export async function proxyToOrigin(req, reply, origin, opts = {}) {
|
||||
const target = `${origin.replace(/\/$/, "")}${req.raw.url ?? "/"}`;
|
||||
const headers = {};
|
||||
for (const [key, value] of Object.entries(req.headers)) {
|
||||
if (value == null || HOP_BY_HOP.has(key.toLowerCase())) continue;
|
||||
headers[key] = Array.isArray(value) ? value.join(", ") : String(value);
|
||||
}
|
||||
|
||||
const method = req.method.toUpperCase();
|
||||
const hasBody = method !== "GET" && method !== "HEAD";
|
||||
let body;
|
||||
if (hasBody) {
|
||||
if (Buffer.isBuffer(req.body)) body = req.body;
|
||||
else if (typeof req.body === "string") body = req.body;
|
||||
else if (req.body != null) {
|
||||
body = JSON.stringify(req.body);
|
||||
if (!headers["content-type"]) headers["content-type"] = "application/json";
|
||||
}
|
||||
}
|
||||
|
||||
let res;
|
||||
try {
|
||||
res = await fetch(target, { method, headers, body });
|
||||
} catch (err) {
|
||||
const message = opts.unreachableMessage ?? "upstream unreachable";
|
||||
req.log.warn({ err, target }, `proxy: ${message}`);
|
||||
return reply.code(502).send({ error: message });
|
||||
}
|
||||
|
||||
reply.code(res.status);
|
||||
res.headers.forEach((value, key) => {
|
||||
if (REPLY_SKIP.has(key.toLowerCase())) return;
|
||||
reply.header(key, value);
|
||||
});
|
||||
return reply.send(Buffer.from(await res.arrayBuffer()));
|
||||
}
|
||||
|
||||
/**
|
||||
* Forward `/ops/*` to the control plane (same-origin UI in production).
|
||||
* @param {import("fastify").FastifyRequest} req
|
||||
* @param {import("fastify").FastifyReply} reply
|
||||
*/
|
||||
export async function proxyOpsToControl(req, reply) {
|
||||
return proxyToOrigin(req, reply, controlOrigin(), {
|
||||
unreachableMessage: "control plane unreachable",
|
||||
});
|
||||
}
|
||||
@@ -11,12 +11,14 @@
|
||||
"start:api": "node server.js",
|
||||
"start:worker": "node worker.js",
|
||||
"start:control": "node control.js",
|
||||
"start:web": "node web-server.js",
|
||||
"migrate": "node -e \"import('./db.js').then((m) => m.migrate().then(() => process.exit(0)))\"",
|
||||
"test:plugins": "JFLOW_PLUGINS_DIR=./data/plugins-smoke-test JFLOW_DB_PATH=./data/plugins-smoke.db node test/plugins-smoke.js",
|
||||
"test:workflow-history": "JFLOW_WORKFLOWS_DIR=./data/workflow-history-smoke JFLOW_DB_PATH=./data/workflow-history-smoke.db node test/workflow-history-smoke.js",
|
||||
"test:plugins": "JFLOW_DATA_DIR=./data JFLOW_PLUGINS_DIR=./data/plugins-smoke-test JFLOW_DB_PATH=./data/plugins-smoke.db node test/plugins-smoke.js",
|
||||
"test:workflow-history": "JFLOW_DATA_DIR=./data JFLOW_WORKFLOWS_DIR=./data/workflow-history-smoke JFLOW_DB_PATH=./data/workflow-history-smoke.db node test/workflow-history-smoke.js",
|
||||
"reset-admin": "node reset-admin.js",
|
||||
"test:profiles": "node test/profiles-smoke.js",
|
||||
"test:set-dry-run": "node test/set-dry-run-smoke.js"
|
||||
"test:set-dry-run": "node test/set-dry-run-smoke.js",
|
||||
"test:config-refs": "node test/config-refs-smoke.js"
|
||||
},
|
||||
"dependencies": {
|
||||
"@jerapah-flow/shared": "workspace:*",
|
||||
@@ -41,7 +43,7 @@
|
||||
"nodemailer": "^9.0.5",
|
||||
"pino": "^10.3.1",
|
||||
"pino-roll": "^4.0.0",
|
||||
"pm2": "^6.0.13",
|
||||
"pm2": "6.0.14",
|
||||
"rss-parser": "^3.13.0",
|
||||
"ssh2-sftp-client": "^12.1.1",
|
||||
"webdav": "^5.10.0",
|
||||
|
||||
+39
-17
@@ -1,27 +1,49 @@
|
||||
import fs from "fs";
|
||||
import path from "path";
|
||||
import { fileURLToPath } from "url";
|
||||
|
||||
export const SERVER_ROOT = path.dirname(fileURLToPath(import.meta.url));
|
||||
export const REPO_ROOT = path.resolve(SERVER_ROOT, "../..");
|
||||
export const SCRIPTS_DIR = path.join(SERVER_ROOT, "scripts");
|
||||
export const DATA_DIR = path.join(SERVER_ROOT, "data");
|
||||
|
||||
/** Prefer `preferred` unless only `legacy` already has files. */
|
||||
function existingDir(preferred, legacy, probe) {
|
||||
const has = (dir) =>
|
||||
probe ? probe(dir) : fs.existsSync(dir);
|
||||
if (has(preferred) || !has(legacy)) return preferred;
|
||||
return legacy;
|
||||
}
|
||||
|
||||
function hasInstanceData(dir) {
|
||||
return (
|
||||
fs.existsSync(path.join(dir, "jerapah-flow.db")) ||
|
||||
fs.existsSync(path.join(dir, "workflows"))
|
||||
);
|
||||
}
|
||||
|
||||
/** Instance data (SQLite, live workflows, control-state). Not product source. */
|
||||
export const DATA_DIR = path.resolve(
|
||||
process.env.JFLOW_DATA_DIR ??
|
||||
existingDir(
|
||||
path.join(REPO_ROOT, "data"),
|
||||
path.join(SERVER_ROOT, "data"),
|
||||
hasInstanceData,
|
||||
),
|
||||
);
|
||||
/** Live instance workflows (not shipped in git). Override for tests. */
|
||||
export const WORKFLOWS_DIR =
|
||||
process.env.JFLOW_WORKFLOWS_DIR ?? path.join(DATA_DIR, "workflows");
|
||||
/** Pre-0.1 layout; used only for one-shot migrate into WORKFLOWS_DIR. */
|
||||
export const LEGACY_WORKFLOWS_DIR = path.join(SERVER_ROOT, "workflows");
|
||||
export const WORKFLOWS_DIR = path.resolve(
|
||||
process.env.JFLOW_WORKFLOWS_DIR ?? path.join(DATA_DIR, "workflows"),
|
||||
);
|
||||
/** User plugins (repo-root /plugins, outside the pnpm workspace). */
|
||||
export const PLUGINS_DIR =
|
||||
process.env.JFLOW_PLUGINS_DIR ??
|
||||
path.resolve(SERVER_ROOT, "../../plugins");
|
||||
export const PLUGINS_DIR = path.resolve(
|
||||
process.env.JFLOW_PLUGINS_DIR ?? path.join(REPO_ROOT, "plugins"),
|
||||
);
|
||||
/** Example plugin sources shipped with the repo. */
|
||||
export const EXAMPLE_PLUGINS_DIR = path.resolve(
|
||||
SERVER_ROOT,
|
||||
"../../examples/plugins",
|
||||
);
|
||||
export const EXAMPLE_PLUGINS_DIR = path.join(REPO_ROOT, "examples/plugins");
|
||||
/** Example workflow YAML presets (not loaded by the runner). */
|
||||
export const EXAMPLE_WORKFLOWS_DIR = path.resolve(
|
||||
SERVER_ROOT,
|
||||
"../../examples/workflows",
|
||||
export const EXAMPLE_WORKFLOWS_DIR = path.join(REPO_ROOT, "examples/workflows");
|
||||
export const LOGS_DIR = path.resolve(
|
||||
process.env.JFLOW_LOGS_DIR ??
|
||||
existingDir(path.join(REPO_ROOT, "logs"), path.join(SERVER_ROOT, "logs")),
|
||||
);
|
||||
export const LOGS_DIR = path.join(SERVER_ROOT, "logs");
|
||||
export const WEB_DIST = path.resolve(SERVER_ROOT, "../web/dist");
|
||||
export const WEB_DIST = path.join(REPO_ROOT, "packages/web/dist");
|
||||
|
||||
@@ -1,8 +1,6 @@
|
||||
import path from "path";
|
||||
import pm2 from "pm2";
|
||||
import { SERVER_ROOT } from "./paths.js";
|
||||
|
||||
const REPO_ROOT = path.resolve(SERVER_ROOT, "../..");
|
||||
import { REPO_ROOT, SERVER_ROOT } from "./paths.js";
|
||||
|
||||
export const PM2_HTTP_NAME = "jflow-http";
|
||||
export const PM2_WORKER_NAME = "jflow-worker";
|
||||
@@ -164,13 +162,57 @@ export async function restartPm2Process(pmId) {
|
||||
return { name: proc.name, pmId: id };
|
||||
}
|
||||
|
||||
/**
|
||||
* PM2 injects these into process.env of a managed app. Spreading them into
|
||||
* `pm2.start({ env })` overwrites `name` / `pm_exec_path` so God restarts
|
||||
* jflow-control instead of launching http/worker (EADDRINUSE :8600 loop).
|
||||
*/
|
||||
const PM2_META_KEYS = new Set([
|
||||
"name",
|
||||
"namespace",
|
||||
"exec_mode",
|
||||
"exec_interpreter",
|
||||
"instances",
|
||||
"instance_var",
|
||||
"node_app_instance",
|
||||
"unique_id",
|
||||
"status",
|
||||
"username",
|
||||
"windowsHide",
|
||||
"merge_logs",
|
||||
"vizion",
|
||||
"vizion_running",
|
||||
"autostart",
|
||||
"autorestart",
|
||||
"automation",
|
||||
"km_link",
|
||||
]);
|
||||
|
||||
/**
|
||||
* @param {NodeJS.ProcessEnv} env
|
||||
* @returns {NodeJS.ProcessEnv}
|
||||
*/
|
||||
export function withoutPm2Meta(env) {
|
||||
/** @type {NodeJS.ProcessEnv} */
|
||||
const out = {};
|
||||
for (const [key, val] of Object.entries(env)) {
|
||||
if (val == null) continue;
|
||||
if (PM2_META_KEYS.has(key)) continue;
|
||||
if (key.startsWith("pm_") || key.startsWith("axm_") || key.startsWith("PM2_")) {
|
||||
continue;
|
||||
}
|
||||
out[key] = val;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Shared env for child processes.
|
||||
* @param {{ generation: number }} opts
|
||||
*/
|
||||
export function childEnv(opts) {
|
||||
return {
|
||||
...process.env,
|
||||
...withoutPm2Meta(process.env),
|
||||
JFLOW_CONFIG_GENERATION: String(opts.generation),
|
||||
JFLOW_CORS_ORIGIN: process.env.JFLOW_CORS_ORIGIN ?? "http://localhost:8500",
|
||||
PORT: process.env.JFLOW_HTTP_PORT ?? "8700",
|
||||
|
||||
@@ -36,7 +36,9 @@ import {
|
||||
resolveFailureTriggerConfig,
|
||||
} from "./trigger-failure.js";
|
||||
import { enqueueWorkflowJob } from "./workflow-queue.js";
|
||||
import { ensureInitialRevision } from "./workflow-history.js";
|
||||
import { ensureInitialRevision, recordRevision } from "./workflow-history.js";
|
||||
import { workflowIdFromFile } from "./workflow-normalize.js";
|
||||
import { publishReload } from "./control-bus.js";
|
||||
|
||||
/**
|
||||
* @typedef {{ owner: string, file: string, workflow: any }} WorkflowEntry
|
||||
@@ -509,6 +511,7 @@ export function createRegistry(server, opts = {}) {
|
||||
owner,
|
||||
workflowKey: key,
|
||||
context: incomingContext,
|
||||
data: ctx.data,
|
||||
});
|
||||
const stepCtx = {
|
||||
data: ctx.data,
|
||||
@@ -553,6 +556,45 @@ export function createRegistry(server, opts = {}) {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Persist `enabled: false` for a workflow and reload registries across processes.
|
||||
* @param {string} owner
|
||||
* @param {string} file
|
||||
* @param {string} key
|
||||
*/
|
||||
async function disableWorkflowForConsecutiveFailures(owner, file, key) {
|
||||
const content = fsStore.readWorkflowYaml(owner, file);
|
||||
if (content == null) {
|
||||
throw new Error(`workflow file missing for ${key}`);
|
||||
}
|
||||
const doc = yaml.parseDocument(content);
|
||||
if (doc.errors?.length) {
|
||||
throw new Error(doc.errors[0]?.message ?? "invalid yaml");
|
||||
}
|
||||
const parsed = doc.toJSON();
|
||||
if (parsed?.enabled === false) {
|
||||
log.debug({ workflow: key }, "workflow already disabled");
|
||||
return;
|
||||
}
|
||||
doc.set("enabled", false);
|
||||
const nextContent = String(doc);
|
||||
fsStore.writeWorkflowYaml(owner, file, nextContent);
|
||||
await recordRevision({
|
||||
workflowId: workflowIdFromFile(file),
|
||||
owner,
|
||||
file,
|
||||
content: nextContent,
|
||||
reason: "disable-on-consecutive-failures",
|
||||
});
|
||||
reregister();
|
||||
try {
|
||||
await publishReload({ type: "workflows" });
|
||||
} catch {
|
||||
// Redis may be briefly unavailable; local reload already applied.
|
||||
}
|
||||
log.warn({ workflow: key }, "disabled workflow after consecutive failures");
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {{
|
||||
* key: string,
|
||||
@@ -589,6 +631,17 @@ export function createRegistry(server, opts = {}) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (failureConfig.disableOnConsecutiveFailures) {
|
||||
const entry = workflows.get(opts.key);
|
||||
if (entry) {
|
||||
await disableWorkflowForConsecutiveFailures(entry.owner, entry.file, opts.key);
|
||||
} else {
|
||||
log.warn({ workflow: opts.key }, "cannot disable missing workflow entry");
|
||||
}
|
||||
}
|
||||
|
||||
if (!failureConfig.workflowName) return;
|
||||
|
||||
const destKey = resolveWorkflowTriggerKey(opts.owner, failureConfig.workflowName);
|
||||
const alertData = buildFailureAlertData({
|
||||
sourceKey: opts.key,
|
||||
|
||||
@@ -11,6 +11,7 @@ import { isSecret, Secret, unwrapSecretsDeep } from "./secret-value.js";
|
||||
import { getHttpPageByName, getHttpTemplateByName } from "./http-pages-store.js";
|
||||
import { getSecretPlaintext } from "./secrets-store.js";
|
||||
import { getVariablePlain } from "./variables-store.js";
|
||||
import { DEFAULT_OWNER } from "@jerapah-flow/shared";
|
||||
|
||||
const hostRequire = createRequire(import.meta.url);
|
||||
|
||||
@@ -443,7 +444,7 @@ function createScriptSandbox({
|
||||
log,
|
||||
script,
|
||||
workflowName,
|
||||
owner = "default",
|
||||
owner = DEFAULT_OWNER,
|
||||
$workflows = $workflowsStub,
|
||||
pluginDir = null,
|
||||
}) {
|
||||
@@ -563,7 +564,7 @@ export function instantiateScriptSource(script, source, opts = {}) {
|
||||
log: opts.log ?? inspectLog,
|
||||
script,
|
||||
workflowName: opts.workflowName ?? "inspect",
|
||||
owner: opts.owner ?? "default",
|
||||
owner: opts.owner ?? DEFAULT_OWNER,
|
||||
$workflows: opts.$workflows,
|
||||
pluginDir: opts.pluginDir ?? null,
|
||||
});
|
||||
|
||||
@@ -31,6 +31,7 @@ import { getAppVersion } from "../../app-version.js";
|
||||
import { EXAMPLE_PLUGINS_DIR } from "../../paths.js";
|
||||
import { pluginScriptRef } from "../../plugin-manifest.js";
|
||||
import { evaluateJsonata, SET_STEP_SCRIPT } from "../../workflow-parse.js";
|
||||
import { DEFAULT_OWNER } from "@jerapah-flow/shared";
|
||||
|
||||
/**
|
||||
* @param {{ referencedScripts: () => Set<string> }} registry
|
||||
@@ -258,7 +259,7 @@ export default function scriptsPluginFactory(registry) {
|
||||
req.body ?? {}
|
||||
);
|
||||
|
||||
let owner = "local";
|
||||
let owner = DEFAULT_OWNER;
|
||||
if (body.owner != null && body.owner !== "") {
|
||||
try {
|
||||
owner = fsStore.assertOwner(String(body.owner));
|
||||
@@ -304,6 +305,7 @@ export default function scriptsPluginFactory(registry) {
|
||||
owner,
|
||||
workflowKey: "dry-run",
|
||||
context: incomingContext,
|
||||
data: incomingData,
|
||||
},
|
||||
);
|
||||
const ctx = {
|
||||
@@ -365,6 +367,7 @@ export default function scriptsPluginFactory(registry) {
|
||||
owner,
|
||||
workflowKey: "dry-run",
|
||||
context: incomingContext,
|
||||
data: incomingData,
|
||||
});
|
||||
const ctx = {
|
||||
data: incomingData,
|
||||
|
||||
@@ -76,6 +76,7 @@ function triggerSummary(owner, workflow, nameById) {
|
||||
schedule: t?.schedule ?? null,
|
||||
onConsecutiveFailures: t?.onConsecutiveFailures ?? null,
|
||||
onFailureWorkflow: t?.onFailureWorkflow ?? null,
|
||||
disableOnConsecutiveFailures: t?.disableOnConsecutiveFailures === true,
|
||||
auth: isHttp ? authLabel(t?.auth, nameById) : null,
|
||||
};
|
||||
});
|
||||
@@ -705,15 +706,14 @@ export default function workflowsPluginFactory(registry) {
|
||||
return reply.code(err.statusCode ?? 400).send({ error: err.message });
|
||||
}
|
||||
const raw = fsStore.readWorkflowYaml(owner, file);
|
||||
if (raw == null) {
|
||||
return reply.code(404).send({ error: "workflow not found" });
|
||||
}
|
||||
let name = null;
|
||||
try {
|
||||
const parsed = yaml.parse(raw);
|
||||
name = parsed?.name ?? null;
|
||||
} catch {
|
||||
// ignore
|
||||
if (raw != null) {
|
||||
try {
|
||||
const parsed = yaml.parse(raw);
|
||||
name = parsed?.name ?? null;
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
try {
|
||||
const item = await moveWorkflowToTrash({
|
||||
|
||||
@@ -31,7 +31,6 @@ import {
|
||||
getRedisUrlForLog,
|
||||
} from "./workflow-queue.js";
|
||||
import { purgeExpiredTrash } from "./workflow-trash.js";
|
||||
import { migrateLegacyWorkflowsIfNeeded } from "./workflow-migrate.js";
|
||||
import {
|
||||
getConfigGeneration,
|
||||
startHeartbeatLoop,
|
||||
@@ -129,12 +128,6 @@ export async function startApp(opts = {}) {
|
||||
}
|
||||
});
|
||||
|
||||
try {
|
||||
migrateLegacyWorkflowsIfNeeded();
|
||||
} catch (err) {
|
||||
log.warn({ err }, "legacy workflow migrate failed");
|
||||
}
|
||||
|
||||
const registry = createRegistry(server, {
|
||||
queue: workflowQueue,
|
||||
// Cron + HTTP triggers enqueue jobs; only the API process may own them.
|
||||
|
||||
@@ -2,7 +2,7 @@ import { migrate, db } from "../db.js";
|
||||
import { upsertSecret, deleteSecret } from "../secrets-store.js";
|
||||
import { deleteVariable, upsertVariable } from "../variables-store.js";
|
||||
import { Secret } from "../secret-value.js";
|
||||
import { parseConfigRef, resolveConfigRefs } from "../config-refs.js";
|
||||
import { resolveConfigRefs } from "../config-refs.js";
|
||||
|
||||
await migrate();
|
||||
|
||||
@@ -23,126 +23,138 @@ async function assertRejects(fn, match) {
|
||||
throw new Error(`expected to reject (${match ?? "any error"})`);
|
||||
}
|
||||
|
||||
const owner = "default";
|
||||
const workflowKey = "default/config-refs-smoke.yaml";
|
||||
const ctx = { owner, workflowKey, context: {} };
|
||||
|
||||
function assertParse(value, expected) {
|
||||
const got = parseConfigRef(value);
|
||||
if (expected == null) {
|
||||
assert(got == null, `expected null parse for ${JSON.stringify(value)}, got ${JSON.stringify(got)}`);
|
||||
return;
|
||||
}
|
||||
assert(got != null, `expected parse for ${JSON.stringify(value)}`);
|
||||
assert(got.kind === expected.kind, `kind ${got.kind} !== ${expected.kind}`);
|
||||
assert(got.name === expected.name, `name ${JSON.stringify(got.name)} !== ${JSON.stringify(expected.name)}`);
|
||||
}
|
||||
|
||||
assertParse("password123", null);
|
||||
assertParse("$FOO_bar", null);
|
||||
assertParse("$SECRET", null);
|
||||
assertParse(" password123 ", null);
|
||||
assertParse("$SECRET_zte_modem_password", { kind: "secret", name: "zte_modem_password" });
|
||||
assertParse(" $SECRET_zte_modem_password ", { kind: "secret", name: "zte_modem_password" });
|
||||
assertParse("Bearer $SECRET_x", null);
|
||||
assertParse("$KV_modem password", null);
|
||||
assertParse("$VAR_ntfy_url", { kind: "var", name: "ntfy_url" });
|
||||
assertParse("$CONTEXT_token", { kind: "context", name: "token" });
|
||||
assertParse("$SECRET_", { kind: "secret", name: "" });
|
||||
assertParse("$CONTEXT_SECRET_foo", { kind: "context", name: "SECRET_foo" });
|
||||
const owner = "config_refs_smoke_owner";
|
||||
const ctx = { owner, workflowKey: `${owner}/config-refs-smoke.yaml`, context: {}, data: {} };
|
||||
|
||||
{
|
||||
const literal = await resolveConfigRefs("password123", ctx);
|
||||
assert(literal === "password123", "literal passthrough");
|
||||
const unknown = await resolveConfigRefs("$FOO_bar", ctx);
|
||||
assert(unknown === "$FOO_bar", "$FOO_bar stays literal");
|
||||
const kvLiteral = await resolveConfigRefs("$KV_modem_password", ctx);
|
||||
assert(kvLiteral === "$KV_modem_password", "$KV_ stays literal");
|
||||
const embedded = await resolveConfigRefs("Bearer $SECRET_x", ctx);
|
||||
assert(embedded === "Bearer $SECRET_x", "mid-string stays literal");
|
||||
const number = await resolveConfigRefs(42, ctx);
|
||||
assert(number === 42, "number passthrough");
|
||||
}
|
||||
|
||||
const secret = await upsertSecret({
|
||||
owner,
|
||||
name: "config_refs_smoke_token",
|
||||
value: "s3cret-ok",
|
||||
});
|
||||
const varUrl = await upsertVariable({
|
||||
owner,
|
||||
name: "config_refs_smoke_url",
|
||||
type: "string",
|
||||
value: "https://example.test",
|
||||
});
|
||||
const varRetry = await upsertVariable({
|
||||
owner,
|
||||
name: "config_refs_smoke_retry",
|
||||
type: "number",
|
||||
value: 3,
|
||||
});
|
||||
const varDebug = await upsertVariable({
|
||||
owner,
|
||||
name: "config_refs_smoke_debug",
|
||||
type: "boolean",
|
||||
value: false,
|
||||
});
|
||||
|
||||
const created = [];
|
||||
try {
|
||||
const secret = await upsertSecret({
|
||||
owner,
|
||||
name: "config_refs_smoke_token",
|
||||
value: "s3cret-ok",
|
||||
});
|
||||
created.push(["secret", secret.id]);
|
||||
|
||||
const varUrl = await upsertVariable({
|
||||
owner,
|
||||
name: "config_refs_smoke_url",
|
||||
type: "string",
|
||||
value: "https://example.test",
|
||||
});
|
||||
created.push(["var", varUrl.id]);
|
||||
|
||||
const varRetry = await upsertVariable({
|
||||
owner,
|
||||
name: "config_refs_smoke_retry",
|
||||
type: "number",
|
||||
value: 3,
|
||||
});
|
||||
created.push(["var", varRetry.id]);
|
||||
|
||||
const varDebug = await upsertVariable({
|
||||
owner,
|
||||
name: "config_refs_smoke_debug",
|
||||
type: "boolean",
|
||||
value: false,
|
||||
});
|
||||
created.push(["var", varDebug.id]);
|
||||
|
||||
{
|
||||
const resolved = await resolveConfigRefs("$SECRET_config_refs_smoke_token", ctx);
|
||||
assert(resolved === "s3cret-ok", "secret resolve");
|
||||
const resolved = await resolveConfigRefs("{{ secrets.config_refs_smoke_token }}", ctx);
|
||||
assert(resolved === "s3cret-ok", "secret whole-value");
|
||||
}
|
||||
{
|
||||
const resolved = await resolveConfigRefs(" $SECRET_config_refs_smoke_token ", ctx);
|
||||
assert(resolved === "s3cret-ok", "secret resolve trimmed");
|
||||
}
|
||||
{
|
||||
const resolved = await resolveConfigRefs("$VAR_config_refs_smoke_url", ctx);
|
||||
const resolved = await resolveConfigRefs("{{ vars.config_refs_smoke_url }}", ctx);
|
||||
assert(resolved === "https://example.test", "var string");
|
||||
}
|
||||
{
|
||||
const resolved = await resolveConfigRefs("$VAR_config_refs_smoke_retry", ctx);
|
||||
assert(resolved === 3, "var number stays number");
|
||||
assert(typeof resolved === "number", "var number type");
|
||||
const resolved = await resolveConfigRefs("{{ vars.config_refs_smoke_retry }}", ctx);
|
||||
assert(resolved === 3, "var number keeps type");
|
||||
}
|
||||
{
|
||||
const resolved = await resolveConfigRefs("$VAR_config_refs_smoke_debug", ctx);
|
||||
assert(resolved === false, "var boolean stays false");
|
||||
assert(typeof resolved === "boolean", "var boolean type");
|
||||
const resolved = await resolveConfigRefs("{{ vars.config_refs_smoke_debug }}", ctx);
|
||||
assert(resolved === false, "var boolean keeps type");
|
||||
}
|
||||
{
|
||||
const resolved = await resolveConfigRefs("$CONTEXT_token", {
|
||||
const resolved = await resolveConfigRefs("{{ context.token }}", {
|
||||
...ctx,
|
||||
context: { token: "ctx-token-ok" },
|
||||
});
|
||||
assert(resolved === "ctx-token-ok", "context string");
|
||||
}
|
||||
{
|
||||
const resolved = await resolveConfigRefs("$CONTEXT_n", {
|
||||
const resolved = await resolveConfigRefs("{{ context.n }}", {
|
||||
...ctx,
|
||||
context: { n: 7 },
|
||||
});
|
||||
assert(resolved === "7", "context number stringify");
|
||||
assert(resolved === 7, "context number keeps type");
|
||||
}
|
||||
{
|
||||
const wrapped = new Secret("wrapped-secret-ok");
|
||||
const resolved = await resolveConfigRefs("$CONTEXT_tok", {
|
||||
const resolved = await resolveConfigRefs("{{ context.tok }}", {
|
||||
...ctx,
|
||||
context: { tok: wrapped },
|
||||
});
|
||||
assert(resolved === "wrapped-secret-ok", "context Secret unwrap");
|
||||
}
|
||||
|
||||
{
|
||||
const resolved = await resolveConfigRefs("{{ context.user }}", {
|
||||
...ctx,
|
||||
context: { user: { id: "u1", role: "admin" } },
|
||||
});
|
||||
assert(
|
||||
resolved && typeof resolved === "object" && resolved.id === "u1",
|
||||
"whole-value object pass-through",
|
||||
);
|
||||
}
|
||||
{
|
||||
const resolved = await resolveConfigRefs("{{ context.user.id }}", {
|
||||
...ctx,
|
||||
context: { user: { id: "nested-id" } },
|
||||
});
|
||||
assert(resolved === "nested-id", "nested context path");
|
||||
}
|
||||
{
|
||||
const resolved = await resolveConfigRefs("{{ data.items.0.id }}", {
|
||||
...ctx,
|
||||
data: { items: [{ id: "row-0" }] },
|
||||
});
|
||||
assert(resolved === "row-0", "array index path");
|
||||
}
|
||||
{
|
||||
const resolved = await resolveConfigRefs(
|
||||
"{{ vars.config_refs_smoke_url }}/{{ data.channel }}",
|
||||
{ ...ctx, data: { channel: "alerts" } },
|
||||
);
|
||||
assert(resolved === "https://example.test/alerts", "concatenation");
|
||||
}
|
||||
{
|
||||
const resolved = await resolveConfigRefs("Bearer {{ context.token }}", {
|
||||
...ctx,
|
||||
context: { token: "abc" },
|
||||
});
|
||||
assert(resolved === "Bearer abc", "mixed string");
|
||||
}
|
||||
{
|
||||
const nested = await resolveConfigRefs(
|
||||
{
|
||||
url: "$VAR_config_refs_smoke_url",
|
||||
retry: "$VAR_config_refs_smoke_retry",
|
||||
debug: "$VAR_config_refs_smoke_debug",
|
||||
password: "$SECRET_config_refs_smoke_token",
|
||||
url: "{{ vars.config_refs_smoke_url }}",
|
||||
retry: "{{ vars.config_refs_smoke_retry }}",
|
||||
debug: "{{ vars.config_refs_smoke_debug }}",
|
||||
password: "{{ secrets.config_refs_smoke_token }}",
|
||||
headers: { Authorization: "$KV_modem_password" },
|
||||
extra: ["$CONTEXT_token", "plain"],
|
||||
extra: ["{{ context.token }}", "plain"],
|
||||
},
|
||||
{ ...ctx, context: { token: "ctx-token-ok" } },
|
||||
);
|
||||
@@ -155,59 +167,47 @@ try {
|
||||
assert(nested.extra[1] === "plain", "nested array literal");
|
||||
}
|
||||
|
||||
const data = { password: "$SECRET_config_refs_smoke_token" };
|
||||
const config = { password: "$SECRET_config_refs_smoke_token" };
|
||||
const data = { password: "{{ secrets.config_refs_smoke_token }}" };
|
||||
const config = { password: "{{ secrets.config_refs_smoke_token }}" };
|
||||
const resolvedConfig = await resolveConfigRefs(config, ctx);
|
||||
assert(resolvedConfig.password === "s3cret-ok", "config resolved");
|
||||
assert(data.password === "$SECRET_config_refs_smoke_token", "data not walked");
|
||||
assert(config.password === "$SECRET_config_refs_smoke_token", "input config not mutated");
|
||||
assert(data.password === "{{ secrets.config_refs_smoke_token }}", "data not walked");
|
||||
assert(config.password === "{{ secrets.config_refs_smoke_token }}", "input config not mutated");
|
||||
|
||||
await assertRejects(
|
||||
() => resolveConfigRefs("$SECRET_does_not_exist_xyz", ctx),
|
||||
() => resolveConfigRefs("{{ secrets.does_not_exist_xyz }}", ctx),
|
||||
'secret "does_not_exist_xyz" not found',
|
||||
);
|
||||
await assertRejects(
|
||||
() => resolveConfigRefs("$SECRET_not valid", ctx),
|
||||
"invalid secret name",
|
||||
);
|
||||
await assertRejects(
|
||||
() => resolveConfigRefs("$SECRET_", ctx),
|
||||
"invalid secret name",
|
||||
);
|
||||
await assertRejects(
|
||||
() => resolveConfigRefs("$CONTEXT_missing", ctx),
|
||||
'context "missing" not found',
|
||||
() => resolveConfigRefs("{{ context.missing }}", ctx),
|
||||
"path not found",
|
||||
);
|
||||
await assertRejects(
|
||||
() =>
|
||||
resolveConfigRefs("$CONTEXT_obj", {
|
||||
resolveConfigRefs("Bearer {{ context.obj }}", {
|
||||
...ctx,
|
||||
context: { obj: { a: 1 } },
|
||||
}),
|
||||
'context "obj" is not a scalar',
|
||||
"not a scalar",
|
||||
);
|
||||
await assertRejects(
|
||||
() => resolveConfigRefs("$CONTEXT_", ctx),
|
||||
"empty context key",
|
||||
() => resolveConfigRefs("{{ vars }}", ctx),
|
||||
"empty var name",
|
||||
);
|
||||
await assertRejects(
|
||||
() => resolveConfigRefs("$VAR_does_not_exist_xyz", ctx),
|
||||
'variable "does_not_exist_xyz" not found',
|
||||
() => resolveConfigRefs("{{ title }}", ctx),
|
||||
"unknown root",
|
||||
);
|
||||
await assertRejects(
|
||||
() => resolveConfigRefs("$VAR_not valid", ctx),
|
||||
"invalid variable name",
|
||||
);
|
||||
await assertRejects(
|
||||
() => resolveConfigRefs("$VAR_", ctx),
|
||||
"empty variable name",
|
||||
() => resolveConfigRefs("{{ context.__proto__.x }}", ctx),
|
||||
"forbidden path segment",
|
||||
);
|
||||
} finally {
|
||||
await deleteSecret(secret.id);
|
||||
await deleteVariable(varUrl.id);
|
||||
await deleteVariable(varRetry.id);
|
||||
await deleteVariable(varDebug.id);
|
||||
for (const [kind, id] of created.reverse()) {
|
||||
if (kind === "secret") await deleteSecret(id);
|
||||
else await deleteVariable(id);
|
||||
}
|
||||
await db.destroy();
|
||||
}
|
||||
|
||||
console.log("config-refs smoke test passed");
|
||||
await db.destroy();
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
* Smoke: core vs plugin scripts, fork, example install, resolve, run.
|
||||
*
|
||||
* Run:
|
||||
* JFLOW_DATA_DIR=packages/server/data \
|
||||
* JFLOW_PLUGINS_DIR=packages/server/data/plugins-smoke-test \
|
||||
* JFLOW_DB_PATH=packages/server/data/plugins-smoke.db \
|
||||
* node packages/server/test/plugins-smoke.js
|
||||
|
||||
@@ -71,11 +71,11 @@ const created = await upsertProfile({
|
||||
owner,
|
||||
name,
|
||||
script: "ntfy.js",
|
||||
config: { url: "$VAR_ntfy_channel" },
|
||||
config: { url: "{{ vars.ntfy_channel }}" },
|
||||
description: "smoke",
|
||||
});
|
||||
assert(created.name === name, "created");
|
||||
assert(created.config.url === "$VAR_ntfy_channel", "config roundtrip");
|
||||
assert(created.config.url === "{{ vars.ntfy_channel }}", "config roundtrip");
|
||||
assert(created.script === "ntfy.js", "script locked on profile");
|
||||
|
||||
const fetched = await getProfilePlain(owner, name);
|
||||
|
||||
@@ -136,6 +136,18 @@ const warnings = collectWorkflowWarnings(
|
||||
);
|
||||
assert.ok(warnings.warnings.some((w) => w.code === "unknown_script"));
|
||||
|
||||
const ghostFile = newWorkflowFilename();
|
||||
fsStore.writeRegisters(owner, [file, ghostFile]);
|
||||
const ghostTrash = await moveWorkflowToTrash({
|
||||
workflowId: workflowIdFromFile(ghostFile),
|
||||
owner,
|
||||
file: ghostFile,
|
||||
name: null,
|
||||
});
|
||||
assert.equal(ghostTrash, null);
|
||||
assert.ok(!fsStore.readRegisters(owner).includes(ghostFile));
|
||||
assert.ok(fsStore.readRegisters(owner).includes(file));
|
||||
|
||||
const trashed = await moveWorkflowToTrash({
|
||||
workflowId,
|
||||
owner,
|
||||
|
||||
@@ -47,12 +47,18 @@ export function resolveFailureTriggerConfig(workflow, owner, runtimeTrigger) {
|
||||
|
||||
const threshold = Number(spec.onConsecutiveFailures);
|
||||
const workflowName = onFailureWorkflowName(spec);
|
||||
if (!Number.isFinite(threshold) || threshold < 1 || workflowName.length === 0) {
|
||||
const disableOnConsecutiveFailures = isDisableOnConsecutiveFailures(spec);
|
||||
if (
|
||||
!Number.isFinite(threshold) ||
|
||||
threshold < 1 ||
|
||||
(workflowName.length === 0 && !disableOnConsecutiveFailures)
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
threshold: Math.floor(threshold),
|
||||
workflowName,
|
||||
workflowName: workflowName.length > 0 ? workflowName : null,
|
||||
disableOnConsecutiveFailures,
|
||||
};
|
||||
}
|
||||
return null;
|
||||
@@ -66,6 +72,13 @@ function onFailureWorkflowName(trigger) {
|
||||
return typeof value === "string" ? value.trim() : "";
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {Record<string, unknown>} trigger
|
||||
*/
|
||||
export function isDisableOnConsecutiveFailures(trigger) {
|
||||
return trigger?.disableOnConsecutiveFailures === true;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {unknown} workflow
|
||||
*/
|
||||
@@ -81,12 +94,13 @@ export async function validateWorkflowFailureTriggers(workflow) {
|
||||
const hasThreshold =
|
||||
trigger.onConsecutiveFailures != null && trigger.onConsecutiveFailures !== "";
|
||||
const hasWorkflow = onFailureWorkflowName(trigger).length > 0;
|
||||
const hasDisable = isDisableOnConsecutiveFailures(trigger);
|
||||
|
||||
if (!hasThreshold && !hasWorkflow) continue;
|
||||
if (!hasThreshold && !hasWorkflow && !hasDisable) continue;
|
||||
|
||||
if (!hasThreshold || !hasWorkflow) {
|
||||
if (!hasThreshold || (!hasWorkflow && !hasDisable)) {
|
||||
const err = new Error(
|
||||
"onConsecutiveFailures and onFailureWorkflow must both be set on a trigger",
|
||||
"onConsecutiveFailures requires onFailureWorkflow and/or disableOnConsecutiveFailures",
|
||||
);
|
||||
err.statusCode = 400;
|
||||
throw err;
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
/**
|
||||
* Production UI server (:8500).
|
||||
* Serves packages/web/dist and proxies /api, /ops, /admin, /u like Vite in dev.
|
||||
* Always-on — survives Ops stop of jflow-http.
|
||||
*/
|
||||
import fs from "fs";
|
||||
import fastify from "fastify";
|
||||
import fastifyStatic from "@fastify/static";
|
||||
import { log } from "./logger.js";
|
||||
import { WEB_DIST } from "./paths.js";
|
||||
import {
|
||||
controlOrigin,
|
||||
httpOrigin,
|
||||
proxyToOrigin,
|
||||
} from "./ops-proxy.js";
|
||||
|
||||
if (!fs.existsSync(WEB_DIST)) {
|
||||
log.error(
|
||||
{ WEB_DIST },
|
||||
"web dist missing — run `pnpm build` before starting the UI server",
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const port = Number(process.env.JFLOW_UI_PORT ?? 8500);
|
||||
const control = controlOrigin();
|
||||
const http = httpOrigin();
|
||||
|
||||
const server = fastify({ loggerInstance: log });
|
||||
|
||||
/**
|
||||
* @param {import("fastify").FastifyRequest} req
|
||||
* @param {import("fastify").FastifyReply} reply
|
||||
*/
|
||||
async function proxyApi(req, reply) {
|
||||
const url = req.raw.url ?? "";
|
||||
// Match Vite: /api/auth → control (login works when HTTP is stopped).
|
||||
if (url === "/api/auth" || url.startsWith("/api/auth/") || url.startsWith("/api/auth?")) {
|
||||
return proxyToOrigin(req, reply, control, {
|
||||
unreachableMessage: "control plane unreachable",
|
||||
});
|
||||
}
|
||||
return proxyToOrigin(req, reply, http, {
|
||||
unreachableMessage: "HTTP API unreachable",
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {import("fastify").FastifyRequest} req
|
||||
* @param {import("fastify").FastifyReply} reply
|
||||
*/
|
||||
async function proxyOps(req, reply) {
|
||||
return proxyToOrigin(req, reply, control, {
|
||||
unreachableMessage: "control plane unreachable",
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {import("fastify").FastifyRequest} req
|
||||
* @param {import("fastify").FastifyReply} reply
|
||||
*/
|
||||
async function proxyHttp(req, reply) {
|
||||
return proxyToOrigin(req, reply, http, {
|
||||
unreachableMessage: "HTTP API unreachable",
|
||||
});
|
||||
}
|
||||
|
||||
server.all("/api", proxyApi);
|
||||
server.all("/api/*", proxyApi);
|
||||
server.all("/ops", proxyOps);
|
||||
server.all("/ops/*", proxyOps);
|
||||
server.all("/admin", proxyHttp);
|
||||
server.all("/admin/*", proxyHttp);
|
||||
server.all("/u", proxyHttp);
|
||||
server.all("/u/*", proxyHttp);
|
||||
|
||||
await server.register(fastifyStatic, {
|
||||
root: WEB_DIST,
|
||||
wildcard: false,
|
||||
});
|
||||
|
||||
server.setNotFoundHandler((req, reply) => {
|
||||
const url = req.raw.url ?? "";
|
||||
if (
|
||||
url.startsWith("/api") ||
|
||||
url.startsWith("/u/") ||
|
||||
url.startsWith("/admin") ||
|
||||
url.startsWith("/ops")
|
||||
) {
|
||||
return reply.code(404).send({ error: "not found" });
|
||||
}
|
||||
return reply.sendFile("index.html");
|
||||
});
|
||||
|
||||
async function shutdown() {
|
||||
try {
|
||||
await server.close();
|
||||
} catch (err) {
|
||||
log.error({ err }, "web-server shutdown error");
|
||||
}
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
process.on("SIGINT", shutdown);
|
||||
process.on("SIGTERM", shutdown);
|
||||
|
||||
try {
|
||||
await server.listen({ host: "0.0.0.0", port });
|
||||
log.info(
|
||||
{ port, WEB_DIST, control, http },
|
||||
"UI server listening (static + proxy)",
|
||||
);
|
||||
} catch (err) {
|
||||
log.error({ err }, "failed to start UI server");
|
||||
process.exit(1);
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
import { HTTP_METHODS } from "@jerapah-flow/shared";
|
||||
import { HTTP_METHODS, DEFAULT_OWNER } from "@jerapah-flow/shared";
|
||||
import {
|
||||
checkAnyHttpAuth,
|
||||
resolveAuthMechanisms,
|
||||
@@ -85,8 +85,14 @@ export function createHttpTriggerHandler({
|
||||
return async function dispatchHttpTrigger(req, reply) {
|
||||
const wildcard = /** @type {{ "*": string }} */ (req.params)["*"] ?? "";
|
||||
const url = `/u/${String(wildcard).replace(/^\/+/, "")}`;
|
||||
// Compat: leftover webhooks still hitting /u/default/... after owner rename.
|
||||
const compatUrl = url.startsWith("/u/default/")
|
||||
? `/u/${DEFAULT_OWNER}/${url.slice("/u/default/".length)}`
|
||||
: url === "/u/default"
|
||||
? `/u/${DEFAULT_OWNER}`
|
||||
: url;
|
||||
const method = String(req.method ?? "GET").toUpperCase();
|
||||
const routeKey = `${method} ${url}`;
|
||||
const routeKey = `${method} ${compatUrl}`;
|
||||
const mapped = httpRoutes.get(routeKey);
|
||||
|
||||
if (!mapped) {
|
||||
@@ -104,7 +110,7 @@ export function createHttpTriggerHandler({
|
||||
if (t?.type !== "HTTP") return false;
|
||||
const m = String(t.method ?? "POST").toUpperCase();
|
||||
const p = namespacedPath(entry.owner, t.path);
|
||||
return m === method && p === url;
|
||||
return m === method && p === compatUrl;
|
||||
}) ?? mapped.trigger;
|
||||
|
||||
if (
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
import fs from "fs";
|
||||
import path from "path";
|
||||
import { LEGACY_WORKFLOWS_DIR, WORKFLOWS_DIR } from "./paths.js";
|
||||
import { log } from "./logger.js";
|
||||
|
||||
/**
|
||||
* Recursively copy a directory.
|
||||
* @param {string} src
|
||||
* @param {string} dest
|
||||
*/
|
||||
function copyDir(src, dest) {
|
||||
fs.mkdirSync(dest, { recursive: true });
|
||||
for (const entry of fs.readdirSync(src, { withFileTypes: true })) {
|
||||
const from = path.join(src, entry.name);
|
||||
const to = path.join(dest, entry.name);
|
||||
if (entry.isDirectory()) copyDir(from, to);
|
||||
else fs.copyFileSync(from, to);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* True when WORKFLOWS_DIR has no owner subdirectories.
|
||||
* @param {string} dir
|
||||
*/
|
||||
function isEmptyWorkflowsDir(dir) {
|
||||
if (!fs.existsSync(dir)) return true;
|
||||
const entries = fs.readdirSync(dir, { withFileTypes: true });
|
||||
return !entries.some((e) => e.isDirectory());
|
||||
}
|
||||
|
||||
/**
|
||||
* One-shot: copy packages/server/workflows → data/workflows when the new
|
||||
* store is empty and the legacy tree still exists.
|
||||
* Does not copy from examples/workflows.
|
||||
*/
|
||||
export function migrateLegacyWorkflowsIfNeeded() {
|
||||
if (!isEmptyWorkflowsDir(WORKFLOWS_DIR)) return false;
|
||||
if (!fs.existsSync(LEGACY_WORKFLOWS_DIR)) return false;
|
||||
const legacyEntries = fs.readdirSync(LEGACY_WORKFLOWS_DIR, {
|
||||
withFileTypes: true,
|
||||
});
|
||||
if (!legacyEntries.some((e) => e.isDirectory())) return false;
|
||||
|
||||
fs.mkdirSync(WORKFLOWS_DIR, { recursive: true });
|
||||
copyDir(LEGACY_WORKFLOWS_DIR, WORKFLOWS_DIR);
|
||||
log.info(
|
||||
{ from: LEGACY_WORKFLOWS_DIR, to: WORKFLOWS_DIR },
|
||||
"migrated legacy workflows into instance store",
|
||||
);
|
||||
return true;
|
||||
}
|
||||
@@ -62,29 +62,49 @@ export async function isInTrash(owner, file) {
|
||||
return Boolean(row);
|
||||
}
|
||||
|
||||
function unregisterWorkflow(owner, file) {
|
||||
const registered = fsStore.readRegisters(owner).filter((f) => f !== file);
|
||||
fsStore.writeRegisters(owner, registered);
|
||||
}
|
||||
|
||||
/**
|
||||
* Soft-delete: move YAML to trash dir, unregister, keep revision history.
|
||||
* Soft-delete: unregister, move YAML to trash when present, keep revision history.
|
||||
* Missing YAML (ghost register entries) is still unregistered so it leaves the list.
|
||||
* @param {{
|
||||
* workflowId: string,
|
||||
* owner: string,
|
||||
* file: string,
|
||||
* name?: string | null,
|
||||
* }} opts
|
||||
* @returns {Promise<ReturnType<typeof rowToItem> | null>} trash item, or null if there was no file to keep
|
||||
*/
|
||||
export async function moveWorkflowToTrash(opts) {
|
||||
unregisterWorkflow(opts.owner, opts.file);
|
||||
|
||||
const sourcePath = path.join(WORKFLOWS_DIR, opts.owner, opts.file);
|
||||
if (!fs.existsSync(sourcePath)) {
|
||||
const err = new Error("workflow not found");
|
||||
err.statusCode = 404;
|
||||
throw err;
|
||||
const existing = await db("workflow_trash")
|
||||
.where({ owner: opts.owner, file: opts.file })
|
||||
.first();
|
||||
return existing ? rowToItem(existing) : null;
|
||||
}
|
||||
|
||||
const trashPath = trashFilePath(opts.owner, opts.file);
|
||||
fs.mkdirSync(path.dirname(trashPath), { recursive: true });
|
||||
fs.renameSync(sourcePath, trashPath);
|
||||
|
||||
const registered = fsStore.readRegisters(opts.owner).filter((f) => f !== opts.file);
|
||||
fsStore.writeRegisters(opts.owner, registered);
|
||||
const existing = await db("workflow_trash")
|
||||
.where({ owner: opts.owner, file: opts.file })
|
||||
.first();
|
||||
if (existing) {
|
||||
await db("workflow_trash").where({ id: existing.id }).update({
|
||||
workflow_id: opts.workflowId,
|
||||
name: opts.name ?? existing.name ?? null,
|
||||
deleted_at: nowIso(),
|
||||
trash_path: trashPath,
|
||||
});
|
||||
return rowToItem(await db("workflow_trash").where({ id: existing.id }).first());
|
||||
}
|
||||
|
||||
const id = randomUUID();
|
||||
const deleted_at = nowIso();
|
||||
|
||||
@@ -2,3 +2,4 @@ export { isPlainObject } from "./is-plain-object.js";
|
||||
export { mergeProfileConfig, overlayFromMerged, configHasOverlay } from "./profile-config.js";
|
||||
export { ensureWorkflowFilename, suggestCopyFilename } from "./workflow-filename.js";
|
||||
export { HTTP_METHODS, namespacedPath, hasWorkflowTrigger } from "./workflow-path.js";
|
||||
export { DEFAULT_OWNER } from "./tenant.js";
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
/** Default owner folder for new resources (latent tenant id). */
|
||||
export const DEFAULT_OWNER = "local";
|
||||
@@ -66,7 +66,8 @@ export function SecretEditorModal({ mode, initial, onClose, onSaved }) {
|
||||
</label>
|
||||
<p className="text-xs opacity-60">
|
||||
Values are encrypted at rest and never shown again after save. Values shorter than 8
|
||||
characters are not redacted from logs.
|
||||
characters are not redacted from logs. In workflows use{" "}
|
||||
<span className="font-mono">{"{{ secrets.name }}"}</span> (quote in YAML).
|
||||
</p>
|
||||
{upsert.isError ? (
|
||||
<p className="text-error text-sm">{errorMessage(upsert.error)}</p>
|
||||
|
||||
@@ -129,7 +129,9 @@ export function VariableEditorModal({ mode, initial, onClose, onSaved }) {
|
||||
</label>
|
||||
<p className="text-xs opacity-60">
|
||||
Stored in plaintext. Use Secrets for credentials. In workflows use{" "}
|
||||
<span className="font-mono">$VAR_name</span> as a whole field.
|
||||
<span className="font-mono">{"{{ vars.name }}"}</span> (quote strings that
|
||||
start with <span className="font-mono">{"{"}</span>). Nested:{" "}
|
||||
<span className="font-mono">{"{{ context.user.id }}"}</span>.
|
||||
</p>
|
||||
{formError ? <p className="text-error text-sm">{formError}</p> : null}
|
||||
{upsert.isError ? (
|
||||
|
||||
@@ -4,22 +4,27 @@ import { LuEye, LuEyeOff, LuExternalLink } from "react-icons/lu";
|
||||
import { useVariables } from "../../api/hooks.js";
|
||||
|
||||
const VAR_PEEK_MAX = 48;
|
||||
const MUSTACHE_RE =
|
||||
/\{\{\s*([A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z0-9_]+)*)\s*\}\}/g;
|
||||
|
||||
const CONFIG_REF_PREFIXES = [
|
||||
{ prefix: "$SECRET_", label: "secret" },
|
||||
{ prefix: "$CONTEXT_", label: "context" },
|
||||
{ prefix: "$VAR_", label: "variable" },
|
||||
];
|
||||
|
||||
/**
|
||||
* @param {unknown} value
|
||||
* @returns {{
|
||||
* kind: "mustache",
|
||||
* path?: string,
|
||||
* root?: string,
|
||||
* name?: string,
|
||||
* } | null}
|
||||
*/
|
||||
function describeConfigRef(value) {
|
||||
if (typeof value !== "string") return null;
|
||||
const trimmed = value.trim();
|
||||
for (const { prefix, label } of CONFIG_REF_PREFIXES) {
|
||||
if (trimmed.startsWith(prefix) && trimmed.length > prefix.length) {
|
||||
return { label, name: trimmed.slice(prefix.length), kind: prefix };
|
||||
}
|
||||
}
|
||||
return null;
|
||||
MUSTACHE_RE.lastIndex = 0;
|
||||
const match = MUSTACHE_RE.exec(trimmed);
|
||||
if (!match) return null;
|
||||
const path = match[1];
|
||||
const root = path.split(".")[0];
|
||||
return { kind: "mustache", path, root, name: path.slice(root.length + 1) };
|
||||
}
|
||||
|
||||
function formatVarDisplay(value) {
|
||||
@@ -33,7 +38,7 @@ function truncatePeek(text, maxLen = VAR_PEEK_MAX) {
|
||||
return `${text.slice(0, Math.max(0, maxLen - 1))}…`;
|
||||
}
|
||||
|
||||
/** Edit-time lookup for `$VAR_` against workflow owner (not a runtime guarantee). */
|
||||
/** Edit-time lookup for `{{ vars.name }}` against workflow owner (not a runtime guarantee). */
|
||||
function lookupVariable(variables, owner, name) {
|
||||
const list = Array.isArray(variables) ? variables : [];
|
||||
const match = list.find((v) => v.owner === owner && v.name === name);
|
||||
@@ -62,7 +67,7 @@ function variablesDeepLink({ owner, name, missing }) {
|
||||
|
||||
export function ConfigRefHint({ value, owner }) {
|
||||
const ref = describeConfigRef(value);
|
||||
const isVar = ref?.kind === "$VAR_";
|
||||
const isVar = ref?.kind === "mustache" && ref.root === "vars" && Boolean(ref.name);
|
||||
const { data: variables = [], isPending } = useVariables(undefined, { enabled: isVar });
|
||||
const [revealed, setRevealed] = useState(false);
|
||||
|
||||
@@ -73,9 +78,17 @@ export function ConfigRefHint({ value, owner }) {
|
||||
if (!ref) return null;
|
||||
|
||||
if (!isVar) {
|
||||
const label =
|
||||
ref.root === "secrets"
|
||||
? "secret"
|
||||
: ref.root === "context"
|
||||
? "context"
|
||||
: ref.root === "data"
|
||||
? "data"
|
||||
: "expression";
|
||||
return (
|
||||
<p className="text-xs opacity-60">
|
||||
from {ref.label} <span className="font-mono">{ref.name}</span>
|
||||
from {label} <span className="font-mono">{ref.path}</span>
|
||||
</p>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -143,10 +143,15 @@ function TriggerCardView({
|
||||
|
||||
function triggerSummary(trigger, owner) {
|
||||
const type = trigger?.type;
|
||||
const failure =
|
||||
trigger.onConsecutiveFailures && trigger.onFailureWorkflow
|
||||
? ` · onFailure@${trigger.onFailureWorkflow}`
|
||||
: "";
|
||||
/** @type {string[]} */
|
||||
const failureParts = [];
|
||||
if (trigger.onConsecutiveFailures && trigger.onFailureWorkflow) {
|
||||
failureParts.push(`onFailure@${trigger.onFailureWorkflow}`);
|
||||
}
|
||||
if (trigger.disableOnConsecutiveFailures) {
|
||||
failureParts.push("auto-disable");
|
||||
}
|
||||
const failure = failureParts.length ? ` · ${failureParts.join(", ")}` : "";
|
||||
if (type === "HTTP") {
|
||||
return `${trigger.method || "POST"} ${namespacedPath(owner || "owner", trigger.path || "/")}${failure}`;
|
||||
}
|
||||
@@ -383,6 +388,20 @@ function FailureAlertFields({ trigger, disabled, onChange, alertDestinations })
|
||||
) : null}
|
||||
</FormSelect>
|
||||
</Field>
|
||||
<label className="label cursor-pointer justify-start gap-3 py-0">
|
||||
<input
|
||||
type="checkbox"
|
||||
className="checkbox checkbox-sm"
|
||||
checked={Boolean(trigger.disableOnConsecutiveFailures)}
|
||||
disabled={disabled}
|
||||
onChange={(e) =>
|
||||
onChange({ ...trigger, disableOnConsecutiveFailures: e.target.checked })
|
||||
}
|
||||
/>
|
||||
<span className="label-text">
|
||||
Disable this workflow when the consecutive failure threshold is reached
|
||||
</span>
|
||||
</label>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1,2 +1 @@
|
||||
/** Default owner folder for new resources (latent tenant id). */
|
||||
export const DEFAULT_OWNER = "local";
|
||||
export { DEFAULT_OWNER } from "@jerapah-flow/shared";
|
||||
|
||||
@@ -171,6 +171,7 @@ export function newHttpTrigger() {
|
||||
unauthorized: null,
|
||||
onConsecutiveFailures: "",
|
||||
onFailureWorkflow: "",
|
||||
disableOnConsecutiveFailures: false,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -186,6 +187,7 @@ export function newCronTrigger() {
|
||||
unauthorized: null,
|
||||
onConsecutiveFailures: "",
|
||||
onFailureWorkflow: "",
|
||||
disableOnConsecutiveFailures: false,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -201,6 +203,7 @@ export function newWorkflowTrigger() {
|
||||
unauthorized: null,
|
||||
onConsecutiveFailures: "",
|
||||
onFailureWorkflow: "",
|
||||
disableOnConsecutiveFailures: false,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -303,9 +306,16 @@ function normalizeTrigger(raw) {
|
||||
"unauthorized",
|
||||
"onConsecutiveFailures",
|
||||
"onFailureWorkflow",
|
||||
"disableOnConsecutiveFailures",
|
||||
])
|
||||
: type === "cron"
|
||||
? new Set(["type", "schedule", "onConsecutiveFailures", "onFailureWorkflow"])
|
||||
? new Set([
|
||||
"type",
|
||||
"schedule",
|
||||
"onConsecutiveFailures",
|
||||
"onFailureWorkflow",
|
||||
"disableOnConsecutiveFailures",
|
||||
])
|
||||
: new Set(["type"]);
|
||||
/** @type {Record<string, unknown>} */
|
||||
const extra = {};
|
||||
@@ -323,6 +333,7 @@ function normalizeTrigger(raw) {
|
||||
? ""
|
||||
: String(raw.onConsecutiveFailures),
|
||||
onFailureWorkflow: readOnFailureWorkflow(raw),
|
||||
disableOnConsecutiveFailures: raw.disableOnConsecutiveFailures === true,
|
||||
auth: Array.isArray(raw.auth) ? raw.auth : null,
|
||||
response: typeof raw.response === "string" ? raw.response : "",
|
||||
unauthorized: raw.unauthorized ?? null,
|
||||
@@ -377,6 +388,9 @@ function dumpFailureTriggerFields(t, out) {
|
||||
if (typeof t.onFailureWorkflow === "string" && t.onFailureWorkflow.trim()) {
|
||||
out.onFailureWorkflow = t.onFailureWorkflow.trim();
|
||||
}
|
||||
if (t.disableOnConsecutiveFailures === true) {
|
||||
out.disableOnConsecutiveFailures = true;
|
||||
}
|
||||
}
|
||||
|
||||
function dumpTrigger(t) {
|
||||
|
||||
@@ -179,7 +179,7 @@ export function VariablesPage() {
|
||||
title={confirmDelete ? `Delete ${confirmDelete.name}?` : ""}
|
||||
message={
|
||||
confirmDelete
|
||||
? `This cannot be undone. Workflows that reference $VAR_${confirmDelete.name} will fail.`
|
||||
? `This cannot be undone. Workflows that reference {{ vars.${confirmDelete.name} }} will fail.`
|
||||
: ""
|
||||
}
|
||||
error={del.isError ? errorMessage(del.error) : null}
|
||||
|
||||
@@ -198,7 +198,7 @@ joplinHttp.meta = {
|
||||
token: {
|
||||
type: "string",
|
||||
required: true,
|
||||
description: "Bearer token ($SECRET_); Bearer prefix is added if missing",
|
||||
description: "Bearer token ({{ secrets.* }}); Bearer prefix is added if missing",
|
||||
},
|
||||
timeoutMs: {
|
||||
type: "number",
|
||||
@@ -237,7 +237,7 @@ joplinHttp.meta = {
|
||||
config: {
|
||||
url: "http://10.8.0.6:3030/notes",
|
||||
method: "GET",
|
||||
token: "$SECRET_joplin_api_token",
|
||||
token: "{{ secrets.joplin_api_token }}",
|
||||
timeoutMs: 60000,
|
||||
},
|
||||
},
|
||||
|
||||
Generated
+1
-1
@@ -77,7 +77,7 @@ importers:
|
||||
specifier: ^4.0.0
|
||||
version: 4.0.0
|
||||
pm2:
|
||||
specifier: ^6.0.13
|
||||
specifier: 6.0.14
|
||||
version: 6.0.14(supports-color@7.2.0)
|
||||
rss-parser:
|
||||
specifier: ^3.13.0
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Run the same PM2 binary control.js `require("pm2")` uses.
|
||||
* A global `pm2` 7.x talking to an in-memory 6.x daemon pegs CPU on start.
|
||||
*/
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import { createRequire } from "node:module";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const require = createRequire(
|
||||
path.join(root, "packages/server/package.json"),
|
||||
);
|
||||
const pm2Root = path.dirname(require.resolve("pm2/package.json"));
|
||||
const pm2Bin = path.join(pm2Root, "bin/pm2");
|
||||
const args = process.argv.slice(2);
|
||||
|
||||
function run(pm2Args, opts = {}) {
|
||||
return spawnSync(process.execPath, [pm2Bin, ...pm2Args], {
|
||||
cwd: root,
|
||||
encoding: "utf8",
|
||||
...opts,
|
||||
});
|
||||
}
|
||||
|
||||
const cmd = args[0];
|
||||
if (cmd === "start" || cmd === "restart" || cmd === "reload") {
|
||||
const probe = run(["ls"], { stdio: ["ignore", "pipe", "pipe"] });
|
||||
const text = `${probe.stdout ?? ""}${probe.stderr ?? ""}`;
|
||||
const mem = text.match(/In memory PM2 version:\s*(\S+)/);
|
||||
const loc = text.match(/Local PM2 version:\s*(\S+)/);
|
||||
if (mem && loc && mem[1] !== loc[1]) {
|
||||
console.error(
|
||||
`[jflow] PM2 daemon ${mem[1]} != CLI ${loc[1]}. Killing the daemon so control.js and the CLI share one version.`,
|
||||
);
|
||||
run(["kill"], { stdio: "inherit" });
|
||||
}
|
||||
}
|
||||
|
||||
const child = spawn(process.execPath, [pm2Bin, ...args], {
|
||||
cwd: root,
|
||||
stdio: "inherit",
|
||||
});
|
||||
child.on("exit", (code, signal) => {
|
||||
if (signal) process.kill(process.pid, signal);
|
||||
process.exit(code ?? 1);
|
||||
});
|
||||
Reference in New Issue
Block a user