nsrb/gmail-hook
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: refresh agent playbooks and project README

Browse Source 
Point agents at the CLI safety rules, mailbox defaults, and the split usage/development guides.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
nsrb
2026-09-22 10:09:25 +07:00
co-authored by Cursor
parent 427636b59b
commit ec3b683786
2 changed files with 76 additions and 89 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+43 -79
View File
@@ -1,104 +1,68 @@
# Agent notes — gmail-hook # Agent notes — gmail-hook
npm workspaces: `apps/api`, `apps/web`, `apps/watcher`. Goal is **e-receipt email → parsed spending → PocketBase**, not a generic Gmail client. npm workspaces: `apps/api`, `apps/web`, `apps/watcher`.
Two jobs:
1. **E-receipt pipeline** — Gmail Pub/Sub → match/parse → PocketBase `spendings`.
2. **Agent Gmail toolbox** — CLI to list/read/summarize, archive, label, and create drafts (never send).
## Layout ## Layout
| App | Role | | App | Role |
|---|---| |---|---|
| `apps/api` | Google OAuth, session cookie, Gmail read API. Tokens in SQLite `apps/api/data/app.db`. | | `apps/api` | Google OAuth (`gmail.modify`), session cookie, read-only Gmail HTTP API. Tokens in SQLite `apps/api/data/app.db`. |
| `apps/web` | React login / inbox UI (Vite). | | `apps/web` | React login / inbox UI (Vite). UI is read-only; mutations are CLI-only. |
| `apps/watcher` | Gmail Pub/Sub push consumer. Matches receipts, parses JSON, writes `spendings`. | | `apps/watcher` | Pub/Sub receipt consumer (gated by `WATCHER_MAILBOXES`) + `npm run gmail` CLI. Reuses the same SQLite tokens. |
Shared MIME helpers live in each app (`mime.js`). Watcher reuses the same SQLite tokens as the API. ## Safety (CLI)
## Watcher pipeline - **Never send** (`messages.send` / `drafts.send` must not exist in this repo).
- **Never trash/delete**.
- Mutating commands (`label`, `archive`, `draft`, `forward`) **require `--reason`**.
- Confirm with the user before bulk archive (e.g. more than ~20 ids).
- Google’s consent screen for `gmail.modify` says the app can send/delete; that is Google’s wording — the app still must not.
`POST /pubsub/gmail` → `history.list` (`messageAdded`) → per message: ## Default mailbox
1. `logEvent` + `fetchAndLogMessage` — fetch outer mail; collect `emlMessages` if any Resolution order for every script (`--user` optional):
2. **Work items:** if `.eml` / `message/rfc822` attachments exist → process **each attachment only** (outer is a wrapper). If none → process the outer message as usual.
3. For each work item (sequential, in-process — no job queue): `matchRule` → `parseMatchedMessage` → `saveSpending`
Handlers return `{ pass: {...} }` to merge into ctx or `{ break: true }` to stop. **Parse full `message.body`** (and `message.html` when needed), never the 500-char log `bodyPreview`. 1. `--user <email|id>`
2. `GMAIL_DEFAULT_USER` env
3. `users.is_default = 1`
4. Sole connected account
5. Else fail — set default: `npm run gmail -- accounts --default you@gmail.com`
Attached `.eml` mails use nested From/Subject from Gmail’s expanded `message/rfc822` part. When nested HTML/text is not inlined (`body.data` missing, only `attachmentId`), the watcher fetches those attachments (`hydrateRfc822Message`) before parse. Synthetic `message_id` looks like `outerId#partId`; dedupe remains on `invoice_id`. ## CLI cheat sheet
## Match rules ```bash
npm run gmail -- accounts
npm run gmail -- accounts --default you@gmail.com
npm run gmail -- list --query 'is:unread newer_than:7d' --max 50
npm run gmail -- read MESSAGE_ID
npm run gmail -- summarize --query 'in:inbox newer_than:7d' --max 20
npm run gmail -- labels
npm run gmail -- label ID1,ID2 --add receipts --reason "tag receipts"
npm run gmail -- archive ID1,ID2 --reason "newsletter noise"
npm run gmail -- draft --to a@b.com --subject "Hi" --body "…" --reason "reply draft"
npm run gmail -- forward MESSAGE_ID --to a@b.com --note "FYI" --reason "forward to accounting"
npm run gmail -- audit --limit 50
npm run process-messages -- --query 'newer_than:2d' --dry-run
```
Resolve original sender: Shared flags: `--user`, `--json`, `--dry-run` (mutations: no Gmail write; audit `status=dry-run`).
1. If body has `---------- Forwarded message ---------`, use its From / Subject / Date. ## Docs
2. Else use Gmail envelope.
| kind | From address | Subject | - [docs/usage.md](docs/usage.md) — how agents use Gmail (commands, audit, recipes, ideas).
|---|---|---| - [docs/development.md](docs/development.md) — receipt pipeline, parsers, PocketBase, Google scope change, adding providers.
| `livin.qr_payment.success` | `noreply.livin@bankmandiri.co.id` | `Pembayaran Berhasil!` |
| `grab.ereceipt.ride` | `no-reply@grab.com` | `Your Grab E-Receipt` |
| `grab.ereceipt.tip` | same From/Subject as ride; detected from tip body copy | tip invoice_id = `{Booking ID}:tip` |
| `grab.ereceipt.food` | same From/Subject; body has GrabFood / `Selamat menikmati makanan` | `Pesanan ID` as invoice_id |
| `bri.transfer.success` | `bankbri@bri.co.id` | `Pemindahan Dana Sesama Rekening BRI` |
| `bri.qris.success` | `bankbri@bri.co.id` | `Pembelian QRIS Berhasil` |
| `tokopedia.order.completed` | `noreply@tokopedia.com` | `Pesanan Selesai` |
Production path is **direct provider mail**. Forwards still work via the Fwd header. Do **not** match on the forwarder envelope From.
Rules: `apps/watcher/src/rules/`. Parsers: `apps/watcher/src/parsers/`.
## Parsing notes
- **Required for a spending row:** `amount` + `reference` (invoice id). Fail soft (log + break) if missing — never throw (Pub/Sub must ack).
- **Livin:** support clean forwarded lines and jammed HTML-stripped labels (`Tanggal13 Sep 2026`, `Tanggal15 Agu 2026`). Indonesian month names/abbreviations (`Agu`/`Agustus`, `Des`, …) are accepted. When merchant and city are glued on one line, keep the whole string as `merchant`; only split location when it is already a separate line ending in `- ID`. Do not maintain a city-name list.
- **Grab:** plain text often has date-only `Picked up on …`; pickup/dropoff times (`10:47AM`) live in HTML. Watcher keeps `message.html` and merges the first standalone AM/PM time into `occurredAt` (WIB). Tip receipts (`Your tip goes a long way…`, timestamp like `13 Sep 26 10:47 +0700`) become `grab.ereceipt.tip` with `invoice_id` `{bookingId}:tip` so they do not collide with the ride.
- **BRI / BRImo:** transfer (`Pemindahan Dana Sesama Rekening BRI`) — amount from `Nominal`, `invoice_id` from `Nomor Referensi`, `description` from `Nama Tujuan`, datetime from `Tanggal`. QRIS (`Pembelian QRIS Berhasil`) — amount from `Nominal` (fallback `Total Transaksi`), `description` from `Nama Merchant`, datetime from `Tanggal Transaksi`. Other BRI subjects stay out of scope until samples appear.
- **Tokopedia:** `Pesanan Selesai` — one row per `No.Invoice`; amount from `Total belanja` (not fee lines); `description` = `{Toko} - Tokopedia`; line items (1+) in `details.items`; `trx_date` from `Tanggal Terima` (date-only).
- Open-ended text (names, products, driver, compliments) is best-effort optional `details`.
## PocketBase
Env: `POCKETBASE_URL`, `POCKETBASE_USER`, `POCKETBASE_PASSWORD` (a `_superusers` account). Collection: `spendings`.
- Unique `invoice_id` — duplicate insert → log skip, no throw.
- Leave `category` empty; user fills it in PocketBase admin.
- Store flat columns for querying; `details` for kind leftovers; `parsed` for the full watcher envelope JSON (audit snapshot). Keep `message_id` so the original Gmail message can be re-fetched if needed — do not store raw email body by default.
- Schema ensure runs on watcher boot (`ensureSpendingsSchema`).
- Never commit secrets.
Parse failures and non-duplicate skips (incomplete record, save errors) POST to ntfy (`NTFY_URL`, default `https://n.0dev.web.id/system`) with `messageId` and message body. Duplicates do not notify.
## Run ## Run
```bash ```bash
npm install npm install
npm run dev # api + web npm run dev # api + web
# sign in once via UI so SQLite has tokens # sign in once per Gmail account (re-consent after scope change)
npm run dev:watcher npm run dev:watcher
npm test -w watcher # unit tests (no Gmail/Pub/Sub) npm test -w watcher
``` ```
### Manual / backfill (forwarded old receipts)
Gmail Date on a forward is “today”; `trx_date` still comes from the receipt body. Duplicates skip on unique `invoice_id`. Forward-as-attachment (multiple `*.eml`) is expanded the same way as the watcher.
```bash
# List id / from / subject (from Gmail API)
npm run list-messages --
npm run list-messages -- --query 'newer_than:14d has:attachment filename:eml' --max 100
# Dry-run last 30 days of inbox (default query: in:inbox newer_than:30d)
npm run process-messages -- --dry-run
# Process a wrapper mail that has .eml attachments
npm run process-messages -- --ids OUTER_MESSAGE_ID
# Process and save
npm run process-messages -- --query 'newer_than:2d subject:"Pembayaran Berhasil!"'
npm run process-messages -- --ids MESSAGE_ID_1,MESSAGE_ID_2
npm run process-messages -- --user eleven16th@gmail.com --max 100
```
## Adding a provider
1. Rule module under `src/rules/` (kind, fromAddress, subject, source).
2. Parser under `src/parsers/`; register in `parse-matched-message.js`.
3. Fixture + tests under `test/`.
4. Map into spendings in `pocketbase/map.js` (`source`, `description`, `details`, `parsed`).
README.md
+33 -10
View File
@@ -1,17 +1,31 @@
# Gmail Reader # Gmail Reader
Monorepo with a React + Vite + DaisyUI client, a Fastify API, and a Gmail push watcher. The API owns Google OAuth, stores tokens in SQLite, and calls the Gmail API with `gmail.readonly`. The browser only receives an HTTP-only session cookie. The watcher reuses those tokens, calls `users.watch`, and runs each new INBOX message through a handler pipeline. Monorepo with a React + Vite + DaisyUI client, a Fastify API, and a Gmail push watcher. The API owns Google OAuth, stores tokens in SQLite, and calls the Gmail API with `gmail.modify`. The browser only receives an HTTP-only session cookie (read-only inbox UI). The watcher reuses those tokens for Pub/Sub receipt processing and for the agent CLI (`npm run gmail`).
Agent playbooks: [AGENTS.md](AGENTS.md), [docs/usage.md](docs/usage.md), [docs/development.md](docs/development.md).
## Prerequisites ## Prerequisites
- Node.js 20+ - Node.js 20+
- A Google Cloud OAuth **Web application** client with: - A Google Cloud OAuth **Web application** client with:
- Scope: `https://www.googleapis.com/auth/gmail.readonly` - Scope: `https://www.googleapis.com/auth/gmail.modify`
- Authorized JavaScript origin: `https://oauth.0dev.web.id` (and `http://localhost:5173` for local) - Authorized JavaScript origin: `https://oauth.0dev.web.id` (and `http://localhost:5173` for local)
- Authorized redirect URIs: - Authorized redirect URIs:
- `https://oauth.0dev.web.id/callback` - `https://oauth.0dev.web.id/callback`
- `http://localhost:5173/callback` (local; Vite proxies this to Fastify) - `http://localhost:5173/callback` (local; Vite proxies this to Fastify)
### Change Google API permissions
If you previously used `gmail.readonly`, update the OAuth consent screen and re-consent:
1. **APIs & Services → Library**: Gmail API enabled.
2. **OAuth consent screen** (or **Google Auth Platform → Data Access**): add `https://www.googleapis.com/auth/gmail.modify`; remove `gmail.readonly` if listed.
3. While the app is **External + Testing**, list every mailbox under **Test users** (`gmail.modify` is sensitive).
4. Keep the same Client ID / secret and redirect URIs.
5. Restart the API and **sign in once per Gmail account** via the web UI so refresh tokens pick up the new scope.
Full detail: [docs/development.md](docs/development.md#change-google-api-permissions-gmailmodify).
## Local development ## Local development
```bash ```bash
@@ -37,7 +51,17 @@ npm run dev
Vite proxies `/auth`, `/callback`, and `/api` to Fastify so the Google redirect stays on the same origin as the SPA. Vite proxies `/auth`, `/callback`, and `/api` to Fastify so the Google redirect stays on the same origin as the SPA.
Sign in via the UI at least once so SQLite has a refresh token before starting the watcher. Sign in via the UI at least once so SQLite has a refresh token before starting the watcher or CLI.
### Agent CLI (optional)
```bash
npm run gmail -- accounts
npm run gmail -- list --query 'newer_than:7d'
npm run gmail -- archive MESSAGE_ID --reason "noise"
```
See [docs/usage.md](docs/usage.md). Mutating commands require `--reason`. The CLI never sends mail.
## Production (one process) ## Production (one process)
@@ -92,13 +116,12 @@ oauth.0dev.web.id {
} }
``` ```
The watcher renews `users.watch` on startup and every 12 hours. Handlers run in sequence per new message: log the event, then fetch and log the message body. A handler may pass fields to the next one or `break` the chain. The watcher renews `users.watch` on startup and every 12 hours. New INBOX mail runs through match → parse → PocketBase (`spendings`). Details: [docs/development.md](docs/development.md).
## What it does ## What it does
- Sign in with Google (server-side authorization code flow) - Sign in with Google (server-side authorization code flow; `gmail.modify`)
- List latest mail - List / search / read mail in the web UI (read-only)
- Search with Gmail query syntax (`from:`, `subject:`, `newer_than:`, …) - Multi-account tokens in SQLite with a configurable default mailbox
- Open a message and read decoded plain-text content - Agent CLI: list, read, summarize, labels, archive, label, draft, forward-as-draft, audit log
- Watch INBOX via Gmail push (Pub/Sub) and run a sequential handler pipeline (log event, then fetch and log the message) - Watch INBOX via Gmail push (Pub/Sub) and parse e-receipts into PocketBase
- Watch INBOX via Gmail push (Pub/Sub) and run a sequential handler pipeline (log event, then fetch and log the message)