nsrb
0fa0ed3380
Implement parsing for BRI QRIS transactions, including direct and forwarded message handling. Update rules to recognize QRIS success messages and adjust spending record mapping accordingly. Enhance tests to cover new functionality.
103 lines
6.0 KiB
Markdown
103 lines
6.0 KiB
Markdown
# Agent notes — gmail-hook
|
||
|
||
npm workspaces: `apps/api`, `apps/web`, `apps/watcher`. Goal is **e-receipt email → parsed spending → PocketBase**, not a generic Gmail client.
|
||
|
||
## Layout
|
||
|
||
| App | Role |
|
||
|---|---|
|
||
| `apps/api` | Google OAuth, session cookie, Gmail read API. Tokens in SQLite `apps/api/data/app.db`. |
|
||
| `apps/web` | React login / inbox UI (Vite). |
|
||
| `apps/watcher` | Gmail Pub/Sub push consumer. Matches receipts, parses JSON, writes `spendings`. |
|
||
|
||
Shared MIME helpers live in each app (`mime.js`). Watcher reuses the same SQLite tokens as the API.
|
||
|
||
## Watcher pipeline
|
||
|
||
`POST /pubsub/gmail` → `history.list` (`messageAdded`) → per message:
|
||
|
||
1. `logEvent` + `fetchAndLogMessage` — fetch outer mail; collect `emlMessages` if any
|
||
2. **Work items:** if `.eml` / `message/rfc822` attachments exist → process **each attachment only** (outer is a wrapper). If none → process the outer message as usual.
|
||
3. For each work item (sequential, in-process — no job queue): `matchRule` → `parseMatchedMessage` → `saveSpending`
|
||
|
||
Handlers return `{ pass: {...} }` to merge into ctx or `{ break: true }` to stop. **Parse full `message.body`** (and `message.html` when needed), never the 500-char log `bodyPreview`.
|
||
|
||
Attached `.eml` mails use nested From/Subject from Gmail’s expanded `message/rfc822` part. When nested HTML/text is not inlined (`body.data` missing, only `attachmentId`), the watcher fetches those attachments (`hydrateRfc822Message`) before parse. Synthetic `message_id` looks like `outerId#partId`; dedupe remains on `invoice_id`.
|
||
|
||
## Match rules
|
||
|
||
Resolve original sender:
|
||
|
||
1. If body has `---------- Forwarded message ---------`, use its From / Subject / Date.
|
||
2. Else use Gmail envelope.
|
||
|
||
| kind | From address | Subject |
|
||
|---|---|---|
|
||
| `livin.qr_payment.success` | `noreply.livin@bankmandiri.co.id` | `Pembayaran Berhasil!` |
|
||
| `grab.ereceipt.ride` | `no-reply@grab.com` | `Your Grab E-Receipt` |
|
||
| `grab.ereceipt.tip` | same From/Subject as ride; detected from tip body copy | tip invoice_id = `{Booking ID}:tip` |
|
||
| `grab.ereceipt.food` | same From/Subject; body has GrabFood / `Selamat menikmati makanan` | `Pesanan ID` as invoice_id |
|
||
| `bri.transfer.success` | `bankbri@bri.co.id` | `Pemindahan Dana Sesama Rekening BRI` |
|
||
| `bri.qris.success` | `bankbri@bri.co.id` | `Pembelian QRIS Berhasil` |
|
||
|
||
Production path is **direct provider mail**. Forwards still work via the Fwd header. Do **not** match on the forwarder envelope From.
|
||
|
||
Rules: `apps/watcher/src/rules/`. Parsers: `apps/watcher/src/parsers/`.
|
||
|
||
## Parsing notes
|
||
|
||
- **Required for a spending row:** `amount` + `reference` (invoice id). Fail soft (log + break) if missing — never throw (Pub/Sub must ack).
|
||
- **Livin:** support clean forwarded lines and jammed HTML-stripped labels (`Tanggal13 Sep 2026`, `Tanggal15 Agu 2026`). Indonesian month names/abbreviations (`Agu`/`Agustus`, `Des`, …) are accepted. When merchant and city are glued on one line, keep the whole string as `merchant`; only split location when it is already a separate line ending in `- ID`. Do not maintain a city-name list.
|
||
- **Grab:** plain text often has date-only `Picked up on …`; pickup/dropoff times (`10:47AM`) live in HTML. Watcher keeps `message.html` and merges the first standalone AM/PM time into `occurredAt` (WIB). Tip receipts (`Your tip goes a long way…`, timestamp like `13 Sep 26 10:47 +0700`) become `grab.ereceipt.tip` with `invoice_id` `{bookingId}:tip` so they do not collide with the ride.
|
||
- **BRI / BRImo:** transfer (`Pemindahan Dana Sesama Rekening BRI`) — amount from `Nominal`, `invoice_id` from `Nomor Referensi`, `description` from `Nama Tujuan`, datetime from `Tanggal`. QRIS (`Pembelian QRIS Berhasil`) — amount from `Nominal` (fallback `Total Transaksi`), `description` from `Nama Merchant`, datetime from `Tanggal Transaksi`. Other BRI subjects stay out of scope until samples appear.
|
||
- Open-ended text (names, products, driver, compliments) is best-effort optional `details`.
|
||
## PocketBase
|
||
|
||
Env: `POCKETBASE_URL`, `POCKETBASE_USER`, `POCKETBASE_PASSWORD` (a `_superusers` account). Collection: `spendings`.
|
||
|
||
- Unique `invoice_id` — duplicate insert → log skip, no throw.
|
||
- Leave `category` empty; user fills it in PocketBase admin.
|
||
- Store flat columns for querying; `details` for kind leftovers; `parsed` for the full watcher envelope JSON (audit snapshot). Keep `message_id` so the original Gmail message can be re-fetched if needed — do not store raw email body by default.
|
||
- Schema ensure runs on watcher boot (`ensureSpendingsSchema`).
|
||
- Never commit secrets.
|
||
|
||
Parse failures and non-duplicate skips (incomplete record, save errors) POST to ntfy (`NTFY_URL`, default `https://n.0dev.web.id/system`) with `messageId` and message body. Duplicates do not notify.
|
||
|
||
## Run
|
||
|
||
```bash
|
||
npm install
|
||
npm run dev # api + web
|
||
# sign in once via UI so SQLite has tokens
|
||
npm run dev:watcher
|
||
npm test -w watcher # unit tests (no Gmail/Pub/Sub)
|
||
```
|
||
|
||
### Manual / backfill (forwarded old receipts)
|
||
|
||
Gmail Date on a forward is “today”; `trx_date` still comes from the receipt body. Duplicates skip on unique `invoice_id`. Forward-as-attachment (multiple `*.eml`) is expanded the same way as the watcher.
|
||
|
||
```bash
|
||
# List id / from / subject (from Gmail API)
|
||
npm run list-messages --
|
||
npm run list-messages -- --query 'newer_than:14d has:attachment filename:eml' --max 100
|
||
|
||
# Dry-run last 30 days of inbox (default query: in:inbox newer_than:30d)
|
||
npm run process-messages -- --dry-run
|
||
|
||
# Process a wrapper mail that has .eml attachments
|
||
npm run process-messages -- --ids OUTER_MESSAGE_ID
|
||
|
||
# Process and save
|
||
npm run process-messages -- --query 'newer_than:2d subject:"Pembayaran Berhasil!"'
|
||
npm run process-messages -- --ids MESSAGE_ID_1,MESSAGE_ID_2
|
||
npm run process-messages -- --user eleven16th@gmail.com --max 100
|
||
```
|
||
|
||
## Adding a provider
|
||
|
||
1. Rule module under `src/rules/` (kind, fromAddress, subject, source).
|
||
2. Parser under `src/parsers/`; register in `parse-matched-message.js`.
|
||
3. Fixture + tests under `test/`.
|
||
4. Map into spendings in `pocketbase/map.js` (`source`, `description`, `details`, `parsed`).
|