nsrb
c446d794a8
Implement parsing for Tokopedia "Pesanan Selesai" messages, including support for single and multi-item orders. Update rules to recognize Tokopedia order completion and enhance spending record mapping. Add tests to validate new functionality.
105 lines
6.3 KiB
Markdown
105 lines
6.3 KiB
Markdown
# Agent notes — gmail-hook
|
||
|
||
npm workspaces: `apps/api`, `apps/web`, `apps/watcher`. Goal is **e-receipt email → parsed spending → PocketBase**, not a generic Gmail client.
|
||
|
||
## Layout
|
||
|
||
| App | Role |
|
||
|---|---|
|
||
| `apps/api` | Google OAuth, session cookie, Gmail read API. Tokens in SQLite `apps/api/data/app.db`. |
|
||
| `apps/web` | React login / inbox UI (Vite). |
|
||
| `apps/watcher` | Gmail Pub/Sub push consumer. Matches receipts, parses JSON, writes `spendings`. |
|
||
|
||
Shared MIME helpers live in each app (`mime.js`). Watcher reuses the same SQLite tokens as the API.
|
||
|
||
## Watcher pipeline
|
||
|
||
`POST /pubsub/gmail` → `history.list` (`messageAdded`) → per message:
|
||
|
||
1. `logEvent` + `fetchAndLogMessage` — fetch outer mail; collect `emlMessages` if any
|
||
2. **Work items:** if `.eml` / `message/rfc822` attachments exist → process **each attachment only** (outer is a wrapper). If none → process the outer message as usual.
|
||
3. For each work item (sequential, in-process — no job queue): `matchRule` → `parseMatchedMessage` → `saveSpending`
|
||
|
||
Handlers return `{ pass: {...} }` to merge into ctx or `{ break: true }` to stop. **Parse full `message.body`** (and `message.html` when needed), never the 500-char log `bodyPreview`.
|
||
|
||
Attached `.eml` mails use nested From/Subject from Gmail’s expanded `message/rfc822` part. When nested HTML/text is not inlined (`body.data` missing, only `attachmentId`), the watcher fetches those attachments (`hydrateRfc822Message`) before parse. Synthetic `message_id` looks like `outerId#partId`; dedupe remains on `invoice_id`.
|
||
|
||
## Match rules
|
||
|
||
Resolve original sender:
|
||
|
||
1. If body has `---------- Forwarded message ---------`, use its From / Subject / Date.
|
||
2. Else use Gmail envelope.
|
||
|
||
| kind | From address | Subject |
|
||
|---|---|---|
|
||
| `livin.qr_payment.success` | `noreply.livin@bankmandiri.co.id` | `Pembayaran Berhasil!` |
|
||
| `grab.ereceipt.ride` | `no-reply@grab.com` | `Your Grab E-Receipt` |
|
||
| `grab.ereceipt.tip` | same From/Subject as ride; detected from tip body copy | tip invoice_id = `{Booking ID}:tip` |
|
||
| `grab.ereceipt.food` | same From/Subject; body has GrabFood / `Selamat menikmati makanan` | `Pesanan ID` as invoice_id |
|
||
| `bri.transfer.success` | `bankbri@bri.co.id` | `Pemindahan Dana Sesama Rekening BRI` |
|
||
| `bri.qris.success` | `bankbri@bri.co.id` | `Pembelian QRIS Berhasil` |
|
||
| `tokopedia.order.completed` | `noreply@tokopedia.com` | `Pesanan Selesai` |
|
||
|
||
Production path is **direct provider mail**. Forwards still work via the Fwd header. Do **not** match on the forwarder envelope From.
|
||
|
||
Rules: `apps/watcher/src/rules/`. Parsers: `apps/watcher/src/parsers/`.
|
||
|
||
## Parsing notes
|
||
|
||
- **Required for a spending row:** `amount` + `reference` (invoice id). Fail soft (log + break) if missing — never throw (Pub/Sub must ack).
|
||
- **Livin:** support clean forwarded lines and jammed HTML-stripped labels (`Tanggal13 Sep 2026`, `Tanggal15 Agu 2026`). Indonesian month names/abbreviations (`Agu`/`Agustus`, `Des`, …) are accepted. When merchant and city are glued on one line, keep the whole string as `merchant`; only split location when it is already a separate line ending in `- ID`. Do not maintain a city-name list.
|
||
- **Grab:** plain text often has date-only `Picked up on …`; pickup/dropoff times (`10:47AM`) live in HTML. Watcher keeps `message.html` and merges the first standalone AM/PM time into `occurredAt` (WIB). Tip receipts (`Your tip goes a long way…`, timestamp like `13 Sep 26 10:47 +0700`) become `grab.ereceipt.tip` with `invoice_id` `{bookingId}:tip` so they do not collide with the ride.
|
||
- **BRI / BRImo:** transfer (`Pemindahan Dana Sesama Rekening BRI`) — amount from `Nominal`, `invoice_id` from `Nomor Referensi`, `description` from `Nama Tujuan`, datetime from `Tanggal`. QRIS (`Pembelian QRIS Berhasil`) — amount from `Nominal` (fallback `Total Transaksi`), `description` from `Nama Merchant`, datetime from `Tanggal Transaksi`. Other BRI subjects stay out of scope until samples appear.
|
||
- **Tokopedia:** `Pesanan Selesai` — one row per `No.Invoice`; amount from `Total belanja` (not fee lines); `description` = `{Toko} - Tokopedia`; line items (1+) in `details.items`; `trx_date` from `Tanggal Terima` (date-only).
|
||
- Open-ended text (names, products, driver, compliments) is best-effort optional `details`.
|
||
## PocketBase
|
||
|
||
Env: `POCKETBASE_URL`, `POCKETBASE_USER`, `POCKETBASE_PASSWORD` (a `_superusers` account). Collection: `spendings`.
|
||
|
||
- Unique `invoice_id` — duplicate insert → log skip, no throw.
|
||
- Leave `category` empty; user fills it in PocketBase admin.
|
||
- Store flat columns for querying; `details` for kind leftovers; `parsed` for the full watcher envelope JSON (audit snapshot). Keep `message_id` so the original Gmail message can be re-fetched if needed — do not store raw email body by default.
|
||
- Schema ensure runs on watcher boot (`ensureSpendingsSchema`).
|
||
- Never commit secrets.
|
||
|
||
Parse failures and non-duplicate skips (incomplete record, save errors) POST to ntfy (`NTFY_URL`, default `https://n.0dev.web.id/system`) with `messageId` and message body. Duplicates do not notify.
|
||
|
||
## Run
|
||
|
||
```bash
|
||
npm install
|
||
npm run dev # api + web
|
||
# sign in once via UI so SQLite has tokens
|
||
npm run dev:watcher
|
||
npm test -w watcher # unit tests (no Gmail/Pub/Sub)
|
||
```
|
||
|
||
### Manual / backfill (forwarded old receipts)
|
||
|
||
Gmail Date on a forward is “today”; `trx_date` still comes from the receipt body. Duplicates skip on unique `invoice_id`. Forward-as-attachment (multiple `*.eml`) is expanded the same way as the watcher.
|
||
|
||
```bash
|
||
# List id / from / subject (from Gmail API)
|
||
npm run list-messages --
|
||
npm run list-messages -- --query 'newer_than:14d has:attachment filename:eml' --max 100
|
||
|
||
# Dry-run last 30 days of inbox (default query: in:inbox newer_than:30d)
|
||
npm run process-messages -- --dry-run
|
||
|
||
# Process a wrapper mail that has .eml attachments
|
||
npm run process-messages -- --ids OUTER_MESSAGE_ID
|
||
|
||
# Process and save
|
||
npm run process-messages -- --query 'newer_than:2d subject:"Pembayaran Berhasil!"'
|
||
npm run process-messages -- --ids MESSAGE_ID_1,MESSAGE_ID_2
|
||
npm run process-messages -- --user eleven16th@gmail.com --max 100
|
||
```
|
||
|
||
## Adding a provider
|
||
|
||
1. Rule module under `src/rules/` (kind, fromAddress, subject, source).
|
||
2. Parser under `src/parsers/`; register in `parse-matched-message.js`.
|
||
3. Fixture + tests under `test/`.
|
||
4. Map into spendings in `pocketbase/map.js` (`source`, `description`, `details`, `parsed`).
|