feat: add gmail-hook monorepo with watcher, api, and web
Ship the receipt pipeline (Livin/Grab/BRI), PocketBase spendings, PM2 ecosystem, and a proper .gitignore that excludes secrets, SQLite data, and media dumps. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -0,0 +1,35 @@
|
||||
# Google OAuth Web client (gmail.readonly)
|
||||
GOOGLE_CLIENT_ID=
|
||||
GOOGLE_CLIENT_SECRET=
|
||||
|
||||
# Must match an Authorized redirect URI in Google Cloud Console.
|
||||
# Local (Vite proxies /callback to Fastify):
|
||||
GOOGLE_REDIRECT_URI=http://localhost:5173/callback
|
||||
# Production:
|
||||
# GOOGLE_REDIRECT_URI=https://oauth.0dev.web.id/callback
|
||||
|
||||
# 32+ random characters. Used to derive the session cookie key.
|
||||
SESSION_SECRET=change-me-to-a-long-random-string
|
||||
|
||||
PORT=3000
|
||||
NODE_ENV=development
|
||||
|
||||
# Watcher (apps/watcher) — Gmail Pub/Sub push
|
||||
WATCHER_PORT=3001
|
||||
# Full topic name, e.g. projects/my-gcp-project/topics/gmail-push
|
||||
# A short name like gmail-push is expanded if GOOGLE_CLOUD_PROJECT is set.
|
||||
GOOGLE_PUBSUB_TOPIC=
|
||||
# GOOGLE_CLOUD_PROJECT=
|
||||
# Shared secret. Push URL: https://host/pubsub/gmail?token=...
|
||||
PUBSUB_VERIFICATION_TOKEN=
|
||||
# Defaults to apps/api/data/app.db (same OAuth tokens as the web app)
|
||||
# SQLITE_PATH=
|
||||
|
||||
# PocketBase (watcher writes parsed receipts to collection `spendings`)
|
||||
# Use a _superusers account (admin), not a users-collection login.
|
||||
POCKETBASE_URL=
|
||||
POCKETBASE_USER=
|
||||
POCKETBASE_PASSWORD=
|
||||
|
||||
# ntfy alerts for parse failures / non-duplicate skips (default: system topic)
|
||||
# NTFY_URL=https://n.0dev.web.id/system
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
# Dependencies
|
||||
node_modules/
|
||||
|
||||
# Build output
|
||||
dist/
|
||||
build/
|
||||
*.tsbuildinfo
|
||||
|
||||
# Env / secrets (keep example)
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
|
||||
# Local SQLite / runtime data
|
||||
apps/api/data/
|
||||
*.db
|
||||
*.db-wal
|
||||
*.db-shm
|
||||
|
||||
# Logs
|
||||
*.log
|
||||
npm-debug.log*
|
||||
pnpm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
|
||||
# OS / editor
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# Test / coverage
|
||||
coverage/
|
||||
.nyc_output/
|
||||
|
||||
# Accidental media dumps
|
||||
*.mp4
|
||||
*.mov
|
||||
*.webm
|
||||
*.mkv
|
||||
|
||||
# PM2 local dumps (process list lives on the host)
|
||||
.pm2/
|
||||
@@ -0,0 +1,102 @@
|
||||
# Agent notes — gmail-hook
|
||||
|
||||
npm workspaces: `apps/api`, `apps/web`, `apps/watcher`. Goal is **e-receipt email → parsed spending → PocketBase**, not a generic Gmail client.
|
||||
|
||||
## Layout
|
||||
|
||||
| App | Role |
|
||||
|---|---|
|
||||
| `apps/api` | Google OAuth, session cookie, Gmail read API. Tokens in SQLite `apps/api/data/app.db`. |
|
||||
| `apps/web` | React login / inbox UI (Vite). |
|
||||
| `apps/watcher` | Gmail Pub/Sub push consumer. Matches receipts, parses JSON, writes `spendings`. |
|
||||
|
||||
Shared MIME helpers live in each app (`mime.js`). Watcher reuses the same SQLite tokens as the API.
|
||||
|
||||
## Watcher pipeline
|
||||
|
||||
`POST /pubsub/gmail` → `history.list` (`messageAdded`) → per message:
|
||||
|
||||
1. `logEvent` + `fetchAndLogMessage` — fetch outer mail; collect `emlMessages` if any
|
||||
2. **Work items:** if `.eml` / `message/rfc822` attachments exist → process **each attachment only** (outer is a wrapper). If none → process the outer message as usual.
|
||||
3. For each work item (sequential, in-process — no job queue): `matchRule` → `parseMatchedMessage` → `saveSpending`
|
||||
|
||||
Handlers return `{ pass: {...} }` to merge into ctx or `{ break: true }` to stop. **Parse full `message.body`** (and `message.html` when needed), never the 500-char log `bodyPreview`.
|
||||
|
||||
Attached `.eml` mails use nested From/Subject from Gmail’s expanded `message/rfc822` part. When nested HTML/text is not inlined (`body.data` missing, only `attachmentId`), the watcher fetches those attachments (`hydrateRfc822Message`) before parse. Synthetic `message_id` looks like `outerId#partId`; dedupe remains on `invoice_id`.
|
||||
|
||||
## Match rules
|
||||
|
||||
Resolve original sender:
|
||||
|
||||
1. If body has `---------- Forwarded message ---------`, use its From / Subject / Date.
|
||||
2. Else use Gmail envelope.
|
||||
|
||||
| kind | From address | Subject |
|
||||
|---|---|---|
|
||||
| `livin.qr_payment.success` | `noreply.livin@bankmandiri.co.id` | `Pembayaran Berhasil!` |
|
||||
| `grab.ereceipt.ride` | `no-reply@grab.com` | `Your Grab E-Receipt` |
|
||||
| `grab.ereceipt.tip` | same From/Subject as ride; detected from tip body copy | tip invoice_id = `{Booking ID}:tip` |
|
||||
| `grab.ereceipt.food` | same From/Subject; body has GrabFood / `Selamat menikmati makanan` | `Pesanan ID` as invoice_id |
|
||||
| `bri.transfer.success` | `bankbri@bri.co.id` | `Pemindahan Dana Sesama Rekening BRI` |
|
||||
|
||||
Production path is **direct provider mail**. Forwards still work via the Fwd header. Do **not** match on the forwarder envelope From.
|
||||
|
||||
Rules: `apps/watcher/src/rules/`. Parsers: `apps/watcher/src/parsers/`.
|
||||
|
||||
## Parsing notes
|
||||
|
||||
- **Required for a spending row:** `amount` + `reference` (invoice id). Fail soft (log + break) if missing — never throw (Pub/Sub must ack).
|
||||
- **Livin:** support clean forwarded lines and jammed HTML-stripped labels (`Tanggal13 Sep 2026`, `Tanggal15 Agu 2026`). Indonesian month names/abbreviations (`Agu`/`Agustus`, `Des`, …) are accepted. When merchant and city are glued on one line, keep the whole string as `merchant`; only split location when it is already a separate line ending in `- ID`. Do not maintain a city-name list.
|
||||
- **Grab:** plain text often has date-only `Picked up on …`; pickup/dropoff times (`10:47AM`) live in HTML. Watcher keeps `message.html` and merges the first standalone AM/PM time into `occurredAt` (WIB). Tip receipts (`Your tip goes a long way…`, timestamp like `13 Sep 26 10:47 +0700`) become `grab.ereceipt.tip` with `invoice_id` `{bookingId}:tip` so they do not collide with the ride.
|
||||
- **BRI / BRImo:** `Pemindahan Dana Sesama Rekening BRI` — amount from `Nominal` (not `Total`), `invoice_id` from `Nomor Referensi`, `description` from `Nama Tujuan`, datetime from combined `Tanggal` (`13 September 2026 , 19:00:00 WIB`). Other BRI subjects are out of scope until samples appear.
|
||||
- Open-ended text (names, products, driver, compliments) is best-effort optional `details`.
|
||||
|
||||
## PocketBase
|
||||
|
||||
Env: `POCKETBASE_URL`, `POCKETBASE_USER`, `POCKETBASE_PASSWORD` (a `_superusers` account). Collection: `spendings`.
|
||||
|
||||
- Unique `invoice_id` — duplicate insert → log skip, no throw.
|
||||
- Leave `category` empty; user fills it in PocketBase admin.
|
||||
- Store flat columns for querying; `details` for kind leftovers; `parsed` for the full watcher envelope JSON (audit snapshot). Keep `message_id` so the original Gmail message can be re-fetched if needed — do not store raw email body by default.
|
||||
- Schema ensure runs on watcher boot (`ensureSpendingsSchema`).
|
||||
- Never commit secrets.
|
||||
|
||||
Parse failures and non-duplicate skips (incomplete record, save errors) POST to ntfy (`NTFY_URL`, default `https://n.0dev.web.id/system`) with `messageId` and message body. Duplicates do not notify.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npm run dev # api + web
|
||||
# sign in once via UI so SQLite has tokens
|
||||
npm run dev:watcher
|
||||
npm test -w watcher # unit tests (no Gmail/Pub/Sub)
|
||||
```
|
||||
|
||||
### Manual / backfill (forwarded old receipts)
|
||||
|
||||
Gmail Date on a forward is “today”; `trx_date` still comes from the receipt body. Duplicates skip on unique `invoice_id`. Forward-as-attachment (multiple `*.eml`) is expanded the same way as the watcher.
|
||||
|
||||
```bash
|
||||
# List id / from / subject (from Gmail API)
|
||||
npm run list-messages --
|
||||
npm run list-messages -- --query 'newer_than:14d has:attachment filename:eml' --max 100
|
||||
|
||||
# Dry-run recent Livin/Grab subjects (default query: newer_than:7d)
|
||||
npm run process-messages -- --dry-run
|
||||
|
||||
# Process a wrapper mail that has .eml attachments
|
||||
npm run process-messages -- --ids OUTER_MESSAGE_ID
|
||||
|
||||
# Process and save
|
||||
npm run process-messages -- --query 'newer_than:2d subject:"Pembayaran Berhasil!"'
|
||||
npm run process-messages -- --ids MESSAGE_ID_1,MESSAGE_ID_2
|
||||
npm run process-messages -- --user eleven16th@gmail.com --max 100
|
||||
```
|
||||
|
||||
## Adding a provider
|
||||
|
||||
1. Rule module under `src/rules/` (kind, fromAddress, subject, source).
|
||||
2. Parser under `src/parsers/`; register in `parse-matched-message.js`.
|
||||
3. Fixture + tests under `test/`.
|
||||
4. Map into spendings in `pocketbase/map.js` (`source`, `description`, `details`, `parsed`).
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"name": "api",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "node --watch src/index.js",
|
||||
"start": "node src/index.js"
|
||||
},
|
||||
"dependencies": {
|
||||
"@fastify/secure-session": "^8.2.0",
|
||||
"@fastify/static": "^8.2.0",
|
||||
"better-sqlite3": "^12.2.0",
|
||||
"dotenv": "^17.2.2",
|
||||
"fastify": "^5.6.0",
|
||||
"googleapis": "^159.0.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
import { config } from "dotenv";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
export const rootDir = resolve(__dirname, "../../..");
|
||||
export const apiDir = resolve(__dirname, "..");
|
||||
export const webDistDir = resolve(__dirname, "../../web/dist");
|
||||
|
||||
config({ path: resolve(rootDir, ".env"), quiet: true });
|
||||
|
||||
export const PORT = Number(process.env.PORT || 3000);
|
||||
export const NODE_ENV = process.env.NODE_ENV || "development";
|
||||
export const GOOGLE_CLIENT_ID = process.env.GOOGLE_CLIENT_ID || "";
|
||||
export const GOOGLE_CLIENT_SECRET = process.env.GOOGLE_CLIENT_SECRET || "";
|
||||
export const GOOGLE_REDIRECT_URI =
|
||||
process.env.GOOGLE_REDIRECT_URI || "http://localhost:5173/callback";
|
||||
export const SESSION_SECRET = process.env.SESSION_SECRET || "";
|
||||
|
||||
export const GMAIL_SCOPE = "https://www.googleapis.com/auth/gmail.readonly";
|
||||
export const OAUTH_SCOPES = [
|
||||
GMAIL_SCOPE,
|
||||
"openid",
|
||||
"https://www.googleapis.com/auth/userinfo.email",
|
||||
"https://www.googleapis.com/auth/userinfo.profile",
|
||||
];
|
||||
|
||||
export function assertConfig() {
|
||||
const missing = [];
|
||||
if (!GOOGLE_CLIENT_ID) missing.push("GOOGLE_CLIENT_ID");
|
||||
if (!GOOGLE_CLIENT_SECRET) missing.push("GOOGLE_CLIENT_SECRET");
|
||||
if (!SESSION_SECRET || SESSION_SECRET.length < 16) {
|
||||
missing.push("SESSION_SECRET (16+ characters)");
|
||||
}
|
||||
if (missing.length) {
|
||||
throw new Error(`Missing required env: ${missing.join(", ")}`);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
import { mkdirSync } from "node:fs";
|
||||
import { resolve } from "node:path";
|
||||
import Database from "better-sqlite3";
|
||||
import { apiDir } from "./config.js";
|
||||
|
||||
const dataDir = resolve(apiDir, "data");
|
||||
mkdirSync(dataDir, { recursive: true });
|
||||
|
||||
const db = new Database(resolve(dataDir, "app.db"));
|
||||
db.pragma("journal_mode = WAL");
|
||||
db.pragma("foreign_keys = ON");
|
||||
|
||||
db.exec(`
|
||||
CREATE TABLE IF NOT EXISTS users (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
google_sub TEXT NOT NULL UNIQUE,
|
||||
email TEXT NOT NULL,
|
||||
name TEXT,
|
||||
picture TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS oauth_tokens (
|
||||
user_id INTEGER PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
|
||||
access_token TEXT NOT NULL,
|
||||
refresh_token TEXT,
|
||||
expiry INTEGER NOT NULL
|
||||
);
|
||||
`);
|
||||
|
||||
const upsertUserStmt = db.prepare(`
|
||||
INSERT INTO users (google_sub, email, name, picture)
|
||||
VALUES (@googleSub, @email, @name, @picture)
|
||||
ON CONFLICT(google_sub) DO UPDATE SET
|
||||
email = excluded.email,
|
||||
name = excluded.name,
|
||||
picture = excluded.picture
|
||||
RETURNING *
|
||||
`);
|
||||
|
||||
const saveTokensStmt = db.prepare(`
|
||||
INSERT INTO oauth_tokens (user_id, access_token, refresh_token, expiry)
|
||||
VALUES (@userId, @accessToken, @refreshToken, @expiry)
|
||||
ON CONFLICT(user_id) DO UPDATE SET
|
||||
access_token = excluded.access_token,
|
||||
refresh_token = COALESCE(excluded.refresh_token, oauth_tokens.refresh_token),
|
||||
expiry = excluded.expiry
|
||||
`);
|
||||
|
||||
const getUserByIdStmt = db.prepare(`SELECT * FROM users WHERE id = ?`);
|
||||
const getTokensStmt = db.prepare(`SELECT * FROM oauth_tokens WHERE user_id = ?`);
|
||||
|
||||
export function upsertUser({ googleSub, email, name, picture }) {
|
||||
return upsertUserStmt.get({ googleSub, email, name, picture });
|
||||
}
|
||||
|
||||
export function saveTokens(userId, { accessToken, refreshToken, expiry }) {
|
||||
saveTokensStmt.run({
|
||||
userId,
|
||||
accessToken,
|
||||
refreshToken: refreshToken || null,
|
||||
expiry,
|
||||
});
|
||||
}
|
||||
|
||||
export function getUserById(id) {
|
||||
return getUserByIdStmt.get(id) || null;
|
||||
}
|
||||
|
||||
export function getTokens(userId) {
|
||||
return getTokensStmt.get(userId) || null;
|
||||
}
|
||||
|
||||
export function publicUser(row) {
|
||||
if (!row) return null;
|
||||
return {
|
||||
id: row.id,
|
||||
email: row.email,
|
||||
name: row.name,
|
||||
picture: row.picture,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
import { google } from "googleapis";
|
||||
import {
|
||||
GOOGLE_CLIENT_ID,
|
||||
GOOGLE_CLIENT_SECRET,
|
||||
GOOGLE_REDIRECT_URI,
|
||||
OAUTH_SCOPES,
|
||||
} from "./config.js";
|
||||
import { getTokens, saveTokens } from "./db.js";
|
||||
import { HttpError } from "./http.js";
|
||||
|
||||
export function createOAuthClient() {
|
||||
return new google.auth.OAuth2(
|
||||
GOOGLE_CLIENT_ID,
|
||||
GOOGLE_CLIENT_SECRET,
|
||||
GOOGLE_REDIRECT_URI,
|
||||
);
|
||||
}
|
||||
|
||||
export function authUrl(state) {
|
||||
return createOAuthClient().generateAuthUrl({
|
||||
access_type: "offline",
|
||||
prompt: "consent",
|
||||
include_granted_scopes: true,
|
||||
scope: OAUTH_SCOPES,
|
||||
state,
|
||||
});
|
||||
}
|
||||
|
||||
export async function exchangeCode(code) {
|
||||
const client = createOAuthClient();
|
||||
const { tokens } = await client.getToken(code);
|
||||
client.setCredentials(tokens);
|
||||
|
||||
const oauth2 = google.oauth2({ version: "v2", auth: client });
|
||||
const { data: profile } = await oauth2.userinfo.get();
|
||||
|
||||
if (!profile.id || !profile.email) {
|
||||
throw new HttpError(400, "Google profile did not include email");
|
||||
}
|
||||
|
||||
return {
|
||||
profile: {
|
||||
googleSub: profile.id,
|
||||
email: profile.email,
|
||||
name: profile.name || profile.email,
|
||||
picture: profile.picture || null,
|
||||
},
|
||||
tokens: {
|
||||
accessToken: tokens.access_token,
|
||||
refreshToken: tokens.refresh_token || null,
|
||||
expiry: tokens.expiry_date || Date.now() + 3600 * 1000,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function persistClientTokens(userId, client) {
|
||||
client.on("tokens", (tokens) => {
|
||||
if (!tokens.access_token) return;
|
||||
saveTokens(userId, {
|
||||
accessToken: tokens.access_token,
|
||||
refreshToken: tokens.refresh_token || null,
|
||||
expiry: tokens.expiry_date || Date.now() + 3600 * 1000,
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
export async function gmailClientForUser(userId) {
|
||||
const stored = getTokens(userId);
|
||||
if (!stored?.access_token && !stored?.refresh_token) {
|
||||
throw new HttpError(401, "Google account is not connected");
|
||||
}
|
||||
|
||||
const client = createOAuthClient();
|
||||
client.setCredentials({
|
||||
access_token: stored.access_token,
|
||||
refresh_token: stored.refresh_token,
|
||||
expiry_date: stored.expiry,
|
||||
});
|
||||
persistClientTokens(userId, client);
|
||||
try {
|
||||
await client.getAccessToken();
|
||||
} catch {
|
||||
throw new HttpError(
|
||||
401,
|
||||
"Gmail access expired or was revoked. Sign in again.",
|
||||
);
|
||||
}
|
||||
return google.gmail({ version: "v1", auth: client });
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
export class HttpError extends Error {
|
||||
constructor(statusCode, message) {
|
||||
super(message);
|
||||
this.statusCode = statusCode;
|
||||
}
|
||||
}
|
||||
|
||||
export function requireUser(request) {
|
||||
const userId = request.session.get("userId");
|
||||
if (!userId) {
|
||||
throw new HttpError(401, "Not authenticated");
|
||||
}
|
||||
return userId;
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { existsSync } from "node:fs";
|
||||
import Fastify from "fastify";
|
||||
import fastifySecureSession from "@fastify/secure-session";
|
||||
import fastifyStatic from "@fastify/static";
|
||||
import {
|
||||
NODE_ENV,
|
||||
PORT,
|
||||
SESSION_SECRET,
|
||||
assertConfig,
|
||||
webDistDir,
|
||||
} from "./config.js";
|
||||
import { HttpError } from "./http.js";
|
||||
import { registerAuthRoutes } from "./routes/auth.js";
|
||||
import { registerMessageRoutes } from "./routes/messages.js";
|
||||
|
||||
assertConfig();
|
||||
|
||||
const app = Fastify({
|
||||
logger: true,
|
||||
trustProxy: true,
|
||||
});
|
||||
|
||||
await app.register(fastifySecureSession, {
|
||||
key: createHash("sha256").update(SESSION_SECRET).digest(),
|
||||
cookieName: "session",
|
||||
expiry: 60 * 60 * 24 * 7,
|
||||
cookie: {
|
||||
path: "/",
|
||||
httpOnly: true,
|
||||
sameSite: "lax",
|
||||
secure: NODE_ENV === "production",
|
||||
},
|
||||
});
|
||||
|
||||
app.get("/api/health", async () => ({ ok: true }));
|
||||
|
||||
await registerAuthRoutes(app);
|
||||
await registerMessageRoutes(app);
|
||||
|
||||
app.setErrorHandler((err, request, reply) => {
|
||||
const status = err.statusCode || 500;
|
||||
if (status >= 500) {
|
||||
request.log.error(err);
|
||||
}
|
||||
const message =
|
||||
err instanceof HttpError || status < 500
|
||||
? err.message
|
||||
: "Internal server error";
|
||||
reply.code(status).send({ error: message });
|
||||
});
|
||||
|
||||
const serveFrontend = existsSync(webDistDir);
|
||||
|
||||
if (serveFrontend) {
|
||||
await app.register(fastifyStatic, {
|
||||
root: webDistDir,
|
||||
wildcard: false,
|
||||
});
|
||||
|
||||
app.setNotFoundHandler((request, reply) => {
|
||||
if (
|
||||
request.method !== "GET" ||
|
||||
request.url.startsWith("/api") ||
|
||||
request.url.startsWith("/auth") ||
|
||||
request.url.startsWith("/callback")
|
||||
) {
|
||||
reply.code(404).send({ error: "Not found" });
|
||||
return;
|
||||
}
|
||||
return reply.sendFile("index.html");
|
||||
});
|
||||
}
|
||||
|
||||
try {
|
||||
await app.listen({ port: PORT, host: "0.0.0.0" });
|
||||
app.log.info(
|
||||
`API listening on http://localhost:${PORT}` +
|
||||
(serveFrontend ? " (serving web dist)" : " (API only; use Vite in dev)"),
|
||||
);
|
||||
} catch (err) {
|
||||
app.log.error(err);
|
||||
process.exit(1);
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
function decodeBase64Url(data) {
|
||||
if (!data) return "";
|
||||
const pad = data.length % 4 === 0 ? "" : "=".repeat(4 - (data.length % 4));
|
||||
return Buffer.from(
|
||||
data.replace(/-/g, "+").replace(/_/g, "/") + pad,
|
||||
"base64",
|
||||
).toString("utf8");
|
||||
}
|
||||
|
||||
function stripHtml(html) {
|
||||
return html
|
||||
.replace(/<style[\s\S]*?<\/style>/gi, "")
|
||||
.replace(/<script[\s\S]*?<\/script>/gi, "")
|
||||
.replace(/<br\s*\/?>/gi, "\n")
|
||||
.replace(/<\/p>/gi, "\n\n")
|
||||
.replace(/<\/div>/gi, "\n")
|
||||
.replace(/<\/tr>/gi, "\n")
|
||||
.replace(/<[^>]+>/g, "")
|
||||
.replace(/ /g, " ")
|
||||
.replace(/&/g, "&")
|
||||
.replace(/</g, "<")
|
||||
.replace(/>/g, ">")
|
||||
.replace(/"/g, '"')
|
||||
.replace(/'/g, "'")
|
||||
.replace(/'/gi, "'")
|
||||
.replace(/\n{3,}/g, "\n\n")
|
||||
.trim();
|
||||
}
|
||||
|
||||
function walkParts(payload, acc = { text: "", html: "" }) {
|
||||
if (!payload) return acc;
|
||||
const mime = payload.mimeType || "";
|
||||
const data = payload.body?.data;
|
||||
if (data) {
|
||||
if (mime === "text/plain") acc.text += decodeBase64Url(data);
|
||||
else if (mime === "text/html") acc.html += decodeBase64Url(data);
|
||||
}
|
||||
for (const part of payload.parts || []) {
|
||||
walkParts(part, acc);
|
||||
}
|
||||
return acc;
|
||||
}
|
||||
|
||||
export function extractBody(payload) {
|
||||
const { text, html } = walkParts(payload);
|
||||
if (text.trim()) return text.trim();
|
||||
if (html.trim()) return stripHtml(html);
|
||||
return "";
|
||||
}
|
||||
|
||||
export function header(payload, name) {
|
||||
const match = payload?.headers?.find(
|
||||
(h) => h.name?.toLowerCase() === name.toLowerCase(),
|
||||
);
|
||||
return match?.value || "";
|
||||
}
|
||||
|
||||
export function summarizeMessage(message) {
|
||||
const payload = message.payload || {};
|
||||
return {
|
||||
id: message.id,
|
||||
threadId: message.threadId,
|
||||
from: header(payload, "From"),
|
||||
to: header(payload, "To"),
|
||||
subject: header(payload, "Subject") || "(no subject)",
|
||||
date: header(payload, "Date"),
|
||||
snippet: message.snippet || "",
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
import { randomBytes } from "node:crypto";
|
||||
import { getUserById, publicUser, saveTokens, upsertUser } from "../db.js";
|
||||
import { HttpError, requireUser } from "../http.js";
|
||||
import { authUrl, exchangeCode } from "../google.js";
|
||||
|
||||
export async function registerAuthRoutes(app) {
|
||||
app.get("/auth/google", async (request, reply) => {
|
||||
const state = randomBytes(16).toString("hex");
|
||||
request.session.set("oauthState", state);
|
||||
return reply.redirect(authUrl(state));
|
||||
});
|
||||
|
||||
app.get("/callback", async (request, reply) => {
|
||||
const { code, state, error } = request.query;
|
||||
if (error) {
|
||||
request.log.warn({ error }, "Google OAuth error");
|
||||
return reply.redirect("/login?error=denied");
|
||||
}
|
||||
if (!code || typeof code !== "string") {
|
||||
return reply.redirect("/login?error=missing_code");
|
||||
}
|
||||
const expected = request.session.get("oauthState");
|
||||
if (!expected || state !== expected) {
|
||||
return reply.redirect("/login?error=invalid_state");
|
||||
}
|
||||
request.session.set("oauthState", undefined);
|
||||
|
||||
try {
|
||||
const { profile, tokens } = await exchangeCode(code);
|
||||
const user = upsertUser(profile);
|
||||
saveTokens(user.id, tokens);
|
||||
request.session.set("userId", user.id);
|
||||
return reply.redirect("/");
|
||||
} catch (err) {
|
||||
request.log.error(err);
|
||||
return reply.redirect("/login?error=exchange_failed");
|
||||
}
|
||||
});
|
||||
|
||||
app.post("/auth/logout", async (request, reply) => {
|
||||
request.session.delete();
|
||||
return reply.send({ ok: true });
|
||||
});
|
||||
|
||||
app.get("/api/me", async (request) => {
|
||||
const userId = requireUser(request);
|
||||
const user = getUserById(userId);
|
||||
if (!user) {
|
||||
throw new HttpError(401, "Not authenticated");
|
||||
}
|
||||
return publicUser(user);
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
import { gmailClientForUser } from "../google.js";
|
||||
import { HttpError, requireUser } from "../http.js";
|
||||
import { extractBody, summarizeMessage } from "../mime.js";
|
||||
|
||||
function fromGoogleError(err) {
|
||||
const status = Number(err.code || err.response?.status || err.statusCode);
|
||||
if (status === 401 || status === 403) {
|
||||
return new HttpError(
|
||||
401,
|
||||
"Gmail access expired or was revoked. Sign in again.",
|
||||
);
|
||||
}
|
||||
if (status === 404) {
|
||||
return new HttpError(404, "Message not found");
|
||||
}
|
||||
return err;
|
||||
}
|
||||
|
||||
export async function registerMessageRoutes(app) {
|
||||
app.get("/api/messages", async (request) => {
|
||||
const userId = requireUser(request);
|
||||
const gmail = await gmailClientForUser(userId);
|
||||
const q = typeof request.query.q === "string" ? request.query.q : "";
|
||||
const pageToken =
|
||||
typeof request.query.pageToken === "string"
|
||||
? request.query.pageToken
|
||||
: undefined;
|
||||
const maxResults = Math.min(
|
||||
50,
|
||||
Math.max(1, Number(request.query.maxResults) || 20),
|
||||
);
|
||||
|
||||
let list;
|
||||
try {
|
||||
list = await gmail.users.messages.list({
|
||||
userId: "me",
|
||||
q: q || undefined,
|
||||
maxResults,
|
||||
pageToken,
|
||||
});
|
||||
} catch (err) {
|
||||
throw fromGoogleError(err);
|
||||
}
|
||||
|
||||
const refs = list.data.messages || [];
|
||||
const messages = [];
|
||||
const settled = await Promise.allSettled(
|
||||
refs.map((ref) =>
|
||||
gmail.users.messages.get({
|
||||
userId: "me",
|
||||
id: ref.id,
|
||||
format: "metadata",
|
||||
metadataHeaders: ["From", "To", "Subject", "Date"],
|
||||
}),
|
||||
),
|
||||
);
|
||||
|
||||
for (const result of settled) {
|
||||
if (result.status === "fulfilled") {
|
||||
messages.push(summarizeMessage(result.value.data));
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
messages,
|
||||
nextPageToken: list.data.nextPageToken || null,
|
||||
resultSizeEstimate: list.data.resultSizeEstimate || 0,
|
||||
};
|
||||
});
|
||||
|
||||
app.get("/api/messages/:id", async (request) => {
|
||||
const userId = requireUser(request);
|
||||
const gmail = await gmailClientForUser(userId);
|
||||
const { id } = request.params;
|
||||
|
||||
let data;
|
||||
try {
|
||||
({ data } = await gmail.users.messages.get({
|
||||
userId: "me",
|
||||
id,
|
||||
format: "full",
|
||||
}));
|
||||
} catch (err) {
|
||||
throw fromGoogleError(err);
|
||||
}
|
||||
|
||||
return {
|
||||
...summarizeMessage(data),
|
||||
body: extractBody(data.payload),
|
||||
};
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
/**
|
||||
* PM2 ecosystem for the Gmail receipt watcher.
|
||||
*
|
||||
* Loads secrets from the monorepo root `.env` via dotenv in src/config.js.
|
||||
* Override port with WATCHER_PORT (default 3001; production on rpi4 uses 8083).
|
||||
*
|
||||
* pm2 start ecosystem.config.cjs
|
||||
* pm2 save
|
||||
*/
|
||||
module.exports = {
|
||||
apps: [
|
||||
{
|
||||
name: "gmail-watcher",
|
||||
cwd: __dirname,
|
||||
script: "src/index.js",
|
||||
interpreter: "node",
|
||||
instances: 1,
|
||||
exec_mode: "fork",
|
||||
autorestart: true,
|
||||
max_restarts: 10,
|
||||
restart_delay: 2000,
|
||||
env: {
|
||||
NODE_ENV: "production",
|
||||
WATCHER_PORT: "8083",
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"name": "watcher",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "node --watch src/index.js",
|
||||
"start": "node src/index.js",
|
||||
"test": "node --test test/**/*.test.js",
|
||||
"process-messages": "node scripts/process-messages.js",
|
||||
"list-messages": "node scripts/list-messages.js"
|
||||
},
|
||||
"dependencies": {
|
||||
"better-sqlite3": "^12.2.0",
|
||||
"dotenv": "^17.2.2",
|
||||
"fastify": "^5.6.0",
|
||||
"googleapis": "^159.0.0",
|
||||
"node-html-parser": "^9.0.4"
|
||||
}
|
||||
}
|
||||
Generated
+1414
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,2 @@
|
||||
allowBuilds:
|
||||
better-sqlite3: true
|
||||
@@ -0,0 +1,140 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* List Gmail messages: id, from, subject (metadata only).
|
||||
*
|
||||
* Usage:
|
||||
* npm run list-messages -w watcher
|
||||
* npm run list-messages -w watcher -- --query 'newer_than:14d has:attachment filename:eml'
|
||||
* npm run list-messages -w watcher -- --max 100 --json
|
||||
*
|
||||
* Options:
|
||||
* --query <q> Gmail search (default: newer_than:7d)
|
||||
* --user <email> Mailbox (default: first user with tokens)
|
||||
* --max <n> Max messages (default 50)
|
||||
* --json Print JSON lines instead of a table
|
||||
*/
|
||||
|
||||
import { GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET } from "../src/config.js";
|
||||
import { getUserByEmail, listUsersWithTokens } from "../src/db.js";
|
||||
import { gmailClientForUser } from "../src/google.js";
|
||||
import { header } from "../src/mime.js";
|
||||
|
||||
const DEFAULT_QUERY = "newer_than:7d";
|
||||
|
||||
function parseArgs(argv) {
|
||||
const out = {
|
||||
query: null,
|
||||
user: null,
|
||||
max: 50,
|
||||
json: false,
|
||||
help: false,
|
||||
};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === "--help" || a === "-h") out.help = true;
|
||||
else if (a === "--json") out.json = true;
|
||||
else if (a === "--query") out.query = argv[++i];
|
||||
else if (a === "--user") out.user = argv[++i];
|
||||
else if (a === "--max") out.max = Math.max(1, Number(argv[++i]) || 50);
|
||||
else if (a.startsWith("--query=")) out.query = a.slice("--query=".length);
|
||||
else if (a.startsWith("--user=")) out.user = a.slice("--user=".length);
|
||||
else if (a.startsWith("--max=")) {
|
||||
out.max = Math.max(1, Number(a.slice("--max=".length)) || 50);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async function listMessageIds(gmail, query, max) {
|
||||
const ids = [];
|
||||
let pageToken;
|
||||
while (ids.length < max) {
|
||||
const { data } = await gmail.users.messages.list({
|
||||
userId: "me",
|
||||
q: query,
|
||||
maxResults: Math.min(50, max - ids.length),
|
||||
pageToken,
|
||||
});
|
||||
for (const m of data.messages || []) {
|
||||
if (m.id) ids.push(m.id);
|
||||
if (ids.length >= max) break;
|
||||
}
|
||||
pageToken = data.nextPageToken;
|
||||
if (!pageToken) break;
|
||||
}
|
||||
return ids;
|
||||
}
|
||||
|
||||
function metaFromPayload(payload) {
|
||||
return {
|
||||
from: header(payload, "From") || "",
|
||||
subject: header(payload, "Subject") || "(no subject)",
|
||||
date: header(payload, "Date") || "",
|
||||
};
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
if (args.help) {
|
||||
console.log(`Usage: list-messages [--query q] [--user email] [--max n] [--json]
|
||||
Default query: ${DEFAULT_QUERY}`);
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (!GOOGLE_CLIENT_ID || !GOOGLE_CLIENT_SECRET) {
|
||||
console.error("Missing GOOGLE_CLIENT_ID / GOOGLE_CLIENT_SECRET in .env");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const users = listUsersWithTokens();
|
||||
if (!users.length) {
|
||||
console.error("No signed-in users with tokens. Sign in via the web UI first.");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
let user = users[0];
|
||||
if (args.user) {
|
||||
const byEmail = getUserByEmail(args.user);
|
||||
const byId = users.find((u) => String(u.id) === String(args.user));
|
||||
user = byEmail || byId || null;
|
||||
if (!user) {
|
||||
console.error(`User not found: ${args.user}`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
const gmail = await gmailClientForUser(user.id);
|
||||
const query = args.query || DEFAULT_QUERY;
|
||||
const ids = await listMessageIds(gmail, query, args.max);
|
||||
|
||||
if (!args.json) {
|
||||
console.error(`# mailbox=${user.email} query=${JSON.stringify(query)} count=${ids.length}`);
|
||||
}
|
||||
|
||||
for (const id of ids) {
|
||||
const { data } = await gmail.users.messages.get({
|
||||
userId: "me",
|
||||
id,
|
||||
format: "metadata",
|
||||
metadataHeaders: ["From", "Subject", "Date"],
|
||||
});
|
||||
const meta = metaFromPayload(data.payload || {});
|
||||
const row = {
|
||||
id,
|
||||
from: meta.from,
|
||||
subject: meta.subject,
|
||||
date: meta.date,
|
||||
};
|
||||
|
||||
if (args.json) {
|
||||
console.log(JSON.stringify(row));
|
||||
} else {
|
||||
console.log(`${row.id}\t${row.from}\t${row.subject}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(err);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -0,0 +1,243 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Manually process Gmail messages through match → parse → PocketBase save.
|
||||
*
|
||||
* Use when forwarding old receipts: Gmail Date may be "today", but trx_date
|
||||
* comes from the receipt body (Tanggal/Jam, Picked up on, …). Existing
|
||||
* invoice_id rows are skipped (unique constraint).
|
||||
*
|
||||
* Also expands .eml / message/rfc822 attachments (forward-as-attachment).
|
||||
*
|
||||
* Usage:
|
||||
* npm run process-messages -w watcher -- --query 'newer_than:2d subject:"Pembayaran Berhasil!"'
|
||||
* npm run process-messages -w watcher -- --ids 1a099bd61efd234d,1a099b8b9d239e5e
|
||||
* npm run process-messages -w watcher -- --dry-run --max 20
|
||||
*
|
||||
* Options:
|
||||
* --query <q> Gmail search (default targets Livin/Grab subjects, newer_than:7d)
|
||||
* --ids <a,b> Explicit message IDs (skips list search)
|
||||
* --user <email> Mailbox to use (default: first user with tokens)
|
||||
* --max <n> Max messages to process (default 50)
|
||||
* --dry-run Parse and log only; do not write PocketBase
|
||||
*/
|
||||
|
||||
import {
|
||||
GOOGLE_CLIENT_ID,
|
||||
GOOGLE_CLIENT_SECRET,
|
||||
POCKETBASE_URL,
|
||||
} from "../src/config.js";
|
||||
import { getUserByEmail, listUsersWithTokens } from "../src/db.js";
|
||||
import { gmailClientForUser } from "../src/google.js";
|
||||
import { fetchAndLogMessage } from "../src/handlers/fetch-and-log-message.js";
|
||||
import { matchRule } from "../src/handlers/match-rule.js";
|
||||
import { parseMatchedMessage } from "../src/handlers/parse-matched-message.js";
|
||||
import { saveSpending } from "../src/handlers/save-spending.js";
|
||||
import { runPipeline } from "../src/pipeline.js";
|
||||
import { processFetchedMessage } from "../src/process-fetched.js";
|
||||
import { pocketbaseConfigured } from "../src/pocketbase/client.js";
|
||||
|
||||
const DEFAULT_QUERY =
|
||||
'newer_than:7d (subject:"Pembayaran Berhasil!" OR subject:"Your Grab E-Receipt" OR subject:"Pemindahan Dana Sesama Rekening BRI")';
|
||||
|
||||
function parseArgs(argv) {
|
||||
const out = {
|
||||
query: null,
|
||||
ids: [],
|
||||
user: null,
|
||||
max: 50,
|
||||
dryRun: false,
|
||||
help: false,
|
||||
};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === "--help" || a === "-h") out.help = true;
|
||||
else if (a === "--dry-run") out.dryRun = true;
|
||||
else if (a === "--query") out.query = argv[++i];
|
||||
else if (a === "--ids") {
|
||||
out.ids = String(argv[++i] || "")
|
||||
.split(",")
|
||||
.map((s) => s.trim())
|
||||
.filter(Boolean);
|
||||
} else if (a === "--user") out.user = argv[++i];
|
||||
else if (a === "--max") out.max = Math.max(1, Number(argv[++i]) || 50);
|
||||
else if (a.startsWith("--query=")) out.query = a.slice("--query=".length);
|
||||
else if (a.startsWith("--ids=")) {
|
||||
out.ids = a
|
||||
.slice("--ids=".length)
|
||||
.split(",")
|
||||
.map((s) => s.trim())
|
||||
.filter(Boolean);
|
||||
} else if (a.startsWith("--user=")) out.user = a.slice("--user=".length);
|
||||
else if (a.startsWith("--max=")) {
|
||||
out.max = Math.max(1, Number(a.slice("--max=".length)) || 50);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function makeLog() {
|
||||
const line = (level, obj, msg) => {
|
||||
const payload =
|
||||
typeof obj === "string" ? { msg: obj } : { ...(obj || {}), msg };
|
||||
const { msg: message, ...rest } = payload;
|
||||
console.log(
|
||||
JSON.stringify({
|
||||
level,
|
||||
time: new Date().toISOString(),
|
||||
...rest,
|
||||
msg: message || msg,
|
||||
}),
|
||||
);
|
||||
};
|
||||
return {
|
||||
info: (obj, msg) => line("info", obj, msg),
|
||||
warn: (obj, msg) => line("warn", obj, msg),
|
||||
error: (obj, msg) => line("error", obj, msg),
|
||||
};
|
||||
}
|
||||
|
||||
async function listMessageIds(gmail, query, max) {
|
||||
const ids = [];
|
||||
let pageToken;
|
||||
while (ids.length < max) {
|
||||
const { data } = await gmail.users.messages.list({
|
||||
userId: "me",
|
||||
q: query,
|
||||
maxResults: Math.min(50, max - ids.length),
|
||||
pageToken,
|
||||
});
|
||||
for (const m of data.messages || []) {
|
||||
if (m.id) ids.push(m.id);
|
||||
if (ids.length >= max) break;
|
||||
}
|
||||
pageToken = data.nextPageToken;
|
||||
if (!pageToken) break;
|
||||
}
|
||||
return ids;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
if (args.help) {
|
||||
console.log(`Usage: process-messages [--query q] [--ids id1,id2] [--user email] [--max n] [--dry-run]
|
||||
Default query: ${DEFAULT_QUERY}`);
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (!GOOGLE_CLIENT_ID || !GOOGLE_CLIENT_SECRET) {
|
||||
console.error("Missing GOOGLE_CLIENT_ID / GOOGLE_CLIENT_SECRET in .env");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const users = listUsersWithTokens();
|
||||
if (!users.length) {
|
||||
console.error("No signed-in users with tokens. Sign in via the web UI first.");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
let user = users[0];
|
||||
if (args.user) {
|
||||
const byEmail = getUserByEmail(args.user);
|
||||
const byId = users.find((u) => String(u.id) === String(args.user));
|
||||
user = byEmail || byId || null;
|
||||
if (!user) {
|
||||
console.error(`User not found: ${args.user}`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
const log = makeLog();
|
||||
const gmail = await gmailClientForUser(user.id);
|
||||
|
||||
let messageIds = args.ids;
|
||||
if (!messageIds.length) {
|
||||
const query = args.query || DEFAULT_QUERY;
|
||||
log.info({ query, max: args.max, user: user.email }, "process-messages: listing");
|
||||
messageIds = await listMessageIds(gmail, query, args.max);
|
||||
} else {
|
||||
messageIds = messageIds.slice(0, args.max);
|
||||
}
|
||||
|
||||
log.info(
|
||||
{
|
||||
count: messageIds.length,
|
||||
user: user.email,
|
||||
dryRun: args.dryRun,
|
||||
pocketbase: pocketbaseConfigured() ? POCKETBASE_URL : null,
|
||||
},
|
||||
"process-messages: start",
|
||||
);
|
||||
|
||||
const fetchHandlers = [fetchAndLogMessage];
|
||||
const processHandlers = args.dryRun
|
||||
? [matchRule, parseMatchedMessage]
|
||||
: [matchRule, parseMatchedMessage, saveSpending];
|
||||
|
||||
const summary = {
|
||||
processed: 0,
|
||||
matched: 0,
|
||||
saved: 0,
|
||||
duplicate: 0,
|
||||
skipped: 0,
|
||||
errors: 0,
|
||||
};
|
||||
|
||||
for (const messageId of messageIds) {
|
||||
try {
|
||||
const fetched = await runPipeline(fetchHandlers, {
|
||||
notification: { emailAddress: user.email, historyId: null },
|
||||
userId: user.id,
|
||||
messageId,
|
||||
gmail,
|
||||
log,
|
||||
});
|
||||
|
||||
if (!fetched.message) {
|
||||
summary.skipped++;
|
||||
continue;
|
||||
}
|
||||
|
||||
const results = await processFetchedMessage(fetched, processHandlers);
|
||||
for (const result of results) {
|
||||
summary.processed++;
|
||||
if (!result.rule) {
|
||||
summary.skipped++;
|
||||
continue;
|
||||
}
|
||||
summary.matched++;
|
||||
|
||||
if (args.dryRun) {
|
||||
log.info(
|
||||
{
|
||||
messageId: result.messageId,
|
||||
kind: result.parsed?.kind,
|
||||
invoice_id: result.parsed?.reference,
|
||||
amount: result.parsed?.amount,
|
||||
occurredAt: result.parsed?.occurredAt,
|
||||
},
|
||||
"process-messages: dry-run parsed",
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (result.saved) summary.saved++;
|
||||
else if (result.duplicate) summary.duplicate++;
|
||||
else if (result.error) summary.errors++;
|
||||
else summary.skipped++;
|
||||
}
|
||||
} catch (err) {
|
||||
summary.errors++;
|
||||
log.error(
|
||||
{ messageId, err: err.message || String(err) },
|
||||
"process-messages: failed",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
log.info(summary, "process-messages: done");
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(err);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -0,0 +1,73 @@
|
||||
import { config } from "dotenv";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
export const rootDir = resolve(__dirname, "../../..");
|
||||
export const watcherDir = resolve(__dirname, "..");
|
||||
export const defaultSqlitePath = resolve(__dirname, "../../api/data/app.db");
|
||||
|
||||
config({ path: resolve(rootDir, ".env"), quiet: true });
|
||||
|
||||
export const PORT = Number(process.env.WATCHER_PORT || 3001);
|
||||
export const NODE_ENV = process.env.NODE_ENV || "development";
|
||||
export const GOOGLE_CLIENT_ID = process.env.GOOGLE_CLIENT_ID || "";
|
||||
export const GOOGLE_CLIENT_SECRET = process.env.GOOGLE_CLIENT_SECRET || "";
|
||||
export const GOOGLE_REDIRECT_URI =
|
||||
process.env.GOOGLE_REDIRECT_URI || "http://localhost:5173/callback";
|
||||
export const GOOGLE_CLOUD_PROJECT =
|
||||
process.env.GOOGLE_CLOUD_PROJECT || process.env.GCLOUD_PROJECT || "";
|
||||
export const PUBSUB_VERIFICATION_TOKEN =
|
||||
process.env.PUBSUB_VERIFICATION_TOKEN || "";
|
||||
export const SQLITE_PATH = process.env.SQLITE_PATH || defaultSqlitePath;
|
||||
|
||||
export const POCKETBASE_URL = (process.env.POCKETBASE_URL || "").replace(
|
||||
/\/$/,
|
||||
"",
|
||||
);
|
||||
export const POCKETBASE_USER = process.env.POCKETBASE_USER || "";
|
||||
export const POCKETBASE_PASSWORD = process.env.POCKETBASE_PASSWORD || "";
|
||||
|
||||
/** ntfy topic URL, e.g. https://n.0dev.web.id/system */
|
||||
export const NTFY_URL = (process.env.NTFY_URL || "https://n.0dev.web.id/system").replace(
|
||||
/\/$/,
|
||||
"",
|
||||
);
|
||||
|
||||
export const WATCH_RENEW_MS = 12 * 60 * 60 * 1000;
|
||||
|
||||
const PUBSUB_TOPIC_RE = /^projects\/[^/]+\/topics\/[^/]+$/;
|
||||
|
||||
export function resolvePubsubTopic(
|
||||
topic,
|
||||
{ projectId = GOOGLE_CLOUD_PROJECT } = {},
|
||||
) {
|
||||
const trimmed = (topic || "").trim();
|
||||
if (!trimmed) return "";
|
||||
if (PUBSUB_TOPIC_RE.test(trimmed)) return trimmed;
|
||||
if (trimmed.includes("/")) return trimmed;
|
||||
if (!projectId) return trimmed;
|
||||
return `projects/${projectId}/topics/${trimmed}`;
|
||||
}
|
||||
|
||||
export const GOOGLE_PUBSUB_TOPIC = resolvePubsubTopic(
|
||||
process.env.GOOGLE_PUBSUB_TOPIC || "",
|
||||
);
|
||||
|
||||
export function assertConfig() {
|
||||
const missing = [];
|
||||
if (!GOOGLE_CLIENT_ID) missing.push("GOOGLE_CLIENT_ID");
|
||||
if (!GOOGLE_CLIENT_SECRET) missing.push("GOOGLE_CLIENT_SECRET");
|
||||
if (!GOOGLE_PUBSUB_TOPIC) missing.push("GOOGLE_PUBSUB_TOPIC");
|
||||
if (NODE_ENV === "production" && !PUBSUB_VERIFICATION_TOKEN) {
|
||||
missing.push("PUBSUB_VERIFICATION_TOKEN");
|
||||
}
|
||||
if (missing.length) {
|
||||
throw new Error(`Missing required env: ${missing.join(", ")}`);
|
||||
}
|
||||
if (!PUBSUB_TOPIC_RE.test(GOOGLE_PUBSUB_TOPIC)) {
|
||||
throw new Error(
|
||||
`GOOGLE_PUBSUB_TOPIC must be a full Pub/Sub name like projects/my-gcp-project/topics/gmail-push (got "${process.env.GOOGLE_PUBSUB_TOPIC || ""}")`,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
import { mkdirSync } from "node:fs";
|
||||
import { dirname } from "node:path";
|
||||
import Database from "better-sqlite3";
|
||||
import { SQLITE_PATH } from "./config.js";
|
||||
|
||||
mkdirSync(dirname(SQLITE_PATH), { recursive: true });
|
||||
|
||||
const db = new Database(SQLITE_PATH);
|
||||
db.pragma("journal_mode = WAL");
|
||||
db.pragma("foreign_keys = ON");
|
||||
db.pragma("busy_timeout = 5000");
|
||||
|
||||
db.exec(`
|
||||
CREATE TABLE IF NOT EXISTS users (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
google_sub TEXT NOT NULL UNIQUE,
|
||||
email TEXT NOT NULL,
|
||||
name TEXT,
|
||||
picture TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS oauth_tokens (
|
||||
user_id INTEGER PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
|
||||
access_token TEXT NOT NULL,
|
||||
refresh_token TEXT,
|
||||
expiry INTEGER NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS watch_state (
|
||||
user_id INTEGER PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
|
||||
history_id TEXT NOT NULL,
|
||||
expiration INTEGER NOT NULL
|
||||
);
|
||||
`);
|
||||
|
||||
const getUserByIdStmt = db.prepare(`SELECT * FROM users WHERE id = ?`);
|
||||
const getUserByEmailStmt = db.prepare(
|
||||
`SELECT * FROM users WHERE LOWER(email) = LOWER(?)`,
|
||||
);
|
||||
const getTokensStmt = db.prepare(`SELECT * FROM oauth_tokens WHERE user_id = ?`);
|
||||
const listUsersWithTokensStmt = db.prepare(`
|
||||
SELECT u.*
|
||||
FROM users u
|
||||
INNER JOIN oauth_tokens t ON t.user_id = u.id
|
||||
`);
|
||||
const saveTokensStmt = db.prepare(`
|
||||
INSERT INTO oauth_tokens (user_id, access_token, refresh_token, expiry)
|
||||
VALUES (@userId, @accessToken, @refreshToken, @expiry)
|
||||
ON CONFLICT(user_id) DO UPDATE SET
|
||||
access_token = excluded.access_token,
|
||||
refresh_token = COALESCE(excluded.refresh_token, oauth_tokens.refresh_token),
|
||||
expiry = excluded.expiry
|
||||
`);
|
||||
const getWatchStateStmt = db.prepare(
|
||||
`SELECT * FROM watch_state WHERE user_id = ?`,
|
||||
);
|
||||
const saveWatchStateStmt = db.prepare(`
|
||||
INSERT INTO watch_state (user_id, history_id, expiration)
|
||||
VALUES (@userId, @historyId, @expiration)
|
||||
ON CONFLICT(user_id) DO UPDATE SET
|
||||
history_id = excluded.history_id,
|
||||
expiration = excluded.expiration
|
||||
`);
|
||||
|
||||
export function getUserById(id) {
|
||||
return getUserByIdStmt.get(id) || null;
|
||||
}
|
||||
|
||||
export function getUserByEmail(email) {
|
||||
if (!email) return null;
|
||||
return getUserByEmailStmt.get(email) || null;
|
||||
}
|
||||
|
||||
export function getTokens(userId) {
|
||||
return getTokensStmt.get(userId) || null;
|
||||
}
|
||||
|
||||
export function listUsersWithTokens() {
|
||||
return listUsersWithTokensStmt.all();
|
||||
}
|
||||
|
||||
export function saveTokens(userId, { accessToken, refreshToken, expiry }) {
|
||||
saveTokensStmt.run({
|
||||
userId,
|
||||
accessToken,
|
||||
refreshToken: refreshToken || null,
|
||||
expiry,
|
||||
});
|
||||
}
|
||||
|
||||
export function getWatchState(userId) {
|
||||
return getWatchStateStmt.get(userId) || null;
|
||||
}
|
||||
|
||||
export function saveWatchState(userId, { historyId, expiration }) {
|
||||
saveWatchStateStmt.run({
|
||||
userId,
|
||||
historyId: String(historyId),
|
||||
expiration: Number(expiration),
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
import { google } from "googleapis";
|
||||
import {
|
||||
GOOGLE_CLIENT_ID,
|
||||
GOOGLE_CLIENT_SECRET,
|
||||
GOOGLE_PUBSUB_TOPIC,
|
||||
GOOGLE_REDIRECT_URI,
|
||||
} from "./config.js";
|
||||
import { getTokens, getWatchState, saveTokens, saveWatchState } from "./db.js";
|
||||
import { HttpError } from "./http.js";
|
||||
|
||||
export function createOAuthClient() {
|
||||
return new google.auth.OAuth2(
|
||||
GOOGLE_CLIENT_ID,
|
||||
GOOGLE_CLIENT_SECRET,
|
||||
GOOGLE_REDIRECT_URI,
|
||||
);
|
||||
}
|
||||
|
||||
function persistClientTokens(userId, client) {
|
||||
client.on("tokens", (tokens) => {
|
||||
if (!tokens.access_token) return;
|
||||
saveTokens(userId, {
|
||||
accessToken: tokens.access_token,
|
||||
refreshToken: tokens.refresh_token || null,
|
||||
expiry: tokens.expiry_date || Date.now() + 3600 * 1000,
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
export async function gmailClientForUser(userId) {
|
||||
const stored = getTokens(userId);
|
||||
if (!stored?.access_token && !stored?.refresh_token) {
|
||||
throw new HttpError(401, "Google account is not connected");
|
||||
}
|
||||
|
||||
const client = createOAuthClient();
|
||||
client.setCredentials({
|
||||
access_token: stored.access_token,
|
||||
refresh_token: stored.refresh_token,
|
||||
expiry_date: stored.expiry,
|
||||
});
|
||||
persistClientTokens(userId, client);
|
||||
try {
|
||||
await client.getAccessToken();
|
||||
} catch {
|
||||
throw new HttpError(
|
||||
401,
|
||||
"Gmail access expired or was revoked. Sign in again.",
|
||||
);
|
||||
}
|
||||
return google.gmail({ version: "v1", auth: client });
|
||||
}
|
||||
|
||||
export async function startWatch(userId, { resetCursor = false } = {}) {
|
||||
const gmail = await gmailClientForUser(userId);
|
||||
const { data } = await gmail.users.watch({
|
||||
userId: "me",
|
||||
requestBody: {
|
||||
topicName: GOOGLE_PUBSUB_TOPIC,
|
||||
labelIds: ["INBOX"],
|
||||
},
|
||||
});
|
||||
|
||||
const existing = getWatchState(userId);
|
||||
const historyId =
|
||||
resetCursor || !existing?.history_id
|
||||
? String(data.historyId)
|
||||
: existing.history_id;
|
||||
|
||||
saveWatchState(userId, {
|
||||
historyId,
|
||||
expiration: Number(data.expiration) || Date.now() + 7 * 24 * 3600 * 1000,
|
||||
});
|
||||
|
||||
return { ...data, historyId };
|
||||
}
|
||||
|
||||
export async function listHistory(gmail, startHistoryId) {
|
||||
const messageIds = new Set();
|
||||
let pageToken;
|
||||
let latestHistoryId = String(startHistoryId);
|
||||
|
||||
do {
|
||||
const { data } = await gmail.users.history.list({
|
||||
userId: "me",
|
||||
startHistoryId: String(startHistoryId),
|
||||
pageToken,
|
||||
historyTypes: ["messageAdded"],
|
||||
});
|
||||
if (data.historyId) latestHistoryId = String(data.historyId);
|
||||
for (const record of data.history || []) {
|
||||
for (const added of record.messagesAdded || []) {
|
||||
const id = added.message?.id;
|
||||
if (id) messageIds.add(id);
|
||||
}
|
||||
}
|
||||
pageToken = data.nextPageToken;
|
||||
} while (pageToken);
|
||||
|
||||
return { messageIds: [...messageIds], historyId: latestHistoryId };
|
||||
}
|
||||
|
||||
export function googleStatus(err) {
|
||||
return Number(err.code || err.response?.status || err.statusCode || 0);
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
import {
|
||||
extractBody,
|
||||
extractHtml,
|
||||
hydrateRfc822Message,
|
||||
listRfc822Parts,
|
||||
summarizeMessage,
|
||||
} from "../mime.js";
|
||||
import { googleStatus } from "../google.js";
|
||||
|
||||
const BODY_PREVIEW = 500;
|
||||
|
||||
export async function fetchAndLogMessage(ctx) {
|
||||
const { gmail, messageId, log } = ctx;
|
||||
try {
|
||||
const { data } = await gmail.users.messages.get({
|
||||
userId: "me",
|
||||
id: messageId,
|
||||
format: "full",
|
||||
});
|
||||
const message = {
|
||||
...summarizeMessage(data),
|
||||
body: extractBody(data.payload),
|
||||
html: extractHtml(data.payload),
|
||||
};
|
||||
|
||||
const rfcParts = listRfc822Parts(data.payload);
|
||||
const emlMessages = [];
|
||||
for (let index = 0; index < rfcParts.length; index++) {
|
||||
const part = rfcParts[index];
|
||||
const hydrated = await hydrateRfc822Message(gmail, message.id, part, index);
|
||||
emlMessages.push(hydrated);
|
||||
}
|
||||
|
||||
const emptyEmls = emlMessages.filter((m) => !m.body && !m.html).length;
|
||||
|
||||
log.info(
|
||||
{
|
||||
messageId,
|
||||
from: message.from,
|
||||
subject: message.subject,
|
||||
date: message.date,
|
||||
snippet: message.snippet,
|
||||
bodyPreview: (message.body || "").slice(0, BODY_PREVIEW),
|
||||
emlCount: emlMessages.length,
|
||||
emptyEmlCount: emptyEmls || undefined,
|
||||
},
|
||||
"gmail-watch: message",
|
||||
);
|
||||
|
||||
if (emlMessages.length) {
|
||||
log.info(
|
||||
{
|
||||
messageId,
|
||||
emlCount: emlMessages.length,
|
||||
filenames: emlMessages.map((m) => m.attachmentFilename),
|
||||
},
|
||||
"gmail-watch: eml attachments",
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
pass: {
|
||||
message,
|
||||
emlMessages,
|
||||
gmailPayload: data.payload,
|
||||
},
|
||||
};
|
||||
} catch (err) {
|
||||
const status = googleStatus(err);
|
||||
if (status === 404) {
|
||||
log.warn({ messageId }, "gmail-watch: message not found");
|
||||
return { break: true };
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
export async function logEvent(ctx) {
|
||||
ctx.log.info(
|
||||
{
|
||||
emailAddress: ctx.notification?.emailAddress,
|
||||
historyId: ctx.notification?.historyId,
|
||||
messageId: ctx.messageId,
|
||||
userId: ctx.userId,
|
||||
},
|
||||
"gmail-watch: event",
|
||||
);
|
||||
return { pass: { loggedAt: Date.now() } };
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
import { extractEmail, resolveOriginal, subjectMatches } from "../lib/email.js";
|
||||
import { rules } from "../rules/index.js";
|
||||
|
||||
/**
|
||||
* Match a fetched message against known receipt rules using original From + Subject.
|
||||
* Breaks the pipeline when nothing matches.
|
||||
*/
|
||||
export async function matchRule(ctx) {
|
||||
const { message, log, messageId } = ctx;
|
||||
if (!message) {
|
||||
log.warn({ messageId }, "gmail-watch: match skipped (no message)");
|
||||
return { break: true };
|
||||
}
|
||||
|
||||
const original = resolveOriginal(message);
|
||||
const fromEmail = extractEmail(original.from);
|
||||
|
||||
const rule = rules.find(
|
||||
(r) =>
|
||||
fromEmail === r.fromAddress.toLowerCase() &&
|
||||
subjectMatches(original.subject, r.subject),
|
||||
);
|
||||
|
||||
if (!rule) {
|
||||
log.info(
|
||||
{
|
||||
messageId,
|
||||
from: original.from,
|
||||
subject: original.subject,
|
||||
viaForward: original.viaForward,
|
||||
},
|
||||
"gmail-watch: no rule match",
|
||||
);
|
||||
return { break: true };
|
||||
}
|
||||
|
||||
log.info(
|
||||
{
|
||||
messageId,
|
||||
kind: rule.kind,
|
||||
from: original.from,
|
||||
subject: original.subject,
|
||||
viaForward: original.viaForward,
|
||||
},
|
||||
"gmail-watch: rule matched",
|
||||
);
|
||||
|
||||
return {
|
||||
pass: {
|
||||
rule,
|
||||
original: {
|
||||
from: original.from,
|
||||
subject: original.subject,
|
||||
date: original.date,
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
import { parseBriTransfer } from "../parsers/bri.js";
|
||||
import { parseGrabReceipt } from "../parsers/grab.js";
|
||||
import { parseLivinReceipt } from "../parsers/livin.js";
|
||||
import { notifyReceiptIssue } from "../notify/ntfy.js";
|
||||
|
||||
const PARSERS = {
|
||||
"livin.qr_payment.success": parseLivinReceipt,
|
||||
"grab.ereceipt.ride": parseGrabReceipt,
|
||||
"bri.transfer.success": parseBriTransfer,
|
||||
};
|
||||
|
||||
/**
|
||||
* Parse a matched message body into the shared envelope JSON.
|
||||
* On failure, logs a warning, notifies ntfy, and breaks (Pub/Sub stays healthy).
|
||||
*/
|
||||
export async function parseMatchedMessage(ctx) {
|
||||
const { message, rule, original, log, messageId, notification } = ctx;
|
||||
const parse = PARSERS[rule?.kind];
|
||||
if (!parse) {
|
||||
log.warn({ messageId, kind: rule?.kind }, "gmail-watch: no parser for kind");
|
||||
await notifyReceiptIssue(
|
||||
{
|
||||
reason: "no parser for kind",
|
||||
messageId,
|
||||
kind: rule?.kind,
|
||||
message,
|
||||
},
|
||||
log,
|
||||
);
|
||||
return { break: true };
|
||||
}
|
||||
|
||||
const fields = parse(message.body || "", { html: message.html || "" });
|
||||
if (!fields) {
|
||||
log.warn(
|
||||
{ messageId, kind: rule.kind },
|
||||
"gmail-watch: parse failed",
|
||||
);
|
||||
await notifyReceiptIssue(
|
||||
{
|
||||
reason: "parse failed",
|
||||
messageId,
|
||||
kind: rule.kind,
|
||||
message,
|
||||
},
|
||||
log,
|
||||
);
|
||||
return { break: true };
|
||||
}
|
||||
|
||||
// Grab tip shares From/Subject with ride; parser sets kind/reference.
|
||||
const kind = fields.kind || rule.kind;
|
||||
|
||||
const parsed = {
|
||||
kind,
|
||||
messageId,
|
||||
mailbox: notification?.emailAddress || "",
|
||||
matched: {
|
||||
from: message.from || "",
|
||||
subject: message.subject || "",
|
||||
},
|
||||
original: {
|
||||
from: original?.from || "",
|
||||
subject: original?.subject || "",
|
||||
date: original?.date || "",
|
||||
},
|
||||
amount: fields.amount,
|
||||
currency: fields.currency || "IDR",
|
||||
reference: fields.reference,
|
||||
occurredAt: fields.occurredAt,
|
||||
details: fields.details || {},
|
||||
};
|
||||
|
||||
log.info({ messageId, kind: parsed.kind, parsed }, "gmail-watch: parsed");
|
||||
return { pass: { parsed } };
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
import { pocketbaseConfigured, createRecord } from "../pocketbase/client.js";
|
||||
import { toSpendingRecord } from "../pocketbase/map.js";
|
||||
import { notifyReceiptIssue } from "../notify/ntfy.js";
|
||||
|
||||
/**
|
||||
* Insert a parsed receipt into PocketBase spendings.
|
||||
* Unique invoice_id conflicts are logged and skipped (no throw).
|
||||
* Non-duplicate skips / errors notify ntfy.
|
||||
*/
|
||||
export async function saveSpending(ctx) {
|
||||
const { parsed, rule, log, messageId, message } = ctx;
|
||||
|
||||
if (!parsed) {
|
||||
log.warn({ messageId }, "gmail-watch: save skipped (no parsed)");
|
||||
await notifyReceiptIssue(
|
||||
{
|
||||
reason: "save skipped (no parsed)",
|
||||
messageId,
|
||||
message,
|
||||
},
|
||||
log,
|
||||
);
|
||||
return { break: true };
|
||||
}
|
||||
|
||||
if (!pocketbaseConfigured()) {
|
||||
log.warn(
|
||||
{ messageId, kind: parsed.kind },
|
||||
"gmail-watch: pocketbase not configured; parsed only",
|
||||
);
|
||||
await notifyReceiptIssue(
|
||||
{
|
||||
reason: "pocketbase not configured",
|
||||
messageId,
|
||||
kind: parsed.kind,
|
||||
message,
|
||||
extra: { parsed },
|
||||
},
|
||||
log,
|
||||
);
|
||||
return { pass: { saved: false } };
|
||||
}
|
||||
|
||||
const record = toSpendingRecord(parsed, rule);
|
||||
if (!record.invoice_id || record.amount == null || !record.trx_date) {
|
||||
log.warn(
|
||||
{ messageId, kind: parsed.kind, record },
|
||||
"gmail-watch: save skipped (incomplete record)",
|
||||
);
|
||||
await notifyReceiptIssue(
|
||||
{
|
||||
reason: "save skipped (incomplete record)",
|
||||
messageId,
|
||||
kind: parsed.kind,
|
||||
message,
|
||||
record,
|
||||
extra: { parsed },
|
||||
},
|
||||
log,
|
||||
);
|
||||
return { break: true };
|
||||
}
|
||||
|
||||
try {
|
||||
const { ok, status, data } = await createRecord("spendings", record);
|
||||
if (ok) {
|
||||
log.info(
|
||||
{
|
||||
messageId,
|
||||
kind: parsed.kind,
|
||||
invoice_id: record.invoice_id,
|
||||
recordId: data.id,
|
||||
},
|
||||
"gmail-watch: spending saved",
|
||||
);
|
||||
return { pass: { saved: true, spendingId: data.id } };
|
||||
}
|
||||
|
||||
// Unique constraint / duplicate — no ntfy
|
||||
const msg = String(data?.message || "");
|
||||
const isUnique =
|
||||
status === 400 &&
|
||||
(/unique|invoice_id/i.test(msg) ||
|
||||
data?.data?.invoice_id ||
|
||||
JSON.stringify(data).includes("unique"));
|
||||
|
||||
if (isUnique) {
|
||||
log.info(
|
||||
{ messageId, invoice_id: record.invoice_id, status },
|
||||
"gmail-watch: spending duplicate skipped",
|
||||
);
|
||||
return { pass: { saved: false, duplicate: true } };
|
||||
}
|
||||
|
||||
log.error(
|
||||
{ messageId, status, data },
|
||||
"gmail-watch: spending save failed",
|
||||
);
|
||||
await notifyReceiptIssue(
|
||||
{
|
||||
reason: "spending save failed",
|
||||
messageId,
|
||||
kind: parsed.kind,
|
||||
message,
|
||||
record,
|
||||
extra: { status, data },
|
||||
},
|
||||
log,
|
||||
);
|
||||
return { pass: { saved: false, error: true } };
|
||||
} catch (err) {
|
||||
log.error({ err, messageId }, "gmail-watch: spending save error");
|
||||
await notifyReceiptIssue(
|
||||
{
|
||||
reason: "spending save error",
|
||||
messageId,
|
||||
kind: parsed.kind,
|
||||
message,
|
||||
record,
|
||||
extra: { error: err.message || String(err) },
|
||||
},
|
||||
log,
|
||||
);
|
||||
return { pass: { saved: false, error: true } };
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
export class HttpError extends Error {
|
||||
constructor(statusCode, message) {
|
||||
super(message);
|
||||
this.statusCode = statusCode;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
import Fastify from "fastify";
|
||||
import { GOOGLE_PUBSUB_TOPIC, PORT, WATCH_RENEW_MS, assertConfig } from "./config.js";
|
||||
import { listUsersWithTokens } from "./db.js";
|
||||
import { startWatch } from "./google.js";
|
||||
import { fetchAndLogMessage } from "./handlers/fetch-and-log-message.js";
|
||||
import { logEvent } from "./handlers/log-event.js";
|
||||
import { matchRule } from "./handlers/match-rule.js";
|
||||
import { parseMatchedMessage } from "./handlers/parse-matched-message.js";
|
||||
import { saveSpending } from "./handlers/save-spending.js";
|
||||
import { HttpError } from "./http.js";
|
||||
import { pocketbaseConfigured } from "./pocketbase/client.js";
|
||||
import { ensureSpendingsSchema } from "./pocketbase/schema.js";
|
||||
import { registerPubsubRoutes } from "./routes/pubsub.js";
|
||||
|
||||
assertConfig();
|
||||
|
||||
const fetchHandlers = [logEvent, fetchAndLogMessage];
|
||||
const processHandlers = [matchRule, parseMatchedMessage, saveSpending];
|
||||
|
||||
const app = Fastify({
|
||||
logger: true,
|
||||
trustProxy: true,
|
||||
});
|
||||
|
||||
app.get("/health", async () => ({ ok: true }));
|
||||
|
||||
await registerPubsubRoutes(app, { fetchHandlers, processHandlers });
|
||||
|
||||
app.setErrorHandler((err, request, reply) => {
|
||||
const status = err.statusCode || 500;
|
||||
if (status >= 500) {
|
||||
request.log.error(err);
|
||||
}
|
||||
const message =
|
||||
err instanceof HttpError || status < 500
|
||||
? err.message
|
||||
: "Internal server error";
|
||||
reply.code(status).send({ error: message });
|
||||
});
|
||||
|
||||
async function renewAllWatches() {
|
||||
const users = listUsersWithTokens();
|
||||
if (!users.length) {
|
||||
app.log.warn("gmail-watch: no signed-in users with tokens");
|
||||
return;
|
||||
}
|
||||
for (const user of users) {
|
||||
try {
|
||||
await startWatch(user.id);
|
||||
app.log.info(
|
||||
{ userId: user.id, email: user.email },
|
||||
"gmail-watch: watch started",
|
||||
);
|
||||
} catch (err) {
|
||||
app.log.error({ err, userId: user.id, email: user.email }, "gmail-watch: watch failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let renewTimer;
|
||||
app.addHook("onClose", async () => {
|
||||
if (renewTimer) clearInterval(renewTimer);
|
||||
});
|
||||
|
||||
try {
|
||||
await app.listen({ port: PORT, host: "0.0.0.0" });
|
||||
app.log.info(
|
||||
{ topic: GOOGLE_PUBSUB_TOPIC },
|
||||
`Watcher listening on http://localhost:${PORT}`,
|
||||
);
|
||||
|
||||
if (pocketbaseConfigured()) {
|
||||
try {
|
||||
await ensureSpendingsSchema(app.log);
|
||||
} catch (err) {
|
||||
app.log.error({ err }, "pocketbase: schema ensure failed");
|
||||
}
|
||||
} else {
|
||||
app.log.warn("pocketbase: not configured; spendings will not be saved");
|
||||
}
|
||||
|
||||
await renewAllWatches();
|
||||
renewTimer = setInterval(() => {
|
||||
renewAllWatches().catch((err) => {
|
||||
app.log.error({ err }, "gmail-watch: renew cycle failed");
|
||||
});
|
||||
}, WATCH_RENEW_MS);
|
||||
renewTimer.unref?.();
|
||||
} catch (err) {
|
||||
app.log.error(err);
|
||||
process.exit(1);
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
/**
|
||||
* Extract a bare email address from a From header value.
|
||||
* e.g. `"Livin'" <noreply.livin@bankmandiri.co.id>` → `noreply.livin@bankmandiri.co.id`
|
||||
*/
|
||||
export function extractEmail(from) {
|
||||
if (!from) return "";
|
||||
const angle = from.match(/<([^>]+)>/);
|
||||
if (angle) return angle[1].trim().toLowerCase();
|
||||
const bare = from.match(/[\w.+-]+@[\w.-]+\.\w+/);
|
||||
return bare ? bare[0].toLowerCase() : "";
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a Gmail "Forwarded message" header block from the body.
|
||||
* Returns null if no forwarded header is present.
|
||||
*/
|
||||
export function parseForwardedHeader(body) {
|
||||
if (!body) return null;
|
||||
const marker = /-{2,}\s*Forwarded message\s*-{2,}/i;
|
||||
const idx = body.search(marker);
|
||||
if (idx < 0) return null;
|
||||
|
||||
const after = body.slice(idx);
|
||||
const headerMatch = after.match(
|
||||
/From:\s*(.+)\r?\nDate:\s*(.+)\r?\nSubject:\s*(.+)\r?\nTo:\s*(.+)/i,
|
||||
);
|
||||
if (!headerMatch) return null;
|
||||
|
||||
return {
|
||||
from: headerMatch[1].trim(),
|
||||
date: headerMatch[2].trim(),
|
||||
subject: headerMatch[3].trim(),
|
||||
to: headerMatch[4].trim(),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the original sender/subject/date for matching.
|
||||
* Prefer forwarded header when present; otherwise use the Gmail envelope.
|
||||
*/
|
||||
export function resolveOriginal(message) {
|
||||
const forwarded = parseForwardedHeader(message?.body || "");
|
||||
if (forwarded) {
|
||||
return {
|
||||
from: forwarded.from,
|
||||
subject: forwarded.subject,
|
||||
date: forwarded.date,
|
||||
viaForward: true,
|
||||
};
|
||||
}
|
||||
return {
|
||||
from: message?.from || "",
|
||||
subject: message?.subject || "",
|
||||
date: message?.date || "",
|
||||
viaForward: false,
|
||||
};
|
||||
}
|
||||
|
||||
export function subjectMatches(actual, expected) {
|
||||
if (!actual || !expected) return false;
|
||||
const a = actual.trim().toLowerCase();
|
||||
const e = expected.trim().toLowerCase();
|
||||
// Strip common Fwd:/Re: prefixes for comparison of the core subject
|
||||
const strip = (s) => s.replace(/^(fwd|re):\s*/i, "").trim();
|
||||
return a === e || strip(a) === strip(e) || a.includes(e);
|
||||
}
|
||||
@@ -0,0 +1,168 @@
|
||||
/** Parse Indonesian Rupiah strings like "Rp 22.000,00" or "Rp16.600" into integers. */
|
||||
export function parseIdrAmount(raw) {
|
||||
if (raw == null) return null;
|
||||
const s = String(raw).replace(/\s/g, "");
|
||||
const match = s.match(/Rp?\s*([\d.]+(?:,\d{2})?)/i) || s.match(/([\d.]+(?:,\d{2})?)/);
|
||||
if (!match) return null;
|
||||
// Indonesian: dots are thousands separators, comma is decimal
|
||||
const whole = match[1].replace(/\./g, "").replace(/,\d+$/, "");
|
||||
const n = Number(whole);
|
||||
return Number.isFinite(n) ? n : null;
|
||||
}
|
||||
|
||||
const MONTHS = {
|
||||
// English
|
||||
jan: 1,
|
||||
january: 1,
|
||||
feb: 2,
|
||||
february: 2,
|
||||
mar: 3,
|
||||
march: 3,
|
||||
apr: 4,
|
||||
april: 4,
|
||||
may: 5,
|
||||
jun: 6,
|
||||
june: 6,
|
||||
jul: 7,
|
||||
july: 7,
|
||||
aug: 8,
|
||||
august: 8,
|
||||
sep: 9,
|
||||
sept: 9,
|
||||
september: 9,
|
||||
oct: 10,
|
||||
october: 10,
|
||||
nov: 11,
|
||||
november: 11,
|
||||
dec: 12,
|
||||
december: 12,
|
||||
// Indonesian (Livin uses e.g. "15 Agu 2026")
|
||||
januari: 1,
|
||||
februari: 2,
|
||||
maret: 3,
|
||||
mei: 5,
|
||||
juni: 6,
|
||||
juli: 7,
|
||||
agustus: 8,
|
||||
agu: 8,
|
||||
ags: 8,
|
||||
oktober: 10,
|
||||
okt: 10,
|
||||
nop: 11,
|
||||
desember: 12,
|
||||
des: 12,
|
||||
};
|
||||
|
||||
/**
|
||||
* Parse dates like "13 Sep 2026", "13 September 2026", or Indonesian
|
||||
* "15 Agu 2026" into YYYY-MM-DD. Also accepts 2-digit years ("13 Sep 26").
|
||||
*/
|
||||
export function parseDayMonthYear(raw) {
|
||||
if (!raw) return null;
|
||||
const m = String(raw)
|
||||
.trim()
|
||||
.match(/(\d{1,2})\s+([A-Za-z]+)\s+(\d{2,4})\b/);
|
||||
if (!m) return null;
|
||||
const day = Number(m[1]);
|
||||
const month = MONTHS[m[2].toLowerCase()];
|
||||
let year = Number(m[3]);
|
||||
if (m[3].length === 2) year += year >= 70 ? 1900 : 2000;
|
||||
if (!month || !day || !year) return null;
|
||||
return `${year}-${String(month).padStart(2, "0")}-${String(day).padStart(2, "0")}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse Grab tip timestamps like "13 Sep 26 10:47 +0700" → ISO with offset.
|
||||
*/
|
||||
export function parseGrabDateTime(raw) {
|
||||
if (!raw) return null;
|
||||
const m = String(raw)
|
||||
.trim()
|
||||
.match(
|
||||
/(\d{1,2})\s+([A-Za-z]+)\s+(\d{2,4})\s+(\d{1,2}):(\d{2})(?::(\d{2}))?\s*([+-]\d{4})?/,
|
||||
);
|
||||
if (!m) return null;
|
||||
const dateYmd = parseDayMonthYear(`${m[1]} ${m[2]} ${m[3]}`);
|
||||
if (!dateYmd) return null;
|
||||
const hh = String(Number(m[4])).padStart(2, "0");
|
||||
const mm = m[5];
|
||||
const ss = m[6] || "00";
|
||||
const offset = m[7]
|
||||
? `${m[7].slice(0, 3)}:${m[7].slice(3)}`
|
||||
: "+07:00";
|
||||
return `${dateYmd}T${hh}:${mm}:${ss}${offset}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse time like "10:43:51 WIB" into HH:MM:SS.
|
||||
*/
|
||||
export function parseTimeWib(raw) {
|
||||
if (!raw) return null;
|
||||
const m = String(raw).trim().match(/(\d{1,2}):(\d{2})(?::(\d{2}))?/);
|
||||
if (!m) return null;
|
||||
const hh = String(Number(m[1])).padStart(2, "0");
|
||||
const mm = m[2];
|
||||
const ss = m[3] || "00";
|
||||
return `${hh}:${mm}:${ss}`;
|
||||
}
|
||||
|
||||
/** Combine YYYY-MM-DD + HH:MM:SS as WIB (+07:00) ISO string. */
|
||||
export function toWibIso(dateYmd, timeHms) {
|
||||
if (!dateYmd) return null;
|
||||
if (!timeHms) return dateYmd;
|
||||
return `${dateYmd}T${timeHms}+07:00`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse "10:47AM" / "10:47 AM" into 24h HH:MM:SS.
|
||||
*/
|
||||
export function parseAmPmTime(raw) {
|
||||
if (!raw) return null;
|
||||
const m = String(raw)
|
||||
.trim()
|
||||
.match(/^(\d{1,2}):(\d{2})(?::(\d{2}))?\s*([AaPp][Mm])$/);
|
||||
if (!m) return null;
|
||||
let hour = Number(m[1]);
|
||||
const minute = m[2];
|
||||
const second = m[3] || "00";
|
||||
const ap = m[4].toUpperCase();
|
||||
if (ap === "AM") {
|
||||
if (hour === 12) hour = 0;
|
||||
} else if (hour !== 12) {
|
||||
hour += 12;
|
||||
}
|
||||
return `${String(hour).padStart(2, "0")}:${minute}:${second}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Capture value after a label, allowing optional whitespace between label and value
|
||||
* (including jammed HTML-stripped forms like Tanggal13 Sep 2026).
|
||||
*/
|
||||
export function captureAfterLabel(body, label, { until } = {}) {
|
||||
const escaped = label.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
const untilPart = until
|
||||
? `(?=${until.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")})`
|
||||
: "(?=\\n|$)";
|
||||
const re = new RegExp(`${escaped}\\s*([^\\n]*?)${untilPart}`, "i");
|
||||
const m = body.match(re);
|
||||
return m ? m[1].trim() : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Capture a multi-line block after a label until the next known label or blank run.
|
||||
*/
|
||||
export function captureBlockAfterLabel(body, label, nextLabels = []) {
|
||||
const escaped = label.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
const next =
|
||||
nextLabels.length > 0
|
||||
? nextLabels.map((l) => l.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")).join("|")
|
||||
: null;
|
||||
const until = next ? `(?=${next}|$)` : "$";
|
||||
const re = new RegExp(`${escaped}\\s*([\\s\\S]*?)${until}`, "i");
|
||||
const m = body.match(re);
|
||||
if (!m) return null;
|
||||
return m[1]
|
||||
.split(/\r?\n/)
|
||||
.map((l) => l.trim())
|
||||
.filter(Boolean);
|
||||
}
|
||||
@@ -0,0 +1,256 @@
|
||||
function decodeBase64Url(data) {
|
||||
if (!data) return "";
|
||||
const pad = data.length % 4 === 0 ? "" : "=".repeat(4 - (data.length % 4));
|
||||
return Buffer.from(
|
||||
data.replace(/-/g, "+").replace(/_/g, "/") + pad,
|
||||
"base64",
|
||||
).toString("utf8");
|
||||
}
|
||||
|
||||
function stripHtml(html) {
|
||||
return html
|
||||
.replace(/<style[\s\S]*?<\/style>/gi, "")
|
||||
.replace(/<script[\s\S]*?<\/script>/gi, "")
|
||||
.replace(/<br\s*\/?>/gi, "\n")
|
||||
.replace(/<\/p>/gi, "\n\n")
|
||||
.replace(/<\/div>/gi, "\n")
|
||||
.replace(/<\/tr>/gi, "\n")
|
||||
.replace(/<[^>]+>/g, "")
|
||||
.replace(/ /g, " ")
|
||||
.replace(/&/g, "&")
|
||||
.replace(/</g, "<")
|
||||
.replace(/>/g, ">")
|
||||
.replace(/"/g, '"')
|
||||
.replace(/'/g, "'")
|
||||
.replace(/'/gi, "'")
|
||||
.replace(/\n{3,}/g, "\n\n")
|
||||
.trim();
|
||||
}
|
||||
|
||||
function walkParts(payload, acc = { text: "", html: "" }) {
|
||||
if (!payload) return acc;
|
||||
const mime = payload.mimeType || "";
|
||||
const data = payload.body?.data;
|
||||
if (data) {
|
||||
if (mime === "text/plain") acc.text += decodeBase64Url(data);
|
||||
else if (mime === "text/html") acc.html += decodeBase64Url(data);
|
||||
}
|
||||
for (const part of payload.parts || []) {
|
||||
walkParts(part, acc);
|
||||
}
|
||||
return acc;
|
||||
}
|
||||
|
||||
/** Return both MIME text and html parts (decoded). */
|
||||
export function extractMimeParts(payload) {
|
||||
const { text, html } = walkParts(payload);
|
||||
return {
|
||||
text: text.trim(),
|
||||
html: html.trim(),
|
||||
};
|
||||
}
|
||||
|
||||
export function extractBody(payload) {
|
||||
const { text, html } = extractMimeParts(payload);
|
||||
if (text) return text;
|
||||
if (html) return stripHtml(html);
|
||||
return "";
|
||||
}
|
||||
|
||||
export function extractHtml(payload) {
|
||||
return extractMimeParts(payload).html;
|
||||
}
|
||||
|
||||
export function header(payload, name) {
|
||||
const match = payload?.headers?.find(
|
||||
(h) => h.name?.toLowerCase() === name.toLowerCase(),
|
||||
);
|
||||
return match?.value || "";
|
||||
}
|
||||
|
||||
export function summarizeMessage(message) {
|
||||
const payload = message.payload || {};
|
||||
return {
|
||||
id: message.id,
|
||||
threadId: message.threadId,
|
||||
from: header(payload, "From"),
|
||||
to: header(payload, "To"),
|
||||
subject: header(payload, "Subject") || "(no subject)",
|
||||
date: header(payload, "Date"),
|
||||
snippet: message.snippet || "",
|
||||
};
|
||||
}
|
||||
|
||||
function isRfc822Part(part) {
|
||||
if (!part) return false;
|
||||
const mime = (part.mimeType || "").toLowerCase();
|
||||
const filename = (part.filename || "").toLowerCase();
|
||||
return mime === "message/rfc822" || filename.endsWith(".eml");
|
||||
}
|
||||
|
||||
/**
|
||||
* Collect message/rfc822 (or *.eml) attachment parts. Does not recurse into
|
||||
* nested .eml inside .eml — only direct attachment parts under the tree.
|
||||
*/
|
||||
export function listRfc822Parts(payload, acc = []) {
|
||||
if (!payload) return acc;
|
||||
for (const part of payload.parts || []) {
|
||||
if (isRfc822Part(part)) {
|
||||
acc.push(part);
|
||||
continue; // do not recurse into nested eml
|
||||
}
|
||||
listRfc822Parts(part, acc);
|
||||
}
|
||||
return acc;
|
||||
}
|
||||
|
||||
function findNestedHeaderSource(part) {
|
||||
if (!part) return null;
|
||||
if (header(part, "From") || header(part, "Subject")) return part;
|
||||
for (const child of part.parts || []) {
|
||||
const found = findNestedHeaderSource(child);
|
||||
if (found) return found;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a synthetic message from a Gmail-expanded message/rfc822 part.
|
||||
* When nested text/html is only available via attachmentId (common on large
|
||||
* batches), call hydrateRfc822Message instead.
|
||||
*/
|
||||
export function messageFromRfc822Part(part, { parentMessageId, index = 0 } = {}) {
|
||||
const headerSource = findNestedHeaderSource(part) || part;
|
||||
const partId = part.partId || String(index);
|
||||
const id = parentMessageId ? `${parentMessageId}#${partId}` : partId;
|
||||
return {
|
||||
id,
|
||||
threadId: null,
|
||||
from: header(headerSource, "From"),
|
||||
to: header(headerSource, "To"),
|
||||
subject: header(headerSource, "Subject") || "(no subject)",
|
||||
date: header(headerSource, "Date"),
|
||||
snippet: "",
|
||||
body: extractBody(part),
|
||||
html: extractHtml(part),
|
||||
attachmentFilename: part.filename || "",
|
||||
parentMessageId: parentMessageId || null,
|
||||
_rfc822Part: part,
|
||||
};
|
||||
}
|
||||
|
||||
/** Collect text/plain + text/html nodes that need attachments.get. */
|
||||
export function listNestedBodyAttachmentRefs(part, acc = []) {
|
||||
if (!part) return acc;
|
||||
const mime = (part.mimeType || "").toLowerCase();
|
||||
const attachId = part.body?.attachmentId;
|
||||
if (attachId && (mime === "text/plain" || mime === "text/html") && !part.body?.data) {
|
||||
acc.push({ mime, attachmentId: attachId });
|
||||
}
|
||||
for (const child of part.parts || []) {
|
||||
listNestedBodyAttachmentRefs(child, acc);
|
||||
}
|
||||
return acc;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode a simple multipart .eml (no nested multipart recursion beyond one level
|
||||
* of multipart/*) into { text, html }. Used when only the top-level rfc822 blob
|
||||
* is downloadable.
|
||||
*/
|
||||
export function extractMimePartsFromRawEml(raw) {
|
||||
if (!raw) return { text: "", html: "" };
|
||||
let text = "";
|
||||
let html = "";
|
||||
|
||||
// Split on MIME boundaries (best-effort)
|
||||
const boundaryMatch = raw.match(/boundary="?([^";\r\n]+)"?/i);
|
||||
const chunks = boundaryMatch
|
||||
? raw.split(new RegExp(`\\r?\\n--${boundaryMatch[1].replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}(?:--)?\\r?\\n`))
|
||||
: [raw];
|
||||
|
||||
for (const chunk of chunks) {
|
||||
const headerEnd = chunk.search(/\r?\n\r?\n/);
|
||||
if (headerEnd < 0) continue;
|
||||
const headers = chunk.slice(0, headerEnd);
|
||||
let body = chunk.slice(headerEnd).replace(/^\r?\n\r?\n/, "");
|
||||
const ct = headers.match(/Content-Type:\s*([^\r\n;]+)/i)?.[1]?.trim().toLowerCase() || "";
|
||||
const cte = headers.match(/Content-Transfer-Encoding:\s*([^\r\n]+)/i)?.[1]?.trim().toLowerCase() || "";
|
||||
|
||||
if (cte === "base64") {
|
||||
body = Buffer.from(body.replace(/\s+/g, ""), "base64").toString("utf8");
|
||||
} else if (cte === "quoted-printable") {
|
||||
body = body
|
||||
.replace(/=\r?\n/g, "")
|
||||
.replace(/=([0-9A-Fa-f]{2})/g, (_, h) => String.fromCharCode(parseInt(h, 16)));
|
||||
}
|
||||
|
||||
if (ct === "text/plain") text += body;
|
||||
else if (ct === "text/html") html += body;
|
||||
}
|
||||
|
||||
return { text: text.trim(), html: html.trim() };
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensure an rfc822-derived message has body/html. Fetches nested Gmail
|
||||
* attachments when Gmail did not inline the content.
|
||||
*/
|
||||
export async function hydrateRfc822Message(gmail, parentMessageId, part, index = 0) {
|
||||
const message = messageFromRfc822Part(part, { parentMessageId, index });
|
||||
if (message.body || message.html) {
|
||||
delete message._rfc822Part;
|
||||
return message;
|
||||
}
|
||||
|
||||
const refs = listNestedBodyAttachmentRefs(part);
|
||||
let text = "";
|
||||
let html = "";
|
||||
|
||||
for (const ref of refs) {
|
||||
const { data } = await gmail.users.messages.attachments.get({
|
||||
userId: "me",
|
||||
messageId: parentMessageId,
|
||||
id: ref.attachmentId,
|
||||
});
|
||||
const decoded = decodeBase64Url(data.data);
|
||||
if (ref.mime === "text/plain") text += decoded;
|
||||
else if (ref.mime === "text/html") html += decoded;
|
||||
}
|
||||
|
||||
// Fallback: download the whole .eml blob
|
||||
if (!text && !html && part.body?.attachmentId) {
|
||||
const { data } = await gmail.users.messages.attachments.get({
|
||||
userId: "me",
|
||||
messageId: parentMessageId,
|
||||
id: part.body.attachmentId,
|
||||
});
|
||||
const raw = decodeBase64Url(data.data);
|
||||
const parts = extractMimePartsFromRawEml(raw);
|
||||
text = parts.text;
|
||||
html = parts.html;
|
||||
}
|
||||
|
||||
message.html = html.trim();
|
||||
message.body = text.trim() || (html ? stripHtml(html) : "");
|
||||
delete message._rfc822Part;
|
||||
return message;
|
||||
}
|
||||
|
||||
/**
|
||||
* Expand a fetched Gmail payload into work items:
|
||||
* - if .eml / message/rfc822 attachments exist → those only
|
||||
* - else → the outer message
|
||||
*/
|
||||
export function expandWorkItems(outerMessage, payload) {
|
||||
const parts = listRfc822Parts(payload);
|
||||
if (!parts.length) return [outerMessage];
|
||||
return parts.map((part, index) =>
|
||||
messageFromRfc822Part(part, {
|
||||
parentMessageId: outerMessage.id,
|
||||
index,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
import { NTFY_URL } from "../config.js";
|
||||
|
||||
/**
|
||||
* POST a notification to ntfy. Never throws — failures are logged only.
|
||||
*/
|
||||
export async function notifyNtfy(
|
||||
{
|
||||
title,
|
||||
message,
|
||||
priority = "default",
|
||||
tags = [],
|
||||
},
|
||||
log = console,
|
||||
) {
|
||||
if (!NTFY_URL) return { skipped: true };
|
||||
|
||||
try {
|
||||
const headers = {
|
||||
Title: title.slice(0, 250),
|
||||
Priority: String(priority),
|
||||
"Content-Type": "text/plain; charset=utf-8",
|
||||
};
|
||||
if (tags.length) headers.Tags = tags.join(",");
|
||||
|
||||
const res = await fetch(NTFY_URL, {
|
||||
method: "POST",
|
||||
headers,
|
||||
body: message,
|
||||
});
|
||||
|
||||
if (!res.ok) {
|
||||
const text = await res.text().catch(() => "");
|
||||
log.warn?.(
|
||||
{ status: res.status, text: text.slice(0, 200) },
|
||||
"ntfy: publish failed",
|
||||
);
|
||||
return { ok: false, status: res.status };
|
||||
}
|
||||
return { ok: true };
|
||||
} catch (err) {
|
||||
log.warn?.({ err: err.message || String(err) }, "ntfy: publish error");
|
||||
return { ok: false, error: true };
|
||||
}
|
||||
}
|
||||
|
||||
/** Build a notification body for a failed/skipped receipt work item. */
|
||||
export function formatReceiptAlert({
|
||||
reason,
|
||||
messageId,
|
||||
kind,
|
||||
message,
|
||||
record,
|
||||
extra,
|
||||
}) {
|
||||
const lines = [
|
||||
`reason: ${reason}`,
|
||||
`messageId: ${messageId || ""}`,
|
||||
`kind: ${kind || ""}`,
|
||||
`from: ${message?.from || ""}`,
|
||||
`subject: ${message?.subject || ""}`,
|
||||
`date: ${message?.date || ""}`,
|
||||
`attachment: ${message?.attachmentFilename || ""}`,
|
||||
`parentMessageId: ${message?.parentMessageId || ""}`,
|
||||
];
|
||||
|
||||
if (record) {
|
||||
lines.push(`record: ${JSON.stringify(record)}`);
|
||||
}
|
||||
if (extra) {
|
||||
lines.push(`extra: ${JSON.stringify(extra)}`);
|
||||
}
|
||||
|
||||
const body = message?.body || "";
|
||||
lines.push("", "----- message body -----", body || "(empty)");
|
||||
|
||||
if (message?.html) {
|
||||
lines.push(
|
||||
"",
|
||||
"----- message html (length) -----",
|
||||
String(message.html.length),
|
||||
);
|
||||
}
|
||||
|
||||
// ntfy payloads should stay reasonably sized
|
||||
const text = lines.join("\n");
|
||||
const max = 12000;
|
||||
if (text.length <= max) return text;
|
||||
return `${text.slice(0, max)}\n\n… truncated (${text.length} chars total)`;
|
||||
}
|
||||
|
||||
export async function notifyReceiptIssue(payload, log) {
|
||||
const body = formatReceiptAlert(payload);
|
||||
return notifyNtfy(
|
||||
{
|
||||
title: `gmail-hook: ${payload.reason}`,
|
||||
message: body,
|
||||
priority: "high",
|
||||
tags: ["warning", "email"],
|
||||
},
|
||||
log,
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
import {
|
||||
captureAfterLabel,
|
||||
captureBlockAfterLabel,
|
||||
parseDayMonthYear,
|
||||
parseIdrAmount,
|
||||
parseTimeWib,
|
||||
toWibIso,
|
||||
} from "../lib/parse-helpers.js";
|
||||
|
||||
/**
|
||||
* Parse BRImo / Bank BRI transfer success body (plain text).
|
||||
* Required: Nominal + Nomor Referensi.
|
||||
*/
|
||||
export function parseBriTransfer(body) {
|
||||
if (!body) return null;
|
||||
|
||||
const payerMatch = body.match(/Halo[,\s*]*([^*\n,]+)/i);
|
||||
const payerName = payerMatch ? payerMatch[1].replace(/\*/g, "").trim() : null;
|
||||
|
||||
const tanggalRaw = captureAfterLabel(body, "Tanggal");
|
||||
const dateYmd = parseDayMonthYear(tanggalRaw);
|
||||
const timeHms = parseTimeWib(tanggalRaw);
|
||||
const occurredAt = toWibIso(dateYmd, timeHms);
|
||||
|
||||
// Prefer Nominal over Total (Total = Nominal + Biaya Admin).
|
||||
const nominalRaw = captureAfterLabel(body, "Nominal");
|
||||
const amount = parseIdrAmount(nominalRaw);
|
||||
|
||||
const reference =
|
||||
captureAfterLabel(body, "Nomor Referensi") ||
|
||||
body.match(/Nomor\s+Referensi\s*([^\s\n]+)/i)?.[1] ||
|
||||
null;
|
||||
|
||||
if (amount == null || !reference) return null;
|
||||
|
||||
const namaTujuan = captureAfterLabel(body, "Nama Tujuan");
|
||||
const jenisTransaksi = captureAfterLabel(body, "Jenis Transaksi");
|
||||
const bankTujuan = captureAfterLabel(body, "Bank Tujuan");
|
||||
const nomorTujuan = captureAfterLabel(body, "Nomor Tujuan");
|
||||
const catatanRaw = captureAfterLabel(body, "Catatan");
|
||||
const biayaAdmin = parseIdrAmount(captureAfterLabel(body, "Biaya Admin"));
|
||||
|
||||
const fundingLines = captureBlockAfterLabel(body, "Sumber Dana", [
|
||||
"Jenis Transaksi",
|
||||
]);
|
||||
const fundingSource =
|
||||
fundingLines?.join(" ").replace(/\s+/g, " ").trim() || null;
|
||||
|
||||
const details = {};
|
||||
if (payerName) details.payerName = payerName;
|
||||
if (namaTujuan) details.namaTujuan = namaTujuan.trim();
|
||||
if (jenisTransaksi) details.jenisTransaksi = jenisTransaksi.trim();
|
||||
if (bankTujuan) details.bankTujuan = bankTujuan.trim();
|
||||
if (nomorTujuan) details.nomorTujuan = nomorTujuan.trim();
|
||||
if (catatanRaw != null && catatanRaw.trim() && catatanRaw.trim() !== "-") {
|
||||
details.catatan = catatanRaw.trim();
|
||||
}
|
||||
if (biayaAdmin != null) details.biayaAdmin = biayaAdmin;
|
||||
if (fundingSource) details.fundingSource = fundingSource;
|
||||
|
||||
return {
|
||||
amount,
|
||||
currency: "IDR",
|
||||
reference: reference.trim(),
|
||||
occurredAt,
|
||||
details,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,247 @@
|
||||
import { parse } from "node-html-parser";
|
||||
import {
|
||||
captureAfterLabel,
|
||||
captureBlockAfterLabel,
|
||||
parseAmPmTime,
|
||||
parseDayMonthYear,
|
||||
parseGrabDateTime,
|
||||
parseIdrAmount,
|
||||
toWibIso,
|
||||
} from "../lib/parse-helpers.js";
|
||||
|
||||
function stripForwardedHeader(body) {
|
||||
const m = body.match(
|
||||
/-{2,}\s*Forwarded message\s*-{2,}[\s\S]*?\nTo:[^\n]*\n+/i,
|
||||
);
|
||||
if (!m) return body;
|
||||
return body.slice(m.index + m[0].length);
|
||||
}
|
||||
|
||||
/** Tip receipts share From/Subject with ride e-receipts but different body copy. */
|
||||
export function isGrabTipBody(body) {
|
||||
if (!body) return false;
|
||||
if (/your tip goes a long way/i.test(body)) return true;
|
||||
if (/thanks!\s*your tip/i.test(body)) return true;
|
||||
// Tip layout: has a Tip section, no ride pickup line
|
||||
if (
|
||||
!/picked up on/i.test(body) &&
|
||||
!/hope you enjoyed your ride/i.test(body) &&
|
||||
!/selamat menikmati makanan/i.test(body) &&
|
||||
!/grabfood/i.test(body) &&
|
||||
/\btip\b/i.test(body) &&
|
||||
/total paid/i.test(body)
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/** GrabFood order receipts (ID/EN) — same From/Subject as rides. */
|
||||
export function isGrabFoodBody(body) {
|
||||
if (!body) return false;
|
||||
if (/selamat menikmati makanan/i.test(body)) return true;
|
||||
if (/jenis kendaraan:\s*\r?\n\s*GrabFood/i.test(body)) return true;
|
||||
if (/\bGrabFood\b/i.test(body) && /pesanan id/i.test(body)) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
function extractProduct(content) {
|
||||
const productMatch = content.match(
|
||||
/^\s*(Bike\s+\w+|Car\s+\w+|Grab\w*|Food|Express|Mart)[^\n]*/im,
|
||||
);
|
||||
return productMatch ? productMatch[1].trim() : null;
|
||||
}
|
||||
|
||||
function extractBookingId(body) {
|
||||
return (
|
||||
captureAfterLabel(body, "Booking ID:") ||
|
||||
body.match(/Booking ID:\s*([A-Z0-9-]+)/i)?.[1] ||
|
||||
null
|
||||
);
|
||||
}
|
||||
|
||||
function extractTotalPaid(body) {
|
||||
return (
|
||||
captureAfterLabel(body, "Total Paid") ||
|
||||
body.match(/Total Paid\s*\r?\n\s*(Rp[\d.,]+)/i)?.[1]
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Grab ride HTML puts pickup/dropoff as standalone "10:47AM" divs next to
|
||||
* locations. Plain text only has "Picked up on 13 September 2026" (date only).
|
||||
* Returns { pickup, dropoff } as HH:MM:SS (24h) when found.
|
||||
*/
|
||||
export function extractGrabTripTimesFromHtml(html) {
|
||||
if (!html) return { pickup: null, dropoff: null };
|
||||
const root = parse(html);
|
||||
const times = [];
|
||||
for (const el of root.querySelectorAll("div")) {
|
||||
const t = (el.text || "").replace(/\u00a0/g, " ").trim();
|
||||
if (!/^\d{1,2}:\d{2}(?::\d{2})?\s*[AaPp][Mm]$/.test(t)) continue;
|
||||
const hms = parseAmPmTime(t);
|
||||
if (hms) times.push(hms);
|
||||
}
|
||||
return {
|
||||
pickup: times[0] || null,
|
||||
dropoff: times[1] || null,
|
||||
};
|
||||
}
|
||||
|
||||
function parseRide(body, content, html) {
|
||||
const product = extractProduct(content);
|
||||
const pickedUpRaw = captureAfterLabel(body, "Picked up on");
|
||||
const dateYmd = parseDayMonthYear(pickedUpRaw);
|
||||
const { pickup, dropoff } = extractGrabTripTimesFromHtml(html);
|
||||
const occurredAt = toWibIso(dateYmd, pickup) || dateYmd;
|
||||
|
||||
const bookingId = extractBookingId(body);
|
||||
const amount = parseIdrAmount(extractTotalPaid(body));
|
||||
|
||||
if (amount == null || !bookingId) return null;
|
||||
|
||||
const details = { variant: "ride" };
|
||||
if (product) details.product = product;
|
||||
if (dropoff) details.dropoffAt = toWibIso(dateYmd, dropoff);
|
||||
|
||||
const driverMatch = body.match(
|
||||
/Receipt issued by the driver\s+([^.\n]+)/i,
|
||||
);
|
||||
if (driverMatch) details.driver = driverMatch[1].trim();
|
||||
|
||||
const ratingMatch = body.match(
|
||||
/(\d(?:\.\d)?)\s*(?:\[image: stars\]|★|stars)/i,
|
||||
);
|
||||
if (ratingMatch) details.driverRating = Number(ratingMatch[1]);
|
||||
|
||||
const complimentsBlock = body.match(
|
||||
/Compliments for driver\s*\r?\n([\s\S]*?)(?=\r?\nBreakdown|\r?\nPassenger|$)/i,
|
||||
);
|
||||
if (complimentsBlock) {
|
||||
const compliments = complimentsBlock[1]
|
||||
.split(/\r?\n/)
|
||||
.map((l) => l.trim())
|
||||
.filter((l) => l && !/^\[image/i.test(l));
|
||||
if (compliments.length) details.compliments = compliments;
|
||||
}
|
||||
|
||||
const passenger = captureAfterLabel(body, "Passenger");
|
||||
if (passenger) {
|
||||
details.passenger = passenger.replace(/\s*Profile.*$/i, "").trim();
|
||||
}
|
||||
|
||||
return {
|
||||
variant: "ride",
|
||||
kind: "grab.ereceipt.ride",
|
||||
amount,
|
||||
currency: "IDR",
|
||||
reference: bookingId.trim(),
|
||||
occurredAt,
|
||||
details,
|
||||
};
|
||||
}
|
||||
|
||||
function parseTip(body, content) {
|
||||
const product = extractProduct(content);
|
||||
const bookingId = extractBookingId(body);
|
||||
const amount = parseIdrAmount(extractTotalPaid(body));
|
||||
|
||||
if (amount == null || !bookingId) return null;
|
||||
|
||||
// Tip timestamp sits on its own line, e.g. "13 Sep 26 10:47 +0700"
|
||||
const tipTs =
|
||||
body.match(
|
||||
/(\d{1,2}\s+[A-Za-z]+\s+\d{2,4}\s+\d{1,2}:\d{2}(?::\d{2})?\s*[+-]\d{4})/,
|
||||
)?.[1] || null;
|
||||
const occurredAt = parseGrabDateTime(tipTs);
|
||||
|
||||
const details = { variant: "tip" };
|
||||
if (product) details.product = product;
|
||||
|
||||
return {
|
||||
variant: "tip",
|
||||
kind: "grab.ereceipt.tip",
|
||||
amount,
|
||||
currency: "IDR",
|
||||
// Separate invoice from the ride spend for the same Booking ID
|
||||
reference: `${bookingId.trim()}:tip`,
|
||||
occurredAt,
|
||||
details,
|
||||
};
|
||||
}
|
||||
|
||||
function parseFood(body, content) {
|
||||
const orderId =
|
||||
captureAfterLabel(body, "Pesanan ID") ||
|
||||
body.match(/Pesanan ID\s*\r?\n\s*([A-Z0-9-]+)/i)?.[1] ||
|
||||
captureAfterLabel(body, "Order ID") ||
|
||||
null;
|
||||
|
||||
const totalRaw =
|
||||
body.match(/TOTAL\s*\(INCL\.?\s*TAX\)\s*IDR\s*([\d.]+)/i)?.[0] ||
|
||||
body.match(/TOTAL\s*\r?\n\s*IDR\s*([\d.]+)/i)?.[0] ||
|
||||
body.match(/IDR\s*[\d.]+/)?.[0];
|
||||
const amount = parseIdrAmount(totalRaw);
|
||||
|
||||
if (amount == null || !orderId) return null;
|
||||
|
||||
const occurredAt = parseGrabDateTime(
|
||||
body.match(
|
||||
/(\d{1,2}\s+[A-Za-z]+\s+\d{2,4}\s+\d{1,2}:\d{2}(?::\d{2})?\s*[+-]\d{4})/,
|
||||
)?.[1],
|
||||
);
|
||||
|
||||
const details = { variant: "food", product: "Food" };
|
||||
|
||||
const vehicle =
|
||||
captureAfterLabel(body, "Jenis Kendaraan:") ||
|
||||
body.match(/Jenis Kendaraan:\s*\r?\n\s*([^\n]+)/i)?.[1];
|
||||
if (vehicle) details.vehicleType = vehicle.trim();
|
||||
|
||||
const forName =
|
||||
captureAfterLabel(body, "Diterbitkan untuk") ||
|
||||
body.match(/Diterbitkan untuk\s*\r?\n\s*([^\n]+)/i)?.[1];
|
||||
if (forName) details.passenger = forName.trim();
|
||||
|
||||
const merchantLines = captureBlockAfterLabel(body, "Pesanan Dari:", [
|
||||
"Profil:",
|
||||
"Detail Tagihan",
|
||||
"Detail Pembayaran",
|
||||
]);
|
||||
if (merchantLines?.length) {
|
||||
details.merchant = merchantLines[0];
|
||||
if (merchantLines.length > 1) {
|
||||
details.location = merchantLines
|
||||
.slice(1)
|
||||
.join(" ")
|
||||
.replace(/<https?:\/\/[^>]+>/g, "")
|
||||
.replace(/\s+/g, " ")
|
||||
.trim();
|
||||
}
|
||||
}
|
||||
|
||||
const payment =
|
||||
captureAfterLabel(body, "Detail Pembayaran:") ||
|
||||
body.match(/Detail Pembayaran:\s*\r?\n\s*([^\n]+)/i)?.[1];
|
||||
if (payment && !/Jumlah/i.test(payment)) {
|
||||
details.paymentMethod = payment.trim();
|
||||
}
|
||||
|
||||
return {
|
||||
variant: "food",
|
||||
kind: "grab.ereceipt.food",
|
||||
amount,
|
||||
currency: "IDR",
|
||||
reference: orderId.trim(),
|
||||
occurredAt,
|
||||
details,
|
||||
};
|
||||
}
|
||||
|
||||
export function parseGrabReceipt(body, { html = "" } = {}) {
|
||||
if (!body) return null;
|
||||
const content = stripForwardedHeader(body);
|
||||
if (isGrabTipBody(body)) return parseTip(body, content);
|
||||
if (isGrabFoodBody(body)) return parseFood(body, content);
|
||||
return parseRide(body, content, html);
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
import {
|
||||
captureAfterLabel,
|
||||
captureBlockAfterLabel,
|
||||
parseDayMonthYear,
|
||||
parseIdrAmount,
|
||||
parseTimeWib,
|
||||
toWibIso,
|
||||
} from "../lib/parse-helpers.js";
|
||||
|
||||
/**
|
||||
* Split Penerima lines into merchant + location when the body already
|
||||
* separates them. If HTML-stripped glue merges them into one line
|
||||
* (e.g. SNACK BAROKAHJAKARTA BARAT - ID), keep the whole string as merchant
|
||||
* — do not guess Indonesian city names.
|
||||
*/
|
||||
export function splitMerchantLocation(lines) {
|
||||
if (!lines || !lines.length) {
|
||||
return { merchant: null, location: null };
|
||||
}
|
||||
|
||||
if (lines.length >= 2) {
|
||||
const last = lines[lines.length - 1];
|
||||
if (/-\s*ID\s*$/i.test(last)) {
|
||||
return {
|
||||
merchant: lines.slice(0, -1).join(" ").trim() || null,
|
||||
location: last.trim(),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
return { merchant: lines.join(" ").trim() || null, location: null };
|
||||
}
|
||||
|
||||
export function parseLivinReceipt(body) {
|
||||
if (!body) return null;
|
||||
|
||||
const payerMatch = body.match(/Halo\s+([^,\n]+)/i);
|
||||
const payerName = payerMatch ? payerMatch[1].trim() : null;
|
||||
|
||||
const penerimaLines = captureBlockAfterLabel(body, "Penerima", [
|
||||
"Tanggal",
|
||||
"Jam",
|
||||
"Nominal",
|
||||
]);
|
||||
const { merchant, location } = splitMerchantLocation(penerimaLines);
|
||||
|
||||
const tanggalRaw = captureAfterLabel(body, "Tanggal");
|
||||
const jamRaw = captureAfterLabel(body, "Jam");
|
||||
const dateYmd = parseDayMonthYear(tanggalRaw);
|
||||
const timeHms = parseTimeWib(jamRaw);
|
||||
const occurredAt = toWibIso(dateYmd, timeHms);
|
||||
|
||||
const nominalRaw = captureAfterLabel(body, "Nominal Transaksi");
|
||||
const amount = parseIdrAmount(nominalRaw);
|
||||
|
||||
const reference =
|
||||
captureAfterLabel(body, "No. Referensi") ||
|
||||
body.match(/No\.\s*Referensi\s*([^\s\n]+)/i)?.[1] ||
|
||||
null;
|
||||
|
||||
if (amount == null || !reference) return null;
|
||||
|
||||
const details = {};
|
||||
if (payerName) details.payerName = payerName;
|
||||
if (merchant) details.merchant = merchant;
|
||||
if (location) details.location = location;
|
||||
|
||||
const qrisRef =
|
||||
captureAfterLabel(body, "No. Ref. QRIS") ||
|
||||
body.match(/No\.\s*Ref\.\s*QRIS\s*([^\s\n]+)/i)?.[1];
|
||||
const merchantPan = captureAfterLabel(body, "Merchant PAN");
|
||||
const customerPan = captureAfterLabel(body, "Customer PAN");
|
||||
const acquirer = captureAfterLabel(body, "Pengakuisisi");
|
||||
const terminalId = captureAfterLabel(body, "Terminal ID");
|
||||
const fundingLines = captureBlockAfterLabel(body, "Sumber Dana", [
|
||||
"Simpan email",
|
||||
]);
|
||||
const fundingSource = fundingLines?.join(" ").replace(/\s+/g, " ").trim() || null;
|
||||
|
||||
if (qrisRef) details.qrisRef = qrisRef;
|
||||
if (merchantPan) details.merchantPan = merchantPan;
|
||||
if (customerPan) details.customerPan = customerPan;
|
||||
if (acquirer) details.acquirer = acquirer;
|
||||
if (terminalId) details.terminalId = terminalId.trim();
|
||||
if (fundingSource) details.fundingSource = fundingSource;
|
||||
|
||||
return {
|
||||
amount,
|
||||
currency: "IDR",
|
||||
reference: reference.trim(),
|
||||
occurredAt,
|
||||
details,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
export async function runPipeline(handlers, ctx) {
|
||||
let current = { ...ctx };
|
||||
for (const handler of handlers) {
|
||||
const result = await handler(current);
|
||||
if (!result) continue;
|
||||
if (result.pass && typeof result.pass === "object" && !Array.isArray(result.pass)) {
|
||||
current = { ...current, ...result.pass };
|
||||
}
|
||||
if (result.break) break;
|
||||
}
|
||||
return current;
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
import {
|
||||
POCKETBASE_PASSWORD,
|
||||
POCKETBASE_URL,
|
||||
POCKETBASE_USER,
|
||||
} from "../config.js";
|
||||
|
||||
let cachedToken = null;
|
||||
let cachedTokenAt = 0;
|
||||
const TOKEN_TTL_MS = 50 * 60 * 1000; // refresh before typical 1h expiry
|
||||
|
||||
export function pocketbaseConfigured() {
|
||||
return Boolean(POCKETBASE_URL && POCKETBASE_USER && POCKETBASE_PASSWORD);
|
||||
}
|
||||
|
||||
async function authSuperuser() {
|
||||
const res = await fetch(
|
||||
`${POCKETBASE_URL}/api/collections/_superusers/auth-with-password`,
|
||||
{
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
identity: POCKETBASE_USER,
|
||||
password: POCKETBASE_PASSWORD,
|
||||
}),
|
||||
},
|
||||
);
|
||||
const data = await res.json().catch(() => ({}));
|
||||
if (!res.ok || !data.token) {
|
||||
throw new Error(
|
||||
`PocketBase auth failed (${res.status}): ${data.message || "unknown"}`,
|
||||
);
|
||||
}
|
||||
cachedToken = data.token;
|
||||
cachedTokenAt = Date.now();
|
||||
return cachedToken;
|
||||
}
|
||||
|
||||
async function getToken() {
|
||||
if (cachedToken && Date.now() - cachedTokenAt < TOKEN_TTL_MS) {
|
||||
return cachedToken;
|
||||
}
|
||||
return authSuperuser();
|
||||
}
|
||||
|
||||
export async function pbRequest(path, { method = "GET", body, retry = true } = {}) {
|
||||
if (!pocketbaseConfigured()) {
|
||||
throw new Error("PocketBase is not configured");
|
||||
}
|
||||
const token = await getToken();
|
||||
const res = await fetch(`${POCKETBASE_URL}${path}`, {
|
||||
method,
|
||||
headers: {
|
||||
Authorization: token,
|
||||
...(body ? { "Content-Type": "application/json" } : {}),
|
||||
},
|
||||
body: body ? JSON.stringify(body) : undefined,
|
||||
});
|
||||
|
||||
if (res.status === 401 && retry) {
|
||||
cachedToken = null;
|
||||
await authSuperuser();
|
||||
return pbRequest(path, { method, body, retry: false });
|
||||
}
|
||||
|
||||
const text = await res.text();
|
||||
let data = {};
|
||||
try {
|
||||
data = text ? JSON.parse(text) : {};
|
||||
} catch {
|
||||
data = { raw: text };
|
||||
}
|
||||
return { ok: res.ok, status: res.status, data };
|
||||
}
|
||||
|
||||
export async function getCollection(name) {
|
||||
return pbRequest(`/api/collections/${name}`);
|
||||
}
|
||||
|
||||
export async function updateCollection(name, patch) {
|
||||
return pbRequest(`/api/collections/${name}`, {
|
||||
method: "PATCH",
|
||||
body: patch,
|
||||
});
|
||||
}
|
||||
|
||||
export async function createRecord(collection, record) {
|
||||
return pbRequest(`/api/collections/${collection}/records`, {
|
||||
method: "POST",
|
||||
body: record,
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
/**
|
||||
* Map parsed receipt envelope → PocketBase spendings record.
|
||||
*/
|
||||
export function toSpendingRecord(parsed, rule) {
|
||||
const source = rule?.source || parsed.kind?.split(".")[0] || "";
|
||||
let description = "";
|
||||
if (source === "livin") {
|
||||
description = parsed.details?.merchant || "";
|
||||
} else if (source === "bri") {
|
||||
description = parsed.details?.namaTujuan || "";
|
||||
} else if (source === "grab") {
|
||||
if (parsed.kind === "grab.ereceipt.food" || parsed.details?.variant === "food") {
|
||||
const merchant = parsed.details?.merchant;
|
||||
description = merchant ? `Grab Food ${merchant}` : "Grab Food";
|
||||
} else {
|
||||
const product = parsed.details?.product;
|
||||
const base = product ? `Grab ${product}` : "Grab";
|
||||
description =
|
||||
parsed.kind === "grab.ereceipt.tip" || parsed.details?.variant === "tip"
|
||||
? `${base} tip`
|
||||
: base;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
invoice_id: parsed.reference,
|
||||
amount: parsed.amount,
|
||||
trx_date: toPocketBaseDate(parsed.occurredAt),
|
||||
source,
|
||||
description,
|
||||
category: "",
|
||||
message_id: parsed.messageId || "",
|
||||
details: parsed.details || {},
|
||||
// Full watcher envelope snapshot for audit / debugging (not the raw email body).
|
||||
parsed,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* PocketBase date fields accept "YYYY-MM-DD HH:mm:ss.SSSZ" (UTC) or date-only.
|
||||
*/
|
||||
export function toPocketBaseDate(occurredAt) {
|
||||
if (!occurredAt) return null;
|
||||
// Date-only (Grab)
|
||||
if (/^\d{4}-\d{2}-\d{2}$/.test(occurredAt)) {
|
||||
return `${occurredAt} 00:00:00.000Z`;
|
||||
}
|
||||
const d = new Date(occurredAt);
|
||||
if (Number.isNaN(d.getTime())) return null;
|
||||
// PocketBase: "2026-09-13 07:48:31.000Z"
|
||||
return d.toISOString().replace("T", " ");
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
import { getCollection, pocketbaseConfigured, updateCollection } from "./client.js";
|
||||
|
||||
const SPENDINGS = "spendings";
|
||||
|
||||
const DESIRED_FIELDS = [
|
||||
{
|
||||
name: "invoice_id",
|
||||
type: "text",
|
||||
required: true,
|
||||
min: 0,
|
||||
},
|
||||
{
|
||||
name: "amount",
|
||||
type: "number",
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
name: "trx_date",
|
||||
type: "date",
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
name: "source",
|
||||
type: "text",
|
||||
required: true,
|
||||
min: 0,
|
||||
},
|
||||
{
|
||||
name: "description",
|
||||
type: "text",
|
||||
required: false,
|
||||
min: 0,
|
||||
},
|
||||
{
|
||||
name: "category",
|
||||
type: "text",
|
||||
required: false,
|
||||
min: 0,
|
||||
},
|
||||
{
|
||||
name: "message_id",
|
||||
type: "text",
|
||||
required: false,
|
||||
min: 0,
|
||||
},
|
||||
{
|
||||
name: "details",
|
||||
type: "json",
|
||||
required: false,
|
||||
},
|
||||
{
|
||||
name: "parsed",
|
||||
type: "json",
|
||||
required: false,
|
||||
},
|
||||
];
|
||||
|
||||
const UNIQUE_INVOICE_INDEX =
|
||||
"CREATE UNIQUE INDEX IF NOT EXISTS idx_spendings_invoice_id ON spendings (invoice_id)";
|
||||
|
||||
/**
|
||||
* Ensure spendings has the fields and unique invoice_id index we need.
|
||||
* Safe on an empty collection. Idempotent.
|
||||
*/
|
||||
export async function ensureSpendingsSchema(log = console) {
|
||||
if (!pocketbaseConfigured()) {
|
||||
log.warn?.("pocketbase: skipped schema ensure (not configured)");
|
||||
return { skipped: true };
|
||||
}
|
||||
|
||||
const { ok, status, data } = await getCollection(SPENDINGS);
|
||||
if (!ok) {
|
||||
throw new Error(
|
||||
`pocketbase: cannot load spendings collection (${status}): ${data.message || ""}`,
|
||||
);
|
||||
}
|
||||
|
||||
const existing = data.fields || [];
|
||||
const byName = new Map(existing.map((f) => [f.name, f]));
|
||||
|
||||
// Keep system/autodate fields; merge desired custom fields by name.
|
||||
const systemNames = new Set(["id", "created", "updated"]);
|
||||
const nextFields = existing.filter((f) => systemNames.has(f.name));
|
||||
|
||||
for (const desired of DESIRED_FIELDS) {
|
||||
const prev = byName.get(desired.name);
|
||||
if (prev) {
|
||||
nextFields.push({
|
||||
...prev,
|
||||
type: desired.type,
|
||||
required: desired.required,
|
||||
...(desired.min != null ? { min: desired.min } : {}),
|
||||
});
|
||||
} else {
|
||||
nextFields.push({ ...desired });
|
||||
}
|
||||
}
|
||||
|
||||
// Preserve any other unknown custom fields the user may have added.
|
||||
for (const f of existing) {
|
||||
if (systemNames.has(f.name)) continue;
|
||||
if (DESIRED_FIELDS.some((d) => d.name === f.name)) continue;
|
||||
nextFields.push(f);
|
||||
}
|
||||
|
||||
const indexes = Array.isArray(data.indexes) ? [...data.indexes] : [];
|
||||
if (!indexes.some((idx) => /invoice_id/i.test(String(idx)))) {
|
||||
indexes.push(UNIQUE_INVOICE_INDEX);
|
||||
}
|
||||
|
||||
const patch = { fields: nextFields, indexes };
|
||||
const result = await updateCollection(SPENDINGS, patch);
|
||||
if (!result.ok) {
|
||||
throw new Error(
|
||||
`pocketbase: failed to update spendings schema (${result.status}): ${result.data.message || JSON.stringify(result.data)}`,
|
||||
);
|
||||
}
|
||||
|
||||
log.info?.({ collection: SPENDINGS }, "pocketbase: spendings schema ensured");
|
||||
return { ok: true };
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
import { runPipeline } from "./pipeline.js";
|
||||
|
||||
/**
|
||||
* After fetch: if .eml attachments exist, process each; otherwise process outer.
|
||||
* `fetchResult` is the ctx after [logEvent, fetchAndLogMessage].
|
||||
*/
|
||||
export async function processFetchedMessage(fetchResult, processHandlers) {
|
||||
if (!fetchResult?.message) return [];
|
||||
|
||||
const workItems = fetchResult.emlMessages?.length
|
||||
? fetchResult.emlMessages
|
||||
: [fetchResult.message];
|
||||
|
||||
const results = [];
|
||||
for (const item of workItems) {
|
||||
const result = await runPipeline(processHandlers, {
|
||||
...fetchResult,
|
||||
message: item,
|
||||
messageId: item.id,
|
||||
// Clear prior match/parse/save state from a previous item
|
||||
rule: undefined,
|
||||
original: undefined,
|
||||
parsed: undefined,
|
||||
saved: undefined,
|
||||
duplicate: undefined,
|
||||
error: undefined,
|
||||
spendingId: undefined,
|
||||
});
|
||||
results.push(result);
|
||||
}
|
||||
return results;
|
||||
}
|
||||
@@ -0,0 +1,169 @@
|
||||
import { timingSafeEqual } from "node:crypto";
|
||||
import { PUBSUB_VERIFICATION_TOKEN } from "../config.js";
|
||||
import { getUserByEmail, getWatchState, saveWatchState } from "../db.js";
|
||||
import {
|
||||
gmailClientForUser,
|
||||
googleStatus,
|
||||
listHistory,
|
||||
startWatch,
|
||||
} from "../google.js";
|
||||
import { HttpError } from "../http.js";
|
||||
import { runPipeline } from "../pipeline.js";
|
||||
import { processFetchedMessage } from "../process-fetched.js";
|
||||
|
||||
const userLocks = new Map();
|
||||
|
||||
function withUserLock(userId, fn) {
|
||||
const prev = userLocks.get(userId) || Promise.resolve();
|
||||
const next = prev.then(fn, fn);
|
||||
userLocks.set(
|
||||
userId,
|
||||
next.then(
|
||||
() => {},
|
||||
() => {},
|
||||
),
|
||||
);
|
||||
return next;
|
||||
}
|
||||
|
||||
function tokenFromRequest(request) {
|
||||
const q = request.query?.token;
|
||||
if (typeof q === "string" && q) return q;
|
||||
const header =
|
||||
request.headers["x-pubsub-verification-token"] ||
|
||||
request.headers["x-goog-pubsub-verification-token"];
|
||||
if (typeof header === "string" && header) return header;
|
||||
const auth = request.headers.authorization;
|
||||
if (typeof auth === "string" && auth.startsWith("Bearer ")) {
|
||||
return auth.slice(7);
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
function tokensMatch(provided, expected) {
|
||||
const a = Buffer.from(provided);
|
||||
const b = Buffer.from(expected);
|
||||
if (a.length !== b.length) return false;
|
||||
return timingSafeEqual(a, b);
|
||||
}
|
||||
|
||||
function verifyPush(request) {
|
||||
if (!PUBSUB_VERIFICATION_TOKEN) return;
|
||||
if (!tokensMatch(tokenFromRequest(request), PUBSUB_VERIFICATION_TOKEN)) {
|
||||
throw new HttpError(403, "Invalid Pub/Sub verification token");
|
||||
}
|
||||
}
|
||||
|
||||
function decodeNotification(body) {
|
||||
const data = body?.message?.data;
|
||||
if (typeof data !== "string" || !data) return null;
|
||||
try {
|
||||
const json = Buffer.from(data, "base64").toString("utf8");
|
||||
const parsed = JSON.parse(json);
|
||||
if (!parsed?.emailAddress || parsed.historyId == null) return null;
|
||||
return {
|
||||
emailAddress: parsed.emailAddress,
|
||||
historyId: String(parsed.historyId),
|
||||
};
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async function processNotification(log, { fetchHandlers, processHandlers }, notification) {
|
||||
const user = getUserByEmail(notification.emailAddress);
|
||||
if (!user) {
|
||||
log.warn(
|
||||
{ email: notification.emailAddress },
|
||||
"gmail-watch: unknown mailbox, acking",
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
await withUserLock(user.id, () =>
|
||||
handleUserNotification(log, { fetchHandlers, processHandlers }, user, notification),
|
||||
);
|
||||
}
|
||||
|
||||
async function handleUserNotification(
|
||||
log,
|
||||
{ fetchHandlers, processHandlers },
|
||||
user,
|
||||
notification,
|
||||
) {
|
||||
const gmail = await gmailClientForUser(user.id);
|
||||
|
||||
const state = getWatchState(user.id);
|
||||
const startHistoryId = state?.history_id;
|
||||
if (!startHistoryId) {
|
||||
log.info(
|
||||
{ userId: user.id },
|
||||
"gmail-watch: no cursor yet, starting watch and skipping backlog",
|
||||
);
|
||||
await startWatch(user.id, { resetCursor: true });
|
||||
return;
|
||||
}
|
||||
|
||||
let result;
|
||||
try {
|
||||
result = await listHistory(gmail, startHistoryId);
|
||||
} catch (err) {
|
||||
const status = googleStatus(err);
|
||||
if (status === 404) {
|
||||
log.warn(
|
||||
{ userId: user.id, startHistoryId },
|
||||
"gmail-watch: history expired, re-watching without backfill",
|
||||
);
|
||||
await startWatch(user.id, { resetCursor: true });
|
||||
return;
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
|
||||
for (const messageId of result.messageIds) {
|
||||
const fetched = await runPipeline(fetchHandlers, {
|
||||
notification,
|
||||
userId: user.id,
|
||||
messageId,
|
||||
gmail,
|
||||
log,
|
||||
});
|
||||
await processFetchedMessage(fetched, processHandlers);
|
||||
}
|
||||
|
||||
saveWatchState(user.id, {
|
||||
historyId: result.historyId || notification.historyId,
|
||||
expiration: state.expiration || Date.now() + 7 * 24 * 3600 * 1000,
|
||||
});
|
||||
}
|
||||
|
||||
export async function registerPubsubRoutes(app, { fetchHandlers, processHandlers }) {
|
||||
app.post("/pubsub/gmail", async (request, reply) => {
|
||||
verifyPush(request);
|
||||
|
||||
const notification = decodeNotification(request.body);
|
||||
if (!notification) {
|
||||
request.log.warn("gmail-watch: undecodable Pub/Sub payload, acking");
|
||||
return reply.code(204).send();
|
||||
}
|
||||
|
||||
try {
|
||||
await processNotification(
|
||||
request.log,
|
||||
{ fetchHandlers, processHandlers },
|
||||
notification,
|
||||
);
|
||||
} catch (err) {
|
||||
const status = googleStatus(err) || err.statusCode || 500;
|
||||
if (status === 401 || status === 403) {
|
||||
throw new HttpError(
|
||||
500,
|
||||
"Gmail access expired or was revoked. Sign in again.",
|
||||
);
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
|
||||
return reply.code(204).send();
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
export const briTransfer = {
|
||||
kind: "bri.transfer.success",
|
||||
source: "bri",
|
||||
fromAddress: "bankbri@bri.co.id",
|
||||
subject: "Pemindahan Dana Sesama Rekening BRI",
|
||||
};
|
||||
@@ -0,0 +1,6 @@
|
||||
export const grabEreceipt = {
|
||||
kind: "grab.ereceipt.ride",
|
||||
source: "grab",
|
||||
fromAddress: "no-reply@grab.com",
|
||||
subject: "Your Grab E-Receipt",
|
||||
};
|
||||
@@ -0,0 +1,5 @@
|
||||
import { briTransfer } from "./bri-transfer.js";
|
||||
import { grabEreceipt } from "./grab-ereceipt.js";
|
||||
import { livinQrPayment } from "./livin-qr-payment.js";
|
||||
|
||||
export const rules = [livinQrPayment, grabEreceipt, briTransfer];
|
||||
@@ -0,0 +1,6 @@
|
||||
export const livinQrPayment = {
|
||||
kind: "livin.qr_payment.success",
|
||||
source: "livin",
|
||||
fromAddress: "noreply.livin@bankmandiri.co.id",
|
||||
subject: "Pembayaran Berhasil!",
|
||||
};
|
||||
@@ -0,0 +1,327 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { describe, it } from "node:test";
|
||||
import {
|
||||
expandWorkItems,
|
||||
listRfc822Parts,
|
||||
messageFromRfc822Part,
|
||||
} from "../src/mime.js";
|
||||
import { processFetchedMessage } from "../src/process-fetched.js";
|
||||
import { matchRule } from "../src/handlers/match-rule.js";
|
||||
import { parseMatchedMessage } from "../src/handlers/parse-matched-message.js";
|
||||
|
||||
const silentLog = {
|
||||
info() {},
|
||||
warn() {},
|
||||
error() {},
|
||||
};
|
||||
|
||||
function textPart(text, partId = "0.0") {
|
||||
return {
|
||||
partId,
|
||||
mimeType: "text/plain",
|
||||
body: {
|
||||
data: Buffer.from(text, "utf8")
|
||||
.toString("base64")
|
||||
.replace(/\+/g, "-")
|
||||
.replace(/\//g, "_")
|
||||
.replace(/=+$/, ""),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function htmlPart(html, partId = "0.1") {
|
||||
return {
|
||||
partId,
|
||||
mimeType: "text/html",
|
||||
body: {
|
||||
data: Buffer.from(html, "utf8")
|
||||
.toString("base64")
|
||||
.replace(/\+/g, "-")
|
||||
.replace(/\//g, "_")
|
||||
.replace(/=+$/, ""),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function rfc822Part({
|
||||
partId,
|
||||
filename,
|
||||
from,
|
||||
subject,
|
||||
date,
|
||||
bodyText,
|
||||
bodyHtml,
|
||||
}) {
|
||||
return {
|
||||
partId,
|
||||
mimeType: "message/rfc822",
|
||||
filename,
|
||||
parts: [
|
||||
{
|
||||
partId: `${partId}.0`,
|
||||
mimeType: "multipart/alternative",
|
||||
headers: [
|
||||
{ name: "From", value: from },
|
||||
{ name: "Subject", value: subject },
|
||||
{ name: "Date", value: date },
|
||||
],
|
||||
parts: [
|
||||
textPart(bodyText, `${partId}.0.0`),
|
||||
...(bodyHtml ? [htmlPart(bodyHtml, `${partId}.0.1`)] : []),
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
const grabRideBody = `Bike Hemat
|
||||
Hope you enjoyed your ride!
|
||||
Picked up on 13 September 2026
|
||||
Booking ID: A-TESTBOOKING1
|
||||
Total Paid
|
||||
Rp16.600
|
||||
Receipt issued by the driver Mahendra.
|
||||
Passenger
|
||||
Bintang
|
||||
`;
|
||||
|
||||
const grabFoodBody = `Selamat menikmati makanan Anda!
|
||||
TOTAL
|
||||
IDR 49000 TANGGAL | WAKTU
|
||||
11 Sep 26 18:24 +0700
|
||||
Detail Pesanan
|
||||
Jenis Kendaraan:
|
||||
GrabFood
|
||||
Diterbitkan untuk
|
||||
Bintang
|
||||
Pesanan ID
|
||||
00112712674-TESTFOOD1
|
||||
Pesanan Dari:
|
||||
Doner Kebab - Kalibata City
|
||||
Profil:
|
||||
Personal
|
||||
Detail Tagihan
|
||||
Detail Pembayaran:
|
||||
OVO Cash
|
||||
TOTAL (INCL. TAX) IDR 49000
|
||||
`;
|
||||
|
||||
describe("listRfc822Parts", () => {
|
||||
it("finds message/rfc822 attachments and does not treat outer as eml", () => {
|
||||
const payload = {
|
||||
mimeType: "multipart/mixed",
|
||||
parts: [
|
||||
{
|
||||
mimeType: "multipart/alternative",
|
||||
parts: [textPart("wrapper")],
|
||||
},
|
||||
rfc822Part({
|
||||
partId: "1",
|
||||
filename: "Your Grab E-Receipt.eml",
|
||||
from: "Grab <no-reply@grab.com>",
|
||||
subject: "Your Grab E-Receipt",
|
||||
date: "Wed, 9 Sep 2026 06:09:32 +0000",
|
||||
bodyText: grabRideBody,
|
||||
}),
|
||||
rfc822Part({
|
||||
partId: "2",
|
||||
filename: "Your Grab E-Receipt.eml",
|
||||
from: "Grab <no-reply@grab.com>",
|
||||
subject: "Your Grab E-Receipt",
|
||||
date: "Thu, 10 Sep 2026 11:28:24 +0000",
|
||||
bodyText: grabFoodBody,
|
||||
}),
|
||||
],
|
||||
};
|
||||
const parts = listRfc822Parts(payload);
|
||||
assert.equal(parts.length, 2);
|
||||
assert.equal(parts[0].filename, "Your Grab E-Receipt.eml");
|
||||
});
|
||||
});
|
||||
|
||||
describe("messageFromRfc822Part", () => {
|
||||
it("builds synthetic message with nested From/Subject/body", () => {
|
||||
const part = rfc822Part({
|
||||
partId: "1",
|
||||
filename: "Your Grab E-Receipt.eml",
|
||||
from: "Grab <no-reply@grab.com>",
|
||||
subject: "Your Grab E-Receipt",
|
||||
date: "Wed, 9 Sep 2026 06:09:32 +0000",
|
||||
bodyText: grabRideBody,
|
||||
});
|
||||
const msg = messageFromRfc822Part(part, {
|
||||
parentMessageId: "outer123",
|
||||
index: 0,
|
||||
});
|
||||
assert.equal(msg.id, "outer123#1");
|
||||
assert.equal(msg.from, "Grab <no-reply@grab.com>");
|
||||
assert.equal(msg.subject, "Your Grab E-Receipt");
|
||||
assert.match(msg.body, /Booking ID: A-TESTBOOKING1/);
|
||||
assert.equal(msg.parentMessageId, "outer123");
|
||||
assert.equal(msg.attachmentFilename, "Your Grab E-Receipt.eml");
|
||||
});
|
||||
});
|
||||
|
||||
describe("expandWorkItems", () => {
|
||||
it("returns outer only when no eml", () => {
|
||||
const outer = {
|
||||
id: "outer",
|
||||
from: "me@x.com",
|
||||
subject: "hi",
|
||||
body: "x",
|
||||
html: "",
|
||||
};
|
||||
const payload = {
|
||||
mimeType: "multipart/alternative",
|
||||
parts: [textPart("hi")],
|
||||
};
|
||||
const items = expandWorkItems(outer, payload);
|
||||
assert.equal(items.length, 1);
|
||||
assert.equal(items[0], outer);
|
||||
});
|
||||
|
||||
it("returns eml messages when attachments exist", () => {
|
||||
const outer = {
|
||||
id: "outer123",
|
||||
from: "NSRB <nasyarobby@gmail.com>",
|
||||
subject: "Test attachment",
|
||||
body: "Regards",
|
||||
html: "",
|
||||
};
|
||||
const payload = {
|
||||
mimeType: "multipart/mixed",
|
||||
parts: [
|
||||
{ mimeType: "multipart/alternative", parts: [textPart("Regards")] },
|
||||
rfc822Part({
|
||||
partId: "1",
|
||||
filename: "a.eml",
|
||||
from: "Grab <no-reply@grab.com>",
|
||||
subject: "Your Grab E-Receipt",
|
||||
date: "Wed, 9 Sep 2026 06:09:32 +0000",
|
||||
bodyText: grabRideBody,
|
||||
}),
|
||||
],
|
||||
};
|
||||
const items = expandWorkItems(outer, payload);
|
||||
assert.equal(items.length, 1);
|
||||
assert.equal(items[0].from, "Grab <no-reply@grab.com>");
|
||||
assert.equal(items[0].id, "outer123#1");
|
||||
});
|
||||
});
|
||||
|
||||
describe("processFetchedMessage fan-out", () => {
|
||||
it("processes each eml attachment and skips outer wrapper", async () => {
|
||||
const fetched = {
|
||||
notification: { emailAddress: "eleven16th@gmail.com" },
|
||||
log: silentLog,
|
||||
message: {
|
||||
id: "outer123",
|
||||
from: "NSRB <nasyarobby@gmail.com>",
|
||||
subject: "Test attachment",
|
||||
body: "Regards",
|
||||
html: "",
|
||||
},
|
||||
emlMessages: [
|
||||
{
|
||||
id: "outer123#1",
|
||||
from: "Grab <no-reply@grab.com>",
|
||||
subject: "Your Grab E-Receipt",
|
||||
date: "Wed, 9 Sep 2026 06:09:32 +0000",
|
||||
body: grabRideBody,
|
||||
html: "",
|
||||
parentMessageId: "outer123",
|
||||
},
|
||||
{
|
||||
id: "outer123#2",
|
||||
from: "Grab <no-reply@grab.com>",
|
||||
subject: "Your Grab E-Receipt",
|
||||
date: "Thu, 10 Sep 2026 11:28:24 +0000",
|
||||
body: grabFoodBody,
|
||||
html: "",
|
||||
parentMessageId: "outer123",
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
const results = await processFetchedMessage(fetched, [
|
||||
matchRule,
|
||||
parseMatchedMessage,
|
||||
]);
|
||||
assert.equal(results.length, 2);
|
||||
assert.equal(results[0].parsed?.kind, "grab.ereceipt.ride");
|
||||
assert.equal(results[0].parsed?.reference, "A-TESTBOOKING1");
|
||||
assert.equal(results[1].parsed?.kind, "grab.ereceipt.food");
|
||||
assert.equal(results[1].parsed?.reference, "00112712674-TESTFOOD1");
|
||||
});
|
||||
|
||||
it("processes outer message when no eml attachments", async () => {
|
||||
const fetched = {
|
||||
notification: { emailAddress: "eleven16th@gmail.com" },
|
||||
log: silentLog,
|
||||
message: {
|
||||
id: "direct1",
|
||||
from: `"Livin'" <noreply.livin@bankmandiri.co.id>`,
|
||||
subject: "Pembayaran Berhasil!",
|
||||
date: "Sun, 13 Sep 2026 14:48:36 +0700",
|
||||
body: `Halo TEST,
|
||||
Penerima
|
||||
Shop
|
||||
JAKARTA SELATAN - ID
|
||||
Tanggal 13 Sep 2026
|
||||
Jam 10:00:00 WIB
|
||||
Nominal Transaksi Rp 10.000,00
|
||||
No. Referensi 9999999999999999999`,
|
||||
html: "",
|
||||
},
|
||||
emlMessages: [],
|
||||
};
|
||||
|
||||
const results = await processFetchedMessage(fetched, [
|
||||
matchRule,
|
||||
parseMatchedMessage,
|
||||
]);
|
||||
assert.equal(results.length, 1);
|
||||
assert.equal(results[0].parsed?.kind, "livin.qr_payment.success");
|
||||
});
|
||||
});
|
||||
|
||||
describe("matchRule on synthetic eml", () => {
|
||||
it("matches Grab from nested headers without Fwd", async () => {
|
||||
const result = await matchRule({
|
||||
messageId: "outer#1",
|
||||
log: silentLog,
|
||||
message: {
|
||||
id: "outer#1",
|
||||
from: "Grab <no-reply@grab.com>",
|
||||
subject: "Your Grab E-Receipt",
|
||||
body: grabRideBody,
|
||||
html: "",
|
||||
},
|
||||
});
|
||||
assert.equal(result.pass.rule.kind, "grab.ereceipt.ride");
|
||||
assert.equal(result.pass.original.from, "Grab <no-reply@grab.com>");
|
||||
});
|
||||
});
|
||||
|
||||
describe("extractMimePartsFromRawEml", () => {
|
||||
it("pulls text/html from a simple multipart eml", async () => {
|
||||
const { extractMimePartsFromRawEml } = await import("../src/mime.js");
|
||||
const raw = [
|
||||
'Content-Type: multipart/alternative; boundary="bnd"',
|
||||
"",
|
||||
"--bnd",
|
||||
"Content-Type: text/plain; charset=UTF-8",
|
||||
"",
|
||||
"Hello plain",
|
||||
"--bnd",
|
||||
"Content-Type: text/html; charset=UTF-8",
|
||||
"",
|
||||
"<b>Hello html</b>",
|
||||
"--bnd--",
|
||||
"",
|
||||
].join("\r\n");
|
||||
const parts = extractMimePartsFromRawEml(raw);
|
||||
assert.match(parts.text, /Hello plain/);
|
||||
assert.match(parts.html, /Hello html/);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,233 @@
|
||||
export const livinDirectBody = `Livin' by Mandiri
|
||||
|
||||
Pembayaran Berhasil
|
||||
Halo NASYAROBBY NUGRAHA PUTRA,
|
||||
|
||||
Berikut adalah detail transaksi Anda dengan QR:
|
||||
|
||||
Penerima
|
||||
|
||||
SNACK BAROKAHJAKARTA BARAT - ID
|
||||
|
||||
Tanggal13 Sep 2026
|
||||
|
||||
|
||||
Jam14:48:31 WIB
|
||||
|
||||
|
||||
Nominal TransaksiRp 14.000,00
|
||||
|
||||
|
||||
No. Referensi2609131121540314991
|
||||
|
||||
|
||||
No. Ref. QRIS609532729372
|
||||
|
||||
|
||||
Merchant PAN9360091436372513569
|
||||
|
||||
|
||||
Customer PAN9360000822021345851
|
||||
|
||||
|
||||
PengakuisisiGoPay
|
||||
|
||||
|
||||
Terminal ID A01
|
||||
|
||||
Sumber Dana
|
||||
|
||||
Kartu Kredit - Mandiri Platinum ****4585
|
||||
|
||||
Simpan email ini sebagai ref`;
|
||||
|
||||
export const livinForwardedBody = `---------- Forwarded message ---------
|
||||
From: Livin' <noreply.livin@bankmandiri.co.id>
|
||||
Date: Sun, Sep 13, 2026 at 10:44 AM
|
||||
Subject: Pembayaran Berhasil!
|
||||
To: <nasyarobby@gmail.com>
|
||||
|
||||
|
||||
[image: livinbymandiri]
|
||||
Pembayaran Berhasil
|
||||
|
||||
Halo NASYAROBBY NUGRAHA PUTRA,
|
||||
|
||||
Berikut adalah detail transaksi Anda dengan QR:
|
||||
|
||||
Penerima
|
||||
Eleven Chicken Cabang Gah
|
||||
|
||||
JAKARTA SELATAN - ID
|
||||
|
||||
Tanggal 13 Sep 2026
|
||||
|
||||
|
||||
Jam 10:43:51 WIB
|
||||
|
||||
|
||||
Nominal Transaksi Rp 22.000,00
|
||||
|
||||
|
||||
No. Referensi 2609131121539559589`;
|
||||
|
||||
export const grabBody = `Bike Hemat
|
||||
[image: Grab]
|
||||
|
||||
Hope you enjoyed your ride!
|
||||
Picked up on 13 September 2026
|
||||
Booking ID: A-9RXV4K2GWCLOAV
|
||||
Total Paid
|
||||
Rp16.600
|
||||
Receipt issued by the driver Mahendra.
|
||||
5.0 [image: stars]
|
||||
Compliments for driver
|
||||
Layanan Mantap
|
||||
Breakdown
|
||||
Fare
|
||||
16.600
|
||||
Fare
|
||||
16.600
|
||||
Total Paid
|
||||
16.600
|
||||
Passenger
|
||||
Bintang
|
||||
Profile
|
||||
`;
|
||||
|
||||
export const grabForwardedBody = `---------- Forwarded message ---------
|
||||
From: Grab <no-reply@grab.com>
|
||||
Date: Sun, Sep 13, 2026 at 10:58 AM
|
||||
Subject: Your Grab E-Receipt
|
||||
To: <nasyarobby@gmail.com>
|
||||
|
||||
|
||||
${grabBody}`;
|
||||
|
||||
export const grabTipForwardedBody = `---------- Forwarded message ---------
|
||||
From: Grab <no-reply@grab.com>
|
||||
Date: Sun, Sep 13, 2026 at 11:51 AM
|
||||
Subject: Your Grab E-Receipt
|
||||
To: <nasyarobby@gmail.com>
|
||||
|
||||
|
||||
Bike Hemat
|
||||
[image: Grab]
|
||||
|
||||
Thanks! Your tip goes a long way for your driver.
|
||||
13 Sep 26 10:47 +0700
|
||||
Total Paid
|
||||
Rp2.000
|
||||
Tip
|
||||
From: Apartemen Kalibata City, Jl. Kalibata Raya No.1, Rawajati, Pancoran,
|
||||
Kota Jakarta Selatan, DKI Jakarta, Indonesia, 12750
|
||||
Booking ID: A-9RXV4K2GWCLOAV
|
||||
`;
|
||||
|
||||
/** Minimal Grab ride HTML: pickup/dropoff times only exist here, not in plain text. */
|
||||
export const grabRideHtml = `
|
||||
<div>Picked up on 13 September 2026</div>
|
||||
<div>Booking ID: A-9RXV4K2GWCLOAV</div>
|
||||
<table>
|
||||
<tr>
|
||||
<td>
|
||||
<div>Gate II Apartemen Kalibata City</div>
|
||||
<div style="font-weight:300">10:47AM</div>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>
|
||||
<div>Badan Koordinasi Penanaman Modal</div>
|
||||
<div style="font-weight:300">10:58AM</div>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
`;
|
||||
|
||||
export const grabFoodForwardedBody = `---------- Forwarded message ---------
|
||||
From: <no-reply@grab.com>
|
||||
Date: Fri, Sep 11, 2026 at 7:01 PM
|
||||
Subject: Your Grab E-Receipt
|
||||
To: <nasyarobby@gmail.com>
|
||||
|
||||
|
||||
[image: E-receipt]
|
||||
Selamat menikmati makanan Anda!
|
||||
TOTAL
|
||||
IDR 49000 TANGGAL | WAKTU
|
||||
11 Sep 26 18:24 +0700
|
||||
Detail Pesanan
|
||||
Jenis Kendaraan:
|
||||
GrabFood
|
||||
Diterbitkan untuk
|
||||
Bintang
|
||||
Pesanan ID
|
||||
00112712674-C8EUEZDGEEAZTX
|
||||
Pesanan Dari:
|
||||
Doner Kebab - Kalibata City
|
||||
Kalibata City, Foodcourt Teras Kuliner, Lt. Dasar Units Fb. 23 (Doner
|
||||
Kebab), Jl. Raya Kalibata No. 1, Rawajati, Pancoran, Jakarta Selatan
|
||||
Profil:
|
||||
Personal
|
||||
Detail Tagihan
|
||||
Detail Pembayaran:
|
||||
OVO Cash
|
||||
Detail Pembayaran: Jumlah:
|
||||
1x Fish n Chips IDR 45000
|
||||
|
||||
Subtotal IDR 45000
|
||||
Restaurant packaging charge IDR 4000
|
||||
TOTAL (INCL. TAX) IDR 49000
|
||||
`;
|
||||
|
||||
export const briTransferDirectBody = `Halo,*NASYAROBBY NUGRAHA PUTRA*
|
||||
Berikut ini adalah informasi transaksi yang telah Anda lakukan di Aplikasi
|
||||
BRImo.
|
||||
Transaksi Berhasil
|
||||
Tanggal 13 September 2026 , 19:00:00 WIB
|
||||
Nomor Referensi 212189333864
|
||||
------------------------------
|
||||
Sumber Dana NASYAROBBY NUGRAHA PUTRA
|
||||
0121 **** **** 500
|
||||
Jenis Transaksi Transfer Bank BRI
|
||||
Bank Tujuan BANK BRI
|
||||
Nomor Tujuan 1060 0100 2070 507
|
||||
Nama Tujuan NELLY ARISANTI
|
||||
Catatan -
|
||||
------------------------------
|
||||
Nominal Rp4.000.000
|
||||
Biaya Admin Rp0
|
||||
*Total* *Rp4.000.000*
|
||||
|
||||
|
||||
Semoga informasi ini dapat bermanfaat bagi Anda.`;
|
||||
|
||||
export const briTransferForwardedBody = `Nasyarobby N. Putra
|
||||
Boston University MSSD Program
|
||||
|
||||
---------- Forwarded message ---------
|
||||
From: Bank BRI <BankBRI@bri.co.id>
|
||||
Date: Sun, Sep 13, 2026, 7:00 PM
|
||||
Subject: Pemindahan Dana Sesama Rekening BRI
|
||||
To: <nsrb@bu.edu>
|
||||
|
||||
|
||||
Halo,*NASYAROBBY NUGRAHA PUTRA*
|
||||
Berikut ini adalah informasi transaksi yang telah Anda lakukan di Aplikasi
|
||||
BRImo.
|
||||
Transaksi Berhasil
|
||||
Tanggal 13 September 2026 , 19:00:00 WIB
|
||||
Nomor Referensi 212189333864
|
||||
------------------------------
|
||||
Sumber Dana NASYAROBBY NUGRAHA PUTRA
|
||||
0121 **** **** 500
|
||||
Jenis Transaksi Transfer Bank BRI
|
||||
Bank Tujuan BANK BRI
|
||||
Nomor Tujuan 1060 0100 2070 507
|
||||
Nama Tujuan NELLY ARISANTI
|
||||
Catatan -
|
||||
------------------------------
|
||||
Nominal Rp4.000.000
|
||||
Biaya Admin Rp0
|
||||
*Total* *Rp4.000.000*
|
||||
`;
|
||||
@@ -0,0 +1,171 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { describe, it } from "node:test";
|
||||
import {
|
||||
extractEmail,
|
||||
parseForwardedHeader,
|
||||
resolveOriginal,
|
||||
subjectMatches,
|
||||
} from "../src/lib/email.js";
|
||||
import { matchRule } from "../src/handlers/match-rule.js";
|
||||
import {
|
||||
briTransferDirectBody,
|
||||
briTransferForwardedBody,
|
||||
grabBody,
|
||||
grabForwardedBody,
|
||||
livinDirectBody,
|
||||
livinForwardedBody,
|
||||
} from "./fixtures.js";
|
||||
|
||||
const silentLog = {
|
||||
info() {},
|
||||
warn() {},
|
||||
error() {},
|
||||
};
|
||||
|
||||
describe("extractEmail", () => {
|
||||
it("parses angle-bracket addresses", () => {
|
||||
assert.equal(
|
||||
extractEmail(`"Livin'" <noreply.livin@bankmandiri.co.id>`),
|
||||
"noreply.livin@bankmandiri.co.id",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseForwardedHeader", () => {
|
||||
it("reads original From/Subject from a Fwd block", () => {
|
||||
const h = parseForwardedHeader(livinForwardedBody);
|
||||
assert.ok(h);
|
||||
assert.match(h.from, /noreply\.livin@bankmandiri\.co\.id/);
|
||||
assert.equal(h.subject, "Pembayaran Berhasil!");
|
||||
});
|
||||
|
||||
it("returns null when there is no Fwd block", () => {
|
||||
assert.equal(parseForwardedHeader(livinDirectBody), null);
|
||||
});
|
||||
});
|
||||
|
||||
describe("resolveOriginal", () => {
|
||||
it("uses envelope for direct mail", () => {
|
||||
const o = resolveOriginal({
|
||||
from: `"Livin'" <noreply.livin@bankmandiri.co.id>`,
|
||||
subject: "Pembayaran Berhasil!",
|
||||
date: "Sun, 13 Sep 2026 14:48:36 +0700 (WIB)",
|
||||
body: livinDirectBody,
|
||||
});
|
||||
assert.equal(o.viaForward, false);
|
||||
assert.equal(extractEmail(o.from), "noreply.livin@bankmandiri.co.id");
|
||||
});
|
||||
|
||||
it("uses Fwd header for forwarded mail", () => {
|
||||
const o = resolveOriginal({
|
||||
from: "NSRB <nasyarobby@gmail.com>",
|
||||
subject: "Fwd: Your Grab E-Receipt",
|
||||
body: grabForwardedBody,
|
||||
});
|
||||
assert.equal(o.viaForward, true);
|
||||
assert.equal(extractEmail(o.from), "no-reply@grab.com");
|
||||
assert.equal(o.subject, "Your Grab E-Receipt");
|
||||
});
|
||||
});
|
||||
|
||||
describe("subjectMatches", () => {
|
||||
it("matches exact and Fwd-prefixed subjects", () => {
|
||||
assert.equal(subjectMatches("Pembayaran Berhasil!", "Pembayaran Berhasil!"), true);
|
||||
assert.equal(subjectMatches("Fwd: Pembayaran Berhasil!", "Pembayaran Berhasil!"), true);
|
||||
assert.equal(subjectMatches("Hello", "Pembayaran Berhasil!"), false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("matchRule", () => {
|
||||
it("matches direct Livin", async () => {
|
||||
const result = await matchRule({
|
||||
messageId: "1a099bd61efd234d",
|
||||
log: silentLog,
|
||||
message: {
|
||||
from: `"Livin'" <noreply.livin@bankmandiri.co.id>`,
|
||||
subject: "Pembayaran Berhasil!",
|
||||
date: "Sun, 13 Sep 2026 14:48:36 +0700 (WIB)",
|
||||
body: livinDirectBody,
|
||||
},
|
||||
});
|
||||
assert.equal(result.pass.rule.kind, "livin.qr_payment.success");
|
||||
});
|
||||
|
||||
it("matches forwarded Livin via original header", async () => {
|
||||
const result = await matchRule({
|
||||
messageId: "1a099b0c04c4397a",
|
||||
log: silentLog,
|
||||
message: {
|
||||
from: "NSRB <nasyarobby@gmail.com>",
|
||||
subject: "Fwd: Pembayaran Berhasil!",
|
||||
body: livinForwardedBody,
|
||||
},
|
||||
});
|
||||
assert.equal(result.pass.rule.kind, "livin.qr_payment.success");
|
||||
});
|
||||
|
||||
it("matches direct Grab", async () => {
|
||||
const result = await matchRule({
|
||||
messageId: "grab-direct",
|
||||
log: silentLog,
|
||||
message: {
|
||||
from: "Grab <no-reply@grab.com>",
|
||||
subject: "Your Grab E-Receipt",
|
||||
body: grabBody,
|
||||
},
|
||||
});
|
||||
assert.equal(result.pass.rule.kind, "grab.ereceipt.ride");
|
||||
});
|
||||
|
||||
it("matches forwarded Grab via original header", async () => {
|
||||
const result = await matchRule({
|
||||
messageId: "1a099b8b9d239e5e",
|
||||
log: silentLog,
|
||||
message: {
|
||||
from: "NSRB <nasyarobby@gmail.com>",
|
||||
subject: "Fwd: Your Grab E-Receipt",
|
||||
body: grabForwardedBody,
|
||||
},
|
||||
});
|
||||
assert.equal(result.pass.rule.kind, "grab.ereceipt.ride");
|
||||
});
|
||||
|
||||
it("matches direct BRI transfer", async () => {
|
||||
const result = await matchRule({
|
||||
messageId: "bri-direct",
|
||||
log: silentLog,
|
||||
message: {
|
||||
from: "Bank BRI <BankBRI@bri.co.id>",
|
||||
subject: "Pemindahan Dana Sesama Rekening BRI",
|
||||
body: briTransferDirectBody,
|
||||
},
|
||||
});
|
||||
assert.equal(result.pass.rule.kind, "bri.transfer.success");
|
||||
});
|
||||
|
||||
it("matches forwarded BRI transfer via original header", async () => {
|
||||
const result = await matchRule({
|
||||
messageId: "1a09b6cd9a3bc6bd",
|
||||
log: silentLog,
|
||||
message: {
|
||||
from: "Nasyarobby Putra <nsrb@bu.edu>",
|
||||
subject: "Fwd: Pemindahan Dana Sesama Rekening BRI",
|
||||
body: briTransferForwardedBody,
|
||||
},
|
||||
});
|
||||
assert.equal(result.pass.rule.kind, "bri.transfer.success");
|
||||
});
|
||||
|
||||
it("breaks on unknown from/subject", async () => {
|
||||
const result = await matchRule({
|
||||
messageId: "x",
|
||||
log: silentLog,
|
||||
message: {
|
||||
from: "Someone <other@example.com>",
|
||||
subject: "Hello",
|
||||
body: "hi",
|
||||
},
|
||||
});
|
||||
assert.equal(result.break, true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,23 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { describe, it } from "node:test";
|
||||
import { formatReceiptAlert } from "../src/notify/ntfy.js";
|
||||
|
||||
describe("formatReceiptAlert", () => {
|
||||
it("includes reason, messageId, and body", () => {
|
||||
const text = formatReceiptAlert({
|
||||
reason: "parse failed",
|
||||
messageId: "abc#1",
|
||||
kind: "grab.ereceipt.ride",
|
||||
message: {
|
||||
from: "Grab <no-reply@grab.com>",
|
||||
subject: "Your Grab E-Receipt",
|
||||
body: "Total Paid\nRp1.000",
|
||||
html: "<b>x</b>",
|
||||
},
|
||||
});
|
||||
assert.match(text, /reason: parse failed/);
|
||||
assert.match(text, /messageId: abc#1/);
|
||||
assert.match(text, /Total Paid/);
|
||||
assert.match(text, /html \(length\)/);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,17 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { describe, it } from "node:test";
|
||||
import { parseDayMonthYear } from "../src/lib/parse-helpers.js";
|
||||
|
||||
describe("parseDayMonthYear", () => {
|
||||
it("parses English months", () => {
|
||||
assert.equal(parseDayMonthYear("13 Sep 2026"), "2026-09-13");
|
||||
assert.equal(parseDayMonthYear("13 September 2026"), "2026-09-13");
|
||||
});
|
||||
|
||||
it("parses Indonesian months", () => {
|
||||
assert.equal(parseDayMonthYear("15 Agu 2026"), "2026-08-15");
|
||||
assert.equal(parseDayMonthYear("15 Agustus 2026"), "2026-08-15");
|
||||
assert.equal(parseDayMonthYear("1 Des 26"), "2026-12-01");
|
||||
assert.equal(parseDayMonthYear("1 Mei 2026"), "2026-05-01");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,276 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { describe, it } from "node:test";
|
||||
import { parseBriTransfer } from "../src/parsers/bri.js";
|
||||
import { parseGrabReceipt, extractGrabTripTimesFromHtml } from "../src/parsers/grab.js";
|
||||
import { parseLivinReceipt, splitMerchantLocation } from "../src/parsers/livin.js";
|
||||
import { toSpendingRecord, toPocketBaseDate } from "../src/pocketbase/map.js";
|
||||
import { briTransfer } from "../src/rules/bri-transfer.js";
|
||||
import { livinQrPayment } from "../src/rules/livin-qr-payment.js";
|
||||
import { grabEreceipt } from "../src/rules/grab-ereceipt.js";
|
||||
import {
|
||||
briTransferDirectBody,
|
||||
briTransferForwardedBody,
|
||||
grabBody,
|
||||
grabFoodForwardedBody,
|
||||
grabForwardedBody,
|
||||
grabRideHtml,
|
||||
grabTipForwardedBody,
|
||||
livinDirectBody,
|
||||
livinForwardedBody,
|
||||
} from "./fixtures.js";
|
||||
|
||||
describe("splitMerchantLocation", () => {
|
||||
it("splits clean multi-line penerima", () => {
|
||||
const r = splitMerchantLocation([
|
||||
"Eleven Chicken Cabang Gah",
|
||||
"JAKARTA SELATAN - ID",
|
||||
]);
|
||||
assert.equal(r.merchant, "Eleven Chicken Cabang Gah");
|
||||
assert.equal(r.location, "JAKARTA SELATAN - ID");
|
||||
});
|
||||
|
||||
it("keeps glued merchant+city as merchant only", () => {
|
||||
const r = splitMerchantLocation(["SNACK BAROKAHJAKARTA BARAT - ID"]);
|
||||
assert.equal(r.merchant, "SNACK BAROKAHJAKARTA BARAT - ID");
|
||||
assert.equal(r.location, null);
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseLivinReceipt", () => {
|
||||
it("parses direct HTML-stripped body", () => {
|
||||
const r = parseLivinReceipt(livinDirectBody);
|
||||
assert.ok(r);
|
||||
assert.equal(r.amount, 14000);
|
||||
assert.equal(r.reference, "2609131121540314991");
|
||||
assert.equal(r.occurredAt, "2026-09-13T14:48:31+07:00");
|
||||
assert.equal(r.details.payerName, "NASYAROBBY NUGRAHA PUTRA");
|
||||
assert.equal(r.details.merchant, "SNACK BAROKAHJAKARTA BARAT - ID");
|
||||
assert.equal(r.details.location, undefined);
|
||||
assert.equal(r.details.qrisRef, "609532729372");
|
||||
assert.equal(r.details.acquirer, "GoPay");
|
||||
});
|
||||
|
||||
it("parses forwarded clean body", () => {
|
||||
const r = parseLivinReceipt(livinForwardedBody);
|
||||
assert.ok(r);
|
||||
assert.equal(r.amount, 22000);
|
||||
assert.equal(r.reference, "2609131121539559589");
|
||||
assert.equal(r.occurredAt, "2026-09-13T10:43:51+07:00");
|
||||
assert.equal(r.details.merchant, "Eleven Chicken Cabang Gah");
|
||||
assert.equal(r.details.location, "JAKARTA SELATAN - ID");
|
||||
});
|
||||
|
||||
it("returns null without amount/reference", () => {
|
||||
assert.equal(parseLivinReceipt("Halo someone\nPenerima\nFoo"), null);
|
||||
});
|
||||
|
||||
it("parses Indonesian month abbreviation (Agu)", () => {
|
||||
const body = `Halo NASYAROBBY NUGRAHA PUTRA,
|
||||
Penerima
|
||||
RM Bu Sum Ebony, KalibataJAKARTA SELATAN - ID
|
||||
Tanggal15 Agu 2026
|
||||
Jam09:45:25 WIB
|
||||
Nominal TransaksiRp 15.000,00
|
||||
No. Referensi2608151121545704363
|
||||
No. Ref. QRIS608550990105`;
|
||||
const r = parseLivinReceipt(body);
|
||||
assert.ok(r);
|
||||
assert.equal(r.amount, 15000);
|
||||
assert.equal(r.reference, "2608151121545704363");
|
||||
assert.equal(r.occurredAt, "2026-08-15T09:45:25+07:00");
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseGrabReceipt", () => {
|
||||
it("parses direct Grab body", () => {
|
||||
const r = parseGrabReceipt(grabBody);
|
||||
assert.ok(r);
|
||||
assert.equal(r.kind, "grab.ereceipt.ride");
|
||||
assert.equal(r.amount, 16600);
|
||||
assert.equal(r.reference, "A-9RXV4K2GWCLOAV");
|
||||
assert.equal(r.occurredAt, "2026-09-13");
|
||||
assert.equal(r.details.product, "Bike Hemat");
|
||||
assert.equal(r.details.driver, "Mahendra");
|
||||
assert.equal(r.details.passenger, "Bintang");
|
||||
assert.deepEqual(r.details.compliments, ["Layanan Mantap"]);
|
||||
});
|
||||
|
||||
it("merges pickup time from HTML into occurredAt", () => {
|
||||
const r = parseGrabReceipt(grabBody, { html: grabRideHtml });
|
||||
assert.ok(r);
|
||||
assert.equal(r.occurredAt, "2026-09-13T10:47:00+07:00");
|
||||
assert.equal(r.details.dropoffAt, "2026-09-13T10:58:00+07:00");
|
||||
});
|
||||
|
||||
it("extracts standalone AM/PM trip times from HTML", () => {
|
||||
const t = extractGrabTripTimesFromHtml(grabRideHtml);
|
||||
assert.equal(t.pickup, "10:47:00");
|
||||
assert.equal(t.dropoff, "10:58:00");
|
||||
});
|
||||
|
||||
it("parses forwarded Grab body", () => {
|
||||
const r = parseGrabReceipt(grabForwardedBody);
|
||||
assert.ok(r);
|
||||
assert.equal(r.amount, 16600);
|
||||
assert.equal(r.reference, "A-9RXV4K2GWCLOAV");
|
||||
assert.equal(r.details.product, "Bike Hemat");
|
||||
assert.equal(r.kind, "grab.ereceipt.ride");
|
||||
});
|
||||
|
||||
it("parses Grab tip as a separate kind/invoice", () => {
|
||||
const r = parseGrabReceipt(grabTipForwardedBody);
|
||||
assert.ok(r);
|
||||
assert.equal(r.kind, "grab.ereceipt.tip");
|
||||
assert.equal(r.variant, "tip");
|
||||
assert.equal(r.amount, 2000);
|
||||
assert.equal(r.reference, "A-9RXV4K2GWCLOAV:tip");
|
||||
assert.equal(r.occurredAt, "2026-09-13T10:47:00+07:00");
|
||||
assert.equal(r.details.product, "Bike Hemat");
|
||||
assert.equal(r.details.variant, "tip");
|
||||
});
|
||||
|
||||
it("parses Grab Food receipt", () => {
|
||||
const r = parseGrabReceipt(grabFoodForwardedBody);
|
||||
assert.ok(r);
|
||||
assert.equal(r.kind, "grab.ereceipt.food");
|
||||
assert.equal(r.variant, "food");
|
||||
assert.equal(r.amount, 49000);
|
||||
assert.equal(r.reference, "00112712674-C8EUEZDGEEAZTX");
|
||||
assert.equal(r.occurredAt, "2026-09-11T18:24:00+07:00");
|
||||
assert.equal(r.details.merchant, "Doner Kebab - Kalibata City");
|
||||
assert.equal(r.details.vehicleType, "GrabFood");
|
||||
assert.equal(r.details.passenger, "Bintang");
|
||||
assert.equal(r.details.paymentMethod, "OVO Cash");
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseBriTransfer", () => {
|
||||
it("parses direct BRImo transfer body", () => {
|
||||
const r = parseBriTransfer(briTransferDirectBody);
|
||||
assert.ok(r);
|
||||
assert.equal(r.amount, 4000000);
|
||||
assert.equal(r.reference, "212189333864");
|
||||
assert.equal(r.occurredAt, "2026-09-13T19:00:00+07:00");
|
||||
assert.equal(r.details.payerName, "NASYAROBBY NUGRAHA PUTRA");
|
||||
assert.equal(r.details.namaTujuan, "NELLY ARISANTI");
|
||||
assert.equal(r.details.jenisTransaksi, "Transfer Bank BRI");
|
||||
assert.equal(r.details.bankTujuan, "BANK BRI");
|
||||
assert.equal(r.details.nomorTujuan, "1060 0100 2070 507");
|
||||
assert.equal(r.details.biayaAdmin, 0);
|
||||
assert.equal(
|
||||
r.details.fundingSource,
|
||||
"NASYAROBBY NUGRAHA PUTRA 0121 **** **** 500",
|
||||
);
|
||||
assert.equal(r.details.catatan, undefined);
|
||||
});
|
||||
|
||||
it("parses forwarded BRImo transfer body", () => {
|
||||
const r = parseBriTransfer(briTransferForwardedBody);
|
||||
assert.ok(r);
|
||||
assert.equal(r.amount, 4000000);
|
||||
assert.equal(r.reference, "212189333864");
|
||||
assert.equal(r.details.namaTujuan, "NELLY ARISANTI");
|
||||
});
|
||||
|
||||
it("returns null without amount/reference", () => {
|
||||
assert.equal(parseBriTransfer("Halo someone\nTransaksi Berhasil"), null);
|
||||
});
|
||||
});
|
||||
|
||||
describe("toSpendingRecord", () => {
|
||||
it("maps Livin envelope to spendings fields", () => {
|
||||
const parsed = {
|
||||
kind: "livin.qr_payment.success",
|
||||
messageId: "1a099bd61efd234d",
|
||||
amount: 14000,
|
||||
reference: "2609131121540314991",
|
||||
occurredAt: "2026-09-13T14:48:31+07:00",
|
||||
details: {
|
||||
merchant: "SNACK BAROKAHJAKARTA BARAT - ID",
|
||||
qrisRef: "609532729372",
|
||||
},
|
||||
};
|
||||
const rec = toSpendingRecord(parsed, livinQrPayment);
|
||||
assert.equal(rec.invoice_id, "2609131121540314991");
|
||||
assert.equal(rec.amount, 14000);
|
||||
assert.equal(rec.source, "livin");
|
||||
assert.equal(rec.description, "SNACK BAROKAHJAKARTA BARAT - ID");
|
||||
assert.equal(rec.category, "");
|
||||
assert.equal(rec.message_id, "1a099bd61efd234d");
|
||||
assert.equal(rec.trx_date, toPocketBaseDate(parsed.occurredAt));
|
||||
assert.deepEqual(rec.parsed, parsed);
|
||||
});
|
||||
|
||||
it("maps Grab envelope to spendings fields", () => {
|
||||
const parsed = {
|
||||
kind: "grab.ereceipt.ride",
|
||||
messageId: "1a099b8b9d239e5e",
|
||||
amount: 16600,
|
||||
reference: "A-9RXV4K2GWCLOAV",
|
||||
occurredAt: "2026-09-13",
|
||||
details: { product: "Bike Hemat", driver: "Mahendra" },
|
||||
};
|
||||
const rec = toSpendingRecord(parsed, grabEreceipt);
|
||||
assert.equal(rec.invoice_id, "A-9RXV4K2GWCLOAV");
|
||||
assert.equal(rec.source, "grab");
|
||||
assert.equal(rec.description, "Grab Bike Hemat");
|
||||
assert.equal(rec.trx_date, "2026-09-13 00:00:00.000Z");
|
||||
assert.deepEqual(rec.parsed, parsed);
|
||||
});
|
||||
|
||||
it("maps Grab tip envelope to spendings fields", () => {
|
||||
const parsed = {
|
||||
kind: "grab.ereceipt.tip",
|
||||
messageId: "1a099e731cb511dc",
|
||||
amount: 2000,
|
||||
reference: "A-9RXV4K2GWCLOAV:tip",
|
||||
occurredAt: "2026-09-13T10:47:00+07:00",
|
||||
details: { product: "Bike Hemat", variant: "tip" },
|
||||
};
|
||||
const rec = toSpendingRecord(parsed, grabEreceipt);
|
||||
assert.equal(rec.invoice_id, "A-9RXV4K2GWCLOAV:tip");
|
||||
assert.equal(rec.source, "grab");
|
||||
assert.equal(rec.description, "Grab Bike Hemat tip");
|
||||
assert.equal(rec.trx_date, toPocketBaseDate(parsed.occurredAt));
|
||||
});
|
||||
|
||||
it("maps Grab Food envelope to spendings fields", () => {
|
||||
const parsed = {
|
||||
kind: "grab.ereceipt.food",
|
||||
messageId: "1a09a07cbd6f58cb",
|
||||
amount: 49000,
|
||||
reference: "00112712674-C8EUEZDGEEAZTX",
|
||||
occurredAt: "2026-09-11T18:24:00+07:00",
|
||||
details: {
|
||||
variant: "food",
|
||||
product: "Food",
|
||||
merchant: "Doner Kebab - Kalibata City",
|
||||
},
|
||||
};
|
||||
const rec = toSpendingRecord(parsed, grabEreceipt);
|
||||
assert.equal(rec.invoice_id, "00112712674-C8EUEZDGEEAZTX");
|
||||
assert.equal(rec.source, "grab");
|
||||
assert.equal(rec.description, "Grab Food Doner Kebab - Kalibata City");
|
||||
});
|
||||
|
||||
it("maps BRI transfer envelope to spendings fields", () => {
|
||||
const parsed = {
|
||||
kind: "bri.transfer.success",
|
||||
messageId: "1a09b6cd9a3bc6bd",
|
||||
amount: 4000000,
|
||||
reference: "212189333864",
|
||||
occurredAt: "2026-09-13T19:00:00+07:00",
|
||||
details: {
|
||||
namaTujuan: "NELLY ARISANTI",
|
||||
jenisTransaksi: "Transfer Bank BRI",
|
||||
},
|
||||
};
|
||||
const rec = toSpendingRecord(parsed, briTransfer);
|
||||
assert.equal(rec.invoice_id, "212189333864");
|
||||
assert.equal(rec.amount, 4000000);
|
||||
assert.equal(rec.source, "bri");
|
||||
assert.equal(rec.description, "NELLY ARISANTI");
|
||||
assert.equal(rec.trx_date, toPocketBaseDate(parsed.occurredAt));
|
||||
assert.deepEqual(rec.parsed, parsed);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,12 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Gmail Reader</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
<script type="module" src="/src/main.jsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"jsx": "react-jsx",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "bundler"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"name": "web",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
"build": "vite build",
|
||||
"preview": "vite preview"
|
||||
},
|
||||
"dependencies": {
|
||||
"react": "^19.1.1",
|
||||
"react-dom": "^19.1.1",
|
||||
"react-router-dom": "^7.8.2"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@tailwindcss/vite": "^4.1.13",
|
||||
"@vitejs/plugin-react": "^5.0.2",
|
||||
"daisyui": "^5.1.13",
|
||||
"tailwindcss": "^4.1.13",
|
||||
"vite": "^7.1.5"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
import { Navigate, Outlet, Route, Routes } from "react-router-dom";
|
||||
import { useAuth } from "./auth.jsx";
|
||||
import Layout from "./components/Layout.jsx";
|
||||
import Inbox from "./pages/Inbox.jsx";
|
||||
import Login from "./pages/Login.jsx";
|
||||
import Message from "./pages/Message.jsx";
|
||||
|
||||
function Protected() {
|
||||
const { user, loading } = useAuth();
|
||||
if (loading) {
|
||||
return (
|
||||
<div className="grid min-h-screen place-items-center">
|
||||
<span className="loading loading-spinner loading-lg" />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
if (!user) return <Navigate to="/login" replace />;
|
||||
return (
|
||||
<Layout>
|
||||
<Outlet />
|
||||
</Layout>
|
||||
);
|
||||
}
|
||||
|
||||
function GuestOnly({ children }) {
|
||||
const { user, loading } = useAuth();
|
||||
if (loading) {
|
||||
return (
|
||||
<div className="grid min-h-screen place-items-center">
|
||||
<span className="loading loading-spinner loading-lg" />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
if (user) return <Navigate to="/" replace />;
|
||||
return children;
|
||||
}
|
||||
|
||||
export default function App() {
|
||||
return (
|
||||
<Routes>
|
||||
<Route
|
||||
path="/login"
|
||||
element={
|
||||
<GuestOnly>
|
||||
<Login />
|
||||
</GuestOnly>
|
||||
}
|
||||
/>
|
||||
<Route element={<Protected />}>
|
||||
<Route path="/" element={<Inbox />} />
|
||||
<Route path="/messages/:id" element={<Message />} />
|
||||
</Route>
|
||||
<Route path="*" element={<Navigate to="/" replace />} />
|
||||
</Routes>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
export class AuthError extends Error {
|
||||
constructor() {
|
||||
super("Not authenticated");
|
||||
this.name = "AuthError";
|
||||
}
|
||||
}
|
||||
|
||||
export async function api(path, options = {}) {
|
||||
const headers = { ...(options.headers || {}) };
|
||||
if (options.body && !headers["Content-Type"]) {
|
||||
headers["Content-Type"] = "application/json";
|
||||
}
|
||||
|
||||
const res = await fetch(path, {
|
||||
credentials: "include",
|
||||
...options,
|
||||
headers,
|
||||
});
|
||||
|
||||
if (res.status === 401) {
|
||||
throw new AuthError();
|
||||
}
|
||||
|
||||
const data = await res.json().catch(() => ({}));
|
||||
if (!res.ok) {
|
||||
throw new Error(data.error || res.statusText);
|
||||
}
|
||||
return data;
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
import { createContext, useContext, useEffect, useState } from "react";
|
||||
import { api, AuthError } from "./api.js";
|
||||
|
||||
const AuthContext = createContext(null);
|
||||
|
||||
export function AuthProvider({ children }) {
|
||||
const [user, setUser] = useState(null);
|
||||
const [loading, setLoading] = useState(true);
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
api("/api/me")
|
||||
.then((me) => {
|
||||
if (!cancelled) setUser(me);
|
||||
})
|
||||
.catch((err) => {
|
||||
if (!cancelled && !(err instanceof AuthError)) {
|
||||
console.error(err);
|
||||
}
|
||||
if (!cancelled) setUser(null);
|
||||
})
|
||||
.finally(() => {
|
||||
if (!cancelled) setLoading(false);
|
||||
});
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, []);
|
||||
|
||||
async function logout() {
|
||||
try {
|
||||
await api("/auth/logout", { method: "POST" });
|
||||
} finally {
|
||||
setUser(null);
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<AuthContext.Provider value={{ user, loading, logout }}>
|
||||
{children}
|
||||
</AuthContext.Provider>
|
||||
);
|
||||
}
|
||||
|
||||
export function useAuth() {
|
||||
const ctx = useContext(AuthContext);
|
||||
if (!ctx) throw new Error("useAuth must be used within AuthProvider");
|
||||
return ctx;
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
import { Link } from "react-router-dom";
|
||||
import { useAuth } from "../auth.jsx";
|
||||
|
||||
export default function Layout({ children }) {
|
||||
const { user, logout } = useAuth();
|
||||
|
||||
return (
|
||||
<div className="min-h-screen bg-base-200">
|
||||
<header className="navbar border-b border-base-300 bg-base-100 px-4">
|
||||
<div className="flex-1">
|
||||
<Link to="/" className="text-lg font-semibold tracking-tight">
|
||||
Gmail Reader
|
||||
</Link>
|
||||
</div>
|
||||
<div className="flex items-center gap-3">
|
||||
<span className="hidden text-sm opacity-70 sm:inline">
|
||||
{user?.email}
|
||||
</span>
|
||||
{user?.picture ? (
|
||||
<div className="avatar">
|
||||
<div className="w-8 rounded-full">
|
||||
<img src={user.picture} alt="" referrerPolicy="no-referrer" />
|
||||
</div>
|
||||
</div>
|
||||
) : null}
|
||||
<button type="button" className="btn btn-ghost btn-sm" onClick={logout}>
|
||||
Log out
|
||||
</button>
|
||||
</div>
|
||||
</header>
|
||||
<main className="mx-auto max-w-4xl px-4 py-6">{children}</main>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
@import "tailwindcss";
|
||||
@plugin "daisyui" {
|
||||
themes: corporate --default, business --prefersdark;
|
||||
}
|
||||
|
||||
html,
|
||||
body,
|
||||
#root {
|
||||
min-height: 100%;
|
||||
}
|
||||
|
||||
.message-body {
|
||||
white-space: pre-wrap;
|
||||
word-break: break-word;
|
||||
font-family: ui-sans-serif, system-ui, sans-serif;
|
||||
line-height: 1.55;
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
import { StrictMode } from "react";
|
||||
import { createRoot } from "react-dom/client";
|
||||
import { BrowserRouter } from "react-router-dom";
|
||||
import App from "./App.jsx";
|
||||
import { AuthProvider } from "./auth.jsx";
|
||||
import "./index.css";
|
||||
|
||||
createRoot(document.getElementById("root")).render(
|
||||
<StrictMode>
|
||||
<BrowserRouter>
|
||||
<AuthProvider>
|
||||
<App />
|
||||
</AuthProvider>
|
||||
</BrowserRouter>
|
||||
</StrictMode>,
|
||||
);
|
||||
@@ -0,0 +1,169 @@
|
||||
import { useEffect, useState } from "react";
|
||||
import { Link, useSearchParams } from "react-router-dom";
|
||||
import { api } from "../api.js";
|
||||
|
||||
function formatDate(value) {
|
||||
if (!value) return "";
|
||||
const parsed = new Date(value);
|
||||
if (Number.isNaN(parsed.getTime())) return value;
|
||||
return parsed.toLocaleString(undefined, {
|
||||
month: "short",
|
||||
day: "numeric",
|
||||
hour: "2-digit",
|
||||
minute: "2-digit",
|
||||
});
|
||||
}
|
||||
|
||||
export default function Inbox() {
|
||||
const [searchParams, setSearchParams] = useSearchParams();
|
||||
const q = searchParams.get("q") || "";
|
||||
const [draft, setDraft] = useState(q);
|
||||
const [messages, setMessages] = useState([]);
|
||||
const [nextPageToken, setNextPageToken] = useState(null);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [loadingMore, setLoadingMore] = useState(false);
|
||||
const [error, setError] = useState(null);
|
||||
|
||||
useEffect(() => {
|
||||
setDraft(q);
|
||||
}, [q]);
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
setLoading(true);
|
||||
setError(null);
|
||||
const params = new URLSearchParams();
|
||||
if (q) params.set("q", q);
|
||||
api(`/api/messages?${params}`)
|
||||
.then((data) => {
|
||||
if (cancelled) return;
|
||||
setMessages(data.messages);
|
||||
setNextPageToken(data.nextPageToken);
|
||||
})
|
||||
.catch((err) => {
|
||||
if (!cancelled) setError(err.message);
|
||||
})
|
||||
.finally(() => {
|
||||
if (!cancelled) setLoading(false);
|
||||
});
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, [q]);
|
||||
|
||||
function onSearch(event) {
|
||||
event.preventDefault();
|
||||
const next = draft.trim();
|
||||
if (next) setSearchParams({ q: next });
|
||||
else setSearchParams({});
|
||||
}
|
||||
|
||||
async function loadMore() {
|
||||
if (!nextPageToken) return;
|
||||
setLoadingMore(true);
|
||||
setError(null);
|
||||
try {
|
||||
const params = new URLSearchParams({ pageToken: nextPageToken });
|
||||
if (q) params.set("q", q);
|
||||
const data = await api(`/api/messages?${params}`);
|
||||
setMessages((prev) => [...prev, ...data.messages]);
|
||||
setNextPageToken(data.nextPageToken);
|
||||
} catch (err) {
|
||||
setError(err.message);
|
||||
} finally {
|
||||
setLoadingMore(false);
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="space-y-4">
|
||||
<form onSubmit={onSearch} className="join w-full">
|
||||
<input
|
||||
className="input join-item input-bordered w-full"
|
||||
value={draft}
|
||||
onChange={(e) => setDraft(e.target.value)}
|
||||
placeholder='Search Gmail, e.g. from:alice subject:invoice'
|
||||
aria-label="Search mail"
|
||||
/>
|
||||
<button type="submit" className="btn btn-primary join-item">
|
||||
Search
|
||||
</button>
|
||||
</form>
|
||||
|
||||
{q ? (
|
||||
<p className="text-sm opacity-70">
|
||||
Results for <span className="font-mono">{q}</span>{" "}
|
||||
<button
|
||||
type="button"
|
||||
className="link link-hover"
|
||||
onClick={() => setSearchParams({})}
|
||||
>
|
||||
clear
|
||||
</button>
|
||||
</p>
|
||||
) : (
|
||||
<h2 className="text-sm font-medium opacity-70">Latest mail</h2>
|
||||
)}
|
||||
|
||||
{error ? (
|
||||
<div role="alert" className="alert alert-error text-sm">
|
||||
{error}
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
{loading ? (
|
||||
<div className="flex justify-center py-16">
|
||||
<span className="loading loading-spinner loading-lg" />
|
||||
</div>
|
||||
) : messages.length === 0 ? (
|
||||
<div className="card bg-base-100">
|
||||
<div className="card-body items-center text-center opacity-70">
|
||||
<p>No messages found.</p>
|
||||
</div>
|
||||
</div>
|
||||
) : (
|
||||
<ul className="menu rounded-box bg-base-100 p-0 shadow">
|
||||
{messages.map((message) => (
|
||||
<li
|
||||
key={message.id}
|
||||
className="border-b border-base-200 last:border-b-0"
|
||||
>
|
||||
<Link
|
||||
to={`/messages/${message.id}`}
|
||||
className="flex flex-col items-stretch gap-1 py-3"
|
||||
>
|
||||
<div className="flex items-baseline justify-between gap-3">
|
||||
<span className="truncate font-medium">{message.from}</span>
|
||||
<time className="shrink-0 text-xs opacity-60">
|
||||
{formatDate(message.date)}
|
||||
</time>
|
||||
</div>
|
||||
<span className="truncate">{message.subject}</span>
|
||||
<span className="line-clamp-1 text-sm opacity-60">
|
||||
{message.snippet}
|
||||
</span>
|
||||
</Link>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
|
||||
{nextPageToken && !loading ? (
|
||||
<div className="flex justify-center">
|
||||
<button
|
||||
type="button"
|
||||
className="btn btn-outline"
|
||||
onClick={loadMore}
|
||||
disabled={loadingMore}
|
||||
>
|
||||
{loadingMore ? (
|
||||
<span className="loading loading-spinner" />
|
||||
) : (
|
||||
"Load more"
|
||||
)}
|
||||
</button>
|
||||
</div>
|
||||
) : null}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
import { useSearchParams } from "react-router-dom";
|
||||
|
||||
const ERRORS = {
|
||||
denied: "Google access was denied.",
|
||||
missing_code: "OAuth callback was missing a code.",
|
||||
invalid_state: "OAuth state did not match. Try signing in again.",
|
||||
exchange_failed: "Could not complete Google sign-in.",
|
||||
};
|
||||
|
||||
export default function Login() {
|
||||
const [params] = useSearchParams();
|
||||
const error = ERRORS[params.get("error")] || null;
|
||||
|
||||
return (
|
||||
<div className="grid min-h-screen place-items-center bg-base-200 px-4">
|
||||
<div className="card w-full max-w-md bg-base-100 shadow-xl">
|
||||
<div className="card-body gap-4">
|
||||
<h1 className="card-title text-2xl">Gmail Reader</h1>
|
||||
<p className="text-sm opacity-70">
|
||||
Sign in with Google to list, search, and read mail. Access is
|
||||
read-only.
|
||||
</p>
|
||||
{error ? (
|
||||
<div role="alert" className="alert alert-error text-sm">
|
||||
{error}
|
||||
</div>
|
||||
) : null}
|
||||
<a href="/auth/google" className="btn btn-primary">
|
||||
Sign in with Google
|
||||
</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
import { useEffect, useState } from "react";
|
||||
import { Link, useParams } from "react-router-dom";
|
||||
import { api } from "../api.js";
|
||||
|
||||
function formatDate(value) {
|
||||
if (!value) return "";
|
||||
const parsed = new Date(value);
|
||||
if (Number.isNaN(parsed.getTime())) return value;
|
||||
return parsed.toLocaleString();
|
||||
}
|
||||
|
||||
export default function Message() {
|
||||
const { id } = useParams();
|
||||
const [message, setMessage] = useState(null);
|
||||
const [error, setError] = useState(null);
|
||||
const [loading, setLoading] = useState(true);
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
setLoading(true);
|
||||
setError(null);
|
||||
api(`/api/messages/${encodeURIComponent(id)}`)
|
||||
.then((data) => {
|
||||
if (!cancelled) setMessage(data);
|
||||
})
|
||||
.catch((err) => {
|
||||
if (!cancelled) setError(err.message);
|
||||
})
|
||||
.finally(() => {
|
||||
if (!cancelled) setLoading(false);
|
||||
});
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, [id]);
|
||||
|
||||
return (
|
||||
<div className="space-y-4">
|
||||
<Link to="/" className="btn btn-ghost btn-sm">
|
||||
← Inbox
|
||||
</Link>
|
||||
|
||||
{loading ? (
|
||||
<div className="flex justify-center py-16">
|
||||
<span className="loading loading-spinner loading-lg" />
|
||||
</div>
|
||||
) : error ? (
|
||||
<div role="alert" className="alert alert-error text-sm">
|
||||
{error}
|
||||
</div>
|
||||
) : message ? (
|
||||
<article className="card bg-base-100 shadow">
|
||||
<div className="card-body gap-4">
|
||||
<h1 className="card-title text-xl">{message.subject}</h1>
|
||||
<dl className="grid gap-1 text-sm">
|
||||
<div className="flex gap-2">
|
||||
<dt className="opacity-60">From</dt>
|
||||
<dd>{message.from}</dd>
|
||||
</div>
|
||||
{message.to ? (
|
||||
<div className="flex gap-2">
|
||||
<dt className="opacity-60">To</dt>
|
||||
<dd>{message.to}</dd>
|
||||
</div>
|
||||
) : null}
|
||||
<div className="flex gap-2">
|
||||
<dt className="opacity-60">Date</dt>
|
||||
<dd>{formatDate(message.date)}</dd>
|
||||
</div>
|
||||
</dl>
|
||||
<div className="divider my-0" />
|
||||
<div className="message-body text-sm">
|
||||
{message.body || message.snippet || "(empty message)"}
|
||||
</div>
|
||||
</div>
|
||||
</article>
|
||||
) : null}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
import { defineConfig } from "vite";
|
||||
import react from "@vitejs/plugin-react";
|
||||
import tailwindcss from "@tailwindcss/vite";
|
||||
|
||||
export default defineConfig({
|
||||
plugins: [react(), tailwindcss()],
|
||||
server: {
|
||||
port: 5173,
|
||||
strictPort: true,
|
||||
proxy: {
|
||||
"/api": "http://localhost:3000",
|
||||
"/auth": "http://localhost:3000",
|
||||
"/callback": "http://localhost:3000",
|
||||
},
|
||||
},
|
||||
});
|
||||
Generated
+5537
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"name": "gmail-summarizer",
|
||||
"private": true,
|
||||
"workspaces": [
|
||||
"apps/*"
|
||||
],
|
||||
"scripts": {
|
||||
"dev": "concurrently -k -n api,web -c cyan,magenta \"npm run dev -w api\" \"npm run dev -w web\"",
|
||||
"dev:api": "npm run dev -w api",
|
||||
"dev:web": "npm run dev -w web",
|
||||
"dev:watcher": "npm run dev -w watcher",
|
||||
"test": "npm run test -w watcher",
|
||||
"test:watcher": "npm run test -w watcher",
|
||||
"process-messages": "npm run process-messages -w watcher --",
|
||||
"list-messages": "npm run list-messages -w watcher --",
|
||||
"build": "npm run build -w web",
|
||||
"start": "npm run start -w api",
|
||||
"start:watcher": "npm run start -w watcher"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
},
|
||||
"devDependencies": {
|
||||
"concurrently": "^9.2.1"
|
||||
}
|
||||
}
|
||||
Generated
+206
@@ -0,0 +1,206 @@
|
||||
lockfileVersion: '9.0'
|
||||
|
||||
settings:
|
||||
autoInstallPeers: true
|
||||
excludeLinksFromLockfile: false
|
||||
|
||||
importers:
|
||||
|
||||
.:
|
||||
devDependencies:
|
||||
concurrently:
|
||||
specifier: ^9.2.1
|
||||
version: 9.2.4
|
||||
|
||||
packages:
|
||||
|
||||
ansi-regex@5.0.1:
|
||||
resolution: {integrity: sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==}
|
||||
engines: {node: '>=8'}
|
||||
|
||||
ansi-styles@4.3.0:
|
||||
resolution: {integrity: sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==}
|
||||
engines: {node: '>=8'}
|
||||
|
||||
chalk@4.1.2:
|
||||
resolution: {integrity: sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==}
|
||||
engines: {node: '>=10'}
|
||||
|
||||
cliui@8.0.1:
|
||||
resolution: {integrity: sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ==}
|
||||
engines: {node: '>=12'}
|
||||
|
||||
color-convert@2.0.1:
|
||||
resolution: {integrity: sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==}
|
||||
engines: {node: '>=7.0.0'}
|
||||
|
||||
color-name@1.1.4:
|
||||
resolution: {integrity: sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==}
|
||||
|
||||
concurrently@9.2.4:
|
||||
resolution: {integrity: sha512-TZ0CEhyzvFjgtAvHTusDMgj7wNdihCh7LLLrzdUOXIhdlnL2JBBGA9eJxR24rtqgmdjh3OA3hrN1rCHj6HM8qA==}
|
||||
engines: {node: '>=18'}
|
||||
hasBin: true
|
||||
|
||||
emoji-regex@8.0.0:
|
||||
resolution: {integrity: sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==}
|
||||
|
||||
escalade@3.2.0:
|
||||
resolution: {integrity: sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==}
|
||||
engines: {node: '>=6'}
|
||||
|
||||
get-caller-file@2.0.5:
|
||||
resolution: {integrity: sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==}
|
||||
engines: {node: 6.* || 8.* || >= 10.*}
|
||||
|
||||
has-flag@4.0.0:
|
||||
resolution: {integrity: sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ==}
|
||||
engines: {node: '>=8'}
|
||||
|
||||
is-fullwidth-code-point@3.0.0:
|
||||
resolution: {integrity: sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==}
|
||||
engines: {node: '>=8'}
|
||||
|
||||
require-directory@2.1.1:
|
||||
resolution: {integrity: sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q==}
|
||||
engines: {node: '>=0.10.0'}
|
||||
|
||||
rxjs@7.8.2:
|
||||
resolution: {integrity: sha512-dhKf903U/PQZY6boNNtAGdWbG85WAbjT/1xYoZIC7FAY0yWapOBQVsVrDl58W86//e1VpMNBtRV4MaXfdMySFA==}
|
||||
|
||||
shell-quote@1.9.0:
|
||||
resolution: {integrity: sha512-Iov+JwFv/2HcTpcwNMKd8+IWNb8tboQJNQTkAY/LLVK7gGH9jy+LGkVqPxfekHl+yMmiqXszdGWXgkfml7hjqA==}
|
||||
engines: {node: '>= 0.4'}
|
||||
|
||||
string-width@4.2.3:
|
||||
resolution: {integrity: sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==}
|
||||
engines: {node: '>=8'}
|
||||
|
||||
strip-ansi@6.0.1:
|
||||
resolution: {integrity: sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==}
|
||||
engines: {node: '>=8'}
|
||||
|
||||
supports-color@7.2.0:
|
||||
resolution: {integrity: sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==}
|
||||
engines: {node: '>=8'}
|
||||
|
||||
supports-color@8.1.1:
|
||||
resolution: {integrity: sha512-MpUEN2OodtUzxvKQl72cUF7RQ5EiHsGvSsVG0ia9c5RbWGL2CI4C7EpPS8UTBIplnlzZiNuV56w+FuNxy3ty2Q==}
|
||||
engines: {node: '>=10'}
|
||||
|
||||
tree-kill@1.2.2:
|
||||
resolution: {integrity: sha512-L0Orpi8qGpRG//Nd+H90vFB+3iHnue1zSSGmNOOCh1GLJ7rUKVwV2HvijphGQS2UmhUZewS9VgvxYIdgr+fG1A==}
|
||||
hasBin: true
|
||||
|
||||
tslib@2.8.1:
|
||||
resolution: {integrity: sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==}
|
||||
|
||||
wrap-ansi@7.0.0:
|
||||
resolution: {integrity: sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==}
|
||||
engines: {node: '>=10'}
|
||||
|
||||
y18n@5.0.8:
|
||||
resolution: {integrity: sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA==}
|
||||
engines: {node: '>=10'}
|
||||
|
||||
yargs-parser@21.1.1:
|
||||
resolution: {integrity: sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==}
|
||||
engines: {node: '>=12'}
|
||||
|
||||
yargs@17.7.2:
|
||||
resolution: {integrity: sha512-7dSzzRQ++CKnNI/krKnYRV7JKKPUXMEh61soaHKg9mrWEhzFWhFnxPxGl+69cD1Ou63C13NUPCnmIcrvqCuM6w==}
|
||||
engines: {node: '>=12'}
|
||||
|
||||
snapshots:
|
||||
|
||||
ansi-regex@5.0.1: {}
|
||||
|
||||
ansi-styles@4.3.0:
|
||||
dependencies:
|
||||
color-convert: 2.0.1
|
||||
|
||||
chalk@4.1.2:
|
||||
dependencies:
|
||||
ansi-styles: 4.3.0
|
||||
supports-color: 7.2.0
|
||||
|
||||
cliui@8.0.1:
|
||||
dependencies:
|
||||
string-width: 4.2.3
|
||||
strip-ansi: 6.0.1
|
||||
wrap-ansi: 7.0.0
|
||||
|
||||
color-convert@2.0.1:
|
||||
dependencies:
|
||||
color-name: 1.1.4
|
||||
|
||||
color-name@1.1.4: {}
|
||||
|
||||
concurrently@9.2.4:
|
||||
dependencies:
|
||||
chalk: 4.1.2
|
||||
rxjs: 7.8.2
|
||||
shell-quote: 1.9.0
|
||||
supports-color: 8.1.1
|
||||
tree-kill: 1.2.2
|
||||
yargs: 17.7.2
|
||||
|
||||
emoji-regex@8.0.0: {}
|
||||
|
||||
escalade@3.2.0: {}
|
||||
|
||||
get-caller-file@2.0.5: {}
|
||||
|
||||
has-flag@4.0.0: {}
|
||||
|
||||
is-fullwidth-code-point@3.0.0: {}
|
||||
|
||||
require-directory@2.1.1: {}
|
||||
|
||||
rxjs@7.8.2:
|
||||
dependencies:
|
||||
tslib: 2.8.1
|
||||
|
||||
shell-quote@1.9.0: {}
|
||||
|
||||
string-width@4.2.3:
|
||||
dependencies:
|
||||
emoji-regex: 8.0.0
|
||||
is-fullwidth-code-point: 3.0.0
|
||||
strip-ansi: 6.0.1
|
||||
|
||||
strip-ansi@6.0.1:
|
||||
dependencies:
|
||||
ansi-regex: 5.0.1
|
||||
|
||||
supports-color@7.2.0:
|
||||
dependencies:
|
||||
has-flag: 4.0.0
|
||||
|
||||
supports-color@8.1.1:
|
||||
dependencies:
|
||||
has-flag: 4.0.0
|
||||
|
||||
tree-kill@1.2.2: {}
|
||||
|
||||
tslib@2.8.1: {}
|
||||
|
||||
wrap-ansi@7.0.0:
|
||||
dependencies:
|
||||
ansi-styles: 4.3.0
|
||||
string-width: 4.2.3
|
||||
strip-ansi: 6.0.1
|
||||
|
||||
y18n@5.0.8: {}
|
||||
|
||||
yargs-parser@21.1.1: {}
|
||||
|
||||
yargs@17.7.2:
|
||||
dependencies:
|
||||
cliui: 8.0.1
|
||||
escalade: 3.2.0
|
||||
get-caller-file: 2.0.5
|
||||
require-directory: 2.1.1
|
||||
string-width: 4.2.3
|
||||
y18n: 5.0.8
|
||||
yargs-parser: 21.1.1
|
||||
Reference in New Issue
Block a user