The incremental importer was measured at 24.72 min wall / 186s user time for
rockyou (14.3M rows), and Postgres logs showed checkpoints firing every ~22s
due to WAL pressure from random-order B-tree maintenance and full-page-write
amplification. bulkImport.ts (`npm run import:bulk`) avoids all of that:
- Loads raw rows via COPY into a TEMP staging table (no WAL, no index).
- Merges with the existing table's contents into a fresh, unindexed shadow
table via one INSERT ... SELECT ... GROUP BY.
- Adds the primary key and prefix index only after the table is populated,
so Postgres builds them via a single sorted bulk pass instead of 14M
random-order incremental inserts.
- Swaps the shadow table into place at the end.
- The entire run is one transaction: since pwned_passwords_new doesn't exist
outside that transaction, Postgres skips WAL-logging its data entirely,
and any failure rolls back to the exact starting state instead of leaving
a half-migrated table.
- Wordlist hashing runs across a worker_threads pool (--jobs, shared
resolveWorkerUrl helper with the incremental importer's hashWorker) instead
of blocking the single thread that also drives the COPY stream.
- Each phase (hash+copy, merge, reindex, swap) reports its own wall time.
Net effect measured against the same rockyou file/DB: 42 min -> 24.72 min
(fsync fix) -> ~7.3 min (WAL-skip transaction) -> 186.17s (parallel hashing).
- Backend: Koa + @koa/router serving the HIBP-style /range/:prefix k-anonymity
API, backed by Postgres via pg, with migrate/seed scripts run through tsx.
- Frontend: React + wouter, bundled with webpack (dev server + prod build),
thin client that hashes passwords locally and only sends the hash prefix.
- Import package: CLI to bulk-load a wordlist or hash list into the database,
plus a separate script to pull the full dataset from the HIBP range API.
Both support parallelism (worker threads for wordlist hashing, concurrency
limiter for DB/HTTP fan-out).
- Dev tooling: docker compose for a local Postgres, .env.local/.env.example,
DATABASE_URL support as an alternative to discrete PG* vars.
codinget