rockyou import was ~195s of actual CPU work spread across 42 minutes wall
clock: Postgres was fsync-bound, not CPU/parallelism-bound (each batch was
its own implicit-commit transaction). Wrapping each batch in an explicit
transaction with synchronous_commit=off, and raising the default batch size
20000 (also used for wordlist worker chunking, previously hardcoded to 2000),
brought the same import down to ~25 minutes wall clock / ~186s user time.
Next bottleneck to address: B-tree index maintenance cost as it outgrows
shared_buffers (confirmed via pg_stat_user_tables: 14.3M live tuples, zero
dead tuples on a fresh DB, so it's not autovacuum/bloat). Random-order
inserts into the (prefix, suffix) index cause disk-bound page faults once the
index no longer fits in cache, which matched the observed end-of-run
slowdown and the "N-at-a-time" stalls lining up with --jobs concurrency.
- Backend: Koa + @koa/router serving the HIBP-style /range/:prefix k-anonymity
API, backed by Postgres via pg, with migrate/seed scripts run through tsx.
- Frontend: React + wouter, bundled with webpack (dev server + prod build),
thin client that hashes passwords locally and only sends the hash prefix.
- Import package: CLI to bulk-load a wordlist or hash list into the database,
plus a separate script to pull the full dataset from the HIBP range API.
Both support parallelism (worker threads for wordlist hashing, concurrency
limiter for DB/HTTP fan-out).
- Dev tooling: docker compose for a local Postgres, .env.local/.env.example,
DATABASE_URL support as an alternative to discrete PG* vars.
codinget