From cd4ab3d58258375a6347e6a650ae34e5f1c08602 Mon Sep 17 00:00:00 2001
From: Codinget <codinget@codi.moe>
Date: Wed, 1 Jul 2026 23:19:17 +0000
Subject: [PATCH] Add README

Covers project layout, setup, import tooling, and a summary of the bulk
import performance work, with an upfront disclaimer that this was built
with Claude Code (Sonnet 5).
---
 README.md | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
 create mode 100644 README.md

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..167400b
--- /dev/null
+++ b/README.md
@@ -0,0 +1,57 @@
+# pwned
+
+A small clone of the [HaveIBeenPwned Pwned Passwords API](https://haveibeenpwned.com/API/v3#PwnedPasswords), built for local development and experimentation. It implements the same k-anonymity range API (`GET /range/{5-char-hash-prefix}`), a thin React client to test it against, and tooling to populate the database from a wordlist, a hash list, or the real HIBP range API.
+
+> **Note:** This project was built with [Claude Code](https://claude.com/claude-code) (Sonnet 5), from initial scaffolding through the performance work on the bulk importer. It has not been used in production and has not had a from-scratch human security review; treat it as a learning/testing project rather than something to deploy as-is.
+
+## Layout
+
+This is an npm workspaces monorepo:
+
+- **`packages/backend`** — Koa + `@koa/router` + `pg`. Serves `GET /range/:prefix`, matching the real HIBP API's response shape.
+- **`packages/frontend`** — React + [wouter](https://github.com/molefrog/wouter), bundled with webpack. A thin client that hashes a password locally (SHA-1) and only sends the hash's first 5 characters to the server, same as the real HIBP API's privacy model.
+- **`packages/import`** — CLI tooling to populate `pwned_passwords`:
+  - `index.ts` — incremental import of a wordlist or hash list, with optional worker-thread/concurrency parallelism.
+  - `bulkImport.ts` — a much faster full-table rebuild via `COPY` + deferred index build, for large files (see "Import performance" below).
+  - `importHibp.ts` — pulls the full dataset from the real HIBP range API.
+
+## Getting started
+
+```sh
+npm install
+cp .env.example .env.local   # edit if needed
+npm run db:up                # starts a local Postgres via docker compose
+npm run -w backend migrate   # creates the pwned_passwords table
+npm run db:seed              # optional: a handful of well-known weak passwords
+npm run dev                  # starts backend (:3001) + frontend (:3000)
+```
+
+Point at an externally hosted Postgres instead of the local compose one by setting `DATABASE_URL` in `.env.local` (takes precedence over the discrete `PG*` vars).
+
+## Importing data
+
+```sh
+# incremental, from a plaintext wordlist or a HASH[:COUNT] file
+npm run import -- <file> --format=wordlist --jobs=8
+npm run import -- <file> --format=hashes
+
+# full rebuild, much faster for large files (see below)
+npm run import:bulk -- <file> --format=wordlist --jobs=8
+
+# pull the real dataset from api.pwnedpasswords.com
+npm run import:hibp
+```
+
+### Import performance
+
+The incremental importer (`index.ts`) upserts in batches and is fine for smaller files or topping up an existing table. For large files (e.g. rockyou, ~14.3M lines), it hits real Postgres limits: random-order B-tree maintenance on a table this size becomes commit- and I/O-bound rather than CPU-bound.
+
+`bulkImport.ts` avoids this by loading through `COPY` into an unindexed shadow table, merging it with the existing data in one pass, and only building the primary key/index once at the end (a sorted bulk build, instead of ~14M random-order incremental ones). The whole run is a single transaction, so Postgres can skip WAL-logging the new table's data entirely, and a failure at any point rolls back cleanly instead of leaving a half-migrated table.
+
+Measured on the same file/database across the course of this project's development: **42 minutes → 24.7 minutes → ~7.3 minutes → 186 seconds** wall clock, as (respectively) batching increased and commits became async, the whole operation became one WAL-skipping transaction, and wordlist hashing moved to a worker-thread pool.
+
+## Development
+
+- `npm run dev` — backend + frontend with live reload
+- `npm run build` — production builds
+- `npm run db:up` / `npm run db:down` — local Postgres via docker compose (dev only)