# pwned

A small clone of the [HaveIBeenPwned Pwned Passwords API](https://haveibeenpwned.com/API/v3#PwnedPasswords), built for local development and experimentation. It implements the same k-anonymity range API (`GET /range/{5-char-hash-prefix}`), a thin React client to test it against, and tooling to populate the database from a wordlist, a hash list, or the real HIBP range API.

> **Note:** This project was built with [Claude Code](https://claude.com/claude-code) (Sonnet 5), from initial scaffolding through the performance work on the bulk importer. It has not been used in production and has not had a from-scratch human security review; treat it as a learning/testing project rather than something to deploy as-is.

## Layout

This is an npm workspaces monorepo:

- **`packages/backend`** — Koa + `@koa/router` + `pg`. Serves `GET /range/:prefix`, matching the real HIBP API's response shape.
- **`packages/frontend`** — React + [wouter](https://github.com/molefrog/wouter), bundled with webpack. A thin client that hashes a password locally (SHA-1) and only sends the hash's first 5 characters to the server, same as the real HIBP API's privacy model.
- **`packages/import`** — CLI tooling to populate `pwned_passwords`:
  - `index.ts` — incremental import of a wordlist or hash list, with optional worker-thread/concurrency parallelism.
  - `bulkImport.ts` — a much faster full-table rebuild via `COPY` + deferred index build, for large files (see "Import performance" below).
  - `importHibp.ts` — pulls the full dataset from the real HIBP range API.

## Getting started

```sh
npm install
cp .env.example .env.local   # edit if needed
npm run db:up                # starts a local Postgres via docker compose
npm run -w backend migrate   # creates the pwned_passwords table
npm run db:seed              # optional: a handful of well-known weak passwords
npm run dev                  # starts backend (:3001) + frontend (:3000)
```

Point at an externally hosted Postgres instead of the local compose one by setting `DATABASE_URL` in `.env.local` (takes precedence over the discrete `PG*` vars).

## Importing data

```sh
# incremental, from a plaintext wordlist or a HASH[:COUNT] file
npm run import -- <file> --format=wordlist --jobs=8
npm run import -- <file> --format=hashes

# full rebuild, much faster for large files (see below)
npm run import:bulk -- <file> --format=wordlist --jobs=8

# pull the real dataset from api.pwnedpasswords.com
npm run import:hibp
```

### Import performance

The incremental importer (`index.ts`) upserts in batches and is fine for smaller files or topping up an existing table. For large files (e.g. rockyou, ~14.3M lines), it hits real Postgres limits: random-order B-tree maintenance on a table this size becomes commit- and I/O-bound rather than CPU-bound.

`bulkImport.ts` avoids this by loading through `COPY` into an unindexed shadow table, merging it with the existing data in one pass, and only building the primary key/index once at the end (a sorted bulk build, instead of ~14M random-order incremental ones). The whole run is a single transaction, so Postgres can skip WAL-logging the new table's data entirely, and a failure at any point rolls back cleanly instead of leaving a half-migrated table.

Measured on the same file/database across the course of this project's development: **42 minutes → 24.7 minutes → ~7.3 minutes → 186 seconds** wall clock, as (respectively) batching increased and commits became async, the whole operation became one WAL-skipping transaction, and wordlist hashing moved to a worker-thread pool.

## Development

- `npm run dev` — backend + frontend with live reload
- `npm run build` — production builds
- `npm run db:up` / `npm run db:down` — local Postgres via docker compose (dev only)