codinget
26c5b4eb9a
The incremental importer was measured at 24.72 min wall / 186s user time for rockyou (14.3M rows), and Postgres logs showed checkpoints firing every ~22s due to WAL pressure from random-order B-tree maintenance and full-page-write amplification. bulkImport.ts (`npm run import:bulk`) avoids all of that: - Loads raw rows via COPY into a TEMP staging table (no WAL, no index). - Merges with the existing table's contents into a fresh, unindexed shadow table via one INSERT ... SELECT ... GROUP BY. - Adds the primary key and prefix index only after the table is populated, so Postgres builds them via a single sorted bulk pass instead of 14M random-order incremental inserts. - Swaps the shadow table into place at the end. - The entire run is one transaction: since pwned_passwords_new doesn't exist outside that transaction, Postgres skips WAL-logging its data entirely, and any failure rolls back to the exact starting state instead of leaving a half-migrated table. - Wordlist hashing runs across a worker_threads pool (--jobs, shared resolveWorkerUrl helper with the incremental importer's hashWorker) instead of blocking the single thread that also drives the COPY stream. - Each phase (hash+copy, merge, reindex, swap) reports its own wall time. Net effect measured against the same rockyou file/DB: 42 min -> 24.72 min (fsync fix) -> ~7.3 min (WAL-skip transaction) -> 186.17s (parallel hashing).
25 lines
809 B
JSON
25 lines
809 B
JSON
{
|
|
"name": "pwned",
|
|
"version": "0.1.0",
|
|
"description": "A simple validator that checks if a password is known to be pwned",
|
|
"private": true,
|
|
"workspaces": [
|
|
"packages/*"
|
|
],
|
|
"scripts": {
|
|
"dev": "concurrently -n backend,frontend -c blue,green \"npm:dev -w backend\" \"npm:dev -w frontend\"",
|
|
"build": "npm run build -w backend && npm run build -w frontend",
|
|
"db:up": "docker compose --env-file .env.local up -d",
|
|
"db:down": "docker compose --env-file .env.local down",
|
|
"db:seed": "npm run seed -w backend",
|
|
"import": "npm run start -w import --",
|
|
"import:hibp": "npm run import-hibp -w import --",
|
|
"import:bulk": "npm run bulk-import -w import --"
|
|
},
|
|
"devDependencies": {
|
|
"concurrently": "^10.0.3",
|
|
"tsx": "^4.22.4",
|
|
"typescript": "^6.0.3"
|
|
}
|
|
}
|