Add COPY-based bulk importer to bypass incremental B-tree update cost
The incremental importer was measured at 24.72 min wall / 186s user time for rockyou (14.3M rows), and Postgres logs showed checkpoints firing every ~22s due to WAL pressure from random-order B-tree maintenance and full-page-write amplification. bulkImport.ts (`npm run import:bulk`) avoids all of that: - Loads raw rows via COPY into a TEMP staging table (no WAL, no index). - Merges with the existing table's contents into a fresh, unindexed shadow table via one INSERT ... SELECT ... GROUP BY. - Adds the primary key and prefix index only after the table is populated, so Postgres builds them via a single sorted bulk pass instead of 14M random-order incremental inserts. - Swaps the shadow table into place at the end. - The entire run is one transaction: since pwned_passwords_new doesn't exist outside that transaction, Postgres skips WAL-logging its data entirely, and any failure rolls back to the exact starting state instead of leaving a half-migrated table. - Wordlist hashing runs across a worker_threads pool (--jobs, shared resolveWorkerUrl helper with the incremental importer's hashWorker) instead of blocking the single thread that also drives the COPY stream. - Each phase (hash+copy, merge, reindex, swap) reports its own wall time. Net effect measured against the same rockyou file/DB: 42 min -> 24.72 min (fsync fix) -> ~7.3 min (WAL-skip transaction) -> 186.17s (parallel hashing).
This commit is contained in:
@@ -7,14 +7,17 @@
|
||||
"scripts": {
|
||||
"start": "tsx --env-file=../../.env.local src/index.ts",
|
||||
"import-hibp": "tsx --env-file=../../.env.local src/importHibp.ts",
|
||||
"bulk-import": "tsx --env-file=../../.env.local src/bulkImport.ts",
|
||||
"build": "tsc -p tsconfig.json",
|
||||
"test": "echo \"Error: no test specified\" && exit 1"
|
||||
},
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"pg": "^8.22.0"
|
||||
"pg": "^8.22.0",
|
||||
"pg-copy-streams": "^7.0.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/pg": "^8.20.0"
|
||||
"@types/pg": "^8.20.0",
|
||||
"@types/pg-copy-streams": "^1.2.5"
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user