Add COPY-based bulk importer to bypass incremental B-tree update cost

The incremental importer was measured at 24.72 min wall / 186s user time for
rockyou (14.3M rows), and Postgres logs showed checkpoints firing every ~22s
due to WAL pressure from random-order B-tree maintenance and full-page-write
amplification. bulkImport.ts (`npm run import:bulk`) avoids all of that:

- Loads raw rows via COPY into a TEMP staging table (no WAL, no index).
- Merges with the existing table's contents into a fresh, unindexed shadow
  table via one INSERT ... SELECT ... GROUP BY.
- Adds the primary key and prefix index only after the table is populated,
  so Postgres builds them via a single sorted bulk pass instead of 14M
  random-order incremental inserts.
- Swaps the shadow table into place at the end.
- The entire run is one transaction: since pwned_passwords_new doesn't exist
  outside that transaction, Postgres skips WAL-logging its data entirely,
  and any failure rolls back to the exact starting state instead of leaving
  a half-migrated table.
- Wordlist hashing runs across a worker_threads pool (--jobs, shared
  resolveWorkerUrl helper with the incremental importer's hashWorker) instead
  of blocking the single thread that also drives the COPY stream.
- Each phase (hash+copy, merge, reindex, swap) reports its own wall time.

Net effect measured against the same rockyou file/DB: 42 min -> 24.72 min
(fsync fix) -> ~7.3 min (WAL-skip transaction) -> 186.17s (parallel hashing).
This commit is contained in:
2026-07-01 23:11:38 +00:00
parent 23a59c216d
commit 26c5b4eb9a
6 changed files with 264 additions and 7 deletions
+21 -2
View File
@@ -1389,6 +1389,17 @@
"pg-types": "^2.2.0"
}
},
"node_modules/@types/pg-copy-streams": {
"version": "1.2.5",
"resolved": "https://registry.npmjs.org/@types/pg-copy-streams/-/pg-copy-streams-1.2.5.tgz",
"integrity": "sha512-7D6/GYW2uHIaVU6S/5omI+6RZnwlZBpLQDZAH83xX1rjxAOK0f6/deKyyUTewxqts145VIGn6XWYz1YGf50G5g==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/node": "*",
"@types/pg": "*"
}
},
"node_modules/@types/qs": {
"version": "6.15.1",
"resolved": "https://registry.npmjs.org/@types/qs/-/qs-6.15.1.tgz",
@@ -4566,6 +4577,12 @@
"integrity": "sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==",
"license": "MIT"
},
"node_modules/pg-copy-streams": {
"version": "7.0.0",
"resolved": "https://registry.npmjs.org/pg-copy-streams/-/pg-copy-streams-7.0.0.tgz",
"integrity": "sha512-zBvnY6wtaBRE2ae2xXWOOGMaNVPkXh1vhypAkNSKgMdciJeTyIQAHZaEeRAxUjs/p1El5jgzYmwG5u871Zj3dQ==",
"license": "MIT"
},
"node_modules/pg-int8": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",
@@ -6410,10 +6427,12 @@
"version": "1.0.0",
"license": "ISC",
"dependencies": {
"pg": "^8.22.0"
"pg": "^8.22.0",
"pg-copy-streams": "^7.0.0"
},
"devDependencies": {
"@types/pg": "^8.20.0"
"@types/pg": "^8.20.0",
"@types/pg-copy-streams": "^1.2.5"
}
}
}