add export/import streaming to the pluggable backends #13

Merged

codinget merged 4 commits from feat/export-import-streaming into master 2026-07-23 20:43:15 +02:00

Conversation 0 Commits 4 Files Changed 17

+2251 -8

4 Commits

Author	SHA1	Message	Date
codingetandClaude	d7e31dfce9	docs: codify the FK-safe stream ordering as a wire-format invariant ... The postgres importer inserts records sequentially with FK enforcement live, so it depends on records arriving in dependency order. That requirement was implicit in each backend's export table list; make it explicit and enforced. - Add EXPORT_KIND_ORDER (user, abode, resident, apikey, note) as a documented single source of truth, with the FK dependency chain spelled out on its doc comment, and note the ordering guarantee on the ExportEnvelope wire-format doc. Derive the isExportKind set from it. - Both backends' export() now iterate EXPORT_KIND_ORDER via a loader map (postgres omits note by leaving it out of the map), so emission order is tied to the constant and can't drift. - Tests: a change-detector on EXPORT_KIND_ORDER, plus assertions that both the sqlite and postgres (fake) exports emit record kinds grouped in FK-safe order (kind rank non-decreasing down the stream, after the leading meta). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-23 00:14:15 +00:00
codingetandClaude	3a56dcd9e5	feat: add export/import to the postgres backend; import auto-detects backend ... The export/import plan predated the postgres backend. Bring it up to parity: - PostgresInterface implements Exportable + Importable, mirroring the sqlite backend. Export is a signal-checked async generator (one query per table); import drives a transaction via `WrappedPool.multi`, which rolls back on any error/abort and commits only after the whole stream is consumed cleanly. - The `note` kind is skipped on postgres (its note CRUD is still unimplemented, so a pg database holds none) — a full dump from sqlite imports its user/abode/resident/apikey records and drops notes. - Unlike sqlite (PRAGMA foreign_keys=off), postgres keeps FK enforcement; the FK-safe insertion order keeps a full dump valid, and truly-dangling partial dumps will (correctly) fail. - abode-import now resolves the backend via getDbInterface instead of constructing SqliteInterface directly; isImportable keeps it from ever running over the remote (api) interface, which has no import. - Add isImportable(); make postgres selectClientApikeys' where optional. - Tests: exercise the real PostgresInterface export/import paths against an in-memory fake WrappedPgClient (no pg service in CI) — NDJSON shape, meta.source, filtering, note-skip, insert dispatch, and error/abort rollback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-22 23:36:21 +00:00
codingetandClaude	aadc950e24	fix: scope apikey export to the caller for non-admins ... A non-admin's forced export filter includes co-resident *user* records so abode/resident data isn't left with dangling references, but the same `users` allowlist was also governing `apikey` records — leaking co-residents' apikey metadata (name/permissions/expiry, though not the secret token). Add a dedicated `apikeys` uid-allowlist to ExportFilter that scopes apikey records specifically, falling back to `users` when absent (so existing unfiltered/voluntary-narrowing behaviour and the round-trip are unchanged). computeForcedExportFilter now sets it to the caller alone (intersected with an apikey credential's restrict_users), so a non-admin can only ever export their own keys. Global admins (forced filter null) are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-22 23:16:31 +00:00
codingetandClaude	ccb970f200	feat: add export/import streaming to the pluggable backends ... Add backend-agnostic data export/import over an NDJSON wire format, plus an inspect utility, exposed via three new CLIs and an HTTP export endpoint. - ExportImport types + filter helpers (kind/record scoping, hard-intersection of filters) in src/db/{types/ExportImport,export/filter}.ts - SqliteInterface implements Exportable + Importable: signal-checked async generator export (one query per table, per-record yield, trailing error sentinel on mid-stream failure) and a manually-driven import transaction that rolls back on any error/abort and never commits partial data - ApiInterface implements Exportable via its own fetch({signal}) - computeForcedExportFilter enforces non-global-admin scope (resided-in abodes + co-resident users, intersected with apikey restrict_*); GET /export intersects it with the caller's filter and wires an AbortController to the response socket - inspectExportStream reports kinds/counts from any stream without a db - abode-export / abode-import / abode-inspect CLIs (import is sqlite-only) - Secrets are not exported: imported users default to '#unset' passwords and apikeys are re-minted a token (ClientApikey view round-trips exactly) - test/tools/export-import.test.ts: round-trip, filter narrowing, forced-scope, export/import cancellation, in-process Koa endpoint, inspect Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-22 22:29:03 +00:00