Adds backend-agnostic export / import streaming to the pluggable-backend abstraction, plus an inspect utility, three CLIs, and an HTTP export endpoint. Enables migrating data off a shared instance, moving between backend engines, and plain-file backups.
Wire format: NDJSON, one {"kind","data"} envelope per line; leading meta line records format version, timestamp, source backend, and the effective (possibly narrowed) filter.
What is in here
ExportImport types + filter helpers (kind/record scoping, hard-intersection of filters).
SqliteInterface — Exportable + Importable: signal-checked async-generator export (one query per table, per-record yield, trailing error sentinel on mid-stream failure); manually-driven import transaction that rolls back on any error/abort and never commits partial data.
ApiInterface — Exportable via its own fetch({signal}) (import stays sqlite-only).
exportScope — computeForcedExportFilter forces non-global-admin callers to their resided-in abodes + co-resident users (∩ apikey restrict_*); GET /export intersects with the caller filter and wires an AbortController to the response socket.
inspectExportStream — kinds/counts from any stream, no db.
CLIs: abode-export, abode-import, abode-inspect.
Notes / decisions
Secrets are never exported: imported users default to #unset passwords; apikeys are re-minted a token (the ClientApikey view round-trips exactly).
note kind is fully supported (Notes CRUD already landed), not a no-op.
Tests live in the existing test/ suite (test/tools/export-import.test.ts), not a fresh setup.
FK enforcement is suppressed for the duration of an import so filtered/partial dumps with out-of-scope created_by/updated_by load cleanly.
Full CLI round-trip (export → inspect → import) verified manually.
Pre-existing unrelated failures: the better-sqlite3 native binding is ABI-mismatched in this env (~8 cancelledByParent in the bs3 wrapped-db suite); node:sqlite path is unaffected.
TODO before merge
Human review of the authorization/scoping logic in exportScope.ts
Review the HTTP client-abort test for flakiness on CI
## Summary
Adds backend-agnostic **export / import streaming** to the pluggable-backend abstraction, plus an inspect utility, three CLIs, and an HTTP export endpoint. Enables migrating data off a shared instance, moving between backend engines, and plain-file backups.
**Wire format:** NDJSON, one `{"kind","data"}` envelope per line; leading `meta` line records format version, timestamp, source backend, and the *effective* (possibly narrowed) filter.
## What is in here
- `ExportImport` types + filter helpers (kind/record scoping, hard-intersection of filters).
- **SqliteInterface** — `Exportable` + `Importable`: signal-checked async-generator export (one query per table, per-record yield, trailing `error` sentinel on mid-stream failure); manually-driven import transaction that rolls back on any error/abort and never commits partial data.
- **ApiInterface** — `Exportable` via its own `fetch({signal})` (import stays sqlite-only).
- **exportScope** — `computeForcedExportFilter` forces non-global-admin callers to their resided-in abodes + co-resident users (∩ apikey `restrict_*`); `GET /export` intersects with the caller filter and wires an `AbortController` to the response socket.
- **inspectExportStream** — kinds/counts from any stream, no db.
- CLIs: `abode-export`, `abode-import`, `abode-inspect`.
## Notes / decisions
- Secrets are never exported: imported users default to `#unset` passwords; apikeys are re-minted a token (the `ClientApikey` view round-trips exactly).
- `note` kind is fully supported (Notes CRUD already landed), not a no-op.
- Tests live in the existing `test/` suite (`test/tools/export-import.test.ts`), not a fresh setup.
- FK enforcement is suppressed for the duration of an import so filtered/partial dumps with out-of-scope `created_by`/`updated_by` load cleanly.
## Testing
- `npm run typecheck` + `typecheck:test` clean.
- 18 new tests pass: round-trip, filter narrowing, forced-scope enforcement, export- and import-side cancellation, in-process Koa endpoint (admin / non-admin / client-abort), inspect.
- Full CLI round-trip (export → inspect → import) verified manually.
- Pre-existing unrelated failures: the better-sqlite3 native binding is ABI-mismatched in this env (~8 `cancelledByParent` in the bs3 wrapped-db suite); node:sqlite path is unaffected.
## TODO before merge
- [ ] Human review of the authorization/scoping logic in `exportScope.ts`
- [ ] Review the HTTP client-abort test for flakiness on CI
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Add backend-agnostic data export/import over an NDJSON wire format, plus an
inspect utility, exposed via three new CLIs and an HTTP export endpoint.
- ExportImport types + filter helpers (kind/record scoping, hard-intersection
of filters) in src/db/{types/ExportImport,export/filter}.ts
- SqliteInterface implements Exportable + Importable: signal-checked async
generator export (one query per table, per-record yield, trailing error
sentinel on mid-stream failure) and a manually-driven import transaction
that rolls back on any error/abort and never commits partial data
- ApiInterface implements Exportable via its own fetch({signal})
- computeForcedExportFilter enforces non-global-admin scope (resided-in abodes
+ co-resident users, intersected with apikey restrict_*); GET /export
intersects it with the caller's filter and wires an AbortController to the
response socket
- inspectExportStream reports kinds/counts from any stream without a db
- abode-export / abode-import / abode-inspect CLIs (import is sqlite-only)
- Secrets are not exported: imported users default to '#unset' passwords and
apikeys are re-minted a token (ClientApikey view round-trips exactly)
- test/tools/export-import.test.ts: round-trip, filter narrowing, forced-scope,
export/import cancellation, in-process Koa endpoint, inspect
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A non-admin's forced export filter includes co-resident *user* records so
abode/resident data isn't left with dangling references, but the same `users`
allowlist was also governing `apikey` records — leaking co-residents' apikey
metadata (name/permissions/expiry, though not the secret token).
Add a dedicated `apikeys` uid-allowlist to ExportFilter that scopes apikey
records specifically, falling back to `users` when absent (so existing
unfiltered/voluntary-narrowing behaviour and the round-trip are unchanged).
computeForcedExportFilter now sets it to the caller alone (intersected with an
apikey credential's restrict_users), so a non-admin can only ever export their
own keys. Global admins (forced filter null) are unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The export/import plan predated the postgres backend. Bring it up to parity:
- PostgresInterface implements Exportable + Importable, mirroring the sqlite
backend. Export is a signal-checked async generator (one query per table);
import drives a transaction via `WrappedPool.multi`, which rolls back on any
error/abort and commits only after the whole stream is consumed cleanly.
- The `note` kind is skipped on postgres (its note CRUD is still unimplemented,
so a pg database holds none) — a full dump from sqlite imports its
user/abode/resident/apikey records and drops notes.
- Unlike sqlite (PRAGMA foreign_keys=off), postgres keeps FK enforcement; the
FK-safe insertion order keeps a full dump valid, and truly-dangling partial
dumps will (correctly) fail.
- abode-import now resolves the backend via getDbInterface instead of
constructing SqliteInterface directly; isImportable keeps it from ever
running over the remote (api) interface, which has no import.
- Add isImportable(); make postgres selectClientApikeys' where optional.
- Tests: exercise the real PostgresInterface export/import paths against an
in-memory fake WrappedPgClient (no pg service in CI) — NDJSON shape,
meta.source, filtering, note-skip, insert dispatch, and error/abort rollback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The postgres importer inserts records sequentially with FK enforcement live,
so it depends on records arriving in dependency order. That requirement was
implicit in each backend's export table list; make it explicit and enforced.
- Add EXPORT_KIND_ORDER (user, abode, resident, apikey, note) as a documented
single source of truth, with the FK dependency chain spelled out on its doc
comment, and note the ordering guarantee on the ExportEnvelope wire-format
doc. Derive the isExportKind set from it.
- Both backends' export() now iterate EXPORT_KIND_ORDER via a loader map
(postgres omits note by leaving it out of the map), so emission order is
tied to the constant and can't drift.
- Tests: a change-detector on EXPORT_KIND_ORDER, plus assertions that both the
sqlite and postgres (fake) exports emit record kinds grouped in FK-safe
order (kind rank non-decreasing down the stream, after the leading meta).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
codinget
marked the pull request as ready for review 2026-07-23 19:49:33 +02:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
Adds backend-agnostic export / import streaming to the pluggable-backend abstraction, plus an inspect utility, three CLIs, and an HTTP export endpoint. Enables migrating data off a shared instance, moving between backend engines, and plain-file backups.
Wire format: NDJSON, one
{"kind","data"}envelope per line; leadingmetaline records format version, timestamp, source backend, and the effective (possibly narrowed) filter.What is in here
ExportImporttypes + filter helpers (kind/record scoping, hard-intersection of filters).Exportable+Importable: signal-checked async-generator export (one query per table, per-record yield, trailingerrorsentinel on mid-stream failure); manually-driven import transaction that rolls back on any error/abort and never commits partial data.Exportablevia its ownfetch({signal})(import stays sqlite-only).computeForcedExportFilterforces non-global-admin callers to their resided-in abodes + co-resident users (∩ apikeyrestrict_*);GET /exportintersects with the caller filter and wires anAbortControllerto the response socket.abode-export,abode-import,abode-inspect.Notes / decisions
#unsetpasswords; apikeys are re-minted a token (theClientApikeyview round-trips exactly).notekind is fully supported (Notes CRUD already landed), not a no-op.test/suite (test/tools/export-import.test.ts), not a fresh setup.created_by/updated_byload cleanly.Testing
npm run typecheck+typecheck:testclean.cancelledByParentin the bs3 wrapped-db suite); node:sqlite path is unaffected.TODO before merge
exportScope.ts🤖 Generated with Claude Code
Add backend-agnostic data export/import over an NDJSON wire format, plus an inspect utility, exposed via three new CLIs and an HTTP export endpoint. - ExportImport types + filter helpers (kind/record scoping, hard-intersection of filters) in src/db/{types/ExportImport,export/filter}.ts - SqliteInterface implements Exportable + Importable: signal-checked async generator export (one query per table, per-record yield, trailing error sentinel on mid-stream failure) and a manually-driven import transaction that rolls back on any error/abort and never commits partial data - ApiInterface implements Exportable via its own fetch({signal}) - computeForcedExportFilter enforces non-global-admin scope (resided-in abodes + co-resident users, intersected with apikey restrict_*); GET /export intersects it with the caller's filter and wires an AbortController to the response socket - inspectExportStream reports kinds/counts from any stream without a db - abode-export / abode-import / abode-inspect CLIs (import is sqlite-only) - Secrets are not exported: imported users default to '#unset' passwords and apikeys are re-minted a token (ClientApikey view round-trips exactly) - test/tools/export-import.test.ts: round-trip, filter narrowing, forced-scope, export/import cancellation, in-process Koa endpoint, inspect Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>421a70780atoccb970f200