diff --git a/docs/nest-storage.md b/docs/nest-storage.md index 3e7e426..30f1703 100644 --- a/docs/nest-storage.md +++ b/docs/nest-storage.md @@ -23,8 +23,8 @@ The desktop app can save nests two ways: first save and updates that same record afterward. Save As creates a new record (a copy). A failed save keeps the previous record association; switching server URLs creates a record on the new server rather than updating an id from the - previous one. Delete on the Open dialog's right-click menu permanently removes - the highlighted server record after confirmation. A separate **File > Export + previous one. Version-aware deletion hides the record and retains every archive + and revision for explicit recovery; it never purges data. A separate **File > Export .nest...** command is always available (in both modes) for producing a local file to share or back up. Export does not change the document's file save path or database record association. @@ -62,6 +62,9 @@ with enums serialized as strings (`JsonSerializerDefaults.Web` + | `madeBy` | string | | | `fileSize` | integer | Server-computed; ignored on upload. | | `savedAt` | datetime | Server-computed; ignored on upload. | +| `revision` | integer | Current immutable revision; zero on a legacy server without history. | +| `deleted` | boolean | Hidden from ordinary reads when true; retained in recovery/backup. | +| `restoredFromRevision` | nullable integer | Historical source promoted by Restore; not the preceding revision. | ## Endpoints @@ -75,7 +78,20 @@ with enums serialized as strings (`JsonSerializerDefaults.Web` + | POST | `/api/nests` | multipart: `metadata` (JSON `NestRecord`) + `file` (`.nest` bytes) | `NestRecord` with server-assigned `id` when the client sends an empty guid | | PUT | `/api/nests/{id}/file` | multipart: `metadata` + `file` | Updated `NestRecord`, or 404 if `id` is unknown | | PUT | `/api/nests/{id}/metadata` | JSON `NestRecord` | Updated `NestRecord` (archive untouched), or 404 | -| DELETE | `/api/nests/{id}` | — | 204, or 404 | +| DELETE | `/api/nests/{id}` | — | Updated hidden `NestRecord`, or 404 | +| GET | `/api/nests/{id}/revisions?offset=0&limit=100` | — | Metadata page, newest first; limit 1–200 | +| GET | `/api/nests/{id}/revisions/{revision}/file` | — | Exact immutable archive bytes | +| POST | `/api/nests/{id}/revisions/{revision}/restore` | — | New current revision of the same ID, with source metadata/bytes | +| GET | `/api/nests/deleted/query` | — | Bounded deleted-record page; same query parameters as ordinary browsing | +| GET | `/api/nests/deleted/{id}` / `/{id}/file` / `/{id}/revisions` / `/{id}/revisions/{revision}/file` | — | Recovery equivalents of current/history reads | +| POST | `/api/nests/deleted/{id}/recover` | — | New nondeleted revision of the same ID | + +All existing-record mutations require `If-Match: ""`. +Missing precondition returns 428 (upgrade/reload), malformed precondition 400, +stale precondition 409 with current metadata; none mutate data. Ordinary reads +of hidden records return 404. Changed file/metadata saves and visibility changes +append snapshots atomically; an identical save at the current revision is a no-op. +Restore appends a new snapshot, never rewrites its source or intervening history. There is no authentication; this is a LAN-only service. Do not expose it outside the shop network without adding one. @@ -122,9 +138,11 @@ HTTP upload/body reading happens outside that lock. `/healthz` executes `SELECT on the live connection; it is a connection check, not a backup, integrity scan, or guarantee of future disk capacity. -Concurrent edits of the same record remain **last-writer-wins**: there is no -optimistic version check or document lock. Coordinate editing with other operators -to avoid overwriting their changes. +Revision preconditions refuse stale edits; the database monitor is not a document +edit lock. After a conflict, explicitly reload the newer revision or Save As a +separate nest. After an ambiguous network failure, fetch current metadata before +retrying; never blindly replay a write. Legacy desktops may read but must upgrade +before updating/deleting. A version-aware client refuses writes to a legacy server. ## Server integration tests @@ -252,21 +270,29 @@ or enable Database mode. Canceling Options afterward does not undo that address. ### Backup, restore, and upgrade SQLite runs in WAL mode, so copying `nests.db` from a running service is not a -backup. Back up during a quiet period with no saves. Each step below is a Bash +backup. Back up during a maintenance period with ALL client writes disabled, +including metadata edits, delete, recover and restore, until capture finishes. +The HTTP manifest is not a transaction across requests; writes during capture +can invalidate it. Keep the entire stopped data directory, including any WAL/SHM +files, original image digest and deployment configuration. Copy +`scripts/Server.Tests/nest_manifest.py` from this source version to the deployment +directory as `nest_manifest.py` (Python 3 standard library only). It enumerates +bounded active AND deleted pages plus ALL revision pages, then streams every +current/historical archive through SHA-256. Canonical JSON includes complete +metadata, IDs, current pointers, visibility, timestamps, byte sizes and restore +source relationships. Missing pages, legacy endpoints, failed downloads, +inconsistent totals and invalid revision chains fail closed, not as empty history. +Protect manifests like the database: they contain names and comments. +Each step below is a Bash function that checks every command and stops at the first failure, however it is called; define them in the shell where `compose` is set: ```bash url=http://: -# " " for every record, sorted by id. Fails on any failed request. +# Canonical complete manifest; stdout is emitted only after all reads succeed. nest_manifest() ( - set -o pipefail - list=$(curl -fsS "$1/api/nests") || exit 1 - for id in $(printf '%s' "$list" | grep -o '"id":"[0-9a-f-]*"' | cut -d'"' -f4 | sort); do - hash=$(curl -fsS "$1/api/nests/$id/file" | sha256sum) || exit 1 - printf '%s %s\n' "$id" "${hash%% *}" - done + python3 ./nest_manifest.py "$1" || exit 1 ) # Records the expected manifest and metadata, then archives the stopped data directory. @@ -274,7 +300,9 @@ nest_manifest() ( backup_nests() ( # usage: backup_nests opennest-data-YYYY-MM-DD set -o pipefail fail() { echo "STOP: $*" >&2; exit 1; } - [[ ! -e "$1.tar" ]] || fail "$1.tar already exists" + for suffix in tar manifest metadata.json; do + [[ ! -e "$1.$suffix" ]] || fail "$1.$suffix already exists" + done nest_manifest "$url" > "$1.manifest" || fail "could not record the manifest" curl -fsS "$url/api/nests" > "$1.metadata.json" || fail "could not record the metadata" $compose stop || fail "could not stop the service" @@ -284,7 +312,7 @@ backup_nests() ( # usage: backup_nests opennest-data-YYYY-MM-DD curl -fs --retry 30 --retry-all-errors --retry-delay 1 "$url/healthz" > /dev/null \ || fail "the restarted service is not healthy" [[ $status == 0 ]] || fail "the archive failed; $1.tar is incomplete" - echo "Backed up $(wc -l < "$1.manifest") records to $1.tar" + echo "Backed up all records and revisions to $1.tar" ) backup_nests opennest-data-YYYY-MM-DD @@ -293,20 +321,25 @@ backup_nests opennest-data-YYYY-MM-DD Restore only into a volume created for that restore, never into an existing one (least of all the active volume). `restore_check` refuses an existing volume, extracts the backup into a new one, starts a temporary loopback-only container on -it, and requires the exact metadata list and every archive hash to match. It +it, and requires complete active/deleted revision metadata and archive hashes to match. It removes only the container it started; the new volume is kept either way: ```bash restore_check() ( # usage: restore_check opennest-data-YYYY-MM-DD [new-volume] set -o pipefail fail() { echo "STOP: $*" >&2; exit 1; } + [[ -r "$1.tar" && -r "$1.manifest" && -r "$1.metadata.json" ]] || fail "backup is incomplete" restore=${3:-opennest-data-restore-$(date +%Y%m%d-%H%M%S)} ! docker volume inspect "$restore" > /dev/null 2>&1 || fail "volume $restore already exists" - docker volume create "$restore" > /dev/null || fail "could not create volume $restore" + docker volume create --label "com.opennest.restore.owner=$restore" "$restore" > /dev/null \ + || fail "could not create volume $restore" + [[ $(docker volume inspect --format '{{index .Labels "com.opennest.restore.owner"}}' "$restore") == "$restore" ]] \ + || fail "restore volume ownership differs" echo "Restoring into new volume $restore" docker run --rm -i --network none --entrypoint tar -v "$restore:/app/data" "$2" \ -C /app/data -xf - < "$1.tar" || fail "extraction into $restore failed" - check=$(docker create -p 127.0.0.1::8090 -v "$restore:/app/data" "$2") \ + check=$(docker create --cap-drop ALL --security-opt no-new-privileges:true \ + -p 127.0.0.1::8090 -v "$restore:/app/data" "$2") \ || fail "could not create the check container" trap 'docker rm -f "$check" > /dev/null' EXIT docker start "$check" > /dev/null || fail "the check container did not start" @@ -315,7 +348,7 @@ restore_check() ( # usage: restore_check opennest-data-YYYY-MM-DD [new- || fail "the restored service is not healthy" curl -fsS "http://127.0.0.1:$port/api/nests" | cmp - "$1.metadata.json" || fail "metadata differs" nest_manifest "http://127.0.0.1:$port" | diff - "$1.manifest" || fail "archives differ" - echo "Verified $restore. Set OPENNEST_DATA_VOLUME=$restore in server.env, then run: \$compose up -d" + echo "Verified $restore. Keep it isolated until the operator explicitly approves switching volumes." ) restore_check opennest-data-YYYY-MM-DD @@ -328,15 +361,37 @@ To upgrade, take a stopped-service backup and keep a copy of the current `server.env`. Prefer a digest in `OPENNEST_SERVER_IMAGE` so that file records exactly what ran; for a tag, record `docker image inspect --format '{{join .RepoDigests " "}}' ` first. Change -only `OPENNEST_SERVER_IMAGE` and run `$compose up -d`. To roll back, restore the -previous `server.env` and run `$compose up -d`; if the new version wrote data the -old one cannot read, also restore the backup as above. Never run +only `OPENNEST_SERVER_IMAGE` after migration/restore rehearsal on a disposable copy, +then run `$compose up -d` only with separate deployment authorization. +**Never run an old image against a migrated database**, even if its current-table +GETs still work: old writers can overwrite snapshots or physically delete rows +without updating history. Rollback requires a NEW volume restored from the verified +PRE-migration backup and the exact old image/configuration. Test its old current +metadata/byte manifest on that isolated copy before switching; the revision-aware +manifest deliberately refuses old endpoints. Archive the migrated volume unchanged +for recovery; reverting loses post-backup changes from the old server's view, so +reconcile them deliberately, never replay them automatically. A pre-version server +backup needs its own current-record metadata/byte oracle before migration; verify +those records become byte-identical revision 1 on the disposable upgraded copy. +Never run `docker compose down -v`: it deletes the data volume. +#### Capacity and retention + +Every revision currently stores a full archive BLOB, including metadata-only +and visibility revisions. Budget approximately the sum of ALL revision archive +sizes plus current-table archives, SQLite indexes/pages, WAL and backups; do not +estimate capacity from active current records alone. Measure volume/database and +WAL growth during the disposable rehearsal, reserve space for migration and a +separate verified restore copy, and monitor free space before rollout. There is +no automatic retention or purge; no revision deletion is authorized. Deduplication +requires a separately reviewed migration after measuring actual growth. An old +backup is not a substitute for retaining the migrated volume during rollback. + ## Isolated container smoke Prerequisites: a local Linux Docker daemon on the default Unix-socket context, -Bash, curl, GNU `timeout`, `mktemp`, `tar`, and a .NET SDK able to build `net8.0` projects +Bash, Python 3, curl, GNU `timeout`, `mktemp`, `tar`, `cmp`, and a .NET SDK able to build `net8.0` projects (.NET 8 or newer). Restore needs NuGet access or cached packages. Run from the repository root after building the image: @@ -346,6 +401,22 @@ scripts/Test-ServerContainer.sh --image opennest-server:local scripts/Test-ServerContainer.sh --image opennest-server:local --results /path/to/results ``` +Manifest refusal tests run without Docker. The optional image variables also +exercise the Bash backup/restore blocks extracted verbatim from this page in a +unique disposable Compose project, including existing-file/volume refusal and +tampered-history rejection. The old image test restores ONLY a synthetic +pre-migration backup to another fresh volume and compares every old record's +metadata/bytes; it never mounts a migrated database into the old image. + +```sh +python3 -m unittest discover -s scripts/Server.Tests -p test_revision_backup.py -v +OPENNEST_TEST_IMAGE=opennest-server:local \ +OPENNEST_TEST_OLD_IMAGE= \ + python3 -m unittest discover -s scripts/Server.Tests -p test_revision_backup.py -v +``` + +Local smoke evidence does not imply registry publication or production deployment. + The wrapper accepts an image, not a URL or an existing volume/container. It uses only the local default Docker context, creates uniquely labeled disposable resources, and publishes to `127.0.0.1` on a Docker-allocated port. Every container @@ -363,16 +434,21 @@ owns `nests.db`, and can write `/app/data` but not `/app` or the server DLL. The test-only `scripts/Server.Tests/Server.Tests.csproj` console exercises the existing .NET storage client; it is not included in the server image. +Before seeding, the wrapper restores two synthetic legacy-schema rows into its +fresh volume, verifies byte-identical revision-1 migration against a precomputed +complete metadata/hash oracle, then checks repeated startup preserves history. The synthetic fixture is generated in code: one plate and one rectangular part, with no customer files. `seed` uses the actual `RemoteNestRepository` and `NestSaveSession` to create, update the same ID, and save a copy with a new ID. +It also creates metadata-only, historical-restore, delete/recover/delete revisions, +leaving a six-revision hidden record with a restore-source relationship. It checks read-back metadata, session binding, exact list membership, raw camelCase/string-enum JSON, identical downloaded bytes, and `NestReader` geometry and counts. `reject` snapshots every record's metadata and archive hash, then checks 400 responses for missing/invalid/null metadata and missing/empty files on both upload and file-update routes; unknown metadata/file IDs must return 404. Every rejected operation must leave the snapshot unchanged. -`limits` stores, downloads byte-for-byte, and deletes a 29,000,000-byte synthetic +`limits` stores, downloads byte-for-byte, and soft-deletes a 29,000,000-byte synthetic archive (buffered to disk by the non-root runtime), then requires 413 for a request over 30,000,000 bytes sent with `Expect: 100-continue`, and a reported failure from the real client's oversized upload and file update. The snapshot @@ -384,6 +460,10 @@ fresh named volume, starts a replacement on another allocated loopback port, and archive hashes, and exact list membership. It then follows the documented stopped-service backup: `tar` of the whole data directory through the image, restored into a second owned volume whose service must pass the same `verify`. +The Python manifest is captured after all mutations and compared exactly after +restart and volume restore, with page size 1 to exercise every pagination boundary. +It covers active and deleted current records and ALL revisions, not just the +ordinary `/api/nests` enumeration; the large soft-deleted archive remains included. The original volume mounted read-only, and a root-owned `tmpfs`, must each make the server exit nonzero at startup with a SQLite error and never report healthy. diff --git a/scripts/Server.Tests/Program.cs b/scripts/Server.Tests/Program.cs index 57d9283..34cf8fe 100644 --- a/scripts/Server.Tests/Program.cs +++ b/scripts/Server.Tests/Program.cs @@ -103,6 +103,20 @@ internal static class Program Check(copy.Id != Guid.Empty && copy.Id != created.Id, "Save copy must create a different id."); CheckBinding(session, copy.Id, options.Url); var copyState = await ReadSaved(repository, nest, archive, copy, ct); + // Keep a hidden nest with metadata, restore-source and visibility history in the backup. + var hidden = await repository.UploadAsync(archive, new NestRecord { Name = "Synthetic hidden history" }, ct); + hidden.Comments = "Metadata-only revision"; + hidden = await repository.UpdateMetadataAsync(hidden.Id, hidden, hidden.Revision, ct); + hidden = await repository.RestoreAsync(hidden.Id, 1, hidden.Revision, ct); + Check(hidden.Revision == 3 && hidden.RestoredFromRevision == 1, "Restore must record its source."); + hidden = await repository.DeleteAsync(hidden.Id, hidden.Revision, ct); + hidden = await repository.RecoverAsync(hidden.Id, hidden.Revision, ct); + hidden = await repository.DeleteAsync(hidden.Id, hidden.Revision, ct); + Check(hidden.Deleted && hidden.Revision == 6, "Visibility changes must retain all revisions."); + var hiddenHistory = await repository.GetDeletedRevisionsAsync(hidden.Id, limit: 2, cancellationToken: ct); + Check(hiddenHistory.Total == 6 && hiddenHistory.Items.Count == 2, "Deleted history must be bounded."); + Check((await repository.GetDeletedRevisionFileAsync(hidden.Id, 1, ct))!.SequenceEqual(archive), + "Deleted history must retain original bytes."); var entries = baseline.Concat(new[] { updateState, copyState }).ToList(); await CheckSnapshot(repository, entries, ct); var state = new SmokeState { Version = 1, Records = entries }; @@ -153,6 +167,8 @@ internal static class Program if (test.File is not null) content.Add(new ByteArrayContent(test.File), "file", "synthetic.nest"); using var request = new HttpRequestMessage(method, new Uri(new Uri(url + "/"), path)) { Content = content }; + if (method == HttpMethod.Put) + request.Headers.TryAddWithoutValidation("If-Match", $"\"{before[0].Metadata.Revision}\""); using var response = await http.SendAsync(request, ct); Check(response.StatusCode == HttpStatusCode.BadRequest, $"{method} {test.Name} must return 400."); await CheckSnapshot(repository, before, ct); @@ -190,14 +206,15 @@ internal static class Program Check(downloaded is not null && downloaded.AsSpan().SequenceEqual(accepted), "A near-limit archive must download byte-for-byte."); await CheckMembership(repository, before.Select(s => s.Metadata.Id).Append(created.Id), ct); - await repository.DeleteAsync(created.Id, ct); + var deleted = await repository.DeleteAsync(created.Id, created.Revision, ct); + Check(deleted.Deleted && deleted.Revision == 2, "Near-limit archive must be retained as deleted history."); await CheckSnapshot(repository, before, ct); Console.WriteLine($"Accepted, downloaded, and deleted a {accepted.Length}-byte archive; all other records unchanged."); // Kestrel rejects an oversized declared length before the app reads the form. With // Expect: 100-continue the client waits for, and must receive, that 413 response. var oversized = new byte[RequestBodyLimit]; - var oversizedRecord = new NestRecord { Name = "Synthetic oversized upload" }; + var oversizedRecord = new NestRecord { Name = "Synthetic oversized upload", Revision = before[0].Metadata.Revision }; var metadata = JsonSerializer.Serialize(oversizedRecord, JsonOptions); foreach (var method in new[] { HttpMethod.Post, HttpMethod.Put }) { @@ -207,6 +224,8 @@ internal static class Program content.Add(new ByteArrayContent(oversized), "file", "synthetic.nest"); using var request = new HttpRequestMessage(method, new Uri(new Uri(url + "/"), path)) { Content = content }; request.Headers.ExpectContinue = true; + if (method == HttpMethod.Put) + request.Headers.TryAddWithoutValidation("If-Match", $"\"{before[0].Metadata.Revision}\""); using var response = await http.SendAsync(request, ct); Check(response.StatusCode == HttpStatusCode.RequestEntityTooLarge, $"{method} of a request over {RequestBodyLimit} bytes must return 413."); @@ -244,6 +263,7 @@ internal static class Program Check(returned.SavedAt != default, "Server must assign SavedAt."); var expected = NestRecordFactory.FromNest(nest, returned.Id, archive.LongLength); expected.SavedAt = returned.SavedAt; + expected.Revision = returned.Revision; CheckMetadata(expected, returned); var readback = await repository.GetMetadataAsync(returned.Id, ct); Check(readback is not null, "Saved metadata must be readable."); @@ -360,6 +380,9 @@ internal static class Program Check(reader.Warnings.Count == 0, "NestReader must parse without warnings."); var parsed = NestRecordFactory.FromNest(nest, metadata.Id, archive.LongLength); parsed.SavedAt = metadata.SavedAt; + parsed.Revision = metadata.Revision; + parsed.Deleted = metadata.Deleted; + parsed.RestoredFromRevision = metadata.RestoredFromRevision; CheckMetadata(metadata, parsed); Check(nest.Drawings.Count == 1 && nest.Plates.Count == 1 && nest.Plates[0].Parts.Count == 1, "Parsed archive must contain one drawing, plate, and part."); @@ -459,7 +482,15 @@ internal static class Program CheckWireRecord(record); } else - CheckWireRecord(json.RootElement); + { + if (json.RootElement.TryGetProperty("items", out var items)) + { + foreach (var record in items.EnumerateArray()) + CheckWireRecord(record); + } + else + CheckWireRecord(json.RootElement); + } CheckedResponses++; } return response; diff --git a/scripts/Server.Tests/legacy_fixture.py b/scripts/Server.Tests/legacy_fixture.py new file mode 100644 index 0000000..e6f01fa --- /dev/null +++ b/scripts/Server.Tests/legacy_fixture.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 +"""Generate only synthetic legacy SQLite files inside caller-owned scratch.""" +import hashlib +import json +from pathlib import Path +import sqlite3 +import sys +import tarfile + +SCHEMA = """ +CREATE TABLE nests ( + id TEXT PRIMARY KEY, name TEXT NOT NULL, customer TEXT NOT NULL DEFAULT '', + dateCreated TEXT NOT NULL, dateModified TEXT NOT NULL, material TEXT NOT NULL DEFAULT '', + thickness REAL NOT NULL DEFAULT 0, status TEXT NOT NULL DEFAULT 'Quote', + plateCount INTEGER NOT NULL DEFAULT 0, partCount INTEGER NOT NULL DEFAULT 0, + comments TEXT NOT NULL DEFAULT '', madeBy TEXT NOT NULL DEFAULT '', + fileSize INTEGER NOT NULL DEFAULT 0, savedAt TEXT NOT NULL, file BLOB NOT NULL); +CREATE INDEX ix_nests_savedAt ON nests(savedAt DESC); +""" + + +def create(directory): + db = directory / 'nests.db' + expected = [] + with sqlite3.connect(db) as connection: + connection.executescript(SCHEMA) + for index in range(1, 3): + ident = f'00000000-0000-4000-8000-{index:012d}' + data = bytes([0, index, 255, 128, 0]) + date = '2026-01-02T03:04:05' + fields = (ident, f'Synthetic legacy {index}', 'Synthetic fixture', date, date, + 'Synthetic alloy', 0.25, 'ToBeCut', 2, 3, 'Legacy notes', 'Fixture', len(data), date) + connection.execute('INSERT INTO nests VALUES (' + ','.join('?' * 15) + ')', (*fields, data)) + metadata: dict[str, object] = dict(zip(('id', 'name', 'customer', 'dateCreated', 'dateModified', 'material', + 'thickness', 'status', 'plateCount', 'partCount', 'comments', + 'madeBy', 'fileSize', 'savedAt'), fields)) + metadata.update(status='toBeCut', revision=1, deleted=False, restoredFromRevision=None) + archive = {'sha256': hashlib.sha256(data).hexdigest(), 'bytes': len(data)} + expected.append({'current': metadata, 'archive': archive, + 'revisions': [{'metadata': metadata, 'archive': archive}]}) + (directory / 'legacy.manifest').write_text(json.dumps({'format': 1, 'records': expected}, sort_keys=True, indent=2) + '\n') + with tarfile.open(directory / 'legacy.tar', 'w') as archive: + info = archive.gettarinfo(str(db), arcname='./nests.db') + info.uid = info.gid = 1654 + info.uname = info.gname = '' + with db.open('rb') as source: + archive.addfile(info, source) + + +if __name__ == '__main__': + create(Path(sys.argv[1])) diff --git a/scripts/Server.Tests/nest_manifest.py b/scripts/Server.Tests/nest_manifest.py new file mode 100644 index 0000000..c417fa9 --- /dev/null +++ b/scripts/Server.Tests/nest_manifest.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +"""Read-only, canonical revision manifest; requires a quiescent version-aware server.""" +import argparse +import hashlib +import json +import sys +import urllib.error +import urllib.request +import uuid + + +def require(condition, message): + if not condition: + raise ValueError(message) + + +def collect(url, limit=200): + require(1 <= limit <= 200, "Invalid page size") + opener = urllib.request.build_opener(urllib.request.ProxyHandler({})) + + def read(path, binary=False): + try: + response = opener.open(url.rstrip('/') + path, timeout=30) + except urllib.error.HTTPError as error: + error.close() + raise ValueError("HTTP request failed") from None + with response: + if binary: + digest = hashlib.sha256() + size = 0 + while chunk := response.read(1024 * 1024): + size += len(chunk) + digest.update(chunk) + return {"sha256": digest.hexdigest(), "bytes": size} + return json.load(response) + + def pages(path): + items, total = [], None + while True: + page = read(f"{path}?offset={len(items)}&limit={limit}") + require(page['offset'] == len(items) and page['limit'] == limit, "Page bounds differ") + require(type(page['total']) is int and page['total'] >= 0, "Invalid total") + if total is None: + total = page['total'] + require(total == page['total'], "Server changed during enumeration") + batch = page['items'] + require(isinstance(batch, list) and len(batch) <= limit, "Invalid page") + require(len(items) + len(batch) <= total, "Page exceeds total") + require(batch or len(items) == total, "Truncated enumeration") + items.extend(batch) + if len(items) == total: + return items + + result, ids = [], set() + for deleted in (False, True): + path = '/api/nests/deleted' if deleted else '/api/nests' + records = pages(path + '/query') + for record in records: + ident = record['id'] + require(str(uuid.UUID(ident)) == ident and ident not in ids, "Invalid or duplicate nest ID") + ids.add(ident) + require(record['deleted'] is deleted, "Visibility differs") + revisions = pages(path + '/' + ident + '/revisions') + require([r['revision'] for r in revisions] == list(range(record['revision'], 0, -1)), + "Missing, duplicate, or unordered revisions") + require(revisions and revisions[0] == record, "Current projection differs from latest revision") + history = [] + for revision in revisions: + number = revision['revision'] + require(revision['id'] == ident, "Revision belongs to another nest") + parent = revision['restoredFromRevision'] + require(parent is None or type(parent) is int and 1 <= parent < number, + "Invalid restore source relationship") + archive = read(f"{path}/{ident}/revisions/{number}/file", binary=True) + require(archive['bytes'] == revision['fileSize'], "Revision archive size differs") + history.append({"metadata": revision, "archive": archive}) + current = read(f"{path}/{ident}/file", binary=True) + require(current == history[0]['archive'], "Current archive differs from latest revision") + result.append({"current": record, "archive": current, + "revisions": sorted(history, key=lambda r: r['metadata']['revision'])}) + return {"format": 1, "records": sorted(result, key=lambda r: r['current']['id'])} + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('url') + parser.add_argument('--page-size', type=int, default=200) + args = parser.parse_args() + try: + manifest = collect(args.url, args.page_size) + # Emit only after every request and invariant succeeds; failure never emits a partial manifest. + print(json.dumps(manifest, sort_keys=True, indent=2)) + except Exception as error: + print(f"Manifest failed: {type(error).__name__}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/scripts/Server.Tests/test_revision_backup.py b/scripts/Server.Tests/test_revision_backup.py new file mode 100644 index 0000000..ee661e4 --- /dev/null +++ b/scripts/Server.Tests/test_revision_backup.py @@ -0,0 +1,277 @@ +"""Manifest refusal tests and opt-in executable documentation/rollback rehearsal. + +OPENNEST_TEST_IMAGE= [OPENNEST_TEST_OLD_IMAGE=] +python3 -m unittest discover -s scripts/Server.Tests -p test_revision_backup.py -v +Only generated loopback containers/volumes are used; no live server URL is accepted. +""" +import copy +import hashlib +import http.server +import json +import os +from pathlib import Path +import re +import shutil +import socket +import subprocess +import tempfile +import threading +import time +import unittest +import urllib.parse +import urllib.request +import uuid + +from nest_manifest import collect +from legacy_fixture import create + +ROOT = Path(__file__).resolve().parents[2] + + +class ManifestTests(unittest.TestCase): + def setUp(self): + self.ident = '00000000-0000-4000-8000-000000000001' + self.record = dict(id=self.ident, revision=3, deleted=True, restoredFromRevision=1, + fileSize=3, comments='restored', savedAt='2026-01-02T03:04:05') + self.revisions = [self.record, dict(self.record, revision=2, deleted=False, + restoredFromRevision=None, comments='edited'), + dict(self.record, revision=1, deleted=False, restoredFromRevision=None)] + self.failure = None + owner = self + + class Handler(http.server.BaseHTTPRequestHandler): + def do_GET(self): + path = urllib.parse.urlsplit(self.path).path + query = urllib.parse.parse_qs(urllib.parse.urlsplit(self.path).query) + if owner.failure == 'legacy' or owner.failure == 'download' and path.endswith('/file'): + self.send_error(500 if owner.failure == 'download' else 404) + return + if path.endswith('/file'): + data = b'abc' + else: + values = owner.revisions if path.endswith('/revisions') else ([owner.record] if '/deleted/' in path else []) + offset, limit = int(query['offset'][0]), int(query['limit'][0]) + batch = values[offset:offset + limit] + if owner.failure == 'truncated' and offset > 0: + batch = [] + total = len(values) + (1 if owner.failure == 'total' and offset else 0) + data = json.dumps(dict(items=batch, total=total, offset=offset, limit=limit)).encode() + self.send_response(200) + self.end_headers() + self.wfile.write(data) + + def log_message(self, format, *args): + pass + + self.server = http.server.ThreadingHTTPServer(('127.0.0.1', 0), Handler) + self.thread = threading.Thread(target=self.server.serve_forever, daemon=True) + self.thread.start() + self.url = f'http://127.0.0.1:{self.server.server_port}' + + def tearDown(self): + self.server.shutdown() + self.server.server_close() + self.thread.join() + + def test_all_deleted_revisions_and_parent_metadata_are_preserved(self): + manifest = collect(self.url, 1) + record = manifest['records'][0] + self.assertEqual(self.record, record['current']) + self.assertEqual([1, 2, 3], [r['metadata']['revision'] for r in record['revisions']]) + self.assertEqual(hashlib.sha256(b'abc').hexdigest(), record['archive']['sha256']) + self.assertEqual(1, record['revisions'][2]['metadata']['restoredFromRevision']) + altered = copy.deepcopy(manifest) + altered['records'][0]['revisions'][0]['metadata']['comments'] = 'tampered' + self.assertNotEqual(manifest, altered) + + def test_failed_download_emits_no_partial_manifest(self): + self.failure = 'download' + result = subprocess.run(['python3', str(ROOT / 'scripts/Server.Tests/nest_manifest.py'), self.url], capture_output=True) + self.assertNotEqual(0, result.returncode) + self.assertEqual(b'', result.stdout) + + def test_legacy_endpoint_is_not_empty_history(self): + self.failure = 'legacy' + with self.assertRaises(Exception): + collect(self.url, 1) + + def test_missing_pages_and_changed_totals_fail(self): + for failure in ('truncated', 'total'): + self.failure = failure + with self.assertRaises(ValueError): + collect(self.url, 1) + + def test_missing_revision_and_invalid_parent_fail(self): + self.revisions.pop(1) + with self.assertRaises(ValueError): + collect(self.url, 1) + self.revisions.insert(1, dict(self.record, revision=2, restoredFromRevision=None)) + self.record['restoredFromRevision'] = 3 + with self.assertRaises(ValueError): + collect(self.url, 1) + + +@unittest.skipUnless(os.environ.get('OPENNEST_TEST_IMAGE'), 'Set OPENNEST_TEST_IMAGE for disposable Docker documentation rehearsal') +class DocumentationTests(unittest.TestCase): + def run_command(self, *args, input=None, check=True): + result = subprocess.run(args, input=input, text=True, capture_output=True, timeout=120, cwd=self.scratch) + if check and result.returncode: + self.fail(f"Command failed ({result.returncode}): {args}\n{result.stdout}\n{result.stderr}") + return result + + def docker(self, *args, **kwargs): + return self.run_command('docker', '--context', 'default', *args, **kwargs) + + def setUp(self): + self.temp = tempfile.TemporaryDirectory(prefix='opennest-backup-doc-', dir=os.environ.get('TMPDIR')) + self.scratch = Path(self.temp.name) + self.token = 'opennest-doc-' + uuid.uuid4().hex + self.image = os.environ['OPENNEST_TEST_IMAGE'] + self.volumes = [] + self.containers = [] + self.compose = None + self.addCleanup(self.cleanup_resources) + endpoint = self.docker('context', 'inspect', 'default', '--format', '{{.Endpoints.docker.Host}}').stdout.strip() + self.assertEqual('unix:///var/run/docker.sock', endpoint) + self.volume = self.token + '-data' + self.new_volume(self.volume) + with socket.socket() as probe: + probe.bind(('127.0.0.1', 0)) + port = probe.getsockname()[1] + configuration = {'services': {'opennest-server': {'image': self.image, + 'ports': [f'127.0.0.1:{port}:8090'], 'volumes': [self.volume + ':/app/data'], + 'cap_drop': ['ALL'], 'security_opt': ['no-new-privileges:true']}}, + 'volumes': {self.volume: {'external': True}}} + (self.scratch / 'compose.json').write_text(json.dumps(configuration)) + self.compose = f'docker --context default compose -p {self.token} -f compose.json' + self.run_command('bash', '-c', self.compose + ' up -d') + self.cid = self.run_command('bash', '-c', self.compose + ' ps -q').stdout.strip() + self.url = self.ready(self.cid) + shutil.copy(ROOT / 'scripts/Server.Tests/nest_manifest.py', self.scratch / 'nest_manifest.py') + doc = (ROOT / 'docs/nest-storage.md').read_text() + blocks = re.findall(r'```bash\n(.*?)```', doc, re.S) + selected = [block for block in blocks if 'backup_nests()' in block or 'restore_check()' in block] + self.assertEqual(2, len(selected)) + self.functions = '\n'.join('\n'.join(line for line in block.splitlines() + if not line.startswith(('url=', 'backup_nests opennest-', 'restore_check opennest-'))) + for block in selected) + (self.scratch / 'procedures.sh').write_text(self.functions) + self.run_command('bash', '-n', 'procedures.sh') + # Opaque synthetic archives exercise history without any customer data. + record = self.request('POST', '/api/nests', multipart=True) + record['comments'] = 'metadata version' + record = self.request('PUT', f"/api/nests/{record['id']}/metadata", record, record['revision']) + record = self.request('POST', f"/api/nests/{record['id']}/revisions/1/restore", revision=record['revision']) + self.request('DELETE', f"/api/nests/{record['id']}", revision=record['revision']) + self.request('POST', '/api/nests', multipart=True) + + def new_volume(self, name): + self.assertNotEqual(0, self.docker('volume', 'inspect', name, check=False).returncode) + self.docker('volume', 'create', '--label', 'com.opennest.doc-owner=' + self.token, name) + self.assertEqual(self.token, self.docker('volume', 'inspect', '--format', '{{index .Labels "com.opennest.doc-owner"}}', name).stdout.strip()) + self.volumes.append(name) + + def ready(self, cid): + port = self.docker('port', cid, '8090/tcp').stdout.strip().split(':')[1] + url = f'http://127.0.0.1:{port}' + for _ in range(60): + try: + with urllib.request.urlopen(url + '/healthz', timeout=1) as response: + if json.load(response) == {'status': 'ok'}: + return url + except OSError: + time.sleep(1) + self.fail('Disposable service did not become ready') + + def request(self, method, path, data=None, revision=None, multipart=False): + headers = {} + if revision: + headers['If-Match'] = f'"{revision}"' + if multipart: + boundary = 'synthetic-boundary' + data = (f'--{boundary}\r\nContent-Disposition: form-data; name="metadata"\r\n\r\n' + '{"name":"Synthetic documentation fixture"}\r\n' + f'--{boundary}\r\nContent-Disposition: form-data; name="file"; filename="fixture.nest"\r\n' + f'Content-Type: application/zip\r\n\r\nabc\r\n--{boundary}--\r\n').encode() + headers['Content-Type'] = 'multipart/form-data; boundary=' + boundary + elif data is not None: + data = json.dumps(data).encode() + headers['Content-Type'] = 'application/json' + request = urllib.request.Request(self.url + path, data=data, headers=headers, method=method) + with urllib.request.urlopen(request, timeout=10) as response: + return json.load(response) + + def procedure(self, command, check=True): + prefix = f'compose={json.dumps(self.compose)}; url={self.url}; source ./procedures.sh; ' + return self.run_command('bash', '-c', prefix + command, check=check) + + def cleanup_resources(self): + # Compose removes only this randomly generated project's containers, never its external volumes. + try: + if self.compose: + self.run_command('bash', '-c', self.compose + ' down') + for cid in self.containers: + self.docker('rm', '-f', cid) + for volume in self.volumes: + labels = json.loads(self.docker('volume', 'inspect', '--format', '{{json .Labels}}', volume).stdout) + self.assertTrue(labels.get('com.opennest.doc-owner') == self.token + or labels.get('com.opennest.restore.owner') == volume and volume.startswith(self.token + '-')) + self.assertEqual('', self.docker('container', 'ls', '-aq', '--filter', 'volume=' + volume).stdout.strip(), + 'A documentation check container was not removed') + self.docker('volume', 'rm', volume) + self.assertNotEqual(0, self.docker('volume', 'inspect', volume, check=False).returncode) + self.assertEqual('', self.docker('container', 'ls', '-aq', '--filter', 'label=com.docker.compose.project=' + self.token).stdout.strip()) + finally: + self.temp.cleanup() + + def test_verbatim_backup_restore_and_refusals(self): + self.procedure('backup_nests backup') + original = json.loads((self.scratch / 'backup.manifest').read_text()) + self.assertEqual(2, len(original['records'])) + self.assertEqual(5, sum(len(r['revisions']) for r in original['records'])) + started = self.docker('inspect', '--format', '{{.State.StartedAt}}', self.cid).stdout + refused = self.procedure('if backup_nests backup; then exit 90; fi') + self.assertIn('already exists', refused.stderr) + self.assertEqual(started, self.docker('inspect', '--format', '{{.State.StartedAt}}', self.cid).stdout) + restored = self.token + '-restored' + refused = self.procedure(f'if restore_check backup {self.image} {self.volume}; then exit 90; fi') + self.assertIn('already exists', refused.stderr) + self.volumes.append(restored) + self.procedure(f'restore_check backup {self.image} {restored}') + tampered = self.token + '-tampered' + self.volumes.append(tampered) + original['records'][0]['revisions'][0]['archive']['sha256'] = '0' * 64 + (self.scratch / 'backup.manifest').write_text(json.dumps(original)) + refused = self.procedure(f'if restore_check backup {self.image} {tampered}; then exit 90; fi') + self.assertIn('archives differ', refused.stderr) + + @unittest.skipUnless(os.environ.get('OPENNEST_TEST_OLD_IMAGE'), 'Set OPENNEST_TEST_OLD_IMAGE for old-image rollback rehearsal') + def test_old_image_only_on_pre_migration_backup_copy(self): + create(self.scratch) + rollback = self.token + '-rollback' + self.new_volume(rollback) + # Restore synthetic pre-migration tar to a NEW owned volume, never the upgraded database. + with (self.scratch / 'legacy.tar').open('rb') as archive: + subprocess.run(['docker', '--context', 'default', 'run', '--rm', '-i', '--network', 'none', + '--entrypoint', 'tar', '-v', rollback + ':/app/data', self.image, + '-C', '/app/data', '-xf', '-'], stdin=archive, check=True, timeout=30, capture_output=True) + cid = self.docker('run', '-d', '--cap-drop', 'ALL', '--security-opt', 'no-new-privileges:true', + '-p', '127.0.0.1::8090', '-v', rollback + ':/app/data', + os.environ['OPENNEST_TEST_OLD_IMAGE']).stdout.strip() + self.containers.append(cid) + url = self.ready(cid) + expected = json.loads((self.scratch / 'legacy.manifest').read_text())['records'] + with urllib.request.urlopen(url + '/api/nests') as response: + actual = {r['id']: r for r in json.load(response)} + self.assertEqual(2, len(actual)) + for entry in expected: + record = {k: v for k, v in entry['current'].items() if k not in ('revision', 'deleted', 'restoredFromRevision')} + self.assertEqual(record, actual[record['id']]) + with urllib.request.urlopen(url + f"/api/nests/{record['id']}/file") as response: + self.assertEqual(entry['archive']['sha256'], hashlib.sha256(response.read()).hexdigest()) + with self.assertRaises(Exception): + collect(url) + + +if __name__ == '__main__': + unittest.main() diff --git a/scripts/Test-ServerContainer.sh b/scripts/Test-ServerContainer.sh index 357c6d1..fffa89c 100755 --- a/scripts/Test-ServerContainer.sh +++ b/scripts/Test-ServerContainer.sh @@ -23,7 +23,7 @@ while (($#)); do esac done [[ -n "$image" && "$image" != -* ]] || { usage; exit 2; } -for command in docker dotnet curl timeout mktemp tar; do +for command in docker dotnet curl timeout mktemp tar python3 cmp; do command -v "$command" >/dev/null || { printf 'Missing prerequisite: %s\n' "$command" >&2; exit 2; } done root=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd) @@ -40,7 +40,7 @@ token=${scratch##*/} owner_label='com.opennest.server-smoke.owner' # Two service generations on one data volume, a stopped-service backup and restore into # a second volume, its service, then two storage locations that must fail startup. -container_roles=(first replacement backup restore restored readonly unwritable) +container_roles=(first replacement backup restore restored readonly unwritable legacy) container_names=() cidfiles=() container_attempted=() @@ -306,16 +306,29 @@ run_tool() { # A fresh named volume must be writable by the non-root image user. create_volume 0 +# Migrate a synthetic prior-schema database, never an existing service's data. +python3 "$root/scripts/Server.Tests/legacy_fixture.py" "$scratch" +create_container 7 --rm --interactive --network none --entrypoint tar \ + --mount "type=volume,src=${volumes[0]},dst=/app/data" "$image" -C /app/data -xf - +docker_local start --attach --interactive "$cid" < "$scratch/legacy.tar" > "$run_results/legacy.log" 2>&1 service=first start_service 0 "${volumes[0]}" -run_tool seed --state "$scratch/state.json" +python3 "$root/scripts/Server.Tests/nest_manifest.py" "$url" --page-size 1 > "$scratch/migrated.manifest" +cmp "$scratch/legacy.manifest" "$scratch/migrated.manifest" +run_tool seed --state "$scratch/state.json" --test-allow-nonempty +python3 "$root/scripts/Server.Tests/nest_manifest.py" "$url" --page-size 1 > "$scratch/before-reject.manifest" run_tool reject +python3 "$root/scripts/Server.Tests/nest_manifest.py" "$url" --page-size 1 > "$scratch/after-reject.manifest" +cmp "$scratch/before-reject.manifest" "$scratch/after-reject.manifest" run_tool limits +python3 "$root/scripts/Server.Tests/nest_manifest.py" "$url" --page-size 1 > "$scratch/all.manifest" retire_service 0 # The replacement gets another Docker-allocated loopback port; only the volume is reused. service=replacement start_service 1 "${volumes[0]}" run_tool verify --state "$scratch/state.json" +python3 "$root/scripts/Server.Tests/nest_manifest.py" "$url" --page-size 1 > "$scratch/reopened.manifest" +cmp "$scratch/all.manifest" "$scratch/reopened.manifest" retire_service 1 # Back up the whole data directory from the stopped service (SQLite WAL makes a live @@ -335,4 +348,7 @@ docker_local start --attach --interactive "$cid" < "$scratch/data-backup.tar" > service=restored start_service 4 "${volumes[1]}" run_tool verify --state "$scratch/state.json" +python3 "$root/scripts/Server.Tests/nest_manifest.py" "$url" --page-size 1 > "$scratch/restored.manifest" +cmp "$scratch/all.manifest" "$scratch/restored.manifest" +printf 'PASS: legacy migration, idempotent restart, active/deleted ALL-revision metadata, parent relationships and byte hashes match after restore.\n' printf 'PASS: non-root healthy runtime, real-client round trips, rejections, upload limit, persistent recreation, stopped-service backup/restore, and storage startup failures.\n'