--- wo: 1 title: Write RFC-OPALFLAME-ARCHIVE-CONTAINER-V1.md -- the parity-protected storage container programme: OPALFLAME-ARCHIVE-CONTAINER depends_on: [] blocks: [2, 3, 4, 5] --- # WO 1 -- the container RFC ## Doing Write `docs/RFC-OPALFLAME-ARCHIVE-CONTAINER-V1.md`: the normative definition of the parity-protected, optionally encrypted and signed archive container that carries a `.ofp3.tgz` (or any payload) through long-term storage. Sibling to `docs/RFC-OPALFLAME-FORMAT-STANDARD-V3.md`, and deliberately separate from it. That RFC defines the logical document; this one defines the storage container, which is indifferent to its payload and reusable for anything else worth archiving. Same standard as the v3 RFC: RFC 2119 keywords, complete enough that an independent implementer could build a conforming packer and unpacker from it alone. ## The pipeline this specifies ``` project/ -> project.tar uncompressed -> project.tar.sha256 plaintext hash, NEVER encrypted -> [encrypt project.tar] optional, BEFORE split -> split -b 1440K -> project.part.000 .. NNN -> project.part.sfv CRC32 over the parts (authoritative list) -> project.nfo release/packaging notes, extract / repair / install instructions, registration, dev contact, user guide, TS FAQ (recovery) -> rm project.tar, project.tar.enc originals removed -> par2 create -r10 over the ENTIRE remaining dir -> full_packaged_asset-.tar uncompressed outer container ``` ## Parameters -- SETTLED BY MEASUREMENT, do not re-derive These were measured on haven-4346 with par2cmdline 1.2.0 (probe: `bin/par2_probe.sh`) against real export sizes. Record them in the RFC with the evidence, and state that they are measured rather than assumed. **Chunk size: 1.44MB (`split -b 1440K`).** **Redundancy: `par2 create -r10`.** **One size tier.** Every real project export is under 100MB, so the earlier DVD/CD tier table is deleted, not carried forward. Measured corpus: | project | size | |---|---| | same_water ACT1 vols | 0.8-0.9 MB | | jade mask | 1.1-9.2 MB | | pgtl-imperial_shield / fifa_fraud | 11-12 MB | | fahu-template | 17 MB | | pgtl-fix_iran | 35 MB | | FAHU video packages (x4) | 39-50 MB | | baseline-v2 fixture | 48 MB | | pgtl-toll_booth | 57 MB | | pgtl-seahorse_syndicate | 75 MB (largest found) | Exports carry images, clips and audio -- NOT a final render -- which is why they stay small. **Why these values.** Measured on a 50MB archive at 1.44MB chunks (34 chunks), `-r10` produced a 6.4MB recovery set in ~1s and survived: | damage | outcome | |---|---| | 3 whole chunks deleted | recovered, payload sha256 exact | | 4 whole chunks deleted | failed ("need 36 more recovery blocks") | | 5 whole chunks deleted | failed ("need 95 more") | | scattered corruption across 12 chunks | recovered, payload sha256 exact | par2 real overhead is ~1.25x nominal: `-r5`=7.3%, `-r10`=12.9%, `-r15`=18.5%, `-r20`=23.7% of payload. The governing relationship, which the RFC MUST state so the numbers are re-derivable if sizes ever change: > Surviving `K` entirely-lost chunks requires redundancy exceeding > `K x (chunk_size / total_size) x 100` percent, plus headroom for par2 > overhead. Sitting *at* that ratio FAILS -- measured: a chunk sized at exactly > 10% of the archive was unrecoverable at `-r10`, short by 9 blocks. At 1.44MB each chunk is ~2.9% of a 50MB archive, which is why `-r10` clears 3 whole chunks instead of zero. This is the entire reason the chunk size is small. ## Required contents 1. **Scope, terminology, conformance.** What makes a container, a packer and an unpacker conforming. 2. **The pipeline**, step by step, normatively, in the order above. 3. **Uncompressed throughout, and why.** Payload is already-compressed media so compression buys almost nothing; and unrepairable damage in a compressed stream destroys everything downstream of it, where damage in a raw tar stays local and surrounding members remain salvageable by hand. Graceful degradation is the design goal. The outer container is uncompressed too. 4. **`split`, not split-zip, and why.** Split-zip keeps the central directory in the final volume, so losing that one volume can make the set unopenable by standard tools; raw split output is sequential bytes that `cat` back together, and a missing middle chunk still leaves the rest walkable. 5. **Integrity layers, and what each is for.** State plainly that these are three different jobs, not redundancy: - `project.part.sfv` (CRC32) -- **triage**: which chunk moved, and the authoritative ordered file list for reassembly. CRC32 is correct here: the job is detection of media decay, not defence against an adversary, and par2 carries its own hashes underneath. - par2 -- **repair**. - `project.tar.sha256` -- **the integrity authority** for the payload. 6. **`project.tar.sha256` stays unencrypted, and there is NO ciphertext hash.** Keyless verification of the stored bytes is already provided by the SFV, with par2 restoring parts to that state when they fail. The plaintext hash has no subject until decryption produces one -- hash and data become available at the same moment, which is the soonest it could be checked. There is no gap to close. Record this reasoning so it is not "fixed" later. 7. **Encryption: BEFORE split, so parity covers ciphertext.** Never parity-then-encrypt: an AEAD tag fails over the whole file on a single flipped bit, and parity sealed inside the ciphertext can never be reached to repair the thing blocking access to it. Parity repairs the stored bytes first; decryption and authentication then run on data already whole. Name ONE cipher path (gpg and openssl are present on beacon; `age` is not) and specify key handling: no key, passphrase or recipient identity is ever persisted in the project, database, `.nfo`, or repo. 8. **Signing.** Sign `project.tar.sha256` (the inner content digest), so a recovered archive can prove it is authentic and not merely intact. Specify whether the outer container is also signed. 9. **`project.nfo`** -- required fields, populated from real data, never a static template: project name, export timestamp, studio version/commit, payload schema version, chunk size, part count, pinned tool names + exact versions + checksums, and the literal verify / repair / reassemble / decrypt commands. This is what a stranger reads in ten years; a placeholder in it is a defect. 10. **Reassembly is driven by the SFV.** The SFV is the authoritative ordered file list. Note the operational consequence: par2 renames each damaged file to a `.1` backup on repair, so **repair needs disk headroom beyond the package size** (one extra copy of every damaged chunk). Specify whether the tool purges them (`par2 -p`) or cleans up after a verified reassembly. 11. **Edge case: payload smaller than one chunk.** Projects under 1.44MB (real: same_water volumes, jade mask at 1.1MB) produce a single part. The split is a no-op and this MUST be valid, not an error. 12. **Error taxonomy** -- every refusal a conforming unpacker can raise, with stable identifiers. WO 3 consumes this. 13. **Recovery procedure**, written for a human with none of our tooling: the exact ordered commands to verify, repair, reassemble, decrypt and check. 14. **Worked example** -- a minimal complete container. 15. **Conformance checklist**, mechanically checkable. ## Hard constraints - **Use par2, do not implement Reed-Solomon.** par2cmdline is mature and specified; a bespoke codec is a large, subtle, security-relevant undertaking and would be exactly the second definition this fleet keeps paying for. - Every par2 claim in the RFC MUST come from the spec or from measurement, with the source named. Do not restate remembered defaults: par2cmdline 1.2.0 documents `-b` block count default **2000**, `-r` redundancy default **5%**, `-s` block size (mutually exclusive with `-b`), `-n` recovery files (max 31), `-R` recurse. par2 derives block size from block count and never inherits input file sizes; damage cost is per BLOCK, so several corrupt regions in one file repair independently at block cost. - The container is payload-agnostic. It MUST NOT reach into `.ofp3.tgz` internals or reference beat/scene semantics. ## Done means - `docs/RFC-OPALFLAME-ARCHIVE-CONTAINER-V1.md` written, all 15 sections. - Chunk size, redundancy and the single tier recorded WITH their measured evidence and the governing relationship. - Encryption ordering, the no-ciphertext-hash decision, and the SFV-as-list rule each recorded with their reasoning. - Error taxonomy enumerated (WO 3 depends on it). - `.nfo` required-field list complete (WO 5 depends on it). - No code written in this WO -- specification only. - Summary appended, ending with `complete` on its own last line, then shipped. --- ## Worklog (2026-08-06, beacon-83eb) Wrote `docs/RFC-OPALFLAME-ARCHIVE-CONTAINER-V1.md` (1150 lines, 20 sections). All 15 required contents from this WO are covered. Specification only -- no code. ### Deviations from the programme notes, recorded in RFC sec 0 1. **Outer tar is opt-in, not the default** (Davey 2026-08-06). Default packer output is the split/parity DIRECTORY, because that is what commits to git under GitHub's hard 100MB per-file limit with no LFS. `--tar` retained for tape/optical handoff. 2. **Tools are `ofCpack` / `ofCverify`**, not `ofpack` / `ofverify` -- those two names already belong to the FORMAT-STANDARD programme and exist as working tools at `bin/ofpack.py` / `bin/ofverify.py`. 3. **The container carries its own tooling** (Davey 2026-08-06: "include the verify and pack/unpack scripts with the archive as well as a par2.nfo with app definition and dl / source location", because "it will be possible to send the package to someone who has no tooling and they can still read it"). RFC sec 10 is normative: `tools/` holds verbatim `ofCpack.py`, `ofCverify.py`, a `.nfo` sidecar per external program (par2.nfo mandatory) carrying definition + exact version + official source + checksum, and a plain-English README.txt. Third-party tools are LINKED, never redistributed. This imposes a NEW HARD CONSTRAINT ON WO 2 AND WO 3: `ofCpack.py` and `ofCverify.py` MUST each be a single self-contained stdlib-only Python 3 file, or the copy riding inside the container will not run on a stranger's machine. ### Decisions made by this RFC - **One cipher path: GnuPG** (beacon has gpg 2.4.4 + openssl 3.0.13; `age` absent and MUST NOT be introduced). Chosen because the container needs symmetric encryption, recipient encryption AND detached signing from one tool; `openssl enc` gives only the first, `age` cannot sign. - Signing covers `.tar.sha256` (the inner digest), not the tar and not the outer container. - Error taxonomy: 21 stable `E_OFC_*` identifiers (sec 15) -- WO 3 and WO 5 consume these and may not invent others. - `.nfo` required-field list complete (sec 11) -- WO 5 depends on it. ### VERIFIED BY RUNNING IT, not asserted -- and it corrected the spec Installed par2 0.8.1, rhash 1.4.3, cksfv 1.3.15 on beacon and built real containers rather than writing the worked example from the spec. That found a real defect in the programme's invocation: **The spec's `par2 create -r10 -R` is wrong for this container.** par2 derives block SIZE from block COUNT and defaults to `-b2000`; on a small payload that yields ~600-byte blocks whose per-block verification packets (duplicated into every recovery file) dominate the set. Measured on a 1.15MB payload: | invocation | recovery set | % of payload | |---|---|---| | `-r10` (defaults, 8 recovery files) | 1,615,464 B | **137%** | | `-r10 -n1` | 501,040 B | **43%** | | `-r10 -n1 -s32768` | ~145,000 B | **~12.6%** | The programme's 12.9% figure was measured on a 50MB archive and does not generalise downward -- and most of the real corpus is small. RFC sec 12.4 now requires `-n1` and an explicit `-s`, defaulting to **32768 bytes**, chosen because it divides the chunk exactly (1474560 / 32768 = 45 blocks per chunk), so the capacity arithmetic is exact. Added the scaling rule for par2's 32768 source-block ceiling. **Re-verified the settled damage thresholds on beacon under the corrected invocation** (50MB payload -> 36 parts, `-r10 -n1 -s32768`, recovery 10.7%): | damage | outcome | |---|---| | 3 whole chunks deleted | REPAIRED, reassembled sha256 EXACT | | 4 whole chunks deleted | REFUSED, "Repair is not possible." | | 5 whole chunks deleted | REFUSED | | 8 bytes corrupted in each of 12 chunks | REPAIRED, sha256 EXACT | The capacity formula in RFC sec 12.3 predicted the boundary exactly (raw 3.56 -> 3 whole chunks). The settled parameters (1.44MB chunks, `-r10`, one tier) stand; only the par2 invocation changed. **Proved the `.1` backup glob trap is real, not theoretical.** Repaired the 12-chunk-damaged container ("Repair complete."), leaving 12 `.1` backups, then reassembled two ways: | method | result | |---|---| | SFV-driven (`grep -v '^;' ... \| xargs cat`) | 52,439,040 B, sha256 MATCH | | glob (`cat t.part.*`) | 69,491,179 B, CORRUPT | A glob silently produced a payload 17MB too large immediately after a repair that reported total success. RFC sec 13 now carries these numbers. **Other claims verified rather than written from memory:** the literal recovery commands in RFC sec 16 (`rhash --check`, `cksfv -f`, SFV-driven reassembly, `sha256sum -c`) were each run against a real container and pass; rhash and cksfv cross-verified the same 36-part SFV ("Everything OK" from both); the SFV banner and CRC32 case in the worked example (sec 17) are actual rhash output, and rhash emits UPPERCASE CRC32 (the RFC had said lowercase -- corrected, readers MUST compare case-insensitively). **Recorded honestly:** beacon carries par2 0.8.1, not the 1.2.0 the spec measured on haven-4346. Its `--help` lists the same flags but does not print defaults; the `-b2000` default was confirmed on beacon by observation (a default `-r10` run produced exactly 200 recovery blocks = 10% of 2000), not by help text. RFC sec 12.1 states this. ### Prerequisite check (this WO's "CHECK, do not assume") Beacon 2026-08-06: par2, cksfv, rhash, age were all still MISSING as the spec recorded six weeks earlier. Installed par2 0.8.1, rhash 1.4.3, cksfv 1.3.15 to do the verification above. `age` deliberately NOT installed -- the RFC names gpg as the single cipher path. This install is ad-hoc and is NOT the deployment: WO 4 still owns pinning, repeatable provisioning, service-account verification and the QuickSFV round trip. ### Not done here, and why WOs 2-5 are separate WOs by the programme's own design and each ships its own PR. This WO is spec-only per its "no code written in this WO" constraint. The corpus pass (export every project, ofCpack it, land it under `//`) follows WO 5. Note for whoever picks up that migration: beacon is at 93% disk with 2.2G free, and the youtube-channel working tree is 4.8G, so the corpus pass needs the disk situation resolved first. complete