vibed/pages/.beans/pages-w7ow--accept-targz-tarzst-tarxz-archive-uploads.md

title	status	type	priority	created_at	updated_at
Accept tar.gz, tar.zst, tar.xz archive uploads	completed	feature	normal	2026-07-21T04:14:51Z	2026-07-21T04:33:48Z

Extend the upload validator (core::upload) to accept gzip-, zstd-, and xz-compressed tar archives in addition to the current .zip and raw .html uploads. Design analysis lives in the 2026-07-20 conversation; summary below.

Hard constraint

core::upload compiles to wasm32-unknown-unknown and runs single-threaded in a Cloudflare Worker. Every decompressor MUST be pure Rust (no *-sys C bindings, no threads). Verified pure-Rust crates:

• gzip: flate2 with default-features = false, features = ["rust_backend"] (miniz_oxide) — streaming GzDecoder: Read
• zstd: ruzstd 0.8 — streaming StreamingDecoder: Read, no_std-capable
• xz: lzma-rs 0.3 — pure Rust but ONE-SHOT (xz_decompress(&mut input, &mut output)), not streaming
• tar: tar 0.4 crate (reads entries from any Read without touching the fs). RISK: pulls in filetime/xattr which may not compile on wasm32-unknown-unknown. If it fails to compile, hand-roll a minimal ustar header parser (512-byte headers; name@0, octal size@124, "ustar" magic@257) — the test module already hand-rolls the zip byte format, same approach.

bzip2 is intentionally OUT OF SCOPE (only unmaintained pure-Rust decoder available; almost nobody uploads .tar.bz2).

Zip-bomb defense must be preserved

The existing read_entry_limited streams each entry and counts decompressed bytes, aborting the instant max_file or max_uncompressed_total is crossed — never trusting declared sizes. This extends to tar because a tar entry is also an io::Read.

• Streaming formats (gz, zst): wrap raw bytes -> decoder -> tar::Archive -> iterate; each entry Read flows through read_entry_limited unchanged.
• One-shot xz: wrap the output sink in a LimitedWriter that errors past max_uncompressed_total so an xz bomb aborts mid-decompress and never materializes; then tar-parse the bounded buffer.

Format detection = magic-byte sniffing (do NOT trust Content-Type)

Matches the module's "never trust the archive" posture. Sniff the buffered body:

• gzip: 1f 8b
• xz: fd 37 7a 58 5a 00
• zstd: 28 b5 2f fd
• zip: 50 4b (PK) — existing
• tar (plain, optional): "ustar" at offset 257 Keep Content-Type as a fallback hint only; the bytes win. routes::api::validate_upload currently maps Content-Type -> UploadKind around api.rs:100-119 — replace/augment with sniffing.

Shared code changes

• core/upload.rs: extend UploadKind (e.g. add a Tar { compression: Compression } kind or per-format variants), add build_tar_manifest mirroring build_zip_manifest (reuse check_entry_path, seen dedup, read_entry_limited, root-level index.html requirement, all limits), add a LimitedWriter, add UploadError variants (InvalidTar, per-format decompression errors).
• routes/api.rs: switch dispatch to magic-byte sniffing; wire new error paths.
• Cargo.toml: add flate2/ruzstd/lzma-rs/tar to [dependencies] (NOT the wasm-gated table — the logic module compiles on both native and wasm).

Validation (run from pages/, must all pass)

cargo fmt; cargo check --target wasm32-unknown-unknown; cargo check; cargo clippy --all-targets -- -D warnings; cargo clippy --target wasm32-unknown-unknown -- -D warnings; cargo test

Tasks

• Shared foundation: magic-byte sniffing, UploadKind refactor, tar parsing, LimitedWriter, build_tar_manifest, error variants
• gzip (.tar.gz / .tgz) support + tests
• zstd (.tar.zst) support + tests
• xz (.tar.xz) support + tests (LimitedWriter-bounded)
• All validation commands pass (native + wasm)

Summary of Changes

Accept gzip/zstd/xz-compressed tar uploads alongside .html and .zip. Dispatch is now magic-byte sniffing (detect_kind), not Content-Type. New build_tar_manifest reuses every zip-path structural/size rule; the streaming zip-bomb guard is preserved end-to-end (lazy for gzip/zstd; LimitedWriter-bounded for one-shot xz). All decoders are pure Rust (flate2 rust_backend, ruzstd, lzma-rs) and the tar crate compiled cleanly for wasm32-unknown-unknown. bzip2 intentionally out of scope. All six validation commands green; 186 unit + 20 doctests pass (~34 new). Files: Cargo.toml, src/core/upload.rs, src/routes/api.rs, static/index.html.