Extend the upload validator to accept .tar.gz/.tgz, .tar.zst, and
.tar.xz archives alongside raw .html and .zip. Upload-kind dispatch is
now magic-byte sniffing (detect_kind) rather than trusting Content-Type,
matching the module's never-trust-the-archive posture.
build_tar_manifest reuses every zip-path structural and size rule
(path-traversal checks, dedup, entry cap, root index.html requirement).
The streaming zip-bomb defense is preserved end-to-end: gzip/zstd decode
lazily as each tar entry is read, and one-shot xz decompresses into a
size-bounded LimitedWriter so a bomb aborts mid-decompress. All decoders
are pure Rust (flate2 rust_backend, ruzstd, lzma-rs) and the tar crate
compiles cleanly for wasm32-unknown-unknown; bzip2 is intentionally out
of scope.
Frontend accept list and Content-Type helper updated; the backend now
sniffs the real format so archives are sent as application/octet-stream.
Beans: pages-w7ow (feature) + pages-r4ob/g39t/wadp/u3zc (tasks).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire gzip-compressed tar archives through the dispatch foundation.
- Crate: `flate2 = { version = \"1\", default-features = false, features = [\"rust_backend\"] }` in `[dependencies]` (pure-Rust miniz_oxide backend; wasm-safe).
- Streaming path: raw bytes -> `flate2::read::GzDecoder` (or `MultiGzDecoder`) -> `build_tar_manifest`. Each tar entry Read flows through `read_entry_limited` unchanged.
- Detect via gzip magic `1f 8b`.
- Tests (mirror the zip tests in core/upload.rs): happy multi-file-with-subdirs, missing index.html, path traversal, per-file/total limits, and a gzip tar-bomb rejected by the streaming guard. Build fixtures in-memory with the `flate2`+`tar` writers.
## Summary of Changes
gzip via `flate2` (rust_backend/miniz_oxide). Streaming path: bytes -> GzDecoder -> read_tar_entries; each tar entry flows through read_entry_limited so the bomb guard is preserved. Tests: happy multi-file+subdirs, missing index, traversal, per-file/total limits, streaming bomb.
title: 'Upload dispatch foundation: magic-byte sniffing + tar pipeline'
status: completed
type: task
priority: high
created_at: 2026-07-21T04:15:03Z
updated_at: 2026-07-21T04:33:38Z
parent: pages-w7ow
---
Shared groundwork all three compressed-tar formats depend on. Do this FIRST.
- Replace Content-Type dispatch in `routes::api::validate_upload` (api.rs ~100-119) with magic-byte sniffing on the buffered body. Magic: gzip `1f 8b`, xz `fd 37 7a 58 5a 00`, zstd `28 b5 2f fd`, zip `50 4b`. Content-Type is a fallback hint only.
- Extend `UploadKind` in `core/upload.rs` to carry compressed-tar variants (suggest a `Compression` enum: Gzip/Zstd/Xz + a `Tar` kind).
- Add `build_tar_manifest(bytes, compression, limits)` mirroring `build_zip_manifest`: reuse `check_entry_path`, the `seen`/DuplicateEntry dedup, `read_entry_limited`, the root-level `index.html` requirement, `max_entries`, and all size limits. Skip tar directory/non-regular entries.
- Add a `LimitedWriter` (impl std::io::Write) that errors once cumulative bytes exceed `max_uncompressed_total` — used by the one-shot xz path.
- Add `UploadError` variants: `InvalidTar` and per-format decompression errors (or one `DecompressionFailed { format }`).
- tar reading: try the `tar` 0.4 crate first (`Archive::new(reader).entries()`, read each entry as a Read, no fs access). If it will not compile on wasm32-unknown-unknown (filetime/xattr), hand-roll a minimal ustar parser instead.
Preserve the zip-bomb guarantee: never trust declared sizes; count streamed decompressed bytes and abort on limit crossing.
## Summary of Changes
Added `detect_kind(bytes, content_type_hint) -> UploadKind` (magic-byte sniffing: PK/gzip/xz/zstd, else Html) and wired `routes::api::manifest_from_request` to it (dropped the Content-Type mapping and the 'unsupported content type' 400). Added `Compression` enum, `UploadKind::Tar(Compression)`, `build_tar_manifest`, `read_tar_entries` (mirrors build_zip_manifest: check_entry_path, seen-dedup, read_entry_limited, max_entries, root index.html), `LimitedWriter`, and `UploadError::{InvalidTar, DecompressionFailed}`. The `tar` 0.4 crate compiled cleanly for wasm32-unknown-unknown, so no hand-rolled ustar parser was needed in the production path.
- lzma-rs is ONE-SHOT: `xz_decompress<R: BufRead, W: Write>(&mut input, &mut output)`. Decompress into a `LimitedWriter` bounded by `max_uncompressed_total` so an xz bomb aborts mid-decompress and never fully materializes. Then feed the bounded buffer to `build_tar_manifest`.
- Detect via xz magic `fd 37 7a 58 5a 00`.
- Tests: happy path, missing index.html, path traversal, size limits, and an xz bomb rejected by the LimitedWriter bound (a small .xz whose decompressed tar exceeds max_uncompressed_total must be rejected before full materialization). Note xz has the largest wasm code-size/CPU cost of the three.
## Summary of Changes
xz via `lzma-rs` 0.3 (one-shot). Decompresses into a LimitedWriter bounded by xz_materialize_cap (= max_uncompressed_total + bounded tar framing) so an xz bomb aborts mid-decompress; bounded buffer then tar-parsed. Fixtures via lzma-rs xz_compress. Tests: happy, missing index, traversal, limits, LimitedWriter bomb rejection.
Extend the upload validator (`core::upload`) to accept gzip-, zstd-, and xz-compressed tar archives in addition to the current `.zip` and raw `.html` uploads. Design analysis lives in the 2026-07-20 conversation; summary below.
## Hard constraint
`core::upload` compiles to `wasm32-unknown-unknown` and runs single-threaded in a Cloudflare Worker. Every decompressor MUST be pure Rust (no `*-sys` C bindings, no threads). Verified pure-Rust crates:
- gzip: `flate2` with `default-features = false, features = ["rust_backend"]` (miniz_oxide) — streaming `GzDecoder: Read`
- xz: `lzma-rs` 0.3 — pure Rust but ONE-SHOT (`xz_decompress(&mut input, &mut output)`), not streaming
- tar: `tar` 0.4 crate (reads entries from any `Read` without touching the fs). RISK: pulls in `filetime`/`xattr` which may not compile on wasm32-unknown-unknown. If it fails to compile, hand-roll a minimal ustar header parser (512-byte headers; name@0, octal size@124, "ustar" magic@257) — the test module already hand-rolls the zip byte format, same approach.
bzip2 is intentionally OUT OF SCOPE (only unmaintained pure-Rust decoder available; almost nobody uploads .tar.bz2).
## Zip-bomb defense must be preserved
The existing `read_entry_limited` streams each entry and counts decompressed bytes, aborting the instant `max_file` or `max_uncompressed_total` is crossed — never trusting declared sizes. This extends to tar because a tar entry is also an `io::Read`.
- Streaming formats (gz, zst): wrap raw bytes -> decoder -> `tar::Archive` -> iterate; each entry Read flows through `read_entry_limited` unchanged.
- One-shot xz: wrap the output sink in a `LimitedWriter` that errors past `max_uncompressed_total` so an xz bomb aborts mid-decompress and never materializes; then tar-parse the bounded buffer.
## Format detection = magic-byte sniffing (do NOT trust Content-Type)
Matches the module's "never trust the archive" posture. Sniff the buffered body:
- gzip: `1f 8b`
- xz: `fd 37 7a 58 5a 00`
- zstd: `28 b5 2f fd`
- zip: `50 4b` (PK) — existing
- tar (plain, optional): "ustar" at offset 257
Keep Content-Type as a fallback hint only; the bytes win. `routes::api::validate_upload` currently maps Content-Type -> UploadKind around api.rs:100-119 — replace/augment with sniffing.
## Shared code changes
- `core/upload.rs`: extend `UploadKind` (e.g. add a `Tar { compression: Compression }` kind or per-format variants), add `build_tar_manifest` mirroring `build_zip_manifest` (reuse `check_entry_path`, `seen` dedup, `read_entry_limited`, root-level `index.html` requirement, all limits), add a `LimitedWriter`, add `UploadError` variants (`InvalidTar`, per-format decompression errors).
- `routes/api.rs`: switch dispatch to magic-byte sniffing; wire new error paths.
- `Cargo.toml`: add flate2/ruzstd/lzma-rs/tar to `[dependencies]` (NOT the wasm-gated table — the logic module compiles on both native and wasm).
- [x] xz (.tar.xz) support + tests (LimitedWriter-bounded)
- [x] All validation commands pass (native + wasm)
## Summary of Changes
Accept gzip/zstd/xz-compressed tar uploads alongside .html and .zip. Dispatch is now magic-byte sniffing (detect_kind), not Content-Type. New build_tar_manifest reuses every zip-path structural/size rule; the streaming zip-bomb guard is preserved end-to-end (lazy for gzip/zstd; LimitedWriter-bounded for one-shot xz). All decoders are pure Rust (flate2 rust_backend, ruzstd, lzma-rs) and the tar crate compiled cleanly for wasm32-unknown-unknown. bzip2 intentionally out of scope. All six validation commands green; 186 unit + 20 doctests pass (~34 new). Files: Cargo.toml, src/core/upload.rs, src/routes/api.rs, static/index.html.
- Streaming path: raw bytes -> `ruzstd::StreamingDecoder` (implements io::Read) -> `build_tar_manifest`. Each tar entry Read flows through `read_entry_limited` unchanged.
- Detect via zstd magic `28 b5 2f fd`.
- Tests: happy path, missing index.html, path traversal, size limits, zstd tar-bomb rejected by the streaming guard. `ruzstd` is decode-focused — build .tar.zst fixtures with a real zstd encoder available in the dev shell, or vendor a small precompressed fixture; prefer generating in-test if an encoder crate is acceptable (else document the fixture).
## Summary of Changes
zstd via `ruzstd` 0.8 StreamingDecoder (pure Rust). Streaming path preserves the bomb guard. Fixtures built with ruzstd's own encoder (no C dev-dep). Tests: happy, missing index, traversal, limits, streaming bomb, corrupt-frame.
<divid="dropzone-label">Drag & drop an <code>.html</code> file or a <code>.zip</code> of static assets here, or click to choose one.</div>
<divid="dropzone-label">Drag & drop an <code>.html</code> file, a <code>.zip</code>, or a compressed tarball (<code>.tar.gz</code>, <code>.tar.zst</code>, <code>.tar.xz</code>) of static assets here, or click to choose one.</div>
dropzoneLabel.textContent = "Drop the new .html file or .zip here, or click to choose one. It replaces this page's current content.";
dropzoneLabel.textContent = "Drop the new .html, .zip, or compressed tarball (.tar.gz, .tar.zst, .tar.xz) here, or click to choose one. It replaces this page's current content.";
nameInput.value = slug;
nameInput.readOnly = true;
uploadSubmit.textContent = "Update";
} else {
uploadModalTitle.textContent = "Upload a page";
dropzoneLabel.textContent = "Drag & drop an .html file or a .zip of static assets here, or click to choose one.";
dropzoneLabel.textContent = "Drag & drop an .html file, a .zip, or a compressed tarball (.tar.gz, .tar.zst, .tar.xz) of static assets here, or click to choose one.";
nameInput.value = "";
nameInput.readOnly = false;
uploadSubmit.textContent = "Upload";
@ -768,13 +768,26 @@
validateNameField();
}
var ALLOWED_EXTENSIONS = [".html", ".htm", ".zip", ".tar.gz", ".tgz", ".tar.zst", ".tar.xz"];
Elijah Voigt
2 days ago