You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
54 lines
4.3 KiB
Markdown
54 lines
4.3 KiB
Markdown
---
|
|
# pages-w7ow
|
|
title: Accept tar.gz, tar.zst, tar.xz archive uploads
|
|
status: completed
|
|
type: feature
|
|
priority: normal
|
|
created_at: 2026-07-21T04:14:51Z
|
|
updated_at: 2026-07-21T04:33:48Z
|
|
---
|
|
|
|
Extend the upload validator (`core::upload`) to accept gzip-, zstd-, and xz-compressed tar archives in addition to the current `.zip` and raw `.html` uploads. Design analysis lives in the 2026-07-20 conversation; summary below.
|
|
|
|
## Hard constraint
|
|
`core::upload` compiles to `wasm32-unknown-unknown` and runs single-threaded in a Cloudflare Worker. Every decompressor MUST be pure Rust (no `*-sys` C bindings, no threads). Verified pure-Rust crates:
|
|
|
|
- gzip: `flate2` with `default-features = false, features = ["rust_backend"]` (miniz_oxide) — streaming `GzDecoder: Read`
|
|
- zstd: `ruzstd` 0.8 — streaming `StreamingDecoder: Read`, no_std-capable
|
|
- xz: `lzma-rs` 0.3 — pure Rust but ONE-SHOT (`xz_decompress(&mut input, &mut output)`), not streaming
|
|
- tar: `tar` 0.4 crate (reads entries from any `Read` without touching the fs). RISK: pulls in `filetime`/`xattr` which may not compile on wasm32-unknown-unknown. If it fails to compile, hand-roll a minimal ustar header parser (512-byte headers; name@0, octal size@124, "ustar" magic@257) — the test module already hand-rolls the zip byte format, same approach.
|
|
|
|
bzip2 is intentionally OUT OF SCOPE (only unmaintained pure-Rust decoder available; almost nobody uploads .tar.bz2).
|
|
|
|
## Zip-bomb defense must be preserved
|
|
The existing `read_entry_limited` streams each entry and counts decompressed bytes, aborting the instant `max_file` or `max_uncompressed_total` is crossed — never trusting declared sizes. This extends to tar because a tar entry is also an `io::Read`.
|
|
- Streaming formats (gz, zst): wrap raw bytes -> decoder -> `tar::Archive` -> iterate; each entry Read flows through `read_entry_limited` unchanged.
|
|
- One-shot xz: wrap the output sink in a `LimitedWriter` that errors past `max_uncompressed_total` so an xz bomb aborts mid-decompress and never materializes; then tar-parse the bounded buffer.
|
|
|
|
## Format detection = magic-byte sniffing (do NOT trust Content-Type)
|
|
Matches the module's "never trust the archive" posture. Sniff the buffered body:
|
|
- gzip: `1f 8b`
|
|
- xz: `fd 37 7a 58 5a 00`
|
|
- zstd: `28 b5 2f fd`
|
|
- zip: `50 4b` (PK) — existing
|
|
- tar (plain, optional): "ustar" at offset 257
|
|
Keep Content-Type as a fallback hint only; the bytes win. `routes::api::validate_upload` currently maps Content-Type -> UploadKind around api.rs:100-119 — replace/augment with sniffing.
|
|
|
|
## Shared code changes
|
|
- `core/upload.rs`: extend `UploadKind` (e.g. add a `Tar { compression: Compression }` kind or per-format variants), add `build_tar_manifest` mirroring `build_zip_manifest` (reuse `check_entry_path`, `seen` dedup, `read_entry_limited`, root-level `index.html` requirement, all limits), add a `LimitedWriter`, add `UploadError` variants (`InvalidTar`, per-format decompression errors).
|
|
- `routes/api.rs`: switch dispatch to magic-byte sniffing; wire new error paths.
|
|
- `Cargo.toml`: add flate2/ruzstd/lzma-rs/tar to `[dependencies]` (NOT the wasm-gated table — the logic module compiles on both native and wasm).
|
|
|
|
## Validation (run from pages/, must all pass)
|
|
cargo fmt; cargo check --target wasm32-unknown-unknown; cargo check; cargo clippy --all-targets -- -D warnings; cargo clippy --target wasm32-unknown-unknown -- -D warnings; cargo test
|
|
|
|
## Tasks
|
|
- [x] Shared foundation: magic-byte sniffing, UploadKind refactor, tar parsing, LimitedWriter, build_tar_manifest, error variants
|
|
- [x] gzip (.tar.gz / .tgz) support + tests
|
|
- [x] zstd (.tar.zst) support + tests
|
|
- [x] xz (.tar.xz) support + tests (LimitedWriter-bounded)
|
|
- [x] All validation commands pass (native + wasm)
|
|
|
|
## Summary of Changes
|
|
Accept gzip/zstd/xz-compressed tar uploads alongside .html and .zip. Dispatch is now magic-byte sniffing (detect_kind), not Content-Type. New build_tar_manifest reuses every zip-path structural/size rule; the streaming zip-bomb guard is preserved end-to-end (lazy for gzip/zstd; LimitedWriter-bounded for one-shot xz). All decoders are pure Rust (flate2 rust_backend, ruzstd, lzma-rs) and the tar crate compiled cleanly for wasm32-unknown-unknown. bzip2 intentionally out of scope. All six validation commands green; 186 unit + 20 doctests pass (~34 new). Files: Cargo.toml, src/core/upload.rs, src/routes/api.rs, static/index.html.
|