From 0aa6a6862e7602d41491f413e06a10a8ec087dff Mon Sep 17 00:00:00 2001 From: Elijah Voigt Date: Mon, 20 Jul 2026 21:34:00 -0700 Subject: [PATCH] feat(pages): accept gzip/zstd/xz compressed tar uploads Extend the upload validator to accept .tar.gz/.tgz, .tar.zst, and .tar.xz archives alongside raw .html and .zip. Upload-kind dispatch is now magic-byte sniffing (detect_kind) rather than trusting Content-Type, matching the module's never-trust-the-archive posture. build_tar_manifest reuses every zip-path structural and size rule (path-traversal checks, dedup, entry cap, root index.html requirement). The streaming zip-bomb defense is preserved end-to-end: gzip/zstd decode lazily as each tar entry is read, and one-shot xz decompresses into a size-bounded LimitedWriter so a bomb aborts mid-decompress. All decoders are pure Rust (flate2 rust_backend, ruzstd, lzma-rs) and the tar crate compiles cleanly for wasm32-unknown-unknown; bzip2 is intentionally out of scope. Frontend accept list and Content-Type helper updated; the backend now sniffs the real format so archives are sent as application/octet-stream. Beans: pages-w7ow (feature) + pages-r4ob/g39t/wadp/u3zc (tasks). Co-Authored-By: Claude Opus 4.8 --- ...es-g39t--support-targz-tgz-uploads-gzip.md | 20 + ...atch-foundation-magic-byte-sniffing-tar.md | 24 + ...ges-u3zc--support-tarxz-uploads-xzlzma2.md | 20 + ...cept-targz-tarzst-tarxz-archive-uploads.md | 53 ++ ...pages-wadp--support-tarzst-uploads-zstd.md | 20 + pages/Cargo.lock | 131 +++ pages/Cargo.toml | 11 + pages/src/core/upload.rs | 852 +++++++++++++++++- pages/src/routes/api.rs | 30 +- pages/static/index.html | 25 +- 10 files changed, 1158 insertions(+), 28 deletions(-) create mode 100644 pages/.beans/pages-g39t--support-targz-tgz-uploads-gzip.md create mode 100644 pages/.beans/pages-r4ob--upload-dispatch-foundation-magic-byte-sniffing-tar.md create mode 100644 pages/.beans/pages-u3zc--support-tarxz-uploads-xzlzma2.md create mode 100644 pages/.beans/pages-w7ow--accept-targz-tarzst-tarxz-archive-uploads.md create mode 100644 pages/.beans/pages-wadp--support-tarzst-uploads-zstd.md diff --git a/pages/.beans/pages-g39t--support-targz-tgz-uploads-gzip.md b/pages/.beans/pages-g39t--support-targz-tgz-uploads-gzip.md new file mode 100644 index 0000000..a211881 --- /dev/null +++ b/pages/.beans/pages-g39t--support-targz-tgz-uploads-gzip.md @@ -0,0 +1,20 @@ +--- +# pages-g39t +title: Support .tar.gz / .tgz uploads (gzip) +status: completed +type: task +priority: normal +created_at: 2026-07-21T04:15:09Z +updated_at: 2026-07-21T04:33:38Z +parent: pages-w7ow +--- + +Wire gzip-compressed tar archives through the dispatch foundation. + +- Crate: `flate2 = { version = \"1\", default-features = false, features = [\"rust_backend\"] }` in `[dependencies]` (pure-Rust miniz_oxide backend; wasm-safe). +- Streaming path: raw bytes -> `flate2::read::GzDecoder` (or `MultiGzDecoder`) -> `build_tar_manifest`. Each tar entry Read flows through `read_entry_limited` unchanged. +- Detect via gzip magic `1f 8b`. +- Tests (mirror the zip tests in core/upload.rs): happy multi-file-with-subdirs, missing index.html, path traversal, per-file/total limits, and a gzip tar-bomb rejected by the streaming guard. Build fixtures in-memory with the `flate2`+`tar` writers. + +## Summary of Changes +gzip via `flate2` (rust_backend/miniz_oxide). Streaming path: bytes -> GzDecoder -> read_tar_entries; each tar entry flows through read_entry_limited so the bomb guard is preserved. Tests: happy multi-file+subdirs, missing index, traversal, per-file/total limits, streaming bomb. diff --git a/pages/.beans/pages-r4ob--upload-dispatch-foundation-magic-byte-sniffing-tar.md b/pages/.beans/pages-r4ob--upload-dispatch-foundation-magic-byte-sniffing-tar.md new file mode 100644 index 0000000..8b676f4 --- /dev/null +++ b/pages/.beans/pages-r4ob--upload-dispatch-foundation-magic-byte-sniffing-tar.md @@ -0,0 +1,24 @@ +--- +# pages-r4ob +title: 'Upload dispatch foundation: magic-byte sniffing + tar pipeline' +status: completed +type: task +priority: high +created_at: 2026-07-21T04:15:03Z +updated_at: 2026-07-21T04:33:38Z +parent: pages-w7ow +--- + +Shared groundwork all three compressed-tar formats depend on. Do this FIRST. + +- Replace Content-Type dispatch in `routes::api::validate_upload` (api.rs ~100-119) with magic-byte sniffing on the buffered body. Magic: gzip `1f 8b`, xz `fd 37 7a 58 5a 00`, zstd `28 b5 2f fd`, zip `50 4b`. Content-Type is a fallback hint only. +- Extend `UploadKind` in `core/upload.rs` to carry compressed-tar variants (suggest a `Compression` enum: Gzip/Zstd/Xz + a `Tar` kind). +- Add `build_tar_manifest(bytes, compression, limits)` mirroring `build_zip_manifest`: reuse `check_entry_path`, the `seen`/DuplicateEntry dedup, `read_entry_limited`, the root-level `index.html` requirement, `max_entries`, and all size limits. Skip tar directory/non-regular entries. +- Add a `LimitedWriter` (impl std::io::Write) that errors once cumulative bytes exceed `max_uncompressed_total` — used by the one-shot xz path. +- Add `UploadError` variants: `InvalidTar` and per-format decompression errors (or one `DecompressionFailed { format }`). +- tar reading: try the `tar` 0.4 crate first (`Archive::new(reader).entries()`, read each entry as a Read, no fs access). If it will not compile on wasm32-unknown-unknown (filetime/xattr), hand-roll a minimal ustar parser instead. + +Preserve the zip-bomb guarantee: never trust declared sizes; count streamed decompressed bytes and abort on limit crossing. + +## Summary of Changes +Added `detect_kind(bytes, content_type_hint) -> UploadKind` (magic-byte sniffing: PK/gzip/xz/zstd, else Html) and wired `routes::api::manifest_from_request` to it (dropped the Content-Type mapping and the 'unsupported content type' 400). Added `Compression` enum, `UploadKind::Tar(Compression)`, `build_tar_manifest`, `read_tar_entries` (mirrors build_zip_manifest: check_entry_path, seen-dedup, read_entry_limited, max_entries, root index.html), `LimitedWriter`, and `UploadError::{InvalidTar, DecompressionFailed}`. The `tar` 0.4 crate compiled cleanly for wasm32-unknown-unknown, so no hand-rolled ustar parser was needed in the production path. diff --git a/pages/.beans/pages-u3zc--support-tarxz-uploads-xzlzma2.md b/pages/.beans/pages-u3zc--support-tarxz-uploads-xzlzma2.md new file mode 100644 index 0000000..ad36167 --- /dev/null +++ b/pages/.beans/pages-u3zc--support-tarxz-uploads-xzlzma2.md @@ -0,0 +1,20 @@ +--- +# pages-u3zc +title: Support .tar.xz uploads (xz/LZMA2) +status: completed +type: task +priority: normal +created_at: 2026-07-21T04:15:18Z +updated_at: 2026-07-21T04:33:38Z +parent: pages-w7ow +--- + +Wire xz-compressed tar archives through the dispatch foundation. + +- Crate: `lzma-rs = \"0.3\"` in `[dependencies]` (pure-Rust xz/lzma2 decoder; wasm-safe). +- lzma-rs is ONE-SHOT: `xz_decompress(&mut input, &mut output)`. Decompress into a `LimitedWriter` bounded by `max_uncompressed_total` so an xz bomb aborts mid-decompress and never fully materializes. Then feed the bounded buffer to `build_tar_manifest`. +- Detect via xz magic `fd 37 7a 58 5a 00`. +- Tests: happy path, missing index.html, path traversal, size limits, and an xz bomb rejected by the LimitedWriter bound (a small .xz whose decompressed tar exceeds max_uncompressed_total must be rejected before full materialization). Note xz has the largest wasm code-size/CPU cost of the three. + +## Summary of Changes +xz via `lzma-rs` 0.3 (one-shot). Decompresses into a LimitedWriter bounded by xz_materialize_cap (= max_uncompressed_total + bounded tar framing) so an xz bomb aborts mid-decompress; bounded buffer then tar-parsed. Fixtures via lzma-rs xz_compress. Tests: happy, missing index, traversal, limits, LimitedWriter bomb rejection. diff --git a/pages/.beans/pages-w7ow--accept-targz-tarzst-tarxz-archive-uploads.md b/pages/.beans/pages-w7ow--accept-targz-tarzst-tarxz-archive-uploads.md new file mode 100644 index 0000000..ee7cbb2 --- /dev/null +++ b/pages/.beans/pages-w7ow--accept-targz-tarzst-tarxz-archive-uploads.md @@ -0,0 +1,53 @@ +--- +# pages-w7ow +title: Accept tar.gz, tar.zst, tar.xz archive uploads +status: completed +type: feature +priority: normal +created_at: 2026-07-21T04:14:51Z +updated_at: 2026-07-21T04:33:48Z +--- + +Extend the upload validator (`core::upload`) to accept gzip-, zstd-, and xz-compressed tar archives in addition to the current `.zip` and raw `.html` uploads. Design analysis lives in the 2026-07-20 conversation; summary below. + +## Hard constraint +`core::upload` compiles to `wasm32-unknown-unknown` and runs single-threaded in a Cloudflare Worker. Every decompressor MUST be pure Rust (no `*-sys` C bindings, no threads). Verified pure-Rust crates: + +- gzip: `flate2` with `default-features = false, features = ["rust_backend"]` (miniz_oxide) — streaming `GzDecoder: Read` +- zstd: `ruzstd` 0.8 — streaming `StreamingDecoder: Read`, no_std-capable +- xz: `lzma-rs` 0.3 — pure Rust but ONE-SHOT (`xz_decompress(&mut input, &mut output)`), not streaming +- tar: `tar` 0.4 crate (reads entries from any `Read` without touching the fs). RISK: pulls in `filetime`/`xattr` which may not compile on wasm32-unknown-unknown. If it fails to compile, hand-roll a minimal ustar header parser (512-byte headers; name@0, octal size@124, "ustar" magic@257) — the test module already hand-rolls the zip byte format, same approach. + +bzip2 is intentionally OUT OF SCOPE (only unmaintained pure-Rust decoder available; almost nobody uploads .tar.bz2). + +## Zip-bomb defense must be preserved +The existing `read_entry_limited` streams each entry and counts decompressed bytes, aborting the instant `max_file` or `max_uncompressed_total` is crossed — never trusting declared sizes. This extends to tar because a tar entry is also an `io::Read`. +- Streaming formats (gz, zst): wrap raw bytes -> decoder -> `tar::Archive` -> iterate; each entry Read flows through `read_entry_limited` unchanged. +- One-shot xz: wrap the output sink in a `LimitedWriter` that errors past `max_uncompressed_total` so an xz bomb aborts mid-decompress and never materializes; then tar-parse the bounded buffer. + +## Format detection = magic-byte sniffing (do NOT trust Content-Type) +Matches the module's "never trust the archive" posture. Sniff the buffered body: +- gzip: `1f 8b` +- xz: `fd 37 7a 58 5a 00` +- zstd: `28 b5 2f fd` +- zip: `50 4b` (PK) — existing +- tar (plain, optional): "ustar" at offset 257 +Keep Content-Type as a fallback hint only; the bytes win. `routes::api::validate_upload` currently maps Content-Type -> UploadKind around api.rs:100-119 — replace/augment with sniffing. + +## Shared code changes +- `core/upload.rs`: extend `UploadKind` (e.g. add a `Tar { compression: Compression }` kind or per-format variants), add `build_tar_manifest` mirroring `build_zip_manifest` (reuse `check_entry_path`, `seen` dedup, `read_entry_limited`, root-level `index.html` requirement, all limits), add a `LimitedWriter`, add `UploadError` variants (`InvalidTar`, per-format decompression errors). +- `routes/api.rs`: switch dispatch to magic-byte sniffing; wire new error paths. +- `Cargo.toml`: add flate2/ruzstd/lzma-rs/tar to `[dependencies]` (NOT the wasm-gated table — the logic module compiles on both native and wasm). + +## Validation (run from pages/, must all pass) +cargo fmt; cargo check --target wasm32-unknown-unknown; cargo check; cargo clippy --all-targets -- -D warnings; cargo clippy --target wasm32-unknown-unknown -- -D warnings; cargo test + +## Tasks +- [x] Shared foundation: magic-byte sniffing, UploadKind refactor, tar parsing, LimitedWriter, build_tar_manifest, error variants +- [x] gzip (.tar.gz / .tgz) support + tests +- [x] zstd (.tar.zst) support + tests +- [x] xz (.tar.xz) support + tests (LimitedWriter-bounded) +- [x] All validation commands pass (native + wasm) + +## Summary of Changes +Accept gzip/zstd/xz-compressed tar uploads alongside .html and .zip. Dispatch is now magic-byte sniffing (detect_kind), not Content-Type. New build_tar_manifest reuses every zip-path structural/size rule; the streaming zip-bomb guard is preserved end-to-end (lazy for gzip/zstd; LimitedWriter-bounded for one-shot xz). All decoders are pure Rust (flate2 rust_backend, ruzstd, lzma-rs) and the tar crate compiled cleanly for wasm32-unknown-unknown. bzip2 intentionally out of scope. All six validation commands green; 186 unit + 20 doctests pass (~34 new). Files: Cargo.toml, src/core/upload.rs, src/routes/api.rs, static/index.html. diff --git a/pages/.beans/pages-wadp--support-tarzst-uploads-zstd.md b/pages/.beans/pages-wadp--support-tarzst-uploads-zstd.md new file mode 100644 index 0000000..c6e02b7 --- /dev/null +++ b/pages/.beans/pages-wadp--support-tarzst-uploads-zstd.md @@ -0,0 +1,20 @@ +--- +# pages-wadp +title: Support .tar.zst uploads (zstd) +status: completed +type: task +priority: normal +created_at: 2026-07-21T04:15:13Z +updated_at: 2026-07-21T04:33:38Z +parent: pages-w7ow +--- + +Wire zstd-compressed tar archives through the dispatch foundation. + +- Crate: `ruzstd = \"0.8\"` in `[dependencies]` (pure-Rust, no_std-capable zstd decoder; wasm-safe). +- Streaming path: raw bytes -> `ruzstd::StreamingDecoder` (implements io::Read) -> `build_tar_manifest`. Each tar entry Read flows through `read_entry_limited` unchanged. +- Detect via zstd magic `28 b5 2f fd`. +- Tests: happy path, missing index.html, path traversal, size limits, zstd tar-bomb rejected by the streaming guard. `ruzstd` is decode-focused — build .tar.zst fixtures with a real zstd encoder available in the dev shell, or vendor a small precompressed fixture; prefer generating in-test if an encoder crate is acceptable (else document the fixture). + +## Summary of Changes +zstd via `ruzstd` 0.8 StreamingDecoder (pure Rust). Streaming path preserves the bomb guard. Fixtures built with ruzstd's own encoder (no C dev-dep). Tests: happy, missing index, traversal, limits, streaming bomb, corrupt-frame. diff --git a/pages/Cargo.lock b/pages/Cargo.lock index 8222108..ba99a98 100644 --- a/pages/Cargo.lock +++ b/pages/Cargo.lock @@ -81,6 +81,12 @@ dependencies = [ "tower-service", ] +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + [[package]] name = "block-buffer" version = "0.10.4" @@ -96,6 +102,12 @@ version = "3.20.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + [[package]] name = "bytes" version = "1.12.1" @@ -128,6 +140,21 @@ dependencies = [ "libc", ] +[[package]] +name = "crc" +version = "3.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5eb8a2a1cd12ab0d987a5d5e825195d372001a4094a0376319d5a0ad71c1ba0d" +dependencies = [ + "crc-catalog", +] + +[[package]] +name = "crc-catalog" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "217698eaf96b4a3f0bc4f3662aaa55bdf913cd54d7204591faa790070c6d0853" + [[package]] name = "crc32fast" version = "1.5.0" @@ -191,6 +218,26 @@ version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "filetime" +version = "0.2.29" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c287a33c7f0a620c38e641e7f60827713987b3c0f26e8ddc9462cc69cf75759" +dependencies = [ + "cfg-if", + "libc", +] + [[package]] name = "flate2" version = "1.1.9" @@ -468,6 +515,12 @@ version = "0.2.186" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + [[package]] name = "litemap" version = "0.8.2" @@ -480,6 +533,16 @@ version = "0.4.33" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" +[[package]] +name = "lzma-rs" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "297e814c836ae64db86b36cf2a557ba54368d03f6afcd7d947c266692f71115e" +dependencies = [ + "byteorder", + "crc", +] + [[package]] name = "matchit" version = "0.7.3" @@ -534,11 +597,15 @@ name = "pages" version = "0.1.0" dependencies = [ "axum", + "flate2", "getrandom", "http", + "lzma-rs", + "ruzstd", "serde", "serde_json", "sha2", + "tar", "thiserror", "tower-service", "worker", @@ -610,12 +677,34 @@ version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags", + "errno", + "libc", + "linux-raw-sys", + "windows-sys", +] + [[package]] name = "rustversion" version = "1.0.23" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" +[[package]] +name = "ruzstd" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a7c1c839d570d835527c9a5e4db7cb2198683a988cb9d7293fc8674e6bd58fc8" +dependencies = [ + "twox-hash", +] + [[package]] name = "ryu" version = "1.0.23" @@ -762,6 +851,17 @@ dependencies = [ "syn", ] +[[package]] +name = "tar" +version = "0.4.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f6221d9a6003c78398e3b239969f352578258df48c8eb051caadae0015bc840" +dependencies = [ + "filetime", + "libc", + "xattr", +] + [[package]] name = "thiserror" version = "2.0.18" @@ -827,6 +927,12 @@ version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" +[[package]] +name = "twox-hash" +version = "2.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8464ec13c3691491391d9fce00f6416c9a48e46972f72d7865688be2080192c9" + [[package]] name = "typenum" version = "1.20.1" @@ -941,6 +1047,21 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + [[package]] name = "worker" version = "0.7.5" @@ -1005,6 +1126,16 @@ version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" +[[package]] +name = "xattr" +version = "1.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32e45ad4206f6d2479085147f02bc2ef834ac85886624a23575ae137c8aa8156" +dependencies = [ + "libc", + "rustix", +] + [[package]] name = "yoke" version = "0.8.3" diff --git a/pages/Cargo.toml b/pages/Cargo.toml index 9eee284..5ace0b8 100644 --- a/pages/Cargo.toml +++ b/pages/Cargo.toml @@ -17,6 +17,17 @@ thiserror = "2" # Zip reading for upload validation (core::upload); deflate is the only # compression method we need to support. Pure Rust, compiles on wasm32. zip = { version = "2", default-features = false, features = ["deflate"] } +# Compressed-tarball decompressors for upload validation (core::upload). All +# three are pure Rust with no C (`*-sys`) bindings and no threads, so they +# compile for wasm32-unknown-unknown and run in the single-threaded Worker. +# gzip via miniz_oxide's rust_backend (no zlib-sys); ruzstd is a pure-Rust +# streaming zstd decoder; lzma-rs is a pure-Rust (one-shot) xz decoder. +flate2 = { version = "1", default-features = false, features = ["rust_backend"] } +ruzstd = "0.8" +lzma-rs = "0.3" +# ustar tar reading for the compressed-tarball formats. Reads entries as +# in-memory `io::Read` streams — entries are never unpacked to a filesystem. +tar = "0.4" # SHA-256 for session token hashing (auth); pure Rust, compiles on wasm32. sha2 = "0.10" # Entropy for session/state token generation (auth.rs). Declared unconditionally diff --git a/pages/src/core/upload.rs b/pages/src/core/upload.rs index 61de0ea..9b38bf4 100644 --- a/pages/src/core/upload.rs +++ b/pages/src/core/upload.rs @@ -58,6 +58,35 @@ impl Default for UploadLimits { } } +/// The compression wrapper around a tar archive, for the +/// [`UploadKind::Tar`] upload formats. +/// +/// Each variant names a **pure-Rust** decompressor (no `*-sys` C bindings, +/// no threads) so it compiles and runs in the single-threaded Worker: gzip +/// via `flate2`'s `miniz_oxide` backend, zstd via `ruzstd`, xz via +/// `lzma-rs`. bzip2 is deliberately unsupported. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Compression { + /// gzip-compressed tar (`.tar.gz` / `.tgz`), magic `1f 8b`. + Gzip, + /// zstd-compressed tar (`.tar.zst`), magic `28 b5 2f fd`. + Zstd, + /// xz-compressed tar (`.tar.xz`), magic `fd 37 7a 58 5a 00`. + Xz, +} + +impl Compression { + /// A short, stable label used in decompression-failure error messages + /// (e.g. `"gzip"`), so a `400` body names the format that failed. + fn label(self) -> &'static str { + match self { + Compression::Gzip => "gzip", + Compression::Zstd => "zstd", + Compression::Xz => "xz", + } + } +} + /// The kind of upload being validated — determines how `bytes` is /// interpreted by [`build_manifest`]. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -66,6 +95,10 @@ pub enum UploadKind { Html, /// A zip archive of static assets, extracted per design §2.2. Zip, + /// A compressed tar archive of static assets. The inner tar is parsed + /// with the exact same structural and size rules as [`UploadKind::Zip`] + /// (see [`build_manifest`]); only the outer decompression differs. + Tar(Compression), } /// One file destined for R2 storage: `path` is the R2 object key suffix @@ -153,6 +186,17 @@ pub enum UploadError { /// The bytes could not be parsed as a zip archive at all. #[error("upload could not be read as a valid zip archive")] InvalidZip, + /// The decompressed bytes could not be parsed as a tar archive at all. + #[error("upload could not be read as a valid tar archive")] + InvalidTar, + /// The outer compression wrapper (gzip/zstd/xz) failed to decompress — + /// either the stream was corrupt, or (for xz) it produced more output + /// than the total-size guard allows and was aborted mid-decompress. + #[error("upload could not be decompressed as a {format} archive")] + DecompressionFailed { + /// The compression format that failed (`"gzip"`, `"zstd"`, `"xz"`). + format: &'static str, + }, /// The archive (or html upload) contained no files. #[error("upload contains no files")] Empty, @@ -173,6 +217,12 @@ pub enum UploadError { /// or running total limit is crossed, never trusting the zip header's /// declared size. The archive must contain a root-level `index.html` and /// at least one file. +/// - [`UploadKind::Tar`]: `bytes` is decompressed (gzip/zstd/xz) and the +/// inner tar parsed with the **exact same** structural and size rules as +/// the zip path — see [`build_tar_manifest`]. The streaming size guard is +/// preserved end-to-end: gzip and zstd decode lazily as each tar entry is +/// read, and xz (which decodes one-shot) is bounded by a [`LimitedWriter`] +/// so an xz bomb aborts mid-decompress rather than fully materializing. /// /// # Examples /// @@ -203,6 +253,58 @@ pub fn build_manifest( match kind { UploadKind::Html => build_html_manifest(bytes, limits), UploadKind::Zip => build_zip_manifest(bytes, limits), + UploadKind::Tar(compression) => build_tar_manifest(bytes, compression, limits), + } +} + +/// Sniffs the [`UploadKind`] of a buffered upload body from its leading +/// **magic bytes**, falling back to [`UploadKind::Html`] when no archive +/// signature matches. +/// +/// This deliberately ignores the client-declared `Content-Type` for the +/// archive formats — it matches this module's "never trust the archive" +/// posture, and prevents a mislabelled body (a zip sent as `text/html`, or +/// vice versa) from being fed to the wrong parser. `content_type_hint` is +/// retained only as documentation of the caller's declared type; because raw +/// HTML has no magic signature, the no-magic case must always fall through to +/// `Html` regardless of the hint (an upload is never rejected merely for an +/// absent or unexpected `Content-Type`). +/// +/// Recognised signatures: +/// - `50 4b` (`PK`) → [`UploadKind::Zip`] +/// - `1f 8b` → [`UploadKind::Tar`]`(`[`Compression::Gzip`]`)` +/// - `28 b5 2f fd` → [`UploadKind::Tar`]`(`[`Compression::Zstd`]`)` +/// - `fd 37 7a 58 5a 00` → [`UploadKind::Tar`]`(`[`Compression::Xz`]`)` +/// - anything else → [`UploadKind::Html`] +/// +/// # Examples +/// +/// ``` +/// use pages::core::upload::{detect_kind, Compression, UploadKind}; +/// +/// assert_eq!(detect_kind(b"PK\x03\x04", ""), UploadKind::Zip); +/// assert_eq!(detect_kind(b"\x1f\x8b\x08", ""), UploadKind::Tar(Compression::Gzip)); +/// assert_eq!(detect_kind(b"", "text/html"), UploadKind::Html); +/// ``` +pub fn detect_kind(bytes: &[u8], content_type_hint: &str) -> UploadKind { + // The hint plays no part in archive detection (magic bytes are + // authoritative); it is accepted so callers can pass it through and so a + // future tiebreaker for the no-magic case has a home without a signature + // change. Reference it to keep that intent explicit. + let _ = content_type_hint; + + if bytes.starts_with(b"PK") { + UploadKind::Zip + } else if bytes.starts_with(&[0x1f, 0x8b]) { + UploadKind::Tar(Compression::Gzip) + } else if bytes.starts_with(&[0xfd, 0x37, 0x7a, 0x58, 0x5a, 0x00]) { + UploadKind::Tar(Compression::Xz) + } else if bytes.starts_with(&[0x28, 0xb5, 0x2f, 0xfd]) { + UploadKind::Tar(Compression::Zstd) + } else { + // Raw HTML (and anything else without an archive signature). Never a + // rejection here — the manifest builder enforces emptiness/size. + UploadKind::Html } } @@ -275,7 +377,13 @@ fn build_zip_manifest( return Err(UploadError::DuplicateEntry { path: name }); } - let data = read_entry_limited(&mut entry, &name, limits, &mut total_used)?; + let data = read_entry_limited( + &mut entry, + &name, + limits, + &mut total_used, + &UploadError::InvalidZip, + )?; manifest.push(ManifestEntry { path: name, bytes: data, @@ -292,6 +400,223 @@ fn build_zip_manifest( Ok(manifest) } +/// Builds the manifest for an [`UploadKind::Tar`] upload: decompresses the +/// outer wrapper, then parses the inner tar with the same rules as +/// [`build_zip_manifest`]. +/// +/// The decompressor choice is the *only* thing that varies by +/// [`Compression`]: +/// +/// - **gzip / zstd** are streaming `Read` decoders. We wrap the raw bytes in +/// the decoder and hand that straight to the tar reader, so decompression +/// is lazy: each tar entry's bytes are only produced as +/// [`read_entry_limited`] pulls them, and the read aborts the instant a +/// per-file or running-total limit is crossed — a compression bomb is +/// caught after at most one extra chunk, never fully materialized. +/// - **xz** (via `lzma-rs`) is one-shot: it can only decompress the *whole* +/// stream at once. To keep the same bomb guarantee we decompress into a +/// [`LimitedWriter`], which returns an error the moment cumulative output +/// crosses the materialization cap ([`xz_materialize_cap`]) — so an xz +/// bomb aborts mid-decompress. The bounded buffer is then parsed as tar, +/// whose per-entry reads still flow through [`read_entry_limited`]. +fn build_tar_manifest( + bytes: Vec, + compression: Compression, + limits: &UploadLimits, +) -> Result, UploadError> { + match compression { + Compression::Gzip => { + let decoder = flate2::read::GzDecoder::new(Cursor::new(bytes)); + read_tar_entries(decoder, limits) + } + Compression::Zstd => { + // `StreamingDecoder::new` reads and validates the frame header up + // front; a body that isn't a zstd frame fails here. + let decoder = + ruzstd::decoding::StreamingDecoder::new(Cursor::new(bytes)).map_err(|_| { + UploadError::DecompressionFailed { + format: Compression::Zstd.label(), + } + })?; + read_tar_entries(decoder, limits) + } + Compression::Xz => { + // One-shot decode into a size-bounded sink so an xz bomb can't + // fully materialize before the tar-level guards ever run. + let mut input = Cursor::new(bytes); + let mut writer = LimitedWriter::new(xz_materialize_cap(limits)); + lzma_rs::xz_decompress(&mut input, &mut writer).map_err(|_| { + UploadError::DecompressionFailed { + format: Compression::Xz.label(), + } + })?; + read_tar_entries(Cursor::new(writer.into_inner()), limits) + } + } +} + +/// Iterates the entries of a (already-decompressed) tar stream, applying the +/// identical structural and size rules as [`build_zip_manifest`]: directory +/// and non-regular-file entries are skipped, every file entry's name is +/// path-checked ([`check_entry_path`]), duplicates are rejected, the total +/// entry count is capped, decompressed bytes are counted as they stream out +/// ([`read_entry_limited`]), and the result must contain a root-level +/// `index.html` and at least one file. +/// +/// `reader` is generic so the caller can pass either a live streaming +/// decoder (gzip/zstd — decompression happens lazily as entries are read) or +/// a `Cursor` over an already-bounded buffer (xz). +fn read_tar_entries( + reader: R, + limits: &UploadLimits, +) -> Result, UploadError> { + let mut archive = tar::Archive::new(reader); + let entries = archive.entries().map_err(|_| UploadError::InvalidTar)?; + + let mut manifest = Vec::new(); + // Mirror `build_zip_manifest`'s duplicate guard. Unlike the zip crate, + // `tar` performs no name-based deduplication, so with a hand-crafted + // archive this check is genuinely reachable. + let mut seen: HashSet = HashSet::new(); + let mut total_used: u64 = 0; + // The zip path caps `archive.len()` (all entries, directories included) + // up front; tar has no up-front count, so we cap it as we go, counting + // every entry (skipped ones too) and bailing the moment the cap is + // crossed — an archive with a huge entry count is rejected after reading + // only `max_entries + 1` headers. + let mut entry_count: usize = 0; + + for entry in entries { + let mut entry = entry.map_err(|_| UploadError::InvalidTar)?; + + entry_count += 1; + if entry_count > limits.max_entries { + return Err(UploadError::TooManyEntries { + max: limits.max_entries, + }); + } + + // Skip anything that isn't a regular file: directories, symlinks, + // hardlinks, devices, and tar's PAX/GNU metadata pseudo-entries. Only + // regular files become R2 objects (matching the zip path's + // `is_dir()` skip, but stricter — tar has more entry types). + if !entry.header().entry_type().is_file() { + continue; + } + + // Use the raw path *bytes* and require valid UTF-8, rather than + // `entry.path()` (which lossily maps invalid sequences and can apply + // OS-specific separator rewriting) — `check_entry_path` must see the + // exact stored name to make its traversal decision. + let name_bytes = entry.path_bytes(); + let name = match std::str::from_utf8(&name_bytes) { + Ok(s) => s.to_string(), + Err(_) => { + return Err(UploadError::PathTraversal { + path: String::from_utf8_lossy(&name_bytes).into_owned(), + }) + } + }; + check_entry_path(&name, limits)?; + + if !seen.insert(name.clone()) { + return Err(UploadError::DuplicateEntry { path: name }); + } + + let data = read_entry_limited( + &mut entry, + &name, + limits, + &mut total_used, + &UploadError::InvalidTar, + )?; + manifest.push(ManifestEntry { + path: name, + bytes: data, + }); + } + + if manifest.is_empty() { + return Err(UploadError::Empty); + } + if !manifest.iter().any(|e| e.path == "index.html") { + return Err(UploadError::MissingIndexHtml); + } + + Ok(manifest) +} + +/// The maximum number of decompressed bytes the one-shot xz path will +/// materialize before aborting: `max_uncompressed_total` plus a bounded +/// allowance for tar framing (a 512-byte header and up to ~512 bytes of +/// content padding per entry, plus the two-block zero trailer). +/// +/// Sizing the allowance off `max_entries` guarantees that an honest archive +/// whose *file contents* fit within `max_uncompressed_total` never trips this +/// materialization cap on framing overhead alone — its precise per-file and +/// total limits are still enforced afterwards by [`read_entry_limited`]. A +/// genuine xz bomb, whose output dwarfs this bound, is aborted here before it +/// can fill memory. +fn xz_materialize_cap(limits: &UploadLimits) -> u64 { + let framing = (limits.max_entries as u64) + .saturating_add(2) + .saturating_mul(1024); + limits.max_uncompressed_total.saturating_add(framing) +} + +/// A [`std::io::Write`] sink that buffers bytes in memory but refuses to grow +/// past `limit`, returning an [`std::io::Error`] on the write that would +/// cross it. +/// +/// Used only by the one-shot xz path in [`build_tar_manifest`]: because +/// `lzma_rs::xz_decompress` decodes the whole stream in a single call, this +/// is the mechanism that stops an xz decompression bomb from fully +/// materializing — the error it returns aborts `xz_decompress` mid-stream, +/// exactly as [`read_entry_limited`]'s early return does for the streaming +/// gzip/zstd formats. +struct LimitedWriter { + /// The bytes accumulated so far (never exceeds `limit`). + buf: Vec, + /// The hard cap on total buffered bytes. + limit: u64, + /// Running count of bytes accepted so far. + written: u64, +} + +impl LimitedWriter { + /// Creates a sink that will accept at most `limit` total bytes. + fn new(limit: u64) -> Self { + Self { + buf: Vec::new(), + limit, + written: 0, + } + } + + /// Consumes the writer, returning the buffered bytes. + fn into_inner(self) -> Vec { + self.buf + } +} + +impl std::io::Write for LimitedWriter { + fn write(&mut self, data: &[u8]) -> std::io::Result { + self.written = self.written.saturating_add(data.len() as u64); + if self.written > self.limit { + return Err(std::io::Error::new( + std::io::ErrorKind::InvalidData, + "decompressed output exceeded the allowed size", + )); + } + self.buf.extend_from_slice(data); + Ok(data.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + /// Validates a raw zip entry name against the path-traversal and /// length/depth rules from design §2.2. /// @@ -350,24 +675,29 @@ fn check_entry_path(name: &str, limits: &UploadLimits) -> Result<(), UploadError /// Chunk size used by [`read_entry_limited`]'s streaming read loop. const READ_CHUNK: usize = 64 * 1024; -/// Reads a zip entry's decompressed content, enforcing `limits.max_file` +/// Reads an archive entry's decompressed content, enforcing `limits.max_file` /// and `limits.max_uncompressed_total` **during** the read rather than /// after — the read loop aborts as soon as either limit is crossed, so a -/// high-compression-ratio "zip bomb" entry is rejected after at most +/// high-compression-ratio "zip/tar bomb" entry is rejected after at most /// `READ_CHUNK` extra bytes are decompressed, never after the full /// (potentially huge) claimed uncompressed size. +/// +/// `read_error` is the [`UploadError`] returned if the underlying reader +/// itself fails mid-stream — [`UploadError::InvalidZip`] for the zip path, +/// [`UploadError::InvalidTar`] for the tar path (a corrupt gzip/zstd stream +/// surfaces as a read error here and folds into the tar error, which is +/// close enough for a `400` body). fn read_entry_limited( reader: &mut R, path: &str, limits: &UploadLimits, total_used: &mut u64, + read_error: &UploadError, ) -> Result, UploadError> { let mut buf = Vec::new(); let mut chunk = [0u8; READ_CHUNK]; loop { - let n = reader - .read(&mut chunk) - .map_err(|_| UploadError::InvalidZip)?; + let n = reader.read(&mut chunk).map_err(|_| read_error.clone())?; if n == 0 { break; } @@ -861,4 +1191,514 @@ mod tests { UploadError::FileTooLarge { .. } | UploadError::TotalTooLarge { .. } )); } + + // ── compressed-tar fixtures ───────────────────────────────────────── + // + // Like `make_zip_raw`, the inner tar is assembled by hand (rather than + // through `tar::Builder`) so fixtures can carry adversarial names the + // builder would sanitize or reject — `../evil.txt`, absolute paths, + // drive letters. The raw ustar bytes are then compressed with each + // pure-Rust codec's *encoder* (`flate2`, `ruzstd`, `lzma-rs`), so no + // binary fixtures and no C dev-dependency are needed. + + /// Writes `val` into a tar header numeric field as zero-padded octal + /// followed by a trailing NUL (the ustar convention): the field holds + /// `buf.len() - 1` octal digits plus the terminator. + fn tar_octal(buf: &mut [u8], val: u64) { + let digits = buf.len() - 1; + let s = format!("{val:0digits$o}"); + buf[..digits].copy_from_slice(s.as_bytes()); + buf[digits] = 0; + } + + /// Hand-builds an uncompressed ustar archive from `(name, content)` + /// pairs. A trailing `/` in a name makes a directory entry (typeflag + /// `5`, no body); everything else is a regular file (typeflag `0`). + /// Each entry is a 512-byte header (with a correct checksum, which the + /// `tar` reader validates) followed by the content padded to the next + /// 512-byte boundary; the stream ends with two all-zero blocks. + fn make_tar_raw(entries: &[(&str, &[u8])]) -> Vec { + let mut out = Vec::new(); + for (name, content) in entries { + let mut header = [0u8; 512]; + let name_bytes = name.as_bytes(); + let n = name_bytes.len().min(100); + header[0..n].copy_from_slice(&name_bytes[..n]); + tar_octal(&mut header[100..108], 0o644); // mode + tar_octal(&mut header[108..116], 0); // uid + tar_octal(&mut header[116..124], 0); // gid + tar_octal(&mut header[124..136], content.len() as u64); // size + tar_octal(&mut header[136..148], 0); // mtime + let is_dir = name.ends_with('/'); + header[156] = if is_dir { b'5' } else { b'0' }; // typeflag + header[257..263].copy_from_slice(b"ustar\0"); // magic + header[263..265].copy_from_slice(b"00"); // version + + // Checksum: the field is treated as 8 spaces while the unsigned + // byte sum of the whole header is computed, then written back as + // 6 octal digits + NUL + space. + for b in &mut header[148..156] { + *b = b' '; + } + let sum: u32 = header.iter().map(|&b| b as u32).sum(); + let cks = format!("{sum:06o}"); + header[148..154].copy_from_slice(cks.as_bytes()); + header[154] = 0; + header[155] = b' '; + + out.extend_from_slice(&header); + if !is_dir { + out.extend_from_slice(content); + let pad = (512 - (content.len() % 512)) % 512; + out.extend(std::iter::repeat_n(0u8, pad)); + } + } + out.extend(std::iter::repeat_n(0u8, 1024)); // two-block trailer + out + } + + /// gzip-compresses `data` with `flate2`'s writer. + fn gzip(data: &[u8]) -> Vec { + let mut enc = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + enc.write_all(data).unwrap(); + enc.finish().unwrap() + } + + /// zstd-compresses `data` with `ruzstd`'s encoder (the `Fastest` level; + /// higher levels are unimplemented in 0.8). + fn zstd(data: &[u8]) -> Vec { + ruzstd::encoding::compress_to_vec( + Cursor::new(data), + ruzstd::encoding::CompressionLevel::Fastest, + ) + } + + /// xz-compresses `data` with `lzma-rs`'s one-shot `xz_compress`. + fn xz(data: &[u8]) -> Vec { + let mut out = Vec::new(); + lzma_rs::xz_compress(&mut Cursor::new(data), &mut out).unwrap(); + out + } + + /// Compresses raw tar bytes with the codec for `comp`. + fn compress_for(comp: Compression, data: &[u8]) -> Vec { + match comp { + Compression::Gzip => gzip(data), + Compression::Zstd => zstd(data), + Compression::Xz => xz(data), + } + } + + /// Builds a compressed-tarball upload fixture: a hand-rolled ustar + /// archive of `entries`, compressed with `comp`'s codec. + fn make_tar(comp: Compression, entries: &[(&str, &[u8])]) -> Vec { + compress_for(comp, &make_tar_raw(entries)) + } + + // Shared test bodies, run once per format by the `#[test]` wrappers + // below — this mirrors the zip coverage for gzip, zstd, and xz without + // triplicating every assertion. + + /// Happy path: a multi-file archive with subdirectories and a root + /// `index.html` yields exactly those files. + fn check_tar_happy_path(comp: Compression) { + let tar = make_tar( + comp, + &[ + ("index.html", b""), + ("assets/app.js", b"console.log(1)"), + ("assets/img/logo.png", b"\x89PNG"), + ], + ); + let manifest = build_manifest(UploadKind::Tar(comp), tar, &limits()).unwrap(); + let mut paths: Vec<&str> = manifest.iter().map(|e| e.path.as_str()).collect(); + paths.sort(); + assert_eq!( + paths, + ["assets/app.js", "assets/img/logo.png", "index.html"] + ); + } + + /// Directory entries are skipped, not stored as objects. + fn check_tar_directory_entries_skipped(comp: Compression) { + let tar = make_tar( + comp, + &[ + ("assets/", b""), + ("index.html", b"hi"), + ("assets/app.js", b"x"), + ], + ); + let manifest = build_manifest(UploadKind::Tar(comp), tar, &limits()).unwrap(); + assert_eq!(manifest.len(), 2); + assert!(manifest.iter().all(|e| !e.path.ends_with('/'))); + } + + /// A `..` segment is rejected as path traversal. + fn check_tar_rejects_traversal(comp: Compression) { + let tar = make_tar(comp, &[("index.html", b"hi"), ("../evil.txt", b"pwn")]); + let err = build_manifest(UploadKind::Tar(comp), tar, &limits()).unwrap_err(); + assert!(matches!(err, UploadError::PathTraversal { .. })); + } + + /// Missing root-level `index.html` is rejected. + fn check_tar_rejects_missing_index(comp: Compression) { + let tar = make_tar(comp, &[("assets/app.js", b"x")]); + let err = build_manifest(UploadKind::Tar(comp), tar, &limits()).unwrap_err(); + assert_eq!(err, UploadError::MissingIndexHtml); + } + + /// A single file over the per-file limit is rejected by the streaming + /// guard. + fn check_tar_rejects_file_over_limit(comp: Compression) { + let tar = make_tar(comp, &[("index.html", b"hi"), ("big.bin", &[1u8; 100])]); + let small = UploadLimits { + max_file: 50, + ..limits() + }; + let err = build_manifest(UploadKind::Tar(comp), tar, &small).unwrap_err(); + assert!(matches!(err, UploadError::FileTooLarge { .. })); + } + + /// The total across files exceeding the total limit is rejected by the + /// streaming guard (never by the xz materialization cap, which has + /// framing headroom above `max_uncompressed_total`). + fn check_tar_rejects_total_over_limit(comp: Compression) { + let tar = make_tar( + comp, + &[ + ("index.html", &[1u8; 40]), + ("a.bin", &[1u8; 40]), + ("b.bin", &[1u8; 40]), + ], + ); + let small = UploadLimits { + max_file: 1000, + max_uncompressed_total: 100, + ..limits() + }; + let err = build_manifest(UploadKind::Tar(comp), tar, &small).unwrap_err(); + assert!(matches!(err, UploadError::TotalTooLarge { .. })); + } + + // ── gzip ───────────────────────────────────────────────────────────── + + #[test] + fn targz_happy_path_multi_file_with_subdirs() { + check_tar_happy_path(Compression::Gzip); + } + + #[test] + fn targz_directory_entries_are_skipped() { + check_tar_directory_entries_skipped(Compression::Gzip); + } + + #[test] + fn targz_rejects_traversal() { + check_tar_rejects_traversal(Compression::Gzip); + } + + #[test] + fn targz_rejects_missing_index_html() { + check_tar_rejects_missing_index(Compression::Gzip); + } + + #[test] + fn targz_rejects_file_over_per_file_limit() { + check_tar_rejects_file_over_limit(Compression::Gzip); + } + + #[test] + fn targz_rejects_total_over_limit() { + check_tar_rejects_total_over_limit(Compression::Gzip); + } + + /// A gzip "bomb": a single highly-compressible entry whose declared + /// (and real) uncompressed size dwarfs a tight per-file limit. Because + /// gzip decodes lazily as the tar entry is read, the streaming guard + /// aborts after one extra chunk — the full content is never + /// materialized. + #[test] + fn targz_bomb_rejected_by_streaming_guard() { + let tar = make_tar( + Compression::Gzip, + &[("index.html", &vec![0u8; 4 * 1024 * 1024])], + ); + assert!( + tar.len() < 100 * 1024, + "fixture not a high compression ratio: {} bytes", + tar.len() + ); + let tight = UploadLimits { + max_file: 1024, + ..limits() + }; + let err = build_manifest(UploadKind::Tar(Compression::Gzip), tar, &tight).unwrap_err(); + assert!(matches!(err, UploadError::FileTooLarge { .. })); + } + + // ── zstd ───────────────────────────────────────────────────────────── + + #[test] + fn tarzst_happy_path_multi_file_with_subdirs() { + check_tar_happy_path(Compression::Zstd); + } + + #[test] + fn tarzst_directory_entries_are_skipped() { + check_tar_directory_entries_skipped(Compression::Zstd); + } + + #[test] + fn tarzst_rejects_traversal() { + check_tar_rejects_traversal(Compression::Zstd); + } + + #[test] + fn tarzst_rejects_missing_index_html() { + check_tar_rejects_missing_index(Compression::Zstd); + } + + #[test] + fn tarzst_rejects_file_over_per_file_limit() { + check_tar_rejects_file_over_limit(Compression::Zstd); + } + + #[test] + fn tarzst_rejects_total_over_limit() { + check_tar_rejects_total_over_limit(Compression::Zstd); + } + + /// A zstd "bomb": as with gzip, `ruzstd`'s `StreamingDecoder` decodes + /// lazily, so the streaming guard aborts the read long before the whole + /// entry is decompressed. + #[test] + fn tarzst_bomb_rejected_by_streaming_guard() { + let tar = make_tar( + Compression::Zstd, + &[("index.html", &vec![0u8; 4 * 1024 * 1024])], + ); + let tight = UploadLimits { + max_file: 1024, + ..limits() + }; + let err = build_manifest(UploadKind::Tar(Compression::Zstd), tar, &tight).unwrap_err(); + assert!(matches!(err, UploadError::FileTooLarge { .. })); + } + + // ── xz ─────────────────────────────────────────────────────────────── + + #[test] + fn tarxz_happy_path_multi_file_with_subdirs() { + check_tar_happy_path(Compression::Xz); + } + + #[test] + fn tarxz_directory_entries_are_skipped() { + check_tar_directory_entries_skipped(Compression::Xz); + } + + #[test] + fn tarxz_rejects_traversal() { + check_tar_rejects_traversal(Compression::Xz); + } + + #[test] + fn tarxz_rejects_missing_index_html() { + check_tar_rejects_missing_index(Compression::Xz); + } + + #[test] + fn tarxz_rejects_file_over_per_file_limit() { + check_tar_rejects_file_over_limit(Compression::Xz); + } + + #[test] + fn tarxz_rejects_total_over_limit() { + check_tar_rejects_total_over_limit(Compression::Xz); + } + + /// The xz "bomb" test, exercising the one-shot path's [`LimitedWriter`] + /// specifically: xz can only decode the whole stream at once, so the + /// guarantee is upheld by the writer aborting decompression once its + /// materialization cap is crossed. With tight limits the cap is a few + /// KiB, far below the multi-MiB decompressed output, so + /// `xz_decompress` fails and the upload is rejected as + /// `DecompressionFailed` — the full output is never buffered. + #[test] + fn tarxz_bomb_rejected_by_limited_writer() { + let tar = make_tar( + Compression::Xz, + &[("index.html", &vec![0u8; 4 * 1024 * 1024])], + ); + let tight = UploadLimits { + max_uncompressed_total: 4096, + max_file: 4096, + max_entries: 2, + ..limits() + }; + // The materialization cap here is small (4 KiB + a bounded framing + // allowance), well under the 4 MiB of decompressed output. + assert!( + xz_materialize_cap(&tight) < 4 * 1024 * 1024, + "cap should be far below the bomb's output" + ); + let err = build_manifest(UploadKind::Tar(Compression::Xz), tar, &tight).unwrap_err(); + assert_eq!(err, UploadError::DecompressionFailed { format: "xz" }); + } + + // ── structural / decompression errors ─────────────────────────────── + + #[test] + fn tar_rejects_empty_archive() { + // A bare two-block trailer decompresses fine but contains no files. + let tar = make_tar(Compression::Gzip, &[]); + let err = build_manifest(UploadKind::Tar(Compression::Gzip), tar, &limits()).unwrap_err(); + assert_eq!(err, UploadError::Empty); + } + + #[test] + fn tar_rejects_too_many_entries() { + let small = UploadLimits { + max_entries: 3, + ..limits() + }; + let tar = make_tar( + Compression::Gzip, + &[ + ("index.html", b"hi"), + ("a.txt", b"1"), + ("b.txt", b"2"), + ("c.txt", b"3"), + ], + ); + let err = build_manifest(UploadKind::Tar(Compression::Gzip), tar, &small).unwrap_err(); + assert_eq!(err, UploadError::TooManyEntries { max: 3 }); + } + + #[test] + fn tar_rejects_duplicate_entries() { + // Unlike zip, tar performs no name deduplication, so the + // `DuplicateEntry` guard is genuinely reachable here. + let tar = make_tar( + Compression::Gzip, + &[ + ("index.html", b"hi"), + ("dup.txt", b"one"), + ("dup.txt", b"two"), + ], + ); + let err = build_manifest(UploadKind::Tar(Compression::Gzip), tar, &limits()).unwrap_err(); + assert!(matches!(err, UploadError::DuplicateEntry { .. })); + } + + #[test] + fn tar_rejects_path_too_long() { + // ustar names cap at 100 bytes, so scale the limit down rather than + // using an over-long name that wouldn't fit the header field. + let small = UploadLimits { + max_path_len: 20, + ..limits() + }; + let long = format!("{}.txt", "a".repeat(30)); + let tar = make_tar(Compression::Gzip, &[("index.html", b"hi"), (&long, b"x")]); + let err = build_manifest(UploadKind::Tar(Compression::Gzip), tar, &small).unwrap_err(); + assert!(matches!(err, UploadError::PathTooLong { .. })); + } + + #[test] + fn tar_rejects_absolute_path() { + let tar = make_tar( + Compression::Zstd, + &[("index.html", b"hi"), ("/etc/passwd", b"pwn")], + ); + let err = build_manifest(UploadKind::Tar(Compression::Zstd), tar, &limits()).unwrap_err(); + assert!(matches!(err, UploadError::PathTraversal { .. })); + } + + #[test] + fn tar_gzip_corrupt_stream_rejected() { + // Valid gzip magic but a truncated/garbage body — the decoder errors + // mid-read, which folds into `InvalidTar`. + let mut bytes = gzip(&make_tar_raw(&[("index.html", b"hi")])); + // Corrupt the compressed payload past the gzip header. + for b in bytes.iter_mut().skip(12) { + *b ^= 0xff; + } + let err = build_manifest(UploadKind::Tar(Compression::Gzip), bytes, &limits()).unwrap_err(); + assert!(matches!( + err, + UploadError::InvalidTar | UploadError::DecompressionFailed { .. } + )); + } + + #[test] + fn tar_zstd_invalid_frame_rejected() { + // zstd magic but not a valid frame. Depending on where the garbage + // trips the decoder, this surfaces either from + // `StreamingDecoder::new` (→ `DecompressionFailed`) or mid-read while + // the tar reader pulls bytes (→ `InvalidTar`); both are correct 400s. + let bytes = vec![0x28, 0xb5, 0x2f, 0xfd, 0x00, 0x11, 0x22, 0x33]; + let err = build_manifest(UploadKind::Tar(Compression::Zstd), bytes, &limits()).unwrap_err(); + assert!(matches!( + err, + UploadError::DecompressionFailed { format: "zstd" } | UploadError::InvalidTar + )); + } + + // ── detect_kind (magic-byte sniffing) ─────────────────────────────── + + #[test] + fn detect_kind_recognizes_zip_magic() { + assert_eq!(detect_kind(b"PK\x03\x04rest", "text/html"), UploadKind::Zip); + } + + #[test] + fn detect_kind_recognizes_gzip_magic() { + assert_eq!( + detect_kind(&[0x1f, 0x8b, 0x08, 0x00], ""), + UploadKind::Tar(Compression::Gzip) + ); + } + + #[test] + fn detect_kind_recognizes_xz_magic() { + assert_eq!( + detect_kind(&[0xfd, 0x37, 0x7a, 0x58, 0x5a, 0x00, 0x00], ""), + UploadKind::Tar(Compression::Xz) + ); + } + + #[test] + fn detect_kind_recognizes_zstd_magic() { + assert_eq!( + detect_kind(&[0x28, 0xb5, 0x2f, 0xfd, 0x00], ""), + UploadKind::Tar(Compression::Zstd) + ); + } + + #[test] + fn detect_kind_falls_back_to_html_without_magic() { + // Raw HTML has no signature: fall back to Html regardless of the + // Content-Type hint (including an absent/empty one). + assert_eq!( + detect_kind(b"", "text/html"), + UploadKind::Html + ); + assert_eq!(detect_kind(b"", ""), UploadKind::Html); + assert_eq!(detect_kind(b"", "application/zip"), UploadKind::Html); + } + + /// The magic-byte sniffer picks the kind that actually round-trips + /// through `build_manifest` for a real fixture of each format. + #[test] + fn detect_kind_agrees_with_build_manifest_per_format() { + for comp in [Compression::Gzip, Compression::Zstd, Compression::Xz] { + let tar = make_tar(comp, &[("index.html", b"hi")]); + assert_eq!(detect_kind(&tar, ""), UploadKind::Tar(comp)); + build_manifest(detect_kind(&tar, ""), tar, &limits()).unwrap(); + } + let zip = make_zip(&[("index.html", b"hi")]); + assert_eq!(detect_kind(&zip, ""), UploadKind::Zip); + } } diff --git a/pages/src/routes/api.rs b/pages/src/routes/api.rs index 592d353..b82aaca 100644 --- a/pages/src/routes/api.rs +++ b/pages/src/routes/api.rs @@ -32,7 +32,7 @@ use super::auth::current_user; use super::AppState; use crate::core::origin::{can_delete, can_replace, origin_allowed}; use crate::core::slug::validate_slug; -use crate::core::upload::{build_manifest, ManifestEntry, UploadKind, UploadLimits}; +use crate::core::upload::{build_manifest, detect_kind, ManifestEntry, UploadLimits}; use crate::db::DbError; /// Maximum accepted raw upload size — 10 MiB (design §2.2, §4.6 step 4). @@ -66,11 +66,16 @@ fn content_length(headers: &HeaderMap) -> Option { /// - `413` if `Content-Length` (when present) *or* the actual buffered body /// exceeds [`MAX_UPLOAD_BYTES`] — both are checked, so a missing or /// understated `Content-Length` can't slip a larger body through. -/// - `400 "unsupported content type"` for anything that isn't `text/html` or -/// a zip content type. /// - `400` with the [`UploadError`](crate::core::upload::UploadError) message -/// if the payload fails validation (zip bomb, no `index.html`, unsafe -/// paths, …). +/// if the payload fails validation (zip/tar bomb, no `index.html`, unsafe +/// paths, corrupt compression, …). +/// +/// The [`UploadKind`](crate::core::upload::UploadKind) is chosen by sniffing +/// the body's leading **magic bytes** ([`detect_kind`]), not the client's +/// `Content-Type` — see that function for why. `Content-Type` is passed +/// through only as a hint; a body with no recognised archive signature (raw +/// HTML included) falls back to `Html`, so an upload is never rejected merely +/// for an absent or unexpected `Content-Type`. /// /// Both callers reach this only *after* their own auth/authorization checks, /// so an unauthorized caller never learns anything about their payload's @@ -97,22 +102,15 @@ fn manifest_from_request( )); } - // Content-Type → UploadKind. + // Sniff the upload kind from the body's magic bytes (never the declared + // Content-Type — see `detect_kind`). The Content-Type is forwarded only + // as a hint for the no-magic (raw-HTML) fallback. let content_type = headers .get(CONTENT_TYPE) .and_then(|v| v.to_str().ok()) .unwrap_or("") .to_ascii_lowercase(); - let kind = if content_type == "text/html" || content_type.starts_with("text/html;") { - UploadKind::Html - } else if content_type == "application/zip" || content_type == "application/x-zip-compressed" { - UploadKind::Zip - } else { - return Err(json_error( - StatusCode::BAD_REQUEST, - "unsupported content type", - )); - }; + let kind = detect_kind(body.as_ref(), &content_type); build_manifest(kind, body.to_vec(), &UploadLimits::default()) .map_err(|e| json_error(StatusCode::BAD_REQUEST, e.to_string())) diff --git a/pages/static/index.html b/pages/static/index.html index a4beb11..72aa875 100644 --- a/pages/static/index.html +++ b/pages/static/index.html @@ -447,9 +447,9 @@
-
Drag & drop an .html file or a .zip of static assets here, or click to choose one.
+
Drag & drop an .html file, a .zip, or a compressed tarball (.tar.gz, .tar.zst, .tar.xz) of static assets here, or click to choose one.
- +
@@ -701,13 +701,13 @@ if (uploadMode === "replace") { uploadModalTitle.textContent = "Update page"; - dropzoneLabel.textContent = "Drop the new .html file or .zip here, or click to choose one. It replaces this page's current content."; + dropzoneLabel.textContent = "Drop the new .html, .zip, or compressed tarball (.tar.gz, .tar.zst, .tar.xz) here, or click to choose one. It replaces this page's current content."; nameInput.value = slug; nameInput.readOnly = true; uploadSubmit.textContent = "Update"; } else { uploadModalTitle.textContent = "Upload a page"; - dropzoneLabel.textContent = "Drag & drop an .html file or a .zip of static assets here, or click to choose one."; + dropzoneLabel.textContent = "Drag & drop an .html file, a .zip, or a compressed tarball (.tar.gz, .tar.zst, .tar.xz) of static assets here, or click to choose one."; nameInput.value = ""; nameInput.readOnly = false; uploadSubmit.textContent = "Upload"; @@ -768,13 +768,26 @@ validateNameField(); } + var ALLOWED_EXTENSIONS = [".html", ".htm", ".zip", ".tar.gz", ".tgz", ".tar.zst", ".tar.xz"]; + function extensionAllowed(filename) { var lower = filename.toLowerCase(); - return lower.slice(-5) === ".html" || lower.slice(-4) === ".htm" || lower.slice(-4) === ".zip"; + return ALLOWED_EXTENSIONS.some(function (ext) { + return lower.slice(-ext.length) === ext; + }); + } + + function isHtmlFile(filename) { + var lower = filename.toLowerCase(); + return lower.slice(-5) === ".html" || lower.slice(-4) === ".htm"; } + // The backend sniffs the real format from the body's magic bytes, so the + // Content-Type only needs to distinguish "raw HTML" (which has no magic and + // relies on this) from "some archive" (zip or tarball). Sending + // application/octet-stream for every archive is correct and future-proof. function contentTypeFor(filename) { - return filename.toLowerCase().slice(-4) === ".zip" ? "application/zip" : "text/html"; + return isHtmlFile(filename) ? "text/html" : "application/octet-stream"; } /** The idle (non-error) hint under the name field, which differs by mode: