Local workflow, smart-HTTP clone/fetch/push and public HTTPS clone/fetch match native git in stdout and exit codes. Only WASI imports (wasi:http, exit-with-code).
138 lines
8.3 KiB
Markdown
138 lines
8.3 KiB
Markdown
# WASI patches for Git foundation (SAU-21)
|
|
|
|
All crates here are complete crates.io source archives, with their original licenses.
|
|
`Cargo.toml` uses local `[patch.crates-io]` paths. No upstream submission has been made.
|
|
Changes apply only to `target_os = "wasi"`; native paths retain upstream behavior.
|
|
No parallel, signal/interrupt, or worktree-mutation features are enabled.
|
|
SAU-24 enables blocking-network-client and a custom wasi:http backend (no curl/reqwest).
|
|
|
|
## gix-tempfile 26.0.0: no OS PID on WASI
|
|
|
|
Exact panic: `src/forksafe.rs`, `ForksafeTempfile::new`, field
|
|
`owning_process_id: std::process::id()`. Ref transactions acquire gix-lock files,
|
|
which register these tempfiles. This is registry fork ownership, **not filename generation**.
|
|
`std::process::id()` panics with `no pids on this platform` on wasm32-wasip2.
|
|
|
|
Patch: a private `process_id()` returns 0 on WASI (no fork, registry is per component
|
|
instance), and the native OS PID elsewhere. Use it in creation and both registry
|
|
cleanup methods. Random tempfile naming and exclusive file creation are unchanged.
|
|
Verified the unpatched ref-updating commit panics; patched commits and index writes pass.
|
|
|
|
## gix-pack 0.77.0: owned pack/index/multi-index bytes
|
|
|
|
`src/lib.rs`: WASI's `MMap` is `Vec<u8>` and the shared `mmap::read_only()` helper
|
|
uses `std::fs::read`. Other targets retain memmap2. This covers the pack data,
|
|
pack index, and multi-pack-index constructors that all use this helper.
|
|
memmap2 0.9.11 selects its unsupported stub on WASI; the original map call returns
|
|
`ErrorKind::Unsupported` rather than reading a pack.
|
|
|
|
Tests generate a tiny pack via `gix-pack`'s `data::output::bytes::FromEntriesIter`;
|
|
the simple v2 index is encoded in the test (fanout, sorted IDs, CRCs, offsets,
|
|
pack checksum, index checksum). This avoids enabling the broken streaming-input
|
|
feature just for the test. A second test packs all commit/tree/blob objects,
|
|
removes every loose object copy and stores the branch only in `packed-refs`,
|
|
reopens the repository and reads history, paths, branches,
|
|
blob contents, and creates/reads its index. No shell or Git executable is used.
|
|
|
|
## gix-odb 0.87.0: owned loose-object bytes
|
|
|
|
Additional breakage found by the full tests: `src/store_impls/loose/find.rs`
|
|
uses memmap2 for *every loose object read and header lookup*, not just packs.
|
|
Without this patch logs and tree/index construction return `unsupported`.
|
|
Use the same platform-dependent `gix_pack::MMap` backing and std file reads on WASI.
|
|
Decompression, missing-object handling, and validation remain upstream code.
|
|
|
|
## gix-index 0.58.0: read index without mmap
|
|
|
|
`src/file/init.rs`: WASI reads into a Vec from the already opened file and rewinds
|
|
that same handle before checksum streaming. This preserves a single inode snapshot
|
|
across concurrent atomic index replacements. Native mmap code is unchanged.
|
|
Nonempty v2 index round trip and checksum-corruption rejection pass under WASI.
|
|
|
|
## filetime 0.2.29: metadata timestamp readers
|
|
|
|
Additional runtime panic found after the mmap fixes: filetime routes WASI to its
|
|
generic wasm backend, whose metadata timestamp readers call `unimplemented!()`.
|
|
`src/lib.rs` now selects `src/wasi.rs` before that backend. It obtains modified,
|
|
accessed, and optional creation times through std Metadata; inaccessible modified/
|
|
accessed times fall back to zero, because the upstream API cannot return an error.
|
|
Symlink time mutation remains an explicit Unsupported error; ordinary file-time
|
|
setters retain the upstream std File::set_times implementation. Metadata reads are
|
|
exercised by index tests; timestamp setters and symlink mutation are not exercised.
|
|
|
|
## gix-ref 0.69.0: disable packed-refs mmap through existing API
|
|
|
|
No additional vendored crate is needed. On WASI, `src/repo.rs::open` calls
|
|
`repo.refs.set_packed_buffer_mmap_threshold(u64::MAX)` before any reference lookup.
|
|
Upstream `store/packed/buffer.rs::Buffer::open` uses owned bytes for files at or
|
|
below that threshold; otherwise it uses unsupported memmap2. The default threshold
|
|
is 32 KiB. Native repository handles retain the upstream default.
|
|
|
|
Regression `large_packed_refs_head_branches_and_log` writes 65,638 bytes of sorted
|
|
packed refs (1,025 branches), removes the loose HEAD branch and reopens through
|
|
our helper. It checks symbolic HEAD, all branch tips/current flag, HEAD log order,
|
|
and log by a non-HEAD packed branch. Before the fix this fails on WASI with
|
|
`HEAD: Could not open packed refs`; with the fix it passes. Packed refs also consume
|
|
owned linear memory; this is not a repository size limit.
|
|
|
|
## Intentionally disabled / work for later cards
|
|
|
|
- `worktree-mutation`: `gix-worktree-state` depends on `io-close 0.3.7`; it lacks
|
|
a WASI File close implementation (`no method close for File`). Add a WASI backend
|
|
or write the checkout engine using std file writes. Do not silently ignore writes.
|
|
- `blocking-network-client` is now enabled by SAU-24: see the streaming-input patch below.
|
|
- `gix-commitgraph` remains unpatched: `src/file/init.rs` maps commit-graph files
|
|
with memmap2, unsupported on WASI. Current `log` explicitly uses
|
|
`use_commit_graph(false)` and does not need this cache. This is NOT disabled
|
|
centrally by `repo::open`. Later merge-base/ahead-behind/revision operations must
|
|
disable or avoid commit-graph loading (including direct graph APIs), or patch
|
|
that crate to read owned bytes before using it. Some high-level gix walks fall
|
|
back to the object database on graph-load errors; direct graph loading can fail.
|
|
Existing native Git `objects/info/commit-graph` files are not fixture-tested here.
|
|
- No signals or threads are provided; keep `interrupt` and `parallel` off.
|
|
- No large-repo memory benchmark was run. Pack + pack-index bytes are held in
|
|
32-bit linear memory, in addition to decoded objects; whole-pack reads have a
|
|
materially higher resident-memory cost than demand-paged native mmap. The tiny
|
|
fixture is 53 pack bytes for one blob. No peak RSS/linear-memory measurement.
|
|
- Multi-pack-index reading uses the patched helper but is not fixture-tested here.
|
|
Delta packs, split index, and external Git-generated fixtures are not tested.
|
|
|
|
## SAU-24: gix-pack 0.77.0 streaming-input on WASI
|
|
|
|
Existing vendored crate extended (no additional crate patch). In `Cargo.toml` and
|
|
`Cargo.toml.orig`, allow gix-tempfile under
|
|
`cfg(any(not(target_arch = "wasm32"), target_os = "wasi"))`. Browser wasm is still
|
|
excluded; gix-tempfile remains `default-features = false`, so no signals are added.
|
|
The foundation's PID patch applies unchanged.
|
|
|
|
Additional compile error exposed by enabling the feature: bundle writer's
|
|
`new_pack_file_resolver` and `resolve_entry` explicitly used `memmap2::Mmap`.
|
|
Change those three type annotations in `src/bundle/write/mod.rs` to `crate::MMap`,
|
|
reusing the existing owned-byte WASI helper and keeping native mmap behavior.
|
|
No pack/index format change. No upstream submission made.
|
|
|
|
Verified by local fake Git HTTP servers through wasi:http in cargo.build: V1
|
|
clone writes a received 227-byte / 3-object pack and real index, updates tracking
|
|
refs and checks out exact bytes; V2 shallow clone writes the shallow boundary.
|
|
This tests the bundle writer, not merely hand-encoded pack indexes. Pack input
|
|
is streamed to a tempfile; indexing still reads a whole-pack Vec on WASI and
|
|
therefore needs 32-bit memory proportional to pack size. Use clone `depth` to
|
|
reduce history. No large-repo memory measurement.
|
|
|
|
## Sources inspected
|
|
|
|
- https://raw.githubusercontent.com/GitoxideLabs/gitoxide/gix-v0.90.0/gix-ref/src/store/file/mod.rs
|
|
- https://raw.githubusercontent.com/GitoxideLabs/gitoxide/gix-v0.90.0/gix-ref/src/store/packed/buffer.rs
|
|
- https://raw.githubusercontent.com/GitoxideLabs/gitoxide/gix-v0.90.0/gix/src/types.rs
|
|
- https://raw.githubusercontent.com/GitoxideLabs/gitoxide/gix-v0.90.0/gix-commitgraph/src/file/init.rs
|
|
- https://raw.githubusercontent.com/GitoxideLabs/gitoxide/gix-v0.90.0/gix/src/revision/walk.rs
|
|
|
|
- https://raw.githubusercontent.com/GitoxideLabs/gitoxide/gix-v0.90.0/gix/Cargo.toml
|
|
- https://static.crates.io/crates/gix-tempfile/gix-tempfile-26.0.0.crate
|
|
- https://static.crates.io/crates/gix-pack/gix-pack-0.77.0.crate
|
|
- https://static.crates.io/crates/gix-index/gix-index-0.58.0.crate
|
|
- https://static.crates.io/crates/gix-odb/gix-odb-0.87.0.crate
|
|
- https://raw.githubusercontent.com/RazrFalcon/memmap2-rs/v0.9.11/src/stub.rs
|
|
- https://raw.githubusercontent.com/alexcrichton/filetime/0.2.29/src/wasm.rs
|
|
- https://static.crates.io/crates/filetime/filetime-0.2.29.crate
|