## Issue
Parallel Nix jobs collapse the worker's ordinary-upstream proxy into 503s. `Upstreams::lookup` fail-fasts on a sixteen-permit semaphore, but guests use the worker as their only substituter and Nix defaults to 25 parallel connections per process — two jobs burst to ~50 concurrent narinfo fetches. The overflow gets instant `503 upstream metadata capacity exhausted: no permits available`, and Nix's retries re-enter the same saturated pool instead of riding it out. The retry amplification turns a burst into sustained failures, and builds die on missing store paths. The gateway 503 also carries no `Retry-After`, so clients hammer at their own retry intervals.
## Fix
- Raise the ordinary-upstream metadata and fetch budgets from 16 to 64, matching the serving side's public metadata budget. Bulk archive streams keep their own smaller budget. 64 absorbs the combined fan-out of the maximum parallel jobs with headroom; budgets stay hardcoded per the existing pattern for these semaphores.
- Gateway 503s now carry `Retry-After: 2` so any remaining overflow backs off instead of amplifying. Scoped to the resolve gateway; public 404s are unchanged.
- Updates `docs/nix-cache-replicas.md` accordingly.
## Validation
- New `sixty_four_parallel_lookups_all_succeed_under_burst_load` regression test: 64 concurrent lookups against a delayed fake upstream — fails on sixteen permits, passes on sixty-four.
- All `cache_upstream` tests pass; `cargo fmt --check` clean.
- `cargo clippy -p ngit-ci-nix-provider --lib -- -D warnings` hits a pre-existing `nonminimal_bool` lint in the untouched `ngit-ci-nar-cache` crate under a newer bare toolchain; unrelated to this change.
Full `test-core` validation left to CI for this PR.
Parallel Nix jobs collapse the worker's ordinary-upstream proxy into 503s. `Upstreams::lookup` fail-fasts on a sixteen-permit semaphore, but guests use the worker as their only substituter and Nix defaults to 25 parallel connections per process — two jobs burst to ~50 concurrent narinfo fetches. The overflow gets instant `503 upstream metadata capacity exhausted: no permits available`, and Nix's retries re-enter the same saturated pool instead of riding it out. The retry amplification turns a burst into sustained failures, and builds die on missing store paths. The gateway 503 also carries no `Retry-After`, so clients hammer at their own retry intervals.
## Fix
- Raise the ordinary-upstream metadata and fetch budgets from 16 to 64, matching the serving side's public metadata budget. Bulk archive streams keep their own smaller budget. 64 absorbs the combined fan-out of the maximum parallel jobs with headroom; budgets stay hardcoded per the existing pattern for these semaphores.
- Gateway 503s now carry `Retry-After: 2` so any remaining overflow backs off instead of amplifying. Scoped to the resolve gateway; public 404s are unchanged.
- Updates `docs/nix-cache-replicas.md` accordingly.
## Validation
- New `sixty_four_parallel_lookups_all_succeed_under_burst_load` regression test: 64 concurrent lookups against a delayed fake upstream — fails on sixteen permits, passes on sixty-four.
- All `cache_upstream` tests pass; `cargo fmt --check` clean.
- `cargo clippy -p ngit-ci-nix-provider --lib -- -D warnings` hits a pre-existing `nonminimal_bool` lint in the untouched `ngit-ci-nar-cache` crate under a newer bare toolchain; unrelated to this change.
Full `test-core` validation left to CI for this PR.