Benchmarks

How xtandard-image is compared with imgproxy OSS under HTTP load. The latest committed results are in bench/RESULTS.md. Micro-benchmarks of single functions are a different thing: they live next to the code as for b.Loop() benchmarks and any performance claim about them needs benchstat output with -count=10 (go-standards §9).

What is measured

Two servers, the same URLs, the same fixtures, the same machine:

• xtandard-image: the app image built from this repository (task release:build).
• imgproxy OSS: darthsim/imgproxy:latest, the reference implementation of the URL dialect.

Both run as containers from bench/compose.yml next to a static origin (Caddy serving bench/fixtures/ over plain HTTP on the compose network). The load generator is k6 in a container on the same network, so no host port forwarding is involved.

Cases (bench/k6/scenario.js), all unsigned imgproxy URLs:

Case	URL options	Source
jpeg-400-jpeg	rs:fit:400:400 → JPEG	photo.jpg, 3072x2048 baseline JPEG, Q85
jpeg-400-webp	rs:fit:400:400 → WebP	same
jpeg-400-avif	rs:fit:400:400 → AVIF	same
png-alpha-400-webp	rs:fit:400:400 → WebP	alpha.png, 1280x853 RGBA
gif-anim-240-webp	rs:fit:240:240 → animated WebP	anim.gif, 480x320, 12 frames
jpeg-1200-fill-jpeg	rs:fill:1200:1200 → JPEG	photo.jpg
info	/info (size, format, dimensions, EXIF, IPTC, XMP)	photo.jpg; xtandard-image only, /info is an imgproxy Pro feature

The fixtures are generated by bench/fixtures.sh from the procedural photo generator in testdata/gen (CC0, no third-party images); they are not committed.

Each case is measured two ways:

1. Fixed arrival rate (k6 constant-arrival-rate) at three rates per case, the same for both servers: about 25 %, 50 % and 90 % of the lower of the two throughput ceilings on the reference machine. This shows latency (p50/p95/p99) at a load the server can sustain, and what happens close to saturation (queueing, dropped iterations, 429/503 load shedding).
2. Throughput ceiling: closed loop with a fixed number of concurrent requests (default 2 x workers), reporting requests per second and latency.

While k6 runs, docker stats samples the loaded server’s container about once a second: CPU (100 % = one core) and memory (cgroup usage as reported by Docker, close to RSS). The host’s 1-minute load average before each run is recorded as a noise indicator.

Matched settings

The servers’ defaults differ; the benchmark pins the ones that change the work done so both do the same job (bench/compose.yml has the full list):

Setting	Both servers	imgproxy default	xtandard-image default
Workers	host cores (BENCH_WORKERS)	2 x cores	cores
Quality	JPEG 80, WebP 80, AVIF 60	80, 79, 63	80, 80, 60
JPEG	baseline	baseline	progressive
AVIF speed	libvips effort 1 (IMGPROXY_AVIF_SPEED=8)	effort 1	effort 4
WebP effort	4	4	4
Animation frames	up to 256	1 (first frame only)	256
Origin cache	off	none	256 MiB LRU of source bytes
Metadata, ICC	stripped	stripped	stripped
libvips	concurrency 1 per image, operation cache off	same	same

Things deliberately left at each server’s default: memory allocator (xtandard-image preloads jemalloc with MALLOC_ARENA_MAX=2; imgproxy’s image uses its own build), Go GC settings, queue behaviour (imgproxy queues without limit by default; xtandard-image queues up to workers x 16 requests for up to 5 s and then answers 429/503). The last one shows up near saturation.

Method

bench/run.sh starts the origin and both servers and keeps them running, but only one server is under load at a time. For every case it warms both up (5 s closed loop each), then for every rate and for the ceiling runs each server back to back, BENCH_REPEAT times (default 3 x 15 s). Interleaving means slow changes in background load on the host hit both servers alike instead of favouring whichever ran first. The tables report the median of the repetitions (and the p99 range across them).

What this does not measure: cold starts, large sources over a slow network, result caching (a CDN does that in production), or behaviour with memory limits. The load generator shares the machine with the servers; on a laptop the numbers are relative, not absolute capacity figures.

Reproducing

task release:build                                   # or XI_APP_BASE=xtandard-base:dev task release:build
bench/run.sh                                         # ~1 h with the defaults
BENCH_CASES="jpeg-400-webp info" BENCH_REPEAT=1 bench/run.sh   # a subset

Variables: XI_IMAGE (default xtandard-image:dev), IMGPROXY_IMAGE, BENCH_WORKERS (default: CPUs Docker sees), BENCH_SERVERS, BENCH_CASES, BENCH_DURATION, BENCH_REPEAT, BENCH_WARMUP, BENCH_MAX_VUS, BENCH_MODES (rate max). The rates live in the case table at the top of bench/run.sh; scale them to the hardware. It needs Docker (compose), jq and awk; fixtures are generated on first use in the base image builder (XI_BASE_IMAGE, as for task docker:*).

Output goes to bench/out/ (gitignored): one k6 summary per run, results.jsonl with every run (k6 numbers, status counts, CPU, memory, host load) and report.md with the machine and image metadata and the median tables. To publish a run, copy report.md into bench/RESULTS.md and write the interpretation by hand: every number in RESULTS.md must come from a run whose results.jsonl you have.

Before a release, rerun it on the release image and update RESULTS.md (docs/RELEASE.md). For figures you want to rely on, use a quiet dedicated Linux machine: no other containers, no builds, and the load average near zero before starting.

k6 (AGPL-3.0) runs as an unmodified external tool from its own image; nothing from it is linked, vendored or shipped (ADR 0006).