Ramjet-ingress: an ingress controller where a config change doesn't reload anything

Announcements Ecosystem
, ,
1

I’ve been building an ingress controller in Rust and would like feedback from people who operate ingress at scale. It implements the same Ingress API surface as ingress-nginx (same path semantics, same canary annotations), but the design starts from a different premise.

The premise: in ingress-nginx, a change to an Ingress spec regenerates nginx.conf and reloads workers. That reload costs you connections, and the cost scales with how much traffic you’re carrying — which is backwards. Here, the controller compiles an immutable route table and publishes it by storing one pointer (ArcSwap). Readers do a single atomic load per request. There is no reload, no drain, no config file. Load-balancer counters deliberately live outside the table, so a config change doesn’t make backends forget what they’re currently serving.

What that buys, measured

Single-node k0s on an EC2 t3.xlarge, both controllers installed in the same cluster, load generated on-box. Read the caveats below before quoting any of this.

Configuration churn — one Ingress spec change every 2 seconds for 120 seconds, under constant load, with 100 idle keep-alive connections held open:

keep-alives survived HTTP errors reloads
ramjet-ingress 100 / 100 0 n/a (90 table publishes)
ingress-nginx 0 / 100 1,829 66

Reproduced twice independently, and on a different kernel/distro than an earlier run. Credit where due: under endpoint-only churn ingress-nginx reloads zero times (its Lua balancer handles that path) and also keeps 100/100 — so this is specifically about Ingress spec changes.

Steady state, c64, median of 3 interleaved 30s runs:

RPS CPU per request RSS
ramjet (io_uring engine) 12,355 69 µs 15.2 MiB
ramjet (default engine) 10,858 100 µs 10.5 MiB
ingress-nginx 7,971 187 µs 67.6 MiB
no proxy (baseline) 22,234 — —

Time from kubectl apply to the route serving traffic: 372 ms vs 2,762 ms (median, 10 trials).

Caveats, because they matter: one 4-vCPU burstable box with the load generator sharing those cores, so nobody was measured in isolation and the engine margins are compressed. ingress-nginx ran at chart defaults with access logging disabled (a choice that favours it) and no memory limit, while ramjet ran under its chart’s 256Mi cap. Full methodology, raw JSON, and the harness are in the repo — including two claims from an earlier run that this one retracted after arm-order rotation showed they were measurement artifacts.

Features that came out of the immutable-snapshot design

  • Instant rollback. POST /admin/rollback {"generation": 41} republishes an earlier table and freezes publication; DELETE resumes. Sub-millisecond, no restart. The controller keeps watching and recording while pinned.
  • Semantic audit trail. Every publish emits a structured diff — routes added/removed, backends changed with endpoint deltas, certs rotated — as logs, Kubernetes Events, and an optional webhook. “Why did traffic shift at 14:32” is answerable.
  • Canary auto-promotion, built in. Annotate a canary Ingress and it ramps 5→10→25→50→100 on healthy windows, holds when there isn’t enough traffic to judge, and rolls back to weight 0 on an error-rate or latency breach, writing the reason onto the object. No extra component to install.
  • Traffic mirroring. Shadow live traffic to another Service, fire-and-forget, with a hard invariant that the primary is never slowed — verified with a dead shadow backend (40/40 served, no added latency).
  • Two data planes, one binary. The default is hyper/tokio. --engine uring runs a completion-based reactor (io_uring on Linux, kqueue on BSD). One :443 listener serves both: the ClientHello is inspected before a config is chosen, and HTTP/2 clients are handed to the other engine mid-connection with their bytes intact.
  • HTTP/3 over QUIC, experimental and off by default.
  • Per-route live stats over a JSON admin API, plus ramjet-top — a terminal cockpit showing per-route RPS, error rates, latency and the generation timeline.

What it does not do yet

  • No leader election. The chart hard-codes replicas: 1. Multiple replicas would fight over Ingress status writes. This is the biggest gap.
  • gRPC works via backend-protocol: GRPC (h2c to the pod), but GRPCS and TLS-to-upstream are not implemented.
  • io_uring is usually blocked in practice. containerd’s default seccomp profile denies io_uring_setup, so on a stock cluster --engine uring logs a warning and falls back to the default engine. Turning it on means seccompProfile: Unconfined — a real security tradeoff, and the reason the default engine is the one that ships enabled.
  • An idle keep-alive connection costs ~20 KiB against nginx’s ~4.4 KiB. Attributed and documented; ~16 KiB of it is a floor in the HTTP library’s buffer sizing.
  • The io_uring engine wins throughput and median latency but consistently loses p99 by ~7%. Unexplained, so no tail-latency claims are being made for it.

Alpha. It passes 1,250 tests, runs a full e2e against a real cluster, and is serving real traffic behind Cloudflare on my test box — but it hasn’t been through anyone else’s production.

What I’d most like feedback on: is the zero-reload property something that has actually bitten you in production, or is it a problem people have already engineered around? And for anyone running ingress on bare metal — is hostNetwork on :80/:443 the right default, or do you expect NodePort?