Sharing one key pool across two machines without making it a dependency

The problem

Free-tier provider keys are individually useless and collectively fine. Fifteen of them, rotated properly, will absorb a real workload. Doing that rotation by hand in every calling script is not maintainable, so it belongs behind one OpenAI-compatible endpoint.

The constraint

The same pool is used from two machines at once. Naive round-robin per process means both hosts independently hammer the same key and trip the same limit.

Provider quotas are also enforced per model, not per key, so a key that is rate-limited on one model is often still perfectly usable on another. Cooldowns keyed only by key throw away capacity you are still entitled to.

The decision

The two hosts coordinate through a shared store, using one atomic rotation ticket plus cooldowns keyed by provider, key and model.

The decision worth defending is what happens when that store is unreachable. It sits behind a circuit breaker and is an optimisation, never a dependency. If it goes away, rotation silently falls back to per-process and every request still completes. A coordination layer that can take down the thing it coordinates is worse than no coordination layer.

What the telemetry taught

The first telemetry design logged request and response bodies. It reached 1.4 GB in five weeks on a machine with 19 GiB of total disk, which is a straightforward way to take out every other service on the box.

It now writes one row per request with no bodies, a size cap and a retention window. The useful questions, which are about rate limits and cost, are all answerable from metadata.

The result

Runs on two machines against roughly fifteen keys across two providers, with automatic failover and model fallback. It is also what generated most of the bulk text in the research behind this site, which is a fair test of whether it works.