FastAPI front doors, container workers, and capacity limits

A production service has at least two responsibilities: deciding whether to accept work and performing accepted work. A FastAPI application can implement authenticated submission and status endpoints, while queue workers execute the agent workflow. The API should validate request shape, tenant scope, job budget, and an idempotent submission key before accepting a job. A quick HTTP response is useful only if accepted work has a durable identity and an honest status.

FastAPI provides BackgroundTasks for work performed after returning a response. Its documentation also discusses using larger task systems for work that need not run in the same process. Do not infer durable distributed job guarantees from that convenience interface. Long-running workflows that must recover after process loss need persisted state and a worker delivery design. Returning accepted before safely recording the job creates a false promise to the client.

Use a capacity model before adding replicas. In a stable system, average work in progress is approximately arrival rate multiplied by average time in the system. If two tasks arrive per second and each occupies a worker slot for six seconds, the workload requires about 12 concurrently occupied slots on average before adding headroom. A single worker process may provide several slots for I/O-bound work, but CPU-bound work has a different resource profile.

If each of four workers has four slots, there are 16 slots. At six seconds per task, the rough maximum service rate is 16 divided by six, or 2.67 tasks per second under the model's assumptions. Running near that limit can produce large queues when durations vary. The calculation assumes comparable tasks and no tighter downstream bottleneck, so validate it with load tests and latency distributions rather than treating it as a sizing guarantee.

Additional workers can increase contention instead of throughput if the service is limited by a provider quota, a database connection pool, or a restricted tool. Suppose every task makes five model calls and the available allowance is ten calls per second. Then the model-call budget supports at most two tasks per second on average, regardless of worker count, before retries and other traffic are considered.

Add admission control, bounded queue length, per-tenant concurrency, and a policy for deadlines that will expire before execution. Monitor oldest job age as well as queue depth: ten long tasks and ten short tasks create different waiting times. CPU can be low while I/O-bound workers are saturated on external requests. Kubernetes autoscaling supports resource and custom metrics through documented interfaces, but the metric and stabilization policy must match the bottleneck you actually observed.

Docker images package application code and runtime dependencies, not live credentials or the full mutable development environment. Use a small appropriate base, pin dependency inputs according to the team's update policy, and rebuild deliberately for security and compatibility updates. Docker's build guidance covers cache behavior, build context, stages, and other reproducibility practices. A dependency cache can accelerate builds while still producing an image from reviewed inputs.

Separate build-time inputs from runtime configuration. Secrets should arrive through the approved runtime mechanism or build-secret facility when truly required, rather than being copied into image layers. Run workers with the restricted identity, mounts, network access, and resource limits designed earlier. A container image that starts successfully demonstrates packaging, while the effective runtime configuration determines what the worker can reach and how much it can consume.

More replicas introduce coordination concerns: lease ownership, shared checkpoints, duplicate deliveries, and graceful shutdown. A worker receiving a shutdown signal should stop accepting new jobs, finish or checkpoint current work within its allowance, and release or let leases expire according to the protocol. The API should expose readiness separately from a mere process-alive check so traffic is not sent to a component that cannot accept work.

The example computes worker and provider ceilings using invented workload numbers. It is an arithmetic model, not a load generator or autoscaler. Use it to identify the likely bottleneck, then measure actual service time, queue delay, retry traffic, and downstream quotas. A production capacity report should state workload mix, concurrency settings, accepted arrival rate, and the conditions under which the service starts rejecting or deferring work. That makes scaling decisions accountable to observed behavior.

The arithmetic assumes homogeneous tasks, six seconds of slot occupancy, and five provider calls per task. The 75% target is an invented planning assumption. Real capacity depends on service-time variability, quotas, retries, batching, and queue policy; no cloud resources are created.