Bound parallel fanout and merge evidence deliberately

Fanout sends independent assignments to several workers and fan-in combines their results. It is attractive for searching separate document collections, checking independent files, or evaluating competing hypotheses. If twenty requests each create eight workers, the service suddenly has one hundred sixty outstanding calls. A per-request semaphore of eight does not protect the shared upstream service. Production admission control needs a global limit as well as per-user fairness and per-run budgets.

Distinguish queued work from active work. A task can exist in memory while waiting for a slot, consuming storage but no outbound connection. If queued work grows without a bound, overload merely moves from the model provider into your process. Set a maximum queued count, an expiration time, and a policy for rejecting or deferring low priority work. These choices determine user-visible behavior during load spikes.

Python's asyncio.TaskGroup provides a scope for related asynchronous tasks. Exiting the scope waits for its tasks; a non-cancellation failure triggers cancellation of remaining tasks and reports grouped exceptions. Cleanup belongs in finally blocks, and cancellation should generally propagate. This differs from collecting every result with gather(return_exceptions=True), where partial outcomes become ordinary values the caller must inspect. [Python coroutine and task documentation](https://docs.python.org/3/library/asyncio-task.html).

Neither behavior is universally correct. A release gate may require all checks and abort if any checker crashes. A literature review can still benefit from three successful collections when a fourth is unavailable. Encode that distinction in the coordinator contract. Swallowing an exception and returning an empty list silently converts unavailable evidence into evidence of absence, which can be much worse than a clearly incomplete report.

Give each result a worker identifier, input version, claim, source references, confidence rationale, and limitations. Confidence scores from different workers are not automatically calibrated. A 0.9 from one prompt cannot be averaged with a 0.7 from another and interpreted as a probability. Prefer decisions based on evidence completeness, source quality, and independently checked constraints before introducing numerical weighting.

Suppose workers A and B both report that endpoint X was removed, citing the same changelog. Worker C reports that X remains available, citing a tested compatibility shim. The first two reports are correlated evidence, not two independent votes. A useful merge groups by claim and provenance, then flags the conflict for resolution. Majority vote would conceal the important implementation detail. Preserve unresolved disagreement in the artifact instead of smoothing it into a unanimous narrative.

Workers often finish in a different order on each run. If the coordinator appends paragraphs as results arrive, both the report structure and later model context may change with network timing. Assign stable task identifiers, collect results, and sort by an intentional key before synthesis. This preserves concurrency while making the aggregation reproducible. The example uses an input index rather than completion time to order results.

Some operations can stream partial results safely. A dashboard can show each completed file independently, provided that the final summary waits for the required set and labels missing checks. An aggregate such as average risk cannot be treated as final before its denominator is known. Define which fields are provisional, which are commutative, and which require a barrier. A reducer that is associative and commutative is easier to parallelize, but many prose synthesis operations have neither property.

Imagine six independent reviews, each using twelve hundred input tokens and three hundred output tokens. The worker stage uses nine thousand tokens before the coordinator reads their outputs. If each worker receives the full ten thousand token conversation instead, input cost grows sharply without guaranteeing better judgments. Send a focused shared brief plus relevant artifacts, then let workers request narrowly scoped missing context.

Record token use per worker alongside useful evidence produced. A seventh worker may contribute a genuinely different source collection, or it may restate the same six conclusions. Stop rules can use coverage rather than agent count: every required subsystem checked, every claim source attached, every conflict resolved or explicitly marked. Bounded fanout makes resource use predictable; disciplined fan-in makes the parallel work worth doing. Both are necessary for a design that remains understandable when one worker fails.

This Python 3.12 compatible teaching simulation counts words instead of calling agents. At most two worker bodies hold the semaphore, and input indices determine output order. Inject a worker exception to inspect TaskGroup behavior; use a real shared limiter for a service-wide cap.