Commoditech Let's talk
ServicesModelsCase StudiesBlogFAQCareersContact
🇵🇱 PL🇬🇧 EN🇩🇪 DE🇫🇷 FR🇪🇸 ES
Let's talk
← Back to the articles

LLM orchestration: inference cost and agents

Published: 27 September 2026 • B2B guide / LLM

Five agents on one helpdesk ticket. A planner, a researcher, a writer, a critic, a verifier. Each calls the model in series. The next call’s input contains the previous call’s output. The p99 latency (the 99th percentile of response time) is the sum of five calls, not the maximum. The invoice is a sum too. A ticket that costs 1500 tokens in one call costs 10,500 tokens in that crew, before anyone says “wrong”.

LLM orchestration (large language model) is the decision about which call is allowed to exist. Inference cost is the product of call count, tokens and retries. It is not the name of the framework on the slide.

Three patterns that briefs collapse into “agents”:

  • Router. One small call decides whether the expensive model runs at all. Ong et al., RouteLLM: Learning to Route LLMs with Preference Data (arXiv:2406.18665, 2024). The abstract: in some cases cost falls by more than 2×, without losing answer quality. The README of the same project (lm-sys) states a stronger figure: up to 85% cheaper while keeping 95% of GPT-4 quality on the MT Bench benchmark. That is a model pair and that benchmark. It is not your traffic.
  • Cascade. A cheap model answers. The expensive one starts only when a verifier rejects the result. Chen, Zaharia and Zou, FrugalGPT (arXiv:2305.05176, 2023). Table 3, matching the best single model: HEADLINES 98.3% cheaper (cost 33.1 versus 0.6, GPT-4 baseline), OVERRULING 73.3% (9.7 versus 2.6, GPT-4 baseline), COQA 59.2% (72.5 versus 29.6, GPT-3 baseline, not GPT-4). The authors describe a range from 50% to 98%. The abstract: up to 98% cheaper, or 4% better at the same cost. A different operating point in the same paper, a budget of one fifth of GPT-4’s cost on HEADLINES: about 80% cheaper and 1.5% higher quality. That is not the same percentage as 98.3%.
  • Role graph. Several agents, one tape, different tools. It makes sense when a step can be stopped by a condition. It does not make sense as five job titles on ticket classification.

The sentence missing from the “multi-agent” slide

The model is not the loop. The loop is the call graph. Without a step limit and a quality sample on your traffic, the percentage on the slide is someone else’s benchmark.

1. Anatomy: why the fifth call is the expensive one

Cost does not grow linearly with the number of roles if each role receives only its own instruction. In a typical graph it does grow, because context is appended. The arithmetic below assumes an 800-token system prompt, a 400-token ticket and 300 tokens of output per step. Without a prefix cache (a cache of the shared start of the request) every call pays for that start again.

input  = 1200 + 1500 + 1800 + 2100 + 2400 = 9000
output = 5 × 300                          = 1500
total  = 10500 tokens per ticket
one call = 800 + 400 + 300                = 1500
ratio  = 7

The factor of seven is not an invoice. It is addition you can repeat on paper. One retry of the last step adds another 2400 input tokens and 300 output tokens, 2700 in total. The tail comes from the retry, not from the first call.

On a serial path, p99 is a sum. An assumption, not a reading from your monitoring: if one call takes 2 seconds, five calls take 10 seconds. Parallel branches shorten the clock only to the slowest branch plus the call that merges them. You pay for every branch, including the one whose result is discarded.

A server you run yourself does not turn five calls into one. Kwon, Li et al., PagedAttention and the vLLM system (SOSP 2023, arXiv:2309.06180), make a single token cheaper: on their 2023 setup, throughput of popular models rises 2–4× versus FasterTransformer and Orca, at the same latency level. The gain is larger on longer sequences. That is packing of the key-value cache (KV cache). It is not a reason to add a fourth agent.

Cemri, Pan, Yang et al., Why Do Multi-Agent LLM Systems Fail? (arXiv:2503.13657, March 2025 version). The MAST taxonomy: 14 failure modes in three baskets, annotator agreement kappa 0.88. The baskets:

  1. Specification and design. An agent ignores the task or its own role. It does not know the stop condition. It loses the conversation history.
  2. Misalignment between agents. One does not ask. Another withholds what it already knows. The conversation leaves the task or starts over.
  3. Verification and termination. Stopping too early, no check, or a check of the wrong thing.

An example from their ChatDev trace. The task: chess in algebraic notation, moves such as “Ke8”. The system returns a game that reads square coordinates. The verifier checks that the code compiles. It does not check the rules of chess. Compilation is a cheap test. It is not a test of the task.

2. Which numbers transfer, and which do not

Table 4 of the same paper, the ProgramDev set (their 32 “write a program” tasks, not HumanEval). ChatDev in the baseline topology finishes the task in 25.0% of runs. A better prompt: 34.4%. A new graph topology: 40.6%. In the text the authors describe an improvement on the order of 14 percentage points and say plainly that this does not close the failures. HumanEval on the same setup moves from 89.6% to 91.5%. A benchmark that is already high will not show the cost of bad orchestration.

A second experiment, AG2 on GSM-Plus, six repetitions. With GPT-4, a better prompt alone goes from 84.75% to 89.75%. A new three-role topology stays at 85.50%, and a Wilcoxon test against the baseline has p = 0.4, so that difference cannot be separated from noise. With GPT-4o both the prompt and the new topology beat the baseline (p = 0.03), both near 89%. “Add an agent” is not a fix that wins regardless of the model.

A model judge (LLM-as-a-judge, scoring a trace with another model) is not a free audit. In their setup, o1 with examples in the prompt has accuracy 0.94 and kappa 0.77 against humans. Without examples kappa falls to 0.58. You can screen traces with it. You cannot sign a quality clause with it.

Architectural conclusion

First a stop condition and a check that knows the task. Then a router or a cascade, calibrated on your sample, not on MT Bench. A role graph only after the steps have different tools and the loop can be cut. vLLM lowers the price of a token on your own GPU. It does not lower the number of calls.

3. Worked example: 10,500 tokens before anyone says “wrong”

From engineering practice: the sum on paper, before you buy a framework

A layout that repeats whenever an “agent crew” sits on a ticket inbox (helpdesk, one model via API, five roles in series). This is not a report from a named deployment. It is the arithmetic from section 1.

  • Happy path, five roles, no prefix cache: 10,500 tokens.
  • The same ticket in one call with tools (function calling: the model invokes a function itself): 1500 tokens. Ratio 7.
  • One retry of the last step: plus 2700 tokens. Nobody has measured how often your critic will force it. MAST only says verification is sometimes empty or checks the wrong thing.
  • The clock, assuming 2 seconds per call and no parallelism: 10 seconds versus 2 seconds. The user’s p99 sees the sum.

A cut that still has a check: one call to a cheap model, an acceptance threshold, the expensive model only after a rejection. That is the FrugalGPT cascade, not a fifth role named “senior”. Take the threshold from your own tickets. The 98.3% on HEADLINES belongs to that dataset and to a 2023 model pair.

Numbers from papers do not move 1:1 onto your API invoice. The split does: first the number of calls, then whether the verifier can say “wrong”, then the price of a token. Starting with vLLM on a five-role graph packs the same loop more cheaply.

4. Decision table: what to call, and what not to automate

Approach Complexity p99 latency Inference cost Load on the team When to use it
One call plus tools Low One call Lowest in the arithmetic above (1500 tokens) Low, as long as the output schema is strict Classification, extraction, JSON. A schema or a rule exists
Router: strong or weak model Medium A small call, sometimes a second RouteLLM: more than 2× in the abstract. Up to 85% and 95% of GPT-4 quality on MT Bench in the project README The threshold has to be calibrated on your traffic Mixed traffic and an evaluation set, not a demo
Cascade with a verifier Medium A sum, until the verifier accepts FrugalGPT: 59.2%, 73.3% or 98.3%, depending on the dataset A verifier that knows the task An automatic check exists: a test, a schema, a rule
Multi-role graph High Sum of steps. Tail on retry Grows with appended context. In the arithmetic above, 7× versus one call A trace, a step limit, an eval set Steps have different tools and can be stopped
Your own server (vLLM) High Depends on the queue, not on the role count A cheaper token. The same number of calls. 2–4× throughput on the Kwon 2023 setup A GPU and an on-call rotation High volume, or data that cannot go to someone else’s API

AI / LLM / RAG (retrieval-augmented generation: an answer grounded in retrieved documents): specialist rate 180–250 PLN/h, client rate 240–350 PLN/h. About 35–50% more than Python for ordinary backend work. Supplier margin in an open model: 10–25%. A market map, not a quote. Breakdown: what IT body leasing costs in 2026. Document retrieval is a different loop from the agent call count. Model retraining is a third: MLOps services.

5. Anti-patterns the framework tutorial skips

  1. Five roles for ticket classification. The framework is up. The evaluation set is not. ChatDev finishes 25.0% of ProgramDev tasks. A better topology reaches 40.6%, and the authors write that this is not enough for production.
  2. A critic with no stop condition. MAST: no termination condition, and verification that checks compilation instead of the task. An empty check is cheap and catches nothing. A check that always demands a retry adds the most expensive call to the tail.
  3. A percentage from someone else’s dataset written into the contract. 98.3% is HEADLINES. 85% is MT Bench and the model pair in the RouteLLM README. On your traffic the threshold is calibrated again. Otherwise the saving is silent: quality drops, and the API returns no error.
  4. Prompt logs with personal data in someone else’s tracing tool. A data-processing agreement does not keep up with an SDK that sends ticket text “for debugging”. The trace stays in your project.

6. Playbook: whom to hire, and in which order

Do not start with a fifth role. Start with the question of what holds the invoice: the number of calls, a missing check, or the price of a token on your own GPU.

  1. The API is already there, you already use one model, and you lack a step limit and an evaluation set. That is LLM developer staff augmentation: one person, your standup, your Definition of Done. Staff augmentation here means renting a specialist into your team, usually billed by time (time and materials, T&M). We do not sell a named bench for Monday morning.
  2. Data cannot go to an API, and the trace, the eval set and the server do not exist. One person will not stitch that in a sprint. A squad: someone for orchestration, someone for the check, someone for serving. That is closer to IT team leasing than to “we will buy one more prompt person”. When a squad, and when one role: team leasing vs staff augmentation 2026.
  3. The API key and the repo do not leave for a laptop. The contractor works in your repository and on your key. Exporting logs “to reproduce the bug faster” is a collection of customer tickets. Clauses: contracts, margins, copyright.
  4. Ramp-up. A person joining an existing team: first profiles within days, start after your interviews and the contract. A squad from zero: weeks, because you are stitching permissions and the production stop condition. A promise of “three senior LLM people from Monday” is either a CV or a bench we do not keep.

Commoditech has done T&M and permanent hiring from Warsaw since 2012. The network has 80+ specialists. We do not keep them idle on a bench. A brief for staff augmentation or for a permanent hire (a success fee of 15–25%, no hire means no fee) can also be filed from the editor, through MCP for AI agents (Model Context Protocol, the protocol an assistant in the editor uses to call tools). A human prices the specific stack. The bands are in the rates article, not in an automatic reply. Contact: the form.

FAQ

How is a router different from a cascade and from an agent graph?

A router picks a model with one small call and the decision ends there. A cascade runs a cheap model and calls the expensive one only after a verifier rejects the answer. An agent graph appends further roles and usually appends their output to the next context. In the arithmetic in this article, five roles without a prefix cache are 10,500 tokens against 1500 for one call. RouteLLM and FrugalGPT cut cost on their datasets. A percentage from their tables does not go into a contract without a sample of your traffic.

Will FrugalGPT’s 98% saving transfer to our helpdesk?

Not as a number for the budget. 98.3% is the HEADLINES dataset, cost 33.1 versus 0.6, at GPT-4 quality. OVERRULING gives 73.3%, COQA 59.2%, and there the baseline is GPT-3. The authors describe a range from 50% to 98%. On HEADLINES, at a budget of one fifth of GPT-4, a separate operating point is about 80% cheaper and +1.5% quality. A cascade needs labelled examples from a distribution close to production. Otherwise the verifier threshold belongs to someone else.

What does LLM developer staff augmentation cost in Poland in 2026?

On the AI / LLM / RAG market map: specialist 180–250 PLN/h, client 240–350 PLN/h, about 35–50% more than Python for ordinary backend work. Supplier margin in an open model: 10–25%. This is not a quote. A human prices the brief. Details: 2026 rates.

When is it one person, when a squad, and can the work sit under an NDA?

One person, when the API and the model are already up and what is missing is a step limit, a router threshold and an evaluation set. A squad, when data cannot go to someone else’s API and nobody has a server, a trace or a check. Yes, under a non-disclosure agreement (NDA): the contractor works in your repository and on your key. Exporting prompt logs to a laptop is a collection of tickets. Clauses: contracts, margins, copyright.

Sources