Grok CLI and terminal agents: MCP and contracting

Q: Does 80% on SWE-bench Verified mean the agent closes 80% of a senior engineer's tickets?

No. The score is a triple: model, agent scaffold, and benchmark hygiene. On 13 August 2024 OpenAI reported GPT-4o resolving 33.2% of the 500 SWE-bench Verified tasks. On 23 February 2026 the same vendor wrote that Verified progress had slowed from 74.9% to 80.9% in six months, and that in an audit of 27.6% of the set at least 59.4% of the reviewed failures have tests that reject a correct fix. On 8 July 2026, on the 731-task public split of SWE-bench Pro, the frontier moved from 23.3% to 80.3% in eight months, while humans marked 249 tasks (34.1%) as broken. A leaderboard percentage does not belong in a contract.

Q: Stdio or a remote MCP server on a company network?

Stdio runs a process on the machine that hosts the agent. It does not open a port. From the MCP revision of 2025-03-26 onward, including 2025-11-25, the remote transport is Streamable HTTP: the server has an address, so it needs authentication and a separate allowlist for tools that write. Reading an issue or a CI status can go remote. A production write and a networked shell do not inherit that grant.

Q: What does it cost to contract an AI-agent engineer or an LLM developer in Poland in 2026?

The published AI / LLM / RAG rate map: specialist 180–250 PLN/h, client rate 240–350 PLN/h, about 35–50% above ordinary backend Python. Supplier margin in the explicit model is 10–20%. That is not a quote for a terminal-agent role. A brief with a step cap, an eval set, and an MCP allowlist is priced by a person. The breakdown is in the 2026 IT staff-augmentation rates article.

Q: Can a terminal agent work in a repository covered by an NDA?

Yes, if the API key, the session transcript, and the shell stay inside your boundary. The contractor works in your repository and on your key. Exporting a session to a laptop copies the diff and sometimes a secret the agent already read. A shell without network, secret files excluded, and no MCP tool that writes to production are technical terms of the contract, not a footnote in the tool's marketing.

Published: 26 October 2026 • Technical article

A terminal agent is given a repository and the order “fix this test”. The loop does not end after one model call. Each step appends a search result, a file window, and stdout to the context. Take a prefix of 2,000 tokens (instruction, AGENTS.md, the task) and 1,000 tokens of new observation per step, with no prefix cache. Twelve steps send 90,000 input tokens in total. One call with the same prefix and one observation sends 3,000. The ratio is 30. The last three steps are 36,000 tokens, 40% of the bill. The first three are 9,000, 10%.

That is addition you can repeat on paper. It is not an invoice from a named rollout. The invoice looks the same when nobody set a step cap and the stop condition is the model saying “done”, not a green test.

The second stake is harder than the token. An agent that reads an issue, a README, or a tool result also executes whatever a third party put there, and it has a shell underneath. Debenedetti, Zhang, Balunović, Beurer-Kellner, Fischer, and Tramèr, AgentDojo (arXiv:2406.13352, 2024): 97 tasks, 629 security test cases. Models fail a share of the tasks even with no attack. A prompt injection carried in data returned by a tool breaks some security properties and not others.

⚡ Architectural point

A leaderboard score is a triple: model, scaffold, and the hygiene of the test set. The API bill is the sum of a growing context. The control is a stop condition that knows the test, plus the list of tools the agent is allowed to call. The name of a binary on $PATH is none of those three.

1. Anatomy: host, loop, MCP

A terminal agent is a loop, not a chat. The host keeps history, calls the model, runs an action, and appends the observation. Grok CLI – the product xAI documents as Grok Build, command grok – is such a host. The docs (x.ai/docs/build/overview, product page x.ai/cli) list an interactive TUI, headless mode grok -p, a plan before edits, subagents, a git worktree (-w), AGENTS.md, skills, hooks, and MCP servers. The model is selected with -m and grok models. grok inspect shows the configuration discovered in the working directory. Authentication is a browser login or the XAI_API_KEY variable. The npm package is @xai-official/grok.

None of that is a resolve rate. The public docs do not assign a SWE-bench score to the grok binary. A score, if someone measures one, belongs to a specific model, a step cap, and a test harness.

Model Context Protocol

The Model Context Protocol (MCP) separates the host from the tools. The specification, revision 2025-11-25 (modelcontextprotocol.io), sits on JSON-RPC 2.0. The host (Grok CLI, or another editor client) holds the client. The server exposes three things: tools (functions the model may call), resources (data to read), and prompts (templates). Lifecycle: initialize negotiates capabilities, tools/list returns schemas, tools/call runs one tool. On the client side the spec also defines sampling (the server asks the host to call a model) and a list of directories (roots).

The local transport is stdio: a process on the same machine, no port. The remote transport from the 2025-03-26 revision, kept in 2025-11-25, is Streamable HTTP. The earlier HTTP+SSE setup is no longer the primary remote transport. One MCP server can be attached to several hosts. It does not replace the decision about which tool is allowed to write.

{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/call",
  "params": {
    "name": "get_issue",
    "arguments": {"key": "PAY-1842"}
  }
}

The message shape is from the MCP tools specification (revision 2025-06-18 and later revisions that keep it). The name get_issue is an example of a read. It is not a list of Grok CLI tools.

step = 0
context = [system, AGENTS.md, task]
while step < LIMIT and not target_test_green():
    action = model(context)
    if action.kind == "mcp":
        result = server.tools_call(action.name, action.arguments)
    elif action.kind == "shell":
        result = sandbox(action.command)  # network, .env, and git push live here
    context.append(truncate(action, result))
    step += 1

A shell-only interface loses to an interface built for the model, even when the weights stay the same. Yang, Jimenez, Wettig, Lieret, Yao, Narasimhan, and Press, SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (arXiv:2405.15793, NeurIPS 2024). On SWE-bench Lite, with GPT-4 Turbo: an agent limited to an ordinary shell resolves 11.00% of tasks, SWE-agent resolves 18.00% (54 of 300). A separate file-viewer ablation: a 100-line window scores 18.0%, pasting the whole file scores 12.7% (down 5.3 points). In the editor ablation the shell-only path falls to 10.3% against 18.0% for an edit that runs a syntax lint. cat of a whole file floods the context with text the model will not use in the patch.

On full SWE-bench the same SWE-agent with GPT-4 Turbo resolves 12.47% (286 of 2,294) at an average cost of $1.59 per instance in the prices reported in the paper. The abstract rounds pass@1 to 12.5%. The other 87.7% in that abstract is HumanEvalFix, short function repairs. That is not the same set as 2,294 GitHub issues.

2. Numbers you can recompute – and numbers you must not carry over

Jimenez, Yang, Wettig, Yao, Pei, Press, and Narasimhan, SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv:2310.06770, ICLR 2024). The set: 2,294 problems from real issues and pull requests, 12 Python repositories. The model receives the repository and the issue text and must return an edit that passes hidden tests. The best model in that paper, Claude 2, resolves 1.96% with no tool loop. That is the ceiling of that non-interactive setup. It is not “the agent from the year before the interface”.

Putting 1.96% next to 12.47% mixes the model, the year, and the scaffold. The ablation that holds the model fixed (GPT-4 Turbo) and changes only the interface is narrower: 11.00% against 18.00% on Lite. Seven percentage points on 300 tasks. Enough to stop treating raw bash as a neutral API for the model. Not enough to write “the agent resolves issues” into a proposal.

OpenAI, 13 August 2024, SWE-bench Verified. A review of 1,699 SWE-bench tasks, three independent annotators, 500 remain. The “easy” slice: 196 tasks under 15 minutes. The “hard” slice: 45 tasks over one hour. GPT-4o resolves 33.2%. The Agentless scaffold doubles its earlier 16% on the original SWE-bench. The annotation report: 38.3% of samples had an underspecified statement, 61.1% had tests that can fail a valid solution, and 68.3% were removed for underspecification, an unfair test, or another defect. Verified is cleaner than the full set. It is still someone else's Python issues, not your backlog.

OpenAI, 23 February 2026, Why we no longer evaluate SWE-bench Verified. After the jump, the frontier moves from 74.9% to 80.9% in six months. The authors audit 27.6% of the set, the tasks models often fail, and find that at least 59.4% of the reviewed items have tests that reject a functionally correct patch. A separate problem is contamination: the model saw the tasks in training. OpenAI stops reporting Verified and points to SWE-bench Pro.

OpenAI, 8 July 2026, Separating signal from noise in coding evaluations. On the public split of SWE-bench Pro, 731 tasks, frontier pass rate rises from 23.3% to 80.3% in eight months. The same text: an automatic pass flags 200 tasks (27.4%) as broken, a human campaign flags 249 (34.1%). Read the jump and the broken-task rate together. Otherwise 80% belongs to the board, not to the team.

The vendor table in the GPT-5.2 Thinking announcement (openai.com/index/introducing-gpt-5-2): SWE-bench Verified 80.0%, SWE-bench Pro public 55.6%. A 24.4-point gap (80.0 minus 55.6) on two sets for the same model is larger than the entire 2024 shell-versus-ACI ablation. The GPT-5.4 announcement (openai.com/index/introducing-gpt-5-4) reports SWE-bench Pro public at 57.7% and Terminal-Bench 2.0 at 75.1%. Terminal-Bench measures work in a terminal, not the same issue distribution as SWE-bench. Three percentages, three definitions of “can program”.

Becker, Rush, Barnes, and Rein, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv:2507.09089, METR, 2025). A randomized trial: 16 developers, on average five years on those repositories (projects above 22k stars), 246 tasks, February–June 2025. When AI tools are allowed – mainly Cursor Pro and Claude 3.5 and 3.7 Sonnet – time increases by 19% (confidence interval +2% to +39%). Before the study the same people forecast a 24% reduction. After the study they still believe it was 20% faster. Economists in the authors' survey forecast a 39% reduction, ML researchers 38%. The authors limit the claim to that setting: an experienced person, a large familiar codebase, early 2025. They do not carry it over to a junior or to greenfield work.

On 24 February 2026 METR writes that a later measurement – from August 2025, newer tools – does not yield a reliable signal: people drop out of the sample, and the new confidence intervals cover both a speedup and a slowdown (metr.org). The +19% figure remains a description of early 2025. It is not a material constant for Q4 2026.

⚡ What you can take into the backlog

The interface (file window, lint on edit, terse search) moves the score at fixed weights: on the order of 7 points on Lite in 2024. A team's perception can have the opposite sign from the stopwatch: −24% in the forecast, +19% on the clock. A Verified or Pro percentage with no scaffold name and no test audit is not a forecast of your sprint throughput.

3. Case: 90,000 tokens before the test says “wrong”

🛠️ From engineering practice: the loop, on paper

The setup that shows up when a terminal agent is pointed at a failing test in a known repository and is allowed to read files and run commands. This is not a report from a named client. The assumptions are explicit so you can replace them with your own token counter.

Prefix P = 2,000 tokens. Delta D = 1,000 tokens appended to the context per step (observation and the previous reply). No prefix cache: input at step k is P + (k−1)·D.
Sum for N = 12: 12·2,000 + 1,000·12·11/2 = 24,000 + 66,000 = 90,000 input tokens. One call: 3,000. Ratio 30.
Steps 1–3: 2,000 + 3,000 + 4,000 = 9,000 (10%). Steps 10–12: 11,000 + 12,000 + 13,000 = 36,000 (40%). The tail is four times the head.
The same equation with a cap of N = 8: 8·2,000 + 1,000·8·7/2 = 44,000. That is 51% less input than 90,000. It is the effect of a shorter loop, not a measured p99 from a dashboard.
A vendor prefix cache discounts the shared beginning. It does not delete a bad step and it does not turn the stop condition into a test. The discount is in the API price list, not here.

The cut that still has a check: a step cap, stop only when the target test passes, a file view on the order of 100 lines (the SWE-agent ablation, not your A/B), MCP limited to reading the issue and the CI status, a shell with no network and no secret files, a separate git worktree per agent, and a human doing the merge. If the test fails at step 8, the loop stops. You do not get a patch. You get a bounded bill and a diff to review.

METR is why a team will still sign off that “it feels faster”. In their RCT the clock showed +19%, and the belief after the fact stayed at −20% time. The gate is therefore not a survey after the session. The gate is a test you knew before the agent started, and a token counter on the session.

A different loop, in which several roles append each other's outputs, multiplies tokens before any shell runs. The arithmetic for five roles on one ticket is in the article on LLM orchestration and inference cost. The bottleneck here is one terminal loop, not a graph of role names.

4. Decision table

Approach / pattern	Implementation complexity	Latency (p95/p99)	Infrastructure cost	Load on the team	When to use it
In-editor suggestion, a human accepts the diff	Low	No API-loop p99. Human time on one stream	One call, or a fraction of one. The lowest figure in the arithmetic above	Review stays with the author of the commit	A local edit, a small context, a diff a person can read in full
Terminal agent: file window, step cap, test as the stop	Medium	Wall-clock p99 = the sum of steps until the test is green or the cap hits	The context triangle. At P=2000, D=1000, N=12: 90,000 input tokens. N=8: 44,000. A sandbox, not a GPU cluster, while the model is someone else's API	A permission rule, a token counter, CI on the same test	The issue has a test, the repository is familiar, the merge stays with a human
The same agent plus read-only MCP	Medium	Sum of steps plus tool RTT. An issue read should not sit inside every step	As above. Every tool result lands in D, so a wide JSON breaks the 1,000-token assumption	A tool schema, an allowlist, a server audit (`grok inspect`, `grok mcp doctor`)	A repeatable read: issue text, a log, CI status. Not a write
Networked shell, no cap, MCP that writes	Low at the start, high in an incident	A tail with no upper bound. Retrying the last step resends the largest context (at N=12: 13,000 input tokens)	The token triangle plus a side effect: a push, a migration, a read of `.env`	On-call and a post-session investigation. AgentDojo: data from a tool can hijack the next call	Outside a repository that holds secrets, and outside production. Default: not this row
Two agents, one working directory	High, even though it looks like a second session	Wall time can fall. Git's p99 (lock, conflict) rises	Two token bills. One of them lands in the bin on conflict	Who merges, and whose test is the source of truth	A separate worktree per session (`grok -w` in the Grok CLI docs). Not two processes in the same tree

AI / LLM / RAG rate map, not a quote: specialist 180–250 PLN/h, client 240–350 PLN/h, about 35–50% above ordinary backend Python, supplier margin 10–20% in the explicit model. Internal source: what IT staff augmentation costs in 2026. The role “terminal agent + MCP + eval” has no separate row on that map. A person prices the brief.

5. Anti-patterns the host tutorial does not mention

A leaderboard percentage written into a contract. GPT-5.2 Thinking in the vendor table: 80.0% Verified and 55.6% Pro public. In February 2026 OpenAI steps away from Verified (the audit: ≥59.4% of reviewed failures have a flawed test). In July 2026 humans mark 34.1% of public Pro as broken, in the same window where the frontier rose from 23.3% to 80.3%. The contract takes your ticket set and your test, not someone else's board.
Shell-only and cat of the whole file. With GPT-4 Turbo on Lite, an ordinary shell finishes at 11.00%, whole-file views at 12.7%, a 100-line window at 18.0%. The gap belongs to that paper and that model. The direction is practical: the observation has to stay short, or D in the arithmetic above stops being 1,000.
Stop = the model said “done”. Steps 10–12 carry 40% of input tokens at N=12. A loop with no test spends the most expensive third on a retry. An empty assertion (“it compiles”) lets a bad patch through in the same way the multi-agent failure taxonomy describes in the orchestration article.
A writing MCP tool and a networked shell under one grant, plus an MCP server committed in the repo with no review. grok inspect exists so you can see what the directory loads. A README and an issue body are an injection channel (AgentDojo: the attack sits in tool data, not in the system prompt). Installing the host with curl | bash on a laptop that can reach production is a policy decision, not an onboarding step.

6. Playbook: who to contract, and in which order

First ask what is actually missing: a step cap and a test on a single model, or a harness (MCP, sandbox, worktree, eval set) for an agent that has to walk a repository. The rate comes after that.

The API is already there, you already call one model, and what is missing is a step cap, a response schema, and an eval set on your tickets. That is staff augmentation of an LLM developer: one person, your standup, your definition of done. Staff augmentation here means renting a specialist into your team, usually time and materials (T&M). We do not sell a named bench for Monday morning.
The agent has to read the repository, call MCP (tracker, CI, logs), sit in a worktree, and hand back a diff under a human gate. That is different work from a prompt for one call: a tool allowlist, a shell sandbox, an eval on your failing tests, and a review of what grok inspect loads from the directory. That brief fits a contract for an AI-agent engineer, not another prompt bolted on.
The key, the repository, and the transcript do not leave for a private laptop. The contractor works in your repository and on your key. Exporting a session “to reproduce the bug” is a copy of the diff and of whatever the agent read, including an accidental secret. The same rule applies to the MCP server: reading an issue is in scope, writing to production is not, until there is a separate path with a human on it.
Ramp-up. One person joining an existing team: first profiles within days, a start after your interviews and the contract. A squad (agent, eval, tool platform) takes weeks, because permissions and the stop condition sit on your production. A promise of three seniors “from Monday” is either a CV or a bench we do not keep.

Commoditech has billed T&M and permanent hiring from Warsaw since 2012. The network holds 80+ specialists. We do not keep them idle on a bench. A brief can also be filed from an editor, through MCP for AI agents – the protocol an assistant uses to call a supplier's tools, not a shortcut around recruiting. A person prices the specific stack. The bands are in the rates article. Contact: the form.

FAQ

Does 80% on SWE-bench Verified mean the agent closes 80% of a senior engineer's tickets?

No. The score is a triple: model, scaffold, and benchmark hygiene. OpenAI, 13 August 2024: GPT-4o resolves 33.2% of the 500 Verified tasks. On 23 February 2026 the same vendor writes that progress slowed from 74.9% to 80.9% in six months, and that in an audit of 27.6% of the set at least 59.4% of reviewed failures have tests that reject a correct fix. On 8 July 2026, on public Pro (731 tasks), the frontier rose from 23.3% to 80.3% in eight months, and humans marked 249 tasks (34.1%) as broken. Your sprint takes your test. The board percentage stays in the bibliography.

Stdio or a remote MCP server on a company network?

Stdio runs a process next to the host. It does not open a port. It fits a tool that is only allowed to read that directory anyway. The remote transport from MCP revision 2025-03-26, including 2025-11-25, is Streamable HTTP: the server has an address, so it also has authentication and a separate list of tools allowed to write. Reading an issue and a CI status can go remote. A networked shell and a production write do not inherit that grant.

What does it cost to contract an AI-agent engineer or an LLM developer in Poland in 2026?

On the published AI / LLM / RAG map: specialist 180–250 PLN/h, client 240–350 PLN/h, about 35–50% above ordinary backend Python. Supplier margin in the explicit model is 10–20%. That is not a quote. A role with a sandbox, an eval set, and MCP has no separate number here. A person prices it after the brief. The table: IT staff-augmentation rates, 2026.

Can a terminal agent work in a repository covered by an NDA?

Yes, if the API key, the transcript, and the shell stay inside your boundary. The contractor works in your repository and on your key. Exporting a session to a laptop copies the diffs and sometimes a secret from .env that the loop already pulled into the context. A shell with no network, secret files excluded, and MCP with no production write belong in the technical terms, not on a “safe AI” slide.

Sources

Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., Narasimhan, K. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. arXiv:2310.06770. arxiv.org/abs/2310.06770. Claude 2: 1.96% of 2,294 tasks.
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., Press, O. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024. arXiv:2405.15793. arxiv.org/abs/2405.15793. GPT-4 Turbo: 12.47% (286/2,294), average $1.59; Lite 18.00% against 11.00% shell-only; 100-line view 18.0% against 12.7% for the whole file.
OpenAI (13 August 2024). Introducing SWE-bench Verified. openai.com/index/introducing-swe-bench-verified. 500 tasks, GPT-4o 33.2%.
OpenAI (23 February 2026). Why we no longer evaluate SWE-bench Verified. openai.com/index/why-we-no-longer-evaluate-swe-bench-verified. 74.9% → 80.9%; ≥59.4% of audited failures have a flawed test.
OpenAI (8 July 2026). Separating signal from noise in coding evaluations. openai.com/index/separating-signal-from-noise-coding-evaluations. Pro public, 731 tasks: 23.3% → 80.3%; 249 tasks (34.1%) marked broken.
OpenAI. Tables in the GPT-5.2 announcement (Verified 80.0%, Pro public 55.6% for GPT-5.2 Thinking) and the GPT-5.4 announcement (Pro public 57.7%, Terminal-Bench 2.0 at 75.1%). Vendor figures, not an independent Commoditech measurement.
Becker, J., Rush, N., Barnes, B., Rein, D. (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. arXiv:2507.09089. arxiv.org/abs/2507.09089. Write-up: METR, 10 July 2025. Update that the follow-up measurement has no reliable signal: METR, 24 February 2026.
Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., Tramèr, F. (2024). AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. arXiv:2406.13352. arxiv.org/abs/2406.13352. 97 tasks, 629 security test cases.
Model Context Protocol, revision 2025-11-25. modelcontextprotocol.io/specification/2025-11-25. Protocol authors: David Soria Parra, Justin Spahr-Summers.
xAI. Grok Build / Grok CLI. Documentation, x.ai/cli.
Commoditech — AI-agent engineers, LLM developers, 2026 rates.