Copilot generated 40 tests in eight minutes. 31 compile. 22 pass on CI. Line coverage jumped from 41% to 58%. Change fail rate in production didn't budge. The assertion in half the files is assertNotNull(result). SAST in the same pipeline has 180 findings, twelve of which are true CWEs. No one has a ticket for triage.
This is not a model problem. This is an oracle problem. AI test automation without a quality gate looks like code quality. It's a green build based on current behavior, including bugs. LLM doesn't know what was supposed to happen. It knows what's in the repo.
Below is a distinction missing from "looking for an AI tester" briefs: when to hire a single SDET (body leasing QA automation), and when the test pyramid and SAST require a squad. We've detailed hourly rates separately: how much IT body leasing costs in 2026. Here we calculate why Copilot coverage is the most expensive way to a calm dashboard.
One sentence not found on the "AI testing" slide
A test that passes proves nothing. A test that can fail proves something. Meta on Instagram and Facebook didn't push LLM output to main. They pushed what passed the filter: build, stable pass, measurable coverage increase, engineer's acceptance. 25% of generated classes increased coverage. The rest failed at the quality gate. This is a product. Not a prompt.
1. Anatomy: green test, dead oracle
A typical quality week in a bank, telco, or fintech doesn't look like a Playwright tutorial. It looks like three disconnected loops:
- Unit. Developer, JUnit or pytest, Copilot in IDE. Artifact: a
*Test.javafile that no one reads because "it passes". - E2E. QA clicks staging or recorded Selenium. Flaky on CI, so the job is
allow_failure: true. - "Security". Sonar / Checkmarx in the pipeline. Quality gate set not to block the release. Findings pile up in "won't fix".
Three symptoms that you have a generator, not test automation:
- Oracle cloned from SUT. The test calls the same helper as production and compares the result with the result. FX rounding, discount, VAT — the bug is in the helper, the test canonizes it.
- Line coverage without mutation score. PIT or Stryker would kill a mutant in three seconds. No one runs it. The team's KPI is % lines, not the number of killed mutants.
- SAST without SLO. A critical CWE in payments hangs for 120 days because of "false positive last time". LLM tests won't catch this: SAST reads the pattern, the test reads the behavior. These are two different alarms.
Therefore, a brief "tester who knows Copilot" without an oracle context is the most expensive way to get another green build. The competencies that actually make up test automation are on the landing pages: testers and QA automation, JS/TS (Playwright, Cypress), Java, Python. SAST and Secure SDLC add cybersecurity / DevSecOps. CI where these jobs actually work: DevOps / SRE.
2. What Research Says, Not "AI Writes Tests" Decks
You don't need another definition of shift-left. You need a threshold at which a generated test is allowed to enter main — and the right for it not to enter if it merely compiles.
- 75 / 57 / 25 — and only then a human. Alshahwan et al., Automated Unit Test Improvement using Large Language Models at Meta (FSE 2024, arXiv:2402.09171): TestGen-LLM improves existing human-written tests, it doesn't write a suite from scratch. On Reels and Stories (Instagram), 75% of generated tests build, 57% pass stably, 25% increase coverage. At Instagram and Facebook test-a-thons, the tool improved 11.5% of the classes it was run against. 73% of recommendations were accepted by Meta engineers for production. The filter (compilation, no flakes, coverage delta) is there to prevent hallucinations from reaching main. The authors call this Assured Offline LLMSE: the model's output is a candidate, not code.
- Coverage increases when the LLM receives a gap, not the entire file. Pizzorno and Berger, CoverUp: Effective High Coverage Test Generation for Python (arXiv:2403.16218): coverage loop → prompt for uncovered fragment → test → measurement. Median line+branch 80% versus 47% for CodaMosa; versus MuTAP 89% to 77% overall. This is not GPT magic. This is feedback from in-loop instrumentation, analogous to PSI in continuous training.
- Pure ChatGPT breaks assertions. Yuan et al., No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation (arXiv:2305.04207): tests from ChatGPT fail on compilation and due to incorrect assertions. ChatTester (generator + iterative refiner) yields 34.3% more compilable tests and 18.7% more with correct assertions. Even the model's "self-repair" doesn't remove the human oracle: the refiner improves syntax and assert, not the business specification.
- A newer model doesn't replace the quality gate. Konstantinou, Degiovanni, and Papadakis (arXiv:2601.09695, 2026) replicate HITS, SymPrompt, TestSpark, and CoverUp on newer LLMs: a naive prompt can be better for line coverage (+17.7%), branch coverage (+19.8%), and mutation score (+20.9%) than older pipelines. The operational conclusion is contrary to the "buy Copilot" slide: a stronger model increases the volume of candidates. Volume without a filter means more green tests for review. Review is the bottleneck, not the token.
Key Architectural Conclusion
AI-powered test automation starts with a quality gate, not a license. The generator writes a candidate. CI checks: compilation, no flakes, coverage delta or mutation score, no oracle clone from the SUT. SAST separately: critical CWE blocks merge. A human (SDET) accepts what the filter cannot: whether the assertion describes the contract or today's bug. If any stage is "paste from Copilot and merge," you don't have code quality. You have speed.
3. Production Case Study: coverage 71%, FX bug in prod
From Engineering Practice: from Copilot to mutation score
Payment API, Java 17 + Spring Boot, E2E with Playwright, SAST: SonarQube in GitLab CI. The team worked as follows:
- unit: developer + Copilot, JUnit 5, 60% line coverage gate,
- E2E: QA, recorded happy-path scenarios, job marked as optional,
- SAST: quality gate disabled on hotfix branch "to meet the deadline",
- sprint metric: % coverage. Not change fail rate. Not number of killed mutants.
Problem: in the sprint, billing module coverage went from 44% to 71%. Two weeks later in production, FX rounding (pennies during currency conversion) understated the amount. Tests called FxRounding.round(amount) — the same helper as production — and compared it with the helper's result. The bug was in the helper. The suite validated it. Sonar had flagged duplicated logic for four months. Ticket: won’t fix.
Change: a gate like Meta's, but on a Spring stack. A new LLM-generated test is accepted when: (1) it compiles, (2) it passes three times in a row, (3) PIT shows a positive mutation score delta on the class, (4) the assertion does not call the SUT as an oracle — the expected value is a constant, a table of cases, or a separate oracle in testdata, (5) an SDET reviews the PR. SAST: CWE in the payments package blocks merge, it doesn't open Slack.
Measurement: coverage dropped on paper (tautological tests were removed). Mutation score for billing increased. The next FX hotfix was caught in CI, not at the client's. PR review time with tests: minutes, not "we accept because it's green". This is not Copilot Enterprise. This is the oracle owner.
The numbers from Meta's paper do not translate 1:1 to your billing. The mechanics translate: 75% "builds successfully" is not yet quality. 25% with coverage delta and 73% acceptance is already a process that can be audited by a risk committee — analogous to the PSI gate in MLOps services.
4. Decision Table: What Test, What Team Composition
| Approach | Complexity | Quality Signal | CI / Token Cost | Team Overhead | When to Apply |
|---|---|---|---|---|---|
| Manual / Exploratory | Low | High for UX and edge cases, zero for regression | Low | High with every release | New surface, no API contract, PoC |
| Recorded E2E (Selenium IDE, codegen) | Low | Happy-path; flaky with CSS | Medium (minutes per job) | Medium (selector repair) | Demo, not a pyramid; not as the sole gate |
| SDET + Pyramid (unit, API, narrow Playwright) | Medium | Regression, contract, p95 job | Medium | Medium, decreases when suite is stable | Product with CI, stable team, SLA for release |
| LLM dump (Copilot → commit, gate = coverage) | Low | False: lines grow, mutants live | Low–Medium (tokens) | Hidden (review and prod debt) | Never as a merge gate. Exercise, not a process |
| Assured LLMSE (Meta filter: build, pass, delta, accept) | High | Coverage delta / mutation score + review | Medium–High (loop + tokens) | High at start, decreases when filter is stable | Large monolith, existing suite, SDET at the gate |
| SAST with SLO (CWE in critical path blocks) | Medium | Pattern in code, not behavior | Low–Medium | Triage; without owner = zero | Payments, IAM, personal data; alongside tests, not instead of |
QA automation / SDET is not a "cheaper developer" and is not a pentest. 2026 rates: specialist 110–160 PLN/h, client 160–240 PLN/h, 26–38k B2B. Manual testers are significantly lower. Pentest and GRC are on a different landing. Margin 10–25%. Map, not a price list — details in 2026 rates.
5. Anti-patterns not found in the Copilot tutorial
- Coverage as a generator's KPI. The team celebrates 71% lines. PIT would leave 40% mutants alive. Meta publishes 25% of classes with coverage delta not because the model is weak. It's because the filter discards the rest. Your dashboard without this filter lies the other way around.
- Expected calculated with the same code as production. A classic in billing, taxes, FX, discount allocation. The test 'documents' a bug. The oracle is a table of cases, an accounting constant, or a separate, reviewed oracle. Not
service.calc(x)versusservice.calc(x). - SAST in informational mode forever. 180 findings, zero owner, quality gate disabled for a hotfix. LLM tests will not replace SAST: it won't read CWE-89 in string concatenation if the assertion checks HTTP 200. Conversely: SAST won't catch an incorrect penny rounding. Two alarms. Two owners or one SDET with DoD for both.
- Brief 'QA with AI' for an SDET role. You'll get someone who clicks and pastes. You won't get a person who rejects 75% of the model's output. The market broke just like with MLOps: a classic tester has supply, an SDET with an oracle and CI does not. An incorrectly labeled job ad collects CVs in 48 h and zero competence regarding mutation score.
- False quality team. A manual tester from vendor A, 'someone for Cypress' from B, SAST '10% from security'. Three onboardings, zero common Definition of Done for merge. We described the anatomy of this error in team leasing vs body leasing 2026. This is not staff augmentation. This is an integration tax.
6. Playbook: Who to hire and in what order
Don't start with a Copilot Enterprise license. Start by asking which loop is blocking the release: unit, E2E, SAST, or the lack of an oracle owner.
- One gap in the existing pyramid. You have JUnit/pytest, Playwright on the critical path, CI, someone who reviews PRs. What's missing is an LLM filter owner and SAST triage. This is classic body leasing of testers and SDETs: one person, your stand-up, your DoD. First profiles in days, not in a quarter — we source on demand, we don't sell named benches for tomorrow morning.
- The only test is a click. There's no unit test, no API contract, Sonar has been stagnant for half a year. One person won't stitch this together. A team of 3–5: SDET (pyramid, CI), exploratory QA (what a generator won't come up with), DevOps if jobs don't exist. This is closer to team leasing than "we'll buy Copilot and a junior".
- Test data does not leave the VPC. Payment fixtures, PESEL, NDA. Contractor in your IAM, your environment, T&M contract with NDA, data processing agreement, and IP on the client's side. A prod dump to Colab "so Copilot guesses tests better" is a leak. A prompt with NDA-protected code to a public model cloud — also.
- Ramp-up. A person for an existing team: first CVs in 24–48 h, start after your interviews and contract. A squad from scratch: weeks, not a sprint, because you're stitching together permissions, test data, and DoD for merge. The lie "three senior SDETs from Monday" is a CV or a bench we don't have — and we won't pretend.
Since 2012, Commoditech has been doing T&M and permanent recruitment from Warsaw. 80+ specialists in the network, not an idle bench. A T&M brief or success fee can be submitted from an IDE via MCP for AI agents, not just from a form. Amounts for a specific stack are calculated by a human; ranges are in the article about rates, not in the agent's JSON.
FAQ
Will LLM replace test automation and QA testers?
No. LLM is a candidate generator. Meta TestGen-LLM: 75% of tests build, 57% pass stably, 25% increase coverage. 73% of recommendations went into production because the filter (compilation, pass, coverage delta) and engineer review cut the rest. Without a quality gate, you're buying green assertions on current behavior, including bugs. ChatTester shows that even the model's self-repair fixes syntax, not specification.
When to hire one SDET, and when a QA team?
One person, when the pyramid is in place (unit + API + narrow E2E in CI), but there's no quality gate owner: mutation score, SAST triage, Copilot test review. A team of 3–5 (SDET + exploratory QA + CI specialist), when the only test is a click on staging, and Sonar has had 200 "won't fix" findings for six months. An SDET alone won't write business oracle for a product owner. A manual tester alone won't maintain a job in GitLab. If you don't have anyone to review a contractor's PR, don't buy body leasing. Buy a lead plus one role or a squad.
How much does SDET / QA automation body leasing cost in Poland in 2026?
QA automation / SDET: specialist 110–160 PLN/h, client 160–240 PLN/h, 26–38k B2B. A manual tester is significantly cheaper; an SDET is not. Pentest and NIS2 are different rates and a different landing. Vendor margin 10–25% — if someone promises 8% for a replacement in 5 days, account for that elsewhere. This is a market map, not an offer. A human calculates the quote for a brief. Details in 2026 rates.
How does SAST differ from tests generated by LLM?
SAST reads code without running it and looks for patterns (SQLi, XSS, secrets, CWE). LLM writes a test designed to fail bad behavior. SAST without an owner is a false positive dashboard. LLM without a filter is coverage vanity. Both need SLO: a critical CWE cannot remain in "informational," a Copilot test cannot be merged without a mutation score delta. These are not substitutes. These are two jobs in one DoD for merge.
Sources
- Alshahwan, N., Chheda, J., Finogenova, A., Gokkaya, B., Harman, M., Harper, I., Marginean, A., Sengupta, S., Wang, E. (2024). Automated Unit Test Improvement using Large Language Models at Meta. FSE 2024. arXiv:2402.09171. arxiv.org/abs/2402.09171
- Pizzorno, J. A., Berger, E. D. (2024). CoverUp: Effective High Coverage Test Generation for Python. arXiv:2403.16218. arxiv.org/abs/2403.16218
- Yuan, Z., Lou, Y., Liu, M., Ding, S., Wang, K., Chen, Y., Peng, X. (2023/2024). No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation. arXiv:2305.04207. arxiv.org/abs/2305.04207
- Konstantinou, M., Degiovanni, R., Papadakis, M. (2026). How well LLM-based test generation techniques perform with newer LLM versions? arXiv:2601.09695. arxiv.org/abs/2601.09695
- DORA / Google Cloud (2024). Accelerate State of DevOps Report.
- Commoditech — body leasing rates 2026, team leasing vs body leasing, QA and SDET rental.