FROM THE INDUSTRY
product. Determinism is the point.
The most efficient deployments I have seen run it on 1 to 2 percent. Almost nobody doing quality assurance (QA) at scale runs it at 100. People hear that and assume something failed, but nothing failed. This is what AI looks like when it stops being a demo and becomes infrastructure. A slider, not a switch The industry debate is usually framed as a choice: scripted automation or agentic AI, the old way or the new way. In production, nobody experiences it as a choice. The share of AI in a test is a parameter, like a timeout value or a retry count. You set it per test, per step, per business goal. At one end sits full agentic execution: describe the goal in plain language and let the agent navigate. At the other end sits pure code: deterministic, version- controlled, consuming zero AI tokens at runtime. There are two dynamics at work here, but it’s lazy to pick sides. The question to ask is actually, ‘what does this specific test need?’ Three things decide it: repeatability, trust, and cost per run.
contact. An AI agent given the same goal found no issues with this and continued regardless, as if nothing had happened. The same goes for exploratory testing, one-off checks around a live event, and teams with no scripting skills who need an answer from a real device today rather than after a development sprint. “Open the first on-demand title, check the metadata, the price and the poster” is a complete test now, which is something that used to involve weeks of scenario building. Live events made this concrete for us this summer. We monitored World Cup delivery for several operators and platforms across multiple regions, and the pattern repeated everywhere: the checks that mattered most were rarely the ones anyone had scripted in advance. Somebody always needs to know, two hours before kickoff, that a specific app on a specific device in a specific market plays the match. Goal- oriented testing is built for exactly that question. But look at what these cases have in common: adaptability matters more than repeatability. Change that, and the whole calculation flips.
Non-regression testing is the same story. A thousand tests a week across twenty device models is not asking “can the app be used?” It is asking “did anything change since Tuesday?”. Code answers that precisely, repeatably, and at near-zero marginal cost. A key performance indicator (KPI) like video start time only means something if the procedure behind it is identical on every run. And then there is the invoice. Agentic execution consumes compute every time it runs. On a test that fires ten times a day across a hundred devices, that adds up quickly, and it buys you nothing if the coded version was already reliable.
The two patterns that actually win
So where does AI belong in a production pipeline? In our deployments, two places.
The first we call by-design AI: the test is 80 to 90 percent deterministic, with AI inserted exactly where code has always been miserable. Picking tonight’s episode of a live show from a rail that reorders itself daily. Judging whether what is on screen is genuinely appropriate for a child’s profile: trivial for a human, painful to code, well suited to a vision model. The workflow stays governed and measurable. The AI takes only the steps that were always brittle. The second pattern is smaller and, I think, more important: recovery. The test runs as pure code on the happy path. AI wakes up only when something unexpected interrupts it: a system pop-up, a firmware
Where 100 percent makes sense
Why the backbone stays deterministic
Full AI is the right tool when the environment will not hold still.
Take channel change monitoring. An operator measuring zap times runs thousands of identical operations a day, across dozens of device models, and needs the results to be comparable week over week, market over market. Put a probabilistic agent in that loop and you add variance to a measurement whose only job is to be stable. The measurement is the
My favorite example from the past year: a large European operator reskinned its entire homepage overnight to promote a blockbuster film. Menus and visual anchors moved around, which would have caused capture-based scripts to break upon
Volume 48 No.23 SEPTEMBER 2026
83
Made with FlippingBook - Online magazine maker