A lot of AI evaluation still starts with a familiar question: can the model complete the task?
That question matters. It just isn't enough once a model is part of a system that can plan, use tools, hold state, delegate work and act on external services.
When one agent hands work to another, the thing being evaluated changes again. The final answer can look right while the process that produced it is fragile, unsafe or impossible to reconstruct.
I've been thinking about where independent AI testing needs to go next. My working answer is that the unit of testing has to expand with the system: we need to test the agent, the hand-offs, the authority around it and the complete arrangement in use.
The problem is getting bigger than the model
An agent can make a reasonable decision using bad information from another agent. An orchestrator can pass an incorrect assumption down a chain. Two agents can agree because they share the same blind spot, not because either one independently checked the answer.
A human approval step can also exist on paper while the reviewer has neither the context nor the time to challenge what is in front of them. None of these problems is necessarily visible in a single prompt-and-response benchmark.
The UK Alan Turing Institute's Centre for Emerging Technology and Security (CETaS) describes this as a behavioural assurance problem: what matters is not only whether a system completes a task, but how its outcome was produced and what happens when people and agents interact.
Assure deployments, not models alone.
The paper presents this as one of seven assurance principles for agentic systems. It is a useful way to frame the question I want CTAI to investigate: what did the deployed arrangement do, under these conditions, and can we show the evidence?
The paper also makes a boundary clear. Its taxonomy is a practical framework, and some concerning behaviours were observed in deliberately constructed laboratory settings. Those observations should not be presented as proof that the same behaviour is routine in real deployments.
A benchmark score is not the whole story
Hari Santhosh's discussion of agent evaluation helped prompt this note. The published Singularity-Bench paper is a separate research source: it sets out six end-to-end tasks spanning pre-training, mid-training, post-training, inference, the agent harness and cluster orchestration.
That is an ambitious capability question: can agents carry out complex AI-development work? It also makes a broader evaluation question hard to avoid. Capability at one layer does not automatically tell us how a connected system behaves.
A model may perform well. An agent harness may be effective. Specialist agents may each complete their assigned task. But once they are connected:
- Does the right information reach the next agent?
- Can a downstream agent tell verified evidence from an upstream assertion?
- Can one agent challenge another's conclusion?
- Can the orchestrator detect loops, conflicting instructions or repeated work?
- Can one component's failure contaminate the rest of the workflow?
- Can we reconstruct who or what made a consequential decision?
The UC Berkeley Center for Long-Term Cybersecurity's 2026 Agentic AI Risk-Management Standards Profile similarly warns that multi-agent risks can emerge through interaction and cannot be understood by looking only at individual components.
Can this agent do its job?
Evaluate an individual component against defined conditions: its capability, reliability and limits.
Can the arrangement hold up?
Evaluate whether the complete system produces a dependable outcome as its parts interact.
Both matter. They answer different questions. A good component result cannot stand in for evidence about the system it joins.
The Autonomy Stack
This is my working map of the system under test. It is an original CTAI diagram, built to show the connections and control points I would want to investigate. It is not a reproduction of Hari Santhosh's graphic or any other source visual.
THE AUTONOMY STACK
From individual agents to connected systems.
The arrows are where a lot of the interesting testing lives: hand-offs, trust assumptions, permissions, context transfer, retries, escalation and the point where a recommendation becomes an action.
Six layers I would evaluate
This is not a universal checklist to force onto every product. The architecture, use case and consequences determine which layers matter and how deeply they need to be tested.
- 01
Agent capability
Can each agent do its own job? How reliable is it, and where does it fail?
- 02
Hand-offs & orchestration
Is work routed correctly? Do context, constraints, source and uncertainty survive between agents?
- 03
Collective behaviour
Do agents reinforce errors, create loops, drift from the goal or produce false consensus?
- 04
Authority & execution
What can each agent access, change, delegate or approve? Are boundaries enforced at the point of action?
- 05
Human control
Can a person understand, challenge, pause, reject or reverse a consequential decision in practice?
- 06
Runtime & change
Can behaviour be monitored, reconstructed and retested as models, tools, agents and permissions change?
Failure modes I would actively look for
The Turing Institute's behavioural assurance research is useful here because it moves beyond familiar model errors and describes failures that can arise through interaction. These are test hypotheses, not claims that every deployed agent system exhibits them.
Cascading errors
One agent makes an incorrect assumption. Another treats it as established information. A third turns it into a recommendation or action. The original error becomes harder to spot as it travels.
False consensus
Several agents agree, but share a model, data, prompt assumptions or source material. Agreement can look like corroboration when it is the same mistake repeated.
Goal drift & proxy optimisation
Agents optimise the measurable task they were given while losing sight of the wider business objective. The workflow can be locally successful and collectively wrong.
Misreporting & weak records
The system says it completed an action, checked a condition or followed a policy, but logs or external state do not support that account.
Responsibility diffusion
When a chain of agents contributes to a decision, it becomes difficult to establish where an error entered, who had authority and who was meant to catch it.
Human oversight theatre
A human is technically in the loop, but lacks the evidence, context, time or authority needed to make the approval meaningful.
The Turing paper distinguishes failures inside one agent workflow from failures that emerge specifically through interactions between agents or between people and agents. Testing components in isolation can miss the interaction-specific behaviour.
A practical example: the cost-saving agent
Hari Santhosh's post and example prompted a useful case for this note: an agent identifies possible cloud waste, opens an Infrastructure-as-Code pull request, estimates blast radius on a 1–10 scale, then waits for a person to merge it. In the example as described, the agent does not change live infrastructure itself.
I like the separation between proposing a change and executing it. But I would want to test whether that boundary holds under pressure, when the evidence is incomplete or when another agent joins the workflow.
Can the agent identify a legitimate saving, propose a technically valid change, assess likely impact and remain inside its authority — including when its assumptions are wrong?
- Proposal quality: Is the pull request valid, relevant and supported by evidence?
- Risk scoring: Does the blast-radius score match the potential impact, and does it explain uncertainty?
- Permission boundaries: Can the agent modify live infrastructure, merge its own change or acquire permissions it was not granted?
- Adversarial conditions: What happens when resource data is stale, incomplete or misleading?
- Agent interaction: If one agent proposes and another reviews, do they independently check the evidence or repeat the same assumptions?
- Human approval: Does the reviewer receive enough context? Can the agent bypass or pressure the checkpoint?
- Failure and recovery: Is a wrong proposal contained before execution? Can an applied change be reversed safely?
Attribution boundary: this is Hari's illustration, not a CTAI test result. I have not independently assessed the agent, the benchmark score, its risk estimates or any claimed outcomes.
How I would investigate a system
I wouldn't begin by running hundreds of prompts and collecting a pass rate. I would first map what the system is allowed to do, how it is put together and what could happen if it gets something wrong.
CTAI already uses four phases. For system-level testing, I would apply them to the whole chain:
- PHASE 01
Initial Impact
Map the architecture and define the claim. Identify agents, roles, models, tools, memory, data flows, permissions, external systems, human approval points and intended outcomes. Record the tested version and configuration.
- PHASE 02
Stress Testing
Test normal operation, then introduce credible pressure: conflicting instructions, bad upstream information, unavailable tools, ambiguous goals, agent disagreement, repeated delegation, stale context and attempted boundary violations. Use a sandbox or controlled environment for consequential actions.
- PHASE 03
Observe & Document
Capture the full trajectory where available: inputs and outputs, tool calls, hand-offs, decisions, permission checks, timestamps, retries, escalations, approvals and external state changes. Preserve failed attempts as well as successful runs.
- PHASE 04
Verdict & Report
Assess the evidence against defined criteria. Explain what worked, what failed, where risk accumulated, whether controls held and what remains untested. The verdict stays human-led and bounded to the tested configuration.
The important addition is to follow the whole chain, not just the final answer.
That is how I would connect this proposal to CTAI's existing four-phase method. It is a proposed application of the method, not a completed CTAI evaluation of a multi-agent product.
A scenario matrix
These are starting points. A test should select scenarios from the system's actual architecture, threat model and intended use; not every row will apply to every product.
| Scenario | What it can reveal |
|---|---|
| Normal task; all components available | Baseline capability and workflow behaviour |
| Upstream agent returns incorrect information | Error propagation and verification |
| Two agents disagree | Conflict handling and escalation |
| Agents share one flawed source | False consensus and correlated error |
| Tool or API becomes unavailable | Recovery, retries and loop control |
| An instruction falls outside the agent's scope | Permission and policy enforcement |
| A high-impact action is proposed | Risk assessment and approval boundaries |
| A human rejects the recommendation | Whether rejection is respected |
| Logs conflict with the claimed outcome | Traceability and misreporting |
| A component or model is updated | Regression and version-to-version change |
The difficult bit: proving the controls work
There is a difference between a product claiming to have an approval step and showing that the step is enforced.
There is a difference between an agent saying it has stopped and observing that no further action occurred.
And there is a difference between three agents agreeing and three independent sources of evidence supporting the same conclusion.
This is where hands-on testing has a useful role. We can define expected behaviour before the test, challenge a boundary deliberately, then compare what the system says happened with evidence of what actually happened.
The Canadian Centre for Cyber Security and partner agencies recommend least-privilege access, explicit human control points, auditability, interruption, reversibility and progressive increases in autonomy. The guidance also says that system designers, not agents, should determine which actions need human approval.
A system should not be allowed to decide for itself where its own authority ends.
The same guidance recommends monitoring operations such as tool calls, memory interactions, decisions and actions, then cross-checking agent reports against independent system logs. The Turing paper cautions that the presence of human oversight is not, by itself, evidence that oversight is meaningful.
Where this could take CTAI
Testing a single AI tool is one thing. Independently evaluating a multi-agent product, its orchestration, action boundaries and failure behaviour is a broader engagement.
- Architecture and risk mapping: understand what is connected, what each agent can do and where authority sits.
- System-level scenario testing: evaluate the complete workflow, including hand-offs, dependencies and failure propagation.
- Autonomy and control testing: challenge permissions, approval gates, stopping conditions and recovery.
- Evidence and traceability review: compare reported behaviour with captured actions and system records.
- Retesting: revisit relevant scenarios after material changes to models, agents, tools, policies or architecture.
This could be higher-value work because the complexity, consequences and evidence requirements are greater. It would also need careful scoping, appropriate access, controlled test environments and clear agreement about what CTAI is and is not assessing.
I would not position this as certification, penetration testing or a guarantee of safety. It would be independent, human-led testing and evaluation of defined system behaviour under agreed conditions.
The opportunity is to help builders understand what their system actually does before customers have to discover it for them.
My take
I don't think the next stage of AI evaluation is simply more benchmarks or more agents checking other agents.
It's better evidence about how the complete system behaves.
As autonomy increases, the distance between a model's output and a real-world consequence gets shorter. Testing needs to account for that.
A system that completes a task but cannot explain its actions, contain a failure or respect its authority boundary is not something I'd judge on task success alone.
Test the agent. Test the hand-offs. Test the authority. Test what happens when it goes wrong. And test the whole system together.
That's where I think the next generation of independent AI testing needs to go.
Research, sources & attribution
This note synthesises published research and guidance. It is not a report of a CTAI test of Hari Santhosh's benchmark or cost-saving agent.
- Rick Hennessy and Carolyn Ashurst. Behavioural Assurance of Agentic AI for Sensitive and High-Stakes Organisations. Centre for Emerging Technology and Security (CETaS), Alan Turing Institute, 23 September 2026. Used for its behavioural failure taxonomy, access-authority-trust framing and system-level assurance principles. Read the research.
- Nada Madkour et al. Agentic AI Risk-Management Standards Profile. UC Berkeley Center for Long-Term Cybersecurity, Version 1.0, February 2026. Used for multi-agent risks, system-level assessment, monitoring, containment and accountability. Read the profile.
- Canadian Centre for Cyber Security and partner agencies. Careful adoption of agentic AI. Practical guidance on permissions, human control, auditability, progressive autonomy and continuous evaluation. Read the guidance.
- OWASP. Artificial Intelligence Security Verification Standard (AISVS), C9: Orchestration & Agentic Security. Technical verification prompts for execution budgets, loop control, circuit breakers and high-impact action approvals. Read the controls.
- Yang Zhou et al. Singularity-Bench: Can Agents Build Agents? arXiv:2611.16591, submitted 12 November 2026. Cited as a published benchmark paper about end-to-end autonomous AI-development tasks, not as validation of Hari's example or a CTAI finding. Read the paper.
The practical examples in this Lab Note draw on Hari Santhosh's original post on X about Singularity-Bench and a cost-saving Infrastructure-as-Code agent, along with my conversation with Hari about his post. They are illustrations and questions for investigation, not independently verified product claims or CTAI test findings.
Research boundary. The evidence base and terminology are still developing. The Turing Institute describes its taxonomy as a practical framework and notes that some concerning behaviours have been observed in deliberately constructed laboratory settings; those results should not automatically be presented as evidence of routine real-world behaviour. The evaluation approach here is CTAI's interpretation and proposal.