Google AQuA: Finding AI Agent Errors Even When Logs Are Green

Google's AQuA is an implementation to observe and diagnose the quality of AI agents in production. The problem it faces is concrete: a request can be completed without technical errors and still leave the user with a wrong answer, an incomplete task or an action that is not useful.

On October 8, 2026, Google published work on the "outer loop", the cycle that analyzes behavior out of the path of requests. AQuA uses telemetry to evaluate sessions and detect recurring defects. It does not apply code corrections and does not independently open pull request. Google Developers Announcement.

Why can an agent go wrong with all the green logs?

Technical logs often answer questions such as "Did the service answer?" or "Did the call end?". They don't always respond to "Did the user get what they needed?". An agent can retrieve an existing document but referred to a different product. You can fill out all the fields in a card using an unauthorized deduction. You can respond politely without resolving the request.

These defects can be invisible to monitoring focused only on exceptions. The execution does not stop, but the meaning of the result does not match the task. The project described by Google separates just the operational path from the quality analysis: the observation takes place in an external cycle, which examines sessions and looks for recurring problems. Official description of the AQuA approach.

For a company that already uses AI agents in processes, this changes the maintenance demand. It is not enough to know that the system is reachable. It is necessary to understand which requests are concluded correctly, which require clarification and which seem to succeed only because they have not generated a computer error. Quality belongs to the work done, not the color of the operating panel.

What data does the AQuA implementation analyze?

The implementation reads telemetry from Cloud Trace, Logging, and BigQuery, evaluates sessions, groups flaws, and verifies insights. The official code describes a system that is adaptable to your Google Cloud project. It is not enough to install it to automatically obtain useful observations: telemetry must contain information that allows the task and relevant steps to be reconstructed. AQuA Official Repository.

A track should allow you to link request, tools used, sources consulted and result, according to what the project is allowed to record. If only one final sentence remains, it can be difficult to tell whether the defect arises from information retrieval, rule or integration. The distributed version also counts: a diagnosis referring to the wrong code risks suggesting an intervention on an already changed behavior.

This does not justify collecting every content without criterion. First, necessary data, access and storage must be defined. A session may contain personal or confidential information, and its analysis is in turn a treatment to be designed. The Guide to Data security with AI remains relevant even when the observation takes place in the organization's project.

What produces the diagnosis and who decides the correction?

A useful insight links a recurring problem to the sessions that show it and to the part of the system to be examined. He doesn't just say "the prompt could be better." It must allow the manager to check whether the description really corresponds to the expected work and whether the proposed change addresses the cause rather than a symptom.

Google describes the possibility of anchoring proposals to a photograph of the distributed code. AQuA does not apply changes and does not independently open pull request: the path comes to diagnosis and proposal, leaving revision and implementation to the team process. This distinction is explicit in the Google Developers ad and in README of implementation.

The decision also requires knowledge of the domain. An agent may have asked for confirmation because an essential requirement was missing; an uncontextualized evaluator might mistake it for inefficiency. The company contact person must be able to explain why the arrest was correct. Quality does not always coincide with fewer steps: sometimes an extra question avoids an inappropriate answer or action.

What is the difference between technical error and quality defect?

These checks complete. It is not necessary to abandon logs to adopt a semantic evaluation, but to link them to the expected result. Even an automation platform can report error events without knowing if the treated information is correct. The event subscriptions of Make concern the operational side of this observation; AQuA addresses a different level of agent diagnosis.

What does the case published by Google show?

The case illustrated in the publication uses 32 synthetic or scripted sessions. It is useful for understanding how the diagnosis works, but it does not demonstrate a universal improvement on the traffic of real businesses. The case data are neither results from Giallo clients nor an effectiveness percentage that can be transferred to a different project. Case described by Google.

A set of representative requests is required to move from demonstration to a business test. There must be successful cases, uncertain cases, and requests that the agent should refuse or transfer. The team notes the expected behavior and then checks to see if the insights spot real problems, ignore major flaws, or report false problems. The evaluator should therefore also be evaluated.

This work should avoid a closed circuit where one model judges another model without external references. Some sessions should be read by competent people on the process, using shared criteria. If the agent manages a automated customer care, the contact person for the assistance must be able to distinguish a formally complete response from a effectively resolved request.

The insight seems convincing, but can't you find a session to prove it?

Don't treat it like a ready fix. Ask for evidence that can be linked to requests, sources and distributed version. Check that the behavior described was really wrong according to the rules of the process. A plausible proposal without a match can become material to be deepened, without automatically entering the release.

A distributor with technical assistance: hypothetical scenario

Let's imagine a distributor using a chatbot to answer questions about spare parts. Service calls succeed and documents are retrieved, but some users receive instructions referring to similar variants of the product. In this hypothetical scenario the problem would not necessarily appear as a technical exception. The answer could be grammatically correct and contain a real link.

The manager would prepare cases that clearly distinguish variants and requests where the necessary code is missing. The expected behavior might be asking for a clarification, not completing the answer anyway. The analysis should look for the reason for the confusion: retrieved sources, information passed to the agent and arrest rule.

If the diagnosis proposes an intervention, the team would compare it with the sessions and try the change on the prepared cases. A better answer in the demonstration would not be enough. It should be verified that the system does not begin to ask for unnecessary clarification when the code is present, or to exclude relevant documents. Monitoring would thus become a controlled learning cycle, linked to the real responsibilities of care.

A service company with collection requests: hypothetical scenario

Consider a company that uses an agent to gather information and prepare boards for the operations team. In this second hypothetical scenario the cards are saved correctly, but some report as confirmed details that the customer had only assumed. The flow ends, the management software contains all the fields and no call fails: the defect concerns the meaning of the information.

The evidence would distinguish explicit statements, missing information, and deductions. Insights should help you understand if the agent loses the difference during the conversation or during the final compilation. An intervention might be about the format of the card or the confirmation step, but it should be chosen after examining the path, without automatically attributing the problem to the initial prompt.

The person in charge would also control what the person taking charge of the request receives. A seemingly complete card may be less useful than a card that clearly signals a gap. The Guide to the chatbot for the company site helps define this boundary: success is not holding back the conversation, but delivering usable and properly qualified information to the process.

How to turn insights into a maintenance cycle?

We need a contact person first who chooses what problems to deal with. A frequent but unremarkable defect may have priorities other than a rare case with important consequences. Priority should consider impact on work, the possibility of reproducing the problem, and clarity of evidence. It shouldn't just depend on how well the system has written the diagnosis.

Then you need a path between insight and modification: verification, hypothesis, implementation, proof and release. Cases that showed the defect enter the next evaluation, along with requests that previously worked. This avoids correcting a case by breaking another behavior. The photograph of the distributed code allows the observation to be linked to the version on which it intervenes, as provided by the implementation.

The same discipline is used when the channel is voiced. The work described for ElevenAgents Architect and voice agent testing is about another platform and another experience of use. The useful link is to maintain a case collection and verification process; it is not assumed that a diagnosis designed for a configuration automatically transfers to any agent.

How to distinguish a repeated defect from an out-of- perimeter request?

Not every dissatisfaction indicates that the agent needs to do more. A request may be out of the intended service, or request an authorization that the user does not own. Before accepting a diagnosis, the team should read the task along with the operating rules. Otherwise you risk correcting a behavior that protects the process, such as asking for confirmation before an action or not showing a confidential source.

To make the distinction verifiable, it is best to note why a case is acceptable or defective. ‘Unuseful response’ is a weak description; ‘did not ask for the code needed to distinguish variants’ identifies an observable rule. Similarly, ‘did not complete the request’ can be corrected if mandatory information is missing. Diagnosis should help to see this difference, not flatten it into a general judgment.

A recurring defect then deserves a check on different sessions. If it only appears with an outdated source, the first action may be to update the archive. If it appears with correct sources and clear requests, the team examines the agent's steps. This separation avoids rewriting instructions to compensate for a data problem. It also allows you to assign the job to the right person: source owner, integration manager or agent developer.

What mistakes to avoid before activating the observation?

The first is to present the published code as a ready-made managed service, with a promise of automatic correction. The second is to record data without having decided who can read it and for how long. The third is to trust any insight without linking it to sessions and expected behavior. The availability of the code allows it to be adapted, it does not eliminate these responsibilities.

Another mistake is forgetting the costs of the same observation: data preparation, services used, model calls and review of proposals. A universal tariff is not provided here, because it depends on the project and consumption. Before extending the analysis, it is best to understand which sessions to include and how to make the outcomes manageable by the team.

Finally, the observer must have access consistent with his task. The discussion on network and web fetch in the Claude Managed Agents shows a different security boundary: what sources and services an agent can reach. Even in diagnosis, an explicit map of permissions is needed, with no large credentials assigned just to make integration easier.

If you already have an active agent, we can set up a maintenance and quality control path starting from representative requests, necessary telemetry and auditing responsibilities. AQuA is interesting when it helps explain a defect with verifiable evidence. The result to look for is knowing what to correct, why and how to prove that change has improved the job.