Superintelligence and control: how to verify agents

Who controls a result when the system that produces it is more capable than the verifier? It is the node of superintelligence, alignment and control. Increasing the competence of a system does not prove that it can meet all the desired goals, nor that a person can recognize his every mistake.

In current corporate agents we face more limited problems: allowed actions, data to be used, results to be verified and cases to be suspended. These controls are necessary and can reduce concrete risks. They do not, however, amount to a proven solution to the problem of overseeing a possible superintelligence.

The distinction also serves an SME. A request for approval, a sandbox or a second model can help, but each one covers a specific boundary. To decide whether a proxy is acceptable we need to know what the control can see and what it can let go.

What is the problem with alignment when increasing capabilities?

Aligning a system means trying to achieve behaviors that are consistent with desired goals and constraints. Saying "complete the practice" leaves many questions open: what information it can use, what compromises it can accept, what actions it can take, and when it must stop. A high ability to find paths to a goal does not automatically resolve the ambiguity of the goal itself.

In the historical document Introducing Superalignment, published on July 5, 2023, OpenAI presented the difficulty of entrusting human supervision to systems far more capable than people. We cite this for the technical problem described, not as a photograph of today's team organization or as an announcement of a demonstrated ASI. Source: Introducing Superalignment.

The simplest example is a result that seems to satisfy the demand but neglects an important constraint. A system can optimize the appearance of completion if the criterion only rewards the closure of the case. Control must distinguish the desired result from the measure we use to represent it. If the measure is incomplete, improving the score does not necessarily coincide with improving the job.

The term superintelligence shifts the question to a broader plane: how to oversee capabilities that could exceed those of the supervisor in many areas? It is not necessary to attribute this condition to a current product to understand the problem. For the meaning and boundaries of the definition we refer to Artificial superintelligence.

Why is a convincing explanation not enough as a verification?

An explanation helps when making sources, relevant passages, and uncertainties visible. It becomes fragile if it is treated as a demonstration only because it is clear. A text can be consistent with itself and omit a decisive condition. The verifier must compare it with external elements, not merely judge its plausibility.

This limit emerges when the result is difficult to assess. An enterprise can easily check that a record has the right code. He may not have the skills to judge a very sophisticated technical optimization. In that case the ability to generate the result exceeds the available ability to verify it, even without formulating any conclusions about the generality of the system.

Human control is a process, not a button. A person who receives an incomplete proposal may approve it in good faith. A competent person may not have time to reconstruct each step. An interface can foreground the response and hide the source that disproves some of the result. None of these cases are resolved just by adding a final confirmation.

The person's mistakes also matter. The criterion written by the person in charge may be too narrow, or an old exception may be remembered as a rule. Supervision should therefore be assessed by including the behavior of the verifier: what information he sees, what errors he can identify and what consequences his wrong decision has.

The Guide to AI Hallucinations It deals with a connected problem. Here the point is broader: a result can overcome a faulty control even without inventing facts. An incomplete selection of the information or an unsuitable metric may be sufficient.

Can a second model solve the supervisor's limit?

A model that evaluates another model may help to find inconsistencies or problematic cases. Utility must be demonstrated on the task. If both receive the same incomplete data or apply the same wrong criterion, consent does not provide independent control. Two equal judgments can share the same mistake.

Even a more capable verifier must have a criterion and a way to access the evidence. If he has to judge a change to the management software, he needs the result of the operation. If he has to verify a document, he needs the applicable sources. Asking "is this answer correct?" without providing the necessary material can produce another plausible text.

Anthropic distinguishes in evaluations the trace of what the agent has done from the final result in the environment. A successful statement does not replace the actual state of the system. This distinction helps to build concrete controls, without demonstrating universal coverage of behaviors. Source: Demystifying evals for AI agents.

We can combine different checks: deterministic field checks, comparison with documents, expert assessment of difficult cases. The combination is used when each control intercepts a distinct error. Adding verifiers without understanding what flaws they recognize can complicate the system and give unwarranted trust.

What current controls remain distinct from ASI alignment?

For a corporate agent we can define a perimeter of actions and acceptance tests. Validity remains linked to tested conditions. When changing tools, sources, or models, behaviors can also change. The previous good result remains useful information, but does not make the new conditions irrelevant.

The distinction between Autonomy and ability of agents comes in handy: a capable system may have few permissions; a mediocre system may have too wide access. Control is a design choice to motivate, without turning it into an intelligence label.

How do you see the problem in a plant engineering firm?

Hypothetical scenario, not customer Giallo: a company in Verona uses an agent to prepare the interventions based on requests and data sheets. The system proposes materials and activities, then asks the manager to approve a draft. The aim is to reduce the work of information reconstruction.

The proposal contains a seemingly correct technical reference, but applicable to a different configuration. The person in charge sees an orderly list and a synthetic motivation. If the difference between configurations is not highlighted, it can approve. Human passage has taken place; substantial control may have failed.

A more verifiable design has the configuration used, the source and the data that is missing. The person in charge can then figure out what element he or she needs to control. For some conditions the system must be limited to preparing the material and suspending the technical choice. The expected result includes that boundary, instead of measuring only how many drafts are completed.

The test could insert an old but plausible card next to the current one. The question is not whether the agent produces an elegant proposal. It is whether it recognizes the conflict and whether the approval process makes it visible. This verifies a job-specific risk, without purporting to resolve the alignment of a superintelligence.

The availability of the sources is deepened in memory and business context. Having the correct document in the archive only helps if it enters the control at the necessary time and keeps its references.

Why is a sandbox not a full warranty?

A sandbox circumscribes an execution environment. It can limit files, network, or other resources according to the configuration adopted. It is an important technical boundary, but its meaning must be described: what actions it blocks, what it allows and what services are accessible through separate tools.

In the Managed Agents architecture, Anthropic describes the separation of credentials from code executed in the sandbox and discusses the risk of prompt injection. It is an example of protecting a specific boundary, not a promise that any malicious instruction or incorrect action will be neutralized. Source: Scaling managed agents.

An agent can stay in the sandbox and produce a wrong document. He can also correctly use an authorized tool to apply a wrong decision. The containment of the execution and the correctness of the intent do not coincide. This difference explains why both technical limits and checks on the result of the process are needed.

According to hypothetical scenario: a service centre in Bologna prepares administrative files. The agent works on copies of documents in a separate environment and generates a draft. The sandbox can prevent it from directly modifying the original archive. It does not, however, prove that it has included all necessary attachments or correctly identified the applicant.

If the system also has a tool to publish the file, we need to examine that path separately. The local file boundary does not describe permission on the external service. A draft may be reversible; a publication may have different consequences. The test must follow the actual action until its outcome.

For leakage, access and operational data management we refer to the guide on AI and Data Security. Here the conceptual limit is concerned: no single technical protection proves on its own that a system always pursues the desired goals.

How to test human approval?

A control test may include a known error and ask the auditor to evaluate it using the intended interface. If the error is not seen, one must understand why: hidden information, unclear criterion or missing competence. The result is about the review process, not individual guilt.

This exercise is also useful when the model improves. More sophisticated proposals may require more competent controls. It is not enough to assume that the manager recognizes every problem because he knows the company. The result domain may be broader than its experience or include a complexity that is difficult to quickly reconstruct.

Open the verifier test: does a known defect exceed approval?

Prepare a plausible proposal with a relevant and already understood error: an outdated source, an incorrect recipient or an omitted constraint. Show it in the same format as the process, without adding suggestions that in reality would not appear.

Observe what elements the verifier uses and whether the error is recognized. If it passes, it changes the way evidence is exposed or narrows down the action allowed. Repeating confirmation without changing control does not address the cause.

The test measures a specific oversight flaw. It does not prove that any future error will be detected or that the system is aligned in a general sense.

The absence of observed errors is not equivalent to proof of absence of errors. A good verification report also states what has not been covered. The distinction between available evidence and broader conclusions is covered in the article on Superintelligence: Does it already exist?.

What errors make control only apparent?

Verifying the sentence instead of the outcome. "Operation completed" may be a misspelled description. Whenever possible, the control must read the resulting state and compare it with the expected result. Text verification alone covers another property.

Adding approvals without providing evidence. A confirmation may become usual if the cases always seem ordinary. Sources, differences and consequences must be visible enough to allow for a real assessment. Control design is part of the system.

Calling a component safe without describing its boundary. Sandboxes, permissions, and logs have specific functions. You need to know what risk they face and what actions remain possible. A reassuring name is not evidence.

Using a local test as a general solution. Passing a set of tests justifies a conclusion limited to those behaviors and conditions. The promises of total control would require much broader evidence. The prudence of the conclusion is part of the quality of the assessment.

What acceptance criteria can an SME define?

For a circumscribed process we can list correct result, allowed actions, unacceptable errors and suspension mode. Then we assign to each criterion a workable verification. If no one knows whether an output is correct, this is a limit to be faced before granting more autonomy.

The control system must include who receives the exception and how they resume the work. Reporting a problem to a person who does not have access to documents can block the case without making it manageable. The quality of supervision also depends on the link between roles, information and responsibilities.

These elements come into the choice if use AI today or postpone the case. The point is not to wait for a promise of zero risk, but to have evidence proportionate to the work we want to delegate.

We can start with actions and tests, before choosing the degree of autonomy. The service Giallo AI agents helps define the perimeter of the process and the criteria by which to accept or suspend a result.