Kimi K3.1 Pro benchmarks: comparison with other LLMs

Kimi K3.1 Pro benchmarks show competitive results on coding, agents and documents, alongside clear differences between tasks. The official collection does not establish general superiority over Claude, GPT, DeepSeek or GLM: it combines different systems, tools and sources. Choosing a business candidate requires reading the score together with the test version and protocol.

This analysis uses the table and notes published by Kimi, checked on October 24, 2026. The four charts isolate eight tests and retain the six models from the original columns. We have not run the benchmarks, made authenticated calls or measured a return on investment. The data are published claims, not an independent replication by Giallo.

For the identity and availability of the new version, you can start with what Kimi K3.1 Pro is. Here the question is more precise: what the results tell us, which comparisons require caution and how to use them to design a test on your own work.

Which data are we comparing, and who measured them?

The common source of the figures shown is the developer’s compilation. Kimi presents its own results and takes values from releases and leaderboards for some competitors. This provenance lets us reconstruct the published comparison, but does not turn the columns into a controlled experiment with the same tools, time and infrastructure. The official notes describe the differences for each test.

The column configurations are labeled max; Claude Fable 5.1, GPT-6 Astra and Claude Opus 5.5 also include the label with fallback. We retain it in the tables. We do not treat fallback as an irrelevant detail or an identical rule across all benchmarks: its effect depends on the source protocol and how problematic cases are handled.

The overview does not include Gemini as a competitor. A model’s presence as an evaluator within a protocol differs from its presence as a system being tested. K3.1 Pro Highspeed also remains outside a separate experimental comparison: the launch table does not give it its own column of measurements. These limits avoid extending the dataset beyond what it publishes.

How should scores, missing data and model differences be read?

A benchmark defines problems, an environment and scoring rules. The score may describe task resolution, answer accuracy or geometric similarity. Two values on the same numerical scale can therefore represent different capabilities. Adding or averaging them would produce a new editorial index, without validation explaining its meaning.

The tables follow the order of the original columns and use N/R for an unreported value. We do not assign scores to missing data. When two results are close, the numerical difference remains a descriptive observation: without an appropriate assessment of uncertainty, it does not by itself demonstrate stable superiority. The official Terminal-Bench website, for example, presents results as measurements of model-agent systems and includes confidence intervals.

Percentage points are used to read the gap between two percentages, when that is the metric adopted. Even an arithmetically correct difference does not measure time saved, revenue or the proportion of a department that can be automated. Those conclusions require business-process data, including costs and checks the benchmark does not observe.

How does Kimi K3.1 Pro compare on coding?

The first panels place two different tests side by side without creating a single ranking. Terminal-Bench concerns operational tasks in a terminal environment; SWE Marathon evaluates software work under its own protocol. Kimi uses Claude Code for K3.1 Pro in both tests, while the other models’ values have mixed sources. Definitions and details are in the Kimi notes, alongside the official reference for Terminal-Bench.

Terminal-Bench 4.0: what does Kimi’s 65.7 mean?

In this snapshot, Opus has the highest value, followed by Kimi. Their proximity is interesting for selecting test candidates, but does not justify treating them as interchangeable. Terminal-Bench evaluates the system tackling the tasks, with tools and an agent: not just an isolated text response. Values and sources are those in the Kimi collection.

A team using a chat without repository access is not reproducing the measurement environment. The same applies to an integration with different tools or a lower operating limit. To bring the benchmark closer to the real case, it is necessary to reconstruct which steps the system can perform and which checks establish that the task is solved.

SWE Marathon v1.1: where does the order change?

Here Kimi reports the highest value among the columns shown. K3.1 Pro and DeepSeek are evaluated with Claude Code, while the other figures come from the SWE Marathon leaderboard according to the publication notes. The dynamic leaderboard page could not be read in full during our extraction: we therefore do not present these rows as independent verification of the individual measurements.

The useful reading is that relative behavior changes across coding tests. For a business, what matters is which task truly resembles maintenance of its products, how much code the system needs to explore and how changes are accepted. The K3.1 Pro coding guide turns these questions into a configuration and checks on the work produced.

Does Kimi beat GPT and Claude on scientific benchmarks?

Not in the two panels reported here. The first concerns scientific workflows with tools; the second concerns knowledge and reasoning questions from the GPQA-Diamond dataset. The collection shows an Astra advantage on both, with very different gaps. The Terminal-Bench Science website describes the nature of the tasks; the values used here remain those of the Kimi compilation.

Terminal-Bench Science 0.1 and the DeepSeek exception

The 15.7 score for DeepSeek belongs to the earlier V4.1-Flash, not the 1017 variant in the general header. Kimi states that it obtained 9,3 in its own evaluation of 1017, but chose the earlier public value. We retain the displayed measurement and correct the variant name in the row: replacing it without explanation would alter the comparison. The distinction is explicit in the official Kimi notes.

This panel also shows why selecting only tests favorable to a newly released model is insufficient. If the need concerns operational scientific research, the published gap deserves a specific check. This does not mean an SME should discard Kimi for all technical work; it means task relevance comes before the model’s general reputation.

GPQA-Diamond: advanced knowledge and workflows are different

The scores are close compared with the previous test. The table does not describe how well an assistant can find an internal document, use specialized software or recognize a missing operational constraint, however. For this row, the published methodology does not establish complete parity of environments and budgets; we stick to the data in the Kimi source.

This is a useful reminder for people in technical roles: a correct answer to an advanced question and execution of a long assignment are related but distinct capabilities. A business test must include the path to the result, alongside the knowledge needed to explain it.

How does K3.1 Pro perform with agents and documents?

Agents’ Last Exam and OfficeQA Pro v2 bring the focus closer to work with software and documents, although they do not reproduce the processes of a specific Italian business. The first test uses operational assignments; the second requires work on a document corpus. Harnesses, materials and acceptance rules remain essential parts of the comparison. The Kimi publication describes the configuration.

Agents’ Last Exam ALL: what do the assignments evaluate?

The official ALE v1.1 version includes 122 active tasks, of which 86 CLI, according to the Agents’ Last Exam repository. Kimi states that environments are aligned with the official release and assigns zero to attempts that bypass instructions, for example retrieving an answer instead of performing the requested analysis. K3.1 Pro uses Kimi Code; DeepSeek uses Claude Code. These choices are described in the Kimi notes.

Kimi has the highest value among those reported, but the Fable result is not published. Its absence prevents a complete comparison on this row. For an SME, the interesting point is different: an agent must respect the required path and assignment limits. Reaching plausible text through a prohibited shortcut can constitute process failure.

OfficeQA Pro v2: why does PDF preparation matter?

Kimi describes an evaluation in which the entire PDF corpus is rendered as images without machine-readable text. Astra uses Codex and the other models Claude Code. Attempts retrieving reference answers online receive zero, according to the official Kimi methodology. These are more precise conditions than the simple phrase “reads documents well.”

In a business, the result suggests a candidate for tests with scans and visual materials. It does not demonstrate knowledge of the company’s archive or the correctness of all citations. The link between answer, page and extracted data must be verified. The guide to K3.1 Pro multimodal agents addresses this transition from benchmark to a working system.

What does the BenchCAD comparison actually measure?

BenchCAD evaluates geometric reconstructions, with and without tools. The metric shown is Voxel IoU, a measure of overlap between volumes represented through voxels. The percentage score expresses geometric similarity according to the pipeline, not the proportion of completed designs, technical compliance or safety for industrial use. Details are in the BenchCAD protocol reported by Kimi.

BenchCAD with tools: why is the model name not enough?

The Kimi protocol for open models uses a random subset of 1,000 cases drawn from the 17900 in Vision2Code. In the tools-enabled panel it uses Claude Code; Claude values come from the Opus publication and Astra’s from the OpenAI report, as explained in the Kimi notes. This is why samples and sources must also remain visible.

A workflow with tools can allow intermediate results to be examined and corrected. The final score describes the system in that mode. It is not correct to attribute it to a single response obtained without the same tools, or to treat a small gap between two values as a guarantee of quality for the company’s next drawing.

BenchCAD without tools: what remains comparable?

The no-tools mode uses a single generation with official prompts and pipeline for the evaluations described by Kimi. Astra and GLM have no value shown in this panel. The source therefore allows comparison only of the available results, without deriving a complete ranking of the six models.

The gap between modes with and without tools is a reminder of how much system design matters. It does not, however, allow the benefit of tools to be calculated automatically for every model: the rows do not constitute an isolated, uniform experiment. In a technical office, verification must include requirements, constraints and checks that geometric similarity does not represent.

Why do some figures differ from other developers’ releases?

Publications may use different snapshots, harnesses or selection criteria. For Astra, Kimi reports 58,2 on Terminal-Bench 4.0 and 68,1 on Science; the OpenAI GPT-6 Astra release consulted presents, respectively, 57,9 and 64,6. We retain the values from the Kimi compilation in the charts, stating this difference rather than mixing the collections.

Choosing a preferred figure from a different publication for every model can construct a comparison no one has ever performed. Correct editorial work retains an identifiable snapshot and explains where it differs from other sources. When a protocol is insufficiently documented, the conclusion must remain limited to what can be observed.

Which results prevent an overall winner from being declared?

Even outside the eight selected panels, there are results less favorable to Kimi. In Program Bench with binary scoring the table publishes 31.5 for K3.1 Pro, 50.0 for Astra and 65.0 for Opus. Kimi assigns success only when the task’s original score reaches the threshold of 95%: these values are not the original ProgramBench score. The source and rule are in the Kimi publication.

The chart selection makes task families readable, rather than hiding unfavorable tests. Changing the activity or success criterion can change the order. For a business, knowing where the candidate needs to be tested is more useful than receiving a general verdict derived from an arbitrary aggregate.

First hypothetical scenario: comparing two software assistants

An Italian software company maintains business management software with specific operating rules. It wants to understand whether a new model helps resolve reports without introducing unrelated changes. Terminal-Bench and SWE Marathon suggest an area to investigate, but do not measure that repository. The scenario is hypothetical and does not describe a test conducted by Giallo.

The team can choose known defects, prepare an isolated environment and give both candidates the same materials. Acceptance criteria come before the test: reproduce the behavior, propose a narrowly scoped correction and show a relevant check. If the system changes requirements or removes a check to obtain a green result, the task must not be considered solved.

The evaluation also records the human intervention required and attempts that are not accepted. The benchmark score helps decide which candidates to invite to the test; the final choice depends on the changes the team is willing to approve. If you already use K3, the K3.1 Pro migration guide lets you organize the comparison against an existing reference.

Second hypothetical scenario: verifying information in a PDF archive

A sales office prepares internal briefs from specifications, attachments and revisions it receives. OfficeQA results make an evaluation with visual documents interesting, but do not demonstrate that the model can distinguish every version of the business material. This too is a design scenario, with no success rates or savings attributed to the product.

A useful sample includes a straightforward request, documents with conflicting information and a case where the requested data are missing. An acceptable answer identifies the document and passage supporting every important point. When the sources do not allow a conclusion, it must leave the question open. The ability to stop is part of quality, not a response to penalize for being less complete.

The person responsible checks correctness, source links and the work needed to reach an accepted brief. Any delivery to the customer remains a separate step. In this way, the test measures the office’s real need and makes errors visible that a general score on documents could not describe.

How do you prepare a repeatable business evaluation?

Define the acceptable result first and retain a fixed version of the sample. Record the channel, model, effort, tools and instructions used. The Kimi API default is max, while Kimi Code specifies high in its detailed configuration: for a comparable test, the setting must be explicit, according to the API quickstart and Code documentation.

A minimal record for reading each result

Assignment: describe the input, required output and constraints that cannot be bypassed.

System: retain the model, variant, channel, effort, tools, instructions and version of the materials.

Acceptance: record the check that demonstrates correctness and the errors that make the result unusable.

Total cost: observe usage, tools, waiting time, attempts and human review through to the accepted result.

For costs, see K3.1 Pro pricing and APIs, keeping the price list separate from observed usage. A model with a lower cost per token may require more attempts; a faster one may leave more work for the reviewer. Without observing the accepted result, the economic comparison remains incomplete.

Which mistakes make an LLM comparison useless?

The first is turning N/R into zero, producing a ranking that penalizes a model without a measurement. The second is forgetting the DeepSeek Science variant or the with-fallback label on proprietary-model columns. The third is adding different metrics, such as task scores and Voxel IoU, simply because both can appear on a scale from zero to one hundred.

The fourth mistake is comparing different configurations without recording them, attributing the advantage of extra tools or budget to the model. The fifth is inferring productivity and return on investment from a public test that does not measure that process. The common correction is to retain context, protocol and acceptance criteria alongside the figure.

What decision should you make after reading the charts?

The data help select relevant tests: software maintenance, assignments with tools, visual documents and geometric reconstruction require different checks. K3.1 Pro deserves attention in more than one of these areas, but the collection’s limits remain part of the decision. A useful next step is a small, repeatable evaluation that can also record unacceptable results.

To understand the product beyond the figures, return to the Kimi K3.1 Pro guide. If you want to move from rankings to a sample reflecting your company’s documents, tools and checks, we can design an AI evaluation of your business process.