Skip to content
Giallo

Chain of thought: what it means when a model reasons step by step

Chain of thought makes a model write intermediate steps before answering. How it works, when it helps, its limits and how to use it in an SME.

Editorial team: Giallo Studio

Published on

Updated on

Reading time: 8 min

Diagram of a problem broken into intermediate steps that lead to a final answer

Chain of thought is a prompting technique that leads an AI model to break a problem into intermediate steps before giving the final answer. Instead of jumping straight to the conclusion, the model writes out the steps before answering. Google Research describes it this way: the model produces intermediate reasoning steps before answering a multi-step problem (Google Research).

In a clear definition: chain of thought is not a new model and not software to install. It is a way of asking. With standard prompting you ask for the direct answer; with chain of thought you ask for, or show through examples, a path that goes through intermediate steps.

It should, however, be read with care. The same sources that show its advantages warn that the explanation written by the model does not always match what the model actually did. In this guide we look at the idea, the mechanism, the limits and a cautious way to use it in a small company or a professional practice.

In summary

  • Chain of thought makes the model produce intermediate steps before the final answer, instead of asking for the result right away.
  • It does not require a large training dataset or changes to the weights: examples included in the prompt are enough, according to Google Research.
  • The advantages shown by the sources concern large-scale models and multi-step problems, such as calculations and common-sense reasoning.
  • Anthropic warns that written reasoning can be convincing even when it does not reflect the model's real steps.
  • For an SME it is a tool to try on real cases and to check, not a guarantee of a correct answer.

What is chain of thought, in simple terms?

The idea is simple. Faced with a multi-step problem, a model can answer in one go or write the steps first. Google Research explains that with standard prompting the model is asked to give the answer directly, while with chain of thought it is induced to break the problem into intermediate steps, in one case leading to a correct final answer (Google Research).

An analogy, flagged as such: it is the difference between telling a colleague only the total of a quote and handing over the sheet with the sum laid out in columns. The sheet does not make the calculation automatically right, but it lets the reader follow where it went.

The same source adds that the idea is to have the model generate a chain of thought that imitates an intuitive process when tackling a multi-step problem (Google Research). The verb matters: imitating a process does not mean the process is identical to the human one.

How does it work compared with a normal prompt?

The mechanism, as Google Research describes it, is this: the prompt includes a few examples in which the answer is preceded by the steps, and the model is induced to produce similar steps for the new problem. The authors specify that the chain-of-thought examples included in the prompt were composed by hand (Google Research).

One important practical point: the source distinguishes this approach from fine-tuning. Producing a thought process was already possible with fine-tuning, but here it is obtained with prompting alone, without a large training dataset and without modifying the model's weights (Google Research). If you want to understand when fine-tuning is really needed, see the guide on when fine-tuning is needed.

Appearance Standard prompting Chain of thought Source
What you ask for The direct answer Intermediate steps, then the answer Google Research
How do you get A simple question Examples with the steps in the prompt Google Research
Training Nothing specific No large dataset, weights unchanged Google Research
Effect with larger models Rather flat performance curves Performance that grows with scale Google Research

For those starting from scratch with writing requests, a good complement is the guide on how to get started with prompt engineering.

Why do the results concern large models?

The sources provided say so explicitly. Google Research writes that, with chain of thought, language models of sufficient scale, around 100B parameters, can solve complex reasoning problems that standard prompting methods cannot solve (Google Research).

On two benchmarks, according to the same page, standard prompting gives relatively flat growth curves: increasing the model's scale does not substantially improve performance. With chain of thought, by contrast, scale leads to improvements that clearly surpass standard prompting for large models (Google Research).

Mind the context. Those results are historical, described by a research publication. The sources provided do not say whether they hold in the same way for today's models, so they should not be turned into promises about your tool. To choose the right model for a task, it is better to start from how to read AI benchmarks and from how to choose an AI model.

Is the reasoning written by the model the real one?

Not always, and this is the part that matters most to anyone working with AI. Anthropic, in its work to observe what happens inside a model, explains that models learn their own strategies during training, encoded in the billions of operations they perform for every word they write, and that we therefore do not understand how they do most of the things they do (Anthropic).

Hence the question posed by the source: when Claude writes out its reasoning step by step, does that explanation represent the real steps, or is it sometimes a plausible fiction for a conclusion already decided? (Anthropic). The answer is nuanced: extended thinking often produces better answers, but sometimes the chain turns out to be misleading and the model invents plausible steps to get where it wants (Anthropic).

The source also shows the positive case. When asked to compute the square root of 0.64, Claude produces a faithful chain, with internal elements representing the intermediate step of computing the root of 64 (Anthropic). So chain of thought can be faithful: the point is that it is not necessarily so.

From a reliability standpoint, Anthropic notes that Claude's "fake" reasoning can be very convincing, and that sometimes the model builds a plausible argument designed to agree with the user rather than to follow logical steps (Anthropic). A tidy, well-written text is not proof of correctness.

The same source recalls a general limit: by talking to a model you can learn only so much, because not even human beings, neuroscientists included, know all the details of how their own brain works (Anthropic). On the broader topic of model errors, also read AI hallucinations and how to avoid them.

What changes with reasoning models?

A related topic is the so-called reasoning models. Epoch AI observes that, with the arrival of these models, citing OpenAI's o1 released in December 2024, it has become more problematic to ignore the distinction between the cost of the model and the cost of thinking (Epoch AI).

Also according to Epoch AI, reasoning models can productively consume many more tokens, but as a result they get good performance from a smaller base model that is cheaper to run per token (Epoch AI). Translated for anyone paying for an API: more written steps mean more tokens. If the term is new to you, you will find the basics in what are AI tokens.

The topic also appears in the settings of commercial models. For an example of how reasoning effort is adjusted, see reasoning effort and what changes in comparisons. For a concrete case of a reasoning-oriented model, there is the article on a reasoning model and its practical limits.

How might an SME use it?

The sources provided do not cover specific business processes, so the examples that follow are working hypotheses, not measured results and not client cases. The common thread is the one indicated by the research: the technique applies to tasks that can be solved with language and that require multiple steps (Google Research).

Example: a window and door fitting company. An owner needs to work out how to arrive at a quote starting from a request full of measurements, options and installation conditions. They ask the assistant to list the items first, then the dependencies between them, and finally the total. The advantage is that the owner can check every line: if an item is missing, they see it before sending the offer.

Example: an accounting firm. A staff member asks a model to explain step by step how it reconciled two lists of transactions. The steps serve as a trail to compare against the documents. If a step finds no match, the result is not used. Here chain of thought is an aid to review, not a substitute.

Example: an online shop. Whoever handles returns asks the model to check, in sequence, the conditions written in the shop's own policy before proposing a reply to the customer. The final text still goes through a person.

For ideas on where to bring AI into processes, also see the guide on what to automate in an SME.

Which mistakes should you avoid?

There are five, and almost all of them stem from confusing a tidy text with a correct one.

Believing the steps are proof of the result

It happens because a linear explanation inspires confidence. Anthropic warns that fabricated reasoning can be very convincing (Anthropic). To avoid it, check the result against the source data, not just the consistency of the text.

Expecting the same effects from any model

Google's research links the improvements to models of sufficient scale, around 100B parameters (Google Research). A smaller model may not benefit in the same way. Always test on the tool you will actually use.

Using it on tasks that do not need it

If the question is simple, writing extra steps lengthens the answer and, with reasoning models, can consume more tokens and therefore cost more (Epoch AI). Reserve it for multi-step tasks.

Taking for granted that the model is not trying to please

Anthropic points out that the model can offer a plausible argument designed to agree with the user (Anthropic). As a precaution, phrase the request neutrally and ask it to also point out whatever contradicts the hypothesis.

Not testing on real cases before use

The sources do not say whether the historical results hold for today's models in your process. Prepare a small set of cases whose answer you already know and measure. The guide on verifying AI answers before using them offers a working checklist.

How do you apply it, step by step?

A cautious procedure, derived from the limits described in the sources and therefore to be read as a working guideline, not a validated method:

  1. Choose a multi-step task with an answer you can verify: a calculation, a comparison of conditions, a sequence of checks.
  2. Write the request asking it to set out the intermediate steps before the final answer, or include one or more worked examples with their steps.
  3. Compare the result with the known answer and read the steps one by one, looking for those that find no match in the data.
  4. Repeat on more cases, including awkward ones, and note where the chain is useful and where it misleads.
  5. Decide the rule of use: where chain of thought helps, where a direct answer is enough, where a person must always verify.

Before starting, it helps to have a few conditions under control:

  • A chosen task with an answer a person can verify.
  • A small set of test cases with an already known outcome.
  • A clear rule on who checks the result before use.
  • Attention to costs, if the tool charges per token.

So, is it worth using?

Yes, as a tool to verify case by case. It can improve some answers on multi-step tasks without retraining the model, and it makes the path to the result more readable. It does not, however, guarantee that the reasoning shown is the real one, and with reasoning models it can push token consumption up.

For a company, the sensible step is a small trial on a concrete process, with a person checking the outcome. If you want to set it up with someone who helps you choose the case, the checks and the tool, you can start from our AI consulting.

Giallo Studio guides: methodology and sources

Want to apply this to your business?In 30 minutes, free of charge: the first process to automate and its estimated return, in writing.

Book the free audit

How we apply it in the company

Related resources

AI consulting services and process automation

FAQ

What is chain of thought in one sentence?

It is a prompting technique in which the model writes out the intermediate steps of a problem before giving the final answer. You get it by asking for it in the prompt or by showing examples with the steps. It does not require retraining the model.

Does chain of thought guarantee correct answers?

No. It can improve some answers on multi-step tasks, but the written steps can be plausible without reflecting what the model actually did. Always check the result against the source data.

Do I need to train or modify the model to use it?

No. According to Google Research, it is enough to include a few examples in the prompt. You do not need a large training dataset or any changes to the model's weights.

Does it work with all AI models?

The sources provided link the improvements to large-scale models. They do not say whether the historical results hold for every current model, so it is best to test it on the tool you actually use.

Does chain of thought cost more?

It can, if your tool charges per token. Epoch AI notes that reasoning models can consume many more tokens, so writing more steps tends to increase consumption.

Can an SME use it without technical skills?

Yes, because it is a way of writing the request. You do need a small test on real cases and a person who checks the results before they are used in day-to-day work.

Put it into practice

Let's turn this guide into your first live workflow.

Tell us what process you want to lighten: we evaluate feasibility, return and first operational step.