Time to first token: what it is and why it decides the perception of speed

The time to first token — often abbreviated TTFT, from English time to first token è is the time that passes between sending the request and the appearance of the first piece of reply. It is the time when the user looks at a still screen, without knowing if something is happening.

It is a metric different from the overall speed, and for the experience of use it is almost always the most important of the two. A system that begins to respond immediately and then proceeds calmly is perceived as fast; one that thinks long and then delivers everything together is perceived as slow, even when it ends first.

Understanding it serves two concrete things: choosing the right model for an interface used by people, and knowing where to intervene when users complain that "AI is slow."

The two metrics that should not be confused

When we talk about the speed of a model, we mix two numbers that behave differently.

The sum of the two gives the total time, but the two entries They don't optimize in the same way and do not weigh in the same way. For a short answer dominates the time to the first token. For a long document it also weighs the generation speed, because the text has to finish exiting.

There is a third number, often decisive in practice: the variability. A service that answers usually in a second but every now and then in ten is perceived as unreliable more than a constantly a little slow.

Why it decides the perception of speed

The answers appear progressively, word after word. This completely changes the psychology of waiting: as soon as the first piece arrives, the user start reading, and from that moment the generation continues while he is busy.

The result is counterintuitive: between two systems, the one that begins to respond first seems faster even if it takes longer overall. The wait that really weighs is just the initial one, because it is the only one where the user has nothing to do.

What Makes It Worse

In order of how much it weighs in practice.

1. The length of what you send. Before producing the first word the model must process the whole prompt. A very long context is paid on hold, as well as in money.
2. The level of reasoning required. If the model is set to reason for a long time before responding, that time adds up to the initial wait.
3. The queue of the service. When the supplier is loaded, the request waits before it is even processed. It is the most frequent cause of abnormal peaks.
4. The server distance. The data journey adds fixed time: it matters especially when everything else is already fast.
5. The intermediate steps. In a system that first searches for documents and then questions the model, the perceived expectation also includes the search.

The most useful point is the first: the number one cause of perceived slowness is too long a prompt, not an unpowerful model. It is also why filling the context window "for safety" is a bad idea twice — see What Are Tokens.

How much does the context affect, in practice

It is worth fixing the mechanism, because it is the main lever and you understand it in thirty seconds.

Before writing the first word the model must read everything you sent him. This reading phase grows with the amount of text: the more material you attach, the more time it takes before something comes out. The next step — producing the response — instead proceeds at a fairly constant pace.

This results in a useful practical rule: shortening prompt improves initial wait, shortening response no. If the user complains about the time before something appears, asking for shorter answers is useless: you have to look at what is sent.

The last line is the one that surprises the most: the best model is often the slowest to start, because it processes more. On a simple task you get a worse experience by paying more — one of the reasons why it suits use different models for different tasks.

How it is reduced

The cache deserves a separate note: when much of the prompt is identical to each call —system instructions, business glossary, examples —reuse it instead of reworking it reduces both the wait and the cost. It is the single optimization with the best ratio of fatigue to result in almost all systems in production.

When it doesn't count

In an automatic process that runs at night on three thousand documents, Nobody's watching. There the time to the first token is irrelevant and count two other things: the cost per task and the total time of the lot.

It is a distinction that is worth making early, because it leads to opposite choices: for the interactive interface a reactive model is convenient, for the night lot it is the cheapest that exceeds the quality threshold. Often the right answer is use both, each where it renders — see when an economic model is enough.

How it is measured

It takes little, but it needs to be done on the real data.

• Record the instant of sending and the instant of the first piece received, for every request
• Do it on Real Traffic, not on a test request with a short prompt
• Look at the typical value and the worst cases: the average hides just those that make users angry
• Measure by type of request: those with a lot of context behave differently
• Re-measure at different times: the supplier's queue varies in the day

A frequent mistake is to measure only once, in the afternoon, with a short prompt, and conclude that the system is fast. Users will experience anything else. For comparison between models see the latency of AI models.

Practical examples

E-commerce with assistant on site. The assistant responded well but "looked slow" and users abandoned the chat. Measuring it, it was seen that with each question were attached the entire product catalog and a long history of the conversation. Reducing the context to only the relevant products and caching the fixed instructions cut the initial wait, without changing the model or making the answers worse.

Professional study, document archive research. The perceived wait was high because before the response the system searched in the documents, and during the search the interface showed nothing. The most effective correction was not technical: it was show now that the search was ongoing and bring up the answer progressively. The real time has changed little, the complaints are gone.

Mistakes to Avoid

• Confusing total speed and initial wait. You optimize the wrong thing.
• Not enabling progressive display. Throw away the main advantage.
• Filling in the context "for safety". You pay twice, on hold and in cash.
• Using the maximum level of reasoning anywhere. On simple tasks is expected gifted.
• Measuring only the average. The worst cases are the ones that generate the complaints.
• Applying the same choices to interface and batch processes. They have opposite goals.

How to apply it in the company

If you are evaluating an AI system that will be used by people —customers or colleagues —put time at the first token among the criteria of choice, alongside quality and cost.

1. Distinguish Uses: interactive interface or automatic processing. The criteria are different.
2. Ask the supplier for a measure on real traffic, with typical value and worst cases, not a brochure number.
3. Check that the interface shows the answer progressively. It's free and it changes perception.
4. Checks if the system prompt is cached. If it isn't, you're paying wait and money on every call.

These are four questions that anyone can ask without technical expertise, and that shift the experience of use more than many seemingly more important choices. As it integrates everything with existing systems see the integration of AI.

Conclusion

The time to the first token is the metric that decides whether an AI system looks fast. It almost never depends on the power of the model: it depends on how much you make it read before answering, how much reasoning you ask it and how the interface handles the wait.

Before changing the model because "it's slow", it measures where you waste time. In most cases the answer is too big a context, and it resolves without spending anything.

AI becomes useful when it enters processes. Giallo Studio helps SMEs and teams to build concrete, measurable and sustainable automations.