How to choose an AI model: latency, context window, accuracy and cost per token
Every few months, a new model tops the rankings. Naturally, you want to use it for everything. But if you move every task in your system to the most powerful model available, your costs will most likely rise, responses will take longer and quality will barely improve.
In the previous chapter, you saw how to divide work among specialised agents. Now it's time to decide which engine each one needs. You'll see why there is no “best model” for everything, which four criteria to consider and what type of model fits each role. You'll also see how to protect your workflows against provider changes.
There is no “best model”
With each new generation of models, public leaderboards publish their results on standardised tests such as MMLU, HumanEval, GSM8K or MATH. Many teams respond by sending all their requests to whichever model is currently in first place.
There are three problems with this approach.
1. The tests don't reflect your real work
Some questions from these tests may have accidentally ended up in the models' own training data. This is known as benchmark contamination. When it happens, the score rises, but that doesn't mean the model performs better with your documents, emails or contracts.
What's more, even without contamination, a general test measures general capabilities. Your system needs to solve specific tasks.
2. You overpay for simple tasks
Using a model with the highest capabilities to classify an email, extract a date or generate JSON is like hiring a surgeon to apply a plaster. It works, but the cost eats into the margin of any service you want to scale.
3. You wait longer than necessary
Larger models, especially those that reason step by step before answering, tend to take longer. In a customer support chat or a workflow with several agents running in sequence, that delay is noticeable.
The key idea: there is no best model overall. There is a suitable model for each step in your workflow.
The balance between reasoning, speed and cost
You can almost never have maximum reasoning capability, maximum speed and minimum cost all at once. Choosing a model means deciding what to prioritise at each step.
The four criteria for choosing a model
1. What type of task it needs to solve
Before looking at models, classify the task:
Task type
Examples
Suitable model
Complex reasoning
Solving difficult logic problems, detecting subtle contradictions in a contract, conducting an expert analysis, redesigning complex software
Frontier models with explicit reasoning or high capabilities
Text transformation
Summarising clear texts, changing the tone, translating, drafting variants from a template
Mid-range models
Mechanical operations and extraction
Classifying, routing, extracting names, amounts or tax identification numbers (NIF), formatting JSON with a fixed structure
Small language models (SLM)
Remember what we covered in chapter 2: no model can reliably perform exact calculations using text alone. For that, connect a calculation tool.
2. The context window and how well the model uses it
Some models accept one or two million tokens of context. But a model's ability to accept a lot of information doesn't mean it uses all of it well.
A study published in 2023 by Nelson F. Liu and other researchers described the “lost in the middle” phenomenon. Models tended to make better use of information at the beginning and end of the context, and poorer use of information in the middle. More recent models have improved in this respect, but you shouldn't assume they read every page of a very long document equally well.
A practical guideline: when working with extensive technical or legal documentation, it's usually better to use a RAG architecture with short, precise chunks, as we saw in the chapter on Brains, than to dump an entire 500-page file into a giant model's context window. It's also cheaper: you pay for every token you send.
A simple way to check this is the “needle in a haystack” test. It involves hiding a specific piece of information in a long document and asking about it. If the model can't find it when it's in the middle, you have a clear signal.
3. Latency
There are two measures worth distinguishing:
- Time to first token (TTFT): how long it takes for the first character of the response to appear. This is what people notice most in a chat or a customer-facing assistant.
- Tokens per second (TPS): the speed at which the rest of the text is generated. This is crucial in workflows with several agents running in sequence, because the second agent can't start until the first one finishes.
Example: in a workflow with five agents running in sequence, if each takes eight seconds, the user waits forty. If you move the first three steps to fast models that take one second each, the wait drops to around nineteen.
4. Cost per token
Providers charge separately for two types of tokens:
- Input tokens: everything you send to the model (instructions, documents, history). These are usually cheaper.
- Output tokens: what the model generates. These usually cost several times more than input tokens; often between 3 and 5 times more, though it depends on the provider and the model.
The basic formula is:
Cost of a request = (input tokens × input price) + (output tokens × output price)
Prices are usually quoted per million tokens.
An example with fictional prices
Imagine an extraction agent that processes 100 case files a day. Each case file involves 8.000 input tokens and 1.000 output tokens. We'll compare two models with made-up prices, just to illustrate the calculation:
Frontier model
Small model
Input price (per million)
3,00 €
0,15 €
Output price (per million)
15,00 €
0,60 €
Cost per case file
0,039 €
0,0018 €
Monthly cost (30 days)
117 €
5,40 €
In this example, the large model costs around 20 times more for the same extraction. And if the small model already extracts the data correctly, that difference does nothing to improve the final result. Now multiply that across the five agents in a workflow.
Always check current prices on each provider's website, as they change frequently.
How to test models with your own cases
Instead of relying on public rankings, create your own test set:
- Gather between 20 and 50 real examples of the task, including some difficult or unusual ones. Remove personal data.
- Define what a correct response looks like for each case before testing any model.
- Test the smallest model first. If it handles the task well, you don't need a larger one.
- Move up a level only if necessary, and measure whether the improvement justifies the cost and the wait.
- Repeat the test whenever you change the model, version or system instruction.
This way, you choose based on data from your business, not general promises.
Protect your workflows: pinned versions and fallback models
Choosing the right model isn't enough. You also need to prevent your system's behaviour from changing without you noticing.
Use pinned versions
Many providers offer generic names, such as modelo-latest, that always point to the latest version. They're convenient for testing, but risky in production. When the provider updates the model, your agent may start following instructions differently. This is known as instruction drift.
Best practice: in your business workflows, always use the specific dated version offered by the provider, for example, nombre-modelo-2026-03. When you want to update, do it deliberately and only after it has passed your tests.
Also bear in mind that providers retire older models from time to time. Review their notices and plan the switch well in advance.
Have a fallback model
Define a compatible alternative model for each critical step. You'll use it if the primary model fails, the service is interrupted or you exceed the provider's usage limits. These limits are usually measured in tokens per minute (TPM) and requests per minute (RPM).
Test the fallback model with the same set of cases as the primary model. If the output format changes or quality drops significantly, it's not a real fallback.
If you choose models from another provider as fallbacks, also review their data processing terms. This is the data sovereignty we covered in chapter 1.
Quick checklist for choosing a model for each agent
- I know what type of task it handles: reasoning, transformation or extraction.
- I've tested the smallest model that might work first.
- I have a set of real cases for comparing models.
- I know how much information it needs to read and whether using RAG makes sense.
- I know the maximum acceptable wait for that step.
- I've calculated the monthly cost using the provider's current prices.
- I use a pinned version, not a generic name like latest.
- I have a tested fallback model.
Conclusion
Choosing a model isn't about finding the most powerful one, but the most suitable one for each step. A small model can classify and extract data quickly and at very low cost. A frontier model adds value where in-depth reasoning is needed, such as risk reviews. If you assess each step by task type, context, latency and cost, test it with your own cases and pin versions with a fallback ready, you'll build faster, cheaper and more stable systems.
Want to choose the right model for each of your agents? Visit libertadia.com and sign up in the app.